Multi‐stream densely connected network for semantic segmentation

IET Computer VisionVolume 16, Issue 2 p. 180-191 ORIGINAL RESEARCH PAPEROpen Access Multi-stream densely connected network for semantic segmentation Dayu Jia, Dayu Jia School of Electrical and Information Engineering, Tianjin University, Tianjin, ChinaSearch for more papers by this authorJiale Cao, Jiale Cao School of Electrical and Information Engineering, Tianjin University, Tianjin, ChinaSearch for more papers by this authorJing Pan, Jing Pan School of Electronic Engineering, Tianjin University of Technology and Education, Tianjin, ChinaSearch for more papers by this authorYanwei Pang, Corresponding Author Yanwei Pang [email protected] orcid.org/0000-0001-6670-3727 School of Electrical and Information Engineering, Tianjin University, Tianjin, China Correspondence Yanwei Pang, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China. Email: [email protected]Search for more papers by this author Dayu Jia, Dayu Jia School of Electrical and Information Engineering, Tianjin University, Tianjin, ChinaSearch for more papers by this authorJiale Cao, Jiale Cao School of Electrical and Information Engineering, Tianjin University, Tianjin, ChinaSearch for more papers by this authorJing Pan, Jing Pan School of Electronic Engineering, Tianjin University of Technology and Education, Tianjin, ChinaSearch for more papers by this authorYanwei Pang, Corresponding Author Yanwei Pang [email protected] orcid.org/0000-0001-6670-3727 School of Electrical and Information Engineering, Tianjin University, Tianjin, China Correspondence Yanwei Pang, School of Electrical and Information Engineering, Tianjin University, Tianjin 300072, China. Email: [email protected]Search for more papers by this author First published: 02 December 2021 https://doi.org/10.1049/cvi2.12078Citations: 2AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Abstract Semantic segmentation is a challenging task in computer vision which is widely used in autonomous driving and scene understanding. State-of-the-art semantic segmentation networks, like DeepLab and PSPNet, make full use of multiple feature information to improve spatial resolution. However, the feature resolution in the scale-axis is not dense enough for practical applications. To tackle this problem, a multi-stream network is designed with atrous convolutional layers at multiple rates to capture objects and context at multiple scales. Furthermore, intra-connections and inter-connections are designed to fuse multi-scale features densely which produce a feature pyramid with much larger scale diversity and larger receptive field by involving small quantity of computation. The proposed module can be easily used in other methods and it helps to increase the performance. Compared with existing methods, the proposed network, called Multi-stream Densely Connected Network, reaches competitive results on ADE20K dataset, PASCAL VOC 2012 dataset, and Cityscapes dataset. 1 INTRODUCTION Semantic segmentation has become one of the most promising topics in computer vision. In semantic segmentation tasks, each pixel of an image is labelled with the class of its enclosing object or region [1-7]. This topic is of broad interest for potential applications of automatic driving and scene understanding. With the help of Deep Convolutional Neural Networks (DCNN) [8], information with different receptive fields is able to be captured by stacking convolutional layers with non-linear and pooling units. So DCNN becomes the dominant kind of semantic segmentation methods. One of the difficulties in semantic segmentation is scene and label variety [4]. To conquer this difficulty, especially for complex scene understanding, contextual information should be taken into full account [1, 9, 10]. Figure 1 shows two examples of incorrect segmentation situations for lack of contextual information. If more contextual information is involved, the error can be avoided. FIGURE 1Open in figure viewerPowerPoint Visualization of the importance of contextual information for semantic segmentation. For example, the left is the images to be segmented and the right is the segmentation results. The first row indicates that part of the house is misclassified as train (yellow rectangle). The second row indicates that part of the skyscraper is misclassified caused by reflection (yellow rectangle; Visual examples from the result of DilatedNet [12] on ADE20K dataset [13]) In order to make full use of feature maps from multiple scales, DeepLab [3, 11] employs Atrous Spatial Pyramid Pooling (ASPP) to concatenate feature maps generated by atrous convolutional layers with different dilated rates. Atrous convolution [12] is widely used in Convolution Neural Network to achieve large receptive field without increasing the number of parameters and the amount of computation. ASPP encodes multi-scale information and eventually boosts performance. However, ASPP has only four atrous convolutional layers with dilated rates 6, 12, 18, and 24, making the receptive fields not dense. Another disadvantage of ASPP is that a large dilated rate has to be employed to achieve a large receptive field which is ineffective and it loses modelling power [10]. To solve these problems, we propose a Multi-stream Densely Connected Network (MDCN) which is shown in Figure 2. In the proposed network, a multi-stream structure is used to extract the features at different scales. Furthermore, intra-connection and inter-connection are used to make the receptive fields dense. FIGURE 2Open in figure viewerPowerPoint Classical frameworks for generating contextual information. SPP, Spatial Pyramid Pooling Our approach achieves competitive performance on ADE20k dataset [13], PASCAL VOC 2012 dataset [14], and Cityscapes dataset [15]. The experimental results show that the proposed method has advantages on both simple scene as well as high resolution and complex street scene. The two contributions of this study are summarised as follows: 1. We design a multi-stream network. Each stream contains several (e.g., three) atrous convolutional layers with different dilated rates to cover a large range of scales 2. We design intra-connections and inter-connections to make different scales of feature maps connected densely 3. The proposed module has advantage on generality. Our module can be used in other methods to fuse features in different scales 2 RELATED WORK The most successful state-of-the-art deep learning techniques for semantic segmentation stem from a common forerunner: the Fully Convolutional Network (FCN) [2]. FCN adapts contemporary classification networks (AlexNet [16], VGG net [17], and Xception [18]) into FCNs and transfers their learnt representations by fine-tuning to the segmentation task. It has been shown that contextual information plays an important role in classifying the pixels in FCN-based semantic segmentation [3, 4, 19]. There are three manners for FCN to explore multi-scale contextual information: Image Pyramid, Spatial Pyramid Pooling, and Encoder–Decoder. Image pyramid The structure of Image Pyramid is described in Figure 3a. In this structure, an image is resized into several different scales and the resized images are fed to the network as inputs independently. At the end of the network, features generated by different scales of inputs are fused to predict the results [20, 21]. The disadvantage of the model is that it does not scale well for larger/deeper DCNNs due to the limited GPU memory and thus it is usually applied during the inference stage [22]. FIGURE 3Open in figure viewerPowerPoint Overview of our proposed MDCN. Given an input image, we first use a pre-trained CNN to extract feature maps. We build a MDCEM, including three streams of atrous convolutional layers. To generate dense feature maps, we design intra-connections and inter-connections. At the end of MDCN, we fuse low-level features and the output of MDCEM together to predict the final results. CNN, Convolutional Neural Networks; MDCEM, Multi-stream Dense Context Encode Module; MDCN, Multi-stream Densely Connected Network Spatial pyramid pooling The structure of Spatial Pyramid Pooling (SPP) is described in Figure 3b. In this structure, the SPP layer [23] is added on the top of the last convolutional layer. The SPP layer pools the features and generates fixed-length outputs, which are then fed into the fully-connected layers (or other classifiers). PSPNet [4] proposed Pyramid Pooling Model to fuse the features under four different pyramid scales. Global pooling operation is used for the coarsest scale feature map to a single bin output. The following pyramid level separates the feature map into different sub-regions and forms pooled representation for different locations. DeepLab series [3, 11, 24-26] use atrous convolutional layers on the SPP architecture which is called ASPP. Encoder–Decoder The structure of an Encoder–Decoder is described in Figure 3c. There are two parts in this model: the encoder and the decoder. In the encoder part, the spatial dimension is reduced by pooling operation. In the decoder part, the spatial dimension is recovered by upsampling operation. SegNet [19] uses pooling indices computed in the max-pooling step of the corresponding encoder to perform non-linear upsampling. U-Net [27] consists of a contracting path to capture the context and a symmetric expanding path that enables precise localisation which gains extraordinary results in biomedical image segmentation. DeepLabV3+ [25] combines encoder–decoder and SPP together. Moreover, low-level features are introduced to the decoder for better segmentation results. Context Encoding Module [9] is proposed to capture the semantic context of scenes and selectively highlight class-dependent feature maps. BlitzNet [28] completes the detection and segmentation tasks together which also uses the encoder–decoder structure. Shortcut connection is widely used in DCNN [29]. In shortcut connection architecture, a few intermediate layers are directly connected to the auxiliary classifiers to address vanishing/exploding gradients. ResNet [30] and DenseNet [31] are representative networks of this architecture. Densely connected Atrous Spatial Pyramid Pooling Network (DASPP) [10] is one of the state-of-the-art semantic segmentation networks for street scenes. In DASPP, Densely connected Atrous Spatial Pyramid Pooling (DenseASPP) is used to generate different scales of features. The structure connects a set of atrous convolutional layers in a dense way so that it not only covers a larger scale range, but also cover that scale range densely. The structure does not increase the model size significantly. The proposed MDCN is the improved version of DASPP. MDCN shares the advantage of DenseNet and DASPP. The proposed method not only avoids vanishing/exploding gradients but also generates much more dense scales of feature maps compared with DASPP. 3 METHODOLOGY In this section, we first analyse some incorrect segmentation situations from which one can understand the importance of dense contextual information. In order to generate dense contextual information, we propose the Multi-stream Dense Context Encode Module (MDCEM). Finally, our proposed MDCN is introduced which is shown in Figure 2. Incorrect segmentation situations ADE20K dataset [13] contains 21,210 training images and 2000 validation images with 150 stuff/object category labels. The scene described in ADE20K consists of objects with different scales. So we choose ADE20K to support the claim that utilising contextual information with a large receptive field is very important in semantic segmentation. Figure 1 shows two typical incorrect segmentation results caused by a lack of enough contextual information. The first row indicates that some parts of the house are misclassified as a train (yellow rectangle). We can figure out that the house in the image has the similar appearance with a train. If enough contextual information is involved in training such as railways, the misclassification error can be avoided. The second row indicates that some parts of the skyscraper are misclassified as tree and sky (yellow rectangle). The misclassification is caused by reflection. If we only consider local information, it is difficult to distinguish the refection or the real object. Contextual information with large receptive field can solve the problem. To summarise the incorrect segmentation situation, the errors are related to the lack of enough contextual information. Thus, it is necessary to design a module to solve the problem. Multi-stream dense context encode module With the above analysis, we propose MDCEM which is the core structure of our network to improve the ability of generating contextual information with large and dense receptive fields. In order to make it easy to analyse how the proposed MDCEM model works, we choose three streams as examples in this section. 3.2.1 The module In MDCEM, a multi-stream structure is introduced to cover a large range of scales. Each of the streams has several, for example, three stacked 3 × 3 atrous convolutional layers with different dilated rates. We also design intra-connections (blue, orange and green arrows in Figure 2) and inter-connections (red arrows in Figure 2). With the connections, the inputs of an atrous convolutional layer are all the feature maps generated by prior atrous convolutional layers, which makes the scales of receptive fields dense. For the stride of atrous convolution in MDCEM is set to 1, the resolution of feature maps in MDCEM is the same. 3.2.2 The size of the receptive field Different dilated rates make the convolutional layers generate different sizes of the receptive field. For an atrous convolutional layer with the dilated rate d and the kernel size k, the size of the receptive filed can be defined as follows: R = ( d − 1 ) × ( k − 1 ) + k (1) If two atrous convolutional layers are stacked together, a larger receptive field is generated. For size of the receptive fields of two atrous convolutional layers R 1 and R 2 , the new size of receptive fields R n e w can be defined as follows: R n e w = R 1 + R 2 − 1 (2) For example, the largest receptive field in our MDCEM is generated by three stacked atrous convolutional layers with kernel size k = 3 and dilated rates d = [ 6 , 12 , 18 ] . The size of the equivalent receptive field R n e w = R d = 6 + R d = 12 + R d = 18 − 2 = 13 + 15 + 37 − 2 = 73. 3.2.3 The number of the receptive fields In Figure 4, the proposed model MDCEM is compared with ASPP and DASPP. The dilated rates of the three streams in our model are [2, 4, 8], [4, 8, 12] and [6, 12, 18] which are shown in Figure 2. In the proposed MDCEM model, each stream can generate 3 + 3 + 1 = 7 receptive fields and the inter-connections can generate 3 receptive fields. So the total number of receptive fields is 3 × 7 + 3 = 24 (yellow, red, blue, and green bars represent the receptive fields generated by stream 1, stream 2, stream 3, and inter-connections, respectively). We can figure out that the proposed model can generate more dense receptive fields compared with ASPP and DASPP. FIGURE 4Open in figure viewerPowerPoint Comparison of receptive fields generated by ASPP, DASPP, and the proposed MDCEM. The height represents the number of the receptive field and the width of bar represents the size of the receptive field. ASPP, Atrous Spatial Pyramid Pooling; DASPP, Densely connected Atrous Spatial Pyramid Pooling Network; MDCEM, Multi-stream Dense Context Encode Module Network architecture Based on the MDCEM model, we can construct the proposed architecture called MDCN. We adopt the Encoder–Decoder method as the overall framework. In DCNN, several convolutional layers with non-linear and batch normalisation operations are stacked together as the backbone network to generate the feature of an image. VGG [17], ResNet [30] and Xception [18] are widely used as backbone networks in DCNN. The proposed MDCEM is followed by a backbone network to generate dense contextual information. The backbone network and the proposed MDCEM model form the encoder structure of MDCN. In the proposed decoder structure, the output of the encoder structure is upsampled by a factor of 4 and is concatenated with the low-level features generated by DCNN at the bottom of the framework. The advantage of introducing low-level feature is that the details of images can be segmented accurately. It is noted that the number of channels of low-level features is much larger than that of the output of encoder structure. So we apply an 1 × 1 convolutional layer to reduce the number of channels. Then, we use several 3 × 3 convolutional layers to refine the feature maps. Finally, another upsampling layers by a factor of 4 is used to recover the full resolution. 4 EXPERIMENTAL RESULTS In order to prove the effectiveness, the proposed MDCN method is compared with FCN [2], DeepLab [3, 25], SegNet [19], DilatedNet [12], CascadeNet [13], PSPNet [4], DeconvNet [32], CRF-RNN [33], Piecewise [34], ResNet38 [35], EncNet [9], RefineNet [36], GCN [37], DUC [38], and DenseASPP [10]. It is noted that DenseASPP is a special case of the proposed MDCN method. MDCN is reduced to DenseASPP when only one stream is used. The experiments are conducted on PASCAL VOC 2012 [14], ADE20K [13], and Cityscapes [15]. The algorithm is implemented with Tensorflow [39]. Implementation details The network is trained on two Tesla P100 GPUs with 32 GB graphics memory per core for both validation and testing. For PASCAL VOC 2012, the 'poly' learning rate is employed. The network is trained with the initial learning rate 0.003, the crop size 513 × 513, the batch size 8 and fine-tuning batch normalisation parameters. The iteration number is set to 30K. Momentum and weight decay are set to 0.9 and 0.0005, respectively. In testing, after training on the training set with 30K iterations and the initial learning rate = 0.007, we finetune the model on the training set for another 30K iterations and the initial learning rate = 0.001. In order to get the best performance, the training batch size is set to 12 and Xception-65 is used as backbone. Other hyper parameters are the same as the validation. For ADE20K, the number and the category of images are much larger compared with PASCAL VOC 2012 and Cityscapes, so large training batch size and iterations are used. The 'poly' learning rate is employed. We use Xception-65 as backbone. The network is trained with the initial learning rate 0.007, crop size 513 × 513, and fine-tuning batch normalisation parameters. The iteration number is set to 150K and the training batch size is set to 12. Momentum and weight decay are set to 0.9 and 0.0005, respectively. For Cityscapes, we change some settings in training because the resolution of the images is larger compared with PASCAL VOC 2012. In validation, the 'poly' learning rate is employed. We use Xception-65 as backbone. The network is trained with the initial learning rate 0.007, crop size 769 × 769, fine-tuning batch normalisation parameters. The iteration number is set to 90K. Momentum and weight decay are set to 0.9 and 0.0005 respectively. In testing, we first train on the training set with 105K iterations and the initial learning rate = 0.007. Then we finetune the model on the training set with another 105K iterations and the initial learning rate = 0.001. In order to get the best performance, the training batch size is set to 8. Other hyper parameters are the same as the validation. PASCAL VOC 2012 The PASCAL VOC 2012 segmentation dataset [14] contains 20 foreground object categories and 1 background class. The original dataset contains 1464 (train), 1449 (val), and 1456 (test) pixel-level annotated images as well as 10,582 (training) augment training images. In this section, we use the training data to train our proposed network. 4.2.1 Validation results In order to prove the effectiveness of each improvement of our proposed model, we conduct experiments with several settings, including the backbone, the number of streams, adding intra-connections or not, adding inter-connections or not, and adding decoder or not. The results of validation are shown in Table 1. In order to keep the experiments fair, we only use pre-trained models on ImageNet [40] without any other training or evaluation tricks. The first row is the baseline DeepLabV2 with ResNet-50 as backbone which reaches 70.9% mIoU. The second row is the MDCN architecture with one stream which is the simple version of DenseASPP [10]. The second to fourth rows shows the effectiveness of multi-stream. We can figure out that the mIoU of MDCN increases with the number of stream increasing. In row five, we take out the inter-connections of MDCN and the mIoU decreased by 0.6%. The sixth row indicates that the performance of our model is enhanced if we change a complex backbone network. The seventh row is the framework shown in Figure 2 with Xception as backbone, involving intra-connections and inter-connections as well as decoder structure, which reaches 78.2% mIoU. If we use multi-scale evaluation, the result can reach 80.3% mIoU. Some visual results are shown in Figure 5. TABLE 1. Performance of MDCN on PASCAL VOC 2012 with different network settings Method Core module Backbone Number of stream Intra-connections Inter-connections Decoder mIoU (%) DeepLabV2 ASPP ResNet-50 4 × × × 70.9 MDCN(DASPP) MDCEM ResNet-50 1 ✓ × × 71.5 MDCN MDCEM ResNet-50 2 ✓ × × 72.1 MDCN MDCEM ResNet-50 3 ✓ ✓ × 73.6 MDCN MDCEM ResNet-50 3 ✓ × × 73.0 DeepLabV2 ASPP ResNet-101 4 × × × 75.1 MDCN MDCEM ResNet-101 3 ✓ ✓ × 76.6 MDCN MDCEM Xception-65 3 ✓ ✓ ✓ 78.2 MDCN(MS) MDCEM Xception-65 3 ✓ ✓ ✓ 80.3 Note: To make the comparison fair, we do not use any training tricks for all the methods. Abbreviations: ASPP, Atrous Spatial Pyramid Pooling; DASPP, Densely connected Atrous Spatial Pyramid Pooling Network; MDCEM, Multi-stream Dense Context Encode Module; MDCN, Multi-stream Densely Connected Network. FIGURE 5Open in figure viewerPowerPoint Visual results on PASCAL VOC 2012 dataset 4.2.2 Test results In order to compare with other models, we finetune our proposed model with the method described in Section 4.1. The testing results are shown in Table 2. It is noted that we do not pre-train on MS-COCO dataset [46]. Our MDCN achieves 83.3% mIoU which is better than all other methods (i.e., FCN, DeepLab, DeconvNet, CRF-RNN, Piecewise, ResNet38, PSPNet, and EncNet). TABLE 2. Performance comparison on PASCAL VOC 2012 testing set Method Aero Bike Bird Boat Bottle Bus Car Cat Chair Table Horse Mbike Person Plant Sheep Sofa Train mIoU FCN [2] 76.8 34.2 68.9 49.4 60.3 75.3 74.7 77.6 21.4 46.8 63.9 76.5 73.9 45.2 72.4 37.4 70.9 62.2 DeepLab [3] 84.4 54.5 81.5 63.6 65.9 85.1 79.1 83.4 30.7 59.8 76.1 83.2 80.8 59.7 82.2 50.4 73.1 71.6 DeconvNet [32] 89.9 39.3 79.7 63.9 68.2 87.4 81.2 86.1 28.5 62.0 80.3 83.6 80.2 58.8 83.4 54.3 80.7 72.5 CRF-RNN [33] 87.5 39.0 79.7 64.2 68.3 87.6 80.8 84.4 30.4 60.4 77.8 83.1 80.6 59.5 82.8 47.8 78.3 72.0 Piecewise [34] 90.6 37.6 80.0 67.8 74.4 92.0 85.2 86.2 39.1 58.9 83.9 84.3 84.8 62.1 83.2 58.2 80.8 75.3 ResNet38 [35] 94.4 72.9 94.9 68.8 78.4 90.6 90.0 92.1 40.1 71.7 93.7 91.0 89.1 71.3 90.7 61.3 87.7 82.5 PSPNet [4] 91.8 71.9 94.7 71.2 75.8 95.2 89.9 95.9 39.3 71.7 94.5 88.8 89.6 72.8 89.6 64.0 85.1 82.6 EncNet [9] 94.1 69.2 96.3 76.7 86.2 96.3 90.7 94.2 38.8 73.3 92.5 88.8 87.9 68.7 92.6 59.0 86.4 82.9 MDCN(ours) 95.5 73.7 84.1 74.2 80.6 94.4 91.1 95.5 43.4 68.4 90.9 92.0 90.1 72.6 93.0 65.0 90.1 83.3 Abbreviations: FCN, Fully Convolutional Network; MDCN, Multi-stream Densely Connected Network. ADE20K ADE20K dataset [13] is introduced in Section 3.1. Table 3 compares the proposed MDCN method with FCN [2], SegNet [19], DilatedNet [12], CascadeNet [13], and PSPNet [4]. We can figure out that our models which involve contextual information (PSPNet, MDCN) have great advantages compared with those that do not involve enough contextual information (FCN, SegNet, DilatedNet, CascadeNet). For involving more dense contextual information, our model achieves 44.06%, outperforming PSPNet by 0.25% and DenseASPP by 0.08%. TABLE 3. Performance of MDCN on ADE20K validation set Method mIoU (%) FCN [2] 29.39 SegNet [19] 21.64 DilatedNet [12] 32.31 CascadeNet [13] 34.90 PSPNet [4] 43.81 DenseASPP [10] 43.98 MDCN (ours) 44.06 Abbreviations: FCN, Fully Convolutional Network; MDCN, Multi-stream Densely Connected Network. Several examples are shown in Figure 6. For 'skyscrapter' in column one, DilatedNet treats some part as 'building' (yellow rectangle). For 'chair' in column two, our model has advantages in details of chair legs (yellow rectangle). Another interesting thing we find that the middle of the table is labelled incorrectly in ground truth while the segmentation results of our method are more realistic. For 'door' in column three, DilatedNet fails and mistakenly segments some part of the door as window whereas our method performs well (yellow rectangle). For 'tower' in column four, DilatedNet mis-segments the top part as 'building' while our model avoids the mistake (yellow rectangle). FIGURE 6Open in figure viewerPowerPoint Visual results on ADE20K dataset. The first row contains the original images. The second row contains the ground truth. The third row contains the results of DilatedNet [12]. The last row contains the results of our Multi-stream Densely Connected Network. The right column is the colourmap of ADE20K. The mistakes are marked as yellow rectangles Cityscapes The Cityscapes Dataset [15] focuses on the semantic understanding of urban street scenes. It comprises a large, diverse set of stereo video sequences recorded in streets from 50 different cities. The resolution of the street images in Cityscapes is 2048 × 1024. The 5000 street scene images are split into 2975 training images and 500 validation images with high-quality pixel-level annotations, as well as 1525 test images with annotations withheld and comparison with other methods is performed via a dedicated evaluation server. 4.4.1 Validation results In order to prove the effectiveness, the proposed MDCN method is compared with the state-of-the-art method DeepLabv3+ on the validation dataset of Cityscapes. To make the comparison fair, we carry out the experiments under the same conditions and do not use any specific training tricks. The results in terms of mIoU are provided in Table 4. From Table 4, it can be seen that the proposed method outperformed DeepLabV3+ by 0.9% on single scale evaluation and 2.6% mIoU on multiple scales evaluation. Some visual results are shown in Figure 7. It can be seen that the proposed MDCN has advantages in details. Even though the image contains a dense crowd (the situation in the second row), our method can get highly accurate segmentation results. TABLE 4. Performance of MDCN on Cityscapes validation set Method mIoU (%) DeepLabV3+ [25](baseline) 77.4 DenseASPP [10] 77.7 MDCN (single scale evaluation) 78.3 MDCN (multiple scales evaluation) 79.98 Abbreviations: ASPP, Atrous Spatial Pyramid Pooling; MDCN, Multi-stream Densely Connected Network. FIGURE 7Open in figure viewerPowerPoint Visual results on Cityscapes dataset 4.4.2 Test results On the test data of Cityscapes, the proposed MDCN is compared with eight methods. It is noted that we only use the fine annotated data and do not pre-train the data on MS-COCO dataset [46]. It can be found from Table 5 that the proposed method achieves the best segmentation result. In order to compare with state-of-the-art method, we replace the ASP Module in object-contextual representations network (OCRNet) [44] with the proposed MDCEM. It can be seen that the proposed method outperforms many recent state-of-the-art methods, such as CPNet [41], SFNet [42], LSTNet [43] and OCRNet [44]. TABLE 5. Performance comparison on Cityscapes testing set Method Road Swalk Build. Wall Fence Pole Tlight Sign Veg. Terrain Sky Person Rider Car Truck Bus Train mIoU CRF-RNN [33] 96.3 73.9 88.2 47.6 41.3 35.2 49.5 59.7 90.6 66.1 93.5 70.4 34.7 90.1 39.2 57.5 55.4 62.5 FCN [2] 97.4 78.4 89.2 34.9 44.2 47.4 60.1 65.0 91.4 69.3 93.9 77.1 51.4 92.6 35.3 48.6 46.5 65.3 Dilation10 [12] 97.6 79.2 89.9 37.3 47.6 53.2 58.6 65.2 91.8 69.4 93.7 78.9 55.0 93.3 45.5 53.4 47.7 67.1 DeepLab [3] 97.9 81.3 90.3 48.8 47.4 49.6 57.9 67.3 91.9 69.4 94.2 79.8 59.8 93.7 56.5 67.5 57.5 70.4 RefineNet [36] 98.2 83.3 91.3 47.8 50.4 56.1 66.9 71.3 92.3 70.3 94.8 80.9 63.3 94.5 64.6 76.1 64.3 73.6 GCN [37] - - - - - - - - - - - - - - - - - 76.9 DUC [38] 98.5 85.5 92.8 58.6 55.5 65 73.5 77.9 93.3 72 95.2 84.8 68.5 95.4 70.9 78.8 68.7 77.6 PSPNet [4] 98.6 86.2 92.9 50.8 58.8 64.0 75.6 79.0 93.4 72.3 95.4 86.5 71.3 95.9 68.2 79.5