Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation∗
Year: 2022

Abstract:
Semantic segmentation requires per-pixel prediction for a given image. Typically, the output resolution of a segmentation network is severely reduced due to the downsampling operations in the CNN backbone. Most previous methods employ upsampling decoders to recover the spatial resolution. Various decoders were designed in the literature. Here, we propose a novel decoder, termed dynamic neural representational decoder (NRD), which is simple yet significantly more efficient. As each location on the encoder’s output corresponds to a local patch of the semantic labels, in this work, we represent these local patches of labels with compact neural networks. This neural representation enables our decoder to leverage the smoothness prior in the semantic label space, and thus makes our decoder more efficient. Furthermore, these neural representations are dynamically generated and conditioned on the outputs of the encoder networks. The desired semantic labels can be efficiently decoded from the neural representations, resulting in high-resolution semantic segmentation predictions. We empirically show that our proposed decoder outperforms the decoder in DeeplabV3+ with only ∼ 30% computational complexity, and achieves competitive performance with the methods using dilated encoders with only ∼ 15% computational costs. Experiments on the Cityscapes, ADE20K, and PASCAL Context datasets demonstrate the effectiveness and efficiency of our proposed method.

heading: 1 Introduction
text: Semantic segmentation is a fundamental task in computer vision, which requires pixel-level classification on an input image. Fully convolutional networks (FCNs) are the de facto standard approaches to this task, which often consist of an encoder and a decoder. We focus on improving the decoder in this work and assume the encoder to be any backbone networks such as ResNet [HZRS16]. Due to the down-sampling layers (e.g., stridden convolutions or pooling) used in these networks, the encoder’s outputs are often of much lower resolutions than the input image. Thus, a decoder is used to spatially upsample the output. The decoder can simply be a bilinear upsampling, which directly upscales the low-resolution outputs of encoders to desired resolutions, or it can be a sophisticated network with a stack of convolutions and multi-level features. Note that another approach to tackle the issue of low-resolution outputs is the use of dilation convolution as in DeepLab [CPK+17], which balances the need for large receptive fields and maintaining a higher-resolution feature map. The computational cost is significantly heavier introduced by dilation convolutions.
Popular decoders for semantic segmentation include the one in DeepLabV3+ [CPSA17], which fuses the low-level feature maps with 1/4 resolution of the input image, and RefineNet [LMSR17] which gradually combines multi-level feature maps. A potential drawback of these decodes may be that they ∗BZ, YL, ZT contributed equally, and are listed alphabetically. CS is the corresponding author (e-mail: chunhua@me.com).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
do not explicitly exploit the label dependency, thus being less efficient in recovering the pixel-wise prediction accurately.
Let us consider an 8× 8 local patch on a binary semantic label space, denoted by P ∈ {0, 1}64. If we do not consider any structural correlations in the patch, there would be 264 possibilities for this local patch. However, it is clear to see, for any natural images, the vast majority of the possibilities never exist in the real label map and only a tiny fraction of them are really possible (see Fig. 1). Considering the redundancy in the labels, most existing decoders that do not explicitly take this into account would be sub-optimal. This motivates us to design a much more effective decoder by exploiting the prior.
A simple approach is dimensionality reduction techniques. As shown in [THSY19], the authors first apply principal component analysis (PCA) to the label patches and compress them into low-dimension compact vectors. Next, the network is required to predict these low-dimension vectors, which are eventually restored into the semantic labels by inverting the PCA process. Their method achieves some success. However, the simplicity and linearity assumption of PCA also limit its performance.
The semantic label masks for natural images are not random and follow some distributions, as shown in Fig. 1. Therefore, a good mask representation/decoder must exploit this prior. For computational efficiency, we also want the decoder to be in a compact form. Thus, we require the prior to be effectively learnable from data. Recently, many works [SMB+20, MON+19, PNM+20] exploit neural networks to represent 3D shapes. The work of [Mit97] found that neural networks enjoy the inductive bias of smooth interpolation between data points, which means that for two points of the same label, the neural networks tend to assign the same label to the points between them as well. As a result, we can conclude that the above idea of representing 3D shapes with neural networks can implicitly leverage the local smoothness prior. Therefore, inspired by these works, we can also represent the local patches of semantic labels with neural networks.
To be specific, as shown in Fig. 1, we represent each local label patch by a compact neural network gθi with a few convolution layers interleaved with non-linearities. The semantic labels of a local patch can be obtained by forwarding the corresponding network with (x, y)-coordinate maps and a guidance map m (explained later) as inputs. Furthermore, the parameters θ of these neural networks, which represent the local label patches, can be dynamically generated with the encoder network in FCNs, and each location on the encoder’s output feature maps is responsible for generating the parameters of the neural network representing the specific local label patch surrounding it. The dynamic network makes it possible to incorporate the neural representations into the conventional encoder-decoder architectures and enables a compact design of the decoder, resulting in an end-to-end trainable framework. This avoids the separable learning process as done in [THSY19].
Thus, our method is termed dynamic neural representation decoder (NRD) for semantic segmentation. We summarize our main contributions as follows.
• We propose a novel decoder that is effective and compact for semantic segmentation, to recover the spatial resolutions. For the first time, we represent the local label patches using neural networks and make use of dynamic convolutions to parametrize these neural networks.
• Different from previous methods, which often neglect the redundancy in the semantic label space, our proposed decoder NRD can better take advantage of the redundancy, and thus it is able to achieve on par or improved accuracy with significantly reduced computational cost. As shown in Fig. 2(a), we achieve a better trade-off between computational cost and accuracy compared to previous methods.
• Compared with the decoder used in classic encoder-decoder model DeeplabV3+ [CZP+18], we achieve an improvement of 0.9% mIoU on the Cityscapes dataset with less than 30% computational cost. Moreover, on the trimaps, where only the pixels near the object boundaries are evaluated, a 1.8% improvement can be obtained. This suggests that NRD can substantially improve the quality of the object boundaries. Moreover, NRD is even more significant than some methods that use dilated encoders, which usually require 4× more computational cost than ours with similar accuracy. For example, NRD achieves 46.09% mIoU on the competitive ADE20K dataset, which is comparable to that of DeepLabV3+ with a dilated encoder (46.35%) but with only 30% computational cost. We also benchmark our method on the Pascal Context dataset and show excellent performance with much less computational cost.

heading: 1.1 Related Work
text: Neural network representations. Recently, many works [SMB+20, MON+19, PNM+20] exploit neural networks to represent 3D shapes, which follow the idea that a 3D shape can be represented with a classification model and the 3D shape can be restored by forwarding the 3D coordinates through the classification network. These methods can be viewed as representing the point cloud data with the neural network’s parameters.
Dynamic filter networks. Different from traditional convolutions whose filters are fixed during inference once learned, the filters are dynamically generated by another network (namely, the controller). This idea was proposed by [JDBTvG16], which enlarges the capacity of the network and captures more content-dependent information such as contextual information. Recently, CondInst [TSC20] makes use of dynamic convolutions to implement the dynamic mask heads, which are used to predict the masks of individual instances. In this work, we follow in this vein for a different purpose, which is to dynamically generate the parameters of the networks representing local label masks so as to produce high-resolution semantic segmentation results.
Encoder-decoder network architectures. The encoder-decoder architecture is widely used to solve the semantic segmentation task, and almost all the mainstream semantic segmentation methods can be categorized into this family. Typically, the encoder gradually reduces the resolution of feature maps and extracts semantic features, while the decoder is applied to the output features of the encoder
to decode the desired semantic labels and recover the spatial resolution. Our work here focuses on the decoder.
The most commonly used bilinear upsampling can be viewed as the simplest decoder, which assumes that the semantic label maps are smooth to a large extent and the linear interpolation is sufficient to approximate them. Thus, using bilinear upsampling here is effective when the semantic label maps are simple, but the performance is not satisfactory if the label maps are complicated. DeconvNet [NHH15] introduces deconvolutional layers in its decoder to step-by-step recover the resolution of the prediction, which can result in much better performance. UPerNet [XLZ+18] uses an FPNlike structure to fuse feature maps of different scales, and obtains high-resolution feature maps. DeepLabv3+ [CZP+18] designs an effective decoder module that makes use of both encoder-decoder structure and dilation/atrous convolution, which is still one of the most competitive segmentation methods to date, especially in the trade-off between accuracy and computation complexity. CARAFE [WCX+19] first upsamples feature maps with parameter-free methods and then applies a learnable content-aware kernel mask to the upsampled feature maps. Thus far, despite achieving some success, we believe that there is much room for improvement in terms of taking full advantage of label space prior and designing highly effective and compact decoders for semantic segmentation. The proposed NRD attempts to narrow this gap.

heading: 2 Our Method

heading: 2.1 Overall Architecture
text: Given an input image I ∈ RH×W×3, the goal of semantic segmentation is to provide the pixel-level classification score map of the shape of H ×W × C, where C equals the number of categories to be classified into. As mentioned above, mainstream semantic segmentation methods are often based on encoder-decoder architectures. We also follow this line. Fig. 3 shows the overall framework of the proposed model for semantic segmentation.
Our work focuses on the decoder part, and thus we simply make our encoder the same as DeeplabV3+ [CZP+18]. The encoder consists of a CNN backbone (e.g., ResNet) and some optional modules such as ASPP [CPSA17], which can enhance the output features. By forwarding an input image I ∈ RH×W×C through the encoder, it generates feature maps with the shape of H/r × W/r × D,
where D is the number of the channels of the feature maps and r is the downsampling ratio of the encoder.
The downsampling ratio is determined by the down-sampling operators in the encoder and can be adjusted by reducing the stride of these down-sampling operators. Dilated convolutions are often used to compensate for the reduction of receptive fields after reducing the strides, with the price of computation overhead. An encoder that reduces the strides and uses dilation convolutions is often referred to as a dilated encoder. For example, an encoder based on the standard ResNet backbone produces the feature maps with r = 32. Most methods [YCW19, CZP+18, ZDS+18] dilate the encoder and reduce r to 16 or 8. By using the dilated encoder, these methods can output higher-resolution results while the dilated encoder would significantly increase the computational cost. In our work, we do not dilate the encoders (e.g., using r = 32) for faster computation and our proposed NRD is expected to better predict the semantic mask at a high resolution.
Let us denote the encoder’s output feature maps by F ∈ RH32×W32×D, whose resolution is 1/32 of the input image and the desired semantic label map (i.e., the final results). Thus, we make each spatial location on F responsible for a 32× 32 local patch surrounding the location and predict the local label map of the patch with our proposed NRD. Finally, the label maps of these patches are merged into the full-resolution segmentation results.

heading: 2.2 Dynamic Neural Representational Decoders (NRD)
text: In this section, we provide the details of our NRD and how we generate the parameters for it. The core idea here is to make use of a neural network to represent a local label patch. Thus, given a ground-truth semantic label map Y ∈ {0, 1, ..., C − 1}H×W , following the convention, we first convert it to the one-hot label map Y ′ ∈ {0, 1}H×W×C , where C is the number of classes. Next, Y ′ is divided into a number of H ′ ×W ′ local patches, and let P ∈ Rr×r×C be one of the patches, where H ′ and W ′ are the height and width of the encoder’s outputs and r is 32 in our work. Let us take Cityscapes as an example, and thus we have C = 19 and P ∈ R32×32×19. Next, a compact network gθ(·) is designed to represent the local mask patch P , as shown in Fig. 1. To be specific, in our experiment, gθ(·) is composed of three 1× 1 convolutions interleaved with the non-linearity ReLU. Except for the input and output channels, all the hidden layers in gθ(·) have 16 channels. The output channels of gθ(·) is equal to the number of classes (i.e., C). To recover the local patch P , we apply gθ(·) to a (x, y)-coordinate map Q = [0 : 1s : 1] × [0 : 1 s : 1] ∈ Rs×s×2, where [0 : 1s : 1] is the range from 0 to 1 with step 1 s (s is the desired upsampling rate, being 8 in this work ) and ‘×’ means the Cartesian multiplication. Since gθ(·) is composed of 1× 1 convolutions, the outputs of gθ(·) also have size s× s and can be denoted as G ∈ Rs×s×19. As shown in Fig. 3-(b), gθ(·) takes guidance maps m ∈ Rs×s×Cm as additional inputs. m is generated by applying two convolutional layers to the low-level feature maps, which reduce the channels of the feature maps to Cm, being 16 in this work. We use the same low-level feature maps as in DeepLabv3+, whose resolutions are 1/4 of the input image. Afterward, a bilinear upsampling is used to upscale G by 4 times to obtain P ′ ∈ R32×32×19. Next, we compute the loss between P ′ and P , which, through the back-propagation, adjusts the network’s parameter θ so that P ′ is as similar to P as possible. In this way, the network parameters θ can be viewed as the representation of the local semantic label patch P . Although it is possible to remove the bilinear upsampling here and, by increasing the resolution of Q and m, to make the network directly output the desired resolution 32× 32, we do not adopt this because using bilinear is sufficient when the upsampling factor is small (e.g., being 4 here). We note that in the above case the network gθ(·) has 899 parameters in total (#weights = (2 + 16) · 16(conv1) + 16 · 16(conv2) + 16 · 19(conv3) and #biases = 16(conv1) + 16(conv2) + 19(conv3)).
As shown in previous works [THSY19, WCY+18], each location on the encoder’s output feature maps can encode the information of the local patch surrounding it. Therefore, inspired by dynamic filter networks [YWP+18b], we can use the decoder’s output features at each location to dynamically generate the parameters of the representational network for the label patch of the location. To be specific, given the encoder’s output feature maps F ∈ RH′×W ′×D, where H ′ = H/32, W ′ = W/32 and D are height, width, and the number of channels of F .
Controller. We apply a 3×3 convolution with 512 channels, which is followed by a 1×1 convolution to generate the parameters θ (shown as the ‘controller’ in Fig. 3). The number of output channels of
the convolution is equal to the number of parameters in θ. The generated parameters are then split and reshaped into the weights and biases in gθ(·), and then gθ(·) is forwarded to obtain the semantic prediction P ′. P ′ is supervised by the ground-truth label patch P , making the whole framework end-to-end trainable. The overall architecture is shown in Fig. 3.

heading: 3 Experiments
text: The proposed model is evaluated on three semantic segmentation benchmarks. The performance is measured in terms of intersection-over-union averaged across the present classes (mIoU). We also evaluate the performance near the object boundaries by calculating mIoU on the trimap following [CZP+18]. We evaluate our method on the following benchmarks.
ADE20K [ZZP+17] is a dataset that contains more than 20K images exhaustively annotated with pixel-level annotation. It has 20, 210 images for training and 2, 000 images for validation. The number of categories is 150.
PASCAL Context [MCL+14] is a dataset with 4, 998 images for training and 5, 105 images for validation. We use default settings in [MMS20] that chose the most frequent 59 classes plus one background class (60 classes in total) as the targets.
Cityscapes [COR+16] is a benchmark for semantic urban scene parsing. The training, validation and test splits contain 2, 975, 500 and 1, 525 images with fine annotations, respectively. All images from this dataset are 1024× 2048 pixels in size. Implementation details. We use ResNet-50 and ResNet-101 [HZRS16] as our backbone networks and initialize them with the ImageNet pre-trained weights. The training and testing settings as well as data augmentations inherit the default settings in [MMS20] unless specified. Specifically, for all datasets, we use ‘poly’ as our learning policy. The initial learning rate is set at 0.01, the weight decay is set to 0.0005 for Cityscapes and ADE20K. For PASCAL Context, the initial learning rate is 0.004 and the weight decay is 0.0001. We train ADE20K, PASCAL-Context and Cityscapes for 160k, 80k and 80k iterations, with the crop size of 512× 512, 480× 480 and 512× 1024, respectively. The training and testing environment is on a workstation with four Volta 100 GPU cards. For test time augmentation, we employ the horizontal flip and multi-scale inference. The scale factors are {0.5, 0.75, 1.0, 1.25, 1.5, 1.75}.

heading: 3.1 Ablation Study
text: In this section, we conduct the ablation study to show the effectiveness of our proposed NRD. Here, we first compare NRD with the decoder of DeeplabV3+ [CZP+18] since it is widely-used in practice. Then, we compare with other decoder methods. Note that when these methods are compared, we use the same encoder for them. Finally, we investigate the hyper-parameters of our model design.
Compared to the DeepLabV3+ decoder. Since we do not use dilation convolutions in our encoder, we also remove the dilation in the DeeplabV3+ encoder for a fair comparison. The results are shown in Table 1. As shown in the table, with exactly the same settings, NRD outperforms the decoder in DeeplabV3+ by 0.9% mIoU on the Cityscpaes val. split with less than 1/3 computational cost (20.4 vs. 76.4 GFlops), and the total computational cost including the encoder and decoder is reduced from 290.6 to 234.6 GFlops. In addition, on the trimap, NRD is 1.8% mIoU better than the DeeplabV3+ decoder, which suggests that our method is able to produce boundaries of higher quality.
Compared to the bilinear decoder. We also compare our method with the simplest decoder which uses a 1× 1 convolution to map the outputs of the encoder to the desired segmentation predictions and then simply uses the bilinear upsampling to upscale the predictions to the desired resolutions. Again, both encoders’ output resolutions are 1/32 of the input image. To make a fair comparison, we also remove the guidance map in NRD (e.g., the low-level features). Thus, only coordinate maps are taken as the input of NRD. As shown in Table 1, NRD surpasses the bilinear decoder by a large margin (+3.5% mIoU). Note that although NRD has a higher computational cost than the bilinear decoder (1.3 vs. 2.6 GFlops), the overall computational cost is almost the same (215.5 vs. 216.8 GFlops) as most of the computational cost is in the encoder. Additionally, the mIoU on the trimap is improved by 5.4%.
Compared to other decoder methods. We also compare NRD with some other decoder methods. ResNet-50 is used as the backbone and we do not use the dilated encoders in all these methods. The results are shown in Table 2. As shown in the table, compared to CARAFE [WCX+19], we improve the mIoU on Cityscapes from 72.1% to 75.0% with similar computational complexity (203.0 vs. 203.2 GFlops) and the number of parameters. In addition, compared to DUC [WCY+18], which outputs multiple channels and use the “depth-to-space” operation to increase the spatial resolutions, our NRD is superior to it (75.0% vs. 74.7% mIoU) with only 60% computational complexity (203.2 vs. 336.1 GFlops) and ∼33% parameters. Ablation study of architectures of NRD. Here, we investigate the hyper-parameters of our NRD. Table 3 shows the performance as we vary the number of channels Cr of the representational network gθ(·). As we can see in the table, the performance is not very sensitive to the number of channels (within 0.4% mIoU). We also experiment by varying the number of channels Cm of the guidance map m. As shown in Table 3, using Cm = 16 can result in slightly better performance than Cm = 8 (79.8% vs. 79.5% mIoU), but increasing Cm to 32 cannot improve the performance further.
Table 4 shows the effect of the inputs to the representational network gθ(·). As we can see, if no guidance maps are given and gθ(·) only takes as input the coordinate maps, NRD can already achieve descent performance (78.2% mIoU), which is already much better than the bilinear decoder as shown in Table 1. In addition, if gθ(·) only takes the guidance map as input, NRD can achieve similar performance (78.3% mIoU). However, it can be seen that there is a significant improvement on the trimap mIoU (+3.9% mIoU), which suggests that the guidance map plays an import role in preserving the details. Finally, if both the coordinate maps and the guidance maps are used, NRD can achieve the best performance (79.8% mIoU).

heading: 3.2 Comparisons with state-of-the-art methods
text: In this section, we compare our method with other state-of-the-art methods on three dataset: ADE20K, PASCAL-Context and Cityscapes.
ADE20K. Table 5 shows the comparisons with state-of-the-art methods on ADE20K. Our method achieves 45.62% in terms of mIoU with ResNet-101 as the backbone. It is 0.95% better than the recent SFNet [LYZ+20], with the same ResNet-101 backbone. Besides, due to the strong ability of NRD to recover the spatial information, we do not need to use the multi paths complex decoder as
in SFNet and thus our method only spends 50% computational cost of SFNet. Our method is also better than other methods with dilated encoders, including DMNet [HDQ19], ANLNet [ZXB+19], CCNet [HWH+19] and EncNet [ZDS+18], and needs only 20% ∼ 30% computational cost of these methods. Additionally, by using a larger backbone ResNext-101, our performance can be further improved to 46.09% mIoU. Note that even with the larger backbone, our method still has much lower computational complexity than other methods with dilated encoders. As a result, we can achieve competitive performance among state-of-the-art methods with significantly less computational cost.
PASCAL-Context. Table 6 shows the results on the PASCAL-Context dataset. We follow HRNet [SZJ+19] to evaluate our method and report the results under 59 classes (without background) and 60 classes (with background). Our methods achieve 54.1% (59 classes) and 49.0% (60 classes) mIoU. The results are even better than the sophisticated high-resolution network HRNet with ∼50% computational complexity (42.9 vs. 82.7 GFlops). Note that HRNet stacks some hourglass networks and is much complicated than ours. Our method also achieves better results with less computational cost than other methods, as shown in the table.
Cityscapes. Table 7 shows the performance of our method on the Cityscapes test split. We train our model with the trainval split and only the fine annotations. As we can see, the proposed method can achieve competitive performance with much less computational complexity. Compared to the recent RGNet [YLG+20], our method achieves comparable performance with less than 30% computational cost (>1500 vs. 390.0 GFlops). Our method also has competitive performance with SFNet less than 50% computational complexity (821.2 vs. 390 GFlops). In addition, it is worth noting that our method based on ResNet-50 can have better performance than ResNet-18 based SFNet (79.5% vs. 80.0% mIoU) while having even less computational complexity (234.6 vs. 243.9 GFlops). This suggests that our proposed method has a better speed-accuracy trade-off as shown in Fig. 2(a).

heading: 4 Conclusion
text: We have proposed a compact yet very effective decoder, termed Neural Representational Decoders (NRD), for the semantic segmentation task. For the first time, we use the idea of neural representations for designing the segmentation decoder, which is able to better exploit the structure in the semantic
segmentation label space. To implement this idea, we dynamically generate the neural representations with dynamic convolution filter networks so that the neural representations can be incorporated into the standard encoder-decoder segmentation architectures, enabling end-to-end training. We show on a number of semantic segmentation benchmarks that our method is highly efficient and achieves state-of-the-art accuracy. We believe that our method can be a strong decoder in high-resolution semantic segmentation and may inspire other dense prediction tasks such as depth estimation and super-resolution. Last but not the least, our method still has some limitations. One of the limitations is that the dynamic filter networks have not been well-supported in some mobile devices, which might restrict the applicability of this method.

Review summaries:
In this paper, the authors design a simple and effective decoder named NRD for the Semantic Segmentation task. They represent the local patch of the label map with a neural network, which makes it possible to restore details in the prediction with a low computational cost. Experiments on three benchmarks show the effectiveness of the proposed approach.

This paper introduces one light-weight feature decoder (NRD) for semantic segmentation. Compared with the dilated encoder, the  proposed NRD is effective for semantic segmentation due to it can be directly applied on the  standard feature encoding network. In the experimental validation, this work shows satisfied performance with much less computational cost. 

In this paper, the authors propose a novel decoder that is effective and compact for semantic segmentation, to recover the spatial resolutions. For the first time, we represent the local label patches using neural networks and make use of dynamic convolutions to parametrize these neural networks. The experiments show on a number of semantic segmentation benchmarks that the method is highly efficient.