Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation

1 Introduction . Semantic segmentation is a fundamental task in computer vision , which requires pixel-level classification on an input image . Fully convolutional networks ( FCNs ) are the de facto standard approaches to this task , which often consist of an encoder and a decoder . We focus on improving the decoder in this work and assume the encoder to be any backbone networks such as ResNet [ HZRS16 ] . Due to the down-sampling layers ( e.g. , stridden convolutions or pooling ) used in these networks , the encoder ’ s outputs are often of much lower resolutions than the input image . Thus , a decoder is used to spatially upsample the output . The decoder can simply be a bilinear upsampling , which directly upscales the low-resolution outputs of encoders to desired resolutions , or it can be a sophisticated network with a stack of convolutions and multi-level features . Note that another approach to tackle the issue of low-resolution outputs is the use of dilation convolution as in DeepLab [ CPK+17 ] , which balances the need for large receptive fields and maintaining a higher-resolution feature map . The computational cost is significantly heavier introduced by dilation convolutions . Popular decoders for semantic segmentation include the one in DeepLabV3+ [ CPSA17 ] , which fuses the low-level feature maps with 1/4 resolution of the input image , and RefineNet [ LMSR17 ] which gradually combines multi-level feature maps . A potential drawback of these decodes may be that they ∗BZ , YL , ZT contributed equally , and are listed alphabetically . CS is the corresponding author ( e-mail : chunhua @ me.com ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . do not explicitly exploit the label dependency , thus being less efficient in recovering the pixel-wise prediction accurately . Let us consider an 8× 8 local patch on a binary semantic label space , denoted by P ∈ { 0 , 1 } 64 . If we do not consider any structural correlations in the patch , there would be 264 possibilities for this local patch . However , it is clear to see , for any natural images , the vast majority of the possibilities never exist in the real label map and only a tiny fraction of them are really possible ( see Fig . 1 ) . Considering the redundancy in the labels , most existing decoders that do not explicitly take this into account would be sub-optimal . This motivates us to design a much more effective decoder by exploiting the prior . A simple approach is dimensionality reduction techniques . As shown in [ THSY19 ] , the authors first apply principal component analysis ( PCA ) to the label patches and compress them into low-dimension compact vectors . Next , the network is required to predict these low-dimension vectors , which are eventually restored into the semantic labels by inverting the PCA process . Their method achieves some success . However , the simplicity and linearity assumption of PCA also limit its performance . The semantic label masks for natural images are not random and follow some distributions , as shown in Fig . 1 . Therefore , a good mask representation/decoder must exploit this prior . For computational efficiency , we also want the decoder to be in a compact form . Thus , we require the prior to be effectively learnable from data . Recently , many works [ SMB+20 , MON+19 , PNM+20 ] exploit neural networks to represent 3D shapes . The work of [ Mit97 ] found that neural networks enjoy the inductive bias of smooth interpolation between data points , which means that for two points of the same label , the neural networks tend to assign the same label to the points between them as well . As a result , we can conclude that the above idea of representing 3D shapes with neural networks can implicitly leverage the local smoothness prior . Therefore , inspired by these works , we can also represent the local patches of semantic labels with neural networks . To be specific , as shown in Fig . 1 , we represent each local label patch by a compact neural network gθi with a few convolution layers interleaved with non-linearities . The semantic labels of a local patch can be obtained by forwarding the corresponding network with ( x , y ) -coordinate maps and a guidance map m ( explained later ) as inputs . Furthermore , the parameters θ of these neural networks , which represent the local label patches , can be dynamically generated with the encoder network in FCNs , and each location on the encoder ’ s output feature maps is responsible for generating the parameters of the neural network representing the specific local label patch surrounding it . The dynamic network makes it possible to incorporate the neural representations into the conventional encoder-decoder architectures and enables a compact design of the decoder , resulting in an end-to-end trainable framework . This avoids the separable learning process as done in [ THSY19 ] . Thus , our method is termed dynamic neural representation decoder ( NRD ) for semantic segmentation . We summarize our main contributions as follows . • We propose a novel decoder that is effective and compact for semantic segmentation , to recover the spatial resolutions . For the first time , we represent the local label patches using neural networks and make use of dynamic convolutions to parametrize these neural networks . • Different from previous methods , which often neglect the redundancy in the semantic label space , our proposed decoder NRD can better take advantage of the redundancy , and thus it is able to achieve on par or improved accuracy with significantly reduced computational cost . As shown in Fig . 2 ( a ) , we achieve a better trade-off between computational cost and accuracy compared to previous methods . • Compared with the decoder used in classic encoder-decoder model DeeplabV3+ [ CZP+18 ] , we achieve an improvement of 0.9 % mIoU on the Cityscapes dataset with less than 30 % computational cost . Moreover , on the trimaps , where only the pixels near the object boundaries are evaluated , a 1.8 % improvement can be obtained . This suggests that NRD can substantially improve the quality of the object boundaries . Moreover , NRD is even more significant than some methods that use dilated encoders , which usually require 4× more computational cost than ours with similar accuracy . For example , NRD achieves 46.09 % mIoU on the competitive ADE20K dataset , which is comparable to that of DeepLabV3+ with a dilated encoder ( 46.35 % ) but with only 30 % computational cost . We also benchmark our method on the Pascal Context dataset and show excellent performance with much less computational cost . 1.1 Related Work . Neural network representations . Recently , many works [ SMB+20 , MON+19 , PNM+20 ] exploit neural networks to represent 3D shapes , which follow the idea that a 3D shape can be represented with a classification model and the 3D shape can be restored by forwarding the 3D coordinates through the classification network . These methods can be viewed as representing the point cloud data with the neural network ’ s parameters . Dynamic filter networks . Different from traditional convolutions whose filters are fixed during inference once learned , the filters are dynamically generated by another network ( namely , the controller ) . This idea was proposed by [ JDBTvG16 ] , which enlarges the capacity of the network and captures more content-dependent information such as contextual information . Recently , CondInst [ TSC20 ] makes use of dynamic convolutions to implement the dynamic mask heads , which are used to predict the masks of individual instances . In this work , we follow in this vein for a different purpose , which is to dynamically generate the parameters of the networks representing local label masks so as to produce high-resolution semantic segmentation results . Encoder-decoder network architectures . The encoder-decoder architecture is widely used to solve the semantic segmentation task , and almost all the mainstream semantic segmentation methods can be categorized into this family . Typically , the encoder gradually reduces the resolution of feature maps and extracts semantic features , while the decoder is applied to the output features of the encoder to decode the desired semantic labels and recover the spatial resolution . Our work here focuses on the decoder . The most commonly used bilinear upsampling can be viewed as the simplest decoder , which assumes that the semantic label maps are smooth to a large extent and the linear interpolation is sufficient to approximate them . Thus , using bilinear upsampling here is effective when the semantic label maps are simple , but the performance is not satisfactory if the label maps are complicated . DeconvNet [ NHH15 ] introduces deconvolutional layers in its decoder to step-by-step recover the resolution of the prediction , which can result in much better performance . UPerNet [ XLZ+18 ] uses an FPNlike structure to fuse feature maps of different scales , and obtains high-resolution feature maps . DeepLabv3+ [ CZP+18 ] designs an effective decoder module that makes use of both encoder-decoder structure and dilation/atrous convolution , which is still one of the most competitive segmentation methods to date , especially in the trade-off between accuracy and computation complexity . CARAFE [ WCX+19 ] first upsamples feature maps with parameter-free methods and then applies a learnable content-aware kernel mask to the upsampled feature maps . Thus far , despite achieving some success , we believe that there is much room for improvement in terms of taking full advantage of label space prior and designing highly effective and compact decoders for semantic segmentation . The proposed NRD attempts to narrow this gap . 2 Our Method . 2.1 Overall Architecture . Given an input image I ∈ RH×W×3 , the goal of semantic segmentation is to provide the pixel-level classification score map of the shape of H ×W × C , where C equals the number of categories to be classified into . As mentioned above , mainstream semantic segmentation methods are often based on encoder-decoder architectures . We also follow this line . Fig . 3 shows the overall framework of the proposed model for semantic segmentation . Our work focuses on the decoder part , and thus we simply make our encoder the same as DeeplabV3+ [ CZP+18 ] . The encoder consists of a CNN backbone ( e.g. , ResNet ) and some optional modules such as ASPP [ CPSA17 ] , which can enhance the output features . By forwarding an input image I ∈ RH×W×C through the encoder , it generates feature maps with the shape of H/r × W/r × D , where D is the number of the channels of the feature maps and r is the downsampling ratio of the encoder . The downsampling ratio is determined by the down-sampling operators in the encoder and can be adjusted by reducing the stride of these down-sampling operators . Dilated convolutions are often used to compensate for the reduction of receptive fields after reducing the strides , with the price of computation overhead . An encoder that reduces the strides and uses dilation convolutions is often referred to as a dilated encoder . For example , an encoder based on the standard ResNet backbone produces the feature maps with r = 32 . Most methods [ YCW19 , CZP+18 , ZDS+18 ] dilate the encoder and reduce r to 16 or 8 . By using the dilated encoder , these methods can output higher-resolution results while the dilated encoder would significantly increase the computational cost . In our work , we do not dilate the encoders ( e.g. , using r = 32 ) for faster computation and our proposed NRD is expected to better predict the semantic mask at a high resolution . Let us denote the encoder ’ s output feature maps by F ∈ RH32×W32×D , whose resolution is 1/32 of the input image and the desired semantic label map ( i.e. , the final results ) . Thus , we make each spatial location on F responsible for a 32× 32 local patch surrounding the location and predict the local label map of the patch with our proposed NRD . Finally , the label maps of these patches are merged into the full-resolution segmentation results .

Summary: In this paper, the authors propose a novel decoder that is effective and compact for semantic segmentation, to recover the spatial resolutions. For the first time, we represent the local label patches using neural networks and make use of dynamic convolutions to parametrize these neural networks. The experiments show on a number of semantic segmentation benchmarks that the method is highly efficient.