Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation

1 Introduction . Semantic segmentation is a fundamental task in computer vision , which requires pixel-level classification on an input image . Fully convolutional networks ( FCNs ) are the de facto standard approaches to this task , which often consist of an encoder and a decoder . We focus on improving the decoder in this work and assume the encoder to be any backbone networks such as ResNet [ HZRS16 ] . Due to the down-sampling layers ( e.g. , stridden convolutions or pooling ) used in these networks , the encoder ’ s outputs are often of much lower resolutions than the input image . Thus , a decoder is used to spatially upsample the output . The decoder can simply be a bilinear upsampling , which directly upscales the low-resolution outputs of encoders to desired resolutions , or it can be a sophisticated network with a stack of convolutions and multi-level features . Note that another approach to tackle the issue of low-resolution outputs is the use of dilation convolution as in DeepLab [ CPK+17 ] , which balances the need for large receptive fields and maintaining a higher-resolution feature map . The computational cost is significantly heavier introduced by dilation convolutions . Popular decoders for semantic segmentation include the one in DeepLabV3+ [ CPSA17 ] , which fuses the low-level feature maps with 1/4 resolution of the input image , and RefineNet [ LMSR17 ] which gradually combines multi-level feature maps . A potential drawback of these decodes may be that they ∗BZ , YL , ZT contributed equally , and are listed alphabetically . CS is the corresponding author ( e-mail : chunhua @ me.com ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . do not explicitly exploit the label dependency , thus being less efficient in recovering the pixel-wise prediction accurately . Let us consider an 8× 8 local patch on a binary semantic label space , denoted by P ∈ { 0 , 1 } 64 . If we do not consider any structural correlations in the patch , there would be 264 possibilities for this local patch . However , it is clear to see , for any natural images , the vast majority of the possibilities never exist in the real label map and only a tiny fraction of them are really possible ( see Fig . 1 ) . Considering the redundancy in the labels , most existing decoders that do not explicitly take this into account would be sub-optimal . This motivates us to design a much more effective decoder by exploiting the prior . A simple approach is dimensionality reduction techniques . As shown in [ THSY19 ] , the authors first apply principal component analysis ( PCA ) to the label patches and compress them into low-dimension compact vectors . Next , the network is required to predict these low-dimension vectors , which are eventually restored into the semantic labels by inverting the PCA process . Their method achieves some success . However , the simplicity and linearity assumption of PCA also limit its performance . The semantic label masks for natural images are not random and follow some distributions , as shown in Fig . 1 . Therefore , a good mask representation/decoder must exploit this prior . For computational efficiency , we also want the decoder to be in a compact form . Thus , we require the prior to be effectively learnable from data . Recently , many works [ SMB+20 , MON+19 , PNM+20 ] exploit neural networks to represent 3D shapes . The work of [ Mit97 ] found that neural networks enjoy the inductive bias of smooth interpolation between data points , which means that for two points of the same label , the neural networks tend to assign the same label to the points between them as well . As a result , we can conclude that the above idea of representing 3D shapes with neural networks can implicitly leverage the local smoothness prior . Therefore , inspired by these works , we can also represent the local patches of semantic labels with neural networks . To be specific , as shown in Fig . 1 , we represent each local label patch by a compact neural network gθi with a few convolution layers interleaved with non-linearities . The semantic labels of a local patch can be obtained by forwarding the corresponding network with ( x , y ) -coordinate maps and a guidance map m ( explained later ) as inputs . Furthermore , the parameters θ of these neural networks , which represent the local label patches , can be dynamically generated with the encoder network in FCNs , and each location on the encoder ’ s output feature maps is responsible for generating the parameters of the neural network representing the specific local label patch surrounding it . The dynamic network makes it possible to incorporate the neural representations into the conventional encoder-decoder architectures and enables a compact design of the decoder , resulting in an end-to-end trainable framework . This avoids the separable learning process as done in [ THSY19 ] . Thus , our method is termed dynamic neural representation decoder ( NRD ) for semantic segmentation . We summarize our main contributions as follows . • We propose a novel decoder that is effective and compact for semantic segmentation , to recover the spatial resolutions . For the first time , we represent the local label patches using neural networks and make use of dynamic convolutions to parametrize these neural networks . • Different from previous methods , which often neglect the redundancy in the semantic label space , our proposed decoder NRD can better take advantage of the redundancy , and thus it is able to achieve on par or improved accuracy with significantly reduced computational cost . As shown in Fig . 2 ( a ) , we achieve a better trade-off between computational cost and accuracy compared to previous methods . • Compared with the decoder used in classic encoder-decoder model DeeplabV3+ [ CZP+18 ] , we achieve an improvement of 0.9 % mIoU on the Cityscapes dataset with less than 30 % computational cost . Moreover , on the trimaps , where only the pixels near the object boundaries are evaluated , a 1.8 % improvement can be obtained . This suggests that NRD can substantially improve the quality of the object boundaries . Moreover , NRD is even more significant than some methods that use dilated encoders , which usually require 4× more computational cost than ours with similar accuracy . For example , NRD achieves 46.09 % mIoU on the competitive ADE20K dataset , which is comparable to that of DeepLabV3+ with a dilated encoder ( 46.35 % ) but with only 30 % computational cost . We also benchmark our method on the Pascal Context dataset and show excellent performance with much less computational cost . 1.1 Related Work . Neural network representations . Recently , many works [ SMB+20 , MON+19 , PNM+20 ] exploit neural networks to represent 3D shapes , which follow the idea that a 3D shape can be represented with a classification model and the 3D shape can be restored by forwarding the 3D coordinates through the classification network . These methods can be viewed as representing the point cloud data with the neural network ’ s parameters . Dynamic filter networks . Different from traditional convolutions whose filters are fixed during inference once learned , the filters are dynamically generated by another network ( namely , the controller ) . This idea was proposed by [ JDBTvG16 ] , which enlarges the capacity of the network and captures more content-dependent information such as contextual information . Recently , CondInst [ TSC20 ] makes use of dynamic convolutions to implement the dynamic mask heads , which are used to predict the masks of individual instances . In this work , we follow in this vein for a different purpose , which is to dynamically generate the parameters of the networks representing local label masks so as to produce high-resolution semantic segmentation results . Encoder-decoder network architectures . The encoder-decoder architecture is widely used to solve the semantic segmentation task , and almost all the mainstream semantic segmentation methods can be categorized into this family . Typically , the encoder gradually reduces the resolution of feature maps and extracts semantic features , while the decoder is applied to the output features of the encoder to decode the desired semantic labels and recover the spatial resolution . Our work here focuses on the decoder . The most commonly used bilinear upsampling can be viewed as the simplest decoder , which assumes that the semantic label maps are smooth to a large extent and the linear interpolation is sufficient to approximate them . Thus , using bilinear upsampling here is effective when the semantic label maps are simple , but the performance is not satisfactory if the label maps are complicated . DeconvNet [ NHH15 ] introduces deconvolutional layers in its decoder to step-by-step recover the resolution of the prediction , which can result in much better performance . UPerNet [ XLZ+18 ] uses an FPNlike structure to fuse feature maps of different scales , and obtains high-resolution feature maps . DeepLabv3+ [ CZP+18 ] designs an effective decoder module that makes use of both encoder-decoder structure and dilation/atrous convolution , which is still one of the most competitive segmentation methods to date , especially in the trade-off between accuracy and computation complexity . CARAFE [ WCX+19 ] first upsamples feature maps with parameter-free methods and then applies a learnable content-aware kernel mask to the upsampled feature maps . Thus far , despite achieving some success , we believe that there is much room for improvement in terms of taking full advantage of label space prior and designing highly effective and compact decoders for semantic segmentation . The proposed NRD attempts to narrow this gap . 2 Our Method . 2.1 Overall Architecture . Given an input image I ∈ RH×W×3 , the goal of semantic segmentation is to provide the pixel-level classification score map of the shape of H ×W × C , where C equals the number of categories to be classified into . As mentioned above , mainstream semantic segmentation methods are often based on encoder-decoder architectures . We also follow this line . Fig . 3 shows the overall framework of the proposed model for semantic segmentation . Our work focuses on the decoder part , and thus we simply make our encoder the same as DeeplabV3+ [ CZP+18 ] . The encoder consists of a CNN backbone ( e.g. , ResNet ) and some optional modules such as ASPP [ CPSA17 ] , which can enhance the output features . By forwarding an input image I ∈ RH×W×C through the encoder , it generates feature maps with the shape of H/r × W/r × D , where D is the number of the channels of the feature maps and r is the downsampling ratio of the encoder . The downsampling ratio is determined by the down-sampling operators in the encoder and can be adjusted by reducing the stride of these down-sampling operators . Dilated convolutions are often used to compensate for the reduction of receptive fields after reducing the strides , with the price of computation overhead . An encoder that reduces the strides and uses dilation convolutions is often referred to as a dilated encoder . For example , an encoder based on the standard ResNet backbone produces the feature maps with r = 32 . Most methods [ YCW19 , CZP+18 , ZDS+18 ] dilate the encoder and reduce r to 16 or 8 . By using the dilated encoder , these methods can output higher-resolution results while the dilated encoder would significantly increase the computational cost . In our work , we do not dilate the encoders ( e.g. , using r = 32 ) for faster computation and our proposed NRD is expected to better predict the semantic mask at a high resolution . Let us denote the encoder ’ s output feature maps by F ∈ RH32×W32×D , whose resolution is 1/32 of the input image and the desired semantic label map ( i.e. , the final results ) . Thus , we make each spatial location on F responsible for a 32× 32 local patch surrounding the location and predict the local label map of the patch with our proposed NRD . Finally , the label maps of these patches are merged into the full-resolution segmentation results .

Summary: In this paper, the authors design a simple and effective decoder named NRD for the Semantic Segmentation task. They represent the local patch of the label map with a neural network, which makes it possible to restore details in the prediction with a low computational cost. Experiments on three benchmarks show the effectiveness of the proposed approach.