Model-Based Robust Adaptive Semantic Segmentation

1 INTRODUCTION . In recent years , computer vision has become one of the most promising research areas in deep learning because it has empowered a great amount of industry-level applications . In the context of computer vision , semantic segmentation is a core task formulated as a dense labeling problem Luc et al . ( 2016 ) , targeting to allocate every pixel with a label Hsin et al . ( 2019 ) of what is being represented . There are many applications now being used have made great progress with the help of semantic segmentation , such as medical image processing Xue et al . ( 2018 ) , autonomous vehicles Zhao et al . ( 2018 ) and robotics Wang et al . ( 2019 ) . However , many applications of semantic segmentation are life-critical , which means that low model accuracy may pose direct threat to human safety Oakden-Rayner et al . ( 2020 ) . Therefore , it is critical to design training algorithms that can enhance robustness of segmentation methods against input perturbations . A significant number of researches focusing on the robustness of computer vision have been conducted in recent years Arnab et al . ( 2018 ) Kamann & Rother ( 2020 ) Robey et al . ( 2020 ) Tramer et al . ( 2020 ) . However , the majority of existing works focus on image classification tasks Hendrycks & Dietterich ( 2019 ) . Robust semantic segmentation methods against adversarial attacks have been proposed in Hsin et al . ( 2019 ) Xue et al . ( 2018 ) Hung et al . ( 2018 ) Xie et al . ( 2017 ) . Work conducted by Goodfellow et al . ( 2015 ) proved that neural networks can be easily misled by some intentionally designed yet imperceptible perturbations to generate an incorrect answer with high confidence . In computer vision , adversarial attacks apply pixel-level changes onto the image that lead the model to wrong predictions , where the attacked image looks perceptually similar to the original one Ma et al . ( 2020 ) . Perturbation-based robust training algorithms have already tackled this problem . Nevertheless , in real life , there may be changes that raised by some out-of-distribution variations such as snow weather or extreme brightness that can not be represented using small pixel-level changes . Current works on robustness mainly focus on adapting the domain gap on a single side , i.e. , either from image or label side . For instance , Robey et al . ( 2020 ) proposed a model-based robust learning architecture which is applied on Convolutional Neural Networks ( CNN ) to maintain high prediction accuracy under natural variations for image classification tasks . Yet it is highly dependent on the performance of the natural variation model that capture the changes from source to target domain , which makes the algorithm less robust and less scalable in semantic segmentation task . Meanwhile , AdaptSegNet proposed by Tsai et al . ( 2020 ) minimizes the distribution gap on the output side . In this paper , we build upon Robey et al . ( 2020 ) to design a new robust training algorithm for semantic segmentation tasks . The objective is to build a segmentation model that generates high accuracy predictions under natural variation effects . Our method also utilize the idea from Tsai et al . ( 2020 ) that images from different domains with great appearance difference may share some similarity on the label side such as spatial layout and local context . We showed that the semantic feature map of an image under any natural variations remains unchanged which we refer to as semantic meaning invariance . Our proposed training algorithm minimizes the gap on both image and label side . On image side , the model-based robust training algorithm is applied to train a model using the simulated target domain images to enhance robustness . On label side , we apply generative adversarial networks ( GANs ) to minimize the feature map gap between simulated natural variation images and target domain images . We have also included extensive comparisons showing that our method outperforms related state-of-the-art works in domain adaptation . Our contributions are : 1 ) We propose MRTAdapt , a new model-based training algorithm to enhance robustness of DNN-based semantic segmentation methods against natural variations . 2 ) We build on top of generative adversarial networks and model-based robust training algorithms to minimize the gap on both image and label side to enhance robustness . 3 ) Our results on Cityscapes Cordts et al . ( 2016 ) and Synthia Ros et al . ( 2016 ) datasets show that our method outperforms multiple state-ofthe-art domain adaptation techniques , such as AdaptSegNet Tsai et al . ( 2020 ) , ADVENT Vu et al . ( 2019 ) and FDA Yang & Soatto ( 2020 ) . 2 RELATED WORK . Semantic Segmentation . In the past decades , Convolution Neural Networks ( CNN ) are widely used in semantic segmentation . Current state-of-the-art semantic segmentation frameworks are mostly developed from Fully Convolutional Network ( FCN ) by Long et al . ( 2015 ) . ResNet proposed by He et al . ( 2015 ) used a residual block to sum the nonlinear activation output and identity mapping , which is proved to improve the gradient propagation and increase the accuracy of semantic segmentation . DenseNet Huang et al . ( 2018a ) builds upon ResNet and uses the concatenation of previous feature maps called dense block . This gives each layer in DenseNet information from all preceding layers . Jégou et al . ( 2017 ) extended DenseNet into FCN architecture . SegNet proposed by Badrinarayanan et al . ( 2017 ) introduces the deep convolution encoder-decoder architecture to the field of semantic segmentation . Also , in Zhou et al . ( 2015 ) , it is proved that empirical size of receptive field is much smaller than the theoretical size . Chen et al . ( 2016 ) and Yu & Koltun ( 2016 ) used dilated convolution to enlarge the receptive field . ParseNet by Liu et al . ( 2015 ) adds global context to CNNs for semantic segmentation . He et al . ( 2014 ) introduced spatial pyramid pooling in DCNN . PSPNet introduced by Zhao et al . ( 2017 ) uses a novel global pyramid pooling module to capture both global context information . Duta et al . ( 2020 ) extended the idea by combining both local and global Pyramidal Convolution blocks in the neural network model . Chen et al . ( 2017b ) introduced atrous spatial pyramid pooling . Depth image is also used along with original RGB information for semantic segmentation by Wang et al . ( 2019 ) . For actual applications , Zhao et al . ( 2018 ) proposed ICNet for real-time semantic segmentation accomplishing fast inference without sacrificing too much quality left behind . Azimi et al . ( 2020 ) proposed aerial perspective dataset for dense semantic segmentation . Domain Adaptation . Combining with the techniques of Generative Adversarial Network ( GAN ) Goodfellow et al . ( 2014 ) , Ganin & Lempitsky ( 2015 ) proposed DANN to reduce the distribution gap between different domains by using discriminator to make the prediction can not be identified between source and target domain . Pan et al . ( 2020 ) first separate target domain into splits based on entropy-based ranking and then deploy self-supervised adaptation technique to reduce the domain gap between synthetic data and real images . PIT proposed by Lv et al . ( 2020 ) constructs pivot information shared across domains . Chen et al . ( 2019 ) used depth image as guided information to build adaptation method from synthetic to real dataset . In the field of utilizing synthetic datasets to auxiliate the training process of real-world images , MUNIT proposed by Huang et al . ( 2018b ) learn conditional distribution of target domain which can separate domain-invariant semantic content of an image from domain-specific properties . Vu proposed ADVENT model which maximize prediction certainty in target domain by introducing entropy loss Vu et al . ( 2019 ) . Zhu proposed CycleGAN Zhu et al . ( 2020 ) which was trying to learn a mapping such that the distribution of generated image is indistinguishable from the target domain distribution using adversarial loss . AdaptSegNet proposed by Tsai et al . ( 2020 ) aimed to reduce the gap between the outputs from source and target domain given that images might be very different in appearance . FDA proposed by Yang & Soatto ( 2020 ) uses Fast Fourier Transform ( FFT ) to adapt source and target domain . Robustness . As for robustness against corrupted images , Hendrycks & Dietterich ( 2019 ) established rigorous benchmarks for image classification and proposed a series of image corruption examples . Kamann & Rother ( 2020 ) showed that robustness increases with the performance of the semantic segmentation model and dense prediction cell was only designed to improve performance on clean data . Regarding robustness on semantic segmentation , Arnab et al . ( 2018 ) did the first evaluation of adversarial attacks on semantic segmentation and analyzed multi-scale processing . Robey et al . ( 2020 ) proposed model-based robust training architecture focusing on the topic of image classification . Wong & Kolter ( 2020 ) bridged the gap between real-world perturbations and adversarial defenses by learning perturbation sets from data , through common image corruptions . 3 METHOD . In this section we propose a new robust training algorithm for semantic segmentation tasks . Particularly , in section 3.1 , we define the semantic segmentation task and the natural variations . In section 3.2 , we give the outline of our proposed algorithm . In section 3.3 , we define semantic meaning invariance and clarify its connection to our algorithm . In section 3.4 , we introduce the detailed training procedure of our model-based robust adaptive training algorithm ( MRTAdapt ) . 3.1 SEMANTIC SEGMENTATION AND NATURAL VARIATION . Consider an input x ∈ RH×W×3 representing an RGB image with three channels and its corresponding label map y ∈ RH×W which is annotated for every pixel in the image . A semantic segmentation task is to train a neural network that generates pixel-wise segmentation feature map ŷ ∈ RH×W . We assume the dataset is drawn i.i.d . from distribution ( x , y ) ∼ P. The optimization objective in semantic segmentation is to find the best weight w that minimizes the loss function L ( x , y ; w ) with respect to input x , label y and weight w , which can be written as : min w 1 n n∑ j=1 [ L ( x , y ; w ) ] ( 1 ) Next we denote by δ ∈ ∆ a natural variation applied to x that is derived from a nuisance space ∆ . A model of natural variation can be learned from the difference of source and target domain images . Particularly , we denote the natural variation model that transforms the input image x from the source domain into an image in the ( perturbed ) target domain as V ( x , δ ) . Note that , in general , it is hard to write a natural variation model learned from data into an analytical form because there is no closed form expression capturing the natural perturbation . Natural variation model can be customized regarding the needs of each dataset . In this paper , our goal is to design a semantic segmentation method that is robust to natural variations δ . 3.2 ALGORITHM OVERVIEW . Our proposed model-based robust adaptive training algorithm ( MRTAdapt ) shown in Figure 1 has two modules : a segmentation network G using a model-based robust training algorithm to enhance robustness on image side , and a discriminator network D to enhance robustness on label side . Two sets of images from source and target domain will have the notation of Is and It , while the image generated using natural variation model will be denoted as Inv . After using source domain images to train the model , we transform images from source domain with natural variation model . Then we use simulated natural variation images to find the highest segmentation loss by applying the model-based robust training algorithm to optimize the segmentation network . Then we predict the segmentation softmax output for the unlabeled target domain images It by forwarding the target images into the same segmentation network . To minimize the gap between the output of natural variation images and target domain images , we use these two predictions as the input to the discriminator to distinguish whether the feature map generated by segmentation network is from natural variation model or target domain . Using an adversarial loss for discriminator , the loss will back-propagate to segmentation network to help segmentation network to generate similar distributions in target domain to natural variation . Detailed information will be covered in the following sections .

Summary:  The paper proposes a discriminator-generator-based approach for robust semantic segmentation of images against natural variations such as snow, day/night (brightness). The input training image is first modified (perturbation in brightness, generating images with snow), and then fed to PSPNet to generate feature map for these perturbed images. Similarly, the original images (without perturbation) are fed to the same PSPNet to generate the original source feature map. The segmentation loss is a weighted sum of the loss computed on the original features and perturbed features. The loss on perturbed features finds the optimal perturbation parameter that maximizes the loss between predicted and original labels, while the loss on original image features is the traditional segmentation loss. The discriminator module attempts to distinguish between the features generated by perturbed images or target images. The proposed work is compared with the existing works on two standard datasets, Cityscapes and Synthia. The experimental results demonstrate the proposed approach outperforms the existing methods in terms of mIoU.  The main novelty of the proposed work is the combined use of adversarial networks and model-based training (Robey et al 2020) to achieve a robust semantic segmentation in the presence of variation in brightness and snow in the scene.