Model-Based Robust Adaptive Semantic Segmentation
Authors: Jun Wang
Yiannis Kantaros
Venue: ICLR 2022 Submitted (2022)
Keywords: Semantic Segmentation
Robustness
Natural Variation

Abstract: Semantic image segmentation enjoys a wide range of applications such as autonomous vehicles and medical imaging while it is typically accomplished by deep neural networks (DNNs). Nevertheless, DNNs are known to be fragile to input perturbations that are adversarially crafted or occur due to natural variations, such as changes in weather or lighting conditions. This issue of lack of robustness prevents the application of learning-based semantic segmentation methods on safety-critical applications. To mitigate this challenge, in this paper, we propose model-based robust adaptive training algorithm (MRTAdapt), a new training algorithm to enhance the robustness of DNN-based semantic segmentation methods against natural variations that leverages model-based robust training algorithms and generative adversarial networks. Natural variation effects are minimized from both image and label sides. We provide extensive experimental results on both real-world and synthetic datasets demonstrating that model-based robust adaptive training algorithm outperforms multiple state-of-the-art models under various natural variations. 

Under review as a conference paper at ICLR 2022
MODEL-BASED ROBUST ADAPTIVE SEMANTIC SEGMENTATION
Anonymous authors Paper under double-bling review
ABSTRACT
Semantic image segmentation enjoys a wide range of applications such as au- tonomous vehicles and medical imaging while is typically accomplished by deep neural networks (DNNs). Nevertheless, DNNs are known to be fragile to input perturbations that are adversarially crafted or occur due to natural variations, such as changes in weather or lighting conditions. This issue of lack of robust- ness prevents the application safety-critical applications. To mitigate this challenge, in this paper, pro- pose a new robust training algorithm, called MRTAdapt, for Model-based Robust Adaptive training, to enhance the robustness of DNN-based semantic segmenta- tion methods against natural variations that leverages model-based robust training algorithms and generative adversarial networks. Natural variation effects mini- mized from both image and label sides. We provide extensive experimental results on both real-world and synthetic datasets demonstrating that the proposed training algorithm result in robust models that outperform multiple state-of-the models under various natural variations.
learning-basee semantic segmentation methods
INTRODUCTION
In recent years, computer vision has become one of the most promising research areas in deep learning because it has empowered great amount of industry-level applications. In the context of computer vision, semantic segmentation is a core task formulated as a dense labeling problem Luc et al. (016)), targeting to allocate every pixel with a label Hsin et al. (2019) of what is being represented. There are many applications now being used have made great progress with the help of semantic segmentation, such as medical image processing Xue et al. (20T8), autonomous vehicles Zhao et al. (018)) and robotics Wang et al. 2019). However, many applications of semantic seg- mentation are life-critical, which means that low model accuracy may pose direct threat to human safety Bakden--Rayer et al. (2020). Therefore, is critical to design training algorithms that can enhance robustness of segmentation methods against input perturbations.
significant number of researches focusing on the robustness of computer vision have been con- ducted in recent years Arna et al. Kaman Kamann & Rother (220) Robey al. 2020 Tramer al. (2020). However, the majority of existing works focus on image classification tasks Hendrycks & Dietterich 2 Robust semantic segmentation methods against adversarial attacks have been pro- posed in Hsin et al. (2019) Xue et al. (2018) Hung et al. (2018) Xie et al. (2017). Work conducted by Goodfellow al. (2015) proved that neural networks can easily misled by some intentionally designed yet imperceptible perturbations to generate an incorrect answer with high confidence. In computer vision, adversarial attacks apply pixel-level changes onto the image that lead the model to wrong predictions, where the attacked image looks perceptually similar to the original one Ma robust. training algorithms have already tackled this problem. Nev- ertheless, in real life, there may be changes that raised by some out-of-distrrbution variations such as snow weather or extreme brightness that can not be represented using small pixel-level changes.
al. (2020). Perturbation-base
Current works robustness mainly focus on adapting the domain gap on single side, i.e., either from image or label side. For instance, Robey et al. (2020) proposed a model-based robust learning architecture which is applied on Convolutional Neural NetWorks(CNN) to maintain high prediction accuracy under natural variations for image classification tasks. Yet is highly dependent on the performance of the natural variation model that capture the changes from source target domain,

---

Under review as a conference paper at ICLR 2022
which makes the algorithm less robust and less scalable in semantic segmentation task. Meanwhile, AdaptSegNet proposed by Tsai et al. (2020) minimizes the distribution gap on the output side. In this paper, we build upon[Rober et al. (2020) to design a new robust training algorithm for semantic segmentation tasks. The objective is to build a segmentation model that generates high accuracy predictions under natural variation effects. Our method also utilize the idea from [sai et al. (2020) that images from different domains with great appearance difference may share some similarity on the label side such as spatial layout and local context. We showed that the semantic feature map of an image under any natural variations remains unchanged which we refer to as semantic meaning invariance. Our proposed training algorithm minimizes the gap on both image and label side. On image side, the model-based robust training algorithm is applied to train model using the simulated target domain images to enhance robustness. On label side, we apply generative adversarial networks (GANs) to minimize the feature map gap between simulated natural variation images and target domain images. We have also included extensive comparisons showing that our method outperforms related state-of-the-ar works in domain adaptation.
Our contributions are: 1) We propose MRTAdapt, a new model-based training algorithm to enhance robustness of DNN-based semantic segmentation methods against natural variations. 2) We build on top of generative adversarial networks and model- robust training algorithms to minimize the gap on both image and label side enhance robustness. 3) Our results on Cityscapes Cordts et al.) (2016) and Synthia Ros et al. (2016) datasets show that our method outperforms multiple state-of- the-art domain adaptation techniques, such as AdaptSegNet Tsai et al. (2020). ADVENT Vu et al. (2019) and FDA Yang & Soatto (2220).
2 RELATED WORK
Semantic Segmentation. In the past decades, Convolution Neural Networks(CNN) are widely used in semantic segmentation. Current state-of-the-an semantic segmentation frameworks are mostly developed from Fully Convolutional Network (FCN) by Long et al. (2015). ResNet proposed by He 20155) used a residual block to sum the nonlinear activation output and identity mapping, which proved improve the gradient propagation and increase the accuracy of semantic segmentation. DenseNet Huang et al. (2018a) builds upon ResNet and uses the concatenation of previous feature maps called dense block. This gives each layer in DenseNet information from all preceding layers. Jégou et al. (2017) extended DenseNet into FCN architecture. SegNet proposed by |Badrinarayanan ct al. (2017) introduces the deep convolution encoder-decoder architecture the field of semantic segmentation. Also, in Zhou et al. (015)). it is proved that empirical size of receptive field is much smaller than the theoretical size. Chen et al. (2016) and Yu & Koltun (2016) used dilated convolution to enlarge the receptive field. ParseNet by Liu et al. (2015) adds global context to CNNs for semantic segmentation. He et al.] (2014) introduced spatial pyramid pooling in DCNN. PSPNet introduced
by Zhao et al. (2017) uses novel global pyramid pooling module to capture both global context information. Duta et al. (2020) extended the idea by combining both local and global Pyramidal Convolution blocks in the neural network model. Chen et al. (2017b) introduced atrous spatial pyramid pooling. Depth image is also used along with original RGB information for semantic segmentation by Wang et al. (2019). For actual applications, /hao et al. (2018) proposed ICNet for real-time semantic segmentation accomplishing fast inference without sacrificing too much quality left behind. Azimi et al. (2020) proposed acrial perspective dataset for dense semantic segmentation.
Domain Adaptation. Combining with the techniques of Generative Adversarial Network (GAN) Goodfellow et al. (2014). Ganin & Lempitsky (2015) proposed DANN reduce the distribution gap between different domains by using discriminator to make the prediction cannot be identified between source and target domain. Pan et al. (2020) first separate target domain into splits based on entropy-based ranking and then deploy self self-super adaptation technique to reduce the domain gap between synthetic data and real images. PIT proposed by Lv et al.; (2020) constructs pivot information shared across domains. [Cenn et al. (2019) used depth image as guided information to build adaptation method from synthetic to real dataset. In the field of utilizing synthetic datasets to auxiliate the training process of real-world images, MUNIT proposed by Huang et al. (2018b) learn conditional distribution of target domain which can separate domain-invariant invariant semantic content of an image from Nomain-speciff properties. Vu proposed ADVENT model which maximize prediction certainty in target domain by introducing entropy loss Vu et al. (2019. Zhu proposed CycleGAN Zhu al. (2020) which was trying to learn a mapping such that the distribution of generated image is
2

---

Under review as a conference paper at ICLR 2022
indistinguishable from the target domain distribution using adversarial loss. AdaptSegNet proposed by Tsai et al. (2020) aimed to reduce the gap between the outputs from source and target domain given that images might be very different in appearance. FDA proposed by Yang & Soatto (2020)
uses Fast Fourier Transform(FFT) to adapt source and target domain.
Robustness. As for robustness against corrupted images, Hendrycks & Dietterich 2019) established rigorous benchmarks for image classification and proposed a series of image corruption examples. Kamann & Rother (220) showed that robustness increases with the performance of the semantic segmentation model and dense prediction cell was only designed to improve performance on clean data. Regarding robustness on semantic segmentation, |Amab et al. (2018) did the first evaluation of adversarial attacks on semantic segmentation and analyzed multi-scale processing. Robey et al.) (2200) proposed model-based robust training architecture focusing on the topic of image classifi- cation. WWong & Kolter (2020) bridged the gap between real-world perturbations and adversarial defenses by learning perturbation sets from data, through common image corruptions.
METHOD
In this section we propose a new robust training algorithm for semantic segmentation tasks. Par- ticularly, in section 3.1, we define the semantic segmentation task and the natural variations. In section 3.2, we give the outline of our proposed algorithm. section 3.3, we define semantic mean- ing invariance and clarify its connection to our algorithm. In section 3.4, we introduce the detailed training procedure of our model-basee robust adaptive training algorihm((MRTAdap))
3.1 SEMANTIC SEGMENTATION AND NATURAL VARIATION
Consider an input x H representing an RGB image with three channels and its correspond- ing label map E & which is annotated for every pixel in the image. A semantic segmentation task is to train a neural network that generates pixel-wise segmentation feature map IR "xW. We assume the dataset is drawn i.i.d. from distribution (x, ~ P. The optimization objective in semantic segmentation to find the best weight w that minimizes the loss function L(x, y; w) with respect to input label and weight w, which can be written as:
min 12
C(x, yı w)
Next we denote by E A a natural variation applied to r that is derived from a nuisance space A. A model of natural variation can be learned from the difference of source and target domain images. Particularly, we denote the natural variation model that transforms the input image from the source domain into an image in the (perturbed) target domain as V(z, 8). Note that, in general, it is hard to write natural variation model learned from data into an analytical form because there no closed form expression capturing the natural perturbation. Natural variation model can be customized regarding the needs of each dataset. In this paper, our goal is to design a semantic segmentation method that robust to natural variations 8.
3.2 ALGORITHM OVERVIEW
Our proposed model-based robust adaptive training algorithm (MRTAdapt) shown in Figure 1 has two modules: a segmentation network G using a model-based robust training algorithm to enhance robustness on image side, and a discriminator network D enhance robustness on label side. Two sets of images from source and target domain will have the notation of I, and 4,, while the image generated using natural variation model will be denoted as T After using source domain images to train the model, we transform images from source domain with natural variation model. Then we use simulated natural variation images to find the highest segmentation loss by applying the model-based robust training algorithm to optimize the segmentation network. Then we predict the segmentation softmax output for the unlabeled target domain images I by forwarding the target images into the same segmentation network. To minimize the gap between the output of natural variation images and target domain images, we use these two predictions as the input to the discriminator to distinguish whether the feature map generated by segmentation network is from natural variation model target
3

---

Under review as a conference paper at ICLR 2022
PSPNet (Generator)
Source Feature Map
Cori
Shared
seg
Training Images (Source)
Natural Variation Model
Robust Training Algorithim
PSPNet (Generator)
Natural Variation Feature Map
LNv
Shared
Discriminator
adv
larget Domain Images
PSPNot (Generator)
Target Festure Map
Figure I: Model-Based Robust Adaptive Training Algorithm (MRTAdapt)
domain. Using an adversarial loss for discriminator, the loss will back-propagate to segmentation network to help segmentation network generate similar distributions in target domain to natural variation. Detailed information will be covered in the following sections.
3.3 SEMANTIC MEANING INVARIANCE
Images under different kinds of natural variations have huge visual perspective difference. For domain adaptation problem computer vision, the motivation of most works is based on the fact that annotating images in new domain setting is time-consuming and extremely expensive. In this paper, we are trying to simulate the real-world setting when the model is trained with annotated images under normal condition in the source domain but natural variation conditions from target domain occur during inference time. Most of the current works focus on improving the performance from the image side. AdaptSegNet tries solve this problem from the prospective of the label side. The experiment conducted in this work has proved that spatial layout and local context of labels are similar for different datasets for semantic segmentation problem T'sai al. (2220).
Inspired by the idea of minimizing the gap on label size, we found the semantic information will remain the same from the label perspective regardless of the types of natural variation the image. For example, the car in the image under strong brightness will remain the semantic meaning as a car compared normal setting. Take Figure 2 as an example, left side is the same image with different brightness condition compared to the right side. Since the image is only changed into brighter version while all the objects remain the same, the semantic information will be unchanged under natural variation. In the context of semantic segmentation, unchanged semantic information means labels will be the same. Thus, labels the feature map will remain unchanged when we apply natural variation model on source domain image to transform into target domain.
Figure 2: Semantic meaning of an object will not change with natural variation
4

---

Under review as a conference paper at ICLR 2022
3.4 MODEL-BASED ROBUST ADAPTIVE TRAINING ALGORITHM
Different from AdaptSegNet where the goal was to minimize the gap between the label from source domain and target domain, we build top of model-based robust training algorithm with different optimization goal. Using the idea of semantic meaning invariance, we treat the generated natural variation image generated "source" domain image. For the reason that the generated natural varia- tion image is chosen by model-based robust training algorithm is aiming to represent the worst-case scenario of natural variation. The objective of our proposed model-based robust adaptive training algorithm is trying to reduce the gap on both image and label side.
Here we formulate the optimization task as a min-max problem, which will be minimizing the segmentation loss for source and natural variation images, while maximizing the probability of natural variation predictions being treated as target domain feature map. Our proposed algorithm an integration of model-based robust training algorithm and generative adversarial network.
max min C(I,, Jt, In)) D
G
3.4.1 MODEL-BASEH ROBUST TRAINING FORMULATION
The objective of the model-based robust semantic segmentation algorithm is to train the model so that it can be robust to natural variations such changes in the weather or lightning conditions. Given that only have labeled data for source domain images while labels for target domain are unavailable, our goal is to simulate and transform the normal condition training data into natural variation cases and then, the model can be trained until generates high accuracy prediction under abnormal conditions. In general, the concept of this algorithm can be treated as a data augmentation method that widens the range of conditions under which the model can attain high performance. The objective of the robust model-bases training algorithm can be formulated as Robey et al. (2020):
L = min W
Σ
max (V 6), SEA
The inner maximization represents the loss inferred by the natural variation model. Given an input image from source domain, we need search the nuisance space A for a local optimal parameter 5 that generates the worst-case scenario of natural variation. The optimal nuisance parameter is denoted as 8" that infers biggest loss between prediction and original label map. The objective of the inner maximization problem is to maximize the segmentation loss of the model under natural variation. The objective outer minimization problem is to train the weights of the model w to be robust against worst-case natural variation condition upon convergence, which represented in the form of minimizing the sum of the loss in this case.
During actual implementation, ber N. This means a dataset being recognized as training set will contain N image-label pairs: {zi, 3i L. Also, the natural variation model being applied in the algorithm learned from data. Model-based Robust Training generates random domain-specific style code for input images from source domain and filters the style codes which infer maximal loss out for weights update. That means, for every mini-batch size of input images (x. y) from source domain, we apply the natural variation model to transform the image into target domain, where we will treat the transformed im- age and original label new image-label pair denoted as (V(r,, 8), v). Then we utilize both pairs of datapoints to train the network for the outer minimization problem.
assume the number of datapoints being trained is finite num-
3.4.2 NETWORK TRAINING PROCEDURES
For the segmentation network G, we define the segmentation loss weighted sum from both the loss inferred from source domain images as well as the highest loss derived from natural variation images using model- robust training algorithm. Two losses are connected via An and Aoni which represent the contributions of generated natural variation images and source domain images the segmentation network respectively:
Leeg (1oy Inv) = Any Lnv + Cori min
Anv max (2, w) + )on A
(4)
5

---

Under review as a conference paper at ICLR 2022
Algorithm Model-Based Robust Training Algorithm
1: 8* O, Lmax f- 0
2: foreach Batch (Ir, y) do
3: for k steps do
4: Sample from A uniformly random
5: Leutrent f- L(V w)
6:
if Lecnrent > Imax then
7:
Lemax Leurrent 8 +
8:
9:
end if
10:
end for
(v + Aor y)
12:
end
For target domain images, we feed them into segmentation network and obtain an adversarial loss, which aims to fool the discriminator and maximize the probability of target domain predictions Pa being considered natural variation predictions. For the reason that we do not have the ground truth feature map of target domain images, segmentation loss will not be calculated.
Lude (I)) = -4)=-im log (D (P.)
(5)
h,
For the discriminator training. we forward a given segmentation softmax output P to a fully- convolutional discriminator using cross-entropy loss for two classes (natural variation and target domain). The loss can be written as:
Ca(P) - E0 (D(P)(h)u))) log
(6)
So the min-max problem can be rewritten into the following form, where segmentation loss Long and adversarial loss Ladv is summed by applying corresponding weights representing the contribution the network:
max min
Inv) + Aadufadd
D
4 IMPLEMENTATION
4.1 DATASET PREPROCESSING
Brightness Characteristic of an image can be represented using Snte-Saturaiion- Value (HSV), where hue, saturation and value represent the color, grayness and brightness respectively Latifah et al. (020). In this paper, we use natural variation model to simulate target domain images. To achieve better simulation result, we arrange our data preprocessing for brightness condition into two phase: Phase I: divide original dataset into three subsets: bright, medium and dark by calculating the av- erage Aine-Satration- Value of each image. In this paper, we divide the three subsets by their HSV values with 72 and 90 representing the boundary of dark-medium and medium- dataset, using 75 and 130 Synthia dataset. For training set, use bright subset as source domain and dark subset target domain to simulate the fact that most current dataset is constrained in day- time condition. Dark subset referring to the case when a vehicle is moving nighttime under low brightness condition. There are 931 adjusted source domain images representing bright condi- tion, and 486 target domain images representing dark condition for Cityscapes dataset. For Synthia dataset, bright and dark images are 1200 and 1174 respectively. Phase 2: adjust the overall bright- ness of bright subset by applying OpenCV library to enlarge the difference of brightness between training set and testing set.
on Cityscapes

---

Under review as a conference paper at ICLR 2022
Snow For snow weather natural variation, eliminate the effect of brightness and rearrange the training set with different configuration. For Cityscapes, we have source domain subset with 1046 images and target domain subset with 1064 images. For Synthia, source domain and target domain have 1000 and 1 1170 images respectively. Both subsets contain images in bright and dark condition, which allows the network to be exposed to images from different brightness condition. So the natural variation of brightness effect discussed in previous section is minimized. We remained unchanged for source domain images, referring to the case that we train our model in normal condition. Then we apply the snow corruption operation being used in ImageNet-C with severity of to target domain images, referring to the case that our model would be exposed to natural variation images during inference. Due to the limitation of computing resources, the image size being trained in this paper is relatively small. So we apply a proper level of severity for snow effect so that the performance will not break down completely.
Brightness
Snow
MUNIT
MUNIT
Figure 3: Example frames of both natural variations
4.2 IMPLEMENTATION DETAILS
In this paper, natural variation model powered by MUNIT Huang et al. (2018b) on Cityscapes dataset and RAIN proposed by Luo et al. (2020) on Synthia dataset.
PSPNet With the natural variation model derived from either known a priori or model learning from data. For the latter one, we apply MUNIT Cityscapes dataset and RAIN on Synthia dataset, we use PSPNet Zhao et al. (017) to do semantic segmentation. In PSPNet, we use ResNet He et al. (2015) model as backbone with the dilated network method Yu & Koltun (216)), then plug in the robust training module to generate new datapoints and obtain feature map using the weighted combination. Pyramid pooling module with 4 level representing different size of the sub-regions in the feature map is applied to gather both local and global region information. Then we concatenate the feature map obtained by pyramid pooling module with original feature map to generate final feature map.
MUNIT For MUNIT [Huang et al. (2018b) model that captures natural variation model, we set the training iterations into 300000. Weight decay is set to 0.0001 with image size fixed at 200 * 200. The architecture of MUNIT is consisted of two auto-encoders, one for each domain. For each auto- encoder. it is composed by one content encoder, one style encoder and one decoder. Content encoder consisted of strided convolutional layers for down-sampling purposes followed by residual blocks. Instance normalization is attached at the end of each convolutional layer in content encoder. Style encoder first down-samples the images by convolutional layer, then global pooling and fully con- nected layer are applied. For the decoder, style code being generated is fed to multilayer perceptron to generate parameters for Adaptive Instance Normalization (AdaINN. The AdaIN is applied to the residual blocks which are processing the content code. The output will then be ua-sampled and serve as the final reconstructed image. LSGAN is serving as discriminator while VGG is applied for the domain- inniiiinn perceptual loss calculation.
7

---

Under review as a conference paper at ICLR 2022
RAIN For RAIN Luo et al. (2020) model proposed by Luo. We set the training iterations to 100K and learning rate as 0.0001. This model is the composition of AdaIN Huang & Belongie (2017) and
auto-encoder. For AdaIN, VGG model is applied as encoder.
EXPERIMENT
Due to computation resources limitation, the experiments in this paper are done on Google Cloud Platform (GCP) using 16 GB of memory and one NVIDIA Tesla T4 GPU with 16 GB GDDR6 of graphic memory. The operating system being used in the experiment is Ubuntu 18.04. The code implemented based on Python 3.6 and Py 1.5.1. For fair comparison, all the images being trained in this paper will have the same size of 201 * 201 with batch size of 8 for all the methods being applied in the experiment. For evaluation metries, we use pixel-wise accuracy and mean of class-wise intersection over union (mloU) to define evaluation metrics. Pixel-wise accuracy is calculated using the number of pixels common between prediction output and ground truth divided by the number of pixels in ground truth feature map. The Intersection over Union metric is also named as Jaccard index to calculate the overlap between prediction and ground truth. To be specific, IoU (IoU Area of Overlup) calculates the number of pixels common between prediction output and Area of Union
ground truth divided by number of pixels across both feature maps. Inspired by /Zhao et al. (2017), we use ResNet-101 to generate basic feature map. ResNet-101 keeps a proper balance between high performance and proper complexity. Serving as baseline of the experiment, PSPNet using source domain images as training set and test the performance in target domain. Source domain only PSPNet tends to present how natural variation affects the performance in normal configuration. To compare our model in a comprehensive way, we also evaluate the performance of PSPNet when it is trained using labeled images from target domain (dark condition) and test the performance using images from target domain. We train all models for 200 epochs with basic learning rate of 0.01. For the MUNIT network being trained in this experiment, of 1, weight decay of 0.0001 and learning rate of 0,0001 with step side of 10000 referring to the frequency learning rate decaying.
train and test
batch size
We compare our method with several recent domain adaptation methods, in- cluding AdaptSegNet Tsai et al. (2020). ADVENT Vu et al. (2019), and FDA Yang & Soatto (2000) AdaptSegNet introduces generative adversarial network technique to reduce the gap on the label side of source domain and target domain. Two methods are proposed in ADVENT, namely MinEnt and AdvEnt. MinEnt maximize prediction certainty in the target domain by using a proposed entropy loss fent- AdvEnt shares similarity with the concept of AdaptSegNet and minimize the entropy by making target entropy distribution similar to the source domain. We evaluate the performance of both methods being proposed by ADVENT in this paper. FDA utilize Fast Fourier Transform (FFT) and replace the low-level frequencies of target domain images into source domain images. From the original literature of these proposed methods, the semantic segmentation baseline being applied DeepLab Chen et al. (2017a) with ResNet101 using SGD, To compare the performance from the same backbone of segmentation network, we replace the DeepLabV2 into PSPNet in this paper. All the configurations including initial learning rate of 0.1, momentum of 0.9 and weight decay of 0,0001 keep unchanged.
Comparison Methods
Cityscapes
From Table [ Segmentation model that only trained with source domain images will have huge performance drop during the inference time under natural variations. Model-based robust training algorithm (MRT) has 11% and 7% of improvement in terms of IoU compared to the model only trained with source domain images. The other domain adaptation methods such as AdaptSegNet, AdvEnt and MinEnt have stable and better performance compared to the model-based robust training algorithm. FDA method has 15% of improvement in mloU compared to source only PSPNet for brightness case yet not promising result snow effeetss Our proposed method has 16% and 12% of mIoU increase compared to baseline, which outperforms all other comparison methods. For brightness condition, our proposed MRTAdapt even has higher accuracy than segmentation model trained with target domain images.
Synthia For Synthia dataset, we apply RAIN model in stead of MUNIT model to capture natu- ral variation effect. As results shown in Table 2 comparing with all other methods, our proposed MRTAdapt outperforms other state-of-the-art domain adaptation models and also reach similar per-
8

---

Under review as a conference paper at ICLR 2022
Brightness
Snow
method
mloU
mAcc
AllAcc
mloU
mAcc
AllAcc
PSPNet(Source)
0.2221
0.2663
0.7801
0.2230
0.2731
0.7785
MRT
0.3384
0.4037
0.8642
0.2928
0.3511
0.8372
AdaptSegNet
0.3509
0.4061
0.8476
0.3222
0.3849
0.8356
AdvEnt
0.3570
0.4153
0.8714
0.3091
0.3787
0.8250
MinEnt
0.3536
0.4088
0.8642
0.3254
0.3947
0.8492
FDA 0.3706
0.4308
0.8804
0.1892
0.2348
0.7175
ORTA
0.3804
0.4453
0.8805
0.3443
0.4108
0.8591
PSPNet(Target) 0.3753
0.4288
0.8922
0.3936
0.4466
0.8959
Table 1: Performance Comparison for Brightness and Snow on Cityscapes
formance to the case when we have the ground truth label of target domain. Meanwhile, our method shows high robustness in different abnormal conditions.
Brightness
Snow
method
mloU
mAcc
AllAcc
mloU
mAcc
AllAcc
PSPNet(Source)
0.4910
0.5741
0.8567
0.2675
0.3436
0.7490
AdaptSegNet
0.4506
0.5278
0.8555
0.2471
0.3279
0.7367
AdvEnt
0.3250
0.4018
0.7956
0.3503
0.4322
0.8272
MinEnt 0.3049
0.3944
0.7698
0.2526
0.3280
0.6990
FDA 0.5888
0.6521
0.9159
0.2895
0.3758
0.7621
MRTs 0.6861
0.7577
0.9430
0.4516
0.5292
0.8788
PSPNet(Target) 0.7178
0.7076
0.9540
0.3173
0.4144
0.7494
Table Performance Comparison for Brightness and Snow on Synthia
CONCLUSION
In this paper, we addressed the problem of fragility in current learning-basse semantic segmenta- tion methods against natural variation effects that typically occur in real-world conditions. Using industry-level segmentation architecture PSPNet as backbone network, we adopted a model-based robust training algorithm that originally proposed for image classification problem design a robust semantic segmentation method. By applying a model which can capture the changes that li within source and target domain, we challenge the network with the worst-case scenario natural variation images by finding the biggest loss inferred. Also, building upon the AdaptSegNet that min- imizes the distribution gap of source and target domain from the label perspective, we showed that natural variation will not alter the semantic meaning of the label. Based on this idea, we proposed a model- -ased robust adaptive training algorithm that achieves higher performance than several do- main adaptation methods do. Our future research directions will focus on reducing and eventually climinating the dependence of the proposed method on the performance of the model that captures natural variation effects
REFERENCES
Anurag Arnab, Ondrej Miksik, and Philip H. S. Torr. On the robustness of semantic segmentation models to adversarial attacks, 2018
Seyed Majid Azimi, Corentin Henry, Lars Sommer, Arne Schumann, and Eleonora Vig. Skyscapes fine- grained semantic understanding of aerial scenes, 2020.

---

Under review as a conference paper at ICLR 2022
V. Badrinarayanan, A. Kendall, and R. Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 39(12)244812244 2017. doi: 10. (0.1109/PAAM2220.26644000
Liang- Cang-Chiehhhh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected crfs, 2016.
Liang-Chiae Chen, George Papandreou. lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs, 20
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation, 2017b.
Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach, 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding, 2016.
Ionut Cosmin Duta, Li Liu, Fan Zhu, and Ling Shao. Pyramidal convolution: Rethinking convolu- tional neural networks for visual recognition, 2020.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation, 2015.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convo- lutional networks for visual recognition. Lecture Notes in Computer Science, pp. 346-361, 2014. ISSN 1611-3349, doi: 10. 1-00099778---19--057------ 1007 1
URL http: //x: doi org/1C.
__2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations, 2019.
Hsien-Kas Hsin, Usiao-Chien Chiu, Chun-Chen Lin, Chih-Was Wei Chen, and Pei-Kuei Tsung. Ad- versarial self-training with domain mask for semantic segmentation. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 3689-3695. IEEE, 2019.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2018a.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization, 2017.
Xun Huang, Ming-Yu Yu Liu, Serge Belongie, and Jan Kautz, Multimodal unsupervised image-to- image translation, 2018b.
Wei-Chih Hung, Yi-Hsuan Tsai, Yan-Ting Liou, Yen- Yu Lin, and Ming-Hsuar Yang. Adversarial learning for supervupe semantic segmentation, 2018.
Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation, 2017.
Christoph Kamann and Carsten Rother. Benchmarking the robustness of semantic segmentation models. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), Jun 2020. doi: 10.1 00.109/0pp426000220000000
URL lttt:://e /x doi ottp./o.gg//0........
CVPR426000 2020. 00885
10

---

Under review as a conference paper at ICLR 2022
K Latifah, A Rochim, Joko Siswanto, and B Supriyadi. The characteristics of bamboo image traction as an environmentally friendly material. Journal of Physics: Conference Series, 1567: 032035, 06 2020. doi: 04000//174426656///6770000335330
Wei Liu, Andrew Rabinovich, and Alexander C. Berg. Parsenet: Looking wider to see better, 2015.
Jonathan Long, Evan Shelhamer, and Trevor Darrel.. Fully convolutional networks for semantic segmentation, 2015.
Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek. Semantic segmentation using adversarial networks, 2016.
Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, and Yi Yang. Adversarial style mining for one-shot unsupervised domain adaptation. Advances in Neural Information Processing Systems, 2020.
Lv, Liang, X. Chen, and G. Lin. Cross-domain semantic segmentation via interactive relation transfer. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 4333-4342, 2020. doi: 10.1099CCVP422000220000044
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Under- standing adversarial attacks on deep learning based medical image analysis systems, 2020.
Luke Oakden- Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Re. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. Proceedings of the ACM Conference on Health, Inference, and Learning, Apr 2020. doi: 10. 00.114553333553384466.
URL http:://oo.......oo/.//////////3
3384468
Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision, 2020.
Alexander Robey, Hamed Hassani, and George J. Pappas. Model-based robust deep learning, 2020.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio Lopez. The SYN- THIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. 2016.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses, 2020.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation, 2020.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthicu Cord, and Patrick Pérez, Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation, 2019.
Hengli Wang, Yuxiang Sun, and Ming Liu. Self-supervised drivable area and road anomaly segmentation using rgb-d data for robotic wheelchairs. IEEE Robotics and Automation Let- ters, 4(4):4386 4393, Oct 2019. ISSN 2377-3774. doi: 00.1109//a.2011.29222 http://dx. doi org 10. 1109/1RA. 2019. 2932877722274
URL
Eric Wong and Zico Kolter. Learning perturbation sets for robust machine learning, 2020.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection, 2017.
Yuan Xue, Tao Xu, Han Zhang, L. Rodney Long, and Xiaolei Huang. Segan: Adversarial network with multi-scale 11 loss for medical image segmentation. Neuroinformatics, 16(3-4):383-39) May 2018. ISSN 1559-00889 doi: 10. 10.10077/12221--18-99777
URL http://dx.dot
Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation, 2020.
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions, 2016.

---

Under review as a conference paper at ICLR 2022
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network, 2017.
Hengshuang Zhao, Xiaojuan Qi. Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 405-420, 2018.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene enns, 2015.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2020.
12

---

Under review as a conference paper at ICLR 2022
7 APPENDIX
Image
Ground Truth
the
PSPNet (Source)
PSPNet (Target)
MinEnt
AdvEnt
AdaptSegNet
FDA
MRT
MRTADV
Table 3: Brightness Test Example Cityscapes
13

---

Under review as a conference paper at ICLR 2022
Image
Ground Truth
H
PSPNet (Source)
PSPNet (Target)
MinEnt
AdvEnt
NdaptSegNet
FDA
MRT
MRTADV
Table 4: Snow Test Example- Cityscapes
14

---

Under review as a conference paper at ICLR 2022
Image
Ground Truth
PSPNet (Source)
PSPNet (Target)
MinEnt
AdvEnt
AdaptSegNet
FDA
MRTADV
Table Brightness Test Example Synthia
15

---

Under review as a conference paper at ICLR 2022
Image
Ground Truth
PSPNet (Source)
PSPNet (Target)
MinEnt
AdvEnt
AdaptSegNet
MRTADV
Table 6: Snow Test Example- Synthia
16


Reviews:
[{"review_id": "skpuWDA5NAQ", "reviewer": "Reviewer_9pKw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "--Strengths--\n\nThe problems being addressed here are real, and are important -- principled solutions and improvements for model robustness with respect to semantic image segmentation are of high value to the community and additionally likely result in improved performance in many practical applications. I encourage the authors to think about these problems. Global structure of the manuscript is reasonable and system overview figures and pseudocode are appreciated, towards aiding understanding. \n\n--Weaknesses--\n\nThe paper considers an interesting problem (segmentation model robustness) yet unfortunately the composition would appear in a premature state making it difficult to fully grasp the size of the novel contributions. Phrasing issues and sloppy writing currently distract the reader.\n\nThere has been previous work on analyzing the influence of synthetic data quality and usage strategies for semantic segmentation; eg. Zhang et al. (2017).  Additional works also investigate which properties of synthetic training data help generalization to real data, across various tasks. See eg. Su et al. (2015), Movshovitz-Attias et al. (2016). Experimental evidence demonstrating the efficacy of how well synthetic data strategies can generalize to real-world data and imagery are often key to the strength of such works. I believe analogous investigation here would result in more convincing evidence towards the value of the approach. The current lack of such experimental validation leaves the submission somewhat lacking.\n\nMinor (non-exhaustive):\n\n* Section 2 of the manuscript makes an attempt at surveying related work however handfuls of previous papers are each dedicated a terse summary sentence, which may leave the reader somewhat unengaged. Explicitly contrasting these previous works with novel insight towards the problem at hand can help to build a compelling story. What are the key component(s) that are missing from the collective previous work? where are their failings? Explaining more clearly the valuable gap that your contributions, will in contrast, fill can serve to both strengthen the work and help your audience.\n\n* Contradictory statements serve to confuse the reader eg.\n\n 'lack of robustness prevents the application of learning-based semantic segmentation methods on safety-critical applications' ; and \n\n'There are many applications now being used have made great progress with the help of semantic segmentation, such as medical image processing [..], autonomous vehicles [..] and robotics [..].' (pg. 1).\n\n* Table row usage of bold font appears unconventional and rather autocratic.", "summary_of_the_paper": "This paper considers the problem of semantic image segmentation and endeavours to improve the performance of network-based approaches by enhancing model robustness to specific causes of image-space perceptual variance (eg. weather, illumination brightness). A training strategy that leverages generative (GAN) based components, data augmentation and domain adaptation ideas is employed towards encouraging models to represent generated images and target domain images, similarly. Resulting quantitative performance is reported across two standard datsets where comparisons are drawn with recent alternative works. ", "summary_of_the_review": "As stated, the direction is valuable and the ideas somewhat interesting however I feel this submission is currently in a premature state. Lack of clear exposition and thorough real-world experimental evidence weigh rather heavily in the rating. I encourage authors to work on some of the methodological, presentation suggestions.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c1sjOjp4inj", "reviewer": "Reviewer_ChGc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghts:\n+ The work presents a clear motivation for the problem tackled, highlighting the importance of achieving more robust segmentation methods to enable these techniques in real world applications that are very sensitive to mistakes.\n+ The introduction makes a clear statement of the contributions of the work\n+ Related work presents a nice summary of semantic segmentation models based on deep neural networks.\n+ The proposed method obtains better accuracy than other domain adaptation methods in the generated target domain images.\n\nWeaknesses:\n- Problem definition. It would be helpful for the reader to have a more explicit statement earlier in the paper (introduction?) about the fact that the authors model the problem of robustness in the segmentation as a domain adaptation problem: from original data to similar data but with “natural variations expected” applied to the original image domain. This is a core idea that can be learned along the paper, but earlier statement would help understand earlier why related work, evaluation, baselines, … are focused on domain adaptation techniques. \n\n- The related work misses details on two relevant aspects.  First, it presents a nice summary of relevant topics (segmentation, domain adaptation, robustness). However, in the domain adaptation, which is key to position the presented work with respect to the existing work, I miss a bit more organised discussion to point what is general related work on the topic and what is focused on the specific semantic segmentation task tackled in this work. Besides, one of the key components in this work is the proposed simulation of natural variations in the scene images, which is a form of image augmentation. Authors mention this relationship with image augmentation briefly in the method description (sec 3), but no mention until then. I think the related work should discuss a bit this topic as well.\n\n- The method needs to be more clearly explained, specially about how the natural variation model and the target domain images are obtained, how do they differ from each other and how realistic the assumptions made about them are.  \nFirst, authors mention several times while explaining the method (sec 1 and 3) that “A model for the natural variation can be learned from data”. This is a key assumption (as mentioned in the conclusions), which brings a question: to which extent is this realistic? it assumes that we have enough variation in the target domain data to learn this.  It’s not clearly detailed how this model to apply natural variations is learned. Is it part of the training process when learning the segmentation model? Is the GAN-like training mentioned not only to improve the feature maps generated in the segmentation network but also to learn this image generation? This is not clear in the current description.\n\nAnother unclear aspect is about the assumptions on the target domain data. I understood the target domain images are unlabelled. Then, to compare target feature map with natural variation feature map, does the approach assume that those images are aligned somehow with labeled data? i.e., they are different versions (weather, lighting) of the same scene? If this is the case, target domain images are not really unlabelled, since they are aligned to the labeled data. This can be the case in the experimentation, since target domain does not consist of independent images, but images from the training set modified with weather-change simulation effects. It’s not very clear what the assumptions are when reading the method description, one can only see these ideas when reading the experimental set up. \n\n- The experimental validation is not fully convincing on its current form. If I understood correctly, the target domain images are not “real” images but also generated with the preprocessing steps explained in section 4.1, applying certain appearance changes to the original (training) images using existing algorithms to simulate rainy or snowy conditions in the image lighting. \nI find (as previously mentioned) an unclear point here: how are these “simulated” target domain images used to learn the natural variation model?  Both target domain images and the “augmented” training images during training (Natural variation model generated images) come from the same original labeled set.  I see a logical intuition behind it (if I apply certain augmentation (variation) to the training data, if the validation set contains images with similar variation applied, the performance will be better), but it is not a novel insight and current experimentation seems a bit biased if that’s the case.  If I understand the approach, the ideal situation to validate the proposed method would be to use real images from a target domain that does not have the semantic labels (maybe the Mapillary Vistas dataset could be a good source of data? it has data over different seasons and weather conditions), and then apply natural variations learned from this domain to the original labeled domain (labeled but with not such variations included).\n\nMaybe a more detailed description of all this processing (maybe add some math expression?) would help to explain and distinguish these two cases, to present a more convincing validation set up:\n- what’s the natural variation model applied and how is it obtained?\n- what are the target domain images and how are they obtained?", "summary_of_the_paper": "This work presents a new training algorithm for semantic segmentation deep neural networks (MRTAdapt) which is designed to obtain models more robust to changes in the scene due to natural variations, such as weather or lighting conditions. \n\nThe work poses this problem as a domain adaptation task, from the original domain (original labeled data) to the target domain (data with appearance changes due to lighting or weather). \nThe adaptation is achieved by training with augmented data simulating the natural variations learned from data. The training proposed combines a minimization of the segmentation loss (using the existing labels) and a maximisation of the probability of the generated images to be identified as images from the target domain, following GANs-like training, to ensure that the feature maps of the generated images are as similar as possible to target domain image feature maps.\n\nIn the experiments, the lighting and weather changes are simulated by the authors using existing tools applied on two public benchmarks. The approach is compared to other domain adaptation techniques obtaining higher accuracy in the segmentation of the simulated target domains.", "summary_of_the_review": "I currently tend to reject. The problem tackled is relevant, and the intuitions behind the proposed approach interesting, however the technical novelty is limited and the experimental validation not sufficiently convincing on its current form as discussed in the weaknesses. Maybe authors can better discuss and explain the main concerns that hinder clarity on the novelty claims and the validation in the rebuttal.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YYlFoXFk13a", "reviewer": "Reviewer_kE5M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, the following are the strengths of the work:\n 1) The use of the generator-discriminator (min-max problem) approach for tackling the natural variations in semantic segmentation is an interesting idea. \n2) The focus on studying robustness with respect to natural variations (as compared to adversarial attacks) would be extremely useful in practice by eliminating the need for acquiring images in various environmental conditions for training the model. \n\nHowever, there are a few major limitations of this work:\n1) The proposed work utilizes two existing works (Robey et al 2020, and Tsai et al 2020) on domain adaptation for semantic segmentation. It will be better to include a summary of these works and clearly highlight how the proposed work is different from these works. \n2) It is not clear how the natural variations are modeled in this work. At Page 5 (fourth para), it states the “natural variation model…… is learned from the data (Which data?)”. In section 4.1, the images are manually perturbed by varying the brightness or using ImageNet-C (for snow). I believe the way the natural variations are modeled plays an important factor in the robustness of the algorithm.\n3) It is not clear what is the “Target Images” in this work? Does it refer to the perturbed images? If yes, why do only dark images are being considered as the target domain in Section 5 (Page 8, First paragraph).\n\nIn addition to the suggestions above, the following are some minor feedback:\n4)In this work, all the models were trained with the same learning rate and the same number of epochs. I would suggest performing hyper-parameter tuning for all the models by varying the learning rate, weight decay, etc. This might result in a slight improvement in the performance. \n5)It is interesting to note that FDA (Yang & Soatto 2020) performs poorly in the presence of snow, but there is no such effect observed in the dark images. If possible, it will be better to discuss this observation in more detail. This discussion might provide new insights into the performance of GANs. \n6)It will be better to include few images and the corresponding labels obtained using the proposed approach. It will help the reader visualize the performance of the algorithm.\n7) A minor suggestion: For better readability, it is recommended to arrange the existing methods in the same sequence in Tables 1 and 2.\n8) Even though the experimental details are provided (including the hyperparameters), I would also recommend the authors publicly release the code to ensure reproducibility. \n", "summary_of_the_paper": " The paper proposes a discriminator-generator-based approach for robust semantic segmentation of images against natural variations such as snow, day/night (brightness). The input training image is first modified (perturbation in brightness, generating images with snow), and then fed to PSPNet to generate feature map for these perturbed images. Similarly, the original images (without perturbation) are fed to the same PSPNet to generate the original source feature map. The segmentation loss is a weighted sum of the loss computed on the original features and perturbed features. The loss on perturbed features finds the optimal perturbation parameter that maximizes the loss between predicted and original labels, while the loss on original image features is the traditional segmentation loss. The discriminator module attempts to distinguish between the features generated by perturbed images or target images. The proposed work is compared with the existing works on two standard datasets, Cityscapes and Synthia. The experimental results demonstrate the proposed approach outperforms the existing methods in terms of mIoU.  The main novelty of the proposed work is the combined use of adversarial networks and model-based training (Robey et al 2020) to achieve a robust semantic segmentation in the presence of variation in brightness and snow in the scene. \n", "summary_of_the_review": "An interesting work proposing a semantic segmentation approach that is robust to variations in the lighting conditions (day/night) and the presence of snow in the scene.The main novelty of the proposed work is the combined use of adversarial networks and model-based training (Robey et al 2020) to achieve a robust semantic segmentation in the presence of variation in brightness and snow in the scene. However, a few important details about the approach is missing in the work. I would strongly recommend the authors address the limitations mentioned in Pt no 1,2 and 3. A clear definition/description of how the variations are modeled, what is the target images, and how work is different from the existing two works:  are needed before this work can be accepted for publication. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this paper receives negative reviews due to limited technical novelty and contributions. The reviewers discuss extensively on the merits of this work after the rebuttal phase. However, the authors' rebuttal does not address all the raised concerns. As such, the area chair agrees with the reviewers and does not recommend it be accepted at this conference
Decision: Reject