Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  The paper proposes a discriminator-generator-based approach for robust semantic segmentation of images against natural variations such as snow, day/night (brightness). The input training image is first modified (perturbation in brightness, generating images with snow), and then fed to PSPNet to generate feature map for these perturbed images. Similarly, the original images (without perturbation) are fed to the same PSPNet to generate the original source feature map. The segmentation loss is a weighted sum of the loss computed on the original features and perturbed features. The loss on perturbed features finds the optimal perturbation parameter that maximizes the loss between predicted and original labels, while the loss on original image features is the traditional segmentation loss. The discriminator module attempts to distinguish between the features generated by perturbed images or target images. The proposed work is compared with the existing works on two standard datasets, Cityscapes and Synthia. The experimental results demonstrate the proposed approach outperforms the existing methods in terms of mIoU.  The main novelty of the proposed work is the combined use of adversarial networks and model-based training (Robey et al 2020) to achieve a robust semantic segmentation in the presence of variation in brightness and snow in the scene. 

main_review: In my opinion, the following are the strengths of the work:
 1) The use of the generator-discriminator (min-max problem) approach for tackling the natural variations in semantic segmentation is an interesting idea. 
2) The focus on studying robustness with respect to natural variations (as compared to adversarial attacks) would be extremely useful in practice by eliminating the need for acquiring images in various environmental conditions for training the model. 

However, there are a few major limitations of this work:
1) The proposed work utilizes two existing works (Robey et al 2020, and Tsai et al 2020) on domain adaptation for semantic segmentation. It will be better to include a summary of these works and clearly highlight how the proposed work is different from these works. 
2) It is not clear how the natural variations are modeled in this work. At Page 5 (fourth para), it states the “natural variation model…… is learned from the data (Which data?)”. In section 4.1, the images are manually perturbed by varying the brightness or using ImageNet-C (for snow). I believe the way the natural variations are modeled plays an important factor in the robustness of the algorithm.
3) It is not clear what is the “Target Images” in this work? Does it refer to the perturbed images? If yes, why do only dark images are being considered as the target domain in Section 5 (Page 8, First paragraph).

In addition to the suggestions above, the following are some minor feedback:
4)In this work, all the models were trained with the same learning rate and the same number of epochs. I would suggest performing hyper-parameter tuning for all the models by varying the learning rate, weight decay, etc. This might result in a slight improvement in the performance. 
5)It is interesting to note that FDA (Yang & Soatto 2020) performs poorly in the presence of snow, but there is no such effect observed in the dark images. If possible, it will be better to discuss this observation in more detail. This discussion might provide new insights into the performance of GANs. 
6)It will be better to include few images and the corresponding labels obtained using the proposed approach. It will help the reader visualize the performance of the algorithm.
7) A minor suggestion: For better readability, it is recommended to arrange the existing methods in the same sequence in Tables 1 and 2.
8) Even though the experimental details are provided (including the hyperparameters), I would also recommend the authors publicly release the code to ensure reproducibility. 

summary_of_the_review: An interesting work proposing a semantic segmentation approach that is robust to variations in the lighting conditions (day/night) and the presence of snow in the scene.The main novelty of the proposed work is the combined use of adversarial networks and model-based training (Robey et al 2020) to achieve a robust semantic segmentation in the presence of variation in brightness and snow in the scene. However, a few important details about the approach is missing in the work. I would strongly recommend the authors address the limitations mentioned in Pt no 1,2 and 3. A clear definition/description of how the variations are modeled, what is the target images, and how work is different from the existing two works:  are needed before this work can be accepted for publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presents a new training algorithm for semantic segmentation deep neural networks (MRTAdapt) which is designed to obtain models more robust to changes in the scene due to natural variations, such as weather or lighting conditions. 

The work poses this problem as a domain adaptation task, from the original domain (original labeled data) to the target domain (data with appearance changes due to lighting or weather). 
The adaptation is achieved by training with augmented data simulating the natural variations learned from data. The training proposed combines a minimization of the segmentation loss (using the existing labels) and a maximisation of the probability of the generated images to be identified as images from the target domain, following GANs-like training, to ensure that the feature maps of the generated images are as similar as possible to target domain image feature maps.

In the experiments, the lighting and weather changes are simulated by the authors using existing tools applied on two public benchmarks. The approach is compared to other domain adaptation techniques obtaining higher accuracy in the segmentation of the simulated target domains.
main_review: Strenghts:
+ The work presents a clear motivation for the problem tackled, highlighting the importance of achieving more robust segmentation methods to enable these techniques in real world applications that are very sensitive to mistakes.
+ The introduction makes a clear statement of the contributions of the work
+ Related work presents a nice summary of semantic segmentation models based on deep neural networks.
+ The proposed method obtains better accuracy than other domain adaptation methods in the generated target domain images.

Weaknesses:
- Problem definition. It would be helpful for the reader to have a more explicit statement earlier in the paper (introduction?) about the fact that the authors model the problem of robustness in the segmentation as a domain adaptation problem: from original data to similar data but with “natural variations expected” applied to the original image domain. This is a core idea that can be learned along the paper, but earlier statement would help understand earlier why related work, evaluation, baselines, … are focused on domain adaptation techniques. 

- The related work misses details on two relevant aspects.  First, it presents a nice summary of relevant topics (segmentation, domain adaptation, robustness). However, in the domain adaptation, which is key to position the presented work with respect to the existing work, I miss a bit more organised discussion to point what is general related work on the topic and what is focused on the specific semantic segmentation task tackled in this work. Besides, one of the key components in this work is the proposed simulation of natural variations in the scene images, which is a form of image augmentation. Authors mention this relationship with image augmentation briefly in the method description (sec 3), but no mention until then. I think the related work should discuss a bit this topic as well.

- The method needs to be more clearly explained, specially about how the natural variation model and the target domain images are obtained, how do they differ from each other and how realistic the assumptions made about them are.  
First, authors mention several times while explaining the method (sec 1 and 3) that “A model for the natural variation can be learned from data”. This is a key assumption (as mentioned in the conclusions), which brings a question: to which extent is this realistic? it assumes that we have enough variation in the target domain data to learn this.  It’s not clearly detailed how this model to apply natural variations is learned. Is it part of the training process when learning the segmentation model? Is the GAN-like training mentioned not only to improve the feature maps generated in the segmentation network but also to learn this image generation? This is not clear in the current description.

Another unclear aspect is about the assumptions on the target domain data. I understood the target domain images are unlabelled. Then, to compare target feature map with natural variation feature map, does the approach assume that those images are aligned somehow with labeled data? i.e., they are different versions (weather, lighting) of the same scene? If this is the case, target domain images are not really unlabelled, since they are aligned to the labeled data. This can be the case in the experimentation, since target domain does not consist of independent images, but images from the training set modified with weather-change simulation effects. It’s not very clear what the assumptions are when reading the method description, one can only see these ideas when reading the experimental set up. 

- The experimental validation is not fully convincing on its current form. If I understood correctly, the target domain images are not “real” images but also generated with the preprocessing steps explained in section 4.1, applying certain appearance changes to the original (training) images using existing algorithms to simulate rainy or snowy conditions in the image lighting. 
I find (as previously mentioned) an unclear point here: how are these “simulated” target domain images used to learn the natural variation model?  Both target domain images and the “augmented” training images during training (Natural variation model generated images) come from the same original labeled set.  I see a logical intuition behind it (if I apply certain augmentation (variation) to the training data, if the validation set contains images with similar variation applied, the performance will be better), but it is not a novel insight and current experimentation seems a bit biased if that’s the case.  If I understand the approach, the ideal situation to validate the proposed method would be to use real images from a target domain that does not have the semantic labels (maybe the Mapillary Vistas dataset could be a good source of data? it has data over different seasons and weather conditions), and then apply natural variations learned from this domain to the original labeled domain (labeled but with not such variations included).

Maybe a more detailed description of all this processing (maybe add some math expression?) would help to explain and distinguish these two cases, to present a more convincing validation set up:
- what’s the natural variation model applied and how is it obtained?
- what are the target domain images and how are they obtained?
summary_of_the_review: I currently tend to reject. The problem tackled is relevant, and the intuitions behind the proposed approach interesting, however the technical novelty is limited and the experimental validation not sufficiently convincing on its current form as discussed in the weaknesses. Maybe authors can better discuss and explain the main concerns that hinder clarity on the novelty claims and the validation in the rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of semantic image segmentation and endeavours to improve the performance of network-based approaches by enhancing model robustness to specific causes of image-space perceptual variance (eg. weather, illumination brightness). A training strategy that leverages generative (GAN) based components, data augmentation and domain adaptation ideas is employed towards encouraging models to represent generated images and target domain images, similarly. Resulting quantitative performance is reported across two standard datsets where comparisons are drawn with recent alternative works. 
main_review: --Strengths--

The problems being addressed here are real, and are important -- principled solutions and improvements for model robustness with respect to semantic image segmentation are of high value to the community and additionally likely result in improved performance in many practical applications. I encourage the authors to think about these problems. Global structure of the manuscript is reasonable and system overview figures and pseudocode are appreciated, towards aiding understanding. 

--Weaknesses--

The paper considers an interesting problem (segmentation model robustness) yet unfortunately the composition would appear in a premature state making it difficult to fully grasp the size of the novel contributions. Phrasing issues and sloppy writing currently distract the reader.

There has been previous work on analyzing the influence of synthetic data quality and usage strategies for semantic segmentation; eg. Zhang et al. (2017).  Additional works also investigate which properties of synthetic training data help generalization to real data, across various tasks. See eg. Su et al. (2015), Movshovitz-Attias et al. (2016). Experimental evidence demonstrating the efficacy of how well synthetic data strategies can generalize to real-world data and imagery are often key to the strength of such works. I believe analogous investigation here would result in more convincing evidence towards the value of the approach. The current lack of such experimental validation leaves the submission somewhat lacking.

Minor (non-exhaustive):

* Section 2 of the manuscript makes an attempt at surveying related work however handfuls of previous papers are each dedicated a terse summary sentence, which may leave the reader somewhat unengaged. Explicitly contrasting these previous works with novel insight towards the problem at hand can help to build a compelling story. What are the key component(s) that are missing from the collective previous work? where are their failings? Explaining more clearly the valuable gap that your contributions, will in contrast, fill can serve to both strengthen the work and help your audience.

* Contradictory statements serve to confuse the reader eg.

 'lack of robustness prevents the application of learning-based semantic segmentation methods on safety-critical applications' ; and 

'There are many applications now being used have made great progress with the help of semantic segmentation, such as medical image processing [..], autonomous vehicles [..] and robotics [..].' (pg. 1).

* Table row usage of bold font appears unconventional and rather autocratic.
summary_of_the_review: As stated, the direction is valuable and the ideas somewhat interesting however I feel this submission is currently in a premature state. Lack of clear exposition and thorough real-world experimental evidence weigh rather heavily in the rating. I encourage authors to work on some of the methodological, presentation suggestions.
review_title: null

Meta-review: Overall, this paper receives negative reviews due to limited technical novelty and contributions. The reviewers discuss extensively on the merits of this work after the rebuttal phase. However, the authors' rebuttal does not address all the raised concerns. As such, the area chair agrees with the reviewers and does not recommend it be accepted at this conference
Decision: Reject