Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of adversarial domain randomization for urban scene semantic segmentation. Specifically, it proposes an adversarial augmentation method called AdvStyle that injects adversarial signals into the channel-wise mean and standard deviation of images. The style augmentation is used when training on synthetic images from Synthia and GTAV datasets. This paper demonstrates promising results when applied the trained model to real semantic segmentation benchmarks.
main_review: Strengths:
* This paper tackles a structured prediction problem (semantic segmentation) in the domain generation community.
* The idea is clearly stated and straightforward with source code included in the appendix.

Weaknesses:
* The technical novelty is very limited. The key idea (augmenting channel-wise mean and standard deviation) has been explored in the cited reference [MixStyle; Zhou et al., 2021]. The difference is that this paper constrains the augmentation within the first layer with an adversarial formulation. The reviewer fails to perceive a broader impact on the community from this technical contribution.
* At a high level, AdvStyle is a data augmentation method that generates unconstrained adversarial examples. In this sense, a prior work [**Ref1, Bhattad, et al., In ICLR 2020] is highly-related but not discussed in the paper. Bhattad et al. use a pre-trained colorization or texture-transfer network for adversarial example generation. Please elaborate on this in the rebuttal and incorporate the comparisons in the future version of the paper.
* This paper only conducts experiments on sim-to-real transfer but does not evaluate real-to-sim transfer or real-to-real transfer. This makes an unfair comparison to the baseline methods including [Choi et al., 2021]. Please elaborate on this in the rebuttal. The reviewer would kindly suggest incorporating these comparisons in the future version of the paper.

References:

* Ref1: Unrestricted adversarial examples via semantic manipulation. Bhattad et al., In ICLR 2020.
summary_of_the_review: Overall, this is a borderline paper with promising results. The major concerns include the limited technical novelty, missing references, and unfair experimental comparisons. I would like to see the rebuttal and comments from the other reviewers to make a final decision.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the domain generalization problem in semantic segmentation. Starting to synthetic source dataset, the authors propose to apply image-level style transfer to bridge the gap between source and unseen real dataset. Specifically, the image-level style is defined as the mean and variance of each image. And the proposed method learns to reconstruct scenes with their styles and further updates the segmentation model with both original images and reconstructed images.

In experiments, the authors show that the proposed method can:
1. achieve SOTA performance on multiple real datasets 
2. generalize well with various backbones
3. cover the data distributions of real dataset well
main_review: Strengths:
1. Complete and solid experiments 
2. The image-level style transfer part is intuitive 
3. Paper is easy to follow

Weakness:
The main concern I have is the model design and also not sure about some of the authors' intuitions behind:
1. Based on the algorithm in Appendix B, we only will update the adversarial learning once, or the mean and variance once, then we feed the updated mean and variance to robust model part to reconstruct the scene and then further update the segmentation model. My main concern is about what is really learnt in this one update? Personally, I feel that when updating only once, the generator cannot really learn much, e.g. the reconstructed results can be very far away from the original image and cannot be very meaningful. Then if at later stage we force the predictor to minimize the segmentation loss of both original and reconstructed images, will this introduce noise to our predictor, e.g. the predictor then cannot really do well on either image. And is this scenario possible and how does the proposed method handle this problem?
2. Is the segmentation model shared between adversarial and robust model learning process? I believe so, right? Then follow up for the first question, will the segmentation model be drifted away by the one-time update and then introduce additional errors to adversarial learning procedure?
3. How do we decide the maximum number of iterations in algorithm?
4. One corner/extreme case might be that the generator learns very well after many times of iterations and can perfectly reconstruct original images. Then under this scenario, the segmentation model do not need to learn anything but stay with what it has been initialized with would be the right answer for the model. Or we can relax a bit by thinking that what has been generated/reconstructed can be very close to the original images after many epochs and not much changes are needed for segmentation model. How will the proposed method handle this case?
5. What is the difference/relationship between the proposed method and papers like "LEARNING TO SIMULATE"? 
6. Why the proposed method can produces new styles that cover the distributions of different datasets as shown in Fig.5? Any explanation? 
summary_of_the_review: In general, I feel that the paper provides very good experimental results but I have concerns about their model design. I am willing to change my rating if the authors can address my concerns above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a domain generalization method for semantic segmentation, where one synthetic data is given as the source training dataset and the model is tested on unseen real datasets. The authors propose a simple yet effective method by introducing adversarial style augmentations. Specifically, the model decomposes the image into a style feature and a normalized image, and then uses the segmentation loss to update the style feature. With the updated style feature and the normalized image, it can resemble an augmented image, which is then used to optimize the segmentation model along with the original image via the segmentation losses. A few highlights of the proposed method are:
- It appears to be a plug-in module, so that it can be used for different methods. architectures, and tasks
- It shows state-of-the-art performance on the semantic segmentation task
main_review: Strength:
- The paper is very well-written and easy to follow
- The proposed method is simple yet effective with a good motivation
- The proposed augmentation can be integrated into different methods
- The paper shows good performance on two tasks: semantic segmentation and image classification

Weakness:
- The explanation of why the proposed augmentation performs well to cover unseen data distributions is not clear. For example, in Fig 5 of the t-SNE plot, GTAV is the only training data, but it can generate out-of-distribution samples (e.g., Adv GTAV), which does not seem very intuitive. Is it also because other data augmentations (color jittering and gaussian blur) are used? The authors should clarify this.

Experimental comparisons
- In Table 5, although there are different settings, the authors should try efforts to make fair comparisons, e.g., using the whole set of GTAV for training (setting I and II).
- The setting in the paper only uses one source dataset. It would be interesting to report results of using multiple source datasets (like the ISW paper using both GTAV and Synthia).

Minor issue
- The ERM baseline in Sec 4.5 should have the reference, instead of having it in the appendix.

##########################################################################

Post-rebuttal:
The rebuttal well addresses my main concerns. I would encourage the authors to release the code and models if the paper is accepted.
##########################################################################
summary_of_the_review: Although the proposed augmentation method is simple. it shows state-of-the-art performance for domain generalizable semantic segmentation. The proposed method can be also adopted in different methods, backbones, and tasks, which is beneficial to the field. However, the authors should address the raised concerns regarding the technical clarity and experimental comparisons.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper:  The paper presents an adversarial style augmentation approach (referred to as AdvStyle) with the objective of improving the generalization of models from a synthetic domain (training) to a real domain (testing), for the task of scene segmentation in the context of driving. The approach is based on the idea of adversarial style learning and the training of a model using both original and adversarial images. A strong set of experiments showcase the benefits of the approach with respect to the state of the art.

main_review: Pros: 
- Definitely, the main point in favor of this paper is the good results obtained by the approach. I believe the approach is well evaluated, making the right choices of synthetic datasets (training), real datasets (testing), semantic segmentation architectures, and baselines (e.g. ISW, which is one of the best performing approaches for domain generalization in semantic segmentation).

- The approach is orthogonal to other solutions, such as ISW. The combination of ISW and AdvStyle leads to even better accuracy in the downstream task (semantic segmentation).

- The technique is conceptually simple and easy to reproduce.

 
##########################################################################

Cons: 
 
- The t-SNE visualization (Figure 5) doesn’t seem to convey much information regarding what is happening during the training process. I understand it is typical to visualize features spaces in DA/DG papers, but in this case, it is not clear to me that the figure helps make any point. On the contrary, I would argue it distracts from the main point.
 
##########################################################################

Questions during rebuttal period: 
 
I don’t have major concerns with this paper. However, I have a recommendation/thought for the authors.
- Why focus just on driving scenes? It should be relatively straightforward to evaluate this approach in other types of scenes, for instance, indoor scenes, by leveraging NYU dataset + SceneNet or InteriorNet. This will help make an even stronger point.
 
#########################################################################
Miscellanea: 

- Figure 1 is cluttered and hard to understand. I would recommend removing b) and d).

- Figure 5, as mentioned above, doesn’t seem to help with the narrative.

summary_of_the_review: My vote is towards accepting this paper. Similar concepts have been applied in the past to improve domain generalization but never reaching the quality achieved by the proposed approach. It is exciting to see how the application of adversarial style learning can boost results even surpassing existing state-of-the-art solutions, especially considering the ease of implementation of this idea. The experiments section is quite strong, leveraging well-known datasets, relatively modern segmentation architectures, and very recent baselines.
review_title: null

Meta-review: This paper presents a domain generalization method for semantic segmentation. The model is trained on synthetic data (source) and is tested on unseen real datasets (target). The authors propose a simple data augmentation method, AdvStyle, generating unconstrained adversarial examples for the training on the source domain.

There was no consensus on the method among the reviewers. Several issues have been raised. After rebuttal and discussion, no one really changed her/his mind. The motivation of why focus just on driving scenes is still questionable. Definitively, it could be interesting to investigate further why it is not straightforward to have gains on other kinds of scenes. Finally, we encourage the authors to address the raised concerns regarding the discussion with previous works and the comparisons for future publication.
Decision: Reject