Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work aims to address the domain generalization problem in semantic segmentation. A simple and effective adversarial augmentation technic is proposed which changes global statistics at the image level. Through adversarial learning, the adversarial style features seem to capture the characteristics of other datasets successfully and improve the performance significantly.
strengths_and_weaknesses: Strengths
- The main idea of AdvStyle is simple and easy to implement. I believe this can benefit many computer vision tasks, especially the dense prediction tasks.

- The illustration of how style changes affect the segmentation performance in Figure 1 is good. It can be clearly observed that the mIoU decreases a lot when only changing scene colors. 

- The t-SNE visualization in Figure 5 provides good proof for the AdvStyle idea: the adversarial style features can capture some characteristics of other datasets. 

- Sufficient experiments have been done, including comparisons with common data-augmentation technics, SOTA DG methods and extra experiments on classification tasks.

Weaknesses
- In Figure 1, It is claimed that image-level mean-variance is used as the style feature. While it is not clear how to calculate and apply such style features.  In Line 70, it is claimed that “6-dim feature for each example”. So, I wonder if the style features are just mean & variance of R-G-B channels. If so, why not use other color spaces or even texture spaces? I think this is an important ablation study that is missing in the main paper.


questions: I want to double-check if the target domain data are not used at all during training. So the results in Table 1 are obtained by training model on the GTAV training data only, and then applied to other real-world validation data. Right?  

Since the channel-wise augmentation is applied to the global region, I wonder if a local augmentation can further improve the performance.

limitations: The limitations and potential negative societal impact have been well described.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper addresses the general problem of learning a model in one domain and testing it in a second domain, where there is some domain shift between the domains. Specifically, it demonstrates a method that improves testing performance on real, unknown data when learning on synthetic, rendered data in semantic segmentation for autonomous driving. The motivating observation is that in urban semantic segmentation for autonomous driving a concise domain shift is measurable in the per-channel mean and variance of the image data. In other words, the color statistics between different data sets (and acquisition conditions) differ. The authors propose to explicitly target adversarial changes of the mean and variance of the first domain images during training to improve generalization performance in a second, unknown, domain.

The method works in two phases for each batch iteration, where the mean and variance of the input image can be modified during the forward pass: adversarial updating of the mean/variance perturbation based on the current, frozen model prediction, and then regular updating of model weights with both adversarial as well as unperturbed samples.

The paper continues to evaluate the performance of the proposed method on one synthetic versus three real datasets in semantic segmentation for autonomous driving (GTAV versus CityScapes, BDD, Mapillary), as well as several choices for backbones and batch normalization schemes. The empirical performance results demonstrate significant improvements over the chosen baselines. Furthermore, the paper extends the results to image classification on two relevant benchmarks, "Digits" and and "PACS"
strengths_and_weaknesses: The paper is written well and the method is laid out with sufficient clarity and detail. The datasets chosen for comparison are relevant and realistic (in terms of potential real-world applications of the method). I appreciate the motivation: We observe measurable population differences between various datasets and that they can be captured in a compact and semantically meaningful vector already (image mean and variance). How can we use this prior knowledge to improve task performance in semantic segmentation? Based on this motivation, the authors conclude to model the observed perturbation explicitly as it happens to be a natural part of compact image formation models already (in the sense that normalization with these statistics is well known and commonly used). Once it is modeled, the authors show that it is a comparatively simple step to adversarially predict a new set of samples that are harder to predict at the current training state. An appealing property of the method is that the perturbation space is fairly low-dimensional (6-vector) and thus more easily characterized compared to high-dimensional perturbations for instance on the level of individual pixels.

A significant concern with the submission is that it poses its method as domain generalization (DG) rather than image augmentation (IA) and the subsequent analysis and choice of baselines. I believe this confounds the comparison to prior art and comparable methods. If I take the DG perspective I'd have to ask: Will this work for other DG tasks that do not make strong assumptions about the signal (image formation process, meaning of mean/variance)? I can find no evidence for this in the paper and it appears that the main premise is a strong domain assumption (the image formation process) and selection of data to conform with this assumption (i.e. the datasets are chosen such that they differ demonstrably in the chosen and explicitly modeled statistic). Alternatively, the message of the submission could be that for a set of domains, I can a priori examine some low-dimensional formative process that I then can exploit to generate better samples (in terms of generalization across the prior domains). This could be reformulated from the perspective of image augmentation (IA). If I take on the perspective of IA, I'd have to ask if the cited prior art, baselines and benchmarks are chosen appropriately. For prior art, there would be significant references missing. For instance, [1] and [2] below demonstrate generic, in-training adversarial augmentation. Conceivably, per-channel image mean and variance could be included in these approaches as well without fundamental changes to the algorithms. How would the presented method compare in this case? Could it be extended to other explicitly modeled image formation processes (luminance, in-plane shift, etc.)? 
As an example: Let's say my domain shift were in-plane rotations (e.g. relatively static car-based camera vs. smart phone from pedestrian view). The presented method would not be expected to demonstrate much benefit, as the domain assumption of shift in image mean and variance are violated. One could probably model the in-plane rotation with a similar adversarial approach to sample generation, but this essentially requires explicit knowledge of the domain shit in a low-dimensional, parametric, forward way. 

As it is presented, the empirical results are appealing as the method is relatively straightforward and improves over the presented baselines, but the level of contribution is unclear as the choice of baselines lack a thorough sample of and comparison to state of the art image augmentation methods.

```
[1] @inproceedings{
wang2021augmax,
title={AugMax: Adversarial Composition of Random Augmentations for Robust Training},
author={Haotao Wang and Chaowei Xiao and Jean Kossaifi and Zhiding Yu and Anima Anandkumar and Zhangyang Wang},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=P5MtdcVdFZ4}
}

[2] @inproceedings{
zhang2020adversarial,
title={Adversarial AutoAugment},
author={Xinyu Zhang and Qiang Wang and Jian Zhang and Zhao Zhong},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=ByxdUySKvS}
}
```
questions: I would like to hear the author's thoughts on my concerns on image augmentation as a fair baseline above.

Some minor comments that do not affect the rating:
- While inspecting tables 3 and 4, I noticed that the setup is described the same (Source/backbone), but the baselines differ. For instance, CityScapes changes from 21.64 to 28.95. It took a couple of reads to understand that the baseline in table includes Color Jittering and Gaussian Blur. Mentioning this fact in the exposure may make readings smoother.
- Table 3: Would it make sense to add a row combined "Ours" with "AdvPixel"? It may be interesting to see if these are complementary.

limitations: The limitations of the work are fairly clear from the submission.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper propose a new adversarial style augmentation strategy (AdvStyle) for domain generalization in the semantic segmentation task. The approach of AdvStyle can generate hard stylized images during training, preventing the model from overfitting on the source domain. The generation is completed via learning adversarial style feature. Experiments on two semantic segmentation benchmarks demonstrate the effectiveness of AdvStyle. 
strengths_and_weaknesses: Strengths:
The proposed approach can obviously improve the performance of baselines' performance on unseen domains, including segmentation and classification tasks.

Weakness:
1. The proposed AdvStyle is indeed an approach to extend the appearance diversity of the training images, and this approach can not ensure the performance on one unseen domain, i.e., this method lacks the interpretability.
And AdvStyle can improve the generalization ability via color augmentation, but more augmentation effects are needed, such as lighting for the generalization of a model from normal-light training data to low-light testing data.

2. Why not apply the AdvStyle on SOTA classification baselines? like ME-ADA in Table 6 and L2D in Table 7? I wonder whether the performance improvement is highly related with the baseline' performance.

3. The superiority compared with current SOTA augmentation method is not obvious. Like the MixStyle reported in Table 2 of supp. More comparison is needed and such comparisons should be placed in the main paper, not supp.




questions: 1. The proposed method updates the style feature via the direction of the gradient as shown in Eq. (3). This is very similar to the strategy of obtaining adversarial samples (adversarial perturbations). Thus, I wonder how to control the parameter of $\gamma$ in Eq.3? Why use both learning rate for $\mu$ and $\sigma$?
limitations: I agree with authors' stated limitations in the supp, which is the training time. The author can consider how to reduce the training cost.
review_title: null

Meta-review: metareview: Simple and practical way to do better at domain generalization when it comes to semantic segmentation. AdvStyle can generate images that are hard during training, and prevent the model from overfitting on the source domain. Given that it works well, is relatively simple to implement and conceptually sound, I think it will appeal to a large portion of the NeurIPS audience that works on domain generalization.
confidence: Certain
Decision: Accept