Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation

Abstract: In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn a robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) domains. Our study finds that the image style variation can largely influence the model's performance and the style features can be well represented by the channel-wise mean and standard deviation of images. Inspired by this, we propose a novel adversarial style augmentation (AdvStyle) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training. The learned adversarial style feature is used to construct an adversarial image for robust model training. AdvStyle is easy to implement and can be readily applied to different models. Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model performance on unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.

Reviews and author responses:

 Dear Reviewer NDMW,

Thanks for your kind reply. We are delighted that you appreciate our response. We definitely will add all these additional experiments into the final version. We also fully agree with you that the multi-resolution spatial pyramid strategy can bring further improvement and would like to leave it as our extension in future work. We would highly appreciate it if you consider increasing your confidence and upgrading the score based on our proper feedback.

Best,

Authors of Paper #444

 Thanks for the authors’ effort in the feedback. All my concerns have been answered. I would suggest adding these additional experiments into the final version. In addition, if the 4 patches strategy can improve performance from 37.39 to 37.76, I would expect the multi-resolution spatial pyramid strategy can further boost the performance. 

 Dear Reviewer Z9RR,

We sincerely appreciate your review work, which significantly helps us improve our paper. We have carefully addressed each of your concerns in the response. Please let us know if you have any further questions or concerns. We are happy to clarify them.

Many thanks for your comments again.

 Dear Reviewer B93V,

We sincerely appreciate your review work, which significantly helps us improve our paper. We have carefully addressed each of your concerns in the response. Please let us know if you have any further questions or concerns. We are happy to clarify them.

Many thanks for your comments again.

 Dear Reviewer NDMW,

We sincerely appreciate your review work, which significantly helps us improve our paper. We have carefully addressed each of your concerns in the response. Please let us know if you have any further questions or concerns. We are happy to clarify them.

Many thanks for your comments again.

 Thanks for your valuable comments. Please find our detailed response below.

>**Comment 1:** Interpretability of AdvStyle.
>
>**Response 1:** Thanks for these constructive comments. We would like to explain the effectiveness of our AdvStyle in two aspects.
>>
>First, as stated by [34], the domain generalization task can be formulated as solving the worst-case problem, and the solution of the worst-case problem guarantees good performance against data distributions that are different from the source domain. In other words, training the model with data that have different distributions from the source data but with realistic/reasonable variations can improve the performance on unseen domains. In this paper, we follow this statement and design our AdvStyle for learning generalized models. From the visualizations (in Figure 2 of the main paper and more figures in the Appendix), we can observe that our AdvStyle can generate realistic variations that are different from the source domain. This property largely guarantees our AdvStyle to meet the important condition of solving the worst-case problem. Therefore, it is reasonable that training the samples generated by AdvStyle can improve the performance on unseen domains.
>>
> Second, to better understand the benefit of AdvStyle, we have provided a quantitative analysis of the distribution of different datasets in the Appendix (see Section 7 and Table 4). For your convenience, we provide the details below.
>>
> Specifically, we computed the histograms of pixel values of four datasets (GTAV, CityScapes, BDD-100K, Mapillary) and the AdvStyle-augmented dataset of GTAV, which is generated by four epochs. The bin size is set to 8. For each dataset, the histograms of RGB channels are normalized by L1-norm and re-scaled (×) by #bins, and then are concatenated as the histogram feature. We estimate the distribution distance between two datasets by computing the KL-distance between their histogram features. Results are reported in the table below. We can observe that the AdvStyle-augmented dataset has a smaller distance to real datasets, indicating that AdvStyle can generate data closer to the real distributions. This further explains why training the model with AdvStyle-generated samples can improve the performance on these real datasets.
> 
>
>| Source | CityScapes ↓|  BDD ↓| Mapillary ↓| Mean ↓ |
>| -------- | -------- | -------- | -------- | -------- |
>|GTAV |0.5867 |0.3421| 0.3211| 0.4166|
>|Adv-GTAV |**0.5587**| **0.3217** |**0.3058**| **0.3954**|
>
>At last, we would like to point out that our AdvStyle did not explicitly guide the adversarial direction to the real-world distribution, which inevitably will produce unreal or improper variations. We admit this is one drawback of our method and hope to address it in the future study.

----

>**Comment 2:** More augmentation effects are needed, such as lighting.
> 
>**Response 2:** Good suggestion. During rebuttal, we conducted experiments by applying our AdvStyle in the LAB space. Specifically, we first convert the RGB-sample to the counterpart LAB-sample and obtain the learnable mean and standard deviation. Then, we reconvert the LAB-sample to RGB-sample for adversarial learning and model optimization. This manner enables us to implement AdvStyle in the LAB space as well as to use the ImageNet-pretrained parameters. As shown in the Table below, LAB-based AdvStyle also significantly improves the performance on unseen domains but achieves lower results than RGB-based AdvStyle on two of the three benchmarks. Nevertheless, we may expect that LAB-based Advstyle will achieve better results in lighting matters context, such as your suggested one: the generalization from normal-light training data to low-light testing data.
>
>| Method | CityScapes | BDD | Mapillary | Mean |
>| -------- | -------- | -------- |-------- | -------- |
>| Baseline     | 28.95 |25.14| 28.18| 27.42|
>| AdvStyle (LAB)      | 37.09 | 32.89 | **37.13** | 35.70 |
>| AdvStyle (RBG) | **39.62** |**35.54** |37.00 |**37.39**|
>
>In future work, we would like to use/design a module/network that jointly accommodates different types of style variations (e.g., color, lighting, and texture) and employ our AdvStyle strategy on it to learn more robust models.

----

 
>**Comment 3:** Apply AdvStyle on SOTA classification baselines.
>
>**Response 3:** Good suggestion. Following your comment, we apply AdvStyle on another two state-of-the-art classification methods (ME-ADA and L2D) on two classification benchmarks. As shown in the Table below, AdvStyle can yield an improvement of 8.0% in average accuracy on Digits over ME-ADA and 6.2% in average accuracy on PACS over L2D, respectively. In addition, combining AdvStyle with ME-ADA or L2D outperforms the ERM+AdvStyle on both datasets. These results verify the universality of our method. We will add these results in the revision.
>
>
>| Method | SVHN | MNIST-M | SYN| USPS| Avg.|
>| -------- | -------- | -------- | -------- | -------- | -------- |
>| ERM     | 27.8     | 52.7     | 39.7 | 76.9 | 49.3 |
>|ME-ADA | 42.6 | 63.3 | 50.4 | 81.0 | 59.3 |
>|ERM+AdvStyle | 50.4 | 73.4 | 58.7 | **81.6** | 66.0 |
>|ME-ADA+AdvStyle | **55.5**	| **74.1**	| **59.3**	| 80.1	| **67.3** |
>
>
>| Method | Art. | Car. | Ske. | Pho.| Avg.|
>| -------- | -------- | -------- | -------- | -------- | -------- |
>| ERM     | 67.4 | 74.4 | 51.4 | 42.6 | 58.9 |
>|L2D | 74.3 |77.5 | 54.4 | 45.9 |63.0|
>|ERM+AdvStyle | 75.8 | 76.6 | 58.1 | 51.1 | 65.4 |
>|L2D+AdvStyle | **80.6** |	**78.4** | **58.3**	| **59.7**|**69.2**|

----

>**Comment 4:** More comparison with current SOTA augmentation method.
>
>**Response 4:** First, we have compared AdvStyle with different style augmentation methods (RandStyle, MixStyle [43] and CrossStyle [30]) in Table 4 of the main paper. We provide this experiment in the Table below for your convenience, where the models are trained on the GTAV dataset for single-source DG. AdvStyle consistently outperforms other methods on the three datasets and outperforms MixStyle by 2.79% and CrossStyle by 2.81% in mean mIoU, respectively.
>
>| Method | CityScapes | BDD | Mapillary | Mean |
>| -------- | -------- | -------- |-------- | -------- |
>| RandStyle     | 33.40| 34.14 |31.67| 33.07|
>| MixStyle     | 35.53| 32.41 |35.87| 34.60|
>| CrossStyle     | 37.26| 32.40 |34.09| 34.58|
>| AdvStyle (Ours) | **39.62** |**35.54** |**37.00** |**37.39**|
>
> Second, in Table 2 of the supplementary material, AdvStyle consistently outperforms MixStyle on the three datasets. The quantitative improvement is 1.27% in the mean mIoU. We admit that the improvement is not as significant as in the Table above. The reason is that multi-source data provide more styles within the source domains so that MixStyle can be benefited from more diverse styles than single-source DG. Nevertheless, our AdvStyle outperforms MixStyle on both single- and multi-source DG, indicating the effectiveness of our AdvStyle under different contexts. Finally, we would like to indicate that the number of 1.27% is not a marginal improvement due to the difficulty of the DG task. 

----

>**Comment 5:** The update of style features is similar to the strategy of obtaining adversarial samples (adversarial perturbations). How to control the parameter of $\gamma$ in Eq. 3?
>
>**Response 5:** Your understanding is right. The update strategy of our AdvStyle is similar to adversarial perturbation, but their implementations are different due to the different motivations. 
>>
>Adversarial learning requires generating imperceptible adversarial perturbations by several steps, so the update of adversarial samples is controlled by perturbation scope. Instead, AdvStyle aims at generating diverse stylized samples for the current model. For this purpose, we do not need to force strict constraints on the adversarial gradient, since the semantic content of most pixels will be maintained and generating some unrealistic styles is also acceptable. Consequently, we can easily control the AdvStyle by the adversarial style learning rate $\gamma$, and the adversarial samples can be generated by one step.
>>
>We empirically select the adversarial style learning rate $\gamma$ as 3 in Figure 1 of the supplementary material.

----

>**Comment 6:** Why use both learning rate for $\mu$ and $\sigma$?
>
>**Response 6:** Good question. We have tried to use different learning rates for $\mu$ and $\sigma$ but find that the best choices for them are both 3. We thus use one symbol to represent their learning rates in our paper.

----

 Thanks for your valuable comments. Please find our detailed response below.


>**Comment 1:** A significant concern with the submission is that it poses its method as domain generalization (DG) rather than image augmentation (IA) and the subsequent analysis and choice of baselines. I believe this confounds the comparison to prior art and comparable methods. 
>
>**Response 1:** We first would like to clarify that in this paper we focus on solving the problem of domain generalization in the view of data augmentation and that we have stated that our method is a type of data augmentation method. As far as we know, data augmentation techniques are widely used to solve many problems, such as supervised learning (Mixup [A], Random Erasing [B]), semi-supervised learning (MixMatch[C]), domain adaptation (AugGAN [D]), and domain generalization focused on this paper (MixStyle[E]). However, the community commonly does not compare the methods of different tasks, since they are specially designed for the specific tasks, so as our method.
>>
>In addition, in our paper, we have already compared with the state-of-the-art data augmentation methods for domain generalization, including MixStyle, CrossStyle, DRPC, and FSDR. Note that, we have indicated that these four methods are data augmentation methods. We have shown the improvements of our method over them. Note that, all of these compared data augmentation methods in DG do not compare with the data augmentation methods (e.g., Random Erasing [B] that is widely used for supervised learning) of other tasks, due to the different purposes.
>>
>  In sum, we believe that this paper does not confound the comparison to previous state-of-the-art methods, since we have compared with state-of-the-art methods for domain generalization (including both network-designing and augmentation-designing methods) in a fair way. In addition, we believe this paper makes an interesting and reasonable step to investigate how data augmentation can help to learn generalized models, rather than leads a conflict between the tasks of domain generalization and data augmentation.
>
>[A] mixup: Beyond Empirical Risk Minimization. ICLR 2018.
>
>[B] Random Erasing Data Augmentation. AAAI 2020.
>
>[C] MixMatch: a holistic approach to semi-supervised learning. NeurIPS 2019.
>
>[D] AugGAN: Cross Domain Adaptation with
GAN-based Data Augmentation. ECCV 2018.
>
>[E] Domain Generalization with MixStyle. ICLR 2021.

----

>**Comment 2:** If I take the DG perspective I'd have to ask: Will this work for other DG tasks that do not make strong assumptions about the signal (image formation process, meaning of mean/variance)? I can find no evidence for this in the paper and it appears that the main premise is a strong domain assumption (the image formation process) and selection of data to conform with this assumption (i.e. the datasets are chosen such that they differ demonstrably in the chosen and explicitly modeled statistic). Alternatively, the message of the submission could be that for a set of domains, I can a priori examine some low-dimensional formative process that I then can exploit to generate better samples (in terms of generalization across the prior domains). This could be reformulated from the perspective of image augmentation (IA). 
>
>**Response 2:** We would like to clarify that this paper considers the practical tasks along with commonly used data in the computer vision community. In real-world applications, style variations between different domains/datasets are widespread, e.g., autonomous driving with ever-changing conditions (weather), detecting persons at different day-night times, and recognizing objects in different styles (nature or cartoon). In addition, contexts with style variations are widely considered in domain generalization, including image classification (Office-31, Office-Home) and semantic segmentation (GTAV, Cityscapes). Therefore, this paper does not strongly select data to conform to the assumption of our method. Instead, we (1) consider the widely studied and practical task/context in the computer vision community, i.e., domain generalized urban-scene segmentation, and (2) specially design an augmentation method for solving this problem by explicitly considering the particularly crucial style factor. Moreover, we also show the benefit of our method for image classification tasks where the style variations are also significant between different domains.
----

 
>**Comment 3:** If I take on the perspective of IA, I'd have to ask if the cited prior art, baselines and benchmarks are chosen appropriately. For prior art, there would be significant references missing. For instance, [G] and [H] below demonstrate generic, in-training adversarial augmentation.
>
>**Response 3:** As explained in our reply to Comment 1, data augmentation methods are used for many tasks, and their purposes are specially designed for the corresponding tasks/problems. For example, [G] is designed to learn a model that is robust to different corrupted and perturbated samples (e.g., gaussian noise). Instead, in this paper, we aim to improve the model's generalization ability on unseen domains where image style variations are significant. Moreover, both [G] and [H] are largely relied on geometric transformations, such as translating, shearing, and rotating, which can be easily applied to global-based classification tasks but will have significant difficulties in pixel-wise semantic segmentation tasks. In other words, different augmentations have particular functions in solving different tasks. Therefore, we believe it is not essential and sometimes not practical to compare with all of the existing augmentation methods. Instead, comparing with methods focusing on the same task is more important. Recall that, in this paper, we consider the problem of domain generalized urban-scene segmentation. We have conducted experiments on popular benchmarks and have compared our method with state-of-the-art methods, which include methods based on data augmentation.
>
>[G] AugMax: Adversarial Composition of Random Augmentations for Robust Training. NeurIPS 2021.
>
>[H] Adversarial AutoAugment. ICLR 2020.

----

>**Comment 4:** Conceivably, per-channel image mean and variance could be included in these approaches as well without fundamental changes to the algorithms. How would the presented method compare in this case? Could it be extended to other explicitly modeled image formation processes (luminance, in-plane shift, etc.)? As an example: Let's say my domain shift were in-plane rotations (e.g. relatively static car-based camera vs. smart phone from pedestrian view). The presented method would not be expected to demonstrate much benefit, as the domain assumption of shift in image mean and variance are violated. One could probably model the in-plane rotation with a similar adversarial approach to sample generation, but this essentially requires explicit knowledge of the domain shit in a low-dimensional, parametric, forward way.
>
>**Response 4:** We acknowledge the reviewer for proposing these interesting suggestions and directions. We fully agree with you that our method, i.e., the proposed adversarial learning augmentation method, can be extended to other explicitly modeled image formation processes. For example, learning adversarial rotations for the domain generalization contexts where rotation variations are critical and should be carefully concerned. However, as explained in our reply to Comments 1&2&3&4, our goal is to learn a robust model under the autonomous driving context where the style variations are significant between domains. Due to being out of the scope of this paper, we would like to investigate the effectiveness of our method in the tasks with other image variations (such as in-plane rotations) in future work.

----

>**Comment 5:** As it is presented, the empirical results are appealing as the method is relatively straightforward and improves over the presented baselines, but the level of contribution is unclear as the choice of baselines lack a thorough sample of and comparison to state of the art image augmentation methods.
>
>**Response 5:** As explained in the above replies, (1) this paper focuses on designing a data augmentation method for the domain generalized urban-scene segmentation, and (2) existing augmentations methods are specially designed for different tasks. We thus only compare with the baselines and state-of-the-art methods for the studied domain generalized urban-scene segmentation task. In addition, we have already compared our method with the state-of-the-art data augmentation methods for domain generalization, including MixStyle, CrossStyle, DRPC, and FSDR. The results show the superiority of our method over them. Notice that, all of the compared data augmentations (MixStyle, CrossStyle, DRPC, and FSDR) choose not to compare with the other data augmentation methods of other tasks, due to the different purposes. Taking the above explanations, we hope the reviewer can find (1) the motivation of our proposed augmentation method for the domain generalized urban-scene segmentation and (2) the fair comparison with the state-of-the-art methods, especially the data augmentation methods in domain generalization.

----

 >**Minor Comment A:** While inspecting tables 3 and 4, I noticed that the setup is described the same (Source/backbone), but the baselines differ. For instance, CityScapes changes from 21.64 to 28.95. It took a couple of reads to understand that the baseline in table includes Color Jittering and Gaussian Blur. Mentioning this fact in the exposure may make readings smoother.
>
>**Response A:** Thanks for this suggestion. We will add the corresponding description in the captions of Tables 3 and 4 in the revision.

----

>**Minor Comment B:** Table 3: Would it make sense to add a row combined "Ours" with "AdvPixel"? It may be interesting to see if these are complementary.
>
>**Response B:** Good question. We have already included this result in the Table. 3 of the appendix. For your convenience, we provide this experiment in the table below. We can find that AdvPixel can serve to enhance AdvStyle. The performance yields an improvement of 0.81% in mean mIoU.
>
>| Method | CityScapes | BDD | Mapillary | Mean |
>| -------- | -------- | -------- |-------- | -------- |
>| AdvPixel     | 35.42| 33.28 |33.23| 33.97|
>| AdvStyle (Ours) | 39.62 |35.54 |**37.00** |37.39|
>| AdvPixel + AdvStyle | **40.65** | **37.16** | 36.77 | **38.20**|

----

 Thanks for your valuable comments. Please find our detailed response below.

>**Comment 1:** If the style features are just mean & variance of R-G-B channels?
>
>**Response 1:** Yes, the style features are the channel-wise mean and standard deviation of the input RGB images, and each of them is a 3-dim feature. 
>

>**Comment 2:** Why not use other color spaces or even texture spaces? 
>
>**Response 2:** Good question.
>
> *As for the color space*, we conduct experiments by applying our AdvStyle in the LAB space. Specifically, we first convert the RGB-sample to the counterpart LAB-sample and obtain the learnable mean and standard deviation. Then, we reconvert the LAB-sample to RGB-sample for adversarial learning and model optimization. This manner enables us to implement AdvStyle in the LAB space as well as to use the ImageNet-pretrained parameters. As shown in the Table below, LAB-based AdvStyle also significantly improves the performance on unseen domains but achieves lower results than RGB-based AdvStyle on two of the three benchmarks. On the other hand, converting between RGB and LAB will increase the training time due to the extra computation costs. Thus, considering the effectiveness and efficiency, we keep applying our AdvStyle to the RGB space.
>
>| Method | CityScapes | BDD | Mapillary | Mean |
>| -------- | -------- | -------- |-------- | -------- |
>| Baseline     | 28.95 |25.14| 28.18| 27.42|
>| AdvStyle (LAB)      | 37.09 | 32.89 | **37.13** | 35.70 |
>| AdvStyle (RBG) | **39.62** |**35.54** |37.00 |**37.39**|
>
>*As for the texture space*, one option can be directly augmenting the values of image pixels. We have compared the AdvPixel (applying adversarial learning on each pixel) with our AdvStyle in Table 3 of the main paper and Table 3 of the appendix. For your convenience, we provide these results in the table below. Our AdvStyle can outperform AdvPixel by a large margin, while combining AdvStyle and AdvPixel can achieve better performance. 
>
>| Method | CityScapes | BDD | Mapillary | Mean |
>| -------- | -------- | -------- |-------- | -------- |
>| AdvPixel     | 35.42| 33.28 |33.23| 33.97|
>| AdvStyle (Ours) | 39.62 |35.54 |**37.00** |37.39|
>| AdvPixel + AdvStyle | **40.65** | **37.16** | 36.77 | **38.20**|
>
> In future work, we would like to use/design a module/network that jointly accommodates different types of style variations (e.g., color, lighting, and texture) and employ our AdvStyle strategy on it to learn more robust models.

----

>**Comment 3:** If the target domain data are not used at all during training? 
>
>**Response 3:** Yes, the model is trained only on the source domain (GTAV, SYNTHIA, or both of them), and the target domain data (CityScapes, BDD100K and Mapillary) are totally unseen during training. During testing, the trained model is directly applied to the target testing data without further training or fine-tuning.

----

>**Comment 4:** If a local augmentation can further improve the performance? 
>
>**Response 4:** Good suggestion. Following your comment, we apply AdvStyle on local patches instead of the whole image. Specifically, we split each image into 4 patches evenly (top left, top right, bottom left, and bottom right), and regard the channel-wise mean and standard deviation of each patch as learnable parameters (four 6-dim features). Then the model is trained the same as AdvStyle. As shown in the Table below, as you might expect, *AdvStyle + Patches* can further improve the performance on BDD and Mapillary. However, the mean improvement over all domains is marginal. Considering the effectiveness and efficiency, we apply AdvStyle to the whole image in our method.
>
>| Method | CityScapes | BDD | Mapillary | Mean |
>| -------- | -------- | -------- |-------- | -------- |
>| AdvStyle | **39.62** |35.54 | 37.00 |37.39|
>| AdvStyle + Patches | 39.50 | **36.37** | **37.42** | **37.76** |

----

 This work aims to address the domain generalization problem in semantic segmentation. A simple and effective adversarial augmentation technic is proposed which changes global statistics at the image level. Through adversarial learning, the adversarial style features seem to capture the characteristics of other datasets successfully and improve the performance significantly. Strengths
- The main idea of AdvStyle is simple and easy to implement. I believe this can benefit many computer vision tasks, especially the dense prediction tasks.

- The illustration of how style changes affect the segmentation performance in Figure 1 is good. It can be clearly observed that the mIoU decreases a lot when only changing scene colors. 

- The t-SNE visualization in Figure 5 provides good proof for the AdvStyle idea: the adversarial style features can capture some characteristics of other datasets. 

- Sufficient experiments have been done, including comparisons with common data-augmentation technics, SOTA DG methods and extra experiments on classification tasks.

Weaknesses
- In Figure 1, It is claimed that image-level mean-variance is used as the style feature. While it is not clear how to calculate and apply such style features.  In Line 70, it is claimed that “6-dim feature for each example”. So, I wonder if the style features are just mean & variance of R-G-B channels. If so, why not use other color spaces or even texture spaces? I think this is an important ablation study that is missing in the main paper.

 I want to double-check if the target domain data are not used at all during training. So the results in Table 1 are obtained by training model on the GTAV training data only, and then applied to other real-world validation data. Right?  

Since the channel-wise augmentation is applied to the global region, I wonder if a local augmentation can further improve the performance.
 The limitations and potential negative societal impact have been well described.

 The paper addresses the general problem of learning a model in one domain and testing it in a second domain, where there is some domain shift between the domains. Specifically, it demonstrates a method that improves testing performance on real, unknown data when learning on synthetic, rendered data in semantic segmentation for autonomous driving. The motivating observation is that in urban semantic segmentation for autonomous driving a concise domain shift is measurable in the per-channel mean and variance of the image data. In other words, the color statistics between different data sets (and acquisition conditions) differ. The authors propose to explicitly target adversarial changes of the mean and variance of the first domain images during training to improve generalization performance in a second, unknown, domain.

The method works in two phases for each batch iteration, where the mean and variance of the input image can be modified during the forward pass: adversarial updating of the mean/variance perturbation based on the current, frozen model prediction, and then regular updating of model weights with both adversarial as well as unperturbed samples.

The paper continues to evaluate the performance of the proposed method on one synthetic versus three real datasets in semantic segmentation for autonomous driving (GTAV versus CityScapes, BDD, Mapillary), as well as several choices for backbones and batch normalization schemes. The empirical performance results demonstrate significant improvements over the chosen baselines. Furthermore, the paper extends the results to image classification on two relevant benchmarks, "Digits" and and "PACS" The paper is written well and the method is laid out with sufficient clarity and detail. The datasets chosen for comparison are relevant and realistic (in terms of potential real-world applications of the method). I appreciate the motivation: We observe measurable population differences between various datasets and that they can be captured in a compact and semantically meaningful vector already (image mean and variance). How can we use this prior knowledge to improve task performance in semantic segmentation? Based on this motivation, the authors conclude to model the observed perturbation explicitly as it happens to be a natural part of compact image formation models already (in the sense that normalization with these statistics is well known and commonly used). Once it is modeled, the authors show that it is a comparatively simple step to adversarially predict a new set of samples that are harder to predict at the current training state. An appealing property of the method is that the perturbation space is fairly low-dimensional (6-vector) and thus more easily characterized compared to high-dimensional perturbations for instance on the level of individual pixels.

A significant concern with the submission is that it poses its method as domain generalization (DG) rather than image augmentation (IA) and the subsequent analysis and choice of baselines. I believe this confounds the comparison to prior art and comparable methods. If I take the DG perspective I'd have to ask: Will this work for other DG tasks that do not make strong assumptions about the signal (image formation process, meaning of mean/variance)? I can find no evidence for this in the paper and it appears that the main premise is a strong domain assumption (the image formation process) and selection of data to conform with this assumption (i.e. the datasets are chosen such that they differ demonstrably in the chosen and explicitly modeled statistic). Alternatively, the message of the submission could be that for a set of domains, I can a priori examine some low-dimensional formative process that I then can exploit to generate better samples (in terms of generalization across the prior domains). This could be reformulated from the perspective of image augmentation (IA). If I take on the perspective of IA, I'd have to ask if the cited prior art, baselines and benchmarks are chosen appropriately. For prior art, there would be significant references missing. For instance, [1] and [2] below demonstrate generic, in-training adversarial augmentation. Conceivably, per-channel image mean and variance could be included in these approaches as well without fundamental changes to the algorithms. How would the presented method compare in this case? Could it be extended to other explicitly modeled image formation processes (luminance, in-plane shift, etc.)? 
As an example: Let's say my domain shift were in-plane rotations (e.g. relatively static car-based camera vs. smart phone from pedestrian view). The presented method would not be expected to demonstrate much benefit, as the domain assumption of shift in image mean and variance are violated. One could probably model the in-plane rotation with a similar adversarial approach to sample generation, but this essentially requires explicit knowledge of the domain shit in a low-dimensional, parametric, forward way. 

As it is presented, the empirical results are appealing as the method is relatively straightforward and improves over the presented baselines, but the level of contribution is unclear as the choice of baselines lack a thorough sample of and comparison to state of the art image augmentation methods.

```
[1] @inproceedings{
wang2021augmax,
title={AugMax: Adversarial Composition of Random Augmentations for Robust Training},
author={Haotao Wang and Chaowei Xiao and Jean Kossaifi and Zhiding Yu and Anima Anandkumar and Zhangyang Wang},
booktitle={Advances in Neural Information Processing Systems},
editor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},
year={2021},
url={https://openreview.net/forum?id=P5MtdcVdFZ4}
}

[2] @inproceedings{
zhang2020adversarial,
title={Adversarial AutoAugment},
author={Xinyu Zhang and Qiang Wang and Jian Zhang and Zhao Zhong},
booktitle={International Conference on Learning Representations},
year={2020},
url={https://openreview.net/forum?id=ByxdUySKvS}
}
``` I would like to hear the author's thoughts on my concerns on image augmentation as a fair baseline above.

Some minor comments that do not affect the rating:
- While inspecting tables 3 and 4, I noticed that the setup is described the same (Source/backbone), but the baselines differ. For instance, CityScapes changes from 21.64 to 28.95. It took a couple of reads to understand that the baseline in table includes Color Jittering and Gaussian Blur. Mentioning this fact in the exposure may make readings smoother.
- Table 3: Would it make sense to add a row combined "Ours" with "AdvPixel"? It may be interesting to see if these are complementary.
 The limitations of the work are fairly clear from the submission.

 This paper propose a new adversarial style augmentation strategy (AdvStyle) for domain generalization in the semantic segmentation task. The approach of AdvStyle can generate hard stylized images during training, preventing the model from overfitting on the source domain. The generation is completed via learning adversarial style feature. Experiments on two semantic segmentation benchmarks demonstrate the effectiveness of AdvStyle.  Strengths:
The proposed approach can obviously improve the performance of baselines' performance on unseen domains, including segmentation and classification tasks.

Weakness:
1. The proposed AdvStyle is indeed an approach to extend the appearance diversity of the training images, and this approach can not ensure the performance on one unseen domain, i.e., this method lacks the interpretability.
And AdvStyle can improve the generalization ability via color augmentation, but more augmentation effects are needed, such as lighting for the generalization of a model from normal-light training data to low-light testing data.

2. Why not apply the AdvStyle on SOTA classification baselines? like ME-ADA in Table 6 and L2D in Table 7? I wonder whether the performance improvement is highly related with the baseline' performance.

3. The superiority compared with current SOTA augmentation method is not obvious. Like the MixStyle reported in Table 2 of supp. More comparison is needed and such comparisons should be placed in the main paper, not supp.



 1. The proposed method updates the style feature via the direction of the gradient as shown in Eq. (3). This is very similar to the strategy of obtaining adversarial samples (adversarial perturbations). Thus, I wonder how to control the parameter of $\gamma$ in Eq.3? Why use both learning rate for $\mu$ and $\sigma$? I agree with authors' stated limitations in the supp, which is the training time. The author can consider how to reduce the training cost.


Meta-review: Simple and practical way to do better at domain generalization when it comes to semantic segmentation. AdvStyle can generate images that are hard during training, and prevent the model from overfitting on the source domain. Given that it works well, is relatively simple to implement and conceptually sound, I think it will appeal to a large portion of the NeurIPS audience that works on domain generalization.

Decision: Accept