Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation
Authors: Zhun Zhong
Yuyang Zhao
Gim Hee Lee
Nicu Sebe
Venue: NeurIPS 2022 Accept (2022)
Keywords: Domain Generalization
Semantic Segmentation
Adversarial Style Augmentation

Abstract: In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn a robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) domains. Our study finds that the image style variation can largely influence the model's performance and the style features can be well represented by the channel-wise mean and standard deviation of images. Inspired by this, we propose a novel adversarial style augmentation (AdvStyle) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training. The learned adversarial style feature is used to construct an adversarial image for robust model training. AdvStyle is easy to implement and can be readily applied to different models. Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model performance on unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.

Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation
Zhun Zhong 1et
Yuyang Zhao
Gim Hee Lee" Nicu Sebe'
Department of Information Engineering and Computer Science University of Trento 2 Department of Computer Science, National University of Singapore
Abstract
In this paper, we consider the problem of domain generalization in semantic seg- mentation, which aims to learn robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) domains. Our study finds that the image style variation can largely influence the model's 's performance and the style features can be well represented by the channel- mean and standard deviation of images. Inspired by this, we propose novel adver- sarial style augmentation (AdvStyle) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as learnable parameter and updates it by adversarial training. The learned adversarial style feature is used to construct an adversarial image for robust model training. AdvStyle is easy to implement and can be readily applied different models. Experiments on two synthetic-to-re demonstrate that AdvStyle can significantly improve the model performance unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed domain generalized image classification and produces a clear improvement on the considered datasets.
semantic segmentation benchmarks
Introduction
Semantic segmentation plays critical role in autonomous driving, which has achieved impressive improvements with the recent development of deep networks [20, 5, 1, 45]. However, these achieve- ments have been mostly attributed large-scale labeled segmentation datasets, in which annotating pixel-wise labels is very expensive and time-consuming. Moreover, the model trained on one dataset commonly produces poor performance unseen datasets captured in different conditions. This degradation phenomenon is mainly caused by domain shifts [7]. including differences in weather, season, light, etc. For instance, the segmentation model trained on the dataset captured in sunny London will have low accuracy when deployed on the streets of Zurich in rainy weather.
To address the cross-domaie problem, domain adaptation methods [21, 41] are designed transfer the knowledge of labeled source data to unlabeled target data. However, one of their main drawbacks is that they require the use of target data during training, which cannot always be accessible in practice. Another promising line is domain generalization (DG), which focuses on learning a generalizable model using only the labeled source domain. To reduce the annotating cost and protect data privacy, the existing DG works [7, 40, 46] in semantic segmentation commonly choose to learn the robust model with synthetic data, e.g., GTAV [28]. In this paper, focus this synthetic-to-real DG problem for semantic segmentation.
"Equal contribution. i Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Caaps
(a) Examples of diftereni datuees.
(b) Style distribution of difierent dauseess
Style
90
GTAV testing set
Nate
Caybepa Solk
Mad Sole
Orggral
E-Style
Esseplecs changing style GTAV
mlol ofchnnging style.
Figure 1: (a) Examples of different datasets. The image styles from different datasets commonly are very different. (b) Style distribution of different datasets. We use image-level mean-variane as the style feature and show that the style distribution gap between different datasets are large. (c) Examples of changing style feature for a GTAV sample, including adding random noise and replacing the style feature with ones of samples from other datasets. (d) mloU performance of changing styles for GTAV testing set, which is largely reduced after style changing.
To eliminate the impact caused by the large domain gap between synthetic and real data, existing solutions mainly aim at augmenting the synthetic source data with extra real-world samples [40. 14] or learning
features with carefully designed modules [24, 7]. The key idea behind them is to avoid the model overfitting on the source domain. This work follows this idea and introduces a new augmenting approach in the perspective of image style for domain generalized semantic segmentation, which is motivated by the observations in Fig. 1. First, when visualizing the samples of different datasets in Fig. (a) we observe that the image styles are quite different among them, e.g., the road color. The style distribution gap can also be observed in Fig. (b)). Second, the channel-wise mean and standard deviation of an image, which is called style feature in this paper, can well represent the image style. When changing the style feature, the image style of an example varies while the semantic content is well maintained (see Fig. (c)). Third, changing the style features of testing samples will largely deteriorate the model performance (see Fig. (dd).. Importantly, when replacing the style features with the ones of SityScapes, which has a large distribution distance to GTAV (see red and green dots in Fig. 1(b)), the performance is lower than other style changing cases. This indicates that the model performance on testing data is highly related to the style distribution gap between training and testing data.
Taking the above observations into consideration, we argue that the image style an important factor that affects the model performance and propose the adversarial style augmentation (AdvStyle) for domain generalized semantic segmentation. Specifically, AdvStyle contains two steps: adversarial style learning and robust model learning. In adversarial style learning, we first decompose the training sample into style feature and normalized image. The style feature is regarded as a learnable parameter, which is used to reconstruct new training example together with the normalized image. Then, feed the reconstructed example into the segmentation model and optimize the style feature using the adversarial segmentation loss. The updated style feature is called adversarial style feature and is used to produce hard example in the following step. In robust model training, we first generate an adversarial example by de- ne-normalizing the normalized image with the learned adversarial style feature. The adversarial image and the original image are then used to train a robust model using the segmentation loss. In AdvStyle, the adversarial image is dynamically generated based on the current model. In this way, the model is always encouraged to update with difficult styles and thus will be more robust to style variations in unseen domains. In Fig. 2, we show the comparison between AdvStyle and traditional augmentation methods. Our AdvStyle can significantly improve the model performance on unseen target domains and clearly outperforms the other augmentation methods. To summarize, our contributions are threefold:
We propose the novel adversarial style augmentation (AdvStyle) for domain generalized semantic segmentation, which can consistently improve the results on unseen real domains. AdvStyle introduces very limited learnable parameters (6-dim feature for each example) and can be easily implemented with different networks and DG methods.

---

Original (150
(566
Coker Jmering (mbol 20%
Blur Jinrring (atou (5%)
Adv BBur Jitter (mlel-39%)
Figure 2: Illustration of different data augmentation methods. We use GTA5 as the source domain and the ResNet-50 as the backbone. The mloU given in parentheses is evaluated on CityScapes validation set for the model trained with the corresponding augmentation method.
Experiments on two synthetic-tor to-rell DG benchmarks demonstrate the effectiveness of the proposed AdvStyle and show that we achieve new ofate-of-the-arr DG performance.
We show that AdvStyle can also be applied to single DG in image classification and can produce oftt-oof-the-art accuracy two datasets.
2 Related Work
Domain Generalization (DG). To tackle the deficiency of annotated data, DG [40, 14, 7, 33, 46, 4, 43] introduced to learn a robust model with one or multiple source domains, where the model is expected to perform well on unseen domains. Recent DG works in semantic segmentation mostly use synthetic data as the source domain, which can be automatically generated but have a large distribution gap to real-world datasets. One main stream of DG methods [40, 14] focuses on augmenting training samples with extra real-world data from ImageNet [9]. Learning domain-invariant features [7, 24, 33] is another stream to narrow the domain gap. [24] and [7] leverage instance normalization [34] and whitening transformation remove the domain-specific information, respectively. [33] exchanges style features of two samples and adjusts style features with the attention mechanism. Different from the above methods, our AdvStyle generates new samples by learning adversarial styles using only the synthetic source data.
Adversarial Training in DG. Adversarial training [12] is initially proposed to learn robust model that can combat imperceptible perturbations. In recent years, adversarial training is applied to single DG in image classification [36, 27, 26, 11]. by regarding adversarial samples augmented unseen samples. [36] is the first to introduce adversarial samples in DG by max-min iterative training procedure. Later methods form novel domains with the generated adversarial samples and learn nomain-invariant representation by meta-learning [27, 26] or adaptive normalization [1]]. Different from them, this paper adopts adversarial training for semantic segmentation and generates adversarial samples in the perspective of image style.
Style Variation. Style features are widely studied in image translation [16, 10]. By varying the style features, the image style can be changed while semantic content will be maintained. Inspired by this, recent works focus generating data of novel distributions by modifying style features, which are used to train a more robust model. One effective manner to generate new styles by exchanging [33, 44] or mixing styles [47] between samples. On the other hand, new styles can be generated by learnable modules [37]. Instead, we generate novel styles by adversarial training, which encourages the model always optimize with hard stylized examples. This work is also closely related to [2], which generates adversarial examples by colorizing. However, it requires a pre-trained colorization model to change the image color, which much more complex than our AdvStyle. In addition, [2] aims to impair the performance of models by adversarial examples. In contrast, our AdvStyle leverages the adversarial examples improve the generalization ability of the segmentation model. This work also has a connection with "Learning-to- to-Simulate" [30]. However, [30] tries to learn good sets of parameters for an image rendering simulator actual computer vision applications while we attempt to learn generalized model for semantic segmentation. Our work also shares similar idea with AdvBN [32]: generating novel styles through adversarial learning. However, unlike AdvBN, which perturbs feature statistics in the Batch Normalization (BN) [18] layers, we directly generate stylized adversarial examples the image level, which not only avoids the selection of BN layers, but also requires fewer learnable parameters (6-dim vector).
3

---

Original Image
Normed Image
Reconstrocied Imaaee
Freeze
Forward
Backward
Lun (X)
Style Feature
Segmentation
Original Prediction
Model
Adversarial Style Learning
Update
Robust Model Learning
Adversurial
Original Image
Freeze
Adversarial Style Fealure
Advenssrial Imngge
Update
Crag (xr1
DeNorm
Eseg (X)
Segmentation
Normed Image
Model
Original Prediction
Figure 3: The framework of the proposed adversarial style augmentation.
3 Method
Problem Definition. Synthetic-to-real domain generalization (DG) focuses on training robust model with one labeled synthetic domain S, where the model is expected to perform well on unseen domains (Ti, T2. of different real-world distributions. As stated by [36], the DG task can be formulated as solving the worst-case problem:
min sup T:D(S.T)SP
ET [Rusk (0;
where 0 is the model parameters and T is the target domains. Lusk denotes the task-specific loss function, which is the pixel-wise cross-entropy loss in this paper. D(S T) denotes the distribution distance between the source domain and target domains in semantic space. It is constrained to be lower than p for semantic consistency. Inspired by Eq. I, we propose a novel approach to generate a dynamic source domain St, which can help us reduce the domain shifts between the synthetic domain S and real-world domains during training.
3.1 Overview
In the introduction, we show that the image style is an important factor that influences the DG performance. addition, the channel-wise mean and standard deviation of image, which is called style feature, can well represent the image style. Inspired by that, we propose the adversarial style augmentation approach (AdvStyle) for domain generalized semantic segmentation. AdvStyle can dy- namically generate images with new styles during training and effectively improve the generalization ability of the segmentation model. AdvStyle includes two steps: adversarial style learning and robust model training. In adversarial style learning, we first decompose the image into normalized image and style feature and then update the style feature by adversarial segmentation loss. robust model training, we compose the adversarial image by the normalized image and learned adversarial style feature. The model is then optimized with both original and adversarial images. These two steps are implemented in each training iteration, enabling us dynamically generate hard stylized samples for the current model. Our overall framework is illustrated in Fig. 3.
3.2 Adversarial Style Learning
Given an training image r at each iteration, we first compute the channel-wise mean and standard deviation a of and then obtain the normalized image by normalization:
/ HW Σ
Th,w.
NEW.WEW
(2)
Σ
(Ih. 11)2 F
/
1
HW
NEH.WEW
where H and W denote the spatial size of z.

---

After that, we initialize the adversarial style feature μ² and o+ by j and which are regarded as learnable parameters. Then we reconstruct the image with μ²= o and and forward into the network for loss computation. During the backward, the parameters of the network are fixed and the adversarial style feature is updated by:
p t VV jut Luyy (0; a)
(3)
o+ at Logg (0; $),
where 7 is the learning rate for adversarial style learning and Leg is the cross-entropy loss. å T. x+ + /u+ is the reconstructed image. Notice that the style feature is optimized by the adversarial gradient of Leg- Indeed, the adversarial style learning process can be iterated multiple times. In our experiment, we find that one-step adversarial style learning achieves similar results with multi-step ones but is much more efficient. Therefore, we only update the style feature once.
3.3 Robust Model Training
Given the learned adversarial style feature (pct and ott,, we use to generate the adversarial sample xt with the corresponding normalized image:
+ nut.
(4)
Then the original image and the generated adversarial image xt are forwarded to the model for optimization, which can be formulated by,
min (0; + (0;
The detailed training procedure and lytorch-like pseudo-code can be found in the Appendix. During testing, we directly input the original samples into the network without implementing AdyStyle.
3.4 Discussion
Adversarial data augmentation have been studied by several works for DG in image classification. Most of them [27, 36] generate pixel-wise perturbations the training image x; which usually require additional constraint loss to guarantee the semantic consistency in Eq. 1. In addition, these works focus on the image classification task where the recognition result mostly related global feature. However, in semantic segmentation, the model needs to produce the per-pixel predictions so that it more difficult to ensure the pixel-wise semantic consistency during adversarial learning. Instead, our AdvStyle varies the style feature of the image, which will maintain the semantic content of most pixels and thus can well guarantee the pixel- -wiee semantic consistency for semantic segmentation. We conduct experiments in Table 3, which demonstrate the superiority of the proposed AdvStyle over pixel-wise adversarial learning.
4 Experiments
4.1 Experimental Setup
Datasets and Implementation Details. For the synthetic-to-real domain generalization (DG),
we use one of the synthetic datasets (GTAV [28] or SYNTHIA [29]) the source domain and evaluate the model performance on three real-world datasets (CityScapes [8], BDD-100K [39], and Mapillary [23]). The details of the datasets can be found in the Appendix. Following [7]. we use DeepLab V3+ [6] as the segmentation model. The segmentation model is constructed by three backbones, including MobileNetV2 [31], ResNet-50 [13] and ResNet-101. We adopt SGD optimizer with an initial learning rate 0.01, momentum 0.9 and weight decay 5x 10- to optimize the model. The polynomial decay [19] with the power of 0.9 is used as the learning rate scheduler. The learning rate of AdvStyle y is set to 3. All models are trained for 40K iterations with a batch size of 16. Four widely used data augmentation techniques are used during training, including color jittering, Gaussian blur, random cropping and random flipping. The input image is randomly cropped to 768x 768 for training. The original image size is used for testing.
Evaluation Metric. Following [7], the model obtained by the last training iteration used to evaluate the mloU performance the three real-world validation sets. For each method, report the
5

---

Table 1: Evaluation of the proposed AdvStyle on different methods (Baseline, IBN-Net [24] and ISW(7]) and backbones (MobileNetV2 [31]. ResNet--50 [13], and ResNet-101). All models are
trained on the GTAV training set and tested on CityScapes (C), BDD-100K (B), and Mapillary (M) validation sets.
Methods
MobileNetV2
ResNet-50
ResNet-101
GTAV
C
B
M
Mcan
C
B
M
Mcan
C
B
M
Mcan
Baseline
25.92
25.73
26.45
26.03
28.95
25.14
18.18
27.42
32.97
30.77
30.68
31.47
+AdvStyle
31.81
33.01
31.50
32.11
39.62
35.54
37.00
37.39
39.52
36.39
36.10
37.34
IBN-Net
30.14
27.66
27.07
28.29
33.85
32.30
37.75
34.63
37.37
34.21
36.81
36.13
+AdvStyle
32.45
31.55
33.09
32.36
39.32
36.42
40.82
38.85
44.04
39.96
42.67
42.22
ISW 30.86
30.05
30.67
30.53
36.58
35.20
40.33
37.37
37.20
33.36
35.57
35.38
+AdvStyle
33.23
31.84
32.00
32.36
39.60
38.59
41.89
40.03
43.44
40.32
41.96
41.91
Table 2: Results of using SYNTHIA as the source domain. The backbone ResNet-101.
Methods (SYNTHIA->) CityScapes
BDD
Mapillary
Mean
Baseline
34.94
21.96
27.94
28.28
Aaseline+AdSStyle
37.59
27.45
31.76
32.27
IBN-Net
35.83
23.62
28.88
29.44
IBN-Net+AdvStyle
38.72
28,55
33,59
33.62
ISW
35.27
23.54
26.72
28.51
AdvSylel
39.74
28.33
32.87
33.65
result averaged on 3 runs. When using GTAV as the source domain, we use the 19 shared semantic categories for training and evaluation. When using SYNTHIA as the source domain, we use 16 shared categories for training and evaluation, i.e., ignoring the train, truck, and terrain categories.
4.2 Evaluation
Effectiveness of AdvStyle Different Models. Our AdvStyle is model-agnostic method, which can be directly applied to different models without modifying the models. To verify the effectiveness of AdvStyle. we apply it models with different backbones and normalization modules. The backbones include MobileNetV2, ResNet-50 and ResNet- 101. The normalization modules include vanilla batch norm (bascline), instance-batch norm (IBN-Net [24]). and instance selective whitening (ISW [7]). In Table 1, we show the results of using GTAV as the source domain. We can make the following conclusions. First, injecting instance-batel norm (IBN-Net) and instance selective whitening (ISW) modules can consistently improve the performance of the baseline. Second, the proposed AdvStyle can significantly enhance the generalization performance of the baseline model for all backbones. Specifically, the average mloU is increased from 26.03% % 27.42% and 31.47% to 32. 1%, 37.39% and 37.34% for MobileNetV2. ResNet-50 and ResNet- 101, respectively. the baseline with AdvStyle produces higher average mloU than IBN-Net and ISW for all backbones. Third, when adding AdvStyle, the results of IBN-Net and ISW can be further improved for all settings. For example. when using ResNet-101 as the backbone, AdvStyle improves the average mloU of IBN-Net and ISW by 6.09% and 6.53%, respectively. In Table 2, we show the results of using SYNTHIA as the source domain and also observe clear improvements for AdvStyle. These results verify the prominent advantage of our AdvStyle on different models.
addition,
Comparison of Different Augmentation Techniques. We investigate the impact of different augmentation methods, including color jittering, Gaussian blur, AdvPixel [36] and the proposed AdvStyle. AdvPixel is state-of-the--rt method for domain generalized image classification. The main difference between AdvPixel and AdvStyle that AdvPixel learns pixel-wise adversarial example while AdvStyle learss style-wisc adversarial example. We reproduce AdvPixel in our

---

Table 3: Comparison of different augmentations. Source: GTAV; Backbone: ResNet-50. CJ: Color
Jittering, GB: Gaussian Blur, AP: AdvPixel, Ours: AdvStyle.
Table 4: Comparison of different style-aware
methods. Source: GTAV; Backbone: ResNet-
GB AP Ours CityScapes
BDD Mapillary
Mcan
50.
Methods
CityScapes
BDD
Mapillary
Mean
21.64
22.85
24.22
22.91
26.36
23.82
26.33
25.50
Bascline
28.95
25.14
28.18
27.42
25.77
24.05
26.71
25.51
RandStyle
33.40
34.14
31.67
33.07
23.34
28.42
30.64
27.46
MixStyle
35.53
32.41
35.87
34.60
37.51
33.74
34.73
35.32
CrossStyle
37.26
32.40
34.09
34.58
28.95
25.14
28.18
27.42
AdvStyle
39.62
35.54
37.00
37.39
/
35,42
33.28
33.23
33.97
39.62
35.54
37.00
37.39
CityScapes
BDD
Mapillary
Mcan
DAirii - -hac
widd Budiie
BAbdte Becks
8
Adv Learning Rate
Adv Learning Rate
Adv Learning Raie
Adv Leaminng Rate
Figure 4: Parameter analysis on the adversarial learning rate.
setting and select the adversarial learning rate (=10) that achieves the best performance. The random cropping and random flipping used in default.
Results in Table 3 show that all four augmentation methods can improve the generalization per- AdvStyle produces significant improvement compared to other three methods. Specifically, when using AdvStyle, the average mloU is increased from 22.91% to 35.32%. This improvement is about 8% higher than the other methods. Moreover, AdvStyle is well com- plementary to color jittering and Gaussian blur. When combining these three methods, the mloU is further improved in all target domains. Compared to AdvPixel, our AdvStyle achieves clearly higher performance, no matter using color jittering and Gaussian blur. This demonstrates the advantage of learning adversarial style in domain generalized semantic segmentation.
formance. Importantly,
Comparison Different Style-Aware Methods. Table 4, we compare AdvStyle with three style- aware augmentation methods, including MixStyle [47], CrossStyle [33] and RandStyle. MixStyle mixes the styles of two samples with a convex weight while CrossStyle directly swaps the styles of two samples. RandStyle can be regarded a reduction of our AdvStyle, which randomly adds Gaussian noise into the style feature. All style-aware methods are implemented on the image-level consistently improve the
for fair comparison. We find that (1) all style-aware methods performance on all target domains and (2) AdvStyle achieves the best results. The first finding verifies the effectiveness of augmenting image styles and the second finding shows the benefit of learning adversarial styles over other style-aware methods for domain generalized semantic segmentation.
Parameter Analysis the Adversarial Learning Rate. The proposed AdvStyle has one important hyperparameter, i.e., the adversarial learning rate y. To study the impact of y, we vary it the range of [0.1, 40). Results in Fig. 4 show that AdvStyle can significantly improve the performance target domains even with a small value of y. The best results are achieved when y is between I and 10. Assigning a too large value to y (e.g., 40) may produce unrealistic styles and thus hampers the model training.
all
4.3 Comparison with State-of-The-Art Methods
In Table 5, we compare our method with state-of-the-art DG methods in semantic segmentation, including IBN-Net [24], SW [25], IterNorm [15], ISW [7], DPRC [40] and FSDR [14]. The source domain is GTAV and the backbones are ResNet-50 and ResNet-101. 101. Note that, since different
7

---

IV ISW Table Comparison with state-of-the-an domain generalization methods. All models use the GTAV as the source domain. For each backbone, models with the same ID are implemented with the same baseline. Models of "ID=I, and IV use the whole set (24,966) for training while models of "ID=III" only use the training set (12,403). The absolute gain of each model is calculated over the corresponding baseline. & denotes extra using the ImageNet images. indicates using the best trained
checkpoints for evaluating each target domain.
Net ID Methods (GTAV)
CityScapes
BDD
Mapillary
Mean
Baseline
22.20
N/A
N/A
N/A
IBN-Net
29.60 7.40 f
11
Baseline"
32.45
26.73
25.66
28.28
ResNet-50
II
DRPC
37.42 4.971
32.14
5,41
34.12
8.46
34.56
6.281
III
Baseline
28.95
25.14
28. 18
27.42
III Baseline+AdvStyle
39.62
10.67t
35.54
10.4t
37.00
8.821
37.39
9.971
III SW III IterNorm
29.91
0.96t
27.48
2.34t 32.70
29.71
1,53t
29.03
1.61
31.81
2.861
7.56t
33.88
5.7t
32.79
5.371
III IBN-Net TBN-Net+AdvStyle
33.85
4.901
32.30
7.16f
37.75
9,571
34.63
7.21f
III
39.32
10.37
36.42
11.281
40.82
12.64
38.85
11.431
III ISW
36.58
7.631
35.20
10.06
40.33
12.15t
37.37
9.95t
III ISW+AdvStyle
39.60
10.65
38.59
13.45t
41.89
13.71
40.03
12.611
Baseline'
33.4
27.3
27.9
29.53
IBN-Net"
40.3
6.91
35.6
8.3t
35.9
8.01
37.26
7.73
FSDR
44.8
11.41
41.2
13.91
43.4
15.5t
43.13
13.6f
ResNet- 101
II Baseline"
33.56
27.76
28.33
29.88
II DRPC
42.53 8.971
38.72
10.96
38.05
9.72
39.76
9.881
III Baseline
32.97
30.77
30.68
31.47
III Baseline+AdvStyle
39.52
6.55t
36.39
5.621
36.10
5.421
37.34
5.871
III IBN-NNt III IBN-Net+AdvSStlle
37.37
4.401
34.21
3,44f 39.96
36.81
6,13t
36.13
4.66f
44.04
11.07t
9.19t
42.67
11.991
42.22
10.751
III ISW
37.20
4.23f
33.36
2.59t
35.57
4.891
35.38
3.91f
III ISW+AdvStyle
43.44
10.47f
40.32
9.55t
41.96
11.281
41.91
10.441
III ISW+AdvStyle"
45.62
12.65
41.71
10.971
46.69
16.01
44.67
13.201
37.51
33.54
36.12
35.72
IV ISW+AdvStyle
44.51
39.27
43.48
42.42
methods use different segmentation networks (e.g., DeepLab [5], DeepLabV3+ [6] and FCN [20]. different training sets (e.g., the whole GTAV and the training set of GTAV), different training strategies (e.g., learning rate and optimizer), different auxiliary data (e.g., ImageNet samples) and different evaluation manners (e.g., the best model and the last model), absolutely fair way. We show the results of each method as well as the absolute gain against the corresponding baseline.
hard to compare them in an
From Table 5, we can make the following conclusions. First, when using the same baseline model, adding AdvStyle can produce better results than IBN-Net, SW. IterNorm and ISW. Moreover, when applying AdvStyle to IBN-Net or ISW, we achieve new state-of-the-art performance for both ResNet-50 and R011 Second, when compared across basclines. "BvsSline+AdvStylee" achieves the state-of-thh-an mloU for ResNet-50. On the other hand, when using ResNet- 101 as the backbone. ABBN-NetAAvSSyyle produces higher results than DRPC and comparable results with FSDR. Importantly, both FSDR and DRPC use extra ImageNet images and select the best training checkpoints for each target domain. Instead, "IBN-Net+AdyStyle" only utilizes the source data and uses the last training checkpoint to evaluate all target domains. Third, when using the best checkpoint for evaluation, we AdvStlle" produce better performance than DRPC and FSDR, leading to the new state-of-the- reasonable to use the last checkpoint for evaluating all target domains. This is because we can not always have the right labeled validation sets select the best model for unseen domains in practice.
results under the "best checkpoint setting". Even we argue that it is more
8

---

Table Results of using GTAV and SYNTHIA the source domains. The backbone is ResNet-50.
Methods (GTAV+SYNTHIA CityScapes
BDD
Mapillary
Mean
Baseline [7]
35,46
25.09
31.94
30.83
IBN-Net [24]
35.55
32.18
38.09
35.27
ISW [7]
37.69
34.09
38.49
36.75
AdvStyee
39.29
39.26
41.14
39.90
Badiae
Beatte Audiiios
Addrlee
MAdrlgle
CheScees
Figure Qualitative comparison of segmentation results. Source: GTAV; Backbone: ResNet-50.
To make more fair comparisons with the state-of-the-ee methods, we also the whole of GTAV for training ISW and "ISy+AdvStyle". The results with ResNet- 101 are reported in the last two of Table 5. We find that using the whole set of GTAV can produce higher results the CityScapes and the Mapillary datasets.
4.4 Performance under Multi-Source Setting
In Table 6, we evaluate the models under the multi-source domain generalization setting, where both GTAV and SYNTHIA used the source domains. The compared methods include baseline [7], IBN-Net [24], ISW [7], and our AdvStyle. Clearly, AdvStyle consistently improves the results of ISW, further verifying the effectiveness of the proposed AdvStyle.
4.5 Visualization
Qualitative Comparison of Segmentation Results. In Fig. 5, compare the segmentation results for different methods on target domains. It is clear that, the proposed AdvStyle can consistently improve the segmentation results for baseline, IBN and ISW models, especially for the casily-confused classes, e.g., road vs sidewalk and building vs sky.
t-SNE of Styles. In Fig. 6, we visualize the style features generated by AdvStyle during the training phase for the GTAV training set, where ResNet-50 used as the backbone. We can find that AdvStyle continuously generate
style features that are different from the original distribution. The new style features have the chance to be located at the distributions other datasets during the training process. Moreover, AdvStyle will also generate styles that are out of the distributions of the four datasets (G, C. B, M) and may appear in other unseen domains. The visualization results further demonstrate that AdyStyle encourage the model to meet more diverse and unseen styles during training, leading to more robust model.
4.6 Evaluation on Image Classification Task
To verify the versatility of the proposed AdvStyle, we evaluate it on single DG in image classification. Experiments are conducted on two popular DG datasets, i.e., Digits and PACS. The details of the datasets and implementation can be found the Appendix.
Results on Digits. In Table 7, we compare with the baseline (ERM [35]) and 6 state-of-the-ar methods, including CCSA [22], d-SNE [38], JiGen [3], ADA [36], M-ADA [27] and ME-ADA [42]. For AdvStyle, we implement it with ERM and ME-ADA. is clear that, our AdvStyle can significantly improve the accuracy of ERM on all target domains. When applying AdvStyle to the sfate-of-the-art method (ME-ADA), the performance is improved by a large margin, i.e., 8.0% in average accuracy.

---

GEAV
CltyScapes
Mapillary
Ads CITV
Training
Init
Figure 6: t-SNE visualization of adversarial style features during training.
Table Accuracy of single domain generalization on Digits. MNIST is used the training set, and the results on different testing domains are reported in different columns.
Table 8: Accuracy of single domain general- ization on PACS. One domain (name col- umn) is used as the training (source) data and the other domains are used as the testing (tar- get) data.
Method
SVHN MNIST-M SYN
USPS
Avg.
ERM
27.8
52.7
39.7
76.9
49.3
Method
Art.
Car.
Ske.
Pho.
Avg.
CCSA
25.9
49.3
37.3
83.7
49.1
ERM
67.4
74.4
58.9
d-SNE
26.2
51.0
37.8
93.2
52.1
JiGen
69.1
74.6
52.4
41.5
59.4
JiGen
33.8
57.8
43.8
77.2
53.1
ADA
35.5
60.4
45.3
77.3
54.6
RSC
68.8
74.5
53.6
41.9
59.7
M-ADA
42.6
67.9
49.0
78.5
59.5
L2D
74.3
77.5
54.4
45.9
63.0
ME-ADA
42.6
63.3
50.4
81.0
59.3
ERM+AdvStyle
75.8
76.6
58.1
51.1
65.4
ERM+AdvStyle
50.4
73.4
58.7
81.6
66.0
RSC+AdvStyle
75.1
78.0
58.9
55.5
66.8
ME-ADA+Adwstyle
55.5
74.1
59.3
80.1
67.3
1.2D+AdvStyle
80.6 78.4
59.7
69.3
Results on PACS. We compare with the baseline (ERM [35]) and three state-of-the-an DG methods, including JiGen [3], RSC [17] and 1.21) [37]. We reproduce JiGen, RSC and I.21) with their official source codes. All methods use the same baseline (ERM), Results in Table 8 show that JiGen and RSC produce limited improvements. Instead, our AdvStyle can significantly increase the accuracy on all domains for both ERM and RSC. Compared to the recent published work (L22), our method ARRM+AdvStyle outperforms by 2.4% average accuracy. addition, AdvStyle can also be applied to L2D and yields an improvement of 6.3% in average accuracy.
The results on Digits and PACS validate that AdvStyle can also be effectively applied to single domain generalized image classification and can achieve state-of-the the-ttr-rt accuracy.
5 Conclusion
In this paper, we propose a novel augmentation approach, called adversarial style augmentation (AdvStyle), for domain generalization (DG) in semantic segmentation. AdvStyle dynamically generates hard stylized images by learning adversarial image-level style feature, which can encourage the model learning with more diverse samples. With AdvStyle, the model can refrain from the problem of overfitting on the source domain and thus can be more robust to the style variations of unseen domains. AdvStyle is easy implement and can be directly integrated with different models without modifying the network structures and learning strategies. Experiments on two synthetic-to to-aell settings show that AdvStyle can largely improve the generalization performance and achieve state-of -free-of-hh-rrrr- art performance. In addition, AdvStyle can be employed to single DG image classification and obtain significant improvement.
Acknowledgement
This work is supported by the EU H2020 project A14Media (No. 951911) and the PRIN project PREVUE (Prot. 2017N2RK7K).
10

---

References
[1] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. TPAMI, 2017.
[2] Anand Bhattad, Min Jin Chong. Kaizhao Liang, Bo Li, and DA Forsyth. Unrestricted adversarial examples via semantic manipulation. In ICIRR, 2020.
[3] Fabio M Carlueci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
[4] Chen Chen, Zeju Li, Cheng Ouyang. Matt Sinclair, Wenjia Bai, and Daniel Rueckert. Maxstyle: Adversarial style composition for robust medical image segmentation. In MICCAI, 2022.
[5] Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2018.
[6] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[7] Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR, 2021.
[8] Marius Cordis, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urhan scene understanding. In CVPR, 2016.
[9] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[10] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. learned representation for artistic style. In ICLR, 2017.
[11] Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang. Boqing Gong, and Mingyuan Zhou. Adversarially adaptive normalization single domain generalization. In CVPR, 2021.
[12] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamessiin adversarial examples. ICLR, 2015.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[14] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In CVPR, 2021.
[15] Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardization towards efficient whitening. CVPR, 2019.
[16] Xun Huang and Serge Belongie. Arbitrary style transfer in real tile with adaptive instance normalization. In ICCV, 2017.
[17] Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV, 2020.
[18] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML), 2015.
[19] Wei Liu, Andrew Rahinovich, and Alexander C Berg, Parsenet: Looking wider to see better. arXiv preprint ArXivv1506.00m 04579, 2015.
[20] Jomathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015.
[21] Yawei Luo, Liang Zheng, Tao Guan. Junqing Yu. and Yi Yang. Taking closer look at domain shift: Category-level adversaries for semanties consistent domain adaptation. In CVPR, 2019.
[22] Saeid Motiaan, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In ICCV, 2017.

---

[23] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017.
[24] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing leamning and generaliza- tion capacities via ien-net. In ECCV, 2018.
[25] Xingang Pan, Xiaohang Zhan, Jianping Shi, Xiaoou Tang, and Ping Luo. Switchable whitening for deep representation learning. In ICCV, 2019.
[26] Fengchun Qiao and Xi Peng. ensertaiity-guuddee model generalization to unseen domains. In CVPR, 2021.
[27] Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. In CVPR, 2020.
[28] Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
[29] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
[30] Natanicl Ruiz, Samuel Schulter, and Manmohan Chandraker. Learning to simulate. In ICLR, 2019.
[31] Mark Sandler. Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018
[32] Manli Shu, Zuxuan Wu, Micah Goldblum, and Tom Goldstein. Encoding robustness image style via adversarial feature perturbations. In M. Ranzato. A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, NewrlPS, 2021.
[33] Zhiqiang Tang, Yunhe Gao, Yi 7hu, 7hi Zhang, Mu Li, and Dimitris Metaxas. Selfnorm and crossnorm for out-of-distribution robustness. ICCV, 2021.
[34] Dmitry Ulyanow, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 160.00000,,, 2016.
[35] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
[36] Riccardo Volpi, Hongscok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NewrIPS, 2018.
[37] Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. In ICCV, 2021.
[38] Xiang Xu, Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d-sne: Domain adaptation using stochastic neighborhood embedding. In CVPR, 2019.
[391 Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangehen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
[40] Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Vincentiili Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-rcal generalization without access- ing target domain data. In ICCV, 2019.
[41] Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pscudo label denoising and target structure learning for domain adaptive semantic segmentation. CVPR, 2021.
[42] Long Zhao, Ting Liu. Xi Peng, and Dimitris Metaxas. Maximum-entron entropy adversarial data augmentation for improved generalization and robustness. NeurlPS, 2020.
[43] Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhiming Luo, Yaojin Lin, Shaozi Li, and Sebe Nicu. Learning to generalize unseen domains via memory-basee multi multi-source meta-learning for person re-identification. In CVPR, 2021.
[44] Yuyang Zhao, Zhun Zhong, Zhiming Luo, Gim Hee Lee, and Nicu Sehe. Source-free open compound domain adaptation in semantic segmentation. TCSVT, 2022.
[45] Yuyang Zhao, Zhun Zhong, Nicu Sebe, and Gim Hee Lee. Novel class discovery in semantic segmentation. In CVPR, 2022.
[46] Yuyang Zhao, Zhun Zhong, Na Zhao, Nicu Sebe, and Gim Hee Lec. Style- hallucialle dual consistency learning for domain generalized semantic segmentation. In ECCV, 2022.
[47] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu-
tions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] the Appendix.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material URL)? [Yes] See the Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report bars (c.g.. with respect the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, you cite the creators? [Yes]
(h) Did you mention the license of the assets? [No]
(c) Did you include any
either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're 're
ing/curating? [No]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) you describe any potential participant risks, with links to Institutional Review Board (IRB)
approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "gRcSA0hKXc", "reviewer": "Reviewer_NDMW", "summary": "This work aims to address the domain generalization problem in semantic segmentation. A simple and effective adversarial augmentation technic is proposed which changes global statistics at the image level. Through adversarial learning, the adversarial style features seem to capture the characteristics of other datasets successfully and improve the performance significantly.", "questions": "I want to double-check if the target domain data are not used at all during training. So the results in Table 1 are obtained by training model on the GTAV training data only, and then applied to other real-world validation data. Right?  \n\nSince the channel-wise augmentation is applied to the global region, I wonder if a local augmentation can further improve the performance.\n", "limitations": "The limitations and potential negative societal impact have been well described.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The main idea of AdvStyle is simple and easy to implement. I believe this can benefit many computer vision tasks, especially the dense prediction tasks.\n\n- The illustration of how style changes affect the segmentation performance in Figure 1 is good. It can be clearly observed that the mIoU decreases a lot when only changing scene colors. \n\n- The t-SNE visualization in Figure 5 provides good proof for the AdvStyle idea: the adversarial style features can capture some characteristics of other datasets. \n\n- Sufficient experiments have been done, including comparisons with common data-augmentation technics, SOTA DG methods and extra experiments on classification tasks.\n\nWeaknesses\n- In Figure 1, It is claimed that image-level mean-variance is used as the style feature. While it is not clear how to calculate and apply such style features.  In Line 70, it is claimed that “6-dim feature for each example”. So, I wonder if the style features are just mean & variance of R-G-B channels. If so, why not use other color spaces or even texture spaces? I think this is an important ablation study that is missing in the main paper.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "acXP9BAaRk", "reviewer": "Reviewer_Z9RR", "summary": "This paper propose a new adversarial style augmentation strategy (AdvStyle) for domain generalization in the semantic segmentation task. The approach of AdvStyle can generate hard stylized images during training, preventing the model from overfitting on the source domain. The generation is completed via learning adversarial style feature. Experiments on two semantic segmentation benchmarks demonstrate the effectiveness of AdvStyle. ", "questions": "1. The proposed method updates the style feature via the direction of the gradient as shown in Eq. (3). This is very similar to the strategy of obtaining adversarial samples (adversarial perturbations). Thus, I wonder how to control the parameter of $\\gamma$ in Eq.3? Why use both learning rate for $\\mu$ and $\\sigma$?", "limitations": "I agree with authors' stated limitations in the supp, which is the training time. The author can consider how to reduce the training cost.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe proposed approach can obviously improve the performance of baselines' performance on unseen domains, including segmentation and classification tasks.\n\nWeakness:\n1. The proposed AdvStyle is indeed an approach to extend the appearance diversity of the training images, and this approach can not ensure the performance on one unseen domain, i.e., this method lacks the interpretability.\nAnd AdvStyle can improve the generalization ability via color augmentation, but more augmentation effects are needed, such as lighting for the generalization of a model from normal-light training data to low-light testing data.\n\n2. Why not apply the AdvStyle on SOTA classification baselines? like ME-ADA in Table 6 and L2D in Table 7? I wonder whether the performance improvement is highly related with the baseline' performance.\n\n3. The superiority compared with current SOTA augmentation method is not obvious. Like the MixStyle reported in Table 2 of supp. More comparison is needed and such comparisons should be placed in the main paper, not supp.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "5qILJQ3D6eH", "reviewer": "Reviewer_B93V", "summary": "The paper addresses the general problem of learning a model in one domain and testing it in a second domain, where there is some domain shift between the domains. Specifically, it demonstrates a method that improves testing performance on real, unknown data when learning on synthetic, rendered data in semantic segmentation for autonomous driving. The motivating observation is that in urban semantic segmentation for autonomous driving a concise domain shift is measurable in the per-channel mean and variance of the image data. In other words, the color statistics between different data sets (and acquisition conditions) differ. The authors propose to explicitly target adversarial changes of the mean and variance of the first domain images during training to improve generalization performance in a second, unknown, domain.\n\nThe method works in two phases for each batch iteration, where the mean and variance of the input image can be modified during the forward pass: adversarial updating of the mean/variance perturbation based on the current, frozen model prediction, and then regular updating of model weights with both adversarial as well as unperturbed samples.\n\nThe paper continues to evaluate the performance of the proposed method on one synthetic versus three real datasets in semantic segmentation for autonomous driving (GTAV versus CityScapes, BDD, Mapillary), as well as several choices for backbones and batch normalization schemes. The empirical performance results demonstrate significant improvements over the chosen baselines. Furthermore, the paper extends the results to image classification on two relevant benchmarks, \"Digits\" and and \"PACS\"", "questions": "I would like to hear the author's thoughts on my concerns on image augmentation as a fair baseline above.\n\nSome minor comments that do not affect the rating:\n- While inspecting tables 3 and 4, I noticed that the setup is described the same (Source/backbone), but the baselines differ. For instance, CityScapes changes from 21.64 to 28.95. It took a couple of reads to understand that the baseline in table includes Color Jittering and Gaussian Blur. Mentioning this fact in the exposure may make readings smoother.\n- Table 3: Would it make sense to add a row combined \"Ours\" with \"AdvPixel\"? It may be interesting to see if these are complementary.\n", "limitations": "The limitations of the work are fairly clear from the submission.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is written well and the method is laid out with sufficient clarity and detail. The datasets chosen for comparison are relevant and realistic (in terms of potential real-world applications of the method). I appreciate the motivation: We observe measurable population differences between various datasets and that they can be captured in a compact and semantically meaningful vector already (image mean and variance). How can we use this prior knowledge to improve task performance in semantic segmentation? Based on this motivation, the authors conclude to model the observed perturbation explicitly as it happens to be a natural part of compact image formation models already (in the sense that normalization with these statistics is well known and commonly used). Once it is modeled, the authors show that it is a comparatively simple step to adversarially predict a new set of samples that are harder to predict at the current training state. An appealing property of the method is that the perturbation space is fairly low-dimensional (6-vector) and thus more easily characterized compared to high-dimensional perturbations for instance on the level of individual pixels.\n\nA significant concern with the submission is that it poses its method as domain generalization (DG) rather than image augmentation (IA) and the subsequent analysis and choice of baselines. I believe this confounds the comparison to prior art and comparable methods. If I take the DG perspective I'd have to ask: Will this work for other DG tasks that do not make strong assumptions about the signal (image formation process, meaning of mean/variance)? I can find no evidence for this in the paper and it appears that the main premise is a strong domain assumption (the image formation process) and selection of data to conform with this assumption (i.e. the datasets are chosen such that they differ demonstrably in the chosen and explicitly modeled statistic). Alternatively, the message of the submission could be that for a set of domains, I can a priori examine some low-dimensional formative process that I then can exploit to generate better samples (in terms of generalization across the prior domains). This could be reformulated from the perspective of image augmentation (IA). If I take on the perspective of IA, I'd have to ask if the cited prior art, baselines and benchmarks are chosen appropriately. For prior art, there would be significant references missing. For instance, [1] and [2] below demonstrate generic, in-training adversarial augmentation. Conceivably, per-channel image mean and variance could be included in these approaches as well without fundamental changes to the algorithms. How would the presented method compare in this case? Could it be extended to other explicitly modeled image formation processes (luminance, in-plane shift, etc.)? \nAs an example: Let's say my domain shift were in-plane rotations (e.g. relatively static car-based camera vs. smart phone from pedestrian view). The presented method would not be expected to demonstrate much benefit, as the domain assumption of shift in image mean and variance are violated. One could probably model the in-plane rotation with a similar adversarial approach to sample generation, but this essentially requires explicit knowledge of the domain shit in a low-dimensional, parametric, forward way. \n\nAs it is presented, the empirical results are appealing as the method is relatively straightforward and improves over the presented baselines, but the level of contribution is unclear as the choice of baselines lack a thorough sample of and comparison to state of the art image augmentation methods.\n\n```\n[1] @inproceedings{\nwang2021augmax,\ntitle={AugMax: Adversarial Composition of Random Augmentations for Robust Training},\nauthor={Haotao Wang and Chaowei Xiao and Jean Kossaifi and Zhiding Yu and Anima Anandkumar and Zhangyang Wang},\nbooktitle={Advances in Neural Information Processing Systems},\neditor={A. Beygelzimer and Y. Dauphin and P. Liang and J. Wortman Vaughan},\nyear={2021},\nurl={https://openreview.net/forum?id=P5MtdcVdFZ4}\n}\n\n[2] @inproceedings{\nzhang2020adversarial,\ntitle={Adversarial AutoAugment},\nauthor={Xinyu Zhang and Qiang Wang and Jian Zhang and Zhao Zhong},\nbooktitle={International Conference on Learning Representations},\nyear={2020},\nurl={https://openreview.net/forum?id=ByxdUySKvS}\n}\n```", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept