Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation
Authors: Zhun Zhong
Yuyang Zhao
Gim Hee Lee
Nicu Sebe
Venue: ICLR 2022 Submitted (2022)
Keywords: Domain Generalization
Semantic Segmentation
Adversarial Style Augmentation

Abstract: In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn a robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) domains. Our study finds that the image style variation can largely influence the model's performance and the style features can be well represented by the channel-wise mean and standard deviation of images. Inspired by this, we propose a novel adversarial style augmentation (AdvStyle) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training. The learned adversarial style feature is used to construct an adversarial image for robust model training. AdvStyle is easy to implement and can be readily applied to different models. Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model performance on unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.

Under review as a conference paper at ICLR 2022
ADVERSARIAL STYLE AUGMENTATION FOR DOMAIN GENERALIZED URBAN-SCENE SEGMENTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) do- mains. Our study finds that the image style variation can largely influence the model's performance and the style features can be well represented by the channel- wise mean and standard deviation of images. Inspired by this, we propose a novel adversarial style augmentation (AdvStyle) approach, which can dynamically gen- erate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training. The learned adversarial style feature is used construct an adversarial image for ro- bust model training. AdvStyle is easy to implement and can be readily applied different models. Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model per- formance on unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.
INTRODUCTION
Semantic segmentation plays a critical role in autonomous driving, which has achieved impressive improvements with the recent development of deep segmentation networks (Long et al., 2015; Chen al., 201 8a; Badrinarayanan et al., 2017). However, these achievements have been mostly attributed to large-scale labeled segmentation datasets, in which annotating pixel-wise labels is very expensive and time-consuming. In addition, the model trained on one dataset commonly produces poor perfor- mance on unseen datasets captured in different conditions. This degradation phenomenon is mainly caused by domain shifts (Choi et al., 2021), 1), including differences in weather, season, light, category staties, etc. For instance, the segmentation model trained on the dataset captured in sunny London will have low accuracy when deployed on the streets of Zurich in rainy weather.
To address the cross-domain problem, domain adaptation methods (Tsai al., 2018: Vu al., 2019; Luo et al., 2019; Zhang et al., 2021) are designed to transfer the knowledge of labeled source data to unlabeled target data. However, one of their main drawbacks is that they require the use of target data during training, which cannot always be accessible in practice. Another promising line is domain generalization (DG), which focuses on learning a generalizable model using only the labeled source domain. To reduce the annotating cost and protect data privacy, the existing DG works (Choi et al., 2021: Yue et al., 2019) in semantic segmentation commonly choose learn the robust model with synthetic data, e.g., GTAV (Richter et al., 2016). In this paper, we focus on this synthetic-to-real DG problem for semantic segmentation.
To eliminate the impact caused by the large domain gap between synthetic and real data, existing solutions mainly aim at augmenting the synthetic source data with extra real-world samples (Yue et al., 2019; Huang et al., 2021) learning domain-invarian features with carefully designed mod- ules (Pan et al., 2018: Choi et al., 2021). The key idea behind them to avoid the model overfitting on the source domain. This work follows this idea and introduces perspective of image style for domain generalized semantic segmentation, which is motivated by the observations in Fig. 1. First, when showing the samples of different datasets in Fig. 1(a) we observe
augmenting approach in the

---

Under review as a conference paper at ICLR 2022
ChyySapes
RDD
Map
GTAV
Randonn Noite & tots mintt
CoSeyle 110%% mioU)
Change Style
mloUt
miot)
(b) Examples of chunging styles for GTAV, Figure 1: (a) Examples of different datasets. The image styles from different datasets are commonly very different. (b) Examples of changing style feature for GTAV sample, including adding ran- dom noise and replacing the style feature with one of samples from the other datasets. The mloU performance is largely reduced when applying the four style variations to the GTAV testing set.
(a) Examples of different datascts.
that the image styles are quite different among them, e.g., the road color. Second, the channel-wise mean and standard deviation of an image, which is called style feature in this paper, can well repre- sent the image style. When changing the style feature, the image style of an example varies while the semantic content is well maintained (see Fig. 1(b)). Third, changing the style features of testing samples will largely deteriorate the model performance (see the numbers in Fig. 1(b)). This indicates that the model performance is highly related the style distributions of the testing set.
Taking the above observations into consideration, we argue that the image style is an important factor that affects the model performance and propose the adversarial style augmentation (AdvStyle) for domain generalized semantic segmentation. Specifically, AdvStyle contains two steps: adversarial style learning and robust model learning. In adversarial style learning, training sample into style feature and normalized image. The style feature is regarded as a learnable parameter, which used to reconstruct a new training example together with the normalized image. Then, we feed the reconstructed example into the segmentation model and optimize the style feature using the adversarial segmentation loss. The updated style feature is called adversarial style feature and is used to produce hard example in the following step. In robust model training, we first generate an adversarial example by de-normalizing the normalized image with the learned adversarial style feature. The adversarial image and the original image are then used to train robust model using the segmentation loss. In AdvStyle, the adversarial image is dynamically generated based on the current model. In this way, the model is always encouraged to update with difficult styles and thus will be more robust to style variations in unseen domains. In Fig. 2, we show the comparison between AdvStyle and traditional augmentation methods. Our AdvStyle significantly improve the model performance on unseen target domains and clearly outperforms the other augmentation methods. To summarize, our contributions are threefold:
first decompose the
We propose the novel adversarial style augmentation (A (AdvStyle) for domain generalized seman- tie segmentation, which can consistently improve the results on unseen real domains. AdvStyle introduces very limited learnable parameters (6-dim feature for each example) and can be easily implemented with different networks and DG methods.
Experiments on two synthetic-to-real DG benchmarks demonstrate the effectiveness of the pro- posed AdvStyle and show that we achieve new state-of-the-al DG performance. We show that AdvStyle can also be applied to single DG in image classification and can produce oftte-of-theaanee art accuracy two datasets.
Original (m1.412219)
(m5)))225))
Sdyle (m7%))-37%)
Caler lnterin (mbel (-6%)) Mar Waciin (1890
Ae llur kltter (mlal
Figure 2: Illustration of different data augmentation methods. We use GTA5 as the source domain and the ResNet-50 as the backbone. The mloU given in parentheses is evaluated on CityScapes validation set for the model trained with the corresponding augmentation method.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
Domain Generalization (DG) in Semantic Segmentation. To tackle the deficiency of annotated segmentation data, DG is introduced to learn a robust model with one or multiple source domains, where the model is expected perform well unseen domains. Recent works mostly use synthetic data as the source domain, which can be automatically generated but have a large distribution gap to real-world datascts. One main stream of DG methods (Yue al., 2019; Huang et al., 2021) focuses on augmenting training samples with extra real-world data from ImageNet (Deng et al., 2009). Learning domain-invariant features (Choi et al., 2021; Pan al., 2018; Tang et al., 2021) is another stream to narrow the domain gap. Pan et al. (2018) and Choi et al. (2021) leverage instance normalization (Ulyanov et al., 2016) and whitening transformation to remove the domain- specific information, respectively. Tang et al. (2021) exchanges style features of two samples and adjusts style features with the attention mechanism. Different from the above methods, our AdvStyle generates new samples by learning adversarial styles using only the synthetic source data.
Adversarial Training in DG. Adversarial training (Goodfellow et al., 2015) is initially proposed combat imperceptible perturbations. In recent years, adversarial
to learn robust model that training is applied to single DG in image classification (Volpi et al., 2018; Qiao et al., 2020; Qiao & Peng. 2021; Fan et al., 2021), by regarding adversarial samples augmented unseen samples. Volpi et al. (2018) is the first to introduce adversarial samples in DG by max-min iterative training procedure. Later methods form novel domains with the generated adversarial samples and learn domain- invariatn representation by meta-learning (Qiao et al., 2020; Qiao & Peng, 2021) or adaptive normalization (Fan et al., 2021). Different from them, this paper adopts adversarial training for semantic segmentation and generates adversarial samples in the perspective of image style.
Style Variation. Style features are widely studied in image translation (Huang Belongie, 2017; Dumoulin et al., 2017). By varying the style features, the image style can be changed while semantic content will be maintained. Inspired by this, recent works focus on generating data of novel distri- butions by modifying style features, which are used to train more robust model. One effective manner is to generate new styles by exchanging (Zhou et al., 2021; Zhao et al., 2021) or mixing styles (Tang et al., 2021) between samples. On the other hand, new styles can be generated by learn- able modules (Wang et al., 2021). Instead, we generate novel styles by adversarial training, which encourages the model always optimize with hard stylized examples. This work is also closely related to Bhattad et al. (2020), which generates adversarial examples by colorizing. However, requires pre-trained colorization model to change the image color, which is much more complex than our AdvStyle. In addition, Bhattad et al. (2020) aims to impair the performance of models by adversarial examples. In contrast, our AdvStyle leverages the adversarial examples improve the generalization ability of the segmentation model. This work also has a connection with "Learning- Do-Simulatte (Ruiz et al., 2019). However, Ruiz et al. (2019) tries to learn good sets of parameters for an image rendering simulator in actual computer vision applications while we attempt to learn a generalized model for semantic segmentation.
3 METHOD
Problem Definition. Synthetic-to-real domain generalization (DG) focuses on training robust model with one labeled synthetic domain S, where the model is expected to perform well on unseen domains (Ti, T2,-.. ... } of different real- world distributions. As stated by Volpi et al. (2018), the DG task can be formulated as solving the worst-case problem:
min sup ET [Crask (0;T)),
(I)
T:D(S,T)SP
where 0 is the model parameters and T is the target domains. Lusk denotes the task-specific loss function, which is the pixel-wise cross-entropy loss in this paper. D(S, T) denotes the distribution distance between the source domain and target domains in semantic space. It is constrained to be lower than p for semantic consistency. Inspired by Eq. 1. we propose novel approach to generate a dynamic source domain S+, which can help us to reduce the domain shifts between the synthetic domain S and real-world domains TT during training.
3.1 OVERVIEW
In the introduction, we show that the image style is an important factor that influences the DG per- formance. In addition, the whannel-wise mean and standard deviation of image, which is called
3

---

Under review as a conference paper at ICLR 2022
Normed Image
Original Image
Reconstructed image
Freeze
Forward
Backward
Norm
DeNorm
Lose (R)
Style Feature
Segmentation
Original Prediction
Model
Adversarial Style Learning
Update
Model Learning
Adversarial Prediction
Original Image
Adivessaiial Style Feature
Adversarial Image
Update
Freeze
(xrt((
DeNorm
Loze (X)
Segmentation Model
Image
Original Prediction Figure 3: The framework of the proposed adversarial style augmentation.
style feature, can well represent the image style. Inspired by that, we propose the adversarial style augmentation approach (AdvStyle) for domain generalized semantic segmentation. AdvStyle can dynamically generate images with new styles during training and effectively improve the generaliza- tion ability of the segmentation model. AdvStyle includes robust model training. In adversarial style learning, we first decompose the image into normalized image and style feature and then update the style feature by adversarial segmentation loss. In robust model training, we compose the adversarial image by the normalized image and learned adversarial style feature. The model is then optimized with both original and adversarial images. These two steps are implemented in each training iteration, enabling us dynamically generate hard stylized samples the current model. Our overall framework is illustrated in Fig .3.
steps: adversarial style learning and
3.2 ADVERSARIAL STYLE LEARNING
Given an training image r at each iteration, we first compute the channel-wise mean and standard deviation a of and then obtain the normalized image by normalization:
HW
Σ
Fha
V HW
Σ
(Ih,w 11)
σ
p
kEH,wEW
NeH,WEW
where H and W denote the spatial size of
After that, we initialize the adversarial style feature fct and a by and which are regarded as learnable parameters. Then we reconstruct the image with μt, at and and forward into the network for loss computation. During the backward, the parameters of the network are fixed and the adversarial style feature is updated by:
4* + J + +VV,
fut Legg (0; -+ +- Lay Logg 2),
(3)
where 7 is the learning rate for adversarial style learning and Lug is the cross-entropy loss. f T F.o+ + 1x+ is the reconstructed image. Notice that the style feature is optimized by the adversarial gradient of Leeg. Indeed, the adversarial style learning process can be iterated multiple times. In our experiment, we find that one-step adversarial style learning achieves similar results with multi-step ones but is much more efficient. Therefore, we only update the style feature once.
3.3 ROBUST MODEL. TRAINING
Given the learned adversarial style feature (μ² and ot), we use it to generate the adversarial sample x with the corresponding normalized image:
- - + jut.
(4)
Then the original image and the generated adversarial image at are forwarded to the model for optimization, which can be formulated by, min (0: x) + Laeg (0; t
(5)
The detailed training procedure and Pytorch-like pseudo-code can be found in Appendix B. During testing, directly input the original samples into the network without implementing AdvStyle.
4

---

Under review as a conference paper at ICLR 2022
3.4 DISCUSSION
Adversarial data augmentation have been studied by several works for DG in image classification. Most of them (Qiao et al., 2020; Volpi al., 2018) generate pixel-wise perturbations on the training image which usually require additional constraint loss to guarantee the semantic consistency in Eq. 1. In addition, these works focus on the image classification task where the recognition result mostly related to global feature. However, in semantic segmentation, the model needs to produce the per-pixel predictions so that it is more difficult to ensure the pixel-wise semantic consistency during adversarial learning. Instead, our AdvStyle varies the style feature of the image, which will maintain the semantic content of most pixels and thus can well guarantee the pixel-wise semantic consistency for semantic segmentation. We conduct experiments in Table 3, which demonstrate the superiority of the proposed AdvStyle over pixel-wise adversarial learning.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. For the synthetic-to-real domain generalization (DG), we one of the synthetic datasets (GTAV (Richter et al., 2016) or SYNTHIA (Ros et al., 2016)) as the source domain and evaluate the model performance on three real-world datasets (CityScapes (Cordts al., 2016). BDD-100K (Yu et al., 2020), and Mapillary (Neuhold et al., 2017)). GTAV (Richter et al., 2016) contains 24,966 images with the size of 1914x 1052. It splited into 12,403, 6.382, and 6,181 images for training, validating, and testing. SYNTHIA (Ros et al., 2016) contains 9,400 images of 960x where 6,580 images are used for training. We use the validation sets of the three real-worle datasets for evaluation. CityScapes (Cordts et al., 2016) contains 500 validation images of 2048 x 1024, collected primarily in Germany. BDD-100K (Yu et al., 2020) and Mapillary (Neuhold et al., 2017) contain 1,000 validation images of x280x720 and 2,000 validation images of 1920 1080, respectively.
Implementation Details. Following Choi et al. (2021), we use DeepLab V3+ (Chen et al., 2018b) as the segmentation model. The segmentation model is constructed by three backbones, including MobileNetV2 (Sandler et al., 2018). ResNet-50 (He et al., 2016) and ResNet-101. We adopt SGD optimizer with an initial learning rate 0.01, momentum 0.9 and weight decay 5x 10 to optimize the model. The polynomial decay (Liu et al., 2015) with the power of 0.9 is used as the learning rate scheduler. The learning rate of AdvStyle y is set 3. All models are trained for 40K iterations with a batch size of 16. Four widely used data augmentation techniques are used during training, including color jittering, Gaussian blur, random cropping and random flipping. The input image is randomly cropped to 768 for training. The image of the original size is used for testing.
Evaluation Metric. Following Choi et al. (2021), the model obtained by the last training iteration is used to evaluate the mloU performance on the three real-world validation sets. For each method, we report the result averaged on 3 runs. When using GTAV as the source domain, we use the 19 shared semantic categories for training and evaluation. When using SYNTHIA as the source domain, we use 16 shared categories for training and evaluation, i.e., ignoring the train, truck, and terrain categories.
4.2 EVALUATION
Effectiveness of AdvStyle Different Models. Our AdvStyle is model-agnostic method, which can be directly applied to different models without modifying the models. To verify the effective- ness of AdvStyle, we apply it to models with different backbones and normalization modules. The backbones include MobileNetV2, ResNet-50 and ResNet- 101. The normalization modules include vanilla batch norm (baseline), instance-batch norm (IBN-Net (Pan al., 2018)), and instance selec- tive whitening (ISW (Choi et al., 2021)). In Table 1, we show the results of using GTAV as the source domain. We can make the following conclusions. First, injecting instance-batch norm (IBN-Net) and instance selective whitening (ISW) modules can consistently improve the performance of the baseline. Second, the proposed AdvStyle can significantly enhance the generalization performance of the bascline model for all backbones. Specifically, the average mloU is increased from 26.03%, 27.42% and 31.47% to 32. 1% %. 37.39% and 37.34% for MobileNetV2, ResNet-50 and ResNet- 101, respectively. In addition, the baseline with AdyStyle produces higher average mIoU than IBN-Net
5

---

Under review as a conference paper at ICLR 2022
Methods
MobileNetV2
ResNet-50
ResNet-101
GTAV
B
M
Mean
C
B
M
Mean
C
M
Mean
Baseline
25.92
25.73
26.45
26.03
28.95
25,14
28.18
27.4
32.97
30.77
30.68
31.47
+AdvStyle
31.81
33.01
31.50
32.11
39.62
35.54
37.00
37.39
39.52
36.39
36.10
37.34
IBN-Net
30.14
27.66
27.07
28.29
33.85
32.30
37.75
34.63
37.37
34.21
36.81
36.13
+AdvStyle
32.45
31.55
33.09
32.36
39.32
36.42
40.82
38.85
44.04
39.96
42.67
42.22
ISW
30.86
30.05
30.67
30.53
36.58
35.20
40.33
37.37
37.20
33.36
35.57
35.38
+AdvStyle
33.23
31.84
32.00
32.36
39.60
38.59
41.89
40.03
43.44
40.32
41.96
41.91
Table 1: Evaluation of the proposed AdvStyle on different methods (Baseline, IBN-Net (Pan et al., 2018) and ISWiChoi al., 2021)) and backbones (MobileNetV2 (Sandler et al., 2018), ResNet- 50 (He et al., 2016), and ResNet-101), All models are trained on the GTAV training set and tested on CityScapes (C), BDD-100K (B), and Mapillary (M) validation sets.
Methods (SYNTHIA-)
CityScapes
BDD
Mapillary
Mean
Baseline (Choi et al., 2021)
34.94
21.96
27.94
28.28
BavSSlccAAdvSyye
37.59
27.45
31.76
32.27
IBN-Net (Pan et al., 2018)
35.83
23.62
28.88
29.44
IBN-Net+AdvStyle
38.72
28.55
33.59
33.62
ISW (Chai et al., 2021)
35.27
23.54
26.72
28.51
ISW+AdvStyle
39.74
28.33
32.87
33.65
Table 2: Results of using SYNTHIA as the source domain.
The backbone is ResNet-
Color Jittering Gaussian Blur AdvPixel
AdvStyle CityScapes
BDD
Mapillary
Mean
21.64
22.85
24.22
22.91
26.36
23.82
26.33
25.50
25.77
24.05
26.71
25.51
23.34
28.42
30.64
27.46
37.51
33.74
34.73
35.32
28.95
25.14
28.18
27.42
35.42
33.28
33.23
33.97
39.62
35.54
37.00
37.39
Table 3: Comparison of different augmentations. Source: GTAV; Backbone: ResNet-50.
and ISW for all backbones. Third, when adding AdvStyle, the results of IBN-Net and ISW can be further improved for all settings. For example, when using ResNet-101 as the backbone, AdvStyle improves the average mloU of IBN-Net and ISW by 6.09% and 6.53%, respectively. In Table 2, show the results of using SYNTHIA as the source domain and also observe clear improvements for AdvStyle. These results verify the prominent advantage of our AdvStyle different models.
Comparison of Different Augmentation Techniques. We investigate the impact of different aug- mentation methods, including color jittering, Gaussian blur, AdvPixel (Volpi et al., 2018) and the proposed AdvStyle. AdvPixel is a state-of-the-art method for domain generalized image classifi- cation. The main difference between AdvPixel and AdvStyle is that AdvPixel learns pixel-wise adversarial example while AdvStyle learns style- wise adversarial example. We reproduce AdvPixel in our setting and select the adversarial learning rate (=10) that achieves the best performance. The random cropping and random flipping are used in default.
Results in Table 3 show that all four augmentation methods can improve the generalization perfor- mance. Importantly, our AdvStyle produces significant improvement compared to other three meth- ods. Specifically, when using AdvStyle, the average mloU is increased from 22.91% 91% to 35.32%. This improvement is about 8% higher than the other 3 methods. Moreover, AdvStyle is well com- plementary to color jittering and Gaussian blur. When combining these three methods, the mloU is further improved in all target domains. Compared to AdyPixel, our AdvStyle achieves clearly higher performance, no matter using color jittering and Gaussian blur. This demonstrates the advantage of learning adversarial style in domain generalized semantic segmentation.

---

Under review as a conference paper at ICLR 2022
Methods (GTAV) CityScapes
BDD
Mapillary
Mean
Baseline
28.95
25.14|
28.18
27.42
RandStyle
33.40
34.14
31.67
33.07
MixStyle
35.53
32.41
35.87
34.60
CrossStyle
37.26
32.40
34.09
34.58
AdvStyle
39.62
35.54
37.00
37.39
Table 4: Comparison of different style-aware methods. Source: GTAV; Backbone: ResNet-50.
Comparison of Different Style-Aware Methods. In Table 4, we compare AdvStyle with three style-aware augmentation methods, including MixStyle (Zhou al., 2021), CrossStyle (Tang al., 2021) and RandStyle. MixStyle mixes the styles of two samples with a convex weight while CrossStyle directly swaps the styles of two samples. RandStyle our AdvStyle, which randomly adds Gaussian noise into the style feature. All style-aware methods are implemented on the image-level for fair comparison. We can find that (I) all style-aware meth- ods can consistently improve the performance on all target domains and (2) AdvStyle achieves the best results. The first finding verifies the effectiveness of augmenting image styles and the second finding shows the benefit of learning adversarial styles over other style-aware methods for domain generalized semantic segmentation.
be regarded reduction of
4.3 COMPARISON WITH STATE-OF-THE-AN METHODS
In Table 5, we compare our method with state-of-the-art DG methods in semantic segmentation, in- cluding IBN-Net (Pan al., 2018), SW (Pan et al., 2019), IterNorm (Huang et al., 2019), ISW (Choi et al., 2021), DPRC (Yue et al., 2019) and FSDR (Huang et al., 2021). The source domain is GTAV and the backbones are ResNet-50 and ResNet-101. Note that, since different methods use different segmentation networks (e.g., DeepLabV2 (Chen et al., 2018a), DeepLabVV+ (Chen et al., 2018b) and FCN (Long et al., 2015)), different training sets (e.g., the whole GTAV and the training set of GTAV), different training strategies (e.g., learning rate and optimizer). different auxiliary data (e.g., ImageNet samples) and different evaluation manners (e.g., the best model and the last model), is hard to compare them in absolutely fair way. We show the results of each method well as the absolute gain against the corresponding baseline.
From Table 5, can make the following conclusions. First, when using the same baseline model, adding AdvStyle can produce better results than IBN-Net, SW, IterNorm and ISW. More- over, when applying AdvStyle IBN-Net ISW, achieve new state-of-the-art performance for both ResNet-50 and ResNet- 101. Second, when compared across baselines, "Bassline+AdvStyle"" achieves the state-of-the-ar art mloU for ResNet-50. On the other hand, when using ResNet-101 as the backbone, "IBN-Net+AdvStyle produces higher results than DRPC and comparable results with FSDR. Importantly, both FSDR and DRPC use extra ImageNet images and select the best training checkpoints for each target domain. Instead, TIBN-Net+AdvStyy uses the last training checkpoint to evaluate target domains. Third, when using the best check- point for evaluation, we AdvSylll)" produce better performance than DRPC and FSDR, leading to the new state-of-the-art results under the "best checkpoint setting". Even so, we argue that it is more reasonable use the last checkpoint for evaluating all target domains. This is be- cause we can not always have the right labeled validation sets to select the best model for unseen domains in practice.
only utilizes the source data and
To make more fair comparisons with the state-of-the of--oh--ttt-rn--- ahe methods, we also use the whole set of GTAV for training ISW and "ISW "AsSttdvStyle"., The results with ResNet- 101 are reported in the last two rows of Table 5. We can find that using the whole set of GTAV can produce higher results on the CityScapes and the Mapillary datasets.
4.4 VISUALIZATION
Qualitative Comparison of Segmentation Results. In Fig. 4, we compare the segmentation results for diffeernt methods on target domains. is clear that, the proposed AdvStyle can consistently im- prove the semgentation results for baseline, IBN and ISW models, especially for the easily-confused classes, e.g., road vs sidewalk and building vy sky. More results can be found in Appendix E.
7

---

Under review as a conference paper at ICLR 2022
Net ID
Methods (GTAV)
CityScapes
BDD
Mapillary
Mcan
Baseline
22.20
N/A
N/A
N/A
IBN-Net
29.60 7.40
11
Baseline"
32,45
26.73
25.66
28.28
ResNet-50
DRPC
37.42 4.97f
32.14
5.417
34.12
8.461
34,56
6.281
III
Baseline
28.95
25.14
28.18
27.42
BassSine+AAdSSyye
39.62
10.671
35.54
10.41
37.00
8.821
37.39
9.971
III SW IterNorm
29.91
0.96t
27.48
2.341
29.71
1.53t
29.03 161f
III
31.81
2.86
7.56 32.70
33.88
5.71
32.79
5.371
III
IBN-Net IBN-Net+AdvStyle
33.85
4.90f
32.30
7.16
37.75
9.57f
34.63
7.21f
III
39.32
10.371
36.42
11.28
40.82
12.64
38.85
11.43t
III
ISW
36.58
7.63t
35.20
10.06*
40.33
12.
37.37
9.951
III
ISW+AdvStyle
39.60
10.65t
38.59
13.45t
41.89
13.71
40.03
12.61f
Baseline"
33.4
27.3
27.9
29.53
IBN-Net*
40.3
6.9t
35.6
8.31
35.9
8.0
37,26
7.73
FSDRS*
44.8
11.41
41.2
13.9
43.4
15.5t
43.13
13.6
ResNef-100 101
Baseline" DRPC
33.56
27.76
28.33
29.88
II
42.53 8.971
38.72
10.96
9.72f 38.05
39.76
9.881
Baseline
32,97
30.77
30.68
31.47
III
AdvSSiee+AdvSSyye
39.52
6.55t
5.621 36.39
36.10
5.42f
37.34
5.871
III
IBN-Net IBN-Net+AdvStyle
37.37
4.40f 44.04
34.21
3.44
36.81
6.13t
36.13
4.661
III
11.07t
39.96
9.19
42.67
11.99
42.22 10.75t
ISW
37.20
4.231
33.36
2.591
35.57
4.891
35.38
3.91f
III ISW+AdvStyle
43.44
10.471
40.32
9.55*
41.96
11,28*
41,91
10.44
III
ISW+AdvStyle
45.62
12.65t
41.71
10.97f
46.69
16.01
44.67
13.201
IV
ISW
37.51
33.54
36.12
35.72
IV
ISW+AevStyle
44.51
39.27
43.48
42.42
Table 5: Comparison with state-of-the-ard domain generalization methods. All models use the GTAV as the source domain. For each backbone, models with the same ID) are implemented with the same baseline. Models of "II==I, II and IV" use the whole set (24,966) for training while models of "ID=III" only use the training set (12,403). The absolute gain of each model is calculated over the corresponding baseline. $ denotes extra using the ImageNet images. indicates using the best trained checkpoints for evaluating each target domain.
Image
Geond muth
Haselino
IBN-Nat
BIN AriiAd
ISW
50D ClySeere
Mesilay
Figure 4: Qualitative comparison of segmentation results. Source: GTAV; Backbone: ResNet-50.
t-SNE of Styles. In Fig. 5, we visualize the style features generated by AdvStyle during the training phase for the GTAV training set, where ResNet-50 is used as the backbone. We can find that Ad- vStyle can continuously generate new style features that are different from the original distribution. The new style features have the chance to be located at the distributions of other datasets during the training process. Moreover, AdvStyle will also generate styles that are out of the distributions of the four datasets (G, C, B, M) and may appear in other unseen domains. The visualization results further demonstrate that AdyStyle can encourage the model to meet more diverse and unseen styles during training, leading more robust model.
8

---

Under review as a conference paper at ICLR 2022
GTAV Scapps
Mapillary
GTAV
Training
Init
Figure 5: I-SNE visualization of adversarial style features during training.
Method
SVHN MNIST-M
SYN
USPS
Avg.
Method
Art.
Car.
Ske.
Pho.
Avg.
ERM
27.8
52.7
39.7
76.9
49.3
ERM
67.4
74.4
51.4
42.6
58.9
CCSA
25.9
49.3
37.3
83.7
49.1
JiGen
69.1
74.6
52.4
41.5
59.4
d-SNE
26.2
51.0
37.8
93.2
52.1
RSC
68.8
53.6
41.9
59.7
JiGen
33.8
43.8
77.2
53.1
L2D
74.3
77.5
54.4
45.9
63.0
ADA
35.5
60.4
45.3
77.3
54.6
ERM+AdvStyle
75.8
76.6
58.1
51.1
65.4
M-ADA
42.6
67.9
49.0
78.5
59.5
RSC+AdvStyle
75.1
78.0
58.9
55.5
66.8
ME-ADA
42.6
63.3
50.4
81.0
59.3
Table 7: Accuracy of single domain general-
ERM+AdvStyle
50.4
73.4
58.7
81.6
66.0
ization on PACS. One domain (name in col- umn) is used as the training (source) data and
Table 6: Accuracy of single domain generaliza- tion on Digits. MNIST is used as the training set, and the results on different testing domains are ported in different columns.
the other domains are used as the testing (tar- get) data.
4.5 EVALUATION ON IMAGE CLASSIFICATION TASK
To verify the versatility of the proposed AdvStyle, we evaluate on single DG in image classifica- tion. Experiments conducted on two popular DG datasets, i.e., Digits and PACS. The details of the datasets and implementation can found in Appendix D.
Results on Digits. In Table 6, compare with the baseline (ERM (Vapnik, 2013)) and 6 state-of- the-art methods, including CCSA (Motian et al., 2017), d-SNE (Xu et al., 2019). JiGen (Carlucci al., 2019), ADA (Volpi al., 2018), M-ADA (Qiao al., 2020) and ME-ADA (Zhao al., 2020)- For AdvStyle, we implement it with ERM. is clear that, our AdvStyle can significantly improve the accuracy of ERM on all target domains. other state-of-the-aa methods by large margin. For example, AdvStyle is higher than the best competitor (ME-ADA (Zhao al., 2020)) by 6.7% for the accuracy averaged over 4 target domains.
addition, the proposed AdvStyle outperforms the
Results on PACS. We compare with the baseline (ERM (Vapnik, 2013)) and three state-of-the-an DG methods, including JiGen (Carlucci et al., 2019), RSC (Huang et al., 2020) and L2D (Wang et al., 2021). We reproduce JiGen, RSC and L2D with their official source codes. All methods use the same baseline (ERM). Results Table show that JiGen and RSC produce limited improvements. Instead, our AdvStyle can significantly increase the accuracy all domains for both ERM and RSC. Compared to the recent published work (L2D), our method (ERM+AdvStyle) outperforms by 2.4% in average accuracy.
The results on Digits and PACS demonstrate that our AdvStyle can also be effectively applied to single domain generalized image classification and can achieve state-of-the-an accuracy.
CONCLUSION
In this paper, we propose novel augmentation approach. called adversarial style augmentation (AdvStyle), for domain generalization (DG) in semantic segmentation. AdvStyle dynamically gen- erates hard stylized images by learning adversarial image-level style feature, which can encourage the model learning with more diverse samples. With AdvStyle, the model can refrain from the prob- lem overfitting on the source domain and thus can be more robust to the style variations unseen domains. AdvStyle is easy to implement and can be directly integrated with different models with- out modifying the network structures and learning strategies. Experiments on two synthetic-to tetthe largely improve the generalization performance and achieve state-
settings show that AdvStyle of-the-ar performance. In addition, AdvStyle can be employed to single DG in image classification and obtain significant improvement.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper presents an effective approach for synthetic-to-real domain generalization (DG) in se- mantic segmentation, which can help improve safety in autonomous driving. We do not find obvious Ethics issues for our approach since we only use synthetic data for model training. Be that may, one potential issue is that the performance of existing DG approaches (including our approach) in semantic segmentation is still far away from the practical demand, especially when encountering extreme conditions. This may lead the drivers not be completely ease when using the autonomous driving system.
REPRODUCIBILITY STATEMENT
We provide the implementation details and dataset description in Sec. 4.1 and Appendix D for mantic segmentation and image classification, respectively. In addition, the algorithm and Pytorch- like pseudo-code presented in Appendix B, which allow the researchers easily reproduce / integrate our method with existing semantic segmentation and image classification approaches.
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. TPAMI, 2017. 1
Anand Bhattad, Min Jin Chong, Kaizhao Liang, Bo Li, and DA Forsyth. Unrestricted adversarial examples via semantic manipulation. In ICLR, 2020. 3
Fabio M Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In CVPR, 2019. 9
Liang-Chiell Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected cris. TPAMI, 2018a. 1,7
I Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018b. 5,
Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne Kim, Seungryong Kim, and Jaegul Choo. Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR, 2021. 1, 5, 6, 7, 13, 14
Robustnet:
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 5
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 3, 15
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A leamed representation for artistic style. In ICLR, 3017. 3
Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou. Adversarially adaptive normalization for single domain generalization. In CVPR, 2021. 3
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015. 14
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016. 5, 6, 15
10

---

Under review as a conference paper at ICLR 2022
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain random- ization for domain generalization. In CVPR, 2021. 1,3,7
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, and Ling Shao. Iterative normalization: Beyond standardiza- tion towards efficient whitening. In CVPR, 2019. 7
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. ICCV, 3017.
Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV, 2020. 9,
Jonathan J. Hull. A database for handwritten text recognition research. TPAMI, 1994. 14
Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1989. 14
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales, Deeper, broader and artier domain generalization. In ICCV, 2017. 15
Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: Looking wider to see better. In CoRR, 2015. 5
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015. 1,7
Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR, 2019.
Sacid Motian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep super-
vised domain adaptation and generalization. In ICCV, 2017. 9
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NeurlPS Workshop, 2011. 14
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In ICCV, 2017. 5
Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV, 2018. 1, 5, 6, 7, 13, 14
Xingang Pan, Xiaohang Zhan, Jianping Shi, Xiaoou Tang, and Ping Luo. Switchable whitening for deep representation learning. In ICCV, 2019. 7
Fengchun Qiao and Xi Peng. Uncertainty- model generalization unseen domains. CVPR, 2021. 3
Fengchun Qiao, Long Zhao, and Xi Peng. Learning learn single domain generalization. In CVPR, 2020. 3, 5,
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016. 5,5
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016. 5
Nataniel Ruiz, Samuel Schulter, and Manmohan Chandraker. Learning to simulate. In ICLR, 2019.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chinh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 5,

---

Under review as a conference paper at ICLR 2022
Zhiqiang Tang, Yunhe Gao, Yi Zhu, Zhi Zhang, Mu Li, and Dimitris Metaxas. Selfnorm and cross- norm for out-of-distributiin robustness. ICCV, 2021. 3,7
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmohan Chandraker. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprini urXiv: 2016. 3
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013. 9, 15
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS, 2018. 3,5, 6, 9, 14
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.
Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. ICCV, 2021. 3,9 9
Xiang Xu, Xiong Zhou, Ragav Venkatesan, Gurumurthy Swaminathan, and Orchid Majumder. d- sne: Domain adaptation using stochastic neighborhood embedding. In CVPR, 2019. 9
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madha- van, and Trevor Darrell. Bdd100k: diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020. 5
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Snngiooannn-Vinceneee Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consistency: Simulation-to--eal generalization without accessing target domain data. In /CCV, 2019. 1, 7.
Pan Zhang, Zhang, Ting Zhang, Dong Chen, Yong Wang, and Fang Wen. Prototypical pseudo la- bel denoising and target structure learning for domain adaptive semantic segmentation. In CVPR, 2021. 1
Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entree adversarial data augmen- tation for improved generalization and robustness. NewrIPS, 2020.
Yuyang Zhao, Zhun Zhong, Zhiming Luo, Gim Hee Lee, and Nicu Sebe. Source-free open com- pound domain adaptation semantic segmentation. arXiv preprint (.Xiv:2106.03422,, 2021.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021. 3,7
A PARAMETER ANALYSIS
The proposed AdvStyle has one parameter, i.e., the adversarial learning rate To study the impact of y. we vary it in the range of (0.1, Results in Fig. 6 show that AdvStyle can significantly improve the performance on all target domains even with small value of 7. The best results are achieved when y is between 1 and 10. Assigning too large value 7 (e.g. 40) may produce unrealistic styles and thus hampers the model training.
B ALGORITHM AND PYTORCH-LIKE PSEUDO-CODE
The training procedure and Pytorch-like pseudo-code shown in Alg. and Fig. respectively.
12

---

Under review as a conference paper at ICLR 2022
CityScapes
BDD
45
40
40
mlou (%6
mloU (%)
335
30
30
AdvSStyle Baseline
25
AdvStyle Baseline
20
20
0.5
40
Adv Learning Rate
Adv Leamning Rate
Mapillary
Mean
40
40
35
miou (%)
AdyStyle Baseline
mloU (%)
AdvSStyle Baseline
25
25
20
20
0.5
Adv Learning Rate
Adv Learning Rate
Figure 6: Influence of the adversarial learning rate.
Algorithm The training procedure of AdvStyle.
Inputs: labeled source domain S, segmentation model F parameterized by 0, batch size No, total training iterations max_iter adversarial learning rate 7, and model learning rate a. Outputs: Optimized model F parameterized with 0.
1: for in mar_iter do
2: Sample mini-batch X with N₂ images;
3: // Stage 1: Adversarial Style Learning.
4: Compute channel-wise mean μ., standard deviation a and normalized images & with Eq. 2;
5:
Initialize adversarial style feature: 1** μ4, o+ ;;
6:
Compute adversarial segmentation loss - Leegi
7:
Optimize 5+ and + with Eq. 3; I/ Stage 2: Robust Model Training. Generate adversarial images X with X, 1et and at by Eq.
8:
9:
10: Compute the overall training loss Laey (0; Xt') + Lowg X(+ by Eq 5:
Optimize the segmentation model F: <- a (Leeg (0; X) fa (0: x+)));
12: end for
13: Return F parameterized with
C RESULTS OF MULTI-SOURCE SETTING
In Table 8, evaluate the models under the multi-source domain generalization setting, where both GTAV and SYNTHIA are used as the source data. The compared methods include baseline (Choi et al., 2021), IBN-Net (Pan et al., 2018), ISW (Choi et al., 2021) and our AdyStyle. Clearly, Ad- vStyle consistently improves the results of ISW, further verifying the effectiveness the proposed AdyStyle.
13

---

Under review as a conference paper at ICLR 2022
1
Import torch
2
ddf Stylecciinn,,, gt, net, optim, adv
4
5
Args:
impu:: images
7
gt ground-truch labels
8
not: segmentation network
optim: optimizer
adv Ir. learning of AdvSttle
11
Adversarial Style learning
14
# Get style feature and normalized image input.sizc(0)
15
imput inpul. (emmm keepdim- True)
var imput vart 3], keepdim-True)
18
sig (var 1e-5).sur((
sig detachc))) sig detachh)
input normed (input- mu). sig
21
imput normed impat nornmeed cemme)().cmme(
Set learnable style fenture and adv optimizer adv mu, ady sig mu, sig
24
requires grad. (True)
adv sig. sigurequires grad (Trae)
27
adv optim torch.optim. Stt.h.ppiiimSSSDaaaeaeeee SGD mu, adv_sig), sig), Ir=adv Jr. momentum-O, weight docay-D)
28
Optimize adversarial style feature
adv optim. zero grad()
31
adv input input normed sig+ adv mu
adv output net(adi input)
adv loss torch nn fun.ciinnal.coss.. entropy(ady ourput,
34 adv loss)..cckwandd
optim step()
37
# Robust Model Training net.train()
uptim.zero_ggg grad()
40
adv input input normed adv sig + adv mu inputs torch car((input, adv input), dim=0)
worch.aat((p dim-0)
outputs nettinputs)
Feross entropyyourputs, gt)
backward()
optim. step(
Figure 7: The Pytorch-like pseudo-code of AdvStyye.
Methods (CTAV+SYNTHIA CityScapes
BDD
Mapillary
Mean
Baseline (Choi et al., 2021)
35.46
25.09
31.94
30.83
IBN-Net (Pan et al., 2018)
35.55
32.188
38.09
35.27
ISW (Choi et al., 2021)
37.69
34.
38.49
36.75
ASv+ylee
39.29
39.26
41.14
39.90
Table 8: Results of using GTAV and SYNTHIA the source data. The backbone is ResNet-50.
D DETAILS OF SINGLE DOMAIN GENERALIZATION IN IMAGE CLASSIFICATION
Digits includes five domains (MNIST (LeCun et al., 1989), SVHN (Netzer et al., 2011), MNIST-
M (Ganin & Lempitsky, 2015), SYN (Ganin & Lempitsky, 2015), and USPS (Hull, 1994)) of 10 classes. We use MNIST as the source domain and evaluate the model performance on the other domains. Following ADA (Volpi al., 2018), we use the ConvNet architecture (LeCun al., 1989) as the model and use Adam optimizer with learning rate 10 for optimization. The overall training iteration is set 10,000 with batch size of 32. We set the learning rate of AdvStyle to 20.000
Due the absent of batch normalization layer, the gradient is very small on the style feature. Therefore, set large learning for AdvStyle.
14

---

Under review as a conference paper at ICLR 2022
PACS (Li al., 2017) contains four domains (Artpaint, Cartoon, Sketch, and Photo) of 7 classes. For evaluation, we select one of them as the source domain and the other domains as the target the ResNet18 (He et al., 2016) pretrained on
domains. Following RSC (Huang et al., 2020), ImageNet (Deng et al., 2009) as the backbone and add a fully-connected layer as the classification head. We train the model by SGD optimizer. The learning rate is initially set to 0.004 and divided by 10 after 24 epochs. The model is trained for 30 epochs in total with batch size of 128. The learning rate of AdvStyle is 3.
Baseline. The baseline model is the vanilla empirical risk minimization (ERM) (Vapnik, 2013), which directly uses the source domain train the model with classification loss.
E MORE VISUALIZATIONS
Segmentation Results. In Fig. Fig. and Fig. 10, we provide more segmentation results for the baseline and "baselinc+AdvStylee..
Examples of AdvStyle. In Fig. 11, illustrate more examples generated by AdvStyle.
15

---

Under review as a conference paper at ICLR 2022
Image
Ground truth
Baseline
Baseline+AdvSlyye
road sky
swalk
build rider
wall
fence
pole
tlight train
tsign mcycle
veg bicycle
terrain unlabel
person
car
truck
bus
Figure 8: Segmentation results on CityScapes. Source: GTAV; Backbone: ResNet-50.
16

---

Under review as a conference paper at ICLR 2022
Image
Ground truth
Baseline
Baseline AdvStyle
road sky
swalk
build
wall
fence truck
pole
tlight train
tsign
veg bicycle
terrain unlabel
person
rider
car
bus
mcycle
Figure 9: Segmentation results on BDD-100K, Source: GTAV; Backbone: ResNet-50.
17

---

Under review as a conference paper at ICLR 2022
Image
Ground truth
Baseline
Baseline AdvStyle
road
swalk
build
wall
fence
pole
tlight
tsign
veg
terrain
sky
person
rider
car
truck
bus
train
mcycle
bicycle
unlabel
Figure 10: Segmentation results on Mapillary. Source: GTAV; Backbone: ResNet-50.
18

---

Under review as a conference paper at ICLR 2022
+AdvStyle
Oringal
Blurt Jittering
Oringal
Blurt Jittering
Oringal
Blurt Jittering
Oringal
Blurt Jittering
Oringal
Blurt Jittering
Oringal
Blurt
Jittering
Oringal
Blurt
Jittering
Figure 11: Examples adversarial style augmentation. Source: GTAV; Backbone: ResNet-50.
19


Reviews:
[{"review_id": "yHQ0ZKSgFis", "reviewer": "Reviewer_8yWf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Complete and solid experiments \n2. The image-level style transfer part is intuitive \n3. Paper is easy to follow\n\nWeakness:\nThe main concern I have is the model design and also not sure about some of the authors' intuitions behind:\n1. Based on the algorithm in Appendix B, we only will update the adversarial learning once, or the mean and variance once, then we feed the updated mean and variance to robust model part to reconstruct the scene and then further update the segmentation model. My main concern is about what is really learnt in this one update? Personally, I feel that when updating only once, the generator cannot really learn much, e.g. the reconstructed results can be very far away from the original image and cannot be very meaningful. Then if at later stage we force the predictor to minimize the segmentation loss of both original and reconstructed images, will this introduce noise to our predictor, e.g. the predictor then cannot really do well on either image. And is this scenario possible and how does the proposed method handle this problem?\n2. Is the segmentation model shared between adversarial and robust model learning process? I believe so, right? Then follow up for the first question, will the segmentation model be drifted away by the one-time update and then introduce additional errors to adversarial learning procedure?\n3. How do we decide the maximum number of iterations in algorithm?\n4. One corner/extreme case might be that the generator learns very well after many times of iterations and can perfectly reconstruct original images. Then under this scenario, the segmentation model do not need to learn anything but stay with what it has been initialized with would be the right answer for the model. Or we can relax a bit by thinking that what has been generated/reconstructed can be very close to the original images after many epochs and not much changes are needed for segmentation model. How will the proposed method handle this case?\n5. What is the difference/relationship between the proposed method and papers like \"LEARNING TO SIMULATE\"? \n6. Why the proposed method can produces new styles that cover the distributions of different datasets as shown in Fig.5? Any explanation? ", "summary_of_the_paper": "This paper aims to address the domain generalization problem in semantic segmentation. Starting to synthetic source dataset, the authors propose to apply image-level style transfer to bridge the gap between source and unseen real dataset. Specifically, the image-level style is defined as the mean and variance of each image. And the proposed method learns to reconstruct scenes with their styles and further updates the segmentation model with both original images and reconstructed images.\n\nIn experiments, the authors show that the proposed method can:\n1. achieve SOTA performance on multiple real datasets \n2. generalize well with various backbones\n3. cover the data distributions of real dataset well", "summary_of_the_review": "In general, I feel that the paper provides very good experimental results but I have concerns about their model design. I am willing to change my rating if the authors can address my concerns above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tQsEKLutv69", "reviewer": "Reviewer_FUQj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* This paper tackles a structured prediction problem (semantic segmentation) in the domain generation community.\n* The idea is clearly stated and straightforward with source code included in the appendix.\n\nWeaknesses:\n* The technical novelty is very limited. The key idea (augmenting channel-wise mean and standard deviation) has been explored in the cited reference [MixStyle; Zhou et al., 2021]. The difference is that this paper constrains the augmentation within the first layer with an adversarial formulation. The reviewer fails to perceive a broader impact on the community from this technical contribution.\n* At a high level, AdvStyle is a data augmentation method that generates unconstrained adversarial examples. In this sense, a prior work [**Ref1, Bhattad, et al., In ICLR 2020] is highly-related but not discussed in the paper. Bhattad et al. use a pre-trained colorization or texture-transfer network for adversarial example generation. Please elaborate on this in the rebuttal and incorporate the comparisons in the future version of the paper.\n* This paper only conducts experiments on sim-to-real transfer but does not evaluate real-to-sim transfer or real-to-real transfer. This makes an unfair comparison to the baseline methods including [Choi et al., 2021]. Please elaborate on this in the rebuttal. The reviewer would kindly suggest incorporating these comparisons in the future version of the paper.\n\nReferences:\n\n* Ref1: Unrestricted adversarial examples via semantic manipulation. Bhattad et al., In ICLR 2020.", "summary_of_the_paper": "This paper studies the problem of adversarial domain randomization for urban scene semantic segmentation. Specifically, it proposes an adversarial augmentation method called AdvStyle that injects adversarial signals into the channel-wise mean and standard deviation of images. The style augmentation is used when training on synthetic images from Synthia and GTAV datasets. This paper demonstrates promising results when applied the trained model to real semantic segmentation benchmarks.", "summary_of_the_review": "Overall, this is a borderline paper with promising results. The major concerns include the limited technical novelty, missing references, and unfair experimental comparisons. I would like to see the rebuttal and comments from the other reviewers to make a final decision. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qFoG5cqc_ht", "reviewer": "Reviewer_38TQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n- Definitely, the main point in favor of this paper is the good results obtained by the approach. I believe the approach is well evaluated, making the right choices of synthetic datasets (training), real datasets (testing), semantic segmentation architectures, and baselines (e.g. ISW, which is one of the best performing approaches for domain generalization in semantic segmentation).\n\n- The approach is orthogonal to other solutions, such as ISW. The combination of ISW and AdvStyle leads to even better accuracy in the downstream task (semantic segmentation).\n\n- The technique is conceptually simple and easy to reproduce.\n\n \n##########################################################################\n\nCons: \n \n- The t-SNE visualization (Figure 5) doesn’t seem to convey much information regarding what is happening during the training process. I understand it is typical to visualize features spaces in DA/DG papers, but in this case, it is not clear to me that the figure helps make any point. On the contrary, I would argue it distracts from the main point.\n \n##########################################################################\n\nQuestions during rebuttal period: \n \nI don’t have major concerns with this paper. However, I have a recommendation/thought for the authors.\n- Why focus just on driving scenes? It should be relatively straightforward to evaluate this approach in other types of scenes, for instance, indoor scenes, by leveraging NYU dataset + SceneNet or InteriorNet. This will help make an even stronger point.\n \n#########################################################################\nMiscellanea: \n\n- Figure 1 is cluttered and hard to understand. I would recommend removing b) and d).\n\n- Figure 5, as mentioned above, doesn’t seem to help with the narrative.\n", "summary_of_the_paper": " The paper presents an adversarial style augmentation approach (referred to as AdvStyle) with the objective of improving the generalization of models from a synthetic domain (training) to a real domain (testing), for the task of scene segmentation in the context of driving. The approach is based on the idea of adversarial style learning and the training of a model using both original and adversarial images. A strong set of experiments showcase the benefits of the approach with respect to the state of the art.\n", "summary_of_the_review": "My vote is towards accepting this paper. Similar concepts have been applied in the past to improve domain generalization but never reaching the quality achieved by the proposed approach. It is exciting to see how the application of adversarial style learning can boost results even surpassing existing state-of-the-art solutions, especially considering the ease of implementation of this idea. The experiments section is quite strong, leveraging well-known datasets, relatively modern segmentation architectures, and very recent baselines.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dkCX6tN1o-W", "reviewer": "Reviewer_9eF2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The paper is very well-written and easy to follow\n- The proposed method is simple yet effective with a good motivation\n- The proposed augmentation can be integrated into different methods\n- The paper shows good performance on two tasks: semantic segmentation and image classification\n\nWeakness:\n- The explanation of why the proposed augmentation performs well to cover unseen data distributions is not clear. For example, in Fig 5 of the t-SNE plot, GTAV is the only training data, but it can generate out-of-distribution samples (e.g., Adv GTAV), which does not seem very intuitive. Is it also because other data augmentations (color jittering and gaussian blur) are used? The authors should clarify this.\n\nExperimental comparisons\n- In Table 5, although there are different settings, the authors should try efforts to make fair comparisons, e.g., using the whole set of GTAV for training (setting I and II).\n- The setting in the paper only uses one source dataset. It would be interesting to report results of using multiple source datasets (like the ISW paper using both GTAV and Synthia).\n\nMinor issue\n- The ERM baseline in Sec 4.5 should have the reference, instead of having it in the appendix.\n\n##########################################################################\n\nPost-rebuttal:\nThe rebuttal well addresses my main concerns. I would encourage the authors to release the code and models if the paper is accepted.\n##########################################################################", "summary_of_the_paper": "The paper introduces a domain generalization method for semantic segmentation, where one synthetic data is given as the source training dataset and the model is tested on unseen real datasets. The authors propose a simple yet effective method by introducing adversarial style augmentations. Specifically, the model decomposes the image into a style feature and a normalized image, and then uses the segmentation loss to update the style feature. With the updated style feature and the normalized image, it can resemble an augmented image, which is then used to optimize the segmentation model along with the original image via the segmentation losses. A few highlights of the proposed method are:\n- It appears to be a plug-in module, so that it can be used for different methods. architectures, and tasks\n- It shows state-of-the-art performance on the semantic segmentation task", "summary_of_the_review": "Although the proposed augmentation method is simple. it shows state-of-the-art performance for domain generalizable semantic segmentation. The proposed method can be also adopted in different methods, backbones, and tasks, which is beneficial to the field. However, the authors should address the raised concerns regarding the technical clarity and experimental comparisons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a domain generalization method for semantic segmentation. The model is trained on synthetic data (source) and is tested on unseen real datasets (target). The authors propose a simple data augmentation method, AdvStyle, generating unconstrained adversarial examples for the training on the source domain.

There was no consensus on the method among the reviewers. Several issues have been raised. After rebuttal and discussion, no one really changed her/his mind. The motivation of why focus just on driving scenes is still questionable. Definitively, it could be interesting to investigate further why it is not straightforward to have gains on other kinds of scenes. Finally, we encourage the authors to address the raised concerns regarding the discussion with previous works and the comparisons for future publication.
Decision: Reject