Domain Generalization with MixStyle
Authors: Kaiyang Zhou
Yongxin Yang
Yu Qiao
Tao Xiang
Venue: ICLR 2021 Poster (2021)
Keywords: Domain Generalization
Style Mixing

Abstract: Though convolutional neural networks (CNNs) have demonstrated remarkable ability in learning discriminative features, they often generalize poorly to unseen domains. Domain generalization aims to address this problem by learning from a set of source domains a model that is generalizable to any unseen domain. In this paper, a novel approach is proposed based on probabilistically mixing instance-level feature statistics of training samples across source domains. Our method, termed MixStyle, is motivated by the observation that visual domain is closely related to image style (e.g., photo vs.~sketch images). Such style information is captured by the bottom layers of a CNN where our proposed style-mixing takes place. Mixing styles of training instances results in novel domains being synthesized implicitly, which increase the domain diversity of the source domains, and hence the generalizability of the trained model. MixStyle fits into mini-batch training perfectly and is extremely easy to implement. The effectiveness of MixStyle is demonstrated on a wide range of tasks including category classification, instance retrieval and reinforcement learning.

Published as a conference paper at ICLR 2021
DOMAIN GENERALIZATION WITH MIXSTYLE
Kaiyang Zhou', Yongxin Yang', Yu Qiao², Tao Xiang'
'University of Surrey, UK
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China k. zhoo. wwwou.visions. vision@gmail
com
(yongxin. yang, (riiaggesurreyy ac.uk yu. ww.qiaoBsiat.ae ac.cn
ABSTRACT
Though convolutional neural networks (CNNs) have demonstrated remarkable ability in learning discriminative features, they often generalize poorly unseen domains. Domain generalization aims to address this problem by learning from a set of source domains model that is generalizable to any unseen domain. In this paper, novel approach is proposed based on probabilistically mixing instance- level feature statisties of training samples across source domains. Our method, termed MixStyle, is motivated by the observation that visual domain is closely related to image style (e.g., photo vs. sketch images). Such style information is captured by the bottom layers of CNN where our proposed style-mixing takes place. Mixing styles of training instances results in novel domains being synthe- sized implicitly, which increase the domain diversity of the source domains, and hence the generalizability of the trained model. MixStyle fits into mini- train- ing perfectly and is extremely easy to implement. The effectiveness of MixStyle is demonstrated on a wide range of tasks including category classification, instance retrieval and reinforcement learning.
INTRODUCTION
Key to automated understanding of digital images is to compute a compact and informative feature representation. Deep convolutional neural networks (CNNs) have demonstrated remarkable ability representation learning, proven to effective many visual recognition tasks, such as classifying photo images into 1,000 categories from ImageNet (Krizhevsky et al. 2012) and playing Atari games with reinforcement learning (Mnih et al. 2013). However, has long been discovered that the success of CNNs heavily relies the i.id. assumption, i.e. training and test data should be drawn from the same distribution; when such an assumption is violated even just slightly, as in most real- world application scenarios, severe performance degradation is expected (Hendrycks & Dietterich] 2019
Recht et al. 2019).
Domain generalization (DG) aims address such a problem (Zhou al. 2021 Blanchard al.
@017|Maande et al. 2013] Li et al. 2018a| (Zhou et al. 2020b Balaji et al. 2018 Dou et af.] 2019 Carlucci al., 2019). In particular, assuming that multiple source domains containing the same visual classes are available for model training, the goal of DG is to learn models that are robust against data distribution changes across domains, known as domain shift, so that the trained model can generalize well to any unseen domains. Compared to the closely related and more widely stud- ied domain adaptation (DA) problem, DG is much harder in that no target domain data is available for the model to analyze the distribution shift in order to overcome the negative effects. Instead, DG model must rely on the source domains and focus on learning domain-invarian feature repre- sentation in the hope that it would remain discriminative given target domain data.
straightforward solution to DG is to expose a model with a large variety source domains. Specif- ically, the task of learning innain-invariant and thus generalizable feature representation becomes easier when data from more diverse source domains are available for the model. This would reduce the burden designing special models learning algorithms for DG. Indeed, model training with large-scale data of diverse domains is behind the success of existing commercial face recognition or vision-based autonomous driving systems. A recent work by Xu et al. (2021) also emphasizes the

---

Published as a conference paper at ICLR 2021
Visualization of style statistics
Cartoon
Art Painting
art painting
photo
Photo
Figure 1: 2-D t-SNE (Maaten & Hinton 2008
visualization of the style statistics (concatenation of mean and standard deviation) computed from the first residual block's s feature maps of a ResNet- (He al. 2016) trained on four distinct domains (Li et al. 2017). It is clear that different domains are well separated.
importance of diverse training distributions for out-of-distributinn generalization. However, collect- ing data of a large variety of domains is often costly or even impossible. It thus cannot be general solution to DG.
In this paper. a novel approach is proposed based on probabilistically mixing instance-level feature statistics of training samples across source domains. Our model, termed MixStyle, is motivated by the observation that visual domain is closely related to image style. An example is shown in Fig. [ the four images from four different domains depict the same semantic concept, i.e. dog, but with distinctive styles (e.g., characteristies in color and texture). When these images are fed into deep CNN, which maps the raw pixel values into category labels, such style information removed at the output. However, recent style transfer studies (Huang & Belongie 2017] Dumoulin et al. 2017) suggest that such style information is preserved at the bottom layers of the CNN through the instance-level feature statistics, shown clearly in Fig. 4 Importantly, since replacing such statistics would lead to replaced style while preserving the semantic content of the image, it is reasonable assume that mixing styles from images of different domains would result in images of (mixed) new styles. That is, more diverse domains/styles can be made available for training a more comain-generalizable model.
Concretely, our MixStyle randomly selects two instances of different domains and adopts prob- abilistic convex combination between instance-level feature statistics of bottom CNN layers. In contrast to style transfer work (Huang & Belongie 2017 Dumoulin et al.) 2017). no explicit image synthesis is necessary meaning much simpler model design. Moreover, MixStyle perfectly fits into modern mini-batch training. Overall, it is very easy to implement with only few lines of code. To evaluate the effectiveness as well as the general applicability of MixStyle, we conduct extensive experiments wide spectrum of datasets covering category classification (Sec.3.) instance trieval (Sec. 3.2). and reinforcement learning (Sec. 3.3). The results demonstrate that MixStyle can significantly improve CNNs' cross-domain generalization performance
METHODOLOGG
2.1 BACKGROUND
Normalizing feature tensors with instance-specific mean and standard deviation has been found effective for removing image style in style transfer models (Ulyanov al., 2016 Huang & Be- longie 2017 Dumoulin et al. 2017). Such an operation is widely known as instance normalization (IN, Ulyanov et al. (0166)) Let x l BxxxxxX be batch of tensors, with B, C, and W
Source code be found htttp:://giie com/ Kai yang/hhu/mm wwang/hhu/mixstylee... ixstyxstteeeee ease.
2

---

Published as a conference paper at ICLR 2021
denoting the dimension of batch, channel, height and width, respectively, IN is formulated
- x' m(x))
(N
(x) 3,
where 7, IR learnable affine transformation parameters, and xxz)), R _BxC are mean and standard deviation computed across the spatial dimension within each channel of each instance (tensor), i.e.
W
1
p(x)b,c
Jbe,,h,w
HW
h= w=
and
H
W
(x))..
Σ
Σ (Ib,e.ha (x(2)),,)2
(3)
HW
4=1 w=
Huang & Belongie (0117) introduced adaptive instance normalization (AdalN), which simply re- with the feature statistics of style input y achieve
places the scale and shift parameters in Eq. arbitrary style transfer:
(daIN(x)) o(v) m(x)) o
ply)-
2.2 MIXSTYLE
Our method, MixStyle, draws inspiration from AdaIN. However, rather than attaching a decoder for image generation, MixStyle is designed for the purpose of regularizing CNN training by perturbing the style information of source domain training instances. It can be implemented as a plug-and- play module inserted between CNN layers of, c.g., a supervised CNN classifier, without the need to explicitly generate image of new style.
More specifically, MixStyle mixes the feature statistics two instances with random convex weight to simulate new styles. In terms of implementation, MixStyle can be easily integrated into mini-batch training. Given input batch æ, MixStyle first generates a reference batch 7 from s. When domain labels are given, x' is sampled from two different domains and j. e.g., = [x', _r (x' and xd have the same batch size). Then, ă obtained by swapping the position of x and x², followed by shuffling operation along the batch dimension applied to each batch, i.e.
J
(a) Shuffling batch w/ domain label
I
See Fig. [a) for illustration. In cases where domain labels are unknown, randomly sampled from the training data, and sim- ply obtained by ã Shuffle(z) (see Fig. 2()).. Fig. shows that sub-domains exist within each domain, so even if two in- stances of the same domain are sampled, new domain could be synthesized. After shuffling, MixStyle computes the mixed feature statistics by
(Shuffle(2"), "haufff(="),Suuuff((((()))
(b) Shuffling batch w/ random shuffle
Figure 2: A graphical illustration of how reference batch is gener- ated. Domain label is denoted by color.
Tmxi )o +1 (1- ()) Brii = Ax(x))+++ -1 A)m(G),
(5)
(6)
where À E " are instance-wise weights sampled from the Beta distribution, À ~ Beta(a, a)) with E (0,oc) being a hyper- pyreee-arameter. Unless specified otherwise, we set a to 0.1 throughout this paper. Finally, the mixed feature statistics are applied to the style-normalized x,
I ((()) + Bmii.
MixStyle(x) = "/mir o
In practice, we use a probability of 0.5 to decide if MixStyle is activated or not in the forward pass. At test time, MixStyle is applied. Note that gradients blocked in the computational graph of p(-) and (-). MixStyle can be implemented with only few lines of code. See Algorithm in Appendix A.T for the PyTorch-like pseudo-code.
3

---

Published as a conference paper at ICLR 2021
Table Leave-one-domain-owe generalization results on PACS.
Method
Art
Cartoon
Photo
Sketch
Avg
MMD-AAE
75.2
72.7
96.0
64.2
77.0
CCSA
80.5
76.9
93.6
66.8
79.4
JiGen
79.4
75.3
96.0
71.6
80.5
CrossGrad
79.8
76.8
96.0
70.2
80.7
Epi-FCR
82.1
77.0
93.9
73.0
81.5
Metareg
83.7
77.2
95.5
70.3
81.7
L2A-OT
83.3
78.2
96.2
73.6
82.8
ResNet-18
77.0++0.6
7.69
96.0+1..1
69.24.066
79.5
+ Manifold Mixup
75.6+
70.1+0.9
93.5+0.7
65.4+0..
76.2
+ Cutout
74.9=0.4
74.911.6
95,91103
67.710.9
78.3
+ CutMix
74.610.7
71.810.6
95.610.4
65.340.8
76.8
+ Mixup (wo label interpolation)
7..7=11.0
72.310.9
93.0+10.4
69.2+0.2
77.3
+ Mixup
76.8+00.7
74.9+0.7
95.8=0.3
66.6+0.7
78.5
+ DropBlock
76.4+007
75.4=0.7
95.911.33
69.0+0.3
79.2
MixStyle w/ random shuffle
82.310.2
79.0
96.3.O.33
73.8+0.9
82.8
+ MixStyle w/ domain label
84.1
78.8
96.110.3
75.9
83.7
EXPERIMENTS
3.1
GENERALIZATION CATEGORY CLASSIFICATION
Dataset and implementation details. We choose the PACS dataset (Li et al. used domain generalization (DG) benchmark concerned with domain shift in image classification. PACS consists of four domains, i.c. Art Painting, Cartoon, Photo and Sketch, with totally 9,991 im- ages of classes. As shown in Fig. D the domain shift mainly corresponds to image style changes. For evaluation, model is trained
2017), commonly
three domains and tested on the remaining one. Following prior work (Li et al.) 2019) Zhou et al.) 2020a), we use ResNet-18 (He et al.) 2016) as the classi- fier where MixStyle is inserted after the 1st, 2nd and 3rd residual blocks. Our code based on Dysss.pytorch (Zhou et al. 2020c
Baselines. Our main baselines are general-purpose regularization methods including Mixup (Zhang al. 2018b), Manifold Mixup (Verma et al. 2019),, DropBlock (Ghiasi al. (018)) CutMix (Yun et al. 2019 and Cutout (DeVries & Taylor (017). which are trained using the same training parame- as MixStyle and the optimal hyper-parameter setup as reported in their papers. We also compare with the existing DG methods which reported state-of- of-the-an art performance on PACS. These include domain alignment-based CCSA (Motiian al.) (017)) and MMD-AAE (Li al., 2018b). Jigsaw puzzle-based JiGen (Carlucci et al. 2019), adversarial gradient- based CrossGrad (Shankar et al.) 2018), meta-learning-basee Metareg (Balaji et al. 2018) and Epi-FCR (Li al.) 2019), and data nugmentation-basee L2A-OT (Zhou et al. 2020a).
Comparison with general-purpose regularization methods. The results are shown in Table Overall, we observe that the general-purpose regularization methods do not offer any clear advan- tage over the vanilla ResNet-18 in this DG task, while MixStyle improves upon the vanilla ResNet- 18 with a significant margin. Compared with Mixup. MixStyle 5.2% better on average. Recall that Mixup also interpolates the output space, further compare with variant of Mixup in order to demonstrate the advantage of mixing style statistics at the feature level over mixing images at the pixel level for DG- following [Sohn al. 2020), remove the label interpolation in Mixup and sample the mixing weights from a uniform distribution of [o, 1]. Still, MixStyle outperforms this new baseline with large margin, which justifies claim. MixStyle and DropBlock share some commonalities, in that they are both applied to feature maps at multiple layers, but MixStyle signifi- cantly outperforms DropBlock in all domains. The reason why DropBlock ineffective here is because dropping out activations mainly encourages network to mine discriminative patterns, but does not reinforce the ability to cope with unseen styles, which is exactly what MixStyle aims to achieve: by synthesizing "new" styles (domains) MixStyle regularizes the network to become more
Phttps://git thub. com/ ccom///iyyagg/huu/aaassss pytoorch.

---

Published as a conference paper at ICLR 2021
robust to domain shift. In addition, it is interesting to see that Cartoon and Photo, MixStyle w/ random shuffle obtains slightly better results. The reason might be because there exist sub-domains in source domain (see Fig. (a-))) which allow random shuffling to produce more diverse "new" domains that lead to a more domain-generrlizable model.
Comparison with state-of-the-art DG methods. Overall, MixStyle outperforms most DG meth- ods by clear margin, despite being a much simpler method. The performance of MixStyle w/ domain label is nearly 1% better on average than the recently introduced I.2A-OT. From a data augmentation perspective, MixStyle and 1.2A-OT share a similar goal-to synthesize data from pseudo-novel domains. MixStyle accomplishes this goal through mixing style statistics the fea- ture level. Whereas L2A-OT works at the pixel level: it trains an image generator by maximizing the domain difference (measured by optimal transport) between the original and the generated images, which introduces much heavier computational overhead than MixStyle in terms of GPU memory and training time. It worth noting that MixStyle's ' domain label-free version is highly competitive: its 82.8% accuracy is on par with L2A-OT's.
3.2 GENERALIZATION IN INSTANCE RETRIEVAL
Dataset and implementation details. We evaluate MixStyle on the person re-identification (re-ID) problem, which aims to match people across disjoint camera views. As each camera view is itself a distinct domain, person re-ID) is essentially a cross-domain image matching problem. Instead of using the standard protocol where training and test data come from the same camera views, we adopt the cross-dataset setting so test camera views are never seen during training. Specifically, we train a model on one dataset and then test its performance on the other dataset. Two commonly used re-II datasets are adopted: Market 101 (Zheng et al., 2015) and Duke (Ristani et al., 2016 [Zhngg et al.) 2017 Ranking accuracy and mean average precision (mAP) are used as the performance measures (displayed in percentage). We test MixStyle on two CNN architectures: ResNet-50 (He et al. 2016) and OSNet (Zhou et al. [209).. The latter was designed specifically for re-ID. both architectures. MixStyle is inserted after the 1st and 2nd residual blocks. Our code is based Torchreid (Zhou &
Xiang 2019
Baselines. We compare with three baseline methods: 1) The vanilla model, which serves as a strong baseline; 2) DropBlock, which was the top-performing competitor in Table I 3) Ran- domErase (Zhong et al. 2020), widely used regularization method in the re-ID literature (similar to Cutout).
Results. The results are reported Table It clear that only MixStyle consistently outperforms the strong vanilla model under both settings with considerable margins, while DropBlock and Ran- domErase are unable to show any benefit. Notably, RandomErase, which simulates occlusion by erasing pixels in random rectangular regions with random values, has been used as default trick when training re-ID CNNs. However, RandomErase shows a detrimental effect in the cross-dataset re-IID setting. Indeed, similar to DropBlock, randomly erasing pixels offers no guarantee to improve the robustness when it comes to domain shift.
3.3 GENERALIZATION IN REINFORCEMENT LEARNING
Though RL has been greatly advanced by using CNNs for feature learning in raw pixels (Mnih et al., 2013), it has been widely acknowledged that RL agents often overfit training environments while generalize poorly to unseen environments (Cobbe et al. 2019 el al. 2019).
Dataset and implementation details. We conduct experiments on Coinrun (Cobbe et al.) 2019), a recently introduced RL benchmark for evaluating the generalization performance of RL agents. As shown in Fig. 3[a, the goal in Coinrun is to control a character to collect golden coins while avoiding both stationary and non-stationary obstacles. We follow Igl et al. (2019) to construct and train our RL agent: the CNN architecture used in IMPALA (Espeholt et al., 2018) is adopted as the policy network, and is trained by the Proximal Policy Optimization (PPO) algorithm (Schulman al. 2017). Please refer to [gl et al. (2019) for further implementation details. MixStyle is inserted after the 1st and 2nd convolutional sequences. Training data are sampled from 500 levels while test
'tttps:/ thub. com/ Ka wwt.///o./aaaaaaaeeee wwwayyagg///u/eeee
reid.
5

---

Published as a conference paper at ICLR 2021
Table 2: Generalization results on the cross-dataset person re-ID task.
Market1501
Duke
Duke
Market
Model
MAP
R1
R5
R10
mAP
R1
R5
R10
ResNet-50
19.3
35,4
50.3
56.4
20.4
45.2
63.6
70.9
+ RandomErase
14.3
27.8
42.6
49.1
16.1
38.5
56.8
64.5
DropBlock
18.2
33.2
49.1
56.3
19.7
45.3
62.1
69.1
+ MixStyle w/ random shuffle
23.8
42.2
58.8
64.8
24.1
51.5
69.4
76.2
MixStyle w/ domain label
23.4
43.3
58.9
64.7
24.7
53.0
70.9
77.8
OSNet
25.9
44.7
59.6
65.4
24.0
52.2
67.5
74.7
+ RandomErase
20.5
36.2
52.3
59.3
22.4
49.1
66.1
73.0
+ DropBlock
23.1
41.5
56.5
62.5
21.7
48.2
65.4
71.3
+ MixStyle w/ random shuffle
27.2
48.2
62.7
68.4
27.8
58.1
74.0
81.0
+ MixStyle w/ domain label
27.3
47.5
62.0
67.1
29.0
58.2
74.9
80.9
Seen
Baselie Baselin Modityee
35-
Baseline Basaline- MiXtyll
IBAC-SMI IBAC-CNI Moditye
IBAL-SNI
3.0-
IBAC SM Mixttyle
Unseen
6.-
5.0
15k
Prames
(a)
(b)
(c)
Figure 3: (a) Coinrun benchmark. (b) Test performance unseen environments. (c) Difference between training and performance.
data are drawn from new levels of only the highest difficulty. As domain labels are difficult to define, use the random shuffle version of MixStyle. Our code is built top of [gl et al. 2019
Baselines. Following [gl et al. (2019), we train strong baseline models and add MixStyle on top of them to see whether MixStyle can bring further improvements. To this end, we train two baseline models: 1) Baseline, which combines weight decay and data augmentation!51 [51 2) IBAC-SNI (the 0.5 version), the best-performing model [gl al. (2099) which is based on selective noise injection.
Results. The test performance is shown in Fig. 3(b). line+MixStyle (orange), we can see that MixStyle brings significant improvement. Interestingly, the variance is also significantly reduced by using MixStyle, as indicated by the smaller shaded areas (for both orange and red lines). These results strongly demonstrate the effectiveness of MixStyle in enhancing generalization for RL agents. When it comes to the stronger baseline IBAC-SNI (green), MixStyle (red) is able to bring further performance gain, suggesting that MixStyle is complemen- tary to IBAC-SNI. This result also shows the potential of MixStyle to be combined with other advanced RL methods. It is worth noting that Buseline+MixStyle itself is already highly competitive with IBAC-SNI. Fig. B(c) shows the generalization gap from which can been seen that the models trained with MixStyle (orange & red) clearly generalize faster and better than those without using MixStyle (blue & green).
Comparing Baseline (blue) with Base-
plug-and-play component

---

Published as a conference paper at ICLR 2021
res1
res2
ress
res4
(a)
(b)
(c)
(d)
Figure 4: 2-D visualization of flattened feature maps (top) and the corresponding style statistics (bottom). resl- -4 denote the four residual blocks in order in ResNet architecture. We observe that resl to res3 contain domain-related information while res 4 encodes label-related information.
Table 3: Ablation study on where apply MixStyle in the ResNet architecture.
(a) Category classification on PACS.
(b) Cross-datasct person re-ID.
Model
Accuracy
Model
mAP
18
79.5
ResNet-50
19.3
MixStyle (res1)
80.1
MixStyle (res1)
22.6
MixStyle (res12)
81.6
MixStyle (res12)
23.8
+ MixStyle (res123)
82.8
+ MixStyle (res123)
22.0
MixStyle (res1234)
75.6
MixXtyle (res1234)
10.2
MixStyle (res14)
76.3
+ MixStyle (res14)
11.1
MixStyle (res23)
81.7
MixStyle (res23)
20.6
3.4 ANALYSIS
Where to apply MixStyle? We repeat the experiments on PACS (category classification) and the re-ID datasets (instance retrieval) using the ResNet architecture. Given that standard ResNet model has four residual blocks denoted by res1-4, we train different models with MixStyle applied to different layers. For notation, resl means MixStyle is applied after the first residual block; res12 means MixStyle is applied after both the first and second residual blocks; and so forth. The results are shown Table We have the following observations. 1) Applying MixStyle multiple lower- level layers generally achieves better performance- for instance, I resl2 2 is better than resl on achieves the best performance
both tasks. 2) Different tasks favor different combinations--res on PACS, while on the re-ID) datasets res 2 is the best. 3) On both tasks, the performance plunges when applying MixStyle to the last residual block. This makes sense because res4 the closest to the prediction layer and tends to capture semantic content (i.e. label-sensitive) information rather than style. In particular, res4 4 is followed by an average-pooling layer, which essentially forwards the mean vector to the prediction layer and thus forees the mean vector to capture label-related information. As a consequence, mixing the statistics at res4 4 breaks the inherent label space. This is clearer in Fig. 4 the features and style statistics in resl-3 exhibit clustering patterns based on domains while those in res4 have a high correlation with class labels.
*https: / /i thub. cwm/micccccoooom http:iicoooooottttcoccee I
'ee do not use hatch normalization (Toffe & Szegedy 2015 dropout Srivastava et al. 2014) because they are detrimental the performance, as shown by [gl (019

---

Published as a conference paper at ICLR 2021
PAC5
Re-D
Coinrun
47.
Boochne Madipn 0.31
Roceliee Masiyle In 02)
Bnnalite Masoyle Irm 6.)
12AC-M Mictoyle la 6.3
1SAC-5M Macdiyl ja 62)
15Ac SM Meddure 0.5
4002200
Rook: prozt
0.1
82
05
1.4
05
6.-
Fromes
(a)
(b)
(c)
Figure 5: Evaluation the hyper-paramenn on (a) PACS, (b) person re-ID datasets and (c)
Coinrun. In (b), M and D denote M5r and Duke respectively.
Table 4: Mixing vs. replacing.
Mixing vs. replacing. Unlike the AdaIN formulation, which com- pletely replaces one style with the other, MixStyle mixes two styles via a convex combination. Table shows that mixing is better than replacing. This is easy to understand: mixing diversifies the styles (imagine an interpolation between two data points).
Accuracy (%)
Mixing Replacing
02.
82.1
Random vs. fixed shuffle at multiple layers. Applying MixStyle multiple layers, which has been shown advantageous Table 3 raises another question of whether to shuffle the mini-bac batch at dif- ferent layers or use the same shuffled order for all layers. Table [5] suggests that using random shuffle at different layers gives a better performance, which may attributed to the increased noise level that gives better regularization effect.
Table 5: Random vs. fixed
shuffle at multiple layers.
Accuracy (%)
Random Fixed
82.8+0.4
82.4+0.55
Sensitivity of hyper-parameter which has a direct effect on how the convex weights À are sampled. The smaller a is, the more likely the value in À is close to the extreme value of 0 or 1. In other words, a smaller favors the style statistics in Eqs. (5) & (6) to be dominated by one side. We first evaluate a on PACS. Fig. aa shows that with a increasing from 0.1 to 0.4, the accuracy slides from 82.8% to 81.7%. However, further increasing does not impact on the accuracy. Therefore, the results suggest that the performance is not too sensitive to og and selecting a from {0.11 0.2, 0.3) seems be good starting point. We further experiment with E {0.1, 0.2, 0.3} on the re-ID ID datasets and the Coinrun benchmark. Figs. bT & (c) show that in general the variance for the results of different values is small. Therefore, we suggest practitioners to choose from {0.1, 0.33. with 0.1 being good default setting.
Recall that is used control the shape of Beta distribution,
For more analyses and discussions, please see Appendix A.2
4 RELATED WORK
Domain generalization, or DG, studies ofttt--dist (OOD) generalization given only source data typically composed of multiple related but distinct domains. We refer readers to Zhou et al.] 2021 for a comprehensive survey in this topic. Many DG methods are based the idea of aligning teatures between different sources, with a hope that the model can be invariant to domain shift given unseen data. For instance, [Li et al. (2018b) achieved distribution alignment in the hidden represen- tation of an autoencoder using maximum mean discrepancy; Li et al. (2018c) resorted to ad adversarial learning with auxiliary domain classifiers to learn features that are domain-agnostic. Some works explored -peniin- parameterization, such as domain-specific weight matrices (Li al. 2017) and domain-specific BN (Seo et al. 2020). Recently, meta-learning has drawn increasing attention from the DG community (Li et al. 2018a Balaji et al.) 2018 Dou et al., 2019). The main idea is to expose a model to domain shift during training by using pseudo-tran and pseudo-test do- mains, both drawn from source domains. Data augmentation has also been investigated for learning domain- innariannvvaiantt models. Shankar et al, (2018) introduced a cross- gooddigr training method (Cross- Grad) where source data are augmented by adversarial gradients obtained from domain classifier.
8

---

Published as a conference paper at ICLR 2021
Gong al. (0199) proposed DLOW (for the DA problem), which models intermediate domains be- tween source and target via a domainness factor and learns an image translation model to generate coermediate-domaii images. Very recently, Zhou Zhou al. (020aa) introduced L2A-OT to learn a ral network to map source data to pseudo-novel domains by maximizing an optimal transport-based distance measure. Our MixStyle is related to DLOW and L.2A-TT in its efforts to synthesizing novel domains. However, MixStyle differs in the fact that it is done implicitly with a much simpler formulation leveraging the feature-level style statistics and only few lines of extra code on top of standard supervised classifier while being more effective. Essentially, MixStyle can be seen as fea- ture-level augmentation, which is clearly different from the image-level augmentation-ben based DLOW and L2A-OT.
Generalization in deep RL has been a challenging problem where RL agents often overfit training environments, and result, perform poorly in unseen environments with different visual patters or levels (Zhang et al. 2018a). A natural way to improve generalization, which has been shown effective in (Cobbe et al. 2019 Farebrother et al. 2018), is to regularization, e.g., weight decay. However, Igl et al. (2019) suggested that stochastic regularization methods like dropout and batch normalization (which uses estimated population statistics) have adverse effect as the training data in RL are essentially model proeel-dependent. As such, they proposed selective noise injection (SNI), which basically combines stochastic regularization technique with its deterministic counterpart. They further integrated SNT with information bottleneck actor critic (IBAC-SNI) to reduce the variance in gradients. Curriculum learning has been investigated in (Justesen et al. 2018) where the level of training episodes progresses from easy to difficult over the course of training. Gamrian & Goldberg (219) leveraged the advances in GAN-based image-to-image translation (Liu et al. (207)) to map target data the source domain which the agent was trained on. Tobin et al. (2017) introduced domain randomization, which diversifies training data by rendering images with different visual effects via programmable simulator. With similar goal of data augmentation, [ee al. (2020) pre-processed input images with a randomly initialized network. Very recent studies (Laskin et al., 2020 Kostrikov et al. 2020) have shown that it useful to combine diverse set of label-preserving transformations, such as rotation, shifting and Cutout. Different from the aforementioned methods, our MixStyle works the feature level and orthogonal to most existing methods. For instance, we have shown in Sec. 3.3 that MixStyle significantly improves upon IBAC-SNI.
5 CONCLUSION
We presented a simple yet effective domain generalization method, termed MixStyle. MixStyle mixes the feature statistics of two instances to synthesize novel domains, which is inspired by the observation in style transfer work that the feature statistics encode style/domain-relatte information. Extensive experiments covering a wide range of tasks were conducted to demonstrate that MixStyle yields new state-of-the-an on three different tasks.
ACKNOWLEDGMENTS
This work was supported in part by the Shanghai Committee of Science and Technology, China (Grant No. 20DZI 1008000,, in part by the National Natural Science Foundation of China under Grant (61876176, U1713208), the National Key Research and Development Program of China (No. 2020YFC2004800), and in part by the Guangzhou Research Program (201803010066).
REFERENCES
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regularrizatonn. In NeurIPS, 2018.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurIPS, 2011.
Fabio M. Carlucci, Antonio D' I'Inoceetee Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In CVPR, 2019.
Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In ICMI, 2019.

---

Published as a conference paper at ICLR 2021
Terrance VeViies and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 2017.
Qi Dou, Daniel C Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NeurIPS, 2019.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. In ICLR, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In ICMI, 2018.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-image translation. In ICML, 2019.
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolu- tional networks. In NeurIPS, 2018.
Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. Dlow: Domain flow for adaptation and gener- alization. In CVPR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In ICLR, 2019.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization reinforcement learning with selective noise injection and information bottleneck. In NeurIPS, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint 1806 10729, 2018
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. /3649, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. arXiv preprint wwXiv:2004.1499 2020.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applied to document recognition. In IEEE, 1998.
Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. Network randomization: A simple technique for generalization deep reinforcement learning. In ICLR, 2020.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
10

---

Published as a conference paper at ICLR 2021
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI, 2018a.
Li Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, 2019.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In CVPR, 2018b.
Ya Li, Xinmei Tiana, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018c.
Ming-Y Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In NeurIPS, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13 2013.
Saeid Motiann, Marco Piccirilli, Donald A. Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. In ICCV, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Scholkopf. Domain generalization via invariant feature representation. In JCML, 2013.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NeurlPS-W. W, 2011.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In JCML, 2019.
Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance mea- sures and data set for multi-target, multi-camera tracking. In ECCV, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Seonguk Seo, Yumin Suh, Dongwan Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In ECCV, 2020.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradien training. In ICLR, 2018.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 2014.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IROS, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. 1007 2016.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
11

---

Published as a conference paper at ICLR 2021
Keyulu Xu, Mozhi Zhang, Jingling Li, Simon Shaolei Du, Ken-Ichi Kawarabayashi, and Stefanie Jegelka. How neural networks extrapolate: From feedforward to graph neural networks. ICLR, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. arXiv preprint
2018a.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empiri- cal risk minimization. In ICLR, 2018b.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, 2015.
Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In ICCV, 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In AAAI, 2020.
Kaiyang Zhou and Tao Xiang. Torchreid: A library for deep learning person re-identification in pytorch. arXiv preprint arXiv: 10093, 2019.
Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. Omni-scale feature learning for person re-identification. In ICCV, 2019.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, 2020a.
Kaiyang Zhou, Yongxin Yang, Timothy M Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In AAAI, 2020b.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. arXiv preprint arXiv:2003.07325, 2020c.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, arXiv preprint arXiv:2103.02503, 2021.
12

---

Published as a conference paper at ICLR 2021
A
APPENDIX
A.1 PSEUDO-CODE OF MIXSTYLE
Algorithm
I provides PyToch-likke pseudo-code.
Algorithm PyTorch-like pseudo-code for MixStyle.
input features of shape (3, Н, 0)
probabillity apply Mixstyle (default: 0.51
alpha: ppper-parmmeter fer the Beta distribution (defaule: 0.1) ope: small value addad before square root for numerical stability (dofault: 1e-6)
if not training mode:
return x
randon probability p: return x
(05 batch
x.mean (2, 31, keepdim-True) 4 compute instance
x.var (dim- [2, 31, keepdim=True) compute instaace variance sig tear eps) sqrt () # compute instance standard daviation sig metdetach (),, sig.detach.)) 4 block gradients x_nermed
mu) sig normelize imput
Inda setalalpha, alphal aamplee (B, 1, 1, 11) sample instance wise convex weighta
ir domain label is given:
in this case, input x [x x'j]
(8-1, -1, -1) inverse index
pera_), perm_i - perm.chunk(2)
separace indices
pera]] - [am__]ftorca.caddeett (B // 2)] + shuffling
=porr_i [orr_iltoree [ttrcct.aaneprrrmm // $ shuffling
perm carch.cat ((pera)), perm_il,
concatenation
else:
porm torch. ronde..and
gonerate shuffling indicas
mu2, sig2 - mu mu[perm), sig[perm] shuffling
lnda mu2 Imda) + mixed Imda) generate mixed stancard deviation
sig_mix sig
sig2
return K_normed siq_mix mo_nix & decormalize input using the mixea statiatios
13

---

Published as a conference paper at ICLR 2021
Table Test results on the source domains PACS, A: Art. C: Cartoon. P: Photo. Sketch.
Method
C,P.S
A,P.S
A,C,S
A,C.P
Avg
Vanilla
99.49-10.03
99.47=0.04 99.5410.01
99.3810.02 99.47=0.03
99.65=0.03
99.50
MixStyle
99.55=0.02
99.68=0.03
99.56
Table 8: Leave-one-domain-o generalization results on Digits- DG.
Method
MNIST
MNIST-M
SVHN
SYN
Avg
JiGen
96.5
61.4
63.7
74.0
73.9
CCSA
95.2
58.2
65.5
79.1
74.5
MMD-AAE
96.5
58.4
65.0
78.4
74.6
CrossGrad
96.7
61.1
65.3
80.2
75.8
L2A-OT
96.7
63.9
68.6
83.2
78.1
CNN
95.810.3
58,810.5
61.7:0.5
78.6=0.6
73.7
Mixup w/o label interpolation
93.7+00.6
55.2++1.0
61.6+0.9
74.4-0.8
71.2
+ Manifold Mixup
92.7++0.4
53.1+-08
64.4+0.2
76.8-0.5
71.7
+ CutMix
94.911.2
50.1
64.1:0.9
78.1=0.7
71.8
+ Mixup
94.211..5
56.5+0.8
63.310.7
76.7=0.6
72.7
+ Cutout
95.8=10.4
58.410..6
61.910.9
80.6=0.5
74.1
+ DropBlock
96.2#0.1
60.5+0.6
64.120.8
80.2=0.6
75.3
+ MixStyle (ours)
96.5+0.3
63.5+0.8
64.7#0.7
81.2=0.8
76.5
A.2 FURTHER ANALYSIS
MixStyle between same-domain instances. We are interested in knowing mixing styles between same-domain instances helps perfor- mance. To this end, we sample each mini-batch from single domain during training when us- ing MixStyle. The results are shown Table 6 where observe that mixing styles between same-domain instances is about 1% better than the baseline model. This suggests that instance- specific style exists. Nonetheless, the perfor- mance is clearly worse than mixing styles be- tween instances of different domains.
Table 6: Investigation the effect of mixing styles between same-domain instances.
Accuracy (%)
ResNet18
79.5
+ MixStyle w/ same-domain
80.4
MixStyle w/ random shuffle
82.8
MixStyle w/ domain label
83.7
Performance on source domains. To prove that MixStyle does not sacrifice the performance on domains in exchange for gains unseen domains, report the accuracy the held-out validation set of the source domains on PACS in Table
Results on Digits-DG and Office-Home. addition to the experiments on PACS (in Sec. 3.1 we further evaluate MixStyle's effectiveness on two DG datasets, namely Digits-DG (hhou et al. 2020a) and Office-Home /Venkateswara et al. 2017). Digits-DG contains four digit datasets (do- mains) including MNIST (LeCum et al. 1998), MNIST-M (Ganin & &empitsky 2015), SVHN (Net- zer et al. 2011> and SYN (Ganin & Lempitsky 2015). Images from different digit datasets differ drastically in font style, stroke color and background. Office-Home is composed of four domains (Artistic, Clipart, Product and Real World) with around 15,500 images of 65 classes for home and office object recognition. The results are shown in Tables and where domain labels are used in MixStyle. Similar to the results on PACS, here we observe that MixStyle also brings clear im- provements the baseline CNN model and outperforms all general-purpot regularization methods on both Digits-DG DD and Office-Home. Compared with more sophisticated DG methods like L2A -O,, MixStyle's performance is comparable, despite being much simpler to train and consuming much less computing resources.
14

---

Published as a conference paper at ICLR 2021
Table 9: Leave-one-dmmann-oul generalization results on Office-Homes
Method
Artistic
Clipart
Product
Real World
Avg
JiGen
53.0
47.5
71.5
72.8
61.2
CCSA
59.9
49.9
74.1
75.7
64.9
MMD-AAE
56.5
47.3
72.1
74.8
62.7
CrossGrad
58.4
49.4
73.9
75.8
64.4
L2A-OT
60.6
50.1
74.8
77.0
65.6
ResNet18
58.910.3
49.410.1
74.310.1
76.21:.2
64.7
+ Manifold Mixup
56.2+0.44
46.310.3
73.610.1
75.2=0.2
62.8
Mixup w/o label interpolation
57.0+10.2
48.7+0.2
71.4=00.
74.5++0.4
62.9
+ Cutout
57.8+0022
48.1+0.3
73.9-0.2
75.8
63.9
CutMix
57.9+0.1
48.3+0.3
74.5=0.1
75.6:10.4
64.1
DropBlock
58.0-10.1
48.110.11
74.3+00.
75.9
64.1
Mixup
58.2.10.1
49.3:0.2
74.710.1
76.110.1
64.6
MixStyle (ours)
58.740.3
53.4+0.2
74.2=0.1
75.9+0.1
65.5
15


Reviews:
[{"review_id": "g2YJDUhcz-b", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Paper Summary **\n\nThis paper proposed a simple regularization technique for domain generalization tasks, termed MixStyle, based on the observation that domains are determined by image styles. By mixing styles of different instances, which generates synthesized domain samples while preserving the content features, the proposed method achieves the generalizability of the trained model. The MixStyle was applied to numerous applications, such as category classification, instance retrieval, and reinforcement learning, and attained the state-of-the arts. The MixStyle is relatively simple to implement, but effective. \n  \n** Paper Strength **\n+ Simple methodological design, so it is easy to implement.\n+ Understanding the domain shift problems as a style variation makes sense.\n+ Randomizing the styles might be the solution to alleviate the domain generalization problems, but searching all the possible styles and applying them would be challenging and not feasible. So, using different instance samples to extract the styles was nice.\n+ It makes sense that introducing the \\lambda to mix the styles itself and ones of different instances.\n+ The paper is well organized and written.\n\n** Paper Weakness **\n\nI have no major comments on this paper, but minor comments as follows:\n- Even though the authors have shown the ablation study to analyze the levels where the MixStyle should be applied, it is not clear for me yet. The authors applied the MixStyle after 1st, 2nd, and 3rd residual blocks for category classification problems, but applied the MixStyle after 1st and 2nd residual blocks for category classification problems for instance retrieval task. In 3.4 analysis, they only showed the ablation studies on the category classification. Thus, one think the optimal combinations may vary according to the applications. In addition, another combination, e.g., conv34, conv25, would be more interesting.\n- Fig 4 is hard to understand; what do the corresponding style statistics mean? Why does (d) only represent different legends? \n- In Table 1, some experimental settings, e.g., Cartoon or Photo, have shown that MixStyle w/ random shuffle was better? The discussion on this might be interesting. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S94HuhmWRwh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a technique for domain generalization by mixing style of images from different domains. This work adopts a mix up style approach [A] for domain generalization. Different from [A], the paper proposes to conduct mix-up in the intermediate layers, in particular, instance normalization layers. The proposed approach diversifies the data implicitly and the experimental results show that the mix-style can improve domain generalization.\n\nOverall the paper is well-written with plenty of details. I also appreciate the experimental analysis in Sec 3.4 and the variance reported in Table 1. However, I have several concerns regarding the paper:\n- The technical novelty seems rather incremental. This method is an extension of [A] to the instance normalization layer. Similar strategies have been discussed in other works such [B] and [C]. However, these works are not discussed in terms of main similarities/differences.\n- I also found the experimental validation not fully sufficient to grant publication. Currently the validation is only conducted on PACS, the improvement also seems limited. I believe validation on more datasets(such as Digits, Office-Home as used in L2A-OT) can further confirm the effectiveness of the proposed method.\n- I suspect that  interpolating the style parameter might cause performance drop on the domains that have been seen during training. Would it be possible to report performance on the domains that have been seen in the training?\n\n[A] Vikas Verma et al. Manifold Mixup: Better Representations by Interpolating Hidden States. In ICML 2019.\n[B] Rui Gong et al. DLOW: Domain Flow for Adaptation and Generalization. In CVPR 2019.\n[C] Seonguk Seo. Learning to Optimize Domain Specific Normalization for Domain Generalization. In ECCV 2020.\n\n---\nI have read authors' response and other reviews. Some of my concerns are addressed in the response. Especially the added discussion with related work is helpful. Thus I would increase my rating to 6.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "--ls-mdu-U-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** The paper proposes a simple method for domain generalization where multiple source domains are given for a certain task (like image classification) and testing happens on an unseen domain. The authors are inspired by normalization-based style-transfer techniques (Adaptive InstanceNorm) and propose to mix the styles of different source domains to effectively increase diversity of domains during training.\n\n**Pros:**\n\n- Overall, this is a well written paper with a clear idea that is simple but intuitive.\n- The idea is well described, put into context of prior work and empirically validated to improve results over various baselines.\n- It is good to see experiments outside of plain image classification to validate the proposed idea.\n- The analysis where to apply MixStyle is good and makes intuitive sense.\n\n\n**Cons:**\n\n- The relation to MixUp needs to be explained in more details. While related to the proposed MixStyle, MixUp creates a convex combination of both input and output spaces. I can believe that MixUp as a standard data augmentation gives worse results than a vanilla CNN (Table 1) but I would not fully agree with the statement \"... which demonstrates the advantage of mixing style statistics at the feature level over mixing images at the pixel level\" from page 4. MixUp also interpolates the output label space, so the advantage cannot be only attributed the placement of the mixing within the network instead of at the pixel level.\n- As an additional baseline, one could use MixUp with a sampled lambda that is larger than 0.5 in all cases (like in [FixMatch. Sohn et al. NeurIPS'20]) but keeping the label from sample $x$ rather than interpolating with $\\hat{x}$.\n\n- I do not understand why the suffix \"_x\" is added to the analysis in Table 3. Is MixStyle applied after each convolutional layer or after each block in a ResNet architecture? Specifically, for \"conv234_x\", how often is the MixStyle layer added? (3 times or 3 * num_convs_in_block times?)\n\n- For the ReID experiments, I think it should be better highlighted that the cross-dataset setup is the key difference to evaluations in prior work. This somehow gets almost unnoticed because the default setting of ReID is already considered a valid domain generalization task due to the new label space and camera views. This left me a bit confused about how RandomErase can be a widely used data augmentation technique for ReID when it gives worse results in the experiments from Table 2. This became clear to me only after reading the discussion in the last paragraph of Section 3.2.\n\n- I would not make the statement that \"... mixing is CLEARLY better than replacing\" on page 7 (see Table 4) while also stating that \"... with alpha increasing from 0.1 to 0.4, the accuracy SLIGHTLY slides from 82.8% to 81.7%\". That \"slight\" change is larger than the \"clear\" gap before.\n\n\n**Other notes and open questions:**\n- MixUp was used successfully as regularization for semi-supervised learning (SSL) [MixMatch. Berthelot et al. NIPS'19]. Can MixStyle also be used for SSL?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommend acceptance after the rebuttal stage, and the AC found no reason to disagree with them. The proposed method is simple and effective, and the concerns raised about experimental validation and novelty seem well addressed in the rebuttal. 
Decision: Accept (Poster)