Domain Generalization via Independent Regularization from Early-branching Networks
Authors: Liang Chen
Yong Zhang
Yibing Song
Jue Wang
Lingqiao Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: domain generalization
representational learning

Abstract: Learning domain-invariant feature representations is critical for achieving domain generalization, where a model is required to perform well on unseen domains. The critical challenge is that standard training often results in entangled domain-invariant and domain-specific features. To address this issue, we use a dual-branching network to learn two features, one for the domain classification problem and the other for the original target classification problem, and the feature of the latter is required to be independent of the former. While this idea seems straightforward, we show that several factors need to be carefully considered for it to work effectively. In particular, we investigate different branching structures and discover that the common practice of using a shared base feature extractor with two lightweight prediction heads is detrimental to the performance. Instead, a simple early-branching architecture, where the domain classification and target classification branches share the first few blocks while diverging thereafter, leads to better results. Moreover, we also incorporate a random style augmentation scheme as an extension to further unleash the power of the proposed method, which can be seamlessly integrated into the dual-branching network by our loss terms. Such an extension gives rise to an effective domain generalization method. Experimental results show that the proposed method outperforms state-of-the-art domain generalization methods on various benchmark datasets.

Under review as a conference paper at ICLR 2023
DOMAIN GENERALIZATION VIA INDEPENDENT REGU- LARIZATION FROM EARLY-BRANCHING NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning Nomainn-nvvriiane feature representations is critical for achieving domain generalization, where model is required to perform well on unseen domains. The critical challenge is that standard training often results in entangled domain- invariant and domain-specific features (see Figure 2). To address this issue, we use a dual- bra-chhan network learn two features, one for the domain classification problem and the other for the original target classification problem, and the feature of the latter required to independent the former. While this idea seems show that several factors need to be carefully considered for it to work effectively. In particular, we investigate different branching structures and discover that the common practice of using shared base feature extractor with two lightweight prediction heads is detrimental to the performance. Instead, simple early-branching architecture, where the domain classification and target classification branches share the first few blocks while diverging thereafter, leads better results. Moreover, also incorporate random style augmentation scheme as an extension to further unleash the power of the proposed method, which can be seamlessly integrated into the branchban network by our loss terms. Such an extension gives rise to an effective domain generalization method. Experimental results show that the proposed method outperforms state-of-the-art domain generalization methods on various benchmark datasets.
straightforward,
INTRODUCTION
Domain generalization (DG) asks learned models to perform well on unseen domains, which lies its key in learning domain-invariant representations that are robust to domain shift (Ben-David et al., 2006). Standard training often results entangled domain-invarian and domain-specific features, which hinders the model from generalizing to new domains. Existing methods address this issue by introducing various forms of regularization, such as adopting alignment (Muandet et al., 2013; Ghifary et al., 2016; Li al., 2018b; 8b; Hu et al., 2020), using Somain-adversaria training (Ganin et al., 2016; Li et al., 2018b; Yang et al., 2021; Li et al., 2018c), or developing meta-learning methods (Li al., 2018a; Balaji et al., 2018; Dou al., 2019; Li et al., 2019). Despite the success of these arts, DG remains challenging and is far from being solved. For example, as a recent study (Gulrajani & Lopez-Paz, 2021) suggests, under a rigorous evaluation protocol, it turns out that the naive empirical risk minimization (ERM) method (Vapnik, 1999), which aggregates training data from all domains and trains them in an end-to-end manner without additional efforts, can perform competitively against more elaborate alternatives. This observation indicates that more effective approach might be needed disentangle the Hamminn-invarian and domain speciffc features for better DG.
In this paper, we adopt a simple method by leveraging a conventional dual-branching network with one branch predicting image classes (target prediction) and another predicting domain labels. Regarding the features from the target and domain branches as domain-invariant and domain-specific representations, respectively, entanglement will result in an undesired situation where the domain- specific information is also encoded in the target branch, which will inevitably corrupt the prediction when the domain varies during inference. Thus, to explicitly disentangle the domain-ing innaaiiannn and domain-specific features, latter. This idea seems straightforward, but we show that several factors need to be carefully considered for it to work effectively.
impose regularization to require the former to be independent of the

---

Under review as a conference paper at ICLR 2023
Particularly, we first investigate the structure of the dual-branching network and somehow surprisingly discover that the common practice of using a shared base feature extractor with two lightweight prediction heads (Chen et al., 2021; Atzmon et al., 2020) is detrimental to the performance. Instead, a simple early-branching architecture, where the domain classification branch and target classification branch share the first few blocks while diverging thereafter, yields the optimal results. Incorporating this discovery, we propose the basic form of the proposed method. Specifically, we employ Hilbert- Schmidt Information Criterion (HSIC) (Gretton et al., 2005; 2007) as measurement of the feature independence and use two sub-networks (branches) with only few shared convolution blocks for target prediction task and domain prediction task. A glimpse of the basic form is shown in Figure 1.
Next, to further unleash the power of the proposed method, we suggest using domain augmentation to encourage the domain-invariant features to explore sufficient diversity of domain-specific represen- tations. Precisely, we propose a new random style sampling (RDS) scheme that explores augmenting the domain types by incorporating features with randomly modified style statisties. In contrast to previous methods (Zhou et al., 2021; Li et al., 2022) that use mixing or adding noise to synthesize new domains, RDS can directly perturb the mean and variance of feature maps with a controllable perturbing strength. To seamlessly integrate the basic form and the augmentation strategy. we further propose subsequential loss terms to encourage the target branch to be invariant to the original and augmented representations and vice versa for the domain-specifie branch.
Through our experimental studies, we illustrate, (1) the effectiveness of enforcing independence of the class and domain features within the early-branching design; (2) the advantages of the proposed RDS methods compared to existing solutions (Zhou et al., 2021; Li et al., 2022), and effectiveness of the adopted loss functions; (3) our complete method performs favorably against other state-of-the-art algorithms when evaluated in the current benchmark (Gulrajani & Lopez-Paz, 2021).
2 RELATED WORKS
Various methods have been proposed in the DG literature recently (Li et al., 2017; Motiia et al.,
2017; Li et al., 201 8b;a; Gong et al., 2019; Zhou al., 2020a; Zhao al., 2020; Li et al., 2019; Honarvar Nazari & Kovashka, 2020; Li et al., 2021; Zhou et al., 2021; Xu et al., 2021a; Kim et al., 2021; Wang et al., 2020; Bui et al., 2021: Yang et al., 2021; Li et al., 2022: Chen et al., 2022).
Despite the varying details, current DG methods can be roughly categorized into few categories by motivating intuition: invariant representation learning (Ganin et al., 2016; Li et al., 2017; 201 8b;e; Shi et al., 2021), augmentation (Zhou et al., 2021; Li et al., 2022; Xu et al., 2021a; Li et al., 2021). 1), and general machine learning algorithms such as meta-learning (Li et al., 2018a, Balaji et al., 2018;
Dou et al., 2019; Li et al., 2019) and self-supervised learning (Carlucci et al., 2019a; Jeon et al., 2021; Kim et al., 2021). This section briefly reviews methods from the most relevant categories.
Invariant representation learning. The pioneer work (Ben-David et al., 2006) theoretically proved that if the features remain invariant across different domains, then they are general and transferable to different domains. Inspired by this theory, many recent arts aim use deep networks to explore domain- invariaitt features. For example, (Ganin et al., 2016) train a domain- aaversaadvvesarialll neural network (DANN) to obtain domain-invarian features by maximizing the domain classification loss. This idea is further explored by (Li et al., 201 8b). They employ a maximum mean discrepancy constraint for the representation learning of an auto-encoder via adversarial training. Instead of directly obtaining the Nomain-invariant features, some arts (Khosla et al., 2012; Li et al., 2017) suggest decomposing the model parameters into domain-invariant and domain-specific parts and only using the domain- invain-invariant parameters for prediction when confronting unseen domains. Recently, the task has been further explored at a gradient level. Koyama and Yamaguchi (Koyama Yamaguchi, 2020) learn domain-invarial features by minimizing the variances of inter-domain gradients. Inspired by the fact that optimization directions should be similar across domains, (Shi et al., 2021) suggest maximizing the gradient inner products between domains to maintain the invariance. Different from previous approaches, our method is based on a simple intuition that domain and class features should be totally independent. By enforcing straightforward independent constraint, our method can achieve comparable or even better performance against these arts.
It is noteworthy that the carly-branching structure can work with many independence measurements, and we HSIC because it yields better performance. Please see Sec. 4.3 and B for details.
2

---

Under review as a conference paper at ICLR 2023
0c
2c
min
Class Loss
independent constraint
P and 6. class and domain classifler
0b
c and Z class and domain features
0d
Za
min
Domain Loss
tb. 0g 0a
base, class, and domain feature extractors
Figure 1: A glimpse of the proposed framework. The naive ERM is the framework without the independent constraint and the base feature extractor (i.e. 0,- I, where is the identity matrix).
Augmentation. Augmentation is one of the most popular strategies improve the robustness of model to novel domains. Besides the typical augmentation strategies, such as rotation, flipping, and cropping, there are some methods that are specially designed for the DG task. For example, (Yan et al., 2020) apply the mixup idea (Zhang et al., 2018) for DG by directly mixing images and labels from batch. Instead of mixing the image level. (Kim et al., 2021) propose mixup the feature representations, (Xu et al., 2021a) suggest linearly interpolating the amplitude spectrums of different samples. How to new samples for training is also studied in the literature: generative modeling is used in (Zhou et al., 2020b) and (Carlucci et al., 2019b) to create new domain and domain-agnosten images. Inspired by the success in the related area, both works in (Zhou et al., 2021) and (Nam et al., 2021) use AdaIN (Huang & Belongie, 2017) to synthesize new domain information by mixing the statistics of the features. To further explore the idea, (Li et al., 2022) propose to add perturbations to the statistics based on their variances, and (Kang al., 2022) suggest storing seen statistics in queue and consistently generating distinct statistics for mixing. However, despite certain improvements, their augmented domain information may still suffer from homogeneity, thus preventing the model from further generalization. To obtain unlimited and diverse domain information, we assume the statisties of the features to follow normal distributions. Consequently, we can sample unlimited diverse domain information that derives from the original statistics. By integrating the augmented features in an effective framework, our method performs favorably against existing alternatives the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021).
3
OUR METHOD
Problem setup. Let X' denote the image space, and y represents the label space 2, the domain generalization (DG) problem, we are given M source domains D. (D,, DM that sampled from different probability distributions on the joint space X' x V, where each Dm E D, Nm
can be represented as Dm = {(x:", y!"") """ Ji=1 and Nm, is the number of data label pairs in the mh domain, the task is to learn model from D, for making predictions
unseen DM41 domain.
3.1 LEARNING DOMAIN-INVARIAN FEATURES VIA INDEPENDENCE REGULARIIZTIIN
Our method is based on a simple principle: domain- iintaiiniivviiantt features should be independent of domain-specific features. This principle is based on the assumption that domain types and image classes are defined by different sets of visual cues and they can be estimated from different represen- tations. The independent constraint will enforce the Nomain-invariant features avoid focusing on the common cues, so that it can be less affected when the domain type varies during inference.
We implement this idea by learning two branches, one for domain classification and one for target classification, e.g., image classification. Formally, we follow the convention to use shared base feature extractor, denoted as 0,. The two branches are implemented as cascades of feature extractors and classifiers, denoted as {0c, }.} for the target branch and (0,, du} for the domain branch, respec- tively. Then we use Schhmet Information Criterion (HSIC) (Gretton et al., 2005; 2007) as the independence measure to regularize the statistic dependency between features from the domain and target extractors. We show in Sec. 4.3 that the adopted HSIC is more effective than other independent constraints such as orthogonal and correlation minimization. The learning objective (besides the common classification task) can be written as,
min HSIC( za). s.t. 2. = (x))))) za = (z())(z))
(1)
We only investigate DG the classification task, thus instances in y the one-hot class labels.
3

---

Under review as a conference paper at ICLR 2023
Table Evaluations of ERM and the proposed framework with different settings of 0, in the unseen domain from (Li et al., 2017). Here 2.31, is the four blocks in standard ResNet implementation with bec4 close to the classifier. The common practice of using shared base feature extractor with two lightweight prediction heads is detrimental to the performance.
art
cartoon
photo
sketch
avg
ERM
78.0 + 1.3
73.4
+ 0.8
94.1
+ 0.4
73.6 + 2.2
79.8 + 0.4
0b = I
79.1 + 0.7
74.1
+ 1.1
94.6
+ 0.4
74.8 + 0.6
80.7 + 0.5
0g bek1
79.3 + 0.9
74.1 + 2.0
94.8
0.5
75.8 + 1.6
81.0 + 0.6
0, = b
79.1 1.2
73.2 +: 1.8
94.8
74.9
1.3
80.5 0.8
0g 2c
78.3 1 0.2
72.8
0.6
9.9
74.1 0.9
80.0 + 0.2
0, = 2,31,
75.3 - 0.7
72.1
0.2
14.3000
71.5 0.6
78.3 = 0.3
where HSIC(ze; =a) measures the dependence between the representations 2e and zd. Note we do use the domain classification branch when deploying. Thus the objective is only imposed on 8g and 8. Details regarding Eq. (1) can be found in the appendix.
the branching location matters. At first glance, the aforementioned algorithm seems to be quite straightforward, but as will be shown the following parts, there are several factors that need to be carefully considered for it to work effectively. In this section, we focus on the first problem how to choose the neural network architecture for the shared base feature extractor and two branches. The common practice is to use deep sub-network the shared base feature extractor and two lightweight branches with much fewer model parameters (Chen et al., 2021; Atzmon et al., 2020). Surprisingly, empirically find that this strategy fails to produce good performance. To systematically investigate this issue, we conduct an experiment by testing the performance with different branching locations.
Observations:
We conduct experiments with ResNet18 (He et al., 2016) that has four blocks. We try to alter the model architecture by choosing block k k 21, 3, 4} as the shared part. As for the corresponding two branches, we use 4 k blocks plus an adaptive pooling layer and fully-connected layer for feature generation (i.e. denoted 0g or 0a) and a classifier (Øe da)) for each branch. We also consider special case when no shared base feature extractor is used, which is equivalent to using two networks for domain and target prediction. We denote those cases as 0 [I, bek1, bckk,2 22 bek1, ,2,3, bck1,2.3.11 and train the respective architecture with the loss in Eq. 1. Comparisons are shown in Table 1. We note that when the shared backbone deep, i.e., 0g = bek 2,3344 the performances are worse than the ERM baseline (79.8%). Meanwhile, using an earlier branching structure, e.g., 0, = bck, leads to better performances. Similar performances can also be observed when using different independent constraints (i.e. orthogonal and correlation minimization regularizations) and different network structures (i.e. smaller, larger convolution networks and vision transformers). Please refer to Sec. B in our appendix for more details.
This phenomenon can be understood from the role of t.. Idcally, we hope the features for the target branch do not encode any domain information. This is expected to be achieved via the independence regularization in Eq. 1. However, the domain and target branches share a significant part of the network, i.e., the base feature extractor 0g. Then Os will inevitably need to keep both domain and target information, making it harder to disentangle them at the respective branches. By contrast, if an braychhran architecture is used, we could minimize the need for encoding domain information in the pathway of target prediction, i.e., base feature extractor plus target prediction branch. Hereafter, we use 6,, bek; in the following discussion unless otherwise stated.
I-SNE visualizations. To further demonstrate the effectiveness of the aforementioned basic form, we present t-SNE visualizations of features extracted from the target feature extractor. We compare features obtained from our method and ERM. As plotted in Figure 2 (a) and (b), although both methods can separate different classes, our features tend to include less domain information as feature points from different domains are mixed. In comparison, we can see that features from different domains are clearly separated in the ERM case, indicating that ERM encodes more Nomain--speciin information in their target features. These results validate that regularization term indeed help disentangle the domain-invarien and domain-specific features, and the proposed framework is more effective than the ERM method. Please see the appendix for more analysis.
4

---

Under review as a conference paper at ICLR 2023
ort (tsti
Cartoon
photo
photo
skotch
(a) ERM
(b) Proposed
Figure 2: 2D I-SNE visualizations of target representations from the ERM model and our basic form. The PACS dataset (Li et al., 2017) is used with art as the unseen target domain. The seven clusters in (a) and (b) denote the corresponding classes. The domain information more obvious in the target features from the ERM model, indicating that ERM tends to learn entangled Nomain--pecinf and domain-invariant features. In comparison, using the independent constraint can better disentangle the two features, resulting in less domain information in the target features.
3.2 INCORPORATING AUGMENTED DOMAINS
Although the independence regularization and a proper architecture can lead to certain improvements, we are aware that the limited number of domains in the training set could easily overfit the target features, thus hindering our method from achieving its full potential. To overcome this limitation, we proposed
domain augmentation to further strengthen the independence between the features from the domain prediction branch and the target prediction branch. As suggested by recent study (Zhou et al., 2021), random changes in the style statisties of feature maps, e.g., by mixing the mean and variance of feature maps from two samples, can be used to augment the limited image domains in the training set, and thus improve the generalization ability of the model substantially. In this section, we discuss an extension of the aforementioned basic form by further incorporating this kind of domain augmentation technique. the following parts, we start by describing how our framework can be extended with domain augmentation in general, and then we discuss a new variant of domain augmentation scheme.
Formally, we use A(-) to denote domain augmentation operator that could modify image features in way that changes the image style while preserving the target class information. In our design, A(-) is applied to the features extracted from 0, so that both the two branches will be affected. We use f = (x) and A(f) to denote the original image features and the domain- -amaainn--gmeented image feature, respectively. Then the domain augmentation can be seamlessly incorporated into the proposed framework by the following loss functions.
The first is the classification loss Late that includes loss terms for both domain and target classification tasks. We use the cross entropy (CH) loss for the tasks, and Lets can be represented as follows:
Cate = (E())(O.())) y) + CE(O.(O.(A(())) v) + CE())(0d(f))) Dz),
where D2 the domain label æ. Note that this setting, we choose not to treat the augmented domain equally the true domain and thus do not assign domain labels for the augmented domain.
The second term is the sensitivity loss Lyen that regularizes the change of features from the target and domain branches under the domain augmentation. Intuitively, we expect the domain augmentation to have minimal impact on the target feature since supposed to be domain-invariant, On the other hand, expect sufficient change of features for the features for domain classification. Building upon this idea, we propose the following loss:
Leen =1(0.(f) - MLP((O (LP(EE |IMLP(O.())) I-(A (11)a(() MIL((E((A()))II - m)- + [[MLPP(((/)) - -AAP0///(CCAAAAAA)11 n]-),
(3)
where || ||1 is the L2 norm. As suggested by (Grill al., 2020; Kim et al., 2021). use |LLP(a) bll to measure the compatibility between a and b. has shown that this scheme can prevent representation collapse (Grill et al., 2020; Kim et al., 2021). [1- min(:; 0), and the margin is
5

---

Under review as a conference paper at ICLR 2023
considered parammtrr which is set to be the maximum value of the L1 distances computed from the current batch.
The last loss term Lindp brought by the independent constraint. The independence between features from the two branches should be maintained under the augmentation. Therefore, we perform HSIC minimization regardless of whether original data augmented data is used. Expanding the cases, this is equivalent to minimizing the following four-term loss:
Lindp (f) + (a))
HSIC(O:(A(S)) O2(J)) + HSIC(O,(AL))) (z(A))))
In summary, the overall loss Laul for the proposed algorithm can be expressed as, Las = Lota + aLL + Bindds
(5)
where and A are weight parameters.
3.3 A NEW DOMAIN AUGMENTATION STRATEGY
To better unleash the power of the proposed method, we design new domain augmentation strategy dubbed random domain sampling (RDS). Our method is inspired by (Zhou al., 2021), which shows that new domains can be obtained by mixing the style statistics of two different samples. In (Zhou al., 2021), the style statistics are the mean μ RBxC and standard derivation d² E RBxC of activations across all spatial locations and all samples in a batch, given a set of B x C x H x W sized feature maps (where B, C, H, and W are batch, channel, height, and width of the feature map).
However, modifying μ and o² by directly mixing with other image style statistics may create more homogeneous features when confronting similar domain types within batch. RDS solves this problem by perturbing the statistics with controllable strength. Specifically, we first calculate the mean and variance of activations from each image, denoted as μı and of Then, we build probabilistic model for μ4 and of which is done by assuming they follow the Gaussian distribution and then estimating the model parameters within a batch. Finally, we can sample new μ and o² from style statistics. Take μ example. The above procedure can be
the Gaussian distribution written as:
B By)(p= R
N (R Ep), s.t. EE
1
Σ
M..
Σ
1
E(m
(6)
B
B
To obtain inhomogeneous styles, we only accept sampled j whose density is less than e. The accepted samples, denoted as f should satisfy
exp (- (
Ex.) <E.
where is sufficiently small (i.e. the sampled f can be distinct from the original μ. An illustration of the sampling strategy is shown in Figure 3. The same sampling process is applied to obtain
set to be 0.0001 in our implementation) so that
After sampling new style statistics, follow the protocol in AdaIN (Huang & Belongie, 2017) by replacing the style statistics from the original representation with the sampled ones, and the final formulation for our domain augmentation can be written as,
original style sampled style
A(f)
1
(8)
where f is the dimensional activation vectors at each spatial grid of feature Differeet from the previous work (Li et al., 2022; Zhou et al., 2021) that specifically adds normal perturbations to their style statistics or bor- rows style statistics from other samples, RDS manages to control how the sampled statistics deviated from the existing style statistics thanks to its probabilistic modeling of μ4 and of. This prevents the augmentation strategy from creating more homogeneous domains and alleviates the drawback when confronting similar domain information in batch.
Figure A simple demonstration the RDS sampling strategy.

---

Under review as a conference paper at ICLR 2023
Table 2: Evaluations DomainBed (Gulrajani & Lopez-Paz, 2021). All methods examined for 60 trials in each unseen domain. ERM + HSIC is our basic form introduced in Sec. 3.1 without the augmentation and with 0, blkk Top5 accumulates the number of datasets where a method achieves the top 5 performances. Score here accumulates the numbers of the dataset where specific art performs better than ERM. Best and second best results are presented in bold and undelined types.
PACS
VLCS
OfficeHome
Terralcc
DomainNet
Avg.
Top5
Score
MMD
81.3 = 0.8
74,9 = 0.5
59.9 = 0.4
42.0 1.0
7.9 - 6.2
53.2
RSC
80.5 = 0.2
75.4 0.3
58.4 + 0.6
39.4 + 1.3
27.9 2.0
56.3
0
IRM
80.9
58.0 0.1
38.4 0.9
30.4
56.6
0
ARM
80.6
75.9 0.3
59.6 0.3
37.4 1.9
0.1
56.7
0
DANN
0.3
59.5
0.
0.1
56.9
GroupGRO
80.7 0.4
75.4 1.0
60.6 0.3
4..
2.1
57.1
CDANN
80.3
76.0
59.3
38
3.2
VREx
80.2
75.3 0.6
59.5 0.1
43.2
2..1
57.3
CAD
81.9 0.3
75.2 0.6
60.5 0.3
4..5
31.0
57.8
CondCAD
80.8 = 0.5
76.1 = 0.3
61.0 0.4
39.7
3.7
57.9
MTL
80.1 + 0.8
75.2
0.3
59.9 0.5
40.4+100 1.0
3.00
58.1
ERM
79.8 = 0.4
75.8 0.2
60.6 0.2
38.8 1 1.0
0.1
58.1
MixStyle
82.6 0.4
75.2
0.7
59.6 0.8
1.111111111
3.1
58.4
ERM HSIC
81.0 0.6
76.1
0.1
60.7 0.3
38.8 #
3.14
58.4
MLDG
81.3 0.2
75.2 0.3
60.9 0.2
0.
3.0
58.6
Mixup
79.2
76.2 0.3
4.71
3.00
Fishr
81.3 0.3
76.2 0.3
60.9 0.3
42.6
0.3
59.0
SagNet
81.7 +
75.4 0.8
62.5 0.3
40.6 1..
3.1
59.1
SellReg
81.8 = 0.3
76.4 0.7
62.4 0.1
0.3
3.27
59.3
Fish
82.0 = 0.3
76.9 0.2
0.6
4.6
3.0
59,3
5
CORAL
81.7 0.0
75.5 0.4
0.
1.
3.21
59.4
4
SD
81.9 0.3
75.5 044
0.2
42.0
36.3
59.7
Ours
82.4 0.4
76.5 0.4
0.1
43.2
0.1
59.8
EXPERIMENTS
4.1 DATASETS AND DETAILS
Datasets, To evaluate the effectiveness of our method, we conduct extensive experiments on five DG benchmarks, namely PACS (Li al., 2017), VLCS (Fang et al., 2013), OfficeHome (Venkateswara et al., 2017), TerraInc (Beery et al., 2018), and DomainNet (Peng et al., 2019). Specifically, PACS consists 9,991 images which can be divided into 7 classes. This dataset is the most commonly used DG benchmark due to its large distributional shift across 4 domains including art painting, cartoon, photo, and sketch; VLCS collects a total 10,729 images from different datasets (i.e. PASCAL VOC 2007 (Everingham et al., 2010), LabelMe (Russell et al., 2008), Caltech (Fei-Fei et al., 2004), and Sun (Xiao et al., 2010)) which can categorized into 5 classes. Images from different datasets taken under different views and considered different domains in DG; OfficeHome is an object recognition dataset that consists of images in office and home environments, and there are a total of 15,588 images from 65 classes in it. Images from OfficeHome including artistic, clipart, product, and real world; TerraInc contains 24,788 images of wild animals from 10 classes. Those images are taken from different locations (i.c. L100. L38, L43, L46), and the locations are regarded as the varying domains in DG; DomainNet contains 586,575 images. There are a total of 345 classes included in DomainNet whose domains can be depicted styles (i.e. clipart, infograph, painting, quickdraw, real, and sketch).
be divided into domains
Implementation details. We follow the prevalent design by using ResNet18 (ImageNet pretrained) as the backbone, which could enlarge the gaps in DG compared to larger models (Ye et al., 2022). The syper-parameters our model include a and |3 in Eq. (5). We use dynamic range of [0.03, 0.3) and B E [0.1,] for experiments (analysis regarding these two
given in the appendix). Settings for all the compared methods are set according to (Gulrajani & Lopez-Paz, 2021).
Evaluation details. For all the compared arts, follow the default settings in (Gulrajani & Lopez- Paz, 2021). The leave-one-out strategy is used for evaluations. That is, for all the datasets, one domain is selected the held-out for test and the remaining domains treated as souree domains for training. We evaluate the methods 60 times each source domain. The learning rates, augmentation strategies, random seeds, and batch sizes all dynamically set for the compared

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation studies regarding the effectivenesses of the proposed RDS augmentation strategy, the adopted losses, and the adopted HSIC independent constraint. Experiments are conducted in PACS (Li et al., 2017) with the leave-one-out training-test strategy. Here RDS our RDS without Eq. (7); Coth and indp are variants of our method that use orthogonal and correlation as independent measurements.
model
art
cartoon
photo
sketch
avg
Ours
81.3 1 1.4
75.7 = 0.2
94.6 + 0.3
78.1 = 0.7
82.4 + 0.4
Ours w/o augmentation
79.3 0.9
74.1 2.0
0.5
75.8
1.6
81.0 0.6
Ours w/ MixStyle
80.8 0.8
0.3
0.5
77.4 1.5
81.9 0.3
Augments
Ours w/ DSU
81.3 0.5
7.81
9..2
76,3 0.5
81.7 + 0.3
Ours w/ RDS
75.4
9.
76.6 1.7
81.9 + 0.7
Ours w/o Lindp & Laen
79.9 0.7
7..6
9.2
81.5 7.82
Losses
Ours w/o Lindp
80.5 0.5
76,2
9.2
0.2
81.8
Ours w/o Leen
79.3 + 0.4
75.4 0.9
94.5
4 0.2
78.3 4 1.7
81.9 1 0.5
Ours indp
80.6 1.2
75.4 0.5
94.4
77.9
1.4
82.1 0.3
Constraints
Ours w/ Lindp
81.2 0.6
75.8 1 0.2
0.6
77.7 0.5
82.2 + 0.3
methods in reasonable ranges, and the iteration step is fixed as 5,000 for each training. During training, we split the examples from training domains to 8:2 (tra and evaluate the models every 300 iteration steps. Note the training and validation samples are also dynamically selected among different training trials. During test, select the model that performs the best in the validation samples and test it on the whole held-out domain, which is also the "training-domaa model selection method in (Gulrajani & Lopez-Paz, 2021). For each domain in different datasets, the final performance is the average accuracy computed from the 60 trials, and the performance in this dataset is the average of the corresponding domains. To ensure fair comparisons, all the methods are evaluated in same device (8 Nvidia Tesla v v100 GPUs and each with 32G memory).
validate set"
4.2 EXPERIMENTAL RESULTS
Evaluation results are shown in Table 2. We observe that the naive ERM method obtains comparable results against existing arts. In fact, as strong baseline, ERM is ranked I 11th place in the term of average accuracy, and only half of the state-of-the-arts can outperform ERM in most datasets (i.e. Score 2 3). Specially, we note that when using sophisticated designs to learn domain-invarian features on the basis of ERM, the average performances seem to decrease in most cases for IRM (Arjovsky et al., 2019), MMD (Li et al., 2018b), DANN (Ganin et al., 2016), CDANN (Li et al., 2018c), and RSC (Huang et al., 2020). By contrast, when ERM is imposed with straightforward independent constraint (i.e. ERM + HSIC in the table with 0g blk1, which is detailed in Sec. 3.1), the average accuracy increases in most datasets, and the corresponding overall performance is also better than the naive ERM method. The results validate that the simple independent constraint is more effective at disentangling domain-invariant and domain-speciin features than some pioneer arts.
Moreover, compared with alternatives that either use augmentations (i.e. Mixup (Yan al., 2020), SagNet (Nam et al., 2021), SelfReg (Kim et al., 2021), MixStyle (Zhou et al., 2021), and CAD (Ruan al., 2022)) or gradient-level constraints (i.c., Fish (Shi al., 2021), SD (Pezeshki al., 2021), and Fishr (Rame et al., 2022)), our method consistently shows comparable or better performance in out datasets and is ranked 1st place in the
most datasets which obtains the top performances term of average accuracy of all 5 benchmarks. The results validate the effectiveness of our method. present results of average accuracy in each domain from different datasets the appendix.
4.3 ABLATION STUDIES
All experiments in this section conducted on the PACS (Li et al., 2017) benchmark with the same settings illustrated in Sec. 4.1. Please refer to the appendix for more ablation studies.
Effectiveness of our RDS augmentation strategy. Our augmentation strategy RDS aims to further strengthen the generalizability of our method by preventing the target features from overfitting to the limited training domains. We first compare model against the baseline model that does not any augmentation strategies (i.c. Ours w/o augmentation in Table 3). Note when the augmentation step is disabled, the overall algorithm reduces to the framework introduced in Sec. 3.1 with 0, boki
8

---

Under review as a conference paper at ICLR 2023
where Leen, related parts in Cels, and Lindp in Lall are also disabled. As shown in the first row in Table 3, when enabling augmentation. our method can outperform the baseline model by a large margin in almost all domains except for "photo", which might be due the ImageNet pretraining (Xu et al., 2021b). Note that the improvement for "sketch" is more pronounced than others, indicating our RDS can generalize to a target domain that significantly deviates from the source domains.
We also compare variants of our method by replacing RDS with other augmentations based on AdaIN (i.e. Ours w/ MixStyle (Zhou et al.. 2021) and Ours w/ DSU (Li al., 2022)). As listed in the 3rd and 4th rows, RDS consistently outperforms augmentations from (Zhou et al., 2021) and (Li et al., 2022) on average. We postulate that this is due to RDS's capability of synthesizing more diverse domains thanks to the sampling scheme. Moreover, to test if RDS can bring improvements against the naive sampling strategy, we evaluate another variant of our method by disabling the process depicted in Eq. (7) (i.e. Ours w/ RDS Results in the 5th row show that naive sampling is not effective as the proposed sampling strategy, which is not surprising since styles from the naive sampling may not be diverse enough compared to that from RDS. Note that the improvements are more noticeable than others when it comes to the "sketch" domain, which might be due to the reason that the rest three domains close (Zhou et al., 2021), thus requiring less diverse information for generalizing. All these results validate the effectiveness of the proposed RDS against existing methods.
Effectiveness of the losses. In our full algorithm, there are two additional loss terms in addition to the main classification losses: the sensitivity loss Leen and the independent constraint loss Lindp- We study their effectiveness by disabling them separately in our framework. Results listed in the 6th, 7th and 8th rows in Table 3. We observe that disabling either one of them can lead to performance decreases over our original design, demonstrating that each loss can independently contribute to the good performance of our method. Meanwhile, we also note that when the two loss terms are both disabled, the corresponding model is inferior to other variants, meaning these two terms are complementary in our framework and can further improve the generalization.
Effectiveness of the adopted HSIC independent constraint. We minimize the HSIC value between the target and domain features (i.e. &e and zd) to reduce their dependency. As there exist other dependence measurements, such as orthogonal and correlation, one may wonder if HSIC is better choice compared to them. To answer this question, we conduct ablation studies by replacing the HSIC measurement with the other two criteria. The orthogonal constraint requires the multiplication two features to be zero (i.e. Ours w/ C the correlation constraint minimizes the cross- corsecoorel matrix computed between be and 2d, and we use the implementation from (Zbontar et al., 2021) for this experiment (i.e. Ours w/ (ndd)) Experimental results listed in the 9th and 10th rows in Table suggest that the adopted HSIC independent constraint more effective than the other two. We thus use the HSIC criterion in this work.
the
5 DISCUSSIONS AND CONCLUSION
Limitations. The major limitation of this work is that both the target and domain labels are required during training. Since we need the domain- -specffi features to partly guide the target representa- tion learning. it is non-trivial to directly extend our method to occasions where domain labels are unavailable. We leave the exploration of such a setting to future work. Another drawback of the proposed method is that the framework requires another thread for the domain-specifie classification task during training, bringing more parameters to the overall pipeline. Although we can use the same amount of parameters for the target classification task when deploying the model, training both two tasks will inevitably bring extra effort to the system. A promising improvement for the method is to use much fewer parameters to obtain comparable performance against the original implementation.
Conclusion. In this paper, we propose a method for domain generalization by imposing independent regularization for the features learned for domain classification and target prediction. We identify that several factors need to be considered to make this simple idea work effectively: first, an early branching network architecture is more effective than the commonly used deep backbone and lightweight predicting branch structure; second, incorporating augmented domains can further improve the learning outcome, and we introduce a carefully designed domain augmentation method to benefit the generalizability. These two findings are seamlessly integrated by the subsequential loss terms. Through extensive experiments on five benehmark datasets, we show that the proposed algorithm can obtain competitive performance against tfte-offthe-art alternatives.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
For data pre-processing steps, network configurations, and detailed learning settings, we use the existing benchmark (Gulrajani & Lopez-Paz, 2021) for all the implementations. Hyper-parameters specially used in this work, including a and B used in Eq. 5, are fixed for all experiments and are analyzed in Sec. C.4. Please also refer Table 12 for details. We will release the code upon acceptance.
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: arXiv:1907.02893 2019.
Yuval Atzmon, Felix Kreuk, Uri Shalit, and Gal Chechik. A causal view of compositional zero-shot recognition. In NeurlPS, 2020.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain general- ization using metta-rggularization. NeurlPS, 2018.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NeurlPS, 2006.
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. arXiv preprint arXiv:/711.07910. 2017.
Manh-Ha Bui, Toan Tran, Anh Tran, and Dinh Phung. Exploiting domain-specific features to enhance domain generalization. NeurlPS, 2021.
Fabio M Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019a.
Fabio Maria Carlucci, Paolo Russo, Tatiana Tommasi, and Barbara Caputo. Hallucinating agnostic images to generalize across domains. In ICCVW, 2019b.
Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoqing Liu, and Yizhou Yu. Compound domain generalization via meta-knowleege encoding. In CVPR, 2022.
Yang Chen, Yu Wang, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei. A style and semantic memory mechanism for domain generalization. In ICCV, 2021.
Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NeurIPS, 2019.
Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 88(2):303-338, 2010.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV, 2013.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR worksho, 2004.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 17(1):2996-2300 2016.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
10

---

Under review as a conference paper at ICLR 2023
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Seatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE TPAMI, 39 (7):1414-1430, 2016.
Rui Gong. Wen Li, Yuhua Chen, and Luc Van Gool. generalization. In CVPR, 2019.
Dlow: Domain flow for adaptation and
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms, In ALT, 2005.
Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song. Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. In NeurlPS, 2007.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 2020.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Narges Honarvar Nazari and Adriana Kovashka. Domain generalization using shape representation. In ECCV, 2020.
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In UAI, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
Zeyi Huang, Haohan Wang, Eric Xing, and Dong Huang. Self-challenging improves drosin generalization. In ECCV, 2020.
Seogkyu Jeon, Kibeom Hong, Pilhyeon Lee, Jewook Lee, and Hyeran Byun. Feature stylization and domain-aware contrastive learning for domain generalization. ACMMM, 2021.
Juwon Kang, Sohyun Lee, Namyup Kim, and Suha Kwak. Style neophile: Constantly seeking novel styles for domain generalization. CVPR, 2022.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In ECCV, 2012.
Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In ICCV, 2021.
Masanori Koyama and Shoichiro Yamaguchi. When invariance useful in an out-of-distribution generalization problem? arXiv preprint arXiv:2008.0 01888, 2020.
David Krueger, Ethan Caballero, Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In ICML, 2021.
Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, 2017.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI, 2018a.
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, 2019.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In CVPR, 2018b.
11

---

Under review as a conference paper at ICLR 2023
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In ICCV, 2021.
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling- Yu Duan. Uncertainty modeling for out-of-distribution generalization. In ICLR, 2022.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao.
Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018c.
Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In ICCV, 2017.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICMI,, 2013.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, 2021.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019.
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: A learning proclivity in neural networks. In NewrlPS, 2021.
Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of-distributtin generalization. In ICML, 2022.
Yangjun Ruan, Yann Dubois, and Chris J Maddison. Optimal representations for covariate shift. In ICLR, 2022.
Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. IJCV, 7711:1577173,, 2008.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In ICLR, 2021.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Shujun Wang, Lequan Yu, Caizi Li, Chi- Wing Fu, and Pheng-Ann Heng. Learning from extrinsic and intrinsic supervisions for domain generalization. In ECCV, 2020.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. CVPR, 2010.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. fourier-based framework for domain generalization. In CVPR, 2021a.
Zhenlin Xu, Deyi Liu, Junlin Yang. Colin Raffel, and Mare Niethammer. Robust and generalizable visual representation learning via random convolutions. ICLR, 2021b.
Shen Yan. Huan Song. Nanxiang Li. Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint 20X1v:2001.00677, 2020.
12

---

Under review as a conference paper at ICLR 2023
Fu-En Yang, Yuan-Chia Cheng, Zu- Yun Shiau, and Yu-Chiang Frank Wang. Adversarial teacher- student representation learning for domain generalization. In NeurIPS, 2021.
Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In CVPR, 2022.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
Jure Zbontar, Li Jing, Ishan Mísra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group distribution shift. arXiv preprint rcXiv:2007.02931, 2020.
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generalization via entropy regularization. In NeurIPS, 2020.
Fan Zhou, Zhuqing Jiang, Changjian Shui, Boyu Wang, and Brahim Chaib-draa. Domain generaliza- tion with optimal transport and metric learning. arXiv preprint rrXiv:2007.1057: 2020a.
Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, 2020b.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
13

---

Under review as a conference paper at ICLR 2023
In this appendix, provide
Detailed steps for computing the HSIC loss in Eq. (1) in the manuscript in Sec. A;
2. More evaluations of the proposed carly-branching structure in Sec. B;
Further results and analysis in Sec. C.
4. Detailed results of the experimental results from the five benchmark datasets in Sec. D;
A COMPUTING HSIC
The Hilbert-Schmidt Information Criterion (HSIC) (Gretton et al., 2005; 2007) is a non-parametric method for estimating the statistical dependence between samples of two variables. Given two batches of vectors (i.c. and za from the target and domain branches), to compute the linear-kerel HSIC(ze, zd), we first normalize the given vectors: &
So and Ed ||==||
where || denotes 2a
Teall
the L2 norm. Then, HSIC is calculated as the scaled Hilbert-Schmidt norm of their cross-covariance matrix:
obtain two linear kernel matrices: K =c I. Ezzd". Finally, the
HSIC za) = [B 1)a
tr (r(KHH),
where B is the batch size, H - - is centering matrix, and tr(.) computes the trace of the given matric. We refer the reader to the original paper (Gretton al., 2005; 2007) for detailed derivations.
B MORE EVALUATIONS OF THE
STRUCTURE
In the manuscript, we suggest that despite the effectiveness of the adopted independent constraint, an early-branching network structure is essential to the overall performance. This phenomenon can be explained by the role the base feature extractor 0,, If the two branches share significant part of 0b. Then 0, will be asked to provide both domain and target information, making it harder to disentangle them at the respective branches. However, if an early- branching architecture is used, we could minimize the need for encoding domain information in the pathway of target prediction.
B.1 EARLY-BRANCHING STRUCTURE WITH DIFFERENT STRUCTURES.
To further verify our hypothesis, we conduct more evaluations with different network structures in this section. (1) We evaluate the independent constraint on a simple network that consists of convolution layers. Here the domain of 0, (i.e. 0, {I. bck1, bck1,2, bck1, 2ck1,2.3.4) }) is similar to that in the manuscript with boka closest to the classification head. We use the Rotated MNIST dataset (Ghifary et al., 2015) (i.e. RMNIST) in this experiment. RMNIST is originally designed for the handwritten digit recognition task which consists 70,000 samples and can categorized into 6 domains by its rotation degree: {0, 15, 30, 45, 60, 75}. (2) We use larger ResNet50 model with more parameters to evaluate the adopted independent constraint. (3) We adopt vision transformers to evaluate the carly-branchin
the DeiT (Touvron et al., 2021) and T2T (Yuan et al., 2021) transformers that both with 12 attention layers in our evaluations. Considering the consistency with the aforementioned network structures, we divide the 12 attention layers into 4 blocks and each with 3 attention layers. In these two experiments, the PACS dataset (Li et al., 2017) is utilized. Note in all three evaluations, the experimental settings are the same as that in the benchmark (Gulrajani & Lopez-Paz, 2021).
structure. Specifically,
Results are listed in Table 4, 5, and 6. We observe that we can obtain the optimal results on both the smaller (i.e. 4 convs), larger (i.e. ResNet50) convolution networks, and transformers (i.e. DeiT (Touvron et al., 2021) and T2T' (Yuan et al., 2021)) with an carly-branching structure, and the results decrease thereafter. These experiments further validate our hypothesis.
B.2 EARLY-BRANCHING STRUCTURE WITH DIFFERENT INDEPENDENT CONSTRAINTS.
Except for the adopted HSIC, there also exist other dependency measurements, such as orthogonal and correlation. To verify if the carly-branching structure boost the performance despite the
14

---

Under review as a conference paper at ICLR 2023
Table 4: Evaluations of the early-branching structure with lightweight four layer network in the unseen domain from RMNIST (Ghifary et al., 2015). Here 2c31,2.3,1 is the four convolution layers with bok, close to the classifier. The proposed framework achieves the optimal performances with an carly-branching network structure (i.e. 0%. bek: or 0b I).
0
15
30
45
60
75
avg
ERM
95.7 + 0.2
98.6
+ 0.1
98.9 + 0.1
99.0
1 0.1
98.8 1 0.0
96.0
+ 0.1
97.9 1 0.1
0
I
95.8 = 0.0
98.1
0.1
98.8
= 0.0
99.4
1 0.1
99.0
1 0.0
96.6
1 0.2
98.0 1 0.1
0
bek1
96.2 = 1.0
98.4 0.1
99.1 1
0.1
99.0
0.1
96.3
0.330...
98.0 0.1
06
bok1.2
91.4 + 1.0
98.2
98.6 0.1
99.0
98.8 0.1
9..0
96.8 + 0.1
8
boky 2233
88.7 + 0.3
0..5
98.5 0.0
9.0
98.6
94.3
96.1 + 0.1
06
1011,,,3,4
90.4 + 0.6
97.0 + 0.3
98.4 + 0.0
98.5 0.1
98.6 + 0.1
92.8 0.18+0011
95.9 + 0.2
Table 5: Evaluations of the early-branching structure with Resnet50 backbone in the unseen domain from (Li et al., 2017). Here 2ck1,2.3,1 1, is the four layers in a standard ResNet50 implementation with beky close to the classifier. The proposed framework achieves the optimal performance with an carly-branching network structure (i.e. 0, bek: or 0b I).
cartoon
photo
sketch
avg
ERM
85.1 + 1.3
78.0 + 1.2
97.3
+ 0.1
72.2
+ 1.6
83.1 + 0.9
0 I
85.8 + 0.9
77.6
+ 0.1
97.2 1 0.2
17.7
1 0.4
84.5 1 0.3
0
bek1
85.6 + 1.0
77.9 + 1.7
97.5 0.4
77.7
+ 0.1
84.7 + 0.5
06
b
84.2 0.7
77.7
+ 1.1
96.3
+.2
73.5
0.5
82.9 + 0.4
0 bek1,2.
81,2 + 1.2
77.1
+ 0.6
97.1
0.1
72.6
0.7
82.0 + 0.5
06 2,31,2,344
85.4 + 1.6
78.0
+ 0.8
97.1 + 0.2
66.4 2.4
81.7 + 0.7
varying independent constraints, we conduct experiments by replacing HSIC with other criteria. All the experiments conducted the PACS dataset (Li et al., 2017) with the ResNet18 backbone. As shown in Table 7, the parly-branchin structure consistently outperforms the common practice of using lightweight prediction heads when the HSIC measurement is replaced with orthogonal and correlation criteria. These results demonstrate that dependency measurement is not a crucial factor in the early-branching structure.
B.3 KARLY-BRANCHING STRUCTURE WITH AUGMENTATION,
Moreover, since our method incorporates a random domain augmentation scheme into the dual- branching network, one may wonder if the early-branching network structure is still the optimal choice in our method. To answer this question, we conduct experiments on the PACS dataset (Li et al., 2017) using both the ResNet18 and ResNet50 networks (He et al., 2016). Note in these experiments,
Table Evaluations of the barly-branching structure with vision transformer backbones (i.e. DeiT Touvron et al. (2021) and T2T (Yuan et al., 2021)) in the unseen domain from (Li et al., 2017). Here bck1,2,3,4 is the 12 attention layers the transformers, with each block corresponding attention layers and bok, close to the classifier. The proposed framework achieves the optimal performance with an early-branching network structure (i.e. O, bokj or Og I).
art
cartoon
photo
sketch
avg
FRM- VIT
87.3 + 0.4
81.5 + 0.3
98.7 + 0.4
74.5 + 1.9
85.5 + 0.6
9g
87.7 + 0.6
81.2 + 0.4
98.3
+ 0.2
76.0
+ 2.5
85.8 +58+0.4
0, boky
0.8
80.5
0.3
0.9
0.2
Deit
0, = bekj,2
8.79
80.6
9.0
74.6
0.4
0. = bok, 22
86.5
0.8
0.3
0.3
0.3
0, = boky 2.4
83.2 2.0
77.8
97.9 0.4
75.5
83.6 0.5
ERM VIT
8.2
0.7
9.3
7.
83.5
1.5
0, I
86.0
7..0
9.1
75.7
84.0
0. boki
0.9
75.6
9.1
0.5
0.5
T2T
0, bokj,2
8.
774
9.59
74.7
1.5
0.3
0, bok,,2,3
81.6
1.
9.65
75.4
1.
83.2
0, 6ck1,2.3.4
82.6 1 0.6
73.6 08
98.0
74.2 12
2..
15

---

Under review as a conference paper at ICLR 2023
Table 7: Evaluations of early-branching structure with different independent constraints in the unseen domain from (Li et al., 2017) using the ResNet18 backbone. Here ,,,3,2,3,4 is the four layers in standard ResNet implementation with bck, close the classifier.
art
cartoon
photo
sketch
avg
ERM
8.0 +
73.4
0.8
94.1
0.4
73.6
2.2
79.8
0.4
0/
80.6 0.6
73.9 1.8
9.1
73.0 0.8
80.4
0.4
0g beki
78.9 1.3
74.3 0.6
=.5
74.7
80.5 0.3
Orthogonal
0, b
=
73.8 1.9
=4.5=0.4
7.42
80.2 0.4
0 2,3
75.2
72.6=1.0
94.5=0.7
7..1
7.6
0
behy 223,4
75.9 1.4
0.5
=4.0
7.01
7.8
0/
0.51
=
93.5 35.50..5
1.4
80.3 0.5
0. boks
79.7
76.1
94.0 0.6
1.4
-.5
Correlation
bok1,2
76.9
74.7
94.0=
7.94
80.3 00.30055
bck1,2,3
1.3
73.2 1.5
93.3
711
=
0. 1,.1,2,3.44
76.2 0.6
72.8 1.2
94.0 0.4
66.5 0.8
77.4 0.3
Table Evaluations of the early-branching structure with augmentation in the unseen domain from (Li et al., 2017) using the ResNet18 and ResNet50 implementations. Here b 2,,3,4 is the four layers standard ResNet implementation with bck; close the classifier. We do implement O, because it cannot cocxist with our augmentation strategy. Our method consistently performs the best with an early-branching structure when the augmentation is applied.
ResNet18 structure
ResNer50 structure
art
cartoon
photo
sketch
avg
art
cartoon
photo
sketch
avg
ERM
78.0
73.4
94.1
73.6
79.8
85.1
78.0
72.2
0,
boki
81.3
75.7
94.6
78.1
82.4
86.2
81.4
97.3
78.7
85.9
0
bek1,2
81.2
76.4
94.5
76.8
82.2
85.3
80.4
97.3
78.5
85.4
0,
beki, 2,3
75.5
75.8
92.4
75.2
79.7
82.2
78.1
96.5
76.0
83.2
0.
20.1,2,3,4
54.8
50.6
77.7
52.3
58.9
62.4
57.5
80.0
46.4
61.6
we do not implement the setting when 0, I, because it can not coexist with our augmentation strategy. As shown in Table 8, the best performances can be achieved when 0, = book both the two networks. The results suggest that our hypothesis is true in both situations when augmentation strategy is applied or disabled.
B.4 USING THE SAME AMOUNT PARAMETERS FOR THE EARLY- BORLY-BRANCHING STRUCTURE
As the number of parameters for the overall training system is larger when sharing fewer blocks, one may wonder if it can affect the performances. To answer this question, we add extra blocks for the domain-estimation branch (i.e. 02) to ensure that the training pipeline shares the same amount of parameters for different branching locations. Note we do not add parameters in the sharing part and the main classification branch (i.e. 0, and f.) because will use more parameters than the baseline ERM model during the test phase, which is unfair comparison. We conduct ablation studies on the PACS dataset (Li et al., 2017) and list the results in Table 9, The observations are line with the findings in our manuscript: the early-branching structure still performs better than the common practice with lightweight prediction heads when using the same amount of parameters.
C FURTHER RESULTS AND ANALYSIS
C.I RESULTS USING THE RESNET50 BACKBONE
To comprehensively evaluate the effectiveness of our method, we also conduct experiments using the ResNet50 backbone (He et al., 2016). these experiments, we train the basic form of our method (i.e. ERM HSIC), the strong baseline (i.e. ERM method (Vapnik, 1999)), and our method using the DomainBed benchmark (Gulrajani & Lopez-Paz, 2021), and we cite the results from c for other compared models.
16

---

Under review as a conference paper at ICLR 2023
Table 9: Evaluations of the early-branching structure in the unseen domain from (Li et al., 2017) using the same amount of parameters during the training phase. Here ,031,2,3,1 is the four blocks standard ResNet implementation with beky close to the classifier. We add extra blocks for the comain-estimation branch (i.e. 0a) to ensure that the training pipeline shares the same amount of parameters for different branching location. We do not add parameters for the sharing part and the main classification branch (i.e. Og and 0.) to ensure fair comparisons with the ERM method.
cartoon
photo
sketch
avg
ERM
78.0 1 1.3
73.4
1 0.8
94.1 1 0.4
73.6 1 2.2
79.8 1 0.4
0 I
79.1 + 0.7
74.1
+ 1.1
+4.6+0.4
74.8 + 0.6
80.7 + 0.5
0. boki
80.5 + 1.4
74.1
+ 0.9
94.2+00.
75.6 + 0.3
81.1 0.5
0 b
#.111.111
74.0
0.7
9..4
75.7
1.2
80.6 0.4
8 boks, 2.
79.2
0.7
72.4
9.5
75.0
+ 1.2
80.1
+ 0.6
06 2,
76.0 0.9
71.6 0.7
0.3
71.6
0.6
78.2 + 0.5
Table 10: Average accuracies on the DomainBed (Gulrajani & Lopez-Paz, 2021) benchmark using the ResNer50 (He et al., 2016) backbones. ERM + HSIC is our basic form without the augmentation and with 8, blkg Results without are directly cited from previous work (Kim al., 2021).
PACS
VLCS
OfficeHome
TerraInc
DomainNet
Average
ERM (Vapnik, 1999)
83.5 + 0.2
1
66.3 0.3
46.1 = 1.8
40.9 1 0.1
63.3
IRM (Arjovsky al., 2019)
83.5 0.8
78.5
47.6
0.8
33.9 2.8
61.2
GroupGRO (Sagawa et al., 2020)
8..4
0.6
0.7
4.1
0.2
60.7
Mixup (Yan al., 2020)
0.6
0.6
0.3
0.8
39.2
63.4
MLDG (Li all.ll....111 2018a)
1.
0.
0.6
4.9
4.1
63.6
CORAL (Sun Saenko, 2016)
0.3
68.7 0.6
1.0
4.1
64.6
MMD (Li et ul., 2018b)
8.
66.3 0.9
1.6
2.54
58.8
DANN (Ganin al., 2016)
8..6
7.
65.9 0.6
0.5
0.1
62.6
CDANN (Li al., 2018c)
0.9
0.1
65.8 1.3
1.6
0.3
62.0
MTL (Blanchard et al., 2017)
0.5
7.2
66.4 0.5
45.6 1.2
0.1
62.9
SagNet (Nam al., 2021)
8.2
+.5
68.1 0.I
48.6 1.0
4.1
64.2
ARM (Zhang al., 2020)
8..
0.3
64.8 0.3
45.5
0.2
61.7
VREx Krueger
(2021)
0.6
78.3
66.4 0.6
3 0.6
61.9
RSC (Huang al., 2020)
8..2
77.1
65.5 0.9
46.6 1.0
38,9 0.5
62.7
SelfReg (Kim al., 2021)
85.6 0.4
77.8 0.9
67.9 0.7
47.0 0.3
42.8 0.0
64.2
ERM (Vapnik, 1999)
83.1 0.9 0.9
+ 0.8
65.8 0.3
0,9
40.8 0.2
62.8
ERM + HSIC!
84.7 0.5
77.8
+: 0.8
67.3 1 0.4
45.4 0.4
40.6 0.1
63.2
Ours!
85.9 + 0.4
78.8 - 0.4
68.4 0.3
45.7 1.4
40.5 0.4
63.9
Results are listed in Table 10. We note that the independent constraint consistently improves the generalization performance as the basic form outperforms the baseline in most cases, and our method also obtains better performance than the baseline model and the basic form. These results illustrate the effectiveness of the independent constraint and the proposed augmentation strategy. Meanwhile, our method surpasses the baseline ERM model by 1.1 on average, while the reported top 3 methods (i.e. (Sun & Saenko, 2016), (Nam al., 2021), (Kim et al., 2021) lead their ERM model by 1.3, 0.9 and 0.9 in average. These results indicate that our method can also perform favorably against existing arts when implemented with ResNet50 pipeline.
C.2 EFFECTIVENESS OF THE MLP LAYER
We borrow the idea of using an additional MLP layer to prevent representation collapse from (Grill al., 2020). Here conduct an ablation study to examine if it is really necessary. To this end, we compare our method against a variant where the MLP layer disabled in Eq. (3) (i.e. Ours w/o MLP in L.en). As listed in the first row in Table 11, integrating the MLP layer bring improvements in most test domains, validating its effectiveness in improving the DG performance.
C.3 EFFECTIVENESS OF THE SUB LOSS TERMS IN Leen
In the manuscript, we expect the target branch to be insensitive and the domain branch to be sensitive regarding the augmentation operation. Thus the original and augmented features encouraged to
17

---

are set Under review as a conference paper at ICLR 2023
Table 11: Ablation studies regarding the effectiveness of the adopted MLP layer in sens and two sub
loss terms in Eeen (i.e. Lsin and Lary) Experiments are conducted in PACS (Li et al., 2017 with the leave-one-out training-test strategy.
model
art
cartoon
photo
sketch
avg
Ours w/o MLP in Lsen
80.5 + 0.4
75.1
94.2 1 0.3
77.7
1.3
81.9 0.4
Ours w/o Lsim and Lasf
79.3 = 0.4
75.4 = 0.9
94.5 0.2
78.3 = 1.7
81.9 - 0.5
Ours w/o Lasf
81.2 +0.5
74.9
0.6
94.6 + 0.4
77.7 1.0
82.1 + 0.7
Ours w/o Lsim Ours
80.8 + 0.6
76.0 + 0.4
+
77.9
1.4
82.2 + 0.4
81.3 + 1.4
75.7 0.2
94.6 0.3
78.1 + 0.7
82.4 + 0.4
Table 12: Evaluations regarding different syper-parameter (i.e. and [ in Eq. (5)) settings. We fix one parameter and tune another when conducting the experiments which are examined in PACS (Li al., 2017) with the leave-one-out training-test test strategy.
hyper-parameter parameters
art
cartoon
photo
sketch
avg
0.03
80.7 1 0.7
75.7 + 0.9
95.1
0.2
77.8 + 0.7
82.3 + 0.4
B 8=1 0.3
81.5
76.3 1.2
94.5
78.6 1.0
82.7 - 0.4 0.4
79.2 0.8
76.1
9.1
7
0.3
30
58.6
+
+.7
30.3 + 4.5
22.4+599
0.1
8.76
0.3
9.
82.0 0.2 0.7
8.1
7.2
94.5
1.0
22.7
a = 0.3
3 = 10
7.2 8..
9..6
0.9
81.7 0.4
8 100
0.8
76.8 0.7
9..1
75.9
81.8 0.2
3 1000
53.1
58.7 + 8.2
63.3 + 11.9
62.6 + 6.4
59.4 + 8.4
be similar in the target branch and different in the domain branch. Denoting the two sub loss terms as Saim and Laifs we have,
Lee = Deim + Laifs
(10)
where Ceim = - -LP(,))))((( + (MMP (I n] [I)MLP((a(f)) (a(A(f)) n|_
- (.(A())) and Las = [[]Oa(f)
To evaluate the effectiveness of Lsim and Laif. we conduct experiments in the PACS dataset by disabling one and keeping another Leen- Results are shown in Table 11. We observe that either one of the two terms can improve the performance, and the model performs the best when both two terms are integrated. These results indicate the effectiveness of the two sub loss terms.
C.4 SNSSTIVE TO THE HYPER-PARAMER
Our overall algorithm contains two hyper- syper-parameters (i.e. a and /3 in Eq. (5)), which are empirically fixed as 0.3 and / 1. To analyze the sensitivity of our model to these hyper-parameters, we conduct ablation studies by evaluating our model on PACS (Li et al., 2017) using different settings of them. Note we fix the value for Table 12. We observe that our method performs consistency well when the to E [0.03, 0.3] and / E [0.1,
when analyzing another. Results are listed in
C.5 MORE COMPARISONS WITH EXISTING AUGMENTATION STRATEGIES
This section further compares RDS augmentation strategy with other two methods that based on AdaIN (Huang & Belongie, 2017) (i.e. including MixStyle (Zhou et al., 2021) and DSU (Li et al., 2022)). First, for MixStyle, given a batch of training sample æ, the new style statistics Ωmiz and ômiz can be obtained by mixing the statistics of the original sample and their corresponding shuffled one:
Amis
Ay + A)ß, s.t. E Beta(w, w), w E (0, 0o).
(11)
where Beta(-) denotes the Beta distribution, and á compute the mean of the original batch and its shuffled version. A similar method is applied obtain another style statistic â 8mir- Second, for DSU,
18

---

Under review as a conference paper at ICLR 2023
(a) ERM
(b) MixStyle
(c) DSU
(d) RDS
Figure 4: Visualizations of original and augmented style statistics (concatenation of mean and variance). The PACS dataset Li et al. (2017) is used with art as the unseen domain during training. Augmented style statistics from MixStyle (Zhou et al., 202 1) and DSU (Li et al., 2022) can still reveal domain information, while our RDS can create diverse and inhomogeneous statistics.
ceroce
(a) ERM
(b) Proposed
Figure 5: Directly plot the 2d features from the penultimate layer. The PACS dataset (Li al., 2017) is used with art as the unseen target domain. The 2d features are rather close for both two methods, indicating that they are less representative compared to the original 512d version.
their new statistics Pdeu and ddsu obtained by adding perturbations to the original statistics:
Pdean = =1+ DMn, s.t. Φ E (0,
(12)
where Mn is the variance of (similar to our definition in Eq. (6)). The same approach is applied to obtain ddsu-
To better illustrate the differences between their approaches and RDS, we plot the original and augmented style statistics in Figure 4. As shown in Figure 4 (b) and (c), style statistics augmented from MixStyle and DSU can still reveal the domain information, indicating that their augmentations may create homogeneous features. In contrast, statistics from RDS are diverse and do not contain domain information, demonstrating that RDS obtains unlimited inhomogeneous features. These visualizations and the ablation studies in Table 3 can validate the effectiveness of our RDS against other augmentation strategies.
C.6 DIRECTLY PLOTTING THE 2D FEATURES
This section provides an alternate for visualizations of the features provided in Figure Instead of using t-SNE to reduce the dimension of representation from the penultimate layer, we include an extra MLP layer to reduce the dimension into directly and plot the results Figure 5, Compared to the visualizations in Figure 2, we observe that the features in Figure 5 are close, and it is rather hard to distinguish the class and domain information for both the ERM and our method, indicating that the features from the penultimate layer may contain similar information. We believe the main reason is that it will cost a lot of useful information to reduce the dimension the representation from the penultimate layer into 2 in multi-object classification task.
19

---

Under review as a conference paper at ICLR 2023
D DETAILED RESULTS OF THE BENCHMARK DATASETS
In this section, we present the average accuracy in each domain from different datasets. As shown in Table 13, 14, 15, 16, and 17, these results are detailed illustrations of the results in Table 2 in our manuscript. For all the experiments, we use the "training-domas method. A total of 22 methods are examined for 60 trials in each unseen domain, and all methods are trained with the leave-one-our strategy using the ResNet18 (He et al., 2016) backbones.
validate set" as the model selection
Table 13: Average accuracies on the PACS (Li et al., 2017) datasets using the default hyper hyper-paramen parameter settings in DomainBed (Gulrajani & Lopez-Pazz Paz, 2021). Time (min) here is the training time for one trial per GPU for the corresponding method.
cartoon
photo
sketch
Average
Time
ERM ( (Vapnik, 1999)
78.0 1 1,3 73,4 1.0.0008 94.1 0.4 73.6
1 2.2 79,8 1 0.4
24
ERM HSIC
79.3 0.9 74.1 20 94.8 1
0.5
75,8
1 1.6
81.0
0.6
25
IRM (Arjovsky et al., 2019)
76,9
75.1
94.3
0.4
77.4
0.4
80.9
18
GroupGRO (Sagawa et al., 2020)
77.7 76,4
94.0
14.8111
80.7
24
Mixup (Yan et al., 2020)
79.3 74.2
94,9
7.92
18
MLDG (Li et al., 8a)
78.4
75.1
948
76.7
81.3
32
CORAL & Saenko, 2016)
0.5
75.4
95.2
748
81.7
24
MMD et al., 2018b)
0.6
94,0
74.3
81.3
18
DANN (Ganin et al., 2016)
79.0
94,4
3.0
79.2 0.3
17
CDANN et 2018c)
80,4
93.1
1.7
80.3
0.5
24
MTL (Blanchurd et al., 2017)
0.6
73.4
74,4
80.1
0.8
18
SagNet (Nam et al., 2021)
0.4
73.2
94.
76.1
81.7 0.6
24
ARM (Zhang et al., 2020)
0.6
75.0
94.3
73.8
0.6
24
VREx Krueger et al. (2021)
7.74+
75.0
9.3
08.1
0.9
0.5
17
RSC (Huang et al., 2020)
7.1
73.3
016
1.
80.5
0.2
25
SelfReg (Kim et al., 2021)
0.8
74.4
95.4
74,9
81,8
0.3
25
MixStyle (Zhou al., 2021)
8..
76.3
94.2
77.5
82.6
0.4
25
Fish (Shi et al., 2021)
800
75.9
9,.0
76,2
1.0
82.0
0.3
52
SD (Pezeshki et al., 2021)
83. 74.6
0..3
94.6
75.1
1.6
+
25
CAD (Ruan et al., 2022)
83.9 74.2
24.2
0466
94.6
75.0
1.2
81.9
0.3
24
CondCAD (Ruan et al., 2022)
79.7 : 1.0 74.2
1:
94.6
74,8
1.4
80.8
+
0.5
26
Fishr (Rame al., 2022)
81.2 1 0.4
75,8
1
088 94.3
73.8
1
0.6
81.3
0.3
17
Ours
81.3 1.4
75.7
0.2
94.6
0.3
78.1
1
0.7
82.4
0.4
29
Table 14: Average accuracies on the VLCS (Fang et al., 2013) datasets using the default hyper- parameter settings in DomainBed (Gulrajani & Lopez-Paz. 2021).
Caltech
LabelMe
Sun
VOC
Average
ERM (Vapnik, 1999)
97.7 0.3
62.1 0.9
70.3
73.2 + 0.7
75.8
0.2
ERM + HSIC
97.0
62.3
70.1
75.1
0.7
76.1
:.1::00 0.1
IRM (Arjovsky al., 2019)
96.1 0.8
0.3
0.7
+
75.1 75.1
GrouPGRO (Sagawa al., 2020)
96.7
1.5
1.82
72.9
7 0.6
Mixup (Yan al., 2020)
1.5
6..7+0.4
0.3
75.4
0.3
ML.G (Li al., 2018a)
0.5
6.8
0.5
7.81
75.2
CORAL (Sun Saenko, 2016)
0.3
6.1
0.6
1.0
0.4
MMD (Li et al, 201
9.80
6.63
70.8 0.6
0.5
DANN (Ganin al., 2016)
9.1
6..
0.9
7.24+002
0.2
CDANN (L. et al., 2018c)
1.2
6.66
1.3
+.5
76.0 0.5
MTL (Blanchard et al., 2017)
9.34
6.6
0.6
75.2 1.3
SagNet (Nam al., 2021)
9.79
6.7
1.3
75.4 0.6
0.
ARM (Zhang et al., 2020)
9.5
6..9
7.66
7.4
0.3
VREx Krueger et al. (2021)
9.2
1.3
0.9
75.3 1..1
RSC (Huang et al., 2020)
962200
1.
1.0
7.40
75.4
SelfReg (Kim et al., 2021)
95.8
6.
7.6
0.6
0.7
MixStyle (Zhou et al., 2021)
97.3
6..610.1
7.74
71.3119
0.7
Fish (Shi al., 2021)
9.44022
6.410.1
0.4
7.7
7.2
(Pezeshki et al., 2021)
96.5
62.200.0
6.9
7.4
0.4
CAD (Ruan et al., 2022)
94
63.5
7..4112
1.
75.2
CondCAD (Ruan et al.. 2022)
9.5
62.64044
69.1
7..0
0.3
Fishr (Rame et al., 2022)
97.2
6.7
7..4
7..0
0.3
Ours
9.
0.
0.3
7.
0.4
20

---

Under review as a conference paper at ICLR 2023
Table 13: Average accuracies on the OfficeHome (Venkateswara et al., 2017) datasets using the
default apper-parameter settings in DomainBed (Gulrajani & Lopez- Pazz 2021).
art
clipart
product
real
Average
ERM (Vapnik, 1999)
52.2 0.2
48.7
0.5
71.7 0.3
60.6 1 0.2
ERM HSIC
53.1 1: 0.8
47.8
71.4 0.8
60.7 0 0.3
IRM (Arjovsky 2019)
49.7 0.2
46.8
6.4
5.1 0.6
GroupGRO (Sagawa al., 2020)
52.6 1.1
4.9
6.
7..5
0.3
Mixup (Yan al., 2020)
0.7
4.7
7.77
0.66
0.5
MLDG (Li al., 201
0.3
4..4
7.7
7..7
0.2
CORAL (Sun Saenko, 2016)
0.7
4..7
7.2
+3.1
0.4
MMD al.
1.0
4.3
69.3
70.7
0.4
DANN (Ganin al., 2016)
5.5
47.1
69.1
0.7
0.5
CDANN el, 2018c)
5.5
46.
0.5
7.44
0.4
MTL (Blanchard et 2017)
5.56
47.7
0.3
0.6
0.5
SagNet (Nam al., 2021)
55.3
49.6
7.
62.5 7.
ARM (Zhang et 2020)
5..3+
4..5+0.4
6.3
0.1
0.3
VREx Krueger et (2021)
5.31
4.6
0.
59.5 +.
RSC (Huang al., 2020)
0.10
4.
0.7
7.
0.6
SelfReg (Kim al., 2021)
0.81
0.6
0.3
62.4 7..0
MixStyle (Zhou et al., 2021)
0.6
1.1
1.
68.8
0.8
Fish (Shi al., 2021)
54.6
4.0
7.6
+
62.0
(Pezeshki al., 2021)
0.40
5.5
72.7
0.2
CAD (Ruan al,, 2022)
0.6
4.5
6.37
60.5 1.
CondCAD (Ruan al., 2022)
5.6
4..4
6.
0.6
6..0
Fishr (Rame et al., 2022)
5.
4.
6.
7244
6..
0.3
0.2
7.
70.3
0.11
Table 16: Average accuracies on the Terrainc (Beery et al., 2018) datasets using the default hyper- parameter settings in DomainBed (Gulrajani & Lopez-Paz, 2021).
L100
L38
L43
L46
Average
ERM (Vapnik, 1999)
42.1 + 2.5
30.1
+ 1.2
48.9 + 0.6
34.0 + 1.1
38.8 F 1.0
ERM HSIC
40.0 2.6
32.4 1.4
50.0
32.9
38.8 0.9
IRM (Arjovsky al., 2019)
41.8 1: 1.8
3.66
2.1
33.1
1.5
0.9
GroupGRO (Sagawa et al., 2020)
45.3 4.6
3..1
5.8
41.5 3.97
Mixup (Yan al., 2020)
49.4
1.8
5.7
42.1 30.0
MLDG (Li al., 2018a)
39.6
3372
0.5
15
40.1
CORAL (Sun Saenko, 2016)
46.7
36,9
1.9
41.4 3.7 1.8
MMD (Li et al., 2018b)
49.1
3.84
2.14
1.
42.0 1.0
DANN (Ganin al., 2016)
4..3
2.50
1.0
3.6
1.9
CDANN (Li 2018c)
3.4
32.
1.3
3.5
38.6112
MTL (Blanchard et al., 2017)
45.2
31.0
1.1
40.4 34.9
SagNet (Nam al., 2021)
36.3
40.3
0.6
40.6 1.
ARM (Zhang al., 2020)
4..
27.7
1.0
1.5
+
VREx Knueger et (2021)
1.70
41.1
1.5
32.0
43.2
RSC (Huang al., 2020)
42.8
32.2+38
49.6+0.9
1.2
1.3
SelfReg (Kim al., 2021)
46.1
1.6
0.3
34.7
41.3 0.3
MixStyle (Zhou et al., 2021)
1.9
28.0
52.1
3.2
4.
Fish (Shi al., 2021)
4.0
29.0
1.27
32.8 1.0
0.6
SD (Pezeshki
2021)
45.5
33.2+3.1
0.79
42,0 +.8
CAD (Ruan al., 2022)
43.1 2.6
31.1+199
1.6
34.7
+4.7+1.
40.5
+0.5 0.4
CondCAD (Ruan al., 2022)
44.4
32.922
5..5
30.8 008+0..
0.44
Fishr (Rame et al., 2022)
3.3
36
4.2
3..2
1.0
Ours
50.8
35.8
1.3
3.62
1.
21

---

Under review as a conference paper at ICLR 2023
Table 17: Average accuracies on the DomainNet (Peng al., 2019) datasets using the default hyper-
parameter settings in DomainBed (Gulrajani & Lopez-Pas 2021).
clip
info
paint
quick
real
sketch
Avernge
ERM (Vapnik, 1999)
50.4 + 14.0 0.2 40.3 0.5
11.7 4 0.2
52.0 0.2
43.
35.3 0.1
ERM HSIC
50
1.12
39
12.6
0.0
518
0.
IRM (Arjovsky et al., 2019)
0.9 12.
35.0
9,9
43.4
0,
30.4
GroupGRO (Sagawa et al., 2020)
38.2 13.0 1.33
28.7.0.3
0.1
43.4
Mixup (Yan et al, 2020)
4.39 13.6
0.55
10,9
49.9
022
MLDG (Li et 201
51.1 1..1 14.1
0.3
11.7
523
42..0..3.44
CORAI (Sun Saenko, 2016)
51.2++02 15.4
4200
02.7
52.0
0..
MMD al,, 2018b)
a (Ganin at.
16.6 13.3 0.3 00 128
12
128
104
0.0
17.1
137
04
7.9
45
0.
01
CDANN (Li et al, 2018c)
1.2 12.6 1
36.
10.3
47,5
0.1
0,
31.8 0.2
(Blanchard
2017)
SagNet (Nam al., 2021)
5. 14.6
40.
022
12.
0.2
51.5
5
0.3
35.3 0.1 033
424
ARM (Zhang ot al., 2020)
4. 01.7
34.6
98
0.4
43.2
03
299
VREx Krueger al. (2021)
39.2
1.4
31.2
10.2
1
41.5
1.8 34.8
0.8
28.1
1.0
RSC (Hunng et 2020)
39.5 11.4
08
30,5
31
10.2
D&
41.0
14
34.7
2,6 27.9
2.0
SalfReg (Kim
2021)
03 15.1
41.2
02
11.7
+ 0.3
48.8
00
438
34.7
0.2
MixStyle (Zhou et al. 2021)
4..1 00
39.3
0.0
11.4
04
47.7
03
427
33,9
0.1
Fish (Shi al., 2021)
03.5 14.5
40,4
03
11.7
02
0, 35.5 0.0 40 36.3 0.2
SD (Pezeshki ct al., 2021)
5.3
15.5
43
12.6
2
52.
440
et al., 2022)
349
1.6
CondCAD (Ruan el al., 2022)
46.1
13.3
14
10.7
0.2
11
46.8
38.7
31.9 0.7
(Rame al, 2022)
14.6
02
Ours
4.
14.7
4.8
014
49.1
0.
0.6
34.9
22


Reviews:
[{"review_id": "m6O70eShhMI", "reviewer": "Reviewer_VzKA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to solve Domain Generalization (DG) classification problem following the simple idea of decoupling the domain-invariant feature and domain-specific feature by applying an early branching strategy, a new independence measurement HSIC, and a new feature-level data augmentation method generating samples of new domain. Comprehensive experiments on several DG classification benchmarks demonstrate the effectiveness of the proposed method.  ", "summary_of_the_review": "The paper follows a simple idea has limited novelty. But the proposed methods make this simple idea more effective. Comprehensive experiments show the effectiveness of early branching and HISC measurement. This method brings new ideas or approaches to handle how to generate novel domain samples and combined it with previous method to form a feature level data augmentation method.", "strength_and_weaknesses": "Pros:\n1.\tThe logic of this paper is rigorous, and the structure is clear.\n2.\tThe ablation study shows that the HSIC measurement is a better choice compared to the orthogonal constraint and the correlation constraint.\n3.\tThe authors proposed a constrain to obtain inhomogeneous styles to synthesize new domain information.\n4.\tExperiments on benchmarks show the effectiveness of the proposed method.\n\nCons:\n1.\tThe authors try to demonstrate early branch architecture has a relatively better performance. However, in the comparing process of different branching location in the network, the number of parameters of shared base feature extractor and two branches are always varies. Thus, such a comparison cannot eliminate the number of parameters’ influence on the performance.\n2.\tAbout augments ablation study in Table 3. When the photo or carton is the target domain, the performance of RDS with proposed sampling scheme has no noticeable improvement compared to naive sampling without Eq. (7) , which should be discussed.\n3.\tSome new works are not discussed in the paper, e.g., Style Neophile: Constantly Seeking Novel Styles for Domain Generalization (CVPR 2022).\n4.\tThe best results in the experiment table should be in bold type to let others find the results easily. Also, the second-best results should be underlined. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "llapbWGjTSr", "reviewer": "Reviewer_sawe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an interesting study on disentangled representation learning for domain generalization. Technically, the main idea is to build a two-branch network, one for target classification while the other for domain classification. The paper uses carefully designed experiments to demonstrate that separating the two branches at an early stage of a neural network is more effective in learning disentangled representations.\n\nTo motivate the method design, the paper starts from a basic learning framework, which combines a classification loss with a correlation minimization loss (the latter is used for learning disentangled representations). Then, the paper motivates the use of domain augmentation (i.e., MixStyle) to facilitate the disentanglement, followed by the introduction of a modified domain augmentation strategy.\n\nThe main experiments are conducted on the DomainBed benchmark where the proposed method outperforms most baselines.", "summary_of_the_review": "I recommend to accept the paper because the method is simple, well-motivated and well-evaluated. The findings could be useful to future work on learning disentangled representations for domain generalization.\n\n== Post-rebuttal update ==\n\nI have read the rebuttal as well as other reviewers' comments. The authors have done a good job in addressing the questions I raised in the first-round review. My view on the novelty and significance of the paper remains the same: the paper should be accepted.", "strength_and_weaknesses": "**Strengths**\n\nFirst of all, the paper is well-written and well-organized, and both pros and cons of the proposed method are properly discussed in the paper. The idea of learning disentangled representations isn't new in domain generalization. But the study is novel and provides useful tips to the field. The entire framework—including the two-branch network, the correlation minimization loss, and the domain augmentation strategy—is well-motivated. The results clearly show that such a simple method works well in improving model generalization. The two measures, Top-5 and Score used in Table 5, deserve praise and could be of interest to the DomainBed community (where how to properly quantify progress remains an open question).\n\n**Weaknesses**\n\nOverall, there is no major issue that could lead to rejection of the paper. Only a few minor comments are given below, which the authors can use to further improve the paper.\n\n1. The proposed domain augmentation strategy in 3.3 is quite similar to the following work, \"Uncertainty Modeling for Out-of-Distribution Generalization\" published in ICLR'22, which has been cited in the paper. The current discussions in the end of 3.3 aren't very convincing as both methods are based on random sampling. Could the authors point out the major differences, perhaps using math equations?\n\n2. Have the authors tried other correlation minimization losses besides HSIC? Having more discussions about this could greatly help practitioners choose which loss to use.\n\n3. The Augmentation part in Sec. 2 misses those work using generative modeling to achieve data augmentation, e.g., [1, 2].\n\n4. The 2D t-SNE visualization isn't fully convincing because the spreading of the features could well be manipulated by using different parameters in the t-SNE method. Would it be possible to provide the following visualization: reduce the feature dimension to 2 in the last layer and plot the features directly on a 2D plane?\n\n[1] Zhou, K., Yang, Y., Hospedales, T., & Xiang, T. (2020, August). Learning to generate novel domains for domain generalization. In European conference on computer vision (pp. 561-578). Springer, Cham.\n\n[2] Carlucci, F. M., Russo, P., Tommasi, T., & Caputo, B. (2019, October). Hallucinating agnostic images to generalize across domains. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) (pp. 3227-3234). IEEE.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TTy8BaiZAS", "reviewer": "Reviewer_RNoa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a domain dual branch network for domain generalization. The first contribution is to empirically verify which layers should be shared between domain and target classification. Then, authors propose a new augmentation strategy for domain augmentation. Experiments using ResNet-18 demonstrates its effectiveness.", "summary_of_the_review": "This approach is on the right track: identifying the sharing and non-sharable layers between dual-branch networks. But the method is rather ad hoc and I don't think augmentation part is related to this paper. Plus, experimental results are not sufficient since only resnet18 is used.", "strength_and_weaknesses": "### Strength\n\n1. Identifying the sharing and separate layers for dual-branch networks is interesting idea.\n2. The paper is well-written.\n3. The experiments are sufficient, with good performance.\n\n### Weakness\n\n1. The approach is not novel. Empirically finding the sharing layers can not be regarded as a technical contribution. For the second contribution (new augmentation), it is novel, but still very naive.\n2. In page 4, authors spent too many contents explaining the initial results of identifying sharing layers, which I think is useless. It is just experimental observation. Plus, it might not be the case for other backbones like ResNet-50 and Vision transformer.\n3. With regards to backbones, authors used DomainNet, which is good, but they only experiment on ResNet-18, which is rather outdated these days since ResNet-50 is the standard and more popular in DomainBed.\n4. Identifying which layers to share is ad hoc. You should do everything again in face of a new backbone. This is the limitation of this work and cannot be called a contribution.\n5. It will be benfeicial to see the actual running time of this approach compared to others.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H1k53KOZG", "reviewer": "Reviewer_FWNq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a novel approach for domain generalization which relies on an early branching strategy to learn domain invariant features. They show that using HSIC as a domain invariant loss is most effective and also propose a new data augmentation strategy to simulate shifted domains. This work shows that this combination of design decision leads to state-of-the-art performance for domain generalization. While results clearly demonstrating the superiority of early branching for learning domain invariant representation would be quite consequential, the result presented in this work leave room for multiple interpretation as their is significant overlap in performance between branching at over blocks 0-3.", "summary_of_the_review": "This is an interesting work with components that might be useful for future work in domain generalization, a very important problem. \n\nHowever, presenting the results as combination of components makes its hard to distinguish if any individual component represents a meaningful and transferrable contribution. As the main goal is to improve the ability of machine learning model to generalize to unseen domains and not increase placement on leaderboards such as DomainBed its hard to determine the utility of this works contributions. \n\nMore detailed exploration of either Early Branching or RDS could greatly improve this work even if it doesn't yield a new state-of-the-art number.", "strength_and_weaknesses": "This paper presents a new approach for domain generalization through well reasoned design decisions and shows that the method outperforms the strong baseline of ERM and has the highest average accuracy amongst related methods. \n\nThe novel data augmentation strategy seems well reasoned and has strong potential, however it's unclear whether as a standalone it represents a useful contribution. Would be very beneficial to see alternate augmentation strategies combined with losses and branching strategies proposed in this work.  \n\nThe results of the ERM+HSIC represents a weakness of this work, it is the core formulation of proposed approach yet shows marginal benefits over the baseline ERM.\n\nA weakness with this work is their seems to be a conflation of hyperparameter search with the domain bed evaluation strategy, some parameters such as branching depth, alpha, and beta are chosen \"empirically\" in a manner that seems to invalidate the spirit of choosing hyperparameters over random trials in the validation pass over DomainBed. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject