Variational Disentanglement for Domain Generalization
Authors: Yufei Wang
Haoliang Li
Hao Cheng
Bihan Wen
Lap-Pui Chau
Alex Kot
Venue: Accepted by TMLR ()

Abstract: Domain generalization aims to learn a domain-invariant model that can generalize well to the unseen target domain. In this paper, based on the assumption that there exists an invariant feature mapping, we propose an evidence upper bound of the divergence between the category-specific feature and its invariant ground-truth using variational inference. To optimize this upper bound, we further propose an efficient Variational Disentanglement Network (VDN) that is capable of disentangling the domain-specific features and category-specific features (which generalize well to the unseen samples). Besides, the generated novel images from VDN are used to further improve the generalization ability.  We conduct extensive experiments to verify our method on three benchmarks, and both quantitative and qualitative results illustrate the effectiveness of our method.

Hao Cheng Published in Transactions on Machine Learning Research (08/2022)
Variational Disentanglement for Domain Generalization
Yufei Wang
yufe1001 ntu.edu..g
Nanyang Technological University
Haoliang Li
haoliang city..edu..k
City University of Hong Kong
hao006 @ @tw..ennuuu.....
Nanyang Technological University
Bihan Wen"
bihan. wen @ ntu.edu.sg
Nanyang Technological University
Lap-pui Chau The Hong Kong Polytechnic University
lap-pui. chuu @ polyu.eduuuuht edlu.k
Alex C. Kot
eackot edu.edu.sg
Nanyang Technological University
Reviewed on OpenReview:
Abstract
Domain generalization aims learn domain-invarial model that generalize well to the unseen target domain. In this paper, based on the assumption that there exists an invariant feature mapping, we propose an evidence upper bound of the divergence between the category-specino feature and its invariant ground-truth using variational inference. To optimize this upper bound, we further pro- pose an efficient Variational Disentanglement Network (VDN) that is capable of disentangling the domain-specific features and category-specife features (which generalize well to the unseen sam- ples). Besides, the generated novel images from VDN are used to further improve the generalization ability. We conduct extensive experiments verify our method three benchmarks, and both quantitative and qualitative results illustrate the effectiveness of our method.
Introduction
Nowadays, deep neural networks are widely used in numerous tasks and exhibit remarkable performance. However, their performance may degrade rapidly when the deployed environment is different from the training one. How to obtain domain invariant network that can generalize to the data collected from an unseen environment is always a research hotspot (Zhang et al. 2016, Kawaguchi et al., 2017).
Domain generalization (DG) aims tackle the generalization problem where the data from the target domain is inac- cessible. Generally, existing DG research works be categorized into three streams, invariant feature representation learning, meta-lcarning, and data augmentation. The objective of invariant feature representation learning is to extract shareable information across different domains, such, the learned model is expected to be better generalized to the unseen but related domain during evaluation. For example, Li et al. (2018) db aim to minimize the divergence of the latent features between different domains or the divergence with a pre-defined prior distribution. Dou et al. (2019) further propose to minimize the divergence the distributions conditioned on the category label. Although some desired performances have been reported through the aforementioned methods, they may face the risk of overfitting to the source domains without carefully specifying the invariant information learn, e.g., barely ensuring invariance may cause over-matching the observed domains (Shui et al. 2022). On the other hand, data augmentation based
*Corresponding author.

---

Published in Transactions on Machine Learning Research (08/2022)
category-specific features
painting
photo
cartoon
input samples
domain-specific features
generated samples
Given data from multiple source domains, we propose to learn a domain invariant embedding for classi- fication and a domain-specific embedding to encode the style information. The arrows from images to embeddings indicate the encoding process, and the arrows from features to the generated samples represent the generation process. Invariant category-specino features are learned using the proposed variational disentanglement framework.
Figure 1:
methods have also been proved to be effective in the domain generalization setting by enlarging the scale of training data with different augmentation strategies, e.g., data augmentation through adversarial training (Volpi et al., 2018; al. 2020cc). stacked transformation (Zhang et al. (0220), and Fourier transforma- tion (Xu et al. 202T). However, these augmentation strategies can be limited since they only focus on some specific types of data augmentation, such, the diversity of augmented data may be constrained.
Zhou al. (2200)), MixUp (Wang et al.,
In this paper, we propose to learn an invariant feature representation by matching the distribution of the feature space across domains to a distribution which is expected to represent the ground truth (i.e., invariant information) for the problem of domain generalization on a specific task. We first propose to derive the evidence upper bound of the divergence between the distribution of the feature space across domains and its unknown ground-truth distribution. To further optimize the upper bound, we develop an effective framework named Variational Disentanglement Network (VDN), which is capable of alleviating the aforementioned limitations jointly. More specifically, instead of optimizing the latent feature alone, we propose to optimize the calegory(lbbel--seeciie feature =. and domain-specific feature at the same time. The optimization of z, and zạ are twofold: minimizing the redundant information in the task-specific features and maximizing the posterior probability of the generated samples with random styles. By maximizing the posterior probability term, & and 2d are impelled to be disentangled and contain all necessary information to avoid a trivial solution that overfits to source domains. Meanwhile, minimizing our proposed information term that filters out redundant information in 2, can be regarded as the strategy of information bottleneck which benefits the generalization ability of the model (Alemi et al. 20166). Last but not the least, our VDN is also capable of generating different and diverse data across domains based on the specific task by swapping za across domains. The generated novel images can be further used to improve the generalization ability of the model, and are also compatible with existing augmentation-ben based methods. Extensive experiments are conducted on widely-used benchmarks, and both quantitative and qualitative results show the effectiveness of our method.
In summary, our contributions are as follows:
We provide a theoretical insight of how variational disentanglement can benefit the task of domain general- ization based on the basic assumption of the invariant feature representation learning.
We propose to derive a novel evidence upper bound of the divergence between the distribution of task-specific features and its invariant ground truth for domain generalization. Our proposed evidence upper bound further supports the rationality and build the tie between previous feature alignment- based methods (Li et al. 2018 8b
and disentangle-baset methods, e.g., Li et al. (017); Peng et al. (20195).
A novel framework is proposed to optimize the proposed evidence upper bound of the divergence. Extensive experiments demonstrate the effectiveness and superiority of our proposed method.
In al. 101 the author empirically proposes use adversarial autoencoder minimize divergence hetween ph pa where Ph. the latent code distribution and pæ the prior distribution (Laplace distribution the paper).
2

---

Published in Transactions on Machine Learning Research (08/2022)
2 Related works
2.1
Domain generalization
The core ideas of domain generalization (DG) are inherited from domain adaptation (Huang et al., 2006 Pan et al.
2011] Zhang et al., 2015: Ghifary et al., 2017: Blanchard et al. 202T) to some extent, e.g., they assume that there
exists something in common between different domains even if they can look quite different. For example, Khosla ; Seo et al. (2020) aim to jointly learn an unbiased model and a set of bias vectors for each domain, Yang & Gao (0013) use Canonical Correlation Analysis (CCA) to extract the shared feature. MMaandet et al. (0113
et al.
propose a domain invariant analysis method which used MMD and was further extended by [Li et al. (2018b). multi-
task autoencoders were also used by Ghifary et al. (2015) to learn shared feature extractor with multiple decoders.
2020 Wang et al. 20206), e.g., low- rank regularization (Xu et al., 2014 Li al. 2017 2020 Piratla et al. 2020). In addition, data augmentation based methods are also proved to be effective in the domain generalization setting, e.g. GAN generated samples (Volpi et al.
Moreover, various regularization methods of latent code are proposed (Zhao al.
2018 Zhou et al. 2020b), domain mixup (Wang et al.) 2020c /hhou et al. 2021a). stacked transformations (Zhang ct al. 2020). domain-guided perturbation direction (Shankar et al. 2018). Fourier augmentation (Xu ct al.
202 1). and
solving a jigsaw puzzle problem (Carlucci et al. 2019). Meta-learning based methods (Li et al. 2018a Balaji et al. 2018 Li et al. 2019b Dou et al., 2019 Du et al. 2020 are also explored by learning from episodes that simulate the domain gaps. Recently, invariant risk minimization (IRM) is proposed eliminate spurious correlations as such the models are expected to have better generalization performance (Arjovsky et al. 2019 Ahuja et al., 2020 [ellot & van der Schaar 2020 Zunino et al. 2020). As for feature disentanglement, current methods are usually based on decomposition (Li et al. 2017; Khosla et al. 2012; Piratla et | 20200] Chattopadhyay et al. 2020). Generation based disentanglement methods are also explored. For instance, |eng et al. (2019b) conduct the single domain generalization using adversarial disentangled auto-encoder and Wann et al. (2020a) provide a pair of encoders for disentangled features in each domain.
2.2 Feature disentanglement and image translation
Our proposed method is related to the feature disentanglement, which has also been widely adopted in the problem
of cross-domain learning (Bousmalis et al.] 2017: Hoffman et al., 2018 Russo et al.]
2018 Saito et al.) [2018). lot
of progress in domain generalization has been made by applying feature disentanglement (Peng et al. 2019b Khosla
et al. 2012 Li et al., 2017 Piratla et al.) 2020 For instance, Peng et al. (2019b) propose a domain agnostic learning method based on VAE and adversarial learning, and Khosla al.
2012 ; Li et al. (2077); Piratla et al. (2200) assume
that there exists shared model which is regarded as domain invariant and a set of domain-specific weights.
Our work is also related to the image translation (Isola et al. 2017 Choi et al., 2018: Zhu et al. 2017 Liu et al. 2019), which can be treated as conducting feature disentanglement by separating the style feature and content feature. Generally, the existing translation methods can be categorized into two streams, supervised/paires (Isola et al.) 2017) and unsuperviseddumpaire (Zhu al., 2017 Choi et al., 2018). While some progress has been achieved, most of the existing models require large amount of data. Recently, some efforts have been made improve the capability of the translation model by utilizing few samples (Liu at. al. 2019: Saito et al. 2020). Specifically, the framework we propose conducts translation tasks in a more efficient manner given limited and diverse data. This helps us obtain more accurate posterior probability estimation. In addition, the generated high-quality samples with different combinations of image attributes (e.g., samples with a new combination of angles, shape, and color (Zhao et al. 2018)) based on our proposed framework are also used for data augmentation purpose for better generalization capability.
3 Methodology
3.1 Preliminary
Assume we observe K domains and there are N; labeled samples {(r,,,,) 11), in the domain D,. X, is the image set of the domain D, and Y the set of labels. We first introduce the assumption regarding the invariant feature representation for the task of DG.
Assumption 1 (Learnability) Denote the dimension of the image x and the latent feature ze by d, and do, respectively. let de: dd I.2 (Rdde) be the mapping between an image x and its probability density function (PDF) [zelar), where
3

---

Published in Transactions on Machine Learning Research (08/2022)
L' denotes 1-norm integrable function space. A deterministic mapping 0g RR. - N acts classifier to predict the category of the image based on its latent feature zc. For domain generalization problem with n, source domains (X, = X1 U X2... U X and a target domain Xt. we say it learnable if
Bde s.l. EawPray [pc(x)] = E [oc()) [(ow))]=ewP,,, Y, f1,2...., nz,t)
(1)
(g(((((x))) = = E EX,Ux
where Paaly represents the conditional distribution of images with the category y in the domain i, and ((b((z))
=
(x) Ze.
While we assume noiseless case similar to Tachet des Combes et al. (2020), such an assumption is mild and is commonly adopted in the community of domain generalization (Muandet et al. 2013). Based on the assumption that if the task is feasible, there should exist a domain invariant mapping 6.(.). In other words, we can improve the generalization capability of the classifier model if we can find the suggested Kingma & Welling (013), the invariant ground-trut distribution P(ze(z) (where P(ze|=) = (.(x)) can be intractable, as such, we are unable obtain an explicit form of P(zelz). To this end, we propose use variational inference to find an approximation of (-(-))
mapping Φe- However, as
3.2 Motivation
To approximate the conditional distribution of domain-invariant features des propose to minimize the evidence upper bound of the KL divergence [[((ze denotes the feature distribution obtained from the category-specific encoder E. and P(ze(z) is the invariant ground truth distribution. Since we aim to find deterministic classifier, we set Ee(x) = (Q (x) for simplicity. We now introduce the rationality of learning invariant feature representation by minimizing the upper bound of KL divergence. Due to the limited space, proofs of Theorem and and additional details are placed in the Appendix.
derived by Bayes variational inference, where Q (x)
Lemma 1 The KL divergence D(Q(zc)=)IIP(<eH))) can be represented
D(Q(Zel=)]IP((z.) Ez (ogp(r + p(x),
where p(2]2e)) is the PDF of the distribution P(x|2e). Based Lemma 1, derive the evidence upper bound of D(Q(z]]=))]I((<]cc)))
Theorem 1 The evidence upper bound of KL divergence D(Q(zcl=))IP(zelz))) between the distribution Qzzelz) and the ground-truth P(zelæ)) is as follows:
Ezo ((g + C,
O information gain term
posterior probubiliry
where C is constant, [Zey za] and Pa can be an arbitrary prior distribution.
It is worth noting that 2e and 2d are implicitly determined by the category and domain label respectively, i.c., only the realistic image accord with its corresponding category and domain could have a high probability density value. We further show that the performance in the unseen target domain can also benefited from the optimization of the given upper bound in source domains.
SSiilar to other DG works (Li et al. 2018d Hu et al.
2020). there is implicit assumption that the label prior does not vary lot among
source and target domains. The cases that label priors are largely different in the scope of imbalanced classification (Sun al. 2009) and
heterogencous domain generalization (Wang et al. 2020cc Li al. 2019b). In addition, thcoretical analysis can be found in (huui al. (021 which demonstrates that the upper bound of target risk related the discrepuncy of label distribution.
3 For consistency, represent distributions using upper case letter, e.g., Q(ze(ar) means the distribution of random variahle conditioned and it also abbreviated Q 20 The lower represent the exact prohahility density value, e e-g. pc.
4

---

Published in Transactions on Machine Learning Research (08/2022)
cici
Lrec
Zei
Ep
Et
Ltask
Eg
dd
Ed
Zdj
G
GZz Zag
D
L gan
NO0.1)
D
Lreg
Figure 2: Framework of our method for training. For a random sample (æs, Bi) in the training set, we random select another sample (zz, y2) that does not require the same category. We feed the two samples into the task task-specific encoder E. and domain-specific encoder Eg and get the corresponding code 2c. Zeg and 2d, =d,- Then we use the generator G to obtain the reconstructed samples G((ze, 2d,) and samples with random style ( discriminator loss are used to enhance the quality of generated samples which further guarantee good disentanglement between class-specific feature and Cooainn-specitee feature 2d- In addition, for the task-specific feature 2e. it is supervised by the specific task, e.g., classification, and regularized by the information gain term D(Q(Zel<))|
The perceptual loss and
Assumption 2 Given sample from the target domain Xt, there exists a non-empty feasible set which is defined as
I= (1(a(zol=") S Σ Big(zzola) Vz. [R - Vi
where I is the index set, x denotes an arbitrary sample with index i in any source domains, and g(zelar) is the probability density function value of conditioned x² from distribution Q(zelz').
Assumption 2 is mild assumption since it holds as long the task is feasible (the second set is not empty) and ((Felx)) is distribution that satisfies ((=e|z)> > 0 given any ãc, e.g., Gaussian distribution (the first set is not empty).
Theorem 2 Based Assumption 2 the KL divergence between Q(zelze) and the unknown domain- ground truth distribution P P(Zela?) can be hounded as follows
D(Q((elz')))) ((Q(ze)")llP(zael)))) inf Σ
IE1
IEI
The above theorem demonstrates that the KL divergence between ((ze/z) and P(ze x) from source domains con- stitutes the divergence upper bound in the unseen target domain. Therefore, further supports the rationale and effectiveness of our method.
3.3
Optimization strategies
In this section, we introduce the overall framework and the optimization details each term in our proposed evidence upper bound.
3.3.1 Overall framework
The whole framework of our proposed method is illustrated in Fig. which consists of task-specific encoder E., domain-speciffe encoder Eg. generator G, discriminator D3 to distinguish real and generated images, a discrimi- nator D. to justify whether the task-specific feature comes from a predefined distribution, and a task-specific network E, for classification purpose. The overall optimization objective given as
L Xreg
Lposterior
(2)
To minimize
To minimize 3
5

---

Published in Transactions on Machine Learning Research (08/2022)
where the first term Lreg is defined follows
Live = Eza-Pa, Ps.. [Da(zo]] + Ba~x
+1,
(3)
which acts a regularization term that aligns the task-specific feature to a predefined distribution and is further interpreted in section 3.3.2 and poostriir is defined as
ppoostrioo Ltask + Avec Tree Agan I Legan:
(4)
where the definitions of each term in posterior are defined as follows
Ltask = Ex Mlack Eg Ex(x), Es Basauusaaullaasl Ec(G(ze, 2a)).
Lyan = Ez~x (D2 =a]] - -u
Lyee Ex=x E, E,) Ea
The details of each term in Lposterior are illustrated in section 3.3.3 These losses are optimized jointly to disentangle the saec-fpe feature &e and domain-specific feature 2d-
In the test phase, image from unseen but related target domain is passed through the task-specific encoder E, and the task-specific network E,, and the other networks will not be involved. Therefore, network is supposed to have the same inference complexity as the vanilla model.
3.3.2 Estimation of the information gain term
To reduce the risk of learned task-specific feature overfitting to the source domains, similar Li et al. (2018)). we minimize the divergence between Q( (z:) which is introduced by the task-specific encoder Ec, and a predefined prior distribution P(ze). The divergence D(Q [ze) (x) || P(Ze)) can be interpreted as the information gain by introducing specific image Ideally, the optimal features only contain the necessary information for classification, and domain- specific information will be ruled out. To this end, we minimize the task-specific loss, i.e., classification loss in equa- tion 12, and information gain term in our framework simultaneously. For the minimization of information gain term, we empirically find that directly optimizing this term using the reparameterization trick (Kingma Welling 2013) may not be feasible due to the high dimensionality of Ze. To this end, we propose optimize ((Q P(ze)) which is equivalent to optimize the upper bound of the information gain term Lreg in Theorem []] The proof is in Lemmal A.2 and the optimization of the proposed upper bound also supports the core idea of the previous work (Li et al. 201 8b) that aligns the latent feature to a pre-defined distribution. More specifically, we optimize its dual form following the core idea of f-GAN (Nowozin et al. 2016) given as
D(Q(Z))))P(E.)) (p(((z))))( fale.)
a(tx)) sup (plze) ) dza.
(5)
((zo)
(Edomy
where p(-) and q(-) are PDFs of P and Q respectively, f(u) = log(u). and (*) denotes its Fenchel conjugate (Hiriart-Umin & Lemaréchal 2004) which is defined as
(1) = sup uEdomy
{ul - [(u)] = - - log(
(6)
One can replace t in equation 6 using an arbitrary class of functions T. as such, the term ① can be represented
D(Q((z)))P(ze)) = sup (Ero- [T(z,)] Ezo-Quat 1 (T(z)))),
TET
if the capacity of T is large enough.
To optimize equation an adversarial training manner is used. Specifically, we optimize the task-specific encoder E. to maximize this term and the discriminator D. to minimize it. The final goal of the training for the regularization term is as follows
min max Lreg = Eg. P., [D(2e)] + Ex-x [bg() E.
(8)
where the function T in equation 7 implemented by a deep neural network D. with the activation function gy(v) exp(-u) the output retain the original domain domy-.

---

Published in Transactions on Machine Learning Research (08/2022)
Algorithm 1 Variational Disentanglement for domain generalization.
Input: X = { Y = {yi Ji= initialized parameters G, De, Dzs Ecs Ea and Er.
Output: Learned parameters G Dr Dr Ee Eg and E,
1: while Stopping criterion is not met do
2: Sample a minibatch X and y from X and Y respectively.
3: Calculate the task-specific feature Ze (x) ) and domain-specific feature Za = Ex(X).
Shuffle the domain-specific feature Zed = shuffle(Zz) (a) Generate the reconstructed samples Xrec G(Ze, Zu) and the samples with the random style Xrand
G(Ze, Za).
6: Calculate the weighted loss Lgen Lrask + Xveg breg + Aaan Ligan + Avee AreeLree Compute the gradient of Lgen w.r.t G. Egs, Eg and E, and then do the update. if Update frequency is met then Calculate the weighted loss Ldss (NregEreg + Agan
7:
8:
9:
10: Compute the gradient of Leiis w.r.t D. and D2 and then do the update.
11: end if
12:
end while
3.3.3 Estimation of the posterior probability term
For better disentanglement, we propose to maximize the posterior probability term p(z| za). The advantages of maximizing this term can be roughly summarized into two points: first, directly minimizing the information gain term and task-specific term may cause the overfitting to the training samples. For instance, when the dataset is not large enough, directly memorizing all the datasets may have less information gain comparing with extracting discriminate features for classification. By maximizing the posterior probability, we can guarantee that 2e and zd together contain almost all the information of the image which avoid the loss of discriminative features caused by minimizing the in- formation gain term. Second, by improving the quality of generated samples using random combined task-specific feature ,e and style feature zd, the task-specific and domain-specifie features will be disentangled since our discrimi- nator can not only distinguish real and fake images but can also differentiate whether the generated samples are from the specific domains thanks to the domain labels.
maximize the posterior probability term, two obstacles need to solved. First, an efficient sampling strategy is needed on account that the space of 2e and zu large and intractable and it not feasible sample ergodically from them. To this end, inspired by VAE (Kingma & Welling 2013), we propose adopt the task-specific encoder E. and domain-specific encoder Eg instead, i.e., Qz (X) and Qsa Ex(X), to conduct code sampling of and zu which are most likely to generate a realistic sample. To sample in Qe and Qa independently, we shuffle the domain-specific features in a batch and ensure the generated samples G( (d,)) using randomly combined features as realistic as possible. More details can be found in Algorithm !
Unlike VAE (Kinmma & Welling, 2013) which only computes the reconstruction term p(æl =d,), we also need to compute the probability of the generated one G(ze, zd;) using the generator G for Vi + without a corresponding ground-truth. To this end, we estimate the probability p(z|ze: zd) using the following equations
La(z]G(ze, 22), B)) (C(zz. (a =d)
w/ ground-truth
p(alze, za)
w/o ground-truth
where La denotes the PDF of Laplace distribution (corresponding to the LI norm) to measure the pixel reconstruction loss, and (- za) is the estimation from the discriminator based on the corresponding category and domain la- bels/features. To optimize the term in equation Dr needs to have the capability to distinguish between the real and generated images G( =d,) through random combined latent features, and differentiate whether the samples have the desired domain and category. Meanwhile, the generator G is required to produce realistic outputs. To this end, we train the model in an adversarial manner given below
min max = Ex~x Dz 24)l - L. E., Dz
24)]zo; za)].
(10)
For the reconstructed images with ground-truth we empirically find that directly minimizing the pixel level divergence can lead to performance degradation. To this end, we minimize the divergence of semantic features through pre-
7

---

Published in Transactions on Machine Learning Research (08/2022)
&
photo painting
painting photo
(c)
sketch
sketch painting
Figure 3: The generated samples in source domains. The images in the first row are the input images that we want keep the category information and in the last row are the input images that provide the style information. The second row is the reconstructed one and the third row is the translated one.
trained perceptual network Ep instead. Therefore, we maximize the improved version La( (e) =a)). by minimizing its corresponding L1 loss given as
min Lree = Exex (,, E.,Ea,G
(11)
Last but not least, to further enforce the task-specific encoder E, to generate more meaningful embeddings, we use label information to guide the model training by minimizing the following equation
min Leask = Ex-x (rast Ec(x), v)l + Ez, Eso-Qnssrqallaat baask o ((())z))))) Ec,Et
(12)
where y denotes the ground-trune -ruth label of the input x or its task-specific feature Ze, lask is the task-specific loss, e.g.. cross entropy is used in our work for classification tasks. Besides, we also consider the generated samples for data augmentation purpose, which is the second term of equation 12
4 Experiments
We evaluate our method using three benchmarks including Digits-DG (Zhou et al. 2020b), PACS (Li al. 2017) and mini-Domaine (Zhou et al. 2020c; Peng al. (2019)). For the saper-parameters, 0.1, 1.0 and 1.0, respectively, for all experiments. The network architectures and training details are illustrated in the supplementary materials. We report the accuracy of the model at the end of the training.
set Xreg- Arec and Agan
4.1 Digits-DG
Digits-DG (Zhou et al. 2020b) is benchmark composed of MNIST. MNIST-M, SVHN, and SYN where the font and background of the samples are diverse. Except that we additionally use the Fourier-based data augmentation (Xu et al.) 2021), we follow the experiment setting in Zhou et al. (2020b). More specifically, the input images are resized to 32x32 based on RGB channels. The ConvNet used as the backbone for all methods and is divided into two parts: the task-specific encoder F,, which includes the first three conv blocks except for the last max-poolin max-pooling layer and the rest are regarded as the task-specific network E,. For the E, and E,, we use the same learning parameters with Zhou et al. (2020b). SGD is used as the optimizer with an initial learning rate of 0.05 and weight decay of 5e-5. For the rest of the networks, initial learning rate of 5e-5 and the same weight decay. We train the model for 60 epochs using the batch size of 128 and all the learning rate is decayed by 0.1 at the 50th epoch.
Settings:
RmsProp without momentum the optimizer with the
We repeat the experiment times and report the average accuracy in Table ! Our method has about 10% percent improvements MNIST-M, SVHN, and SYN domains compared with the DeepAll method. addition, the results demonstrate that our method achieves the best overall performance, especially the domain of MNIST-M with about 3% improvement compared with other competitors.
Results:
8

---

Published in Transactions on Machine Learning Research (08/2022)
Table I: Evaluation of DG on the Digits-DG benchmark. The average target domain accuracy are reported.
Method
Reference
MNIST MNIST-M
SVHN
SYN
Avg.
DeepAll
95.8
58.8
61.7
78.6
73.7
MMD-AAE [L et al. 2018a
CVPR 2018
96.5
58.4
65.0
78.4
74.6
CrossGrad (Shankar et al. 2018
ICLR 2018
96.7
61.1
65.3
80.2
75.8
DDAIG (Zhou et al. 2020b)
AAAI 2020
96.6
64.1
68.6
81.0
77.6
L2A-OT (Zhou et al. 2020a
ECCV 2020
96.7
63.9
68.6
83.2
78.1
MixStyle Zhou et al. 2021a)
ICLR 2021
96.5
63.5
64.7
81.2
76.5
FACT (Xu al. 2021
CVPR 2021
97.9
65.6
72.4
90.3
81.5
VDN
Ours
97.6
68.1
72.9
87.6
81.6
Table 2: Evaluation of DG on the PACS benchmark. The average target domain accuracy of five repeated experiments is reported.
Method
Reference
Art
Cartoon
Photo
Sketch
Avg.
DeepAll
77.0
75.9
95.5
70.3
79.5
MASF (Dou et al. 2019)
NIPS 2019
80.3
77.2
93.9
71.7
81.0
Epi-FCR (Li ct al. 2019a
ICCV 2019
82.1
77.0
93,9
73.0
81.5
L2A-OT Zhou et al. 2020aa
ECCV 2020
83.3
78.2
96.2
73.6
82.8
RSC Huang et al. 2020
ECCV 2020
78.9
76.9
94.1
76.8
81.7
MixStyle (Zhou et
2021a)
ICLR 2021
84.1
78.8
96.1
75.9
83.7
DAML (Shu al. 202
CVPR 2021
83.0
74.1
95.6
78.1
82.7
FACT Xu et al. 2021
CVPR 2021
85.4
78.4
95.2
79.2
84.5
DSU ct al. 2022)
ICLR 2022
83.6
79.6
95.8
77.6
84.1
VDN
Ours
84.3
79.8
94.6
82.8
85.4
4.2 PACS
Settings: PACS (Li et al. 2017) is benchmark for domain generalization task collected from four different do- mains: photo, painting, sketch, and cartoon with relatively large domain gaps. Following the widely used setting in Carlucci al. (2019), only used the official split of the training
train the model and all the images from the target domain used for the test. RmsProp is used to train all the networks with initial learning rate of 5e-5 without momentum and decrease the learning rate by a factor of 10 at the 50th epoch. The batch size is set to 24 and we sample the same number of images from each domain at the training phase. All the images are cropped to 224x 224 for training and the data augmentations including random crop with a scale factor 1.25, amplitude mix (Xu et al. 2021). and random horizontal flip. Other augmentations such as random grayscale are not used as may conceal the true performance of the model by introducing prior knowledge of the target domain. More specifically, the part from the beginning the second residual block inclusive is regarded the task-specific encoder Eg,, and the remaining part acts as the task task-specific network E,. The discriminators D. and D2 updated once after every 5 updates of other parts in the framework.
Results: The results based on Resnet-18 are reported in Table 2 As we can observe, our method outperforms other ofute-of-the-an methods. Moreover, we observe that we can achieve much better performance in the sketch domain in large margin compared with other baseline methods. We conjecture the reason that the Sketch domain may contain less domain-specifte information. As shown in Fig. 4 shuffling domain-specific features has less impact on image generation in the domain of Sketch. Due limited space, the results based Resnet-50 reported in the Appendix.
4.3
mini-DomainNet
Settings: We then consider larger benchmark nini-DomainNet (Zhou et al. (020c). which a subset of Domain- Net (Peng et al. 2019a) without noisy labels. for evaluation purpose. In mini DomainNet, there are more than 140k images in total belonging 4 different domains, namely sketch, real, clipart, and painting.

---

Published in Transactions on Machine Learning Research (08/2022)
Table 3: Evaluation of DG on mini-DomainNet benchmark. We repeat the experiments three times and report the average accuracy in the unseen target domain in the table.
Method
Reference
Clipart
Real
Painting
Sketch
Avg.
DeepAll
62.86
58.73
47.94
43.02
53.14
MLDG
(Li et al.
2018a)
AAAI 2018
63.54
59.49
48.68
43,41
53.78
JiGen Carlucci et al. 2019
CVPR 2019
63.84
58.80
48.40
44.26
53.83
MASF Dou
2019>
NIPS 2019
63.05
59.22
48.34
43.67
53.58
RSC (Huag al.
2020)
ECCV 2020
64.65
59.37
46.71
42.38
53.94
DSU
Li ct al.
2022
ICLR 2022
63.17
56.03
47.46
47.14
53.45
VDN
Ours
65.08
59.05
48.89
49.21
55.56
For the task-specific encoder Eg and task-specific network E,, we use SGD with momentum of 0.9 and an initial learning rate of 0.005 as the optimizer. For other parts of our framework, RmsProp without momentum is used with the initial learning rate of 0.0001. For the learning rate scheduler towards all the optimizers, we use the same cosine annealing rule (Loshchilov & Hutter, 2016) with the minimum learning rate after 100 epochs. The batch size is 128 with random sampler that roughly samples the same number of images in each domain. We consider data augmentations including the random clip with probability of 0.5, and random crop the data the size 96 x 96 using the scale factor of 1.25. For fair comparison, the division of E, and E, for our model is the same as the setting in PACS. The update frequency of De and D., is the same with PACS that discriminators update once after every 5 updates of other parts in the framework.
the same backbone network Resnet- 18 for all the methods and
Results: We compare our method with MLDG (Li et al. 2018a). JiGen (Carlucei et al., 2019), MASF (Dou et al. 2019). RSC (Huang et al.) 2020), and DSU (Li al. 2022). The results are shown in Table As we can observe, large margin,
we can achieve better performance compared with other baselines, especially in the sketch domain There is interesting finding that similar to the PACS benchmark, the performance improvement in the sketch domain is huge, but in the real domain, the performance of our method has some degradation. This may reveal the potential inductive bias of the model. However, our method still has the best overall performance.
4.4 Ablation study and perceptual results
In this section, we first present the results of the ablation study to illustrate the effectiveness of each component in our proposed method. We further provide some perceptual results image generation to show the significance of feature disentanglement.
4.4.1 Ablation study
We conduct the ablation study using the PACS benchmark. We first explore the effectiveness of each term in the evidence upper bound we proposed in Theorem [ Then, the impacts of optimization strategies for each term are evaluated.
Effectiveness of each term: We first evaluate the effectiveness of optimizing each term we proposed in the EUB, The results are shown in Table 4 where '+0' and '+2' represent that we utilize the information gain term @ defined in equation and the posterior probability term @ defined equation 4 but without data augmentation respectively. '+DAug' means we use the generated samples from our generator G for data augmentation, and '+FAug' represents that we adopt Fourier-based data augmentation strategy (Xu et al., 2021) to do the data augmentation. The results demonstrate that both the information gain term and posterior probability term can improve the generalization capa- bility of the model. In addition, combining these two together can attain larger performance improvements. The results also demonstrate that using the generated samples augmented data can effectively improve the generalization ability of the model.
Impacts of different optimization strategies: As for the evidence upper bound we proposed, there are different optimization strategies for each term. For instance, reparameterization trick (Kingma & Welling (003)) widely used method to optimize the term ①. In addition, directly optimizing the LI reconstruction loss usual way to generate sharp reconstructed images. We investigate the impacts of different optimization strategies and the results are
10

---

Published in Transactions on Machine Learning Research (08/2022)
Table 4: Ablation study regarding the effectiveness of each term in the evidence upper bound we proposed. The first row the DeepAll baseline and the last is the complete version of our method.
+D
+DAug
+FAug
Art
Cartoon
Photo
Sketch
Avg.
77.0
75.9
95.5
70.3
79.5
77.6
77.8
93.5
71.8
80.2
79.7
78.4
94.3
75.2
81.9
81.5
78.2
93.8
77.6
82.8
82.6
78.5
94.0
82.7
84.5
v
84.3
79.8
94.6
82.8
85.4
(a) painting
(b) cartoon
(c) photo
(d) sketch
Figure 4:
The generated samples in the unseen target domains. The rows from up down denote the input image provide the task-specific features, reconstructed images, transformed images, and the images that provide the style features, respectively. Note that the input images all come from the unknown target domain.
shown in Table 3 where the Vone the strategy we adopt. As we can see, directly using the reparameterization trick to align the high-dimensional features can lead to side effects. addition, optimizing the perceptual loss can lead to better performance compared with replacing the reconstruction loss in equation I1 with LI loss.
Table 5: Ablation study regarding the effects of different optimization strategies. 'V: the strategy we use the paper. 'reparam's the reparameterization trick (Kingma & Welling 2013). 'Ll loss': replacing the reconstruction loss in equation II with LI loss.
+0
+
Art
Cartoon
Photo
Sketch
Avg.
77.0
75.9
95.5
70.3
79.5
reparam
74.4
77.6
92.4
68.3
78.2
77.6
77.8
93.5
71.8
80.2
L1 loss
78.2
78.3
93.9
73.4
81.0
79.7
78.4
94.3
75.2
81.9
4.4.2 Perceptual results
To provide intuitive way understand the effect of disentangling, we further give some perceptual results. For limited space, more visualization results are placed in the supplementary materials.
Generated samples based on source-domain images: To demonstrate the effectiveness of our method, we first visualize the generated samples in a cross-domain setting that the pairs of input samples are from different source domains on account that the quality of the samples can reflect the accuracy of the estimated posterior probability and the degree of disentanglement. Some of the generated samples are shown in Fig. The visualization results demonstrate that our method can disentangle the domain dommiin-spccific features and task-specific features well and generate realistic novel samples with high quality and different styles. In addition, observe that the reconstructed samples

---

Published in Transactions on Machine Learning Research (08/2022)
divtt puts
(a) w/o domain label
(b) w/o category label
(c) w/o category label
(d) w/o domain Jabel
Figure 5:
The unsupervised t-SNE visualization results of extracted features from our model. The features in (a.b) are collected from the domain-specific encoder Ed and features in (c,d) are collected from the task specific encoder Ee. PACS benchmark used for visualization and photo domain is selected the target domain. (Best viewed in color.)
may not necessarily be the same as the original one, mainly due to the perceptual loss we adopt, as such, we can prevent the latent features from overfitting to the source domain.
To further demonstrate the effectiveness of our proposed framework, conduct image generation based on the unseen target domain, where the generated samples using the pairs from the same unseen target domain are shown in Fig. based on leave-one-domainonn training manner. From the visualization results, we find that our method can still separate the task-specific features and domain-specific features well even if the networks have never seen the samples from the target domain. More specifically, it can encode the intra-domain style variance based on the observation that the model can generate samples with different styles using the Nomain--pecific features from the same target domain. Meanwhile, the results in Fig. 4 also illustrate that the sketch domain may have little domain-specific features and intra-domain style variance on account that the translated and reconstructed samples are almost the same. This observation further demonstrates the effectiveness of our proposed method by using Sketch the target domain where significant performance improvement can be achieved.
Generated samples based on target-domain images:
4.4.3 The visualization of feature embeddings
In addition to the perceptual results of generated samples, we also utilize t-SNE (Van der Maaten & Hinton 2008 to conduct analysis on the features extracted from two diffcrent branches Eg and Eg using our framework by using PACS where Photo treated as the target domain. The visualization results are shown Fig. and several findings are observed: in Fig. [(a))and and [bb] the features from source domains extracted by Ea can be well separated terms of domain information but cannot be separated in terms of category information. On the contrary, Fig. and F. can separate the features in terms of the category information instead of the domain information, which shows the effectiveness of our disentanglement framework.
can observe from
5 Conclusion
In this paper, we propose to tackle the problem of domain generalization from the perspective of variational disen- tangling. Specifically, we first provide an evidence upper bound regarding the divergence between the distribution of category-specific feature and its invariant ground-truth through Bayes variational inference. Then, we propose an effi- cient framework to optimize the proposed evidence upper bound for domain generalization. Extensive experiments are conducted verify the significance of our proposed method. Besides, conduct experiments of image generation which further justify the effectiveness of our proposed disentanglement framework.
Limitations: While we have the same memory usage and speed for inference, our method may increase the training overhead since the existence of auxiliary networks. In addition, due to the instability of GANs, the tuning of the hyper- parameters may need some experience and effort. Since our method can be regarded as an unpaired image translation network, it has requirements on the data amount, e.g., too many domains with limited data each domain may be challenging for our method. Besides, though the information term our derived evidence upper bound can be exactly calculated, utilize its substitute given the practical concerns. This could be further improved in the future.
12

---

Published in Transactions on Machine Learning Research (08/2022)
Broader Impact:
Our proposed method shows the potential in improving the generalization ability of models. It can be used to alleviate the side effect of data bias, e.g., medical images collected from different devices and areas. In addition, our proposed evidence upper bound could give insights to further works. The generated samples with mixing styles and novel characteristics could mitigate stereotypes. However, like other image generation models, our work may also face the risk of being used to generate malicious samples.
Acknowledgments
This work was carried out at the Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University, Singa- pore. This research is supported in part by the Ministry of Education, Republic of Singapore, under its Start-up Grant. This work is also supported by the Research Grant Council (RGC) of Hong Kong through Early Career Scheme (ECS) under the Grant 21200522 and CityU Applied Research Grant (ARG) 9667244.
References
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In International Conference Machine Learning, pp. 145-155. PMLR, 2020.
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 2016.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893; 2019.
Yogesh Balaji et al. Metareg: Towards domain generalization using meta-regguarizzation. In NuerlPS, pp. 90081008. 2018.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi. Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018.
Alexis Bellot and Mihaela van der Schaar. Accounting for unobserved confounding in domain generalization. arXiv preprint arXiv:2007.10653; 2020.
Gilles Blanchard, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. J. Mach. Learn. Res., 22:2-1, 2021.
Francesco Cappio Borlino et al. Rethinking domain generalization baselines. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9227-9233. IEEE, 2021.
Konstantinos Bousmalis, Nathan Silberman, David Dohan, Dumitru Erhan, and Dilip Krishnan. Unsupervised pixel- level domain adaptation with generative adversarial networks. In CVPR, volume 1, pp. 7, 2017.
Fabio M Carlucci, Antonio D' I'Inncceete, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 1229-2238 2019.
Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference on Computer Vision, pp. 301-318. Springer, 2020.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on 2018.
computer vision and pattern recognition, pp.
Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model- agnostic learning of semantic features. In NuerlPS, 2019.
13

---

Published in Transactions on Machine Learning Research (08/2022)
Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pp. 200-216. Springer, 2020.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pp. 2551-2559, 2015.
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization. T-PAMI, (1):1-1, 2017.
Jean-Baptiste FiriattUUrruty and Claude Lemaréchal. Fundamentals of convex analysis. Springer Science Business Media, 2004.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference machine learning, pp. 1989-1998. PMLR, 2018.
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. Uncertainty in Artificial Intelligence, pp. 292-302. PMLR, 2020.
Jiayuan Huang et al. Correcting sample selection bias by unlabeled data. In NuerIPS, 2006.
Xun Huang, Ming-Yu Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV), 172-189, 2018.
Zeyi Huang. Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. arXiv preprint arXiv:2007.02554 2020.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 1125-11333 1134, 2017.
Kenji Kawaguchi et al. Generalization in deep learning. arXiv preprint arXiv:1710.05468, 2017.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In European Conference on Computer Vision. pp. 158-171. Springer, 2012.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani. Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: benchmark in-the-wild distribution shifts, In International Conference Machine Learning (ICML), 2021.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5542-5550, 2017.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. AAAI, 2018a.
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, pp. 1446-1455, 2019a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pp. 5400-5409, 2018b.
Haoliang Li, YuFei Wang. Renjie Wan, Shiqi Wang, Tie- Rie-Qian Li, and Alex C Kot. Domain generalization for medical imaging classification with linear-dependency regularization. arXiv preprint arXiv:2009. /2829, 2020.
14

---

Published in Transactions on Machine Learning Research (08/2022)
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and LINGYU DUAN. Uncertainty modeling for out- of-distribution generalization. In International Conference on Learning Representations, 2022. URL https: //cpenreview httpeeiiewwwwweeeeeee.ee
net / Forum? ?id- id- 6HN7LHyzGgP
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018c.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain gener- alization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 624-639, 2018d.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In International Conference on Machine Learning, pp. 3915-3924. PMLR, 2019b.
Ming-Y Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsuper- vised image-to-image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10551-10560, 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: arXiv:1608.03398. 2016.
Stochastic
gradient descent with warm restarts. arXiv preprint
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? International conference on machine learning. pp. 3481-3490. PMLR, 2018.
Muandet et al. Domain generalization via invariant feature representation. In JCML, pp. 10-18, 2013.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. 2016.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-2111 2011.
Xingchao Peng, Qinxun Bai. Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi soure- domain adaptation. In ICCV, 2019a.
Xingchao Peng. Zijun Huang, Ximeng Sun, and Kate Saenko. Domain agnostic learning with disentangled represen- tations. In International Conference on Machine Learning, pp. 5102-5112. PMLR, 2019b.
Vihari Piratla et al. Efficient domain generalization via common-specific low-rank decomposition. In International Conference on Machine Learning, pp. 7728-7733, PMLR, 2020.
Paolo Russo, Fabio M Carlucci, Tatiana Tommasi, and Barbara Caputo. From source to target and back: symmetric bi-directional adaptive gan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8099-8108, 2018
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the wilds benchmark for unsupervised adaptation. In NeurIPS Workshop on Distribution Shifts, 2021.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsu- pervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3723-3732, 2018.
Kuniaki Saito, Kate Saenko, and Ming- Yu Liu. Coco-funit: Few-shot unsupervised image translation with content conditioned style encoder. arXiv preprint arXiv:2007.07431, 2, 2020.
Seonguk Seo, Yumin Suh, Dongwan Kim, Geeho Kim, Jongwoo Han, and Bohyung Han. Learning to optimize domain specific normalization for domain generalization. In Computer Vision-ECCV 2020: 16th European Conference. Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pp. 68-83, Springer, 2020.
15

---

Published in Transactions on Machine Learning Research (08/2022)
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Gen- eralizing across domains via cross-gradient training. arXiv preprini arXiv: 10745, 2018.
Yang Shu, Zhangjie Cao, Chenyu Wang. Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmentedd meta-learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9624 9633, 2021.
Changjian Shui, Zijian Li, Jiaqi Li, Christian Gagné, Charles X Ling, and Boyu Wang. Aggregating from multiple larget-shifted sources. In International Conference on Machine Learning, pp. 9638-9648. PMLR, 2021.
Changjian Shui, Boyu Wang, and Christian Gagné. On the benefits of representation regularization in invariance based domain generalization. Machine Learning, pp. 1-21, 2022.
Yanmin Sun, Andrew KC Wong, and Mohamed Kamel. Classification of imbalanced data: review. International journal pattern recognition and artificial intelligence, 23(04))687-719, 2009.
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33:19276- 19289, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9 (11), 2008.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NuerIPS, 2018
Guoqing Wang, Hu Han, Shiguang Shan, and Xilin Chen. Cross-domain face presentation attack detection via multi- domain disentangled representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6678-6687, 2020a.
Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Ann Heng. Learning from extrinsic and intrinsic supervisions for domain generalization. In European Conference Computer Vision pp. 159-176. Springer, 2020b.
Yufei Wang et al. Heterogeneous domain generalization via domain mixup. In ICASSP, pp. 3622-3626. IEEE, 2020c.
Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain gener- alization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14383- 14392, 2021.
Zheng Xu, Wen Li, Li Niu, and Dong Xu. Exploiting low-rank structure from latent domains for domain generaliza- tion. In ECCV. 2014.
Pei Yang and Wei Gao. Multi-view discriminant transfer learning. IJCAI, 2013.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1eXiv:1611.03500 2016.
Kun Zhang, Mingming Gong, and Bernhard Schölkopf. Multi-source domain adaptation: A causal view. In AAAI, volume 1, pp. 3150-3157, 2015.
Ling Zhang, Xiaosong Wang, Dong Yang, Thomas Sanford, Stephanie Harmon, Baris Turkbey, Holger Roth, Andriy Myronenko, Daguang Xu, and Ziyue Xu. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Transactions on Medical Imaging. 2020.
Shanshan Zhao, Mingming Gong, Tongliang Liu, Huan Fu, and Dacheng Tao. Domain generalization via entropy regularization. Advances in Neural Information Processing Systems, 33, 2020.
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song. Noah Goodman, and Stefano Ermon. Bias and generaliza- tion in deep generative models: An empirical study. In NuerlPS, 2018.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European Conference on Computer Vision, pp. 561-578. Springer, 2020a.
Kaiyang Zhou, Yongxin Yang, Timothy M Hospedales, and Tao Xiang. Deep Nomain-adversaae image generation for domain generalisation. In AAAI, 2020b.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. arXiv preprint arXiv:2003.07335, 2020c.
Kaiyang Zhou, Ziwei Liu, et al. Domain generalization: A survey. arXiv preprint arXiv:2/03.025003 2021a.
Kaiyang Zhou, Chen Change Loy, et al. Semi-supervised domain generalization with stochastic stylematch. arXiv preprint urXiv:2106.00592 2021b.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. ICCV, 2017.
Andrea Zunino, Sarah Adel Bargal, Riccardo Volpi, Mehrnoosh Sameki, Jianming Zhang, Stan Sclaroff, Vittorio Murino, and Kate Saenko. Explainable deep classification models for domain generalization. arXiv preprint arXiv:2003.06449 2020.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Contents
Detailed proof
Preliminary
A.2
The upper bound of D (ze Ze x:)
Pi
19
A.3
The upper bound in the unseen target domain
23
Detailed architectures of the networks
23
B.1
The architectures E. and
23
B.2
The architecture of the domain-specilic encoder Ed
25
B.3
The architecture of the discriminator Dg
25
B
4
The architecture of the discriminator D.
26
B.5
The architecture of the generator G
26
C
Extra results
28
Results on PACS benchmark using Resnet-50 backbone
28
C.2
Parameter sensitivity
28
3 Results on Camelyon1
28
Computational cost
28
Extra visualization results
30
C.6 Comparison with other data augmentation-bast based DG methods
Failure cases
31
D Experiment details
31
D. Experiment environment
31
D.2 Tuning strategy for hyper-parameters
A Detailed proof
A.1 Preliminary
Our probabilistic graphical model for classification represented as Fig. 6 Re and xr and the mapping between 2d and æ are all many-to-many, e.g., there are many plausible images if specify ze 2d alone. In addition, given an image there should exist many possible 2, and za, i.e., there exist conditional distributions p(zel=) and p(zalz: instead of deterministic mapping. Or in other word, given specific latent code pair Ize, zd), there exists conditional distribution p(z|2). This assumption is reasonable since the latent codes z = [=c, =a] are usually constrained into a lower-dimensional subspace compared with the image-space so that there may not exist a latent code z that can deterministically correspond to an image r.
is notable that the mapping between
Assumption A.1 (Learnability) Denote the dimension of the image x and the latent feature 2e by d, and de respec- tively, let Φ. Rdd I.A (IR4=) be the mapping between an image and its probability functioon (PDF) f f2.(a). where L' denotes 1-norm integrable function space. deterministic mapping dg Rdd. - N acts as a classifier predict the category of the image based its latent feature 2n- For domain generalization problem with ny source
18

---

Published in Transactions on Machine Learning Research (08/2022)
28
Figure 6: The graphical structure of proposed model for classification, and the arrows represent learnable map- pings. The shaded nodes represent the observed random variables and others are latent random variables. There exist conditional dependences between the image and the category-specifie feature 2, and domain-specific feature za. respectively. We expect be and zd are independent. In addition, the ground-truth label y only depends on the category- specific feature Ze.
domains (X, = U Xn,) and a target domain Xt. we say it learnable if
3dc s.t. E [dc(2)) EgnX,iv (o)())), Vy Y,Vi, f1,2,.... n..t]
(13)
( (x))) Va U.X.
where Xuy represents the conditional distribution of images with the category y in the domain v/(+)) returns the value of the distribution.
In other words, we say the task is learnable if there exists a domain invariant category-specific encoder Φe among source and unseen target domains that can achieve the same marginal distribution p(zely) for each domain. addition, the extracted category-speciff features can be used to correctly predict labels of the images from both source and unseen target domains. This assumption is mild since only assume that the task is feasible. In addition, have additional constraints on the form of the solution.
For a learnable domain generalization task, we aim to find out the domain invariant mapping Φc(-) so that we could further train a domain-agnostic classifier. However, given limited data and limited function space, the mapping di is intractable. To this end, we propose to use an encoder Eg to approximate the unknown ground truth mapping de by Bayes variational inference. More specifically, we give an upper bound of ((Q(z,]=)) (x)) where P(z. ax) (z) and Qzze (z) is the conditional distribution embedded by the encoder E, implemented by a deep neural network. A corresponding framework is further proposed optimize the proposed evidence upper bound.
Our motivation is summarized as follows
We first give an evidence upper bound of the KL divergence ((((zel=)]IP(((e11)) between the approximated conditional distribution of sategory-specifii features 2, and its intractable ground-truth distribution in Lemma and Theorem A.
We further relax the information gain term of the derived upper bound to make a balance between the two terms in the upper bound. Our relaxed loss is proved to be an upper bound of the original one as shown in Lemma A.2
Finally, we demonstrate that under mild assumption, the upper bound of the divergence D(Q(zcl=")][P((z])) in the unseen target domain can also be bounded Theorem A.2
A.2 The upper bound of D(Q(zzl=r)] P(2.(1)))
((2c())) can be represented as Lemma A.1 The KL divergence ((Q(zel (Pz.))))
D(Q(2.l=))IP(2ee)) E Pa.- Oz (lgg Po [c)] + log p(x).
(14)
19

---

Published in Transactions on Machine Learning Research (08/2022)
Proof A.1
/ (<<o|2) log [(2c(2)) d2c p(zalæ)
D(Q(2el<)) P(zala))
/ (zz.|=)[log g(z.(2) log
(15)
((=.| ((ze - logp(x|ze) ((z)) + log p(
([(((z//>)) Q [ogr(<)).)] log)(z)
The term p(z) can come out of the expectation QeeeQuete on account that does not depend on &e. This completes the proof.
Based on Lemma 1, we can derive the evidence upper bound of [(Q(zolz))
Theorem A.1 The evidence upper bound of KL divergence D(Q(2cl=)|IP(ze) i)) between the distribution Q 2x) and the ground-truth P(ze)z) (r) is follows:
DiQ x)) P((o) - Es, Pa logp(zl2) -
-C
(16)
information goin tern
@ posterior probability
where is constant, [Zes zal, and Pa can be an arbitrary prior distribution.
Proof A.2
D(Q(2))=)(P((<1zz) =D(Q(zal))((((za) Peen [logp(z/zz + log p(x)
Using
D(Q(2cl=)](P(za)) BacaRr x [log|
p(x, Kal (alkk)) + log p(x)
pluc, zd, ze) |] + log
D(Q(z)l=)]IP(za)) Baa . .log(
p(ac)
plza)plza /2d (e)pp( 22)
D(Q(2.lz)](P((za) Pa - [log]
+ logp(>)
p(ze)
(17)
p(ze (za)
D[Q(2zl<1)][P Exaw [log(Esa>>p
p [ + logp(*)
p(z.)
S D(Q(2clz)) Exc Pa logl p(zelZa)
p(2.) p[x|2)] p(x)
Using Jossen' innqualiiy aeciinn log connes
D(Q(z)l=)]IP(aa)))
Eize sar plog za)]
+ log p(z)(s)
p(zelza) p(sc) becouse and iu independent
[[)(2c))))IP((e))
Exen P. [log p(x|=)] p[]]]
information
3 posteviar probubiiity term
This completes the proof.
Assumption A.2 For an image x there exists a constant M that satisfies
q(*lae) M given any zc- q(ar)
The Assumption A.2 mild due to the following reasons. First, x is generated based on both 2, and 2ạ- There- fore, the mapping between 2, and z is deterministic, i.e. (xx)2e) can not be the impulse function Second, when the random variables z, and zy extracted from encoders F, and F, disentangled in some degree, i.e.,
20

---

Published in Transactions on Machine Learning Research (08/2022)
dlcltda < k where & is constant, obtain the following upper bound su g(20)
g(rlzo)
g(x, zalzo)
daa
((x)
g(x))
/ g(Z))q (Z)) Re. zd)
ded
(()))(())
(18)
g( Re zd)
S k kF
((x)
q Rey 2d) M
$ k sup
((x)
Rer
where Ze, for a given x. Thus, g(z) also demonstrates that the better the disentanglement, the tighter upper bound we can obtain.
Ee, zd) is the PDF of predefined Laplace/Gaussian distribution so that is bounded, and g(ar) a constant
((a]==) is bounded when and 2d is disentangled to some degree. In addition, Eq. equation 18
A.2 Lemma A.2 The upper bound of the KL. divergence D(Q(ze)=))) based on Assumption A.2 can be represented as
MD(Q(Z=)|I((cc) + M log M.
(19)
Proof A.3
D(Q(ze|z)]I((2)) a(zo(()) log g(Zelx) p(zn) f g( a(<rZz)a(Ze))
dzc
log (0 ((())z))(Ze) dza ((x)
(xx)p(Ze)
(20)
S MI/ g(z)) log g(ze) dāe + log M]
p(zo)
MMP(Q(E)) M log M
The upper bound given in Lemma A.2 is used to balance the weight of two terms in the upper bound of [[Q(z]==)]|((a)
given in Theorem A.1 It is difficult to accurately estimate the posterior probability term in Eq. 16 since the high dimensionality of the image and limited data. On the contrary, the calculation of trivial and accurate, so it leads to a stable gradient. If we do not relax the constraint of the in- formation gain term in Eq. [6] the gradient accumulation will cause over-sparse embedding of ze. The ablation study in the main paper also demonstrate the necessity of the relaxation. To further verify that optimizing D(Q(zz)) P((.)) can help the minimization of [(Q(zc
( |
we provide a synthetic toy example as follows.
We consider a classical problem, exclusive or, which aims to predict the outputs of XOR logic gates. More specifically, in our setting, we assume & E R3 and its label y = (.o > 0) @ (a) > 0) @ (I2 > 0) where x; ~ U(-1, 1) represent the ith dimension of x. For networks, we use the same backbone network for all experiments which includes three fully connected (FC) layers with the (input dimension, output dimension) as follows (3. 3). (3, 2), (2, 1). For each fully connected layer except the last one, there a ReLU activation layer following We adapt the reparameterization trick (Kingma& & Welling 2013) as shown in Fig. so that could explicitly obtain the distribution Q(z (z), i.e., the KL divergence between (ze (z) and the standard Gaussian distribution can be explicitly calculated. For the baseline method, we directly optimize the network by minimizing the cross entropy loss. For the method with the regularization term Lreg. we additionally utilize multilayer perceptron (MLP) with three FC layers as the discriminator D... We keep other papametrrssetrss and settings the same. The experimental results are shown in Fig. [8, As we can see, the model can achieve slightly higher test accuracy by introducing Lreg- addition, even if we do not directly minimize ((( [x)]
can be effectively optimized by minimizing D(Q(ze)) P(z.)) compared with the baseline method, The result in Fig. b)) further supports our Lemma A.2 empiricall..
21

---

Published in Transactions on Machine Learning Research (08/2022)
Fully connected layer (2,1)
Ex.
((0.1) 1)
Mic
Fully connected layer (3,2)
Fully connected layer (3,2)
Fully connected layer (3,3)
Figure 7: The structure of the network for the toy synthetic experiment. The activation layers are omitted for concise.
We utilize the reparameterization trick that could explicitly obtain the distribution (22elz)
wib Log
wi Loes
w/ Log
2.0
Tsst eccurccy
1.8
1.6
0.74
1.4
12
0.70
0.8
2000
4000
6000
8000
10000
Epochs
Iterations
(a) Test accuracy
(b) The KL divergence D[Q(
Figure 8: The comparison of the methods w/ and w/o Lreg-
22

---

Published in Transactions on Machine Learning Research (08/2022)
A.3 The upper bound in the unseen target domain
Assumption A.3 Given sample x' from the target domain Xt. there exists non-empty feasible set I which is defined as
I= (I1g((e)))))) & Big( Vze (Iloo((l) =.l)))))e)))))).))iell).
where the index set, x denotes an arbitrary sample with index any source domains, and q(zela)) the probability density function value of že conditioned from distribution Qzze (z).
Since we target to deal with the target domain samples with only domain shift, i.e., the samples with unseen zd and known =e. the unseen sample has quite a different ground-truth distribution of the feature Σ.. we cannot guarantee the behavior of the classifier even we have an ideal classifier E. that can perfectly disentangle 2e and žd. We assume the task feasible, i.e., there exist samples in source domains that have the same ground -ruth conditional distribution Pacla- In other words, the second set in Assumption 2 is not empty if the task is feasible.
For the first set in Assumption 2, as long e.g.. Gaussian/Laphee Laplace distribution, for any source domain image set in the second set { (x(x') = (x])), vi E f}. there always exist a vector of 3g that makes q(=e(x²) I Eier (=)(= Vzz E IRd hold. Therefore, the fesasible set non-empty the task is feasible.
assume (<=e(<)) is distribution that satisfies ((=e(<) > 0 given any Fes
Theorem A.2 Based on Assumption A.3 the KI. divergence between Q(zelx²) and the unknown Comaiin-invarinn ground truth distribution ((zc(²") can be bounded as follows
((Q(zel=") S inf IEI
>
IEI
The above theorem demonstrates that the KL divergence between Q(ze|z) and P(ze (x) from source domains con- stitutes the divergence upper bound in the unseen target domain. Therefore, further supports the rationale and effectiveness of our method. Since no general upper exists KL divergence, our derived upper bound will also inevitably exist arbitrarily large values. However, it still provides some insights which are intuitive:
We can get tighter bound if we have more diverse samples. By increasing the number of source domain samples, we are likely to obtain larger feasible set to achieve a smaller infimum.
The better disentangle ability the encoder Eg has, the more similar g<ze(=²) and g(zc(=;)) will be. Then are expected to be samller.
B Detailed architectures of the networks
To be more clear about the architecture of our framework, we give simplified diagram about the relationship between class-specific encoder E., domain-specific encoder Ea and generator G. More details about the architecture of each network are elaborated in the following sub-sections.
B.1 The architectures of E. and E,
B.1.1 The architectures of E. and E, for Digits-DG
We use ConvNet as the backbone for Digits- DG and divide into two parts. The details about the division for E, and E, are illustrated in Table 6
B.1.2 The architectures of E. and E, for PACS and mini-Domainnet
For PACS and Aimi-Domannnet we use Resnet- 18 the backbone network and adopt the same division regarding E, and F,, The details about the division for F, and F,, are illustrated in Table
23

---

Published in Transactions on Machine Learning Research (08/2022)
sppoox Eug Aursoh
task-specific encoder E
Ze
concatanation
dopain-specific encoder Ed
expension
generator G
Figure 9: simplified diagram regarding Ec, Ea and G. The input of the generator G consists two parts: the output of the task-specific encoder 2, and the output of the domain-specific encoder zg- The feature is fed into three-layer network to obtain the mean and variance used by AdaIN ResBlk following Liu et al. (2019). In addition, reshape to the same size with the feature map of 2e and then concatenate them together.
Layer
Conv2D(im=3, out=64, kemel size=3, stride=1,
2
Relu
MaxPool2D(kermel
Conv2D(in=64, out=64, kernel_size=33 strde=l.,paddinge
task-specific encoder E2
Relu
MaxPool2DDkermellsizzze Conv2D(in=64, out=64, kernel size=3, Itride=
Relu
MaxPool2D((ermell size=2)
10
Conv2D(in=64, out=64, kernel_ stride=1,
11
Relu
task-specific network Eg
12
MaxPool2Deermeesszee22
flatten the feature
Fully connected layer(nn=64*2*2, category number)
Table 6: The architecture of task task-specific encoder E, and task task-specific network E, for Digits-DG. They are obtained by separating the backbone ConvNet into parts.
24

---

Published in Transactions on Machine Learning Research (08/2022)
#
Layer
Conv2D(in=3, out=64, seomel_sie=7,,sie stid
2
BatchNorm2d
Relu
task-specific encoder E.
5
Relu
3 x 3
64
6
layer1
3 x 3
64
3 x 3
128
7
layer2
x 3
128
x 3
256
layer3
3
256
task-specific network E,
3 x
512
9
layer4
x
512
10
global average pooling
11
Fully connected layer(in=512, category number)
Table 7: The architecture of the task-specific encoder E. and the task-specific network Eg for PACS and mini- DomainNet. They are obtained by separating the backbone Resnet-18 into two parts.
B.2 The architecture of the domain-specific encoder Ed
For all the experiments, we use the same domain-specific encoder. The details of the architecture are shown in Table On account that we global average pooling layer reduce the size of the feature map 1 x 1, the domain- specific code 2a will lose the spatial information and hence only embed the global appearance which is relevant to the domain.
#
Layer
1
Conv2D(in=3, out=64, kemne_size=7,, stidd
2
Relu
3
Conv2D(in=64, out=128, kernel_size-tt [tidd::.nnid
4
Relu
5
Conv2D(in=128, out=256, kernel_size=4, stide=:2=ntd)
6
Relu
7
Conv2D(in=256, out=256, kernel_size=4, (tifee//-ntiddd2..padddnng111]
8
Relu
9
global average pooling
10
Conv2D(in=256, out=128, kernel_size= stride=
Table 8: The architecture of the domain-specific encoder Eg--
B.3 The architecture the discriminator D,:
PatchGan discriminator Isola al. 2017) is adopted De for all experiments. More specifically, the discriminator Dz which is responsible to distinguish real and generated images includes one convolutional layer and 8 activation first residual blocks (Mescheder et al., 2018) as shown in Table g
Our discriminator is not only capable distinguishing real and fake, but it can also distinguish whether the image comes from the specified domain (the number of output channel of our discriminator the domain number as illus- trated in Table g.. 2e and zg are not disentangled well, the arbitrarily combined pairs may not generate realistic images with corresponding domain labels, i.e., the disentanglement is supervised by the posterior probability term.
25

---

Published in Transactions on Machine Learning Research (08/2022)
Layer
Conv2D out-64 http://eeessii....ttscssaaaaaaad
AdffinsiResBlookinnn44,, out=128, activation=-eaayyRllu, norm=None)
AcffirsiResBlookinn122,, out=128, activaation==leakyRRet. norm=None)
ReleectionPu2(((aaddnn-)))
Avg@Pool2@keermllssiee== stride=2)
ActFinstResslllckinn1123, out=26,, activation=leakyRell. norm=None)
ActFirsResSBocc(nn2255,, out=256, activation-leaayRelu, norm=None)
Reflectioonaall((padddnnn
BitPooo@2ddkemmc__siieeeee. stride=2)
AcfFirsiResSlookinn2555. out=512, a, Relll,iiiiii norm=None)
10
ActFissiesBlockiin5522...0 out= activation=-lakyRReuu norm=None)
Reflection/aa2d((aadddnn====]]]]
12 A out= htt.://2000..00044aaaaaaaaaaaaaa norm=None)
AvgPool2d(kermcl_size== stride=2)
13
14
AdtinssRe:sooccii==10044,, 0ut out norm=None)
15
ReffectionPPuddd(poddddn--=)))))
16
AvgPool24(kermielssize--- stride=2)
17
LeakyRelu(1024)
18
1on22Diin=1024, out=domain number, kernel_size=l, stride=1,paddingg
Table 9: The architecture of the discriminator D.-
ActFirstResBlock: activation first residual blocks (Mescheder
et 2018)
B.4 The architecture the discriminator D.
The discriminator D. is built by multiple fully connected layers. The details are illustrated in Table [0] For the fake sample, the input of the discriminator D. the task-specific feature 2 after average-pooling and concated with the domain label which acts supervised information. For the real one, the input is the random variable sampling from standard Gaussian distribution and concated with random domain label.
Layer
Linear(in=input dim, out=512, bias=True) Relu
Dropout(rrobailityy00222
4
Linear(nn=122, out=512, bias=True)
5
Relu Dropou((robabiltty0..)) Linear(512, 2)
6
activation layer g = - exp(
Table 10: The architecture of the discriminator D.. input dim: the number of source domains + channel number of the feature 2e Ex(
B.5 The architecture the generator
We use roughly the same architecture of the generator G for Digits-DG and mainly in the number of channels and the number of upsampling modules. The architecture of the generator is shown in Table 11 Note that the three-layer network for AdaIN ResBlk (Huang al. 2018) is not included in the table.
The difference is
26

---

Published in Transactions on Machine Learning Research (08/2022)
(a) The generator for Digits-DG
layer
AdaIN ResBIk(192)
2
AdalN ResBIk(192)
3
Sisaamplesscaeeffccooeeee nearest)
4
1onv2Diin=192, out=96, kernal_size=5) s=1, p=2)
InstanceNorm
Relu
Upsample(scale_fanne faatfffactr222.m fatp:eeeddeeceefattteeeeeeeeee nearest)
8
Conv2D(in=96, out=48, kernalsize=5, s=1, p=2)
9
InstanceNorm
10
Relu
11
Conv2D(in=48, out=3, kernal size= s=l, p=2)
12
Tanh & Normalize
(b) The generator for PACS and mini- DoiannDomin
#
layer
1
AdaIN ResBik(256)
2
AdaIN ResBIk(256)
3
Upsample(scale 2 nearest)
4
Conv2D(in=256, out=128, kernal_size=5, s=1, p=2)
5
InstanceNorm
6
Relu
7
Upsample(scale factor=2, nearest) Conv2D(in=128, out=64, kernal_ size=5, s=1, p=2)
8
9
InstanceNorm
10
Relu
11
Upsample(scale 2 nearest)
12
Conv2D(in=64, out=32, kernal_size= s=1, p=1)
13
InstanceNorm
14
Relu
15
Conv2D(in=32, out=3, Bernal_size-it s=1, p=2)
16
Tanh & Normalize
Table 11: The architecture of the generator G except the three-layer network for AdaIn ResBlk (Liu et
2019
27

---

Published in Transactions on Machine Learning Research (08/2022)
C Extra results
C.1 Results on PACS benchmark using Resnet-50 backbone
Algorithm
A
P
Avg
ERM
83.2 1.3
76.8 1.7
97.2 0.3
74.8
1.3
83.0
IRM
2.4
77.0 1.3
96.3
71.1 2.2
81.5
GroupDRO
0.74
96.8 0.8
1..6
83.5
Mixup
8.9
1.7
96.8
1.6
83.2
MLDG
33.6
2.3
9.3
76.1
82.9
CORAL
80.5
7.4
9.3
78.6
82.6
MMD
14.9
2.0
9.9
1.5
83.2
DANN
84.3+2.8
2.8
+.8
70.8
81.0
CDANN
288
1.6
9.5
6.5
78.8
MTL
1.5
0.6
97.1
7.71
83.7
SagNet
81.1
75.4
95.7 0.9
77.2
82.3
ARM
0.3
13.3111
06.6+004
72.1
81.7
VREx
8..6
74.1 0.3
96.9
721
81.3
RSC
8.77
82.9 1.1
95.6 0.7
6.5
82.6
VDN(Ourr)
83.5 + 0.7
96.7 + 0.3
85.6 0.6
87.9
Table 12:
Evaluation of domain generalization performance on PACS benchmark using ResNet-50 backbone. The mean value and corresponding standard error of the test accuracy are reported.
We further conduct experiments using ResNet-50 backbone and report the mean accuracy and std the target domain when finish the training. The results repeated experiments are shown in Table 12 and the results demonstrate that our method achieves better performance than SOTA method
C.2 Parameter sensitivity
We explore the influence of two important trained with different Aree and Ayan while keeping other settings the same on the sketch domain PACS, The results are shown Fig I0 As we can see, the performance remains relatively stable among different hyper-parameters. We empirically find that Àgan has relatively larger impact the performance compared with Xrec- We conjec- ture the reason is that inappropriate Agan may cause unnatural artifacts. For larger Xrecs imposes stronger regularization on 2e and zd so it also causes performance degradation.
Arec and Agon. We evaluate the performance models
C.3 Results Camelyon17
We further evaluate the effectiveness of method the WILDS (Koh al. 2021 Sagawa et al. 2021 benchmark. More specifically, utilize Camelyon17 (Bandi al. 2018) dataset which is collected from different hospitals. Each hospital is regarded as domain and the domain variations are mainly in the data collection and processing. We use the official train/val/test split of the dataset and report the test accuracy when we achieved the highest accuracy on the evaluation set. We use DenseNet121 as the backbone which the same for all competitors. The generated samples are shown Fig. [ and the accuracy in the unseen target domain is reported in Table [3] As we can see Fig, I1 our method can well disentangle the domain variations, e.g., the color difference caused by the processing while preserving the necessary details. In addition, the quantitative result exhibits huge improvement compared with the baseline methods, which further demonstrates the effectiveness of our method.
C.4 Computational cost
For inference, we do need the auxiliary networks so that we have the same computational cost and the clock time the baseline method.
The results other methods from camera ready version the paper "In search lost domain generalization ICLR 2021.
28

---

Published in Transactions on Machine Learning Research (08/2022)
82.8
83
82.4
Accuracy
82.5
82
81.8
81.5
81.8
81
10
81.4
10
81.2
gan
81
0.1
rec
Figure 10: Sensitivity analysis of hyper-parametee Avec and Agan-
Figure 11: The generated samples in source domains using Camelyon17 Bandi et al. 2018) dataset. The images in the first row are the input images that we want to keep the category information and in the last row are the input images that provide the style information. The second row is the reconstructed one and the third row is the translated one.
Evaluation Accuracy
Test Accuracy
ERM IRM VDN(OOur)
85.8 (1.9)
70.8 (7.2)
86.2 (1.4)
64.2 (8.1)
88.0 (2.4)
90.3 (3.1)
Table 13: Evaluation domain generalization performance on Camelyon17 dataset using DenseNet-Ie -21 backbone. The mean value and corresponding standard error of the test accuracy are reported.
29

---

Published in Transactions on Machine Learning Research (08/2022)
For training, we compare our method with the baseline method and MLDG using the settings of the experiments Camelyon17 17 as illustrated in Sec.
C.3 All the experiments are conducted on the same server with RTX A5000s.. We use the same settings, c.g., the same batch size and image size, for all experiments. The results are reported in Table [4 As we can see in the table, our method has an acceptable computational cost, which is similar to the previous worth noting that, the extra memory usage of our method is mainly due to the existence of data augmentation, i.e., the batch size of our method can be regarded as twice that of other methods.
work MLDG (Li et al. 2018a).
Parameters
Memory usage
Speed (s/iter)
Baseline
6,955,906
2739MB
0.1153
MLDG (Li al. 2018a)
6,955,906
4013MB
0.4604
VDN (Ours)
8,408.844
5765MB
0.2115
Table 14: The computational cost of our method for training. We train models using mixed precision for all experi- ments using the same platform.
C.5 Extra visualization results
Besides the image generation task in the source domains or in the target domain that we have shown the results in the paper, we are also interested the "cross-domain" performance, i.e., from source domains to unseen target domain and from unseen target domain to source domains. Assuming that we have two images H, and x; and their corresponding task-speeific features Ze, 2e, and domain- speciiic features 2d, 2d, the translated images with different mixture degrees are generated using G(ze,;Aza + (1 - A)za,) where À in [0.0, 0.2, ...; 1.0]. Note that in this work, we do not use the generated samples with mixture style for data augmentation nor use the discriminator to supervise the generated images with a mixture style. How to effectively utilize the sample of mixed style may be a potential research direction.
C.5.1 From the target domain to source domains
We first investigate the performance of our framework that translates the samples from the unseen target domain to source domains. The results are shown in Fig. [4] As we can observe, even if the framework has never seen the samples in the unseen target domain, it can still extract task-specific features that are compatible with the domain- specific features from source domains and then generate the samples with the style of source domains. This further verifies the robustness and generalization ability of our method.
C.5.2 From source domains to the target domain
We then explore the performance of our proposed framework that translates the image from source domains to the unseen target domain. The visualization results are shown in Fig. [5] As we can see, the framework may be difficult to generate samples of the unseen target domain. This is reasonable because our framework has never seen the sample distribution of the target domain and it is difficult to generate OOD samples with large domain gaps. However, our proposed framework can generate samples from new domain, i.e., the image that is not exactly the same as the sketch domain but has some similar characters such as having little task-irrelevant information.
C.6 Comparison with other data augmentation-based DG methods
It is still an open problem and research hotspot to generate realistic images. The conditional generation task or unpaired image translation task usually requires large-scale data or data which are not very diverse. For example, CycleGan uses 939 horse images and 1177 zebra images to train a model that can only transfer the horse image to zebra even if they have relatively high similarity. However, the PACS dataset we used only has 1670 images in the photo domain and there are both different domains and different categories in it. In addition, photo domain images have more details, especially compared with sketch so it will be more difficult to learn and generate.
Most importantly, we aim conduct the classification task instead of the generation task. The generated images are only side products of our method and the perceptual quality of generated images is beyond the scope of our paper.
30

---

Published in Transactions on Machine Learning Research (08/2022)
In addition, ablation study demonstrates that by using generated samples augmented training data, the generalization ability of the model can be further improved and outperform other data generation-based DG methods.
Here we display the visualization results reported in their paper from the methods that also aim to generate novel data using the same PACS dataset in Fig. 12 As we can see, these methods fail to generate data with large divergence, e.g..
the translation to sketch domain in Borlino al. (02T).
Source domains
Cartoon
Stutchh
Treak
1)
G)x,3)
()a.3)
segrantastinn
Tatreng
ReaE (Phutordmmai))
to Pamiiig
Dorran ain Transfo masion Network
unlabeled image
atong
Totple
oragecaiiaiio
(a)
Zhou al.
2020h
Zhou al. 20200
Zhou et al. 202Tb
(d) Borlino et al. 202T
Figure 12: Comparison with other DG methods based novel data generation. Best zoom in for details.
C.7 Failure cases
Some failure cases of our proposed model are displayed in Fig. 13 For the Digits-DG benchmark, we may fail to generate samples with the style that multiple digits exist in single image. We conjecture that it may caused by the inductive bias of network structure, e.g., the utilization of the AdaIn layer. For the natural image, our model may fail transfer images from the sketch domain to the real photo/art painting domains, which include much more information. This may be due to the limitation of the data amount. While some of the samples do not have desirable perceptual quality, they may still include some important features of the desired category, so they are still valuable for acting as augmented data.
6
6
2
6
209
10
Figure 13: Some failure cases of our model.
D Experiment details
D.1 Experiment environment
Parts of the experiments are conducted on a Windows workstation with Ryzen 5900X, 64GB RAM, and an Nvidia RTX 3090. Others are conducted on Linux server with Intel(R) Xeon(R) Silver 4210R CPU, 256GB RAM, and RTX 2080-TIs. For the software, PyTorch 1.9 is used. We do not find significant difference of trained models under two different environments.
31

---

Published in Transactions on Machine Learning Research (08/2022)
D.2
Tuning strategy for hyper-parameters
Since we have multiple hyper-parameters, we elaborate the tuning strategy here for reference.
We first explore the split strategy of E. and Eg. If Ee is too small, then there will be little semantic informa- tion. While Eg is very deep, then we find is difficult reconstruct the image using feature map with small width and height given small but diverse dataset.
After finding the appropriate division of E. and Ee that can well reconstruct the image, we then explore the hyper-parameter of the regularization term Areg since it is relatively individual hyper-parament After finding the Ayeg that can increase the generalization ability, fix it for the following experiments. We also find Avec that does not degrade the classification performance much and well reconstruct the images, and then fix it for the following experiments.
We then introduce the discriminator into the training. We find that the updating frequency of the generator and discriminator is significant. We first find the updating frequency that will not cause model collapse and then finetune Agan-
We search the hyper-parameterd on the PACS benchmark since is a relatively small dataset. We find that the
used in PACS exhibit good robustness and can also be used other benchmarks.
32

---

Published in Transactions on Machine Learning Research (08/2022)
Figure 14: The visualization results of the image translation from the unseen target domain source domains. The photo domain in PACS is used as the unseen target domain. The first row is the input image from the unseen target domain that provides the task- features and the last row is from source domains that provide the domain- specific features. Other rows are images that are generated images G(Eeg Azd, + (1 A)za,) using different mixture degrees.
33

---

Published in Transactions on Machine Learning Research (08/2022)
Figure 15: The visualization results of the image translation from source domains the uneen target domain. The domain sketch in PACS is used the unseen target domain. The first row is the input image from source domains that provide the task- sask-specific features and the last row is from the unseen target domain that provides the domain-specific features. Other rows are images G(zc, Azd, + (1 A)æd,) ) using different mixture degree.
34


Reviews:
[{"review_id": "wvWXCQlhj1", "reviewer": "Reviewer_J5eu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. The whole idea is interesting. DG problem will be impossible if no assumption between domains is made. This paper considers a fundamental assumption regarding DG, i.e., some invariant features exist. Thus, we can consider detaching the label information and style information.\n\n2. The proposed method is based on a developed theoretical bound, which is sound and solid.\n\n3. Experiments verify that the proposed method is effective and better than many sota methods, which further verifies the contribution of the proposed theory and algorithms.\n\nCons:\n\n1. Assumption 1 seems incompleted. Please note that conditional distribution is the same does not mean the joint distribution is the same. For example, if the class prior in the target domain is very different from those in the source domains, DG seems also impossible. How do you avoid this? Some remarks should be added regarding this issue.\n\n2. In Assumption 2, why do you use \\ge rather than =. q is just a pdf function. It is not easy to understand Assumption 2.\n\n3. Section 3.3.2 and Section 3.3.3 are not easy to follow. More details are required for completeness.\n\n4. In Section 3.3.2, this paper uses D(Q(z_c)||P(z_c)) to replace with D(Q(z_c|x)||P(z_c)). More discussions are need here. Some synthetic experiments may be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers disentangling the label information (that is invariant) and domain style for addressing the domain generalization problem better. This idea (in general) could lead more papers to focus on the solvability of the DG problem, which is good for the area. They also first give a theoretical bound and then design their algorithm, making the whole algorithm sound and theoretical solid. The experiments verify that the proposed method works better than existing methods (sota experiments or ablation study). In general, the main idea of this paper and the theory proposed in this paper are interesting. Thus, I would like to give a major revision to this paper.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Cons:\n\n1. Assumption 1 seems incompleted. Please note that conditional distribution is the same does not mean the joint distribution is the same. For example, if the class prior in the target domain is very different from those in the source domains, DG seems also impossible. How do you avoid this? Some remarks should be added regarding this issue.\n\n2. In Assumption 2, why do you use \\ge rather than =. q is just a pdf function. It is not easy to understand Assumption 2.\n\n3. Section 3.3.2 and Section 3.3.3 are not easy to follow. More details are required for completeness.\n\n4. In Section 3.3.2, this paper uses D(Q(z_c)||P(z_c)) to replace with D(Q(z_c|x)||P(z_c)). More discussions are need here. Some synthetic experiments may be helpful.", "broader_impact_concerns": "There is no concern about the ethical implications of the work. DG is a well-known and significant problem in the field.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cHKQHyoagz", "reviewer": "Reviewer_AP9E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n+ The problem of domain generalization is significant and timely. \n+ The work tries to relate it well with the literature. \n\nweaknesses:\n- The writing could be improved (see below). \n- There is no section on limitations and broader impact at the moment. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work develops a framework for domain generalization. The idea is to rely on domain-specific information and category-specific information through feature disentanglement. Specifically, given a multitude of source domains, the framework learns a domain invariant embedding for classification and domain-specific features in a variational setting. The schematic framework can be seen in Fig. 2 and it involves the task-specific encoder, the domain-specific encoder, a decoder (‘generator’) and two discriminators. A number of experiments in small datasets are conducted. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The quality of writing can be substantially improved, as there are multiple mistakes both in language and consistency errors. For instance, no title has a “:” symbol apart from sec. 4.4.3. There is space required in the phrase “the target domain.Meanwhile” (page 11). There are several errors that proofreading would fix, e.g. “all the datasets may has”, “target domain is pass through”, “can also be benefited”. Another error made consistently in the text is the citet package usage. After citet the verb should be in plural form if there are multiple authors, e.g. Li et al. propose. This should be fixed throughout the manuscript. The current writing is not appropriate for a high quality paper. \n\nIs assumption 2 a strict assumption? Could the authors elaborate on the first part of the assumption on why it holds? \n\nWhat are the limitations of the method? How about the broader impact? \n\nIn eq. 9 the objective is max_D min_{E, G}, while for GANs it is usually min_G max_D. Why is it reversed here and are any guarantees lost with this change? \nI think one very commonly used dataset is the WILDS benchmark, however the paper does not include it. Is there some reasoning behind excluding it? \n\nIn sec. 2 the paper mentions that the proposed framework conducts translation in a very efficient manner. However, in the pix2pix paper cited in the same paragraph some of the mappings are learned with few images; are less images used in the experimental validation of this method? \n\nWhat is the computational cost of the proposed method with respect to the baselines? Both in terms of parameters and with respect to the clock time. \n\nHow sensitive is the proposed method to the various hyper-parameters? An ablation study would help to clarify the important hyper-parameters for the interested practitioner. \n", "broader_impact_concerns": "There is no broader impact statement in the paper at the moment. And indeed, similarly to image translation, this framework could have negative impacts, so those need to be described in the revised version. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EOj6ZUY1vW", "reviewer": "Reviewer_DDX9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Update after discussion\n\nAuthors have properly addressed all major concerns. Thus I would recommend acceptance. \n\n## Summary:\n\n### Strong points: \na. [This paper is interesting to the community.] Briefly, it proposed a practical and principled probabilistic framework for addressing domain generalization. Moreover, the framework could generate new style-transfer images. In contrast, the conventional approaches generally did the data augmentation on the embedding space and could not generate real-images.\n\nb. [Theoretical aspects] I would appreciate the authors for clearly illustrating the theoretical assumptions (e.g, assumption 1,2) within the paper. It is quite beneficial to know the limitations and scope.\n\nc. [Practical aspects] The extensive empirical results demonstrate the effectiveness of the proposed approach. Specifically I like the results in Appendix C2.2, which clearly depicts empirical benefits of the proposed approach.\n\n### Weak points:\n\na. [Paper organization/writing] In general, the paper is feasible to follow, while some parts are confusing/unclear or augments are inaccurate. \n\nb. [Theory and practice gap] It is appreciated that authors could provide several theoretical analyses, while there exists several non-ignorable gaps between theory and practice. Following the guidelines of TMLR, authors are required to claim/adjust such gaps. \n\nc. [Practical concerns] The high-level idea sounds correct to me, while there are several practical concerns.\n\n## Detailed comments on weak points\n### Paper organization/writing. \n\n1. [Sec 1]  “Generally, existing DG research works can be categorized into two streams, invariant feature representation learning and data augmentation.” In general the meta-learning based approach could be a third stream.\n2. [Sec 1] “they may face the risk of overfitting to the source domains without carefully specifying the invariant information to learn.” I would refer to a recent paper [1] for discussing the overmatching risk to the source domains, which provides a formal justification about this concern.\n3. [Sec 1]”To further optimize the upper bound, we develop an efficient framework named Variational Disentanglement Network (VDN)”. I think it should be effective rather than efficient, since the efficiency has not been justified in the paper.\n4. [Sec 1] “we propose to optimize the category-specific feature zc and domain-specific feature zd at the same time.” I would suggest category(label)-specific feature zc.\n5. [Sec 1] “Meanwhile, by further minimizing our proposed information term which is to filter out redundant information, zc is expected to have less domain-specific information”. I think it is safe to remove this since zc is defined as independent to the domain.\n6. [Sec 1] “Our proposed evidence upper bound further supports the rationality and builds the tie between previous feature alignment-based methods (Li et al., 2018b) and disentangle-based methods.” It seems I missed something, but I could not find evidence how they are related. This contribution might be adjusted/better illustrated.\n7. [Sec 3] “all source domains are represented as Xi , Yi and X, Y respectively”. Seems like a broken sentence. What is X,Y ? the target ? Besides, the conventional notation assumes X,Y as random variables and P(X), P(Y) as distributions. \n8. [Assumption 1] $\\Phi$ is the means of the random variable, I would suggest avoiding using this function since it is not clear in the definition. It could be simply expressed as the expectation notation $E_{x} \\phi_c(x)$.\n 9. [Lemma 1, Theory 1, Theory 2] The definition of x is consistent with Assumption 1? I.e $x \\in X_s \\cup X_t$ ?\n10. [Sec 3.2] “following the core idea of F-GAN” -> f-GAN, f is a convex function here.\n11. [Paper Organization] The proposed loss, the algorithm as well as the network structure are suggested to be better formed. This reviewer could follow the whole protocol while it requires checking back and forth between different components. Authors are suggested to clearly present the whole losses in Sec 3.1.1, and provide explanations in the following sections.\n12. [Eq (9)] It should be $D_x$ rather than D. Am I correct?\n\n### Theory and practice gap\n\n1. [Assumption 1] Learnability. I agree this assumption is mild but it is suggested to clearly discuss limitations. As far as I understand, it has two limitations (1) It requires the embedding distribution is noise free such that there exists a deterministic labeling function to recover label y from the embedding.(e.g, see the theoretical result in [2]) (2) it implicitly assumes the label distribution should be uniform. (e.g, see the theoretical result in [3]).\n2. [Assumption 2] I have difficulty in understanding Assumption 2. (1) The $I$ set should be the source samples that have the sample covariate w.r.t. the target in the embedding space. I.e need to find all $z_s^{i} = z_t$ to formulate such a set. This assumption in the embedding space sounds reasonable for me since it is a low dimensional data but it is still a bit difficult since $z_s$ is a continuous random variable. (2) The variational distribution w.r.t. $Z_c$ should be upper bound ( $\\beta$ is positive????) by the corresponding source samples. Upon reflection, I understand that there always exists such a coefficient $\\beta$ to guarantee an upper bound. But it is preferable that the author would state the intuitions behind assumption 2.\n3. [Theory 2]  The derivation seems straightforward for me. However, the bound could be vacuous if the $\\beta$ is arbitrarily large in assumption 2. (e.g, set $\\beta$ = 10^6), which induces a non-meaningful OOD generalization bound. Thus a discussion is expected herein. \n4. [Eq (3)] The posterior loss seems not a strong tie with the task, rec and gans loss. Clearly, the posterior loss is purely unsupervised, the variational term does not include the label information. The authors are suggested to make clear illustration between the posterior loss and rec and gans loss. (hint: The prediction loss could be introduced by the information bottleneck/information projection principle to maximize the correlation between $z_c$ and $y$. Or express a joint variable $(x,y)$ in the variational inference. )\n5. [Appendix A.1] The graph seems inaccurate. I think it should be in the anti-causal direction. I.e, y -> z_c -> x <- z_d, then the generative model could model this problem without considering the shift of P(y). This is also aligned with Assumption 1.\n6. [Assumption A.2] I could not understand how q(x|z_c) is bounded. As far as I understand, \n$$q(x|z_c) / q(x) = q(x|z_c)/\\sum_{z_c} q(z)q(x|z_c) \\leq 1/q(z_c)=M,$$ \nthus it should be q(z_c) is positive within its distribution support. Since the upper bound depends on M, thus the upper bound could be involved with the prior distribution q(z_c) and computationally feasible (consider q(z_c) being gaussian for example).\n\n### Practical aspects\n1. If I understand correctly, I think the proposed loss could be fully unsupervised. Is it possible to generate the interpolated samples in Appendix C.2.2 without supervision? This question is for the discussion propose, no additional experiments are required. \n2. It seems the proposed approach has multiple hyper-parameters. The authors are suggested to illustrate the tuning strategy.\n3. [Tab.12] The PACS (S) is surprisingly better than all other baselines (>10%), are there any possible explanations? \n4. As for generated images (C2.2 in appendix), is it possible to illustrate the failure results? I.e some images are poorly generated. I think it would be quite interesting to see such results. \n\nRef \n\n[1] On the benefits of representation regularization in invariance based domain generalization. Machine Learning Journal, 2022.\n\n[2] Domain adaptation with conditional distribution matching and generalized label shift. NeurIPS 2020\n\n[2] Aggregating From Multiple Target-Shifted Sources. ICML 2021\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addressed the OOD (or domain generalization) problem through the ​​disentanglement and variational method.  Specifically, they aim to disentangle style and semantic information, then the framework could both generate new images and improve the prediction accuracy on the unseen domain. Extensive empirical results validated the framework.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall this paper has multiple merits, however, several parts lack discussions/analysis/proper organizations. Based on the aforementioned reviews and guidelines of TMLR, I would suggest the following main revisions:\n\n1. Paper organization/writing. Authors are suggested to revise them. (Q1-12)\n2. Theory and practice gap. Authors are suggested to include discussions or adjust their claims within the paper, to accurately illustrate the contributions. (Q1-6)\n3. Practical aspects. Ideally, *authors are not required to conduct additional experiments but adding relevant discussions/analysis*. As for the failure images, authors are encouraged to propose analysis from the current failure results (if there are).(Q2-4, Q1 for the discussion propose)\n", "broader_impact_concerns": "N/A. Authors are encouraged to make a short statement to illustrate the societal benefits (e.g., mixing style could mitigate stereotype) or risks (e.g, inducing bias) of the proposed framework. \t", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers disentangling the label information and domain style for addressing the domain generalization problem better. The general idea could lead more papers to focus on the solvability of the DG problem, which is good for the area. They also first give a theoretical bound and then design their algorithm, making the whole algorithm sound and theoretical solid. The sufficient experiments verify that the proposed method works better than existing sota methods sota experiments. In general, the main idea of this paper and the theory proposed in this paper are interesting. The authors address reviewers' concerns well in their rebuttal. Thus, I would like to recommend accepting the submission as is.
Decision: Accept (poster)