Domain Invariant Representation Learning with Domain Density Transformations
Authors: A. Tuan Nguyen
Toan Tran
Yarin Gal
Atilim Gunes Baydin
Venue: NeurIPS 2021 Poster (2021)
Keywords: domain generalization
domain invariant representation

Abstract: Domain generalization refers to the problem where we aim to train a model on data from a set of source domains so that the model can generalize to unseen target domains. Naively training a model on the aggregate set of data (pooled from all source domains) has been shown to perform suboptimally, since the information learned by that model might be domain-specific and generalize imperfectly to target domains. To tackle this problem, a predominant domain generalization approach is to  learn some domain-invariant information for the prediction task, aiming at a good generalization across domains. In this paper, we propose a theoretically grounded method to learn a domain-invariant representation by enforcing the representation network to be invariant under all transformation functions among domains. We next introduce the use of generative adversarial networks to learn such domain transformations in a possible implementation of our method in practice. We demonstrate the effectiveness of our method on several widely used datasets for the domain generalization problem, on all of which we achieve competitive results with state-of-the-art models.

Domain Invariant Representation Learning with Domain Density Transformations
A. Tuan Nguyen
Toan Tran VinAI Research Hanoi, Vietnam
University of Oxford: VinAI Research Oxford, United Kingdom
tuan@robots.c ox. wx.accu... ac.uk uk
v. -.toontm@@vinai.ii
Yarin Gal University of Oxford Oxford, United Kingdom yarinêcs ox. ac. uk
Atilim Gunes Baydin
University Oxford
Oxford, United Kingdom gunesdrobots. ox. ac uk
Abstract
Domain generalization refers to the problem where we aim to train model on data from a set of source domains so that the model can generalize to unseen target domains. Naively training a model on the aggregate set of data (pooled from all source domains) has been shown to perform suboptimally, since the information learned by that model might be domain-specnt speeiffc and generalize imperfectly to target domains. To tackle this problem, predominant domain generalization approach is to learn some domain-invariant information for the prediction task, aiming at good generalization across domains. In this paper, we propose theoretically grounded method to learn a domain-invariant representation by enforcing the representation network to be invariant under all transformation functions among domains. We next introduce the use of generative adversarial networks to learn such domain transformations in possible implementation of our method in practice. We demonstrate the effectiveness of our method on several widely used datasets for the domain generalization problem, all of which achieve competitive results with state-of-the-a models.
Introduction
Domain generalization refers to the machine learning scenario where the model trained on multiple source domains that it is expected to generalize well to unseen target domains. The key difference between domain generalization [25, 37, 18] and domain adaptation [49, 48, 14, 45] is that, in domain generalization, the learner does not have access data of the target domain, making the problem much more challenging. One the most common domain generalization approaches to learn an invariant representation across domains, aiming at good generalization performance target domains. For instance, in the representation learning framework, the prediction function = f(x), where x is data and y is label, is obtained composition y h ((x) of deep representation network g(x), where is learned representation of data æ. and a smaller classifier = h(2), predicting label given representation 2, both of which are shared across domains. With this framework, can aim to learn an "invariant" representation across the source domains with the "hope" of a better generalization the target domain.
Most existing distribution alignment [37, 1, 44, 43, 32], which are still prone to distributional shifts when the conditional data distribution not stable. In particular, the marginal alignment refers to making the
-based methods in domain generalization focus on the marginal
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

data test time. Conditional alignment refers to aligning the conditional Domain x/|kll, withcolor indicatingy
Domain 2:2 x/ |x|], with indicating
p(x
p(x)
P())
0
-2
Figure 1: An example of two domains. For each domain, x: uniformly distributed the outer circle (radius 2 domain and radius 3 for domain 2). with color indicating class label y. After transformation = x/ =/||=||a, the marginal of is aligned (uniformly distributed on the unit circle for both domains), but the conditional p(v|z) is not aligned. Thus, using this representation for predicting y would not generalize well across domains.
representation distribution p(2) to be the same across domains. This is essential since if p(2) for the target domain different from that of source domains, the classification network h(2) would face
distribution of the label given the representation p(y|z) to expect that the classification network (trained the source domains) would give accurate predictions at test time. The formal definitions of these two types of alignment are discussed in Section 3.
In Figure I illustrate example where the representation satisfies the marginal alignment but not the conditional aligument. Specifically, x distributed uniformly on the circle with radius 2 (and centered at the origin) for domain I and distributed uniformly on the circle with radius 3 (centered at the origin) for domain The representation 2 defined by the mapping 2 = g(x) = x x/||æ/|a la will the unit circle for both
align the marginal distribution p(2), i.c.. 2 is distributed uniformly domains. However, the conditional distribution p(yle 2) is not aligned between the two domains (y is represented by color), which means using this representation for classification is suboptimal, and in this extreme case would lead to 0% accuracy in the target domain 2. This is an extreme case of misalignment but it does illustrate the importance of the conditional alignment. Therefore, need to align both the marginal and the conditional distributions for a Nomain-invarian representation.
Recently, there have been several attempts [33, 34, 50] to align the joint distribution of the repre- sentation and label p(y, =) a domain generalization problem by aligning the distribution of across domains for each class, i.e., p(>[v) (given that the label distribution p(y) is unchanged across domains). However, the key drawbacks of these methods are that they either do not scale well with the number of classes or have limited performance in real-world computer vision datasets (see Section 5).
In this paper, we focus on learning a Nomain-invariin representation that aligns both the marginal and the conditional distributions in domain generalization problems. We present theoretical results re- garding the necessary and sufficient conditions for the existence a domain-invariant representation; and subsequently propose method to learn such representations by enforcing the invariance of the representation network under domain density transformation functions. A simple intuition for our approach is that if we enforce the representation to be invariant under the transformations among the source domains, the representation will become more robust under other domain transformations.
Furthermore, we introduce an implementation of our method in practice, in which the domain transformation functions are learned through the training process of generative adversarial networks (GANs) [20, 12]. We conduct extensive experiments on several widely used datasets and observe a significant improvement over relevant baselines. We also compare our methods against other state-of-the-aa models and show that our method achieves competitive results.
Our contribution in this work is threefold:
We revisit the domain invariant representation learning problem and shed some light by providing several observations: necessary and sufficient condition for the existence of

---

invaaiiinn representation and a connection between tion and a eenginally-aligneed representation.
representa-
We propose theoretically grounded method for learning a domain-invarian representation based on domain density transformation functions. We also demonstrate that we can learn the domain transformation functions by GANs in order to implement our approach in practice.
We empirically show the effectiveness of our method by performing experiments widely used domain generalization datasets (e.g., Rotated MNIST, VLCS and PACS) and compare our method with relevant baselines (especially CIDG [33], CIDDG [34] and DGER [50]).
2 Related Work
Domain generalization: Domain generalization is an seminal task in real-world machine learning problems where the data distribution of a target domain might vary from that of the training source domains. Therefore, extensive research has been developed to handle that domain-shift problem, aiming at a better generalization performance in the unseen target domain. A predominant approach for domain generalization is domain invariance [37, 33, 34, 3, 47, 2. 24, 50, I, 32, 44, 43] that learns a domain-invariant representation (which we define as to align the marginal distribution of the representation or the conditional distribution of the output given the representation or both). We are particularly interested in CIDG [33], CIDDG [34] and DGER [50], which also learn representation that aligns the joint distribution of the representation and the label given that the class distribution is unchanged across domains. It should be noted that Zhao et al. [50] assume the label is distributed uniformly on all domains, while our proposed method only requires an assumption that the distribution of label is unchanged across domains (and not necessarily uniform). We also show later in our paper that the invariance of the distribution of class label across domains turns out to be the necessary and sufficient condition for the existence of domain-invarian representation. Moreover, we provide a unified theoretical discussion about the two types of alignment, and then propose a method to learn a representation that aligns both the marginal and conditional distributions via domain density transformation functions for the domain generalization problem. Note that there exist several related works, such as Ajakan al. [1]. Ganin et al. [17], that use adversarial loss with a domain discriminator to align the marginal distribution of representation among domains, but they are different from our approach. In particular, our method only uses GANs or normalizing flows to learn the transformation among domains, and learn a representation that is invariant under these functions, without using an adversarial loss on the representation (which can lead to very unstable training [19, 27)). There also exist works [35, 23, 7, 40] in the domain adaptation literature that use generative modeling to learn a domain transformation function from source to target images, and use the transformed images to train classifier. Our method differs from these by enforcing the representation to be invariant under the domain transformation, and we show theoretically that the representation learned that way would be domain-invariant marginally and conditionally, Meanwhile, the above works use the domain transformation to transform the images and train the classifier directly on the transformed data, and are not effective or applicable for domain generalization.
Another line of methods that received a recent surge in interest applying the idea of meta-learning for domain generalization problems [16, 4, 31, 5]. The core idea behind these works is that if we train a model that can adapt among the source domains well, would be more likely to adapt to unseen target domains. Recently, there are approaches [15, 9, 41] that make use of the domain specificity, together with domain invariance, for the prediction problem. The argument here is that domain invariance, while being generalized well between domains, might be insufficient for the prediction of each specific domain and thus domain specificity is necessary. We would like to emphasize that our method is not direct competitor of meta-learning based and domain- specificity based methods. In fact, we expect that our method can be used in conjunction with these methods to get the best of both worlds for better performance.
Density transformation between domains: Since our method is based on domain density trans- formations, we will review briefly some related works here. To transform the data density between domains, one can use several types of generative models. Two common methods are based on GANs [51, 12, 13] and normalizing flows [21]. Although our method is not limited to the choice of the generative model used for learning the domain transformation functions, we opt to use GAN,
3

---

Figure 2: Graphical model. The data distribution is p(z, y for each domain d. Our goal is to learn representation with mapping p(z[zr) from that can be generalized across domains for the prediction task.
specifically StarGAN [12], for scalability. This is just an implementation choice to demonstrate the use and effectiveness of our method in practice, and it is unrelated to our theoretical results.
Connection contrastive learning: Our method can be interpreted intuitively as a way to learn representation network that is invariant (robust) under domain transformation functions. On the other hand, contrastive learning [10, 11, 36] also representation learning paradigm where the model learns images' similarity. In particular, contrastive learning encourages the representation of an input to similar under different transformations (usually image augmentations). However, the transformations in contrastive learning are not learned and do not serve the purpose of making the representation robust under domain transformations. Our method first learns the transformations between domains and then uses them to learn representation that is invariant under domain shifts.
3 Theoretical Approach
3.1 Problem Statement
Let us denote the data distribution for a domain d E 'D by p(r, yld), where the variable x E X represents the data and y E Y is its corresponding label. The graphical model for our domain generalization framework is depicted in Figure 2, in which the joint distribution is presented as follows:
p(d, x, 2) p(d)p(())))))))e d))(2//))
In the domain generalization problem, since the data distribution p(a, y|d) varies between domains, we expect the changes in the marginal data distribution p(z|d) or the conditional data distribution p(y|z, or both. In this paper, we assume that plyld) is invariant across domains, i.e., the marginal distribution of the label y is not dependent on the domain d-this assumption is shown to be the key condition for the existence of a comain-invariant representation (see Remark 1). This is practically reasonable since in many classification datasets, the class distribution can be assumed to be unchanged across domains (usually uniform distribution among the classes, e.g., balanced datasets).
Our aim is to find a invariann representation 2 represented by the mapping p(²[z) that can be used for the classification of label y and be generalized among domains. In practice, this mapping can be deterministic (in that case, p(2[z) = (=)()) (2) with some function Ge. where is the Dirac delta distribution) or probabilistic (e.g., a normal distribution with the mean and standard deviation outputted by a network parameterized by 0). For all of our experiments, we use deterministic mapping for an efficient inference at test time, while in this section, we present our theoretical results with the general case of distribution p(z(2:).
In most existing domain generalization approaches, the innaian representation is defined using one of the two following definitions:
Definition 1. (Marginal Distribution Alignment) The representation is said to satisfy the marginal distribution alignment condition if pp(2)d) is invariant w.r.t. d.
Definition 2. (Conditional Distribution Alignment) The representation is said to satisfy the conditional distribution aligument condition p(y) d) is invariant w.r.t. d.
However, when the joint data distribution varies between domains, crucial to align both the marginal and the conditional distribution of the representation z. To this end, this paper aims to

---

Domain 1
Domain 2
fi transforms data densit from domain to
(Wit the inverse (1)
Xg (2(
Figure 3: Domain density transformation. If we know the function f1,2 that transforms the data density from domain I to domain 2, we can learn domain invariant representation network (x( by enforcing it to be invariant under f1.2, i.e., go (21) go (12) for any 22 f1.2(11)
learn representation that satisfies both the marginal and conditional alignment conditions. We justify our assumption of independence between y and (thus p(yld) = p(y)) by the following remark, which shows that this assumption turns out to be the necessary and sufficient condition for learning a comainninvariann representation. Note that this condition is also used in several existing works [50, 33, 34).
Remark 1. The invariance of p(uld) across domains d is the necessary and sufficient condition for the existence of aamaii--nvariin representation (that aligns both the marginal and conditional distributions).
Proof. provided in the appendix.
It is also worth noting that methods which learn domain independent representation, for example, [24], only align the marginal distribution. This comes directly from the following remark:
Remark 2. representation satisfies the marginal distribution alignment condition if and only if
T(2,d) 0, where T(2, d) is the mutual information between and d.
Proof. provided in the appendix.
The question still remains that how we can learn non-trivial domain invariant representation that satisfies both the distribution alignment conditions. This will be discussed in the following subsection.
3.2 Learning a Domain-Invariall Representation with Domain Density Transformation Functions
To present method, will make some assumptions about the data distribution. Specifically, for any two domains d, d', we assume that there exists an invertible and differentiable function denoted by fa.a that transforms the density p(zly, to p(ø" ly. d'), Vy. Let fa,a" be the inverse of fa d: i.e., far = )
Due to the invertibility and differentiability of f's, we can apply the change of variables theorem [39, 6] for the distributions above. In particular, with x' = fa.a (and thus x = far faza(x') (x")) D), we
have:
p(aly, d) p(æ' y, d') det Jia (x')
where Jf (x') is the Jacobian matrix of the function fax evaluated at x'.
Multiplying both sides of Eq. 2 with p(yld) = p(yld'), we get
p(x, yld) = p(a', wldd") det JS (x0) (+)
(3)
5

---

and marginalizing both sides of the above equation over y gives
p(æ|d) = p(x' (d') det J (a')
(4)
By using Eq. and Eq. 4, we can prove the following theorem, which offers an efficient way to learn a domain-invarian representation, given the transformation functions. f's between domains.
Theorem 1. Given an invertible and differentiable function fa. (with the inverse fø that trans- forms the data density from domain d to d' (as described above). Assuming that the representation satisfies:
p<=|=>) a(=) Vx.
(5)
Then it aligns both the marginal and the conditional of the data distribution for domain d and d'.
Proof. provided in the appendix.
This theorem indicates that, if we can find the functions f that transform the data densities among the domains, we can learn a invariant under all the transformations f. This idea illustrated in Figure 3. We therefore can use the following learning objective to learn
representation by encouraging the representation to be
representation 2 g0(x)):
Ed (( [My ge(x)) Ex (dis(go(x), got.).gofaaa.. (x)))]I]
(6)
Assume that we have a set of sources domain D, {d1, d2, dx }. the objective function in Eq. 6 becomes:
Ed D. p(a.wld) ((()))(x)) - dis(go(x), (o)o(())ogl))) (f (z)))).
where I(y, go (z)) is the prediction loss of a network that predicts y given = (z)),, and dis is a distance metric to enforce the invariant condition in Eq. 5. In our implementation,
squared error distance, e.g., dis(go(z), go(Sa. (x))) = [[go(x) - (f(.a.a(= (x)) since it performs the best in practice. However, also considered other distances such constrastive distance, which discuss in more detail in the appendix.
This theorem motivates us to learn such domain transformation functions for our domain-invarianl representation learning framework. In the next section, we show how one can incorporate this idea into real-world domain generalization problems by learning the transformations with generative adversarial networks.
4 An Practical Implementation using Generative Adversarial Networks
In practice, we can learn the functions f that transform the data distributions between domains by using several generative modeling frameworks, c.g., normalizing flows [21] or GANs [51, 12, 13]. One advantage of normalizing flows is that this transformation naturally invertible by design of the neural network. However, existing frameworks (e.g.. Grover et al. [21]) require two flows to transform between each pair of domains, making it not scalable (scales linearly with the number of domains). Moreover, initial implementation of our method using AlignFlow shows similar performance with the version using GAN. Therefore, we opt to use GANs for better scalability. In particular, we use the StarGAN [12] model, which is a unified network (only requiring single network to transform across all domains) designed for image domain transformations. should be noted that the transformations learned by StarGAN are differentiable everywhere or almost everywhere with typical choices of the activation function (c.g., tanh or ReLU), and the cycle-consistency loss enforces each pair of transformations to approximate pair of inverse functions.
The goal of StarGAN
learn unified network G that transforms the data density among multiple domains. In particular, the network G(z, d, d') (i.e., G is conditioned on the image x and the two different domains d, d') transforms an image x from domain d to domain d'. Different from the original StarGAN model that only takes the image x and the desired destination domain d' as its input, in our implementation, we feed both the original domain and desired destination domain d' together with the original image to the generator G.

---

The generator's goal is to fool a discriminator D into thinking that the transformed image belongs to the destination domain d'. In other words, the equilibrium state of StarGAN, in which G completely fools D, is when G successfully transforms the data density of the original domain to that of the destination domain. After training, we use G(., d, d') as the function fa.a (.) described in the previous section and perform the representation learning via the objective function in Eq. 7.
Three important loss functions of the StarGAN architecture are:
Domain classification loss Cete that encourages the generator to generate images that closely belongs to the desired destination domain d'.
The adversarial loss fade that is the classification loss a discriminator D that tries to distinguish between real images and the synthetic images generated by G. The equilibrium state of StarGAN is when G completely fools D. which means the distribution of the generated images (via G(z, d, d'), p(rld)) becomes the distribution of the real images of the destination domain p(z' [d"). This is our objective, i.e., to learn function that transforms domains' densities.
Reconstruction loss Free d ||| G(z', d', d]Ili] where x' G(x, d, d') to ensure that the transformations preserve the image's content. Note that this also aligns with our interest since want G(.,d', d) to be the inverse of G(., d"), which minimizes Cree-
We can enforce the generator G transform the data distribution within the class y (e.g.. p(xly, d) to p(a' ly. d') Vy) by sampling each minibatch with data from the same class y, so that the discriminator will distinguish the transformed images with the real images from class y and domain d'. However, we found that this constraint can be relaxed in practice, and the generator almost always transforms image within the original class y/.
As mentioned earlier, after training the StarGAN model, we can use the generator Gi., d, d') as our fa.d' (.) function and learn comain-invarian representation via the learning objective in Eq. 7. We name this implementation of our method DIRT-GAN (Domain Invariant Representation learning with domain Transformations via Generative Adversarial Networks).
5 Experiments
5.1 Datasets
To evaluate method, we perform experiments in three datasets that are commonly used in the literature for domain generalization.
Rotated MNIST [18]: In this dataset, 1,000 MNIST images (100 per class) [29] are chosen to form the first domain (denoted Ma), then counter-clockwise rotations of 15", 30%, 45", 60% and 75 applied create five additional domains, denoted M1s, Mao. Mas. Mso and M7s. The task is classification with ten classes (digits 0 to 9).
VLCS [18]: contains 10,729 images from four domains, each domain is subdataset. The four datasets are VOC2007 (V), LabelMe (L., Caltech-101 (C), and SUN09 (S). The task is classification with five classes.
contains 9,991 images from four different domains: art painting, cartoon, photo, sketch. The task is classification with seven classes.
PACS [30]:
5.2 Experimental Setting
For all datasets, perform "feave-one-domain-uu" experiments, where choose one domain as the target domain, train the model on all remaining domains and evaluate it on the chosen domain. Following standard practice, use 90% of available data training data and 10% as validation data, except for the Rotated MNIST experiment where we do not use a validation set and just report the performance of the last epoch.
For the Rotated MNIST dataset, we use a network of two 3x3 convolutional layers and fully connected layer the representation network go to get representation of 64 dimensions. A single
7

---

Table 1: Rotated Mnist. Reported numbers accuracy and standard deviation among 5 runs
Domains
Model
Ma
M15
Ma0
Mas
M60
M7s
Average
HIR [47]
90.34
99.75
99.40
96.17 17
99.25
91.26
96.03
DIVA [24]
93.5
99.3
99.1
99.2
99.3
93.0
97.2
DGER [50]
90.09
99.24
99.27
99.31
99.45
90.81
96.36
DA [17]
86.7
98.0
97.8
97.4
96.9
89.1
94.3
LG [42]
89.7
97.8
98.0
97.1
96.6
92.1
95.3
HEX [46]
90.1
98.9
98.9
98.8
ADV [46]
89.9
98.6
98.8
98.7
98.6
90.4
95.2
DIRT-GAN (ours) 97.2(+10.3) 99.4(10.1) 99.3(110.1) 99.3(10.1) 99.2(110.1) 97.1(10.3)
component PCA
component
component PCA
component: PCA
Piiccpo Component
Princ Pallpall Can pompontnt
Principll Component
Princ
Principal Component
Principal Component
Prticipal Componient
Prinippl Component
(n) Map DIRT-GAN
(b) Domain Mao DIRT-GAN
(6) Domain Map DespAll
(d) Domaii Mgo DespAll
Figure 4: Visualization of the representation space. Each point indicates a representation z an image the two dimensional space and its color indicates the label Two left figures are for our method DIRT-GAN and two right figures are for the naive model DeepAll.
linear layer is then used to map the representation to the ten output classes. This architecture the deterministic version of the network used by lsse et al. [24]. We train our network for 500 epochs with the Adam optimizer [26], using the learning rate 0.001 and minibatch size 64, and report performance on the test domain after the last epoch.
For the VLCS and PACS datasets, for fair comparison against our main baselines, we the most common choices of backbone networks for those datasets in existing works as the representation networks ge. i.e., Alexnet [28] for VLCS and Resnet18 [22] for PACS. We replace the last fully connected layer of the backbone with linear layer of dimension 256 so that our representation has 256 dimensions. As with the Rotated MNIST experiment, we use single layer to map from the representation to the output. We train the network for 100 epochs with plain stochastic gradient descent (SGD) using learning rate 0.001, momentum 0.9, minibatch size 64, and weight decay 0.001. Data augmentation is also standard practice for real-world computer vision datasets like VLCS and PACS, and during the training we augment our data as follows: crops of random size and aspect ratio, resizing to 224 224 pixels, random horizontal flips, random color jitter, randomly converting the image tile to grayscale with 10% probability, and normalization using the ImageNet channel means and standard deviations.
The StarGAN [12] model implementation taken from the authors' original source code with no significant modifications. For each set of source domains, we train the StarGAN model for 100,000 iterations with minibatch of 16 images per iteration.
Our code available at http://gthhb..ommaaaaggggee//// We train our model on a NVIDIA Quadro RTX 6000.
5.3 Results
Rotated MNIST Experiment. Table I shows the performance of our model on the Rotated MNIST dataset. The main baselines we consider this experiment are HIR [47], DIVA [24] and DGER [50], which are domain invariance based methods. Our method recognizably outperforms those, illustrating

---

Table 2: VLCS. Reported numbers are accuracy and standard deviation among
VLCS
Model
Backbone
C
S
Average
CIDG [33]
Alexnet
65.65
60.43
91.12
60.85
69.51
CIDDG [34]
Alexnet
64.38
63.06
88.83
62.10
69.59
DGER [50]
Alexnet
73.24
58.26
96.92
69.10
74.38
HIR [47]
Alexnet
69.10
62.22
95.39
65.71
73.10
JiGen [8]
Alexnet
70.62
60.90
96.93
64.30
73.19
DIRT-GAN (ours)
Alexnet
72.1(#1.0)
64.0(=0.9)
97.3(+0.2)
72.2(+1.1)
76.4
Table 3: PACS. Reported numbers are mean accuracy and standard deviation among rums
PACS
Model
Backbone
Art Painting
Cartoon
Photo
Sketch
Average
DGER [50]
Resnet18
80.70
76.40
96.65
71.77
81.38
JiGen [8]
Resnet18
79.42
75.25
96.03
71.35
79.14
MLDG [31]
Resnet18
79.50
77.30
94.30
71.50
80.70
MetaReg [4]
Resnet18
83.70
77.20
95.50
70.40
81.70
CSD [38]
Resnet18
78.90
75.80
94.10
76.70
81.40
DMG [9]
Resnet18
76.90
80.38
93.35
75.21
81.46
DIRT-GAN (ours)
Resnet18
82,56(:: 0.4)
76.37( 0.3)
95.65(d 0.5)
79.89( 0.2)
83.62
the effectiveness of method in learning domain-invarian representation the existing works. We also include other best-performing models for this dataset the second half of the table. To the best of our knowledge, we set new state-of-the-art performance this Rotated MNIST dataset.
further analyze the distribution of representation by performing principal component analysis to reduce the dimension of from 64 two principal components. We visualize the representation space for two domains M30 and Moo. with each point indicating the representation in the co--immnnsional space and its color indicating the label y. Figures 4a and 4b show the representation space our method (in domains M3n and M60 respectively). the marginal (judged by the general distribution of the points) and the conditional (judged by the positions of colors) are relatively aligned. Meanwhile, Figures 4c and 4d show the representation space with naive training (in domains M30 and Meo respectively), showing the misalignment in the marginal distribution (judged by the general distribution of the points) and the conditional distribution (for example, the distributions of blue points and green points).
an image
clear that both
VLCS and PACS. Tables and 3 show the results for the VLCS and PACS datasets. In these real-world computer vision datasets, we consider HIR [47], CIDG [33], CIDDG [34] and DGER [50] our main domain-invariance baselines. We also include other approaches (meta-learning based or nomain-speciffenn based) in the second half of the tables for references. Our method significantly ourperforms other rearetantrrprresenttttiion baselines, namely CIDG, CIDDG and DGER, with the same backbone architectures, showing the effectiveness of our representation alignment method.
Conclusion
To conclude, in this work we propose a theoretically grounded approach to learn a domain-invariant representation for the domain generalization problem by using domain transformation functions. We also provide insights into domain- innariann representation learning with several theoretical observations. We then introduce implementation for our method in practice with the domain trans- formations learned by a StarGAN architecture and empirically show that our approach outperforms other domain-in inniiiincev based methods. Our method also achieves competitive results on several datasets when compared to other state-of-the-art models. potential limitation of our method is that need to train an additional network (StarGAN) to learn to transform data density among domains.

---

However, this network is only used during training, and the required computation at test time is still the same as other models. In the future, we plan to incorporate our method into meta-learning based and domain-specificii based approaches for improved performance. We also plan to extend the domain-invariant representation learning framework to the more challenging scenarios, for example, where domain information is not available (i.e., we have dataset pooled from multiple source domains but do not know the domain identification of each data instance).
References
[1] H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, and M. Marchand. Domain-adversariall neural networks. arXiv preprint arXiv:1412.4446, 2014.
[2] K. Akuzawa, Y. Iwasawa, and Y. Matsuo. Adversarial invariant feature learning with accuracy constraint for domain generalization. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 315-331. Springer, 2019.
[3] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paza Invariant risk minimization. arXiv preprint 1TXhv:1907.02893, 2019.
[4] Y. Balaji, S. Sankaranarayanan, and R. Chellappa. Metareg: Towards domain generalization using meta-regularrizatoon Advances Neural Information Processing Systems, 31:998-1008, 2018.
[5] H. Behl, A. G. Baydin, and H. Torr. Alpha maml: Adaptive model-agnostic meta-learning In 6th ICML Workshop on Automated Machine Learning. Thirty-sixth International Conference on Machine Learning (ICML 2019), Long Beach, CA, US, 2019.
[6] V. I. Bogachev. Measure theory, volume 1. Springer Science & Business Media, 2007.
[7] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan. Unsupervised pixel-level domain adaptation with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3722-3731, 2017.
[8] F. M. Carlucci, A. D' D'Innocente, S. Bucci, B. Caputo, and T. Tommasi. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2229 2238, 2019.
[9] P. Chattopadhyay, Y. Balaji, and J. Hoffmman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference on Computer Vision, pages 301-318. Springer, 2020.
[10] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[11] T. Chen, S. Korblithh K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
[12] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-imant translation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 8789-8799 2018
[13] Y. Choi, Y. Uh, J. Yoo, and J.W. Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pages 8188-8197 2020.
[14] R. d. Combes, H. Zhao, Y.-.. Wang, and G. Gordon. Domain adaptation with conditional distribution matching and generalized label shift. arXiv preprint arXiv:2003.04475, 2020.
[15] Z. Ding and Y. Fu. Deep domain generalization with structured low-rank constraint. IFEE Transactions on Image Processing, 27(1):04-3313
10

---

[16] Y. Du, J. Xu, H. Xiong, Q. Qiu, X. Zhen, C. G. Snoek, and L. Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pages 200-216. Springer, 2020.
[17] Y. Ganin, Ustinova, H. Ajakan, Germain, Larochelle, Laviolette, M. Marchand, and V. Lempitsky. Domain-adversaria training of neural networks. The Journal of Machine Learning Research, 17(1):2096-222 -030,, 2016.
[18] M. Ghifary, W. B. Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition with multi-task autoencoders. Proceedings of the IEEE International Conference on Computer Vision, pages 2551-2559, 2015.
[19] I. Goodfellow, Nips 2016 tutorial: Generative adversarial networks. arXiv preprint
arXiv:7011.00160, 2016.
[20] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint arXiv: 2661, 2014.
[21] A. Grover, C. Chute, R. Shu, 7. Cao, and S. Ermon. Alignflow: Cycle consistent learning from multiple domains via normalizing flows. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4028-4035, 2020.
[22] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[23] J. Hoffman,, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998 PMLR, 2018.
[24] M. Ilse, J. M. Tomezak, C. Louizos, and M. Welling. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning, pages 322-348. PMLR, 2020.
[25] A. Khosla, T. Zhou, T. Malisiewicz, A. A. Efros, and A. Torralba. Undoing the damage of dataset bias. In European Conference Computer Vision, pages 158-171. Springer, 2012.
[26] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1r2iv:1412.6980, 2014.
[27] N. Kodali, J. Abermethy, J. Hays, and Z. Kira. On convergence and stability gans. arXiv preprint arXiv: 2017.
[28] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[29] Y. LeCun and C. Cortes. MNIST handwritten digit database. 2010. URL http: / l com/ com/exdb//ee mxddb/mnit//..
[30] D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales. Deeper, broader and artier domain generalization. In International Conference on Computer Vision, 2017.
[31] D. Li, Y. Yang, Y.Z. Song, and T. Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[32] H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400-5409, 2018.
[33] Y. Li, M. Gong, X. Tian, T. Liu, and D. Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
11

---

[34] Y. Li, X. Tian, M. Gong, Y. Liu, T. Liu, K. Zhang, and D. Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 624-639, 2018.
[35] M.-Y. Liu and O. Tuzel. Coupled generative adversarial networks. arXiv preprint arXiv:1606.07558 07536, 2016.
[36] I. Misra and L. v. d. Maaten. Self-supervised learning pretext-invariant representations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[37] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International Conference Machine Learning, pages 10-18. PMLR, 2013.
[38] V. Piratla, P. Netrapalli, and S. Sarawagi. Efficient domain generalization via common-specific low-rank decomposition. In International Conference on Machine Learning, pages 7728-7738. PMLR, 2020.
[39] W. Rudin. Real and complex analysis. Tata McGraw-hill education, 2006.
[40] Russo, M. Carlucci, T. Tommasi, and B. Caputo. From source target and back: symmetric bi-directional adaptive gan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8099-8108 8108, 2018.
[41] Seo, Suh, D. Kim, Han, and B. Han. Learning optimize domain specific normalization for domain generalization. arXiv preprint arXiv: 1907.042557 3(6):7, 2019.
[42] S. Shankar, V. Piratla, S. Chakrabarti, S. Chaudhuri, P. Jyothi, and S. Sarawagi. Generalizing across domains via cross-gradient training. arXiv preprint arXiv: 10745, 2018.
[43] Shen, Y. Qu, W. Zhang, and Y. Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[44] B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 450. Springer, 2016.
[45] A. K. Tanwani. Domain-invariant invariant representation learning for sim-to-real transfer. arXiv preprint arXiv:2011.0758% 2020.
[46] H. Wang, Z. He, 7. C. Lipton, and F. P. Xing. Learning robust representations by projecting superficial statisties out, arXiv preprint arXiv: /903. 062566, 2019.
[47] Z. Wang, M. Loog, and J. van Gemert. Respecting domain relations: Hypothesis invariance for domain generalization. arXiv preprint arXiv:2010.0759) 2020.
[48] Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation.
In International Conference on Machine Learning, pages 7404-7413. PMLR, 2019.
[49] H. Zhao, R. T. Des Combes, Zhang, and G. Gordon. On learning invariant representations for domain adaptation. In International Conference on Machine Learning, pages 7523-7533 PMLR, 2019.
[50] Zhao, M. Gong, T. Liu, H. Fu, and D. Tao. Domain generalization via entropy regularization. Advances in Neural Information Processing Systems, 33, 2020.
[51] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
12


Reviews:
[{"review_id": "lfOHMCyADWs", "reviewer": "Reviewer_RKBf", "summary": "The paper proposes a novel method, DRIT-GAN, to learn domain invariant representation for domain generalization.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The task of domain generalization (DG) is classic. The proposed method leverages the well-known technique, StarGAN. The novelty is limited. \n\nQuality: Theoretically, I am afraid that the paper is not technically sound. Remarks 1 & 2 are trivial and immediate from the definition. The author claims several prior works used those conditions without noting them. However, I am not clear about the value of remarks 1 & 2 in developing the proposed method. Here are questions to the author:\n1. You use remark 1 to justify your assumption that y and d are independent. So how does the assumption of y and d being independent guide your algorithm design? Will your method be broken if y and d are not independent?\n2. By remark 2, the author tries to say that enforcing representation z to be independent of domain d is not enough since it only aligns the marginal. However, most prior methods about DG not only enforce 'z' and 'd' being independent. They also use z to predict y. So at least implicitly, they are aligning the conditional distribution. Does the author aware of it?\n\nThen the author put a new assumption, existing invertible transformations between domains, and proposes a method that works under this assumption.\n1. How do you validate your assumption? How likely it satisfies in actual practice?\n2. I notice the author assumes that the transformations only depend on data x, domain d, but are independent of label y. Can you justify why it is independent of y?\n\nThe theorem says the proposed method is valid (correct), but it does not say it is optimal. Theorem gives a sufficient condition but not a necessary one. So I do curious about the optimality of the method. Let's consider the motivating example that the author presented in Figure 1. Does your method work in this case? Can StarGAN learn the correct transformation between two domains? I feel the StarGAN also has the risk of being trapped into the suboptimal as the author described in the introduction and ending with 0% accuracy.\n\nEmpirically, the author does a good job of showing performance improvement. Clearly, your method shows an advantage over prior works on three benchmarks. A deeper analysis would be good if the author can shed more light on why the proposed method improves the performance.\n\nClarity: The paper is well-written. As a reader, I can easily follow it.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned, the paper has limited depth in theory. The author could try to think about theoretical analysis that bounding your model's error. For example, what if your StarGAN cannot perfectly learn the transformation? How its error of learning transformation translates to the error of your final prediction model?", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "CidLf7Tacz", "reviewer": "Reviewer_TszG", "summary": "The paper presents a method for domain generalization in the presence of multiple source domains.  Given a pre-trained domain translation model f, the proposed method regularizes ERM by minimizing the l2 distance between  g(x) and its translation in domain d'; g(f_d'(x)) , where g is the feature extractor. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n-------------------------------\nThe idea of using domain translation as an intervention to learn invariant features is interesting. I think this idea could be expanded to include arbitrary transformations for arbitrary domains. \n\nWeaknesses of the paper:\n-----------------------------------\n- The authors designate a complete page on using invertible function for domain translation but finally opt for stargan. I don't believe stargan learns invertible functions. \n- shouldn't  the goal is to learn invariant functions for p(y|g(x))? I.e for any domain transformation f, p(y|g(x) and p(y|g(f(x))) should be the same. In other words, why is the regularization done in feature space? A better discussion on this is needed. \n- since we do have access to labels in the source domain, what would happen if we just minimize “dist(g(x_{d1}|y=c), g(x_{d2}|y=c))” where d1 and d2 represent domain 1 and 2. Probably this is a big regularization on the model since this way the model would learn common information in x_d1 and x_d2 which relate to class c and this may not generalize well to other domains but an experiment like this could act as an ablation study and highlight the merits of the proposed approach. \n- the paper lacks comparison with IRM and IRM based methods.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly talk about the limitation of their work as having to train an additional domain translation method (STARGAN). \nAnother potential limitation of the model could be its ability to deal with label distribution shift. An experiment is needed to verify if that is indeed the case. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "51kIm7Tozuu", "reviewer": "Reviewer_qiBT", "summary": "This work focuses on solving domain generalization. Given labeled data of multiple source domains, the goal is to predict the labels of target domain data. In domain generalization, the target domain data are not provided at training time. The main challenge of domain generalization is to learn a domain-invariant representation. To solve the problem, most existing approaches focus on the marginal distribution alignment p(z|d) without considering the conditional distribution p(y|z,d). However, it is needed to align both the marginal and the conditional distributions to classify the target data. To address the challenge, they propose a theoretically grounded method for learning a domain-invariant representation based on domain density transformation functions. They enforce the representation to be invariant under the transformations among the source domains to the representation become more robust under other domain transformations. It aligns both the marginal and the conditional distributions. The transformation functions are based on GANs in practice.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Most existing approaches only focus on the marginal distribution alignment. They point out the limitation of the previous approaches with an eye-catching illustration.\n- They propose a novel approach to align both the marginal and the conditional distributions with a theoretical analysis.\n- They empirically show the effectiveness of the proposed method by performing experiments on real-world datasets. The proposed method outperforms the relevant baselines.\n\nWeaknesses\n- They need to more clarify the differences between the meta learning and domain generalization, and to compare briefly the existing approaches for meta learning and their approach.\n- They need to train a complex network (StarGAN) to align the distributions. The classification performance will be greatly affected by training the network.\n- They need to explain the reasons for the performance differences with the key ideas of the baselines.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- They clearly present the limitations of the proposed method.\n- The proposed method is heavily dependent on the transformation network performance. Thus, it requires great effort to set up and train the transformation network.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to align both the marginal and the conditional distributions with a theoretical analysis. Authors have done a good job to address many concerns from reviewers. After the discussion period, all three reviewers are inclined to accept the paper (note that one increased the score from 4 to 6 without editing the original review.) 
Decision: Accept (Poster)