DIVA: Domain Invariant Variational Autoencoder
Authors: Maximilian Ilse
Jakub M. Tomczak
Christos Louizos
Max Welling
Venue:  (2020)
Keywords: representation learning
generative models
domain generalization
invariance

Abstract: We consider the problem of domain generalization, namely, how to learn representations given data from a set of domains that generalize to data from a previously unseen domain. We propose the Domain Invariant Variational Autoencoder (DIVA), a generative model that tackles this problem by learning three independent latent subspaces, one for the domain, one for the class, and one for any residual variations. We highlight that due to the generative nature of our model we can also incorporate unlabeled data from known or previously unseen domains. To the best of our knowledge this has not been done before in a domain generalization setting. This property is highly desirable in fields like medical imaging where labeled data is scarce. We experimentally evaluate our model on the rotated MNIST benchmark and a malaria cell images dataset where we show that (i) the learned subspaces are indeed complementary to each other, (ii) we improve upon recent works on this task and (iii) incorporating unlabelled data can boost the performance even further.

Under review as a conference paper at ICLR 2020
DIVA: DOMAIN INVARIANT VARIATIONAL AUTOEN- CODERS
Anonymous authors Paper under double-blind review
ABSTRACT
We consider the problem of domain generalization, namely, how to learn represen- tations given data from set of domains that generalize data from a previously unseen domain. We propose the Domain Invariant Variational Autoencoder (DIVA), a generative model that tackles this problem by learning three independent latent subspaces, one for the domain, one for the class, and one for any residual variations. We highlight that due to the generative nature of our model we can also incorporate unlabeled data from known previously unseen domains. To the best of our knowledge this has not been done before in domain generalization setting. This property is highly desirable in fields like medical imaging where labeled data is scarce. We experimentally evaluate our model on the rotated MNIST benchmark and malaria cell images dataset where we show that (i) the learned subspaces are indeed complementary to each other, (ii) we improve upon recent works on this task and (iii) incorporating unlabelled data can boost the performance even further.
INTRODUCTION
Deep neural networks (DNNs) led major breakthroughs in variety of areas like computer vision and natural language processing. Despite their big success, recent research shows that DNNs learn the bias present in the training data. As result they are not invariant to cues that are irrelevant the actual task (Azulay & Weiss 2018). This leads to dramatic performance decrease when tested on data from different distribution with different bias.
In domain generalization the goal is to Iearn representations from a set of similar distributions, here called domains, that can be transferred to previously unseen domain during test time. A common motivating application, where domain generalization is crucial, is medical imaging (Blanchard et al.) 2011. Muandet et al.) (213). For instance, in digital histopathology typical task is the classiication of benign and malignant tissue. However, the preparation of histopathology image includes the staining and scanning of tissue which can greatly vary between hospitals. Moreover, a sample from a patient could be preserved in different conditions (Ciompi et al. 2017]. As result, each patient's data could be treated as a separate domain (Lafarge et al. 2017). 7). Another problem commonly encountered in medical imaging is class label scarcity. Annotating medical images
extremely time consuming task that requires expert knowledge. However, obtaining domain labels is surprisingly cheap, since hospitals generally store information about the patient (e.g., age and sex) and the medical equipment (e.g., manufacturer and settings). Therefore, we are interested in extending the domain generalization framework to be able deal with additional unlabeled data, as we hypothesize that it can help to improve performance.
In this paper, we propose tackle domain generalization via a new deep generative model that we refer to as the Domain Invariant Variational Autoencoder (DIVA). We extend the variational autoencoder (VAE) framework (Kingma Welling. 2013 Rezende et al., 2014) by introducing independent latent representations for a domain label, a class label and any residual variations in the input x. Such partitioning of the latent space will encourage and guide the model to disentangle these sources of variation. Finally, by virtue of having generative model we can naturally handle the semi- supervieee scenario, similarly to Kingma et al. (2014). We evaluate our model on a version of the MNIST dataset where each domain corresponds to a specific rotation angle of the digits, as well as on a Malaria Cell Images dataset where each domain corresponds to a different patient. An implementation of DIVA be found under (URL. was removed to preserve anonymity).

---

Under review as a conference paper at ICLR 2020
dd
Zy
Z
Zyy
d
y
d
x
Figure 1: Left: Generative model. According to the graphical model we obtain p(d, x, y, Zd. Zz Zy) = po(x|za, Ze, Zy)pes (za dl)p a )pe, (zyly))(d)pp()) Right: Inference model. We propose to factorize the variational posterior as qoa (z.l qo. (Z, a x) x)qa, (zy xx). Dashed arrows represent the two auxiliary classifiers quy (dlzz) and qw, (y Zy))
2 TOWARDS DOMAIN GENERALIZATION WITH GENERATIVE MODELS
We follow the domain generalization definitions used in Muandet et al. (2013). domain is defined joint distribution p(x, y) on X' x V/ where X' denotes the input space and y denotes the output space. Let Fxxx y be the set of all domains. The training set consists samples S taken from N domains, = (S(d=)) Ji=1- Here, the ith domain píd=i)(x (x, y) represented by n, samples, S(d=i) {(xx (d=i) (d=i)) ) Each of the N distributions p(d=1) (x, y). ................................ p pla=1((x (x,y), y).... ..... p.....ppp.nn p(d=M) y) sampled from Fxxy- We further assume that (d (x, y) * #p pi (x, y). therefore, the samples in S are non-i.i.d. During test time unseen domain (dd==+1)( (x, y). We are interested in learning representations that generalize from pfd=1) (x, w)...... [idd=Ni(x,, (x, y) to this new domain. Training data are given as tuples (d, y) in the case of supervised data or as (d, x) in the case of unsupervised data.
I.
1/
presented with samples S(d=N+1) from a previously
2.1 DIVA: DOMAIN INVARIANT VAE
Assuming perfectly disentangled latent space (Higgins al. 2018), hypothesize that there exists a latent subspace that invariant to changes in d, i.e., it is domain invariant. We propose generative model with three independent sources of variation; 2a. which is domain specific, zy. which is class specific and finally Za. which captures any residual variations left in x. While Z keeps an independent Gaussian prior p(z.). ), Zd and by have conditional priors Pe, (Zald), pe, (2yly) with learnable parameters 0g, fy. This will encourage information about the domain d and label y to be encoded into zu and Zy. respectively. Furthermore. as Zd and Zy are marginally independent by construction, we argue that the model will learn representations Zy that are invariant with respect to the domain d. All three of these latent variables are then used by single decoder po po(x|z.a Zu, Zz Zy) for the reconstruction of x.
Since we are interested in using neural networks to parameterize po po(xzz,, Zz; 2ny, exact inference will be intractable. For this reason, we perform amortized variational inference with an inference network (Kingma Welling 2013 Rezende et al. 2014), i.e., we employ a VAE-type framework. We introduce three separate encoders 4o. [z,) x). Is [zg x)) and go,, (zy that serve variational posteriors over the latent variables. Notice that we do not share their parameters as we empirically found that sharing parameters leads decreased generalization performance. For the prior and variational posterior distributions over the latent variables Zg, Zd, Zy we assume fully factorized Gaussians with parameters given as a function of their input. We coin the term Domain Invariant VAE (DIVA) for our overall model, which is depicted in Figure I
Given specific dataset, all of the aforementioned parameters be optimized by maximizing the following variational lower bound per input x:
L, (d, x, y) B.g,, (x=1x)) (z, xx)4 (x))x) [log po (x Uz, zyy] 3KLL (qo2 (z. x) Ipe. (zzld)) - 3KL (go- (z (x)))(((((z))
3KT (go, (z,, [x)xx|[pp (z.,, (2yly)))
(1)
Notice that we have introduced a weigting term, 3. This is motivated by the 3- (Higgins et al.) 2017) and serves constraint that controls the capacity of the latent spaces of DIVA. Larger
2

---

Under review as a conference paper at ICLR 2020
values of B limit the capacity of each and in the ideal case each dimension of captures one of the conditionally independent factors in x.
To further encourage separation of za and Zy into domain and class specific information respectively, we add two auxiliary objectives. During training zu is used to predict the domain d and By is used to predict the class y for given input x:
FDIVA (d, x, y) = L. (d, x. y) + ( [log qwa (d|za)) + Oiy Basy (x/)x)) [log gw,, (2/2z,))
where Odd ay are weighting terms for each of these auxiliary objectives. Since our main goal is a domain invariant classifier, during inference only use the encoder ge, (2ylx) and the auxiliary classifier qw, (3/z,),. For the prediction of the class y for a new input ø we use the mean Zy. Consequently, we regard the the variational lower bound C, x, y) regularizer. Therefore, evaluating the marginal likelihood p(x) of DIVA is outside the scope of this paper.
Locatello et al. (2018) and Dai & Wipf 2019) claim that learning disentangled representation, 2o (zz), in an unsupervised fashion is impossible for arbitrary generative models. Inductive biases, e.g., some form of supervision constraints on the latent space, are necessary to find a specific set of solutions that matches the true generative model. Consequently, DIVA using domain labels d and class labels y in addition input data during training. Recent work by Khemakhem et al. 2019) shows that conditional priors, like po, (=a(d) and po,, (v) in DIVA, lead to identihabiiity guarantees in VAFs. Furthermore, we enforce the factorization of the marginal distribution of z in the following form: (z) = Toa (2a)4o. (z,) (zy): which prevents the impossibility described in Locatello et al. (2018). We argue that the strong inductive biases in DIVA make it possible to learn disentangled representations that match the ground truth factors of interest, namely, the domain factors Zd and class factors Zy. To highlight the importance of a partitioned latent space we compare DIVA to VAE with single latent space, the results of this comparison can be found in the Appendix.
i.e., 46 (z)
2.2 SEMI-SUPERVISED DIVA
In Kingma et al. 2014)) an extension to the VAE framework was introduced that allows to use labeled as well as un ladeeed data during training. While Kingma et al. (2014) introduced a two step procedure, Louizos al. (2015) presented a way of optimizing the decoder of the VAE and the auxiliary classifier jointly. We use the latter approach to learn from supervised data { Xn, Un)) as well as from unsupervised data ((dm, }. Analogically to Louizos et 2015). we use yw (2)zz,) to impute y:
Lu Lu x)
Ens (mal x)ao (221x))s, (m, (x) [lg Pol Po (zd, Zz; zy]]
BK L(gss (xllx)]loooo (Za (d)) - 3K BKL (Z. xx) ((([)((())))
B (zv x)) (olz, (lox (zuly) log qo, (zy)x))
+ Egg, (x,,(a,)x)) (ulz,) [lo p(y) log yw.,, (wlz,)).
where we use Monte Carlo sampling with the reparametrizaation trick (Kingma & Welling 2013) for the continuous latents Zd, Zu, Zy and explicitly marginalize
the discrete variable y. The final objective combines the supervised and unsupervised variational lower bound as well as the two auxiliary losses. Assuming N labeled and M unlabeled examples, obtain the following objective:
FSS-DIVA
FDIVA (Xn, Vn. dn)
Σ m==
(x(x + aegEys, (x.lxx..) (eulg, |log qwrs (dm [za)]- (4)
RELATED WORK
The majority of proposed deep learning methods for domain generalization fall into one of two
categories: 1) Learning single domain invariant representation, e.g., using ad adversarial methods (Carlucci et al.) 2018 Ghifary et al. 20155 Li et al. 2018 2017 Motiian et al. 2017 Shankar et al. 2018; Wang et al. 2019). While DIVA falls under this category there is key difference: we do not explicitly regularize z, using d. Instead we learn complementary representations Z,. Z, and z, utilizing generative architecture. 2) Ensembling models, each trained on an individual domain from
3

---

Under review as a conference paper at ICLR 2020
the training set (Ding & Fu 2018 Mancini al. 2018). The size of models in this category scales linearly with the amount of training domains. This leads to slow inference if the number of training domains is large. However, the size of DIVA is independent of the number of training domains. In addition, during inference time we only use the mean of the encoder 46 (zy x) and the auxiliary classifier tu, (ylzy)-
Concurrently to DIVA Cai et al. (2019) developed a framework to learn latent disentangled semantic representations (DSR) for domain adaptation. DSR assumes that the data generation process is exclusively controlled by the domain d and class As a result, DSR is lacking a third latent space 2z- We designed DIVA assuming that not all variations in z can be explained by the domain d and the class y. Therefore we introduce 2e in order to capture these residual variations. Furthermore, while DSR uses gradient reversal layers, we directly parameterize the ground truth generative model. As result, the priors in DIVA are conditional which is necessary for guaranteed disentanglement as recent research has shown (Khemakhem et al., 2019). More related work is published under the name of multiple source domain adaptation (Zhao et 2018).
An area that is closely related to domain generalization is that of the statistical parity in fairness. The goal of fair classification is to learn meaningful representation that at the same time cannot be used to associate a data sample to a certain group (Zemel et al., 2013). The major difference to domain generalization is the intention behind that goal, c.g., to protect groups of individuals robust to variations in the input. Consequently, DIVA is closely related to the fair VAE (Louizos) et al. 2015). In contrast the fair VAE, which is using hierarchical latent space, DIVA is using partitioned latent space. Moreover, the fair VAE requires the domain label during inference while DIVA alleviates this issue by learning the classifier without d. Similar DIVA, there is an increasing number of methods showing the benefits of using latent subspaces in generative models (Siddharth
being
al. 2017|
Klys al., 2018 Jacobsen et al., 2018: Bouchacourt et al.) 2018 Atanov al. 2019).
We derived DIVA by following the VAE framework (Kingma & Welling 2013), where the generative
process is the starting point. conditional version of the variational information bottleneck (CVIB) was proposed by Moyer et al. (2018) that likewise leads to objective consisting of a reconstruction loss. However, CVIB suffers from the same limitation as the fair VAE: that the domain must be known during inference, hence, we excluded it from our experiments.
4 EXPERIMENTS
We evaluate the performance of DIVA on two datasets: Rotated MNIST GGhiarr et al., 2015) and Malaria Cell Images (Rajaraman et al., 2018). In both cases we first investigate if DIVA able to successfully learn disentangled representations. Furthermore, we compare DIVA to other methods in a supervised and semi-supervised setting. While for the rotated MNIST dataset DIVA's graphical model is matching the ground truth generative model, the Malaria Cell Images dataset poses more challenging and realistic scenario, where the ground truth generative model is unknown.
4.1 ROTATED MNIST
The construction of the Rotated MNIST dataset follows (Ghifary et al. 2015). We sample 100 images from each of the 10 classes from the original MNIST training dataset. This set of images is denoted Mpe. To create five additional domains the images in Mas rotated by 15, 30, 45, 60 and 75 degrees. In order to evaluate their domain generalization abilities, models are trained on five domains and tested on the remaining 6th domain, e.g., train on Moe, M M13s, Maoe, M45- and Mane, test on M75 The evaluation metric is the classification accuracy on the test domain. All experiments are repeated 10 times. Detailed information about hyperparameters, architecture and training schedule can be found in the Appendix.
4.1.1 QUALITATIVE DISENTANGLEMENT
First of all, we visualize the three latent spaces Zd. Z, and Zy. to see DIVA is able to successfully disentangle them. In addition, we want verify whether DIVA utilizes z, in meaningful way, since it is not directly connected to any downstream task. For the following visualizations we restrict the size of each latent space ndz 2z and n, to 2 dimensions. Therefore, we can plot the latent subspaces
4

---

Under review as a conference paper at ICLR 2020
directly without applying dimensionality reduction, see Figure 2 images from five domains: Muo, M150, , Ma5 and Mao
Here, trained DIVA 5000
2D embeddings of all three latent subspaces. In the top row embeddings colored according to their domain, in the bottom row they are colored according to their class. First column: za encoded by Jo (z) The top plot shows five distinct clusters, where each cluster corresponds to a single domain. In the bottom plot no clustering visible. Second column: 2g encoded by Jo. (z., xx). We observe correlation between the rotation angle of each MNIST digit and zz[(0] in the top plot. Upon visual inspection of the original inputs x, we find a correlation between the line thickness digit and z [0] well correlation between the digit width and 2n[1] in the bottom plot. As result, we observe clustering of embeddings with class '1' at the lower left part of the plot. Third column: Zy encoded by go, (Zyl x)). In the top plot no clustering is visible. The bottom plot shows ten distinct clusters, where each cluster corresponds to class. A plot of the 2D embeddings for the test domain M750 can be found in the Appendix.
Figure 2:
From these initial qualitative results we conclude that DIVA is disentangling the information contained in intended, zu is only containing information about d and Zy only information about y. In the case of the Rotated MNIST dataset Z, captures any residual variation that is not explained by the domain d or the class y. In addition, are able generate conditional reconstructions as well as entirely new samples with DIVA. We provide these in the Appendix.
4.1.2 COMPARISON TO OTHER METHODS
We compare DIVA against the well known domain adversarial neural networks (DA) (Gainn et al. 2015) as well as three recently proposed methods: LG (Shankar et al. 2018), HEX (Wang et al. 2019) and ADV (Wang al., 2019). For the first half of Table (until the vertical line) we only use labeled data. The first column indicates the rotation angle of the test domain. We report test accuracy on for all methods. For DIVA we report the mean and standard error for 10 repetitions. DIVA achieves the highest accuracy across all test domains and the highest average test accuracy among all proposed methods.
The second half of Table showcases the ability of DIVA to unlabeled data. For this experiment we add: The same amount (+1) unlabeled data as well as three (+3), five (+5) and nine (49) times the amount of unlabeled data to training set. We first add the unlabeled data to Mge and create the data for the other domains as described in Section 4.1 In Table we can clearly see performance increase when unlabeled data added the training set. When the amount of unlabeled data is
5

---

Under review as a conference paper at ICLR 2020
Table Comparison with other state-of-the-ar domain generalization methods. Methods in the first half of the table (until the vertical line) use only labeled data. The second half of the table shows results of DIVA when trained semi-supervised (+ X times the amount of unlabeled data). We report the average and standard error of the classification accuracy.
Test
DA
LG
HEX
ADV
DIVA
DIVA(+1)
DIVA(+3)
DIVA(+5)
DIVA(+9)
Mue
86.7
89.7
90.1
89.9
93.5 + 0.3
93,8 + 0.4
93.9 0.5
93.2
93.0
M15
98.0
97.8
98.9
98.6
99.3 = 0.1
9.1
0.1
9.1
9.1
Mage
97.8
98.0
98.9
98.8
99.1 =
9.1
9..
9.1
9.1
M4s
97.4
97.1
98.8
98.7
99.2
9.2
991
0.1
9.1
Magn
96.9
96.6
98.3
98.6
99.3 + 0.1
0.1
9.14
9.14
99.2
M15
89.1
92.1
90.0
90.4
93.0 + 0.4
93.8 0.4
93.8
93.5
93.2+0.33
Avg
94.3
95.3
95.8
97.2
9.
9.
1.
much larger than the amount of labeled data the balancing of loss terms become increasingly more challenging which can lead to decling performance of DIVA, as seen in the last two columns of Table
4.1.3 ADDITIONAL UNLABELED DOMAINS
In Section 4.1.2 we show that the performance of DIVA increases when it presented with additional unlabeled data for each domain. As a result each training domain consists of labeled and unlabeled examples. In this section investigate more challenging scenario: We add an additional domain to our training set that consists of only unlabeled examples. Coming back to our introductory example of medical imaging, here would add unlabeled data from new patient or new hospital the training set. This is in contrast to the experiment in Section 4.1.2 where we would add unlabeled data from each known patient hospital to the training set.
In the following, we are looking at two different experimental setups, in both cases M753 is the test domain: For the first experiment we choose the domains Mar, MM15, M4r and Mag to be part of the labeled training set. In addition, unlabeled data from Mar is used. In Table we can see that even in the case where the additional domain dissimilar to the test domain DIVA able to slightly improve. For the second experiment we choose the domains Mo M150, Mage and M45 to be part of the labeled training set. addition, unlabeled data from MMo used. When comparing the results in Table 2 to the results in Table T we notice drop in accuracy of about 20% for DIVA trained with only labeled data. However, when trained with unlabeled data from MMg we see an improvement of about 7%. The comparison shows that DIVA can successfully learn from samples of a domain without any labels.
Table 2: Comparison of DIVA trained supervised to DIVA trained semi-supervised with additional unlabeled data from Mag and Mogo. We report the average and standard error of the classification accuracy on M75..
Unsupervised domain
DIVA supervised
DIVA semi-supervised
Mag
93.1 0.5
93.3 + 0.4
Mag
73.8 0.8
80.6 + 1.1
4.2 MALARIA CELL. IMAGES
The majority of medical imaging datasets consist of images from a multitude of patients. In domain generalization setting each patient is viewed as an individual domain. While we focus on patients as domains in this paper, this type of reasoning can be extended to, e.g., hospitals domains. We, among others (Rajaraman et al. 2018 Lafarge et al. 2017). argue that machine learning algorithms trained with medical imaging datasets should be evaluated on subset of hold-out patients. This presents a more realistic scenario since the algorithm is tested on images from a previously unseen domain. In the following, example of dataset consisting of samples from multiple patients. The images in this dataset were collected and photographed Chittagong Medical College Hospital, Bangladesh. It consists of
Malaria Cell Images dataset (Rajaraman et al. 2018)

---

Under review as a conference paper at ICLR 2020
27558 single red blood cell images taken from 150 infected and 50 healthy patients. The images were manually annotated by a human expert. A cell has the label y = 1 if shows the parasite and if not. To facilitate the counting of parasitized and uninfected cells, the cells were
the label stained using Giemsa stain which turns the parasites inside the cell pink. In addition, the staining process leads variety of colors of the cell itself. While the color of the cell is relatively constant for single patient, it can vary greatly between patients, see the first row in Figure 3 This variability in appearance of the cells can be easily ignored by human observer, however, machine learning models can fail to generalize across patients. In our experiments we will use the patient ID as the domain label d. We argue that for this specific dataset the patient ID a good proxy of appearance variability. In addition, there is no extra cost for obtaining the patient ID for each cell.
Subsequently, we use a subset of the Malaria Cell Images dataset that consists of the 10 patients with the highest amount of cells. The amount of cells per patient varies between 400 and 700 and there are 5922 cell images in total. The choice of this subset is motivated by the similiar amount of cells as well as the similar marginal label distributions per patient, the latter being necessary condition for successful domain generalization (Zhao et al., 2019p. Furthermore we rescale all images to 64 x 64 pixels. To artificially expand the size of the training dataset data augmentation in the form of vertical flips, horizontal flips and random rotations.
4.2.1 QUALITATIVE DISENTANGLEMENT
We investigate the three latent subspaces Z,, Zμ and Zy to see if DIVA is able to successfully disentangle them. In addition, we want to see if DIVA utilizes zg in a meaningful way, since it is not directly connected any downstream task. Figure shows the reconstructions of x using all three latent subspaces as well as reconstructions of x using only a single latent subspace at a time. First, find that DIVA is able to reconstruct the original cell images using all three subspaces (Figure 3 second row). Second, we find that the three latent subspaces are indeed disentangled: za is containing the color of the cell (Figure 3 third row), 2₂ the shape of the cell (Figure 3 fourth row) and Zy the location of the parasite (Figure 3 fifth row). The holes in the reconstructions using only z, indicate that there no probability mass Zd and Zy at O, similar Figure 2 From the reconstructions in Figure 3 we conclude that DIVA is able to learn disentangled representations that match the ground truth factors of interest, here, the appearance of the cell and the presence of the parasite. In addition to these qualitative results, we show that classifier for y trained on Za Za performs worse than classifier that would always predict the majority class, the results can be found the Appendix.
x ~ po(x| Zzz Zay)
x ~ >>poxxl 0,
x x~ <>po(>(10, Zz.
x ~ xx000
0
Figure 3: Reconstructions of x using all three latent subspaces as well reconstructions of using only a single latent subspace at a time.
4.2.2 SUPERVISED CASE
To further evaluate domain generalization abilities, models are trained on nine domains (patient IDs) and tested on the remaining 10th domain. We choose ROC AUC the hold out test domain as the evaluation metric, since the two classes are highly imbalanced. All experiments are repeated five times.
7

---

Under review as a conference paper at ICLR 2020
Table 3: Results of the supervised experiments for the first part of domains. We report the average and standard error of ROC AUC.
Model
C116P77
C132P93
C137P98
C180P141
C182P143
C184P145
Baseline
90.6 + 0.7
97.8 0.5
98.9 + 0.2
98.5 0.2
96.7 3.770..4 0.4
98.1 0.2
DA
90.6 = 1.7
98.3 0.4
99.0
98.8
97.10088 9.4
DIVA
93.3 = 0.4
98.4 0.3
99.0 0.1
99.0
0.3
9
Table 4: Results of the supervised experiments for the second part of domains. As well as the average across all domains. We report the average and standard error of ROC AUC.
Model
C39P4
C59P20
C68P29
C99P60
Average
Baseline
97.1 + 0.4
82.8 + 2.8
95.3 + 0.6
96.2 + 0.1
95.2 + 1.6
DA
97.4 + 0.3
83.2 + 3.3
96.3 + 0.1
96.1 + 0.3
95.4 + 1.6
DIVA
97.8 + 0.2
82.1 3.0
96.3 + 0.2
96.6 + 0.3
95.8 + 1.6
We compare DIVA with a ResNet-like (He et al. 2013 baseline and DA. During inference all three models have the same architecture, seven ResNet blocks followed by two linear layers. Detailed information about hyperparameters, architecture and training schedule can be found in the Appendix. In Table 3 and 4 we find that the results are not equally distributed across all test domains. In five cases DIVA is able to significantly improve upon the baseline model and DA. Upon visual inspection we find that cells from domain C1 1677 and domain C59P20 stained pink, similar to the stain of the parasite, see Appendix. In case of C116777 DIVA achieves the highest ROC AUC of all three models. In case of domain C59P20, all three methods have difficulties to detect the parasite which leads to the lowest ROC AUC among all domains. We believe that the difficulties arise the poor contrast between cell and parasite. Last, DIVA is able to improve average when compared to the baseline model and DA, although the improvements are within the standard error.
4.2.3 SEMI-SUPERVISED CASE
As described in Section 4.1.3 we interested in learning from domains with no class labels, since such an approach can drastically lower the amount of labeled data needed to learn a domain invariant representation, i.e., a model that generalizes well across patients. For the semi-supervised experiments we choose domain C176777 to be the test domain since its cells show a unique dark pink stain. Furthermore, unlabeled data from domain C59P20 is used since it is visually the closest domain C116P77, see Appendix. The evaluation metric on the hold out test domain is ROC AUC again. In Table S we compare the baseline model, DA and DIVA trained with labeled data from domain C59P20, unlabeled data from domain C59P20 and no data from domain C59P20.
We argue that the improvement of DIVA over DA arises from the way the additional unlabeled data is utilized. In case of DA the unlabeled data (d, x) is only used to train the domain classifier and the feature extractor in an adversarial manner. In Section 2.2 we show that due to DIVA's generative nature 4o, (Z.y) xx), pa, (v,,l)) can be updated using qw, (,y) to marginalize over y for an unlabeled sample x. In addition, the unlabeled data (d, x) is used to update qo (za(x), Pea (zald (d). qua (dlza).
qo. (z. (x) and po (x zz Zz, Zn) in the same way in the supervised case.
Table 5: Results of the semi-supervia experiments for domain C1 Comparison of baseline method, DA and DIVA trained with labeled data from domain C59P20, unlabeled data from domain C59P20 and no data from domain C59P20. We report the average and standard error of ROC AUC.
Training data
Baseline
DA
DIVA
Labeled data from C59P20
90.6 0.7
90.6
1.7
93.3 + 0.4
Unlabeled data from C59P20
72.05
- 2.2
79.4
No data from C59P20
70.0 + 2.6
69.2 1.9
71.9 2.7
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
We have proposed DIVA as generative model with three latent subspaces. We evaluated DIVA on Rotated MNIST and a Malaria Cell Images dataset. In both cases DIVA is able to learn disentangled representations that match the ground truth factors of interest, represented by the class y and the domain d. By learning representations 2, that are invariant with respect to the domain d DIVA is able to improve upon other methods on both datasets. Furthermore, we show that we can boost DIVA's performance by incorporating unlabeled samples, even from entirely new domains for which no labeled examples are available. This property is highly desirable in fields like medical imaging where the labeling process is very time consuming and costly.
In all of our experiments it appears that there is a key difference between interpolation and extrapola- tion, distinction currently not made by the domain generalization community: If we assume that the domains lie in intervals like 105, 30"] or ['red', 'orange', 'yellow'] then the performance for the domains in the center of the interval, e.g., 15e and 'orange', seems to better than for the domains on the ends of the interval. We argue that DIVA can make use of unlabeled data from a domain that is close to the test domain to improve its extrapolation performance, as seen Section 4.1.3 and 4.2.3
REFERENCES
Andrei Atanov, Alexandra Volokhova, Arsenii Ashukha, Ivan Sosnovik, and Dmitry Vetrov. Semi- Conditional Normalizing Flows for Semi-Supervised Learning. arXiv:1905.00505 [cs, stat], May 2019. arXiv: 1905.00505
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly small image transformations? arXiv: 12177 [cs]. May 2018. arXiv: 1805.12177.
Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from Several Related Classification Tasks to New Unlabeled Sample. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp. 2178-2186. Curran Associates, Inc., 2011.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations. pp. 8, 2018.
Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in beta-VAE. arXiv:1804.0359) [cs, stat], April 2018. Xiviv: 1804.03599.
Ruichu Cai, Zijian Li, Pengfei Wei, Jie Qiao, Kun Zhang, and Zhifeng Hao. Learning Disentangled Semantic Representation for Domain Adaptation. In Proceedings of the Twenty-Eighth Interna- tional Joint Conference on Artificial Intelligence, pp. 2060-2066. Macao, China, August 2019, International Joint Conferences on Artificial Intelligence Organization. ISBN 978-0-9992411 1-4-11 doi: 10.24963/jjcai.200 19/255.
Fabio M. Carlucci, Paolo Russo, Tatiana Tommasi, and Barbara Caputo. Agnostic Domain General- ization. / /02 [cs), August 2018. arXiv: 1808.01 102.
Francesco Ciompi, Oscar Geessink, Babak Ehteshami Bejnordi, Gabriel Silva de Souza, Alexi Baidoshvili, Geert Litjens, Bram van Ginneken, Iris Nagtegaal, and Jeroen van der Laak. The importance of stain normalization in colorectal tissue classification with convolutional networks. 1rXiv:1702.05931 [cs], February 2017. arXiv: 1702.05931.
Bin Dai and David Wipf. Diagnosing and Enhancing VAE Models. arXiv: [cs, stat), March 2019. arXiv: 1903.05789.
Zhengming Ding and Yun Fu. Deep Domain Generalization With Structured Low-Rank Constraint. IEEE Transactions on Image Processing, 27(1)304-313, January 2018. ISSN 1057-7149, 1941- 0042. doi: 10. 10.1109/1122011227551181
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain--Aversaan Training of Neural Networks. arXiv: /00:://55008888888 (cs, stat), May 2015. arXiv: 1505.07818.

---

Under review as a conference paper at ICLR 2020
Muhammad Ghifary, W. Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain Generalization for Object Recognition with Multi-task Autoencoders. In 2015 IEEE International Conference Computer Vision (ICCV), pp. 2551-2559, Santiago, Chile, December 2015. IEEE. ISBN 978-1-4673-8391-22 doi: 10. 10.1109/CCV.22015.222
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. urXiv:15/2.03335 [cs], December 2015. arXiv: 1512.03385.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta-VAFE VAE: Learning Basic Visual Concepts with constrained variational framework. pp. 22, 2017.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards a Definition of Disentangled Representations. arXiv: 1812.02230
[cs, stat], December 2018. arXiv: 1812.02230.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv:1502.03167 [cs], February 2015. arXiv: 1502.03167.
Jörn-Heriik Jacobsen, Jens Behrmann, Richard Zemel, and Matthias Bethge. Excessive Invari- ance Causes Adversarial Vulnerability. arXiv:1811.00401 [cs, staf], November 2018. arXiv: 1811.00401.
llyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. Variational Autoencoders and Nonlinear ICA: A Unifying Framework. arXiv:1907.000 190,:000070809 [cs, stat], October 2019. arXiv: 1907.04809.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs], December 2014. arXiv: 1412.6980.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. 132.v11312.6114 [cs, stat], December 2013. arXiv: 1312.6114.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-Supervised Learning with Deep Generative Models, arXiv: 5298 (cs, stat], June 2014. arXiv: 1406.5298.
Jack Klys, Jake Snell, and Richard Zemel. Learning Latent Subspaces in Variational Autoencoders. arXiv: 10li:0/11066999 [cs, stat], December 2018. arXiv: 1812.06190.
Maxime W. Lafarge, Josien
Pluim, Koen J. Eppenhof, Pim Moeskops, and Mitko Veta. Domain-adversarial neural networks to address the appearance variability of histopathology images. arXiv: 1707.06183
1rXiv:1707.06183 [cs], 10553, 2017. doi: 10.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to Generalize: Meta- Learning for Domain Generalization. arXiv: 171.v.1710.03466 [cs], October 2017. arXiv: 1710.03463.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain Generalization via Conditional Invariant Representation. 1r7ivv1807.0844 [cs, stat], July 2018. arXiv: 1807.08479.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disen- tangled Representations. arXiv: 1811.1111225 [cs, stat], November 2018. arXiv: 1835.1
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The Variational Fair 1,00030 [cs, stat), November 2015. arXiv: 1511.00830.
Autoencoder.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling. arXiv:1902.022 102 fcs, stat], February 2019. arXiv: 1902.02102.
Massimiliano Mancini, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci. Best sources forward: domain generalization through source-specific nets. 1rXiv:1806.05810 [cs, stat], June 2018. arXiv: 1806.05810.
10

---

Under review as a conference paper at ICLR 2020
Saeid Motian, Marco Piccirilli,, Donald A. Adjeroh, and Gianfranco Doretto. Unified Deep Super- vised Domain Adaptation and Generalization. arXiv:1709.10190 [cs], September 2017. arXiv: 1709.10190.
Daniel Moyer, Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Invariant Representations without Adversarial Training. 1805 [cs, stat], May 2018. arXiv: 1805.09458.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain Generalization via Invariant Feature Representation. arXiv:1301.2115 [cs, stat], January 2013, arXiv: 1301.2115.
Sivaramakrishnan Rajaraman, Sameer K. Antani, Mahdieh Poostchi, Kamolrat Silamut, Md. A. Hossain, Richard J. Maude, Stefan Jaeger, and George R. Thoma. Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images. PeerJ, 6:e4568, April 2018. ISSN 2167-8359, doi: 10.7717/peerj.4566.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. arXiv preprint 401 2014.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik Kingma. PixeICNN++: Improving the Pix-
cICNN with Discretized Logistic Mixture Likelihood and Other Modifications. arXiv:1701.05517 [cs, stat], January 2017. arXiv: 1701.05517.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing Across Domains via Cross-Gradient Training. arXiv: arXiv:/804. 10745 [cs, staf],
April 2018. arXiv: 1804.10745.
N. Siddharth, Brooks Paige, Jan- Willem van de Meent, Alban Desmaison, Noah D. Goodman, Pushmeet Kohli, Frank Wood, Philip Torr. Learning Disentangled Representations with SemiiSupervised Deep Generative Models. 170iv:/706.00444 [cs, stat], June 2017. arXiv: 1706.00400.
Haohan Wang, Zexue He, Zachary C. Lipton, and Eric P. Xing. Learning Robust Representations by Projecting Superficial Statistics Out, arXiv: arXiv: 990300666666 [cs], March 2019. arXiv: 1903.06256.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 325-333, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Han Zhao, Shanghang Zhang, Guanhang Wu, José M. F. Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial Multiple Source Domain Adaptation. In Bengio, H. Wallach, H. Larochelle, and R. Garnett (eds.), Advances in Neural Information Processing
K. Grauman, N. Systems 3/, pp. 8559-8570. Curran Associates, Inc., 2018
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On Learning Invariant Representation for Domain Adaptation. arX1v:1901.09453 [cs, stat], January 2019. arXiv: 1901.09453.

---

Under review as a conference paper at ICLR 2020
APPENDIX
5.1 ROTATED MNIST
5.1.1 TRAINING PROCEDURE
All DIVA models are trained for 500 epochs. The training terminated if the training loss for has not improved for 100 epochs. As proposed in Burgess et al. (2018), we linearly increase from 0.0 1.0 during the first 100 epochs of training. We set ot = 2000. seen (Maaløe et al. 2019), we adjust og according to the ratio of labeled (N) and unlabeled data (M),
M+N 4y N
(5)
where we set 7 3500. Last, Zdz Zn and 2g each have 64 latent dimensions. All hyperparameters were determined by training DIVA on MMg. MM1.., M300, M43 and testing on Meg- We searched over the following parameters: ad, ad E {1500, 2000, 2500, 3500, 4000); dim = dim (z.) dim(z,,) (zy,) and dim,z.,) (16, 32, 64]; Pmar E {1, 5, 103.
All models were trained using ADAM (Kingma & Ba 2014) (with default settings), a pixel-wise cross entropy loss and batch size of 100.
5.1.2 ARCHITECTURES
To enable fair experiment, the encoder do, (z,|x) and auxiliary classifier qa, (vlzy) form DNN with the same number of layers and weights as described in Wang et al. (2019).
Table 6: Architecture for po xx [d, 2n 2y)) The parameter for Linear is output features. The parame- ters for Conv'raasposee2 are output channels and kernel size. The parameter for Upsample the upsampling factor. The parameters for Conv2d output channels and kernel size.
block
details
Linear( Linear(1024), BatchNorm ReLU
2
Upsample(2) Con/Transposedd (28, 5), BatchNorm2d, ReLU
3
Upsample(2)
5
Con/Trannposse2((566, 5). BatchNorm22, ReLU Conv2d(256, 1)
6
Table 7: Architecture for Po, Poz (z and pe, (Zylv). Each network has two heads one for the mean and one for the scale. The parameter for Linear is output features.
block
details
Linear(64), BatchNorm Id, ReLU
2.1
Linear(64)
2.2
Linear(64), Softplus
12

---

Under review as a conference paper at ICLR 2020
Table Architecture for gou (za(x), 9d= (Zu)xx x) and qd, (zy (2) Each network has two heads one for the mean one and for the scale. The parameters for Conv2d are output channels and kernel size. The parameters for MaxPool2d are kernel size and stride. The parameter for Linear is output features.
block
details
Conv2d(32 5), BatchNorm2d, ReLU
MaxPool2d(2, 2)
Conv2d(64, 5), Batch ReLU
MaxPool2d(2., 2) Lincar(64) Linear(64), Softplus
5.1
5.2
Table 9: Architecture for qun qua (d(zza) and qu,, (ylzy). The parameter for Linear is output features.
block
details
ReLU, Linear(5 for qws (dzza/IO for qu, (3/ Z,)), Softmax
5.1.3 SAMPLES
We present samples from DIVA by sampling 2a, 2, and Zy from their priors and then decoding them. Generated examples on the Rotated MÄIST data are given in the Figure 4 DIVA allows generate
images that are almost indistinguishable from real datapoints.
3
3
Figure 4: Samples from DIVA trained on Rotated MNIST
5.1.4 CONDITIONAL GENERATION
Yet another way to gain insight into the disentanglement abilities of DIVA is conditional generation. We first train DIVA with 3 10 using MMr, M15s, Mare, Mas and MMg- as training domains. After training we perform two experiments. In the first one we are fixing the class and varying the domain. In second experiment we are fixing the domain and varying the class.
Change of class The first row of Figure 5 (left) shows the input images for DIVA. First, generate embeddings Zd, Zn and z for each x using Jo. x), 9o. (z. xx) and Jo, (zylx). Second, we replace 2, with a sample zi, from the conditional prior pa, (z,, 1y). Last, we generate new images from 2d, 2, and n,, using the trained encoder po (x z. 2n, 2ny In Figure (left) rows 2 to 11
13

---

Under review as a conference paper at ICLR 2020
correspond to the classes '0" to '9", We observe that the rotation angle (encoded in Za) and the line thickness (encoded in 2y) are well preserved, while the class the image is changing as intended.
9
3
9
0
0
1
2 7
6
2 9
3
9
2
7
9
3
9 9
2
6
227
4
3
8
a a 9
Figure 5: Reconstructions. Left: First row is input, row 2 to 11 correspond to labels '0" to '9". Right:
First row is input, row correspond to domains 0, 15, 30, 45, 60.
Change of domain We repeat the experiment from above but this time we keep and By fixed while changing the domain. After generating embeddings zd, Z₂ and zy for each x the first row of Figure 5 (right), we replace 2 with a sample z from the conditional prior Pe, (z.ld). Finally, we generate new images from z/ Ze and zy using the trained encoder ps (x Zu. Zz, 2y). In Figure 5 (right) rows to 6 correspond to the domains Mie to MMge. Again, DIVA shows the desired behaviour: While the rotation angle is changing the class and style of the original image is maintained.
5.1.5 QUALITATIVE DISENTANGLEMENTT. TEST DOMAIN
In this section, visualize the zμ and zy for data points x from the test domain M750 for the model trained in Section 4.1.1 Figure shows 1000 embeddings z, encoded by qa, (z,,lx). xx). Figure shows 1000 embeddings 2n encoded by go (z.(x).
14

---

Under review as a conference paper at ICLR 2020
15
30
45
60
75
Figure 6: 1000 two-dimensional embeddings z, encoded by qoa (zalx) for x from the test domain
M75 The color of each point indicates the associated domain.
2
3
4
5
6
7
8
9
Figure 7:
1000 two-dimensional embeddings z, encoded by 4s, (z,/xx) for x from the test domain M750. The color of each point indicates the associated class.
5.1.6 ABLATION STUDY: PARTITIONED LATENT SPACE
We compare DIVA to VAE with single latent space, standard Gaussian prior and two auxillary tasks. The resulting graphical model shown in Figure g The results in Table clearly show the benefits of having partitioned latent space
15

---

Under review as a conference paper at ICLR 2020
x
d
X
y
Figure 8: Left: Generative model. According to the graphical model we obtain p(x, z) = (x))))((z) Right: Inference model. We propose 46 (z(z))) the variational posterior. Dashed arrows represent the two auxiliary classifiers qua (d/z) and quy (y|z).
The objective is given by,
[d, x, == (x(()) [oog)((((2) 3KT. (4o (2) (x)))(((((
- ag (xix) [log Awa (d|z)] + go [log qwi, (u|z)]
(6)
Table 10: Comparison of DIVA with VAE with a single latent space, a standard Gaussian prior and two auxillary tasks on Rotated MNIST. We report the average and standard error of the classification accuracy.
Test
VAE
DIVA
Mg
88.4 0.5
93.5 + 0.3
M15
98.3 1.3300.11
99.3 + 0.1
Ma0a
97.4
99.1 + 0.1
Ma5
97.4
99.2 + 0.1
Moo
97.9
99.3 + 0.1
M750
84.0
93.0 + 0.4
Avg
0.1
97.2 = 1.3
5.1.7 ABLATION STUDY: DIVA WITHOUT 2d AND 2n
We compare DIVA as proposed in Section 2.1 to two ablated versions of DIVA:
1. DIVA without zạ: The domain label d used during training. Therefore, there exist no latent space 2g, no encoder qea (zalz), no prior Pa, (zald) and no classifier que (d]za). The decoder becomes po (ar) Exy Fy)-
2. DIVA without 2g: There exist no latent space 2n, encoder 4o (2x(2r) and prior p(2z). The decoder becomes pe t0|2d, 2yy)
In Table []] compare DIVA and the ablated versions the Rotated MNIST dataset. Surpris- ingly, for Rotated MNIST we could not find a significant difference in performance between DIVA and DIVA without 24, as seen the third column. However, not having zd drastically reduces the interpretability of our model, since without žd we cannot find the variations in that are explained by the domain F.g. in Appendix 5.1.4. we show that we can generate samples conditioned on the domain label that give us clear idea of the meaning of d. Furthermore, as seen in Figure 3 is highly correlated with the color the stain. While the cell shape in
we see that the patient not correlated with d y and therefore is captured by Zn. Without =œ we are unable to gain such (especially from a medical perspective) important insights. the fourth column, we see that for Mye and M75 DIVA with žµ performs significantly better than without. We argue that if 2g does not exist, žn and by will capture the residual variations that are not explained by or y. We believe this makes it harder to predict using my and d using zd-
16

---

Under review as a conference paper at ICLR 2020
Table 11: Results of ablation study.
Test
DIVA
DIVA without Ad
DIVA without Zx
Mge
93.5 + 0.3
93.4 + 0.5
92.7 + 0.5
M 13-
99.3 + 0.1
99.3 0.1
99.4
Ma0
99.1 = 0.1
9.1
99.2
M45
99.2 0.1
9.11
99.1 0.1
Mao
99.3 + 0.1
99.1
9.14
M75
93.0 + 0.4
92.8 0.4
+.
Avg
97.2 1.3
97.1 1.3
97.1
5.2 MALARIA CELL IMAGES
5.2.1
EXAMPLE CELLS FROM EACH DOMAIN
(a) C116P77
(b) C132P93
(c) C137P98
(d) C180P141
(c) C182P143
(f) C184P145
(g) C39P4
(h) C59P20
C68P29
(i) C99P60
5.2.2 TRAINING PROCEDURE: DIVA
All DIVA models trained for 500 epochs. The training terminated if the validation accuracy for y has not improved for 100 epochs. As proposed in Burgess et al. (2018). we linearly increase from 0.0 1.0 during the first 100 epochs of training. We ag 100000 and og 75000.
Last, Zd. 2₂ and Zy each have 64 latent dimensions. We searched over the following parameters:
ad. ad E {25000, 50000, 75000, 100000): dim(za) dim (z.) dim(zy). dim(z.) E {32, 64); 8mar E {1,5, 5, 10). All hyperparameters were determined using validation set that consists of 20 % the training set. All models were trained using ADAM (Kingma & Bal 2014) (with default settings), a mixture of discretized logistics (Salimans et al.
2017) loss and batch size of 100. In
case of the semi-supervised experiment in Section 4.2.3 we adapt or and a, according to Equation
5.2.3 TRAINING PROCEDURE: BASELINE AND DA
In case of the supervised experiments in Section 5.2.5 all models are trained for 500 epochs. The training is terminated if the validation accuracy for has not improved for 100 epochs. In case of the semi-supervised experiments in Section 2.2 the amount of epochs is adjusted to match the number parameter updates of DIVA. For DA we follow the same training procedure as described in Ganin et al. (0155). In the supervised case, first, a labeled batch randomly sampled from the training distributions is used to update the class classifier, domain classifier and the feature extractor in adversarial fashion. Second, second batch randomly sampled from the training distributions used to update only the domain classifier and the feature extractor in an adversarial fashion. In the semi-supervised case samples from the unsupervised domains form the second batch together
17

---

Under review as a conference paper at ICLR 2020
with samples from the supervised domains. We the same domain adaptation parameter schedule as described in Ganin et al. (2015). Determined by hyperparameter search we find that DA performs better when A is used. Here, 0.001. We searched over the following values of E (0.1. 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001). In case of the semi-supervised experiment in Section 4.2.3 0.01 was determined by hyperparameter search.
5.2.4 ARCHITECTURE
In the following we will describe the architecture of DIVA in detail. Note that the architecture for the baseline model the same (o, (2,, xx) (we only use the mean of x.y) followed by Yw, (y) (yyyy) where Zy has 1024 dimensions. DA is using qo, (zylx) without the linear layer as a feature extractor. The class classifier and the domain classifier consist of two linear layers. The feature extractor for all models consist of seven ResNet blocks (He et al. 2015). During training batch norm loffe & Szegedy (2015) is used for all layers.
Table 12:
Architecture for (x(x/za, Zz, Zy). The parameter for Linear is output features. The parameters ResidualCon/Transpoee]) are output channels and kernel size. The parameters for Conv2d are output channels and kernel size.
block
details
Lincar(10244 BatchNormid, Id, LeakyReLU
ResidualConvTranse Trasspss2sss... 3), LeakyReLU
3
3), LeakyReLU
TaniduaaCConnTaaasssseeeelll 3), LeakyReLU
5
Residual CesiidulCCon//aauspsseed(... 3), LeakyReLU
3), LeakyReLU
SonnddaaiCoonnTaasspsseeeeee 3), LeakyReLU
8
ResidualCon 33d2 3), LeakyReLU
Resiidalloo//TTuupose22222 (33 3), LeakyReLU
10
Conv2d 100, 3)
Conv2d( 100,
Table 13: Architecture for (e. (zald) and pe,, (z,lv). Each network has two heads one for the mean and one for the scale. The parameter for Linear output features.
block
details
Linear(64), BatchNormidd d,, LeakyReL.U Linear(64) Linear(64), Softplus
2.1
2.2
18

---

Under review as a conference paper at ICLR 2020
Table 14: Architecture for qdu (Za(x), go. (Ze xx) and qd, (zy] Each network has two heads one for the mean one and for the scale. The parameters for Conv2d are output channels and kernel size. The parameters for ResidualConv2d are output channels and kernel size. The parameter for Linear output features.
block
details
Conv2d(32, 3), BatchNorm22,, LeakhNNo ResidualConn2((33),, LeakyReL.U
ResidualConv2d(32), LeakyReLU
ResidualCoovvdd644.3 3), LeakyReLU
ResidualCConvdd664, 3), LeakyReLU
ResidualConv2d((4, 3), LeakyReLU
ResidualConv2d(64,, 3), LeakyReLU
Residual Cartnaloo222ffff 3), LeakyReLU
9.1
L6near(64)
9.2
Linear(64), Softplus
Table 15: Architecture for Ywa (d/za) and Own (ulzy). The parameter for Linear is output features.
block
details
LeakyReLU, Linear(9 for qw [2lza/2 for qw,, (2) Zyy), Softmax
5.2.5 PREDICTING y USING EITHER Zd, Z5 Zy
We test how predictive Zd, Z, and Zy are for the class y on the Malaria Cell Images dataset. First, we use the trained DIVA models from to create embeddings domain and hold out test domain. Second, we train 2-layer MI.P on the embeddings 2d, 2₂ and Zy from the training domains. We train the MLP for 100 epochs using ADAM (Kingma & Ba 2014 After training we test the MLP embeddings Zd. Zy and Zy from the test domain. In Table 16 we clearly that Zy captures all relevant information in order predict y, while the MLPs traindd using zd and z perform worse than a classifier that would always pick the majority class.
2, and By for every x in the training
Table 16: Prediction of y using 2 layer MIP trained using 2a and Zyy. We report the mean and standard error the classification accuracy on the hold out test domain.
test domain
Z
2
Zy
majority class
0
84.6 + 1.0
85.0
+ 0.2
87.9 = 0.9
0.86
I
89.5 0.4
88.2 + 0.5
96.8 1 0.1
0.9
2
68.2
80.0
96.9 0.5
0.81
3
87.0
2.9
95.5 0.2
0.88
4
82.7 0.3
92.5 = 0.4
0.90
5
88.3 0.2
87.7
90.6 + 0.5
0.88
6
8.7
56.3 5.1
91.1 + 0.1
0.90
7
8.1
8.1
90.8 - 0.8
0.88
8
8.7 0.3
93.5 0.4
0.90
9
99.1002
1.5
94.0 - 0.3
0.89
19


Reviews:
[{"review_id": "SygGzfHpFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose a domain invariant variational autoencoder for domain generalization problem. Specifically, the data is assumed to be constructed from three independent variables, one for the domain, one for the class and one for the residual variations. The method can be used for both unsupervised and semi-supervised cases. Experimental studies on rotated MNIST dataset and a malaria cell images dataset verify the effectiveness of the proposed method.\nThe paper is well-written and easy to follow. The proposed generative model is simple and technically sound. However, I have the following concerns.\n(1)\tIt is not clear how the problem setting, i.e., domain generalization, matters in DIVA. In another words, the proposed DIVA is not specific to domain generalization problem, but can be used for domain adaptation, multiple source transfer learning etc. Actually, I find that the authors compare with DA, which is a conventional domain adaptation method, in the experiment. Moreover, the experimental setups in section 4.1.3 is a multi-source transfer setting, and DIVA can be well be applied. In this sense, I am not very convinced on the claim of the contribution that DIVA is proposed for domain generalization. For me, DIVA is a more general method.\n(2)\tWith point 1, more related works on VAE for domain adaptation need to be discussed.\n(3)\tThe idea of constructing data from disentangled latent variables is not new, see a latest work [ref1]. The main difference of DIVA from [ref1] is the residual variation variable. Two baselines are necessary for comparisons: (1) [ref1], and (2) DIVA without the residual variation variable. Actually, the semantic meaning of z_x is not well discussed. Even in the experimental studies figures 2 and 3, it is hard to tell what Z_x actually represents. \n(4)\tRegarding the 4.1.2, why DA is selected as a baseline? How does DA deal with the multiple domains? Can any other domain adaptation methods, e.g., [ref2], or multiple source transfer methods, e.g. [ref3], be compared?\n(5)\tIn the right side of Table 1, the improvements seem to be very marginal considering the variance. This makes the ability of DIVA to use unlabelled data less convincing. \n(6)\tRegarding 4.1.3, it seems that the domain similarity plays an important role in the performance, comparing the results of M_{30} with M_{60}. Without the labelled M_{60}, which is very similar to the target M_{75}, the performance degenerates dramatically. The current DIVA treats all the domains equally, is it possible to have a weighted form of DIVA that distinguishes the contributions of different domains? \n(7)\tWhat is the task of the malaria cell images experiments? Is it to classify the parasitized and uninfected cells? For a given patient, it makes more sense that all the cells belongs to one category, either infected or healthy. How is the class distribution for a person (in this case a domain)? Is it very unbalanced? \n(8)\tFor figure 3, It is hard to judge the cell images parasitized or uninfected without domain knowledge, can you give the label for each image? Again, the semantic meaning of Z-x is hard to tell. I am not convinced by the shape of the cell for Z-x.\n(9)\tFor 4.2.2, why and how DA is compared? As far as I know, it is for unsupervised domain adaptation. Moreover, the improvements are quite marginal. \nSome minor comments:\n(1)\tPage1, first para, 3rd line, “present” -> “presented”.\n(2)\tPage2, first para, 2nd line, “Y” - > “Y denotes”\n(3)\tSome references lack of page information\nThe paper should be self-contained. I would suggest the authors move some paragraphs in appendix to the paper, for instance, 5.1.1, 5.2.2, and 5.2.3.\nOverall, the paper is presented with extensive empirical evaluations, but less theoretical justification. The significance of the paper is moderate as the key idea of learning disentangled latent variables has been studied, and the paper lacks of evidence to show the pure benefits of introducing Z_x as well as the comparison with the related work [ref1]. \n[ref1] Learning Disentangled Semantic Representation for Domain Adaptation\n[ref2] Conditional Adversarial Domain Adaptation\n[ref3] Multiple Source Domain Adaptation with Adversarial Learning\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkecDId0FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes to solve domain generalization problem in a Bayesian way. The idea is relatively simple: use three hidden variables to encode the domain-related, label-related and the residual information from the original signal. \n\nSome questions:\n-\tMy major concern is the intuition for the proposed algorithm. Though the author uses z_d, z_x and z_y to encode different information from x, it is unclear why they are disentangled. It is also possible that all of them share the same information. In other words, why Figure 2 can be derived from this setting is not well explained. I am also confused on why the model can generalize to unseen target. Only using the learned encoder/decoder from training is hard to generalize.\n-\tFor Figure 2, the first column shows z_d. As a common sense, 30 and 45 degree should be more similar than 30 and 60. However, it seems that the cluster center between 30 and 60 is closer than 30 and 45. Is there any justification? My further concern is whether z_d is meaningful at all. What does z_d look like when applying the model on the unseen testing domain?  (Figure 6 should plot in the context with training domains.)\n-\t A single k-means method can cluster on rotated MNIST by labels. So I think this property should be kept in the feature without any supervised information. However, z_d seems to remove all label information away. I’m not sure why this can happen.\n-\t(Minor) The datasets used for comparison are not discriminative. Maybe the encoder structure is more important than the domain generalization method itself. More challenging datasets are expected. \n\nI would like to improve my score if the author can give a reasonable intuition on why the model can generalize on new domains.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1g6yCkBFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a VAE that can be used in problems in which domain information is available at training time to increase the classification performances on unseen domains. The model can also be used in a semi-supervised setting if unlabelled data is available (even from unseen domains).\n\nAs the authors state, this is a quite common scenario in many interesting applications such as medical imaging. As such, I found the paper interesting to read (the paper is also written quite well).\n\nTo allow the classification of data from any domain, in the inference network the labels d are not used to infer the latent states, but only as an auxiliary loss that forces z_d to capture domain-specific information. However, this makes me wonder if z_d is needed at all? I wouldn't be surprised if the model performed equally well if domain information d was not passed to the model even during training, in which case z_x would also capture domain-specific information. To understand if this is the case, it would be very helpful to add a baseline model in which you use a version of DIVA in which z_d and d are not present both in the generative model and in the inference network.\n\nSince from the technical point of view the novelty of the model is limited, I would have liked to see a stronger experimental section to show the real-life applicability of the model:\n- The MNIST experiments are visually helpful to understand the disentanglement in the model. However you solve a quite simple task. What are the performances of a baseline classifier on this?\n- The malaria cell experiment is definitely more realistic and therefore interesting, and the results are quite convincing (despite being on quite low-resolution images). Since I consider the MNIST experiment a \"toy task\", I think the paper would greatly improve if a new real-world experiment was performed (e.g. other medical imaging datasets).\n\nOverall I liked the paper and I think it is relevant for the ICLR community, therefore I am voting towards acceptance. However, a more convincing experimental section is needed for me to increase the score.  \n\n\nSmall comments:\n- you should make clearer from the abstract that the goal of this paper is to build a domain-invariant classifier (as opposed to solving more VAE-like tasks)\n- in the beginning of section 2.1 you write twice p(x|z_d, z_x, z_d), without z_y\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of domain generalization. The proposed solution, DIVA, introduces a domain invariant variational autoencoder. The latent space can be decomposed into three components: category specific, domain specific, and residual. The authors argue that each component is necessary to capture all relevant information while keeping the latent space interpretable.

This work received mixed scores. Two reviewers recommended weak reject while one reviewer recommended weak accept. There was extensive discussion between the reviewers and authors as well as amongst the reviewers. All reviewers agreed this is an important problem statement and that this work offers a compelling initial approach and experiments for domain generalization. There was disagreement as to whether the contributions as is was sufficient for acceptance. Some reviewers were concerned over similarity to [ref1], this work appears close to the time of ICLR submission and is therefore considered concurrent. However, despite this, there was significant confusion over the proposed solution and whether it is uniquely useful for domain generalization or for other areas like adaptation or transfer learning with reviewers arguing that experiments in these other settings would have helped showcase the benefits of the proposed approach. In addition, there was inconclusive evidence as to whether the two latent components were necessary. 

Considering all discussions, reviews, and rebuttals the AC does not recommend this work for acceptance. The contribution and proposed solution needed substantial clarification and the experiments need additional analysis to explain under what conditions each latent component is needed either to improve performance or for interpretability.

Decision: Reject