Year: 2018

Abstract:
Unsupervised domain adaptation (uDA) models focus on pairwise adaptation settings where there is a single, labeled, source and a single target domain. However, in many real-world settings one seeks to adapt to multiple, but somewhat similar, target domains. Applying pairwise adaptation approaches to this setting may be suboptimal, as they fail to leverage shared information among multiple domains. In this work, we propose an information theoretic approach for domain adaptation in the novel context of multiple target domains with unlabeled instances and one source domain with labeled instances. Our model aims to find a shared latent space common to all domains, while simultaneously accounting for the remaining private, domain-specific factors. Disentanglement of shared and private information is accomplished using a unified information-theoretic approach, which also serves to establish a stronger link between the latent representations and the observed data. The resulting model, accompanied by an efficient optimization algorithm, allows simultaneous adaptation from a single source to multiple target domains. We test our approach on three challenging publicly-available datasets, showing that it outperforms several popular domain adaptation methods.

heading: 1 INTRODUCTION
text: In real-world data, the training and test instances often do not come from the same underlying distribution (Sun et al. (2016)). For example, in the task of object recognition/classification from image data, this may be due to the image noise, changes in the object view, etc., which induce different biases in the observed data sampled during the training and test stage. Consequently, assumptions made by traditional learning algorithms are often violated, resulting in degradation of the algorithms’ performance during inference of test data. Domain Adaptation (DA) approaches (Fernando et al. (2013); Gong et al. (2012); Kodirov et al. (2015); Yoo et al. (2016)) aim to tackle this by transferring knowledge from a source domain (training data) to an unlabeled target domain (test data) to reduce the discrepancy between the source and target data distributions, typically by exploring domain-invariant data structures.
Existing DA methods tackle the adaptation problems in one of the two settings: (semi)supervised DA and unsupervised DA (Csurka (2017)). The former assume that in addition to the labeled data of the source domain, some labeled data from the target domain are also available for training/adapting the classifiers. In contrast, the latter does not require any labels from the target domain but rather explores the similarity in the data distributions of the two domains. In this work, we focus on the unsupervised DA (uDA) scenario, which is more challenging due to the lack of correspondences in source and target labels.
Most works on uDA today focus on a single-source-single-target-domain scenario. However, in many real-world applications, unlabeled data may come from different domains, thus, with different statistical properties but with common task-related content. For instance, we may have access to images of the same class of objects (e.g., cars) recorded by various types of cameras, and/or under different camera views and
at different times, rendering multiple different domains (e.g., datasets). Likewise, facial expressions of emotions, such as joy and surprise, shown by different people and recorded under different views, result in multiple domains with varying data distributions. In most cases, these domains have similar underlying data distributions, which can be leveraged to build more effective and robust classifiers for tasks such as the object or emotion recognition across multiple datasets/domains. To this end, traditional uDA methods focus on the single-source-single-target DA scenario. However, in the presence of multiple domains, as typically encountered in real-world settings, this pair-wise adaptation approach may be suboptimal as it fails to leverage simultaneously the knowledge shared across multiple task-related domains. Recently, Zhao et al. (2017) showed that by having access to multiple source domains can facilitate better adaptation to a single target domain, when compared to the pair-wise DA approach. While this is intuitive due to the access to multiple labelled source domains, offering more adaptation flexibility for the target domain (i.e., by efficiently exploring the data labels across multiple source domains that are most related to the target domain), it comes at the expense of the data labelling in multiple source domains, which can be costly and time-consuming. In either case, a single source domain or readily available multiple source domains, to the best of our knowledge, a simultaneous adaptation to multiple and unlabelled target domains remains an unexplored DA scenario. However, this DA scenario is important as we usually have access to multiple unlabeled domains; yet, the adaptation process is also more challenging due to the lack of supervision in the target domains. Still, multi-target DA can have advantages over a single-target DA when: (i) there is direct knowledge sharing between the source and multiple target domains (fig. 1a), and (ii) the source and a target domain are related through another target domain (fig. 1b). While this seems intuitive, it is critical how the data from multiple unlabelled target domains are leveraged within the multi-target DA approach, in order to improve its performance over the single target DA approaches and naive fusion of multiple target domains.
To this end, we propose a Multi-Target DA-Information-Theoretic-Approach (MTDA-ITA) for single-sourcemulti-target DA. We exploit a single source domain and focus on multiple target domains to investigate the effects of multi-target DA; however, the proposed approach can easily be extended to multiple source domains. This approach leverages the data from multiple target domains to improve performance compared to individually learning from pair-wise source-target domains. Specifically, we simultaneously factorize the information from each available target domain and learn separate subspaces for modeling the shared (i.e., correlated across the domains) and private (i.e., independent between the domains) subspaces of the data (Salzmann et al. (2010)). To this end, we employ deep learning to derive an information theoretic approach where we jointly maximize the mutual information between the domain labels and private (domain-specific) features, while minimizing the mutual information between the the domain labels and the shared (domaininvariant) features. Consequently, the more robust feature representations are learned for each target domain by exploiting dependencies between multiple target domains. We show on benchmark datasets for DA that this approach leads to overall improved performance on each target domain, compared to independent DA for
each pair of source-target domains, or the naive combination of multiple target domains, and state-of-the-art models applicable to the target task.

heading: 2 THE PROPOSED METHOD
text: Without loss of generality, we consider a multi-class (K-class) classification problem as the running example. Furthermore, let (X,Y,D) = {(xi,yi,di)}Ni=0 be a collection of M domains (a labeled source domain, and M − 1 unlabeled target domains), where xi denotes the i-th sample, and yi = [y1i , y2i , ..., yKi ] and di = [d 1 i , d 2 i , ..., d M i ] are the K-D and M -D encoding of the class and domain labels for xi, respectively. Note that the class labels are only available for the source samples.
The latent space representation of the data point x is denoted as z = [zs, zp], where zs and zp are the (latent) shared and private features of the data point x, respectively. By factorizing the joint distribution p(x,y,d, zs, zp) as p(x,y,d, zs, zp) = p(x)p(d)p(zs|x)p(zp|x)p(y|zs), (1) we propose to maximize the following objective function:
L(θs, θp, θc;x,y,d) = λrI(x; z) + λcI(y; zs) + λd ( I(d; zp)− I(d; zs) ) , (2)
where p(x) and p(d) denote the underlying (true) data distribution and domain label distribution, respectively, I(x;y) denotes the Mutual Information between the random variables x and y. λr, λc and λd denote the hyper-parameters controlling the weights of the objective terms. The proposed objective function (2) maximizes the three terms described below:
• I(x; z) : encourages the latent features (both shared and private) to preserve information about the data samples (that can be used to reconstruct x from z).
• I(y; zs): enables to correctly predict the true class label of the samples out of their common shared features.
• I(d; zp) − I(d; zs) : encourages the latent private features to preserve the information about the domain label and penalizes the latent shared features to be domain informative. This not only reduces the redundancy in the shared and private features, but also, penalizes the redundancy of different private spaces, while preserving the shared information.
An additional term could be used to minimize the mutual information between the shared (zs) and private (zp) features. However, computing the mutual information (even approximating it) is intractable due to the highly complex joint distribution p(zs, zp). Since we want zs and zp features to encode different aspects of x, we enforce such constraint by jointly maximizing the term: I(d; zp)− I(d; zs).

heading: 2.1 OPTIMIZATION
text: The following lower bound for mutual information is derived using the non-negativity of KL-divergence (Barber & Agakov (2003)); i.e., Σxp(x|z) ln p(x|z)q(x|z) ≥ 0 gives:
I(x; z) ≥ H(x) + Ep(x,z)[ln q(x|z;φ)] (3)
where H(x) denotes the Shanon Entropy (Lin (1991)) of the random variable x. q(x|z;φ) is any arbitrary distribution parameterized by φ. We need a variational distribution q(x|z;φ) because the posterior distribution p(x|z) = p(z|x)p(x)/p(z) is intractable since the true data distribution p(x) is assumed to be unknown. Similarly, we can derive lower bounds for I(d; zp) ≥ H(d) + Ep(d,zp)[ln q(d|zp;ψ)] and I(d; zs) ≥
H(d) + Ep(d,zs)[ln q(d|zs;ψ)], where q(d|zp;ψ) is any arbitrary distribution parameterized by ψ.1 We further drive lower bound for I(y; zs) as I(y; zs) ≥ H(y) + Ep(y,zs)[ln q(y|zs; θc)], where q(y|zs; θc) is a variational distribution parameterized by θc approximating p(y|zs). Let next Es(x; θs) be a function parameterized by θs that maps a sample x to its corresponding shared feature zs, and Ep(x; θp) be an analogous function which maps x to zp, the feature that is private to each domain (fig. 2). We also define F (zs, zp;φ) as a decoding function mapping the concatenation of the latent features zs and zp to a sample reconstruction x̂, and D(z;ψ) as a decoding function mapping zs and zp to a M -dimensional vector: the predictions of the domain label d̂. Finally, C(zs; θc) is a task-specific function mapping zs to a K-dimensional probability vector of the class label ŷ.
We represent p(d), p(x), p(y) as the empirical distribution of a finite training set (e.g. p(d) = 1 N ∑N i=1 δ(d − di)) as in the case of variational autoencoders (VAE) (Abbasnejad et al. (2017); Pu
et al. (2017)), p(zs|x), p(zp|x) as deterministic functions of x as p(zs|x) = δ ( zs − Es(x; θs) ) and
p(zp|x) = δ ( zp − Ep(x; θp) ) , and the variational distributions q(y|zs), q(x|z) and , q(d|z) as
q(y|zs) = SoftMax(C(zs; θc)), q(d|z) = SoftMax(D(z;ψ)), q(x|z;φ) ∝ exp(‖x− F (z;φ)‖1) (4)
where Softmax(·) denotes the softmax or normalized exponential function (Bridle (1990)), and ‖.‖1 denotes the L1 norm. Then, the optimization task can be posed as a minimax saddle point problem, where we use adversarial training to maximize (2) w.r.t. the parameters (θs, θp, θc), and to minimize (2) w.r.t. the parameters (φ,ψ), using Stochastic Gradient Descent (SGD).
Optimizing the parameters φ of the decoder F
φ̂ = arg min φ LF = λr N N∑ i=1 ‖xi − F ( Es(xi), Ep(xi) ) ‖1. (5)
The decoder F (zs, zp;φ) is trained in such a way so as to minimize the difference between original input x and its decoding from corresponding shared and private features via the decoder F .
1Note that, for simplicity, we shared the parameters ψ between the approximate posterior distributions q(d|zs, ψ) and q(d|zp;ψ).
Optimizing the parameters ψ of the domain classifier D
ψ̂ = arg min ψ LD = − λd N N∑ i=1 d>i lnD ( Es(xi) ) − λd N N∑ i=1 d>i lnD ( Ep(xi) ) . (6)
D(z;ψ) can be considered as a classifier whose task is to distinguish between the shared/private features of the different domains. More precisely, the two terms in Eq. 6 encourage D to correctly predict the domain labels from the shared and private features, respectively.
Optimizing the parameters θc of the label classifier C
θ̂c = arg min θc
{ −H(y)− Ep(y,zs) [ ln q(y|zs) ]} . (7)
Since we have access to the source labels, H(y) is a constant for source samples. we can approximate H[y] for the target samples using the output of the classifier C, leading to the following optimization problem:
θ̂c = arg min θc
LC =− 1
N Ns∑ i=1 yTi lnC ( Es(xi) ) − λc N −Ns N∑ i=Ns+1 C ( Es(xi) )> lnC ( Es(xi) ) +
λc N −Ns N∑ i=Ns+1 C ( Es(xi) )> ln ( 1 N −Ns N∑ i=Ns+1 C ( Es(xi) )) , (8)
where Ns denotes the number of source samples. Intuitively, we enforce the classifier C to correctly predict the class labels of the source samples by the first term in Eq. 8. We use the second term to minimize the entropy of q(y|zs) for the target samples; effectively, reducing the effects of "confusing" labels of target samples, as given by p(y|zs) that leads to decision boundaries occur far away from target data-dense regions in the feature space. The intuition behind the last term is that by minimizing only the entropy (second term), we may arrive at a degenerate solution where every target point xt is assigned to the same class. Hence, the last term encourages the classifier C to have balanced labeling for the target samples where it reaches its minimum, lnK, when each class is selected with uniform probability.
Optimizing the parameter θp of the private encoder Ep
θ̂p = arg min θp LP = λr N N∑ i=1 ‖xi − F ( Es(xi), Ep(xi) ) ‖1 − λd N N∑ i=1 d>i D ( Ep(xi) ) . (9)
The first term in Eq. 9 encourages the private encoder Ep to preserve the recovery ability of the private features. The second term enforces distinct private features be produced for each domain by penalizing the representation redundancy in different private spaces. This, in turn, encourages moving this common information from multiple domains to their shared space.
Optimizing the parameter θs of the shared encoder Es
θ̂s = arg min θs LS = λr N N∑ i=1 ‖x− F ( Es(xi), Ep(xi) ) ‖1 − λc N Ns∑ i=1 yTi lnC ( Es(xi) ) − λd N N∑ i=1 d>i lnD ( Es(xi) ) − λc N −Ns N∑ i=Ns+1 C ( Es(xi) )> lnC ( Es(xi)
) +
λc N −Ns N∑ i=Ns+1 C ( Es(xi) )> ln ( 1 N −Ns N∑ i=Ns+1 C ( Es(xi) )) . (10)
The first term in Eq. 10 encourages the shared encoder Es to preserve the recovery ability of the shared features. The second term is the source domain classification loss penalty that encourages Es to produce discriminative features for the labeled source samples. The third term simulates the adversarial training by trying to fool the domain classifier D when predicting the domain labels d, given the shared features zs. The effect of this is two-fold: (i) the rendered shared features are more distinct from the corresponding private features, (ii) the shared features of different domains are encouraged to be similar to each other. The last two terms encourage Es to produce the shared features for target samples so that the classifier is confident on the unlabeled target data, driving the shared features away from the decision boundaries. To train our model, we alternate between updating the shared encoder Es, the private encoder Ep, the decoder F , the classifier C, and the domain classifier D using the SGD algorithm (see Algorithm 1 in Appendix E for more details).

heading: 3 RELATED WORK
text: There has been extensive prior work on domain adaptation (Csurka (2017)). Recent papers have focused on transferring deep neural network representations from a labeled source dataset to an unlabeled target domain, where the main strategy is to find a feature space such that the confusion between source and target distributions in that space is maximized ( Rebuffi et al. (2017); Benaim & Wolf (2017); Courty et al. (2017); Motiian et al. (2017); Saito et al. (2017); Zhang et al. (2017); Yan et al. (2017); Bousmalis et al. (2017)). For this, it is critical to first define a measure of divergence between source and target distributions. For instance, several methods have used the Maximum Mean Discrepancy (MMD) loss for this purpose (Bousmalis et al. (2017); Zellinger et al. (2017); Long et al. (2014)). MMD computes the norm of the difference between two domain means in the reproducing Kernel Hilbert Space (RKHS) induced by a pre-specified kernel. The Deep Adaptation Network (DAN) (Long et al. (2015)) applied MMD to layers embedded in a RKHS, effectively matching higher order statistics of the two distributions. The deep Correlation Alignment (CORAL) method (Sun & Saenko (2016)) attempts to match the mean and covariance of the two distributions. Deep Transfer Network (DTN) (Zhang et al. (2015)) achieved source/target distribution alignment via two types of network layers based on MMD distance: the shared feature extraction layer, which learns a subspace that matches the marginal distributions of the source and the target samples, and the discrimination layer, which matches the conditional distributions by classifier transduction.
Recently proposed unsupervised DA methods (Rebuffi et al. (2017); Benaim & Wolf (2017); Courty et al. (2017); Motiian et al. (2017); Saito et al. (2017); Zhang et al. (2017)) operate by training deep neural networks using adversarial training, which allows the learning of feature representations that are simultaneously discriminative of source labels, and indistinguishable between the source and target domain. For instance, Ganin & Lempitsky (2015) proposed a DA mechanism called Domain-Adversarial Training of Neural Networks (DANN), which enables the network to learn domain invariant representations in an adversarial way by adding a domain classifier and back-propagating inverse gradients. Adversarial Discriminative Domain Adaptation (ADDA) (Tzeng et al. (2017)) learns a discriminative feature subspace using the source labels, followed by a separate encoding of the target data to this subspace using an asymmetric mapping learned through a domain-adversarial loss. Liu et al. (2017) makes a shared-latent space assumption and proposes an unsupervised image-to-image translation (UNIT) framework based on Coupled GANs (Liu & Tuzel (2016)). Another example is the pixel-level domain adaptation models that perform the distribution alignment not in the feature space but directly in raw pixel space. PixelDA (Bousmalis et al. (2017)) uses adversarial approaches to adapt source-domain images as if drawn from the target domain while maintaining the original content.
While these approaches have shown success in DA tasks with single source-target domains, they are not designed to leverage information from multiple domains simultaneously. More recently, Zhao et al. (2017) introduced an adversarial framework called MDAN for multiple source single target domain adaptation where a domain classifier, induced by minimizing the H-divergence between multiple source and a target domain, is
used to align their feature distributions in a shared space. Instead, in our approach we focus on multi-target DA where we perform adaptation of multiple unlabelled target domains. Although both our model and MDAN use the similar notion of the domain classifier to minimize the domain mismatch in shared space, the domain classifier induced by our information-theoretic (IT) loss also acts to separate domains in the private space (see Eqs. 6 & 9 for more details), improving the essential reconstruction ability, similar to (Bousmalis et al. (2016)). We provided how our model is related to IT representation learning approaches, and multiple domain transfer networks in Appendices A and B respectively. In Appendix C, we also clearly contrasted our model with DSN model which also uses the notion of auto-encoders to explicitly separate the feature representations private to each source/target domain from those that are shared between the domains.

heading: 4 EXPERIMENTAL RESULTS
text: We compare the proposed method with state-of-the-art methods on standard benchmark datasets: a digit classification task that includes 4 datasets: MNIST (LeCun et al. (1998)), MNIST-M (Ganin et al. (2016)), SVHN (Netzer et al. (2011)), USPS (Tzeng et al. (2017)), Multi-PIE expression recognition dataset2, and PACS multi-domain image recognition benchmark (Li et al. (2017)), a new dataset designed for the crossdomain recognition problems (the details for this experiment is available in Appendix G). Fig. 3 illustrates image samples from different datasets and domains. We evaluate the performance of all methods with classification accuracy metric. We repeated each experiment 5 times and report the average and the standard deviation of the accuracy.
We used ADAM (Kingma & Ba (2015)) for training; the learning rate was set to 0.0002 and momentum parameters to 0.5 and 0.999. We used batches of size 16 from each domain, and the input images were meancentered/rescaled to [−1, 1]. The hyper-parameters are empirically set as λr = 1.0, λc = 0.01, λd = 0.20. For the network architecture, our private/shared encoders consisted of three convolutional layers as the front-end and four basic residual blocks as the back-end. The decoder consisted of four basic residual blocks as the front-end and four transposed convolutional layers as the back-end. The discriminator and the classifier consisted of stacks of convolutional layers. We used ReLU for nonlinearity. Tanh function is used as the activation function of the last layer in the decoder F for scaling the output pixels to [−1, 1]. The details of the networks are given in Appendix D.
The quantitative evaluation involves a comparison of the performance of our model to previous work and to Source Only and 1-NN baselines that do not use any domain adaptation. For Source Only baseline, we train our model only on the unaltered source training data and evaluate on the target test data. We compare the proposed method MTDA-ITA with several related methods designed for pair-wise source-target adaptation: CORAL (Sun & Saenko (2016)), DANN (Ganin & Lempitsky (2015)), ADDA (Tzeng et al. (2017)), DTN (Zhang et al. (2015)), UNIT (Liu et al. (2017)), PixelDA (Bousmalis et al. (2017)), and DSN (Bousmalis et al. (2016)). We reported the results of two following baselines: (i) one is to combine all the target domains into a single one and train it using MTDA-ITA, which we denote as (c-MTDA-ITA). (ii) the other one is to train multiple MTDA-ITA separately, where each one corresponds to a source-target pair which we denote as (s-MTDA-ITA). For completeness, we reported the results of the competing methods by combining all the target domains into a single one (denoted by c-DTN, c-ADDA, and c-DSN) as well. We also extend DSN to multiple domains by (i) having one private encoder for all domains denoted by (1p-DSN), (ii) adding multiple private encoders to it denoted by (mp-DSN) and contrast them with our model.

heading: 4.1 DIGITS DATASETS
text: We combine four popular digits datasets (MNIST, MNIST-M, SVHN, and USPS) to build the multitarget domain dataset. All images were uniformly rescaled to 32 × 32. We take each of MNIST-M,
2http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html
SVHN, USPS, and MNIST as source domain in turn, and the rest as targets. We use all labeled source images and all unlabeled target images, following the standard evaluation protocol for unsupervised domain adaptation (Ganin et al. (2016); Long et al. (2016). We show the accuracy of different methods in Table 1. Additional results are available in Appendix F. The results show that first of all cMTDA-ITA has worse performance than sMTDA-ITA and MTDA-ITA. We have similar observations for ADDA, DTN, and DSN that demonstrates a naive combination of different target datasets can sometimes even decrease the performance of the competing methods. Furthermore, MTDA-ITA outperforms the state-of-the-art methods in most of domain transformations. The higher performance of MTDA-ITA compared to other methods is mainly attributed to the joint adaptation of related domains where each domain could benefit of other related domains. Furthermore, from the results obtained, we see that it is beneficial to use information coming from unlabeled target data (see Eq. 8 for updating the classifier C) during the learning process, compared to when no data from target domain is used (See the ablation study section for more information). Indeed, using our scheme, we find a representation space in which embeds the knowledge from the target domain into the learned classifier. By contrast, the competing methods do not provide a principled way of sharing information across all domains, leading to overall lower performance. The results also verify the superiority of MTDA-ITA over both mp-DSN, and 1p-DSN. This can be due to (i) having multiple private encoders increase the number of parameters that may lead to mp-DSN overfitting, (ii) superiority of the MTDA-ITA’s domain adversarial loss over the DSN’s MMD loss to separate the shared and private features, (iii) utilization of the unlabeled target data to regularize the classifier in MTDA-ITA.

heading: 4.2 MULTI-PIE DATASET
text: The Multi-PIE dataset includes face images of 337 individuals captured from different expressions, views, and illumination conditions (fig. 3(c)). For this experiment, we use 5 different camera views (positions) C05, C08, C09, C13, and C14 as different domains (Fig. 3(c)) and the face expressions (normal, smile, surprise, squint, disgust, scream) as labels. Each domain contains 27120 images of size 64× 64× 3. We used each view as the source domain, in turn, and the rest as targets. We expect the face inclination angle to reflect the complexity of transfer learning. Tab. 2 shows the classification accuracy for C13 and C14 as source domain (the results for views C05, C08 and C09 as source domain are available in Appendix F). As can be seen, MTDA-ITA achieves the best performances as well as the best scores in most settings that verifies the effectiveness of MTDA-ITA for multi-target domain adaptation. Clearly, with the increasing camera angle, the image structure changes up to a certain extent (the views become heterogeneous). However, our method produces better results even under such very challenging conditions.

heading: 4.3 ABLATION STUDIES
text: We performed an ablation study on the proposed model measuring impact of various terms on the model’s performance. To this end, we conducted additional experiments for the digit datasets with different components ablation, i.e., training without the reconstruction loss (denoted as MTDA-woR) by setting λr = 0, training without the classifier entropy loss (denoted as MTDA-woE) by setting λc = 0, training without the multidomain separation loss (denoted as MTDA-woD) by setting λd = 0. As can be seen from fig. 4, disabling each of the above components leads to degraded performance. More precisely, the average drop by disabling the classifier entropy loss is ≈ 3.5%. Similarly, by disabling the reconstruction loss and the multi-domain separation loss, we have ≈ 4.5% and ≈ 22% average drop in performance, respectively. Clearly, by disabling the multi-domain separation loss, the accuracy drops
significantly due to the severe data distribution mismatch between different domains. The figure also demonstrates that leveraging the unlabeled data from multiple target domains during training enhances the generalization ability of the model that leads to higher performance. In addition, the performance drop caused by removing the reconstruction loss , i.e., without the private encoder/decoder, indicates (i) the benefit of modeling the latent features as the combination of shared and private features, (ii) the ability of the model’s domain adversarial loss to effectively learn those features. In order to examine the effect of the private features on the model’s classification performance, we took the MTDA-ITA and trained it without the private encoder (denoted as MTDA-woP). As fig. 4 shows, without the private features, the model performed consistently worse (≈ 2% average drop in performance) in all scenarios. This demonstrates explicitly modeling what is unique to each domain can improve the model’s ability to extract domain–invariant features. In summary, this ablation study showed that the individual components bring complimentary information to achieve the best classification results.

heading: 4.4 FEATURE VISUALIZATION
text: We use t-SNE (Maaten & Hinton (2008)) on Digit dataset to visualize shared and private feature representations from different domains. Fig. 6 shows shared and private features from source (SVHN) and target domains before (a),(b) and after adaptation (c),(d). MTDA-ITA significantly reduces the domain mismatch for the shared features (circle markers in fig. 6d, strong mixing of domain labels in this cluster, fig. 6c) and increases it for the private features (triangle markers, pure and well-separated domain clusters in fig. 6c). This is partially due to the proposed multi-domain separation loss through the use of the domain classifier D, which penalizes the domain mismatch for the shared features and rewards the mismatch for the private features. Moreover, as supported by the quantitative results in tab. 1, joint adaptation of related domains and the classifier, accomplished through the model, leads to superior class separability, compared to original features. This is depicted in fig. 6d, where the points in the shared space (large cluster) are grouped into class-specific subgroups (color indicates class label), while they are mixed in private spaces (smaller clusters). This is in contrast to fig. 6b, where original features show no class-specificity.
We also show the learned shared and private features for the models MTDA-woE, MTDA-woP, MTDAwoR, and MTDA-woD, in figs. 6e to 6l. Note that since the private encoder Ep is disabled for MTDA-woR, and MTDA-woP, no private features are depicted in figs. 6g to 6j. The class label separation in the shared space for MTDA-woE, MTDA-woP, and MTDA-woR, figs. 6f, 6h and 6j, is still evident but not as strong as in the full model, fig. 6d, corroborating the small loss in classification accuracy observed in fig. 4a. On the other hand, MTDA-woD has significant mixing of class labels in the shared space, fig. 6l, more so than MTDA-woE, MTDA-woR, and MTDA-woP, implying worse classification prediction in fig. 4a due to the severe mismatch between different domains. Since our model uses one private encoder for all target domains, we also contrasted the visualization of DSN model with one private encoder 1p-DSN in Appendix H.

heading: 5 CONCLUSION
text: This paper presented an information theoretic end-to-end approach to uDA in the context of a single source and multiple target domains that share a common task or properties. The proposed method learns feature representations invariant under multiple domain shifts and simultaneously discriminative for the learning task. This is accomplished by explicitly separating representations private to each domain and shared between source and target domains using a novel discrimination strategy. Our use of a single private domain encoder results in a highly scalable model, easily optimized using established back-propagation approaches. Results on three benchmark datasets for image classification show superiority of the proposed method compared to the state-of-the-art methods for unsupervised domain adaptation of visual domain categories.

heading: A CONNECTION TO INFORMATION THEORETIC REPRESENTATION LEARNING
text: The idea of using information theoretic (IT) objectives for representation learning was originally introduced in (Tishby & Zaslavsky (2015)). Since their approach for optimizing the IT objective functions relied on the iterative Blahut Arimoto algorithm (Tishby & Zaslavsky (2015)), it is not feasible to apply to deep neural network (DNN) frameworks. Similar to our approach, there have been some recent works (Mohamed & Rezende (2015); Chalk et al. (2016); Alemi et al. (2018b; 2016; 2018a)) to approximate the MI by applying variational bounds on MI, though not in the context of domain adaptation.
Mohamed & Rezende (2015) utilized the variational bounds on MI, and apply it to DNNs in the context of reinforcement learning. Chalk et al. (2016) and Alemi et al. (2016), developed the same variational lower bound In the context of Information Bottleneck (IB) principle (Tishby & Zaslavsky (2015)), where the former applied it to sparse coding problems, and used the kernel trick to achieve nonlinear mappings, whereas the latter applied it to DNNs to handle large datasets thanks to the SGD algorithm. Achille & Soatto (2018) proposed a variational bound on the MI in the context of IB, from the perspective of variational dropout and demonstrated its utility in learning disentangled representations for variational autoencoders.
The main difference between our method and the above methods is that these methods throw away the information in the data not related to the task by minimizing the mutual information between the data points and the latent representations that may lead to ignoring the individual characteristics (private features) of the datasets in a multiple dataset regime, whereas our method explicitly models what is unique to each domain (dataset) that improves the model’s ability to extract domain–invariant features.
In the unsupervised representation learning literature, our work is also related to the VAE-based models (Bowman et al. (2015)). However, we propose to tackle the task using our IT approach using deterministic mappings instead of the traditional evidence lower bound (ELBO) optimization with stochastic mappings. In contrast to the unsupervised representation learning approaches, our setting also allows us to further improve the latent representation using the labeled data in the source domain while leveraging the sharing of dependencies across different target domains.

heading: B CONNECTION TO MULTIPLE DOMAIN TRANSFER NETWORKS
text: Recent studies have shown remarkable success in multiple domain transfer (MDT) (Choi et al. (2017); Anoosheh et al. (2017); Kameoka et al. (2018); Hao et al. (2018)) though not in the context of the image classification, rather in the context of image generation. Choi et al. (2017) proposed StarGAN, a generative adversarial network capable of learning mappings among multiple domains in the contest of image to image translation framework. The goal of StarGAN is to train a single generator G though this requires passing in a vector along with each input to the generator specifying the output domain desired, that learns mappings among multiple domains. To achieve this, G is trained to translate an input image x into an output image x′ conditioned on the target domain label d, G(x,d)→ x′. Similar to our domain classifier module D, they introduce an auxiliary classifier that allows a single discriminator to control multiple domains.
Anoosheh et al. (2017) introduced ComboGAN, which decouples the domains and networks from each other. Similar to our encoder/decoder modules, ComboGAN’s generator networks contain encoder/decoders assigning each encoder and decoder to a domain. They combine the encoders and decoders of the trained model like building blocks, taking as input any domain and outputting any other. For example during
inference, to transform an image x from an arbitrary domain X to x′ from domain X′, they simply perform x′ = GX′,X(x) = DecoderX′(EncoderX(x)). The result of EncoderX(x) can even be cached when translating to other domains as not to repeat computation.
The main differences between the MDT methods and ours is that, unlike our method which does domain alignment in feature space, MDT methods adapt representations not in feature space but rather in raw pixel space; translating samples from one domain to the “style” of a other domains. This works well for limited domain shifts where the domains are similar in pixel-space, but can be too limiting for settings with larger domain shifts that results in poor performance in significant structural change of the samples in different domains.

heading: C CONNECTION TO DOMAIN SEPARATION NETWORKS
text: The method closest to our work is Domain Separation Networks (DSN) (Bousmalis et al. (2016)), which use the notion of auto-encoders to explicitly separate the feature representations private to each source/target domain from those that are shared between the domains. Although extending DSN to multiple domains might seem trivial, DSN requires an autoencoder per domain, making the model impractical in the case of more than a couple of domains.
The overall loss of DSN consists of a reconstruction loss for each domain modeled by a shared decoder, a similarity loss such as MMD, which encourages domain invariance modeled by a shared encoder, and a dissimilarity loss modeled by two private encoders: one for the source domain and one for the target domain. While one could attempt to generalize DSN to multiple target domains by having individual per-target domain private encoders, doing so would prove problematic when the number of target domains is large — each private encoder would require a large "private" dataset to learn the private parameters. Precisely, for multiple (M) target domains, we could train a DSN model with one shared encoder, M + 1 private encoder (one for each domain), and one shared decoder. This leads to M + 3 models to train that implies the number of models increases linearly with the number of domains, as does the required training time. Second, DSN uses an orthogonality constraint among the shared and the private representations which may not be strong enough to remove redundancy and enforce disentangling among different private spaces. Precisely, DSN defines the loss via a soft subspace orthogonality constraint between the private and shared representation of each domain. However, it does not enforce the private representation of different domains to be different that may result in redundancy of different private spaces.
In addition, DSN enforces separation of spaces using the notion of Euclidean orthogonality, e.g., ‖zs − zp‖2. In case of multiple target domains, this would result in learning of all pairs of private spaces independently. To address those deficiencies, we first explicitly couple different private encoders into a single private encoder model,Eθp of fig. 2 , which allows us to generalize to an arbitrary number of target domains. To assure that the information among the private and shared spaces is not shared (i.e., "orthogonal"), we define an informationtheoretic criteria enforced by a domain classifier,Dψ of fig. 2, which aims to segment the private space into clusters that correspond to individual target domains. By using Dψ within the adversarial framework, MTDA-ITA learns simultaneously the shared and private features from different domains (see fig. 6). We showed in Sec. 4 that our model performs better than the trivial extension of DSNs to the multi-domain case.

heading: D NETWORK ARCHITECTURE
text: The network architecture used for the experiments is given in tab. 3. We use the following abbreviation for ease of presentation: N=Neurons, K=Kernel size, S=Stride size, D=Number of Domains, C=number of Classes. The transposed convolutional layer is denoted by DCONV. The residual basic block is denoted as RESBLK.

heading: E PROPOSED MODEL’S ALGORITHM
text: The detailed optimization process of the proposed model is shown in Algorithm 1.

heading: F ADDITIONAL EXPERIMENTS FOR DIGIT AND MULTI-PIE DATASETS
text: The additional experiments for Digit dataset where we set MNIST-M and USPS as source domain is available in tab. 4. The additional experiments for Multi-PIE dataset where we set C05, C08 and C09 as source domain is available in tabs. 5 & 6.

heading: G PACS DATASET
text: This dataset contains 9991 images (227 × 227 × 3 dimension) across 7 categories (‘dog’, ‘elephant’, ‘giraffe’,‘guitar’, ‘house’, ‘horse’ and ‘person’) and 4 domains of different stylistic depictions (‘Photo’, ‘Art painting’, ‘Cartoon’ and ‘Sketch’). The very diverse depiction styles provide a significant gap between
domains, coupled with the small number of data samples, making it extremely challenging for domain adaptation. Consequently, the dataset was originally used for multi-source to single target domain adaptation (Li et al. (2017)). Instead, we tackle a significantly more challenging problem of single-source to multiple target adaptation. Tab. 7 shows the classification accuracy of various methods. MTDA-ITA consistently
Algorithm 1 MDTA-ITA Algorithm Require: {X,Y,D}:M domain datasets. λr, λc, λd: Model hyper-parameters. Ensure: θs, θp, θc, φ, ψ: Model parameters.
1: Initialize θs, θp, θc, φ, ψ; 2: repeat 3: Sample a mini-batch from each of source/target domain datasets. 4: Update {θs} by minimizing Ls in Eq.(10) through the gradient descent: θs = θs − η ∂Ls∂θs . 5: Update {θp} by minimizing Lp in Eq.(9) through the gradient descent:θp = θp − η ∂Lp∂θp . 6: Update {θc} by minimizing Lc in Eq.(8) through the gradient descent:θc = θc − η ∂Lc∂θc . 7: Update {φ} by minimizing Lφ in Eq.(6) through the gradient descent:φ = φ− η ∂Lφ∂φ . 8: Update {ψ} by minimizing Lψ in Eq.(5) through the gradient descent:ψ = ψ − η ∂Ls∂ψ . 9: until Convergence;
10: return {θs, θp, θc, φ, ψ}.
achieves the best performance for all transfer tasks. Evaluations were obtained by training all models (ADDA, DSN, and ours) from scratch on the PACS dataset. Note that the overall performance figures are low due to the extreme difficulty of the transfer task, induced by large differences among domains.

heading: H ANALYSIS OF SHARED/PRIVATE SPACE EMBEDDING
text: In the experiments conducted, we showed that our approach is able to achieve better performance than the competing methods including the extended DSN with one private encoder (1p-DSN) which is the most similar method to ours.
Indeed, fig. 5 depicts the embedding of the MTDA-ITA learned private/shared features, those of 1p-DSN and the original features from different domains for Digit datasets (SVHN is the source).
Notice that both MTDA-ITA and 1p-DSN reduces the domain mismatch for the shared features (circle markers in fig. 5) and separate the shared features from private features. On the other hand, MTDA-ITA increases the domain separation for the private features (triangle markers, pure and well-separated domain clusters in fig. 5c) while 1p-DSN is unable to enforce the private representation of different domains to be different (fig. 5e) that may result in redundancy of different private spaces. This is partially due to the proposed multi-domain separation loss through the use of the domain classifierD, which penalizes the domain mismatch for the shared features and rewards the mismatch for the private features, something the 1p-DSN fails to account for. Moreover, as supported by the quantitative results in tab. 1, the class label separation in
the shared space for 1p-DSN, fig. 5f, is still evident but not as strong as in the MTDA-ITA, fig. 5d. This can be attributed to the lack of redundancy in the private space that helps MTDA-ITA to learn more disentangled shared features and usage of the target samples during training, something the 1p-DSN fails to account for.
Domains Classes
21

Review summaries:
This paper investigates multi-target domain adaptation which is an unexplored domain adaptation scenario compared with adapting single/multiple source to single target. A mutual information-based loss is proposed to encourage part of the features to be domain-specific while the other part to be domain-invariant. Instead of optimizing the proposed loss which is intractable, this work proposes to use neural network to model the relative functions and optimize proposed loss’ lower bound by SGD.