Domain Adaptation with Invariant Representation Learning: What Transformations to Learn?
Year: 2022

Abstract:
Unsupervised domain adaptation, as a prevalent transfer learning setting, spans many real-world applications. With the increasing representational power and applicability of neural networks, state-of-the-art domain adaptation methods make use of deep architectures to map the input features X to a latent representation Z that has the same marginal distribution across domains. This has been shown to be insufficient for generating optimal representation for classification, and to find conditionally invariant representations, usually strong assumptions are needed. We provide reasoning why when the supports of the source and target data from overlap, any map of X that is fixed across domains may not be suitable for domain adaptation via invariant features. Furthermore, we develop an efficient technique in which the optimal map from X to Z also takes domain-specific information as input, in addition to the features X . By using the property of minimal changes of causal mechanisms across domains, our model also takes into account the domain-specific information to ensure that the latent representation Z does not discard valuable information about Y . We demonstrate the efficacy of our method via synthetic and real-world data experiments. The code is available at: https://github.com/DMIRLAB-Group/DSAN.

heading: 1 Introduction
text: Unsupervised domain adaptation (UDA) is a common setting for supervised learning, in which the labeled training and unlabeled test data come from different distributions. More formally, given features X ∈ Rd and labels Y ∈ R, we observe labeled source and unlabeled target domain instances, represented by (xS ,yS) = (xSk , y S)mSk=1 and x T k = (x T )mTk=1 respectively, where P S(X,Y ) ̸= P T (X,Y ), and mS and mT are the number of observations in the source and target domains, respectively. The main challenge of domain adaptation is to use the given source domain observations for learning a predictor that will perform well in the target domain. To do so, the procedure needs to make use of some similarities between the two domains. One way to formalize this is by making assumptions about how the joint distribution joint P (X,Y ) changes across domains. For example, in the well-studied setting of covariate shift [31, 44, 20, 35, 2, 8], the marginal distribution P (X) changes while the conditional distribution P (Y |X) (i.e. the optimal predictor) is shared across domains.
However, in many real-world applications, P (Y |X) can also change, and this requires making use of further assumptions. One such assumption is that the factorization P (X,Y ) = P (Y )P (X|Y ) allows for addressing the changes in P (Y ) and P (X|Y ) independently in a situation where their
∗These authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
respective changes are simple and easier to capture [48, 30]. In this particular setting, the problem is generally broken down into: (1) Target shift: P (Y ) changes across domains while P (X|Y ) stay the same [34, 21, 29, 48] (2) Conditional shift: P (X|Y ) changes across domains but P (Y ) stays the same [48, 26, 24]. (3) Conditional-target shift: Both P (X|Y ) and P (Y ) change independently across domains - the most general setting under this generating process assumption [48, 6]. Then, assumptions about the changes of the factors of the joint distrubution can be made so that the problem is solvable, such as location-scale transformation [48, 26], or that the changing parameters lie on a low-dimensional manifold [33], and algorithms can be designed to enforce these constraints and make use of them for prediction in the target domain.
Another fruitful view of the problem is through the lens of representation learning, due to wide-spread applicability of neural architectures for many real-world problems. In particular, state of the art deep learning techniques harness the high representational capacity of neural networks to transform the input data into a latent feature representation which is predictive of the target variable Y in the source domain, and has the same distribution across domains. Formally, this means learning a function (encoder) ϕ : X → Z from the input space to a latent space Z , such that PS(Z) = PT (Z). At the same time, a function h : Z → Y can be learnt to minimize the risk in the labeled source domain [1]. The hope is then, that the overall function g := h ◦ ϕ will have low prediction risk in the target domain. However, the above-described theoretical and methodological framework does not guarantee that the learnt representation Z will have any relevant information for predicting Y in the target domain. Namely, one can easily have a situation in which the learnt representation Z is marginally invariant (PS(Z) = PT (Z)), but not conditionally invariant (PS(Z|Y ) ̸= PT (Z|Y )), as discussed in [53]. This means that the learnt function g := h ◦ ϕ can have very good prediction performance in the source domain, but generalize very poorly to the target domain.
In many methods, the same encoding function ϕ(X) across domains is used to learn invariant latent representations. This enjoys computational benefits and makes the learning procedure relatively simple, and the vast majority of approaches ([12, 22, 25, 23, 17] among many) employ this technique. However, in certain situations, the same encoding function across domains cannot learn a marginally invariant representation that is optimal for classification in the target domain. There are certain studies ([37, 38, 4]) which implement domain-specific encoders ϕS and ϕT . Unfortunately, such methods suffer from the following drawbacks: (1) the exact motivation behind having separate encoders for each domain is not clear; (2) including two separate encoders may be inefficient because it greatly increases the number of parameters that we need to learn; (3) in the field of UDA based on invariant
representation learning, there is still no principled way to guarantee that the learned marginally invariant representation Z has sufficient structural (semantic) information, and has the potential to be conditionally invariant.
In this paper we assume the setting of conditional shift, and we make use of the data-generating process to: (i) justify the use of two separate encoding functions in order to infer the latent representation, (ii) implement the two encoding functions more efficiently, and (iii) constrain the latent representation Z to have meaningful structure which is useful for prediction in the target domain. In Section 2, we first motivate the use of two separate encoders to infer Z, via rigorous treatment and an illustrative example. Subsequently, in Chapter 3, we introduce an efficient way to implement two separate functions for inferring Z. In Chapter 3 we shall also introduce a principled way to ensure that the latent representation Z contains useful information for prediction, and finally, in Chapter 4 we provide empirical evaluation.

heading: 1.1 Related Work
text: As previously mentioned, there is a vast body of work exploring approaches to make use of latent invariant representations for the purposes of unsupervised domain adaptation, using both deep learning and more traditional techniques. Namely, invariant representations were considered as a linear projection in [26], where theoretical guarantees were established regarding the conditional
invariance of the latent representation and the identifiability of the joint distribution in the target domain. Furthermore, this framework was generalized to nonlinear maps to invariant latent representations, parameterized by neural network termed DANN [12]. In this framework, the constraint that PS(Z) = PT (Z) can be enforced in various ways, including adversarial training, Maximum Mean Discrepancy [22], Wassertstein Distance [9] and Margin Disparity Discrepancy [51]. However, none of these studies have considered using two separate encoders for the two domains.
Subsequently, this reasoning has been considered in combination with image translation (CYCADA) [19]. Another assumption frequently made in recent studies is that the input features X have a clustering structure in which each cluster has a different label Y . Therefore, further directions were pursued by incorporating additional information contained in the unlabeled target domain data, such as pseudo-labels provided by the classifier which is initially trained on the labeled source domain data [23, 41] (termed CDAN and SLPP, respectively). In addition, one can also make use of this assumption to enforce that the prediction function g does not pass through high-density regions in input space, as performed by the DIRT-T algorithm [32]. The method by [37] makes use of two separate transformation functions, but does not regularize their output to preserve semantic information in any way. In addition to these, approaches, the DSN algorithm [4] takes into models the changing parameters of P (X,Y ) in the latent representation and enforces that they are different from the invariant representation Z. The only method that we are aware of which takes domain-specific information into account is the study by [38], in which domain information is given and continuous, and therefore contains a lot more information than the discrete domain index that is typically considered. In [17], a spherical architecture is used for ϕ, however there is only one encoder for both domains. The study by [27] performs bridging between the source and the target domains using data-augmentation, but this is a technique specific to image data.
In addition to domain adaptation methodologies in the setting where the is only one labeled source domain, there have been several efforts to tackle the problem of multiple-source unsupervised domain adaptation. Namely, the method by [3] tackles the problem by using modified kernel SVMs to operate on both input features X and marginal distributions PX . Furthermore, the method by [46] assumes that the target domain is a linear mixture of the source domains, and infers the linear coefficients and makes use of them for prediction in the target domain. Furthermore, in [47], data-generating process information is discovered from the multiple-domain data, and made use for recovering the changing parameters in the target domain in order to construct a predictor for it.
While the above-mentioned methods use various approaches to improve the quality of the learnt invariant latent representation and show competitive performance, there is no guarantee that these approaches are general enough to encompass all scenarios. Namely, the challenge that existing methods face is two-fold: (1) there is no principled way to ensure that marginal invariance (PS(Z) = PT (Z)) preserves conditional invariance (PS(Z|Y ) = PT (Z|Y )), (2) there are cases (especially in lower-dimensional datasets) where the encoding function ϕ needs domain-specific information, in addition to the features X . The aim of this paper is to address these two issues. In this paper, we introduce the first invariant representation learning method that makes use of the data-generating process to justify and efficiently use domain-specific information in the encoder ϕ, and to properly constrain it to ensure that Z contains relevant label-specific information.

heading: 2 What Does it Take to Learn a Useful Invariant Representation?
text: To gain insights into what information is necessary for learning an intermediate representation Z via ϕ, let us consider the data-generating process depicted on Figure 1. In this graphical model, the label Y is generated first from its prior distribution P (Y ). Then, the invariant representation Z is generated from Y via the generating mechanism P (Z|Y ) and X is subsequently generated from P (X|Z; θX), where θX are the changing parameters of P (X|Y ) across domains. We consider θX as a parameter vector with a constant value for each point within a domain, and we generally assume that θX contains little information, i.e. the change across domains in P (X|Y ) is minimal. This generating process, in part or in whole, can be exploited in many classification applications. For example, in object classification in images, Y is the label (the object the picture was taken of), X is the observed pixels, and Z can correspond to latent features with semantic relevance, such as some function of the edges and contours on the image. In this case, θX can correspond to different resolution, illumination conditions, or other environment-specific changes that are not relevant for predicting the class Y . In this graphical representation, we see that generally speaking, Z is conditionally dependent from θX given X , although they may be marginally independent. This implies that in order to recover Z, given that X is considered, the information of θX should also be considered in the transformation, because it is in the Markov Blanket of Z (the Markov Blanket of a variable of interest consists of all other variables that are either its parents, its children, or the parents of its children in the probabilistic graphical model. The Markov Blanket represents all of the information needed to predict the variable of interest).
To gain intuition why considering the Markov Blanket is important, consider the following scenario, depicted in Figure 2a. In this setting, we are given the source domain with Uniform distribution, and the target domain with Gaussian distribution. Here, we have a region A ⊂ X (highlighted in red), in which both of the domains’ distributions have support, so there is support overlap. However, each domain has a different density in this region. The following proposition claims that in this situation, we cannot infer an invariant representation Z from X with a single encoder ϕ:
Proposition 1: Let A ⊂ X be a region in input space such that PS(X ∈ A) > 0 and P T (X ∈ A) > 0, and PS(X ∈ A) ̸= P T (X ∈ A). Furthermore, let ϕ : X → Z be an encoder s.t. A = {a : ϕ(a) ∈ B} for some B ⊂ Z . Then, there is no function ϕ s.t. PS(ϕ(X) ∈ B) = P T (ϕ(X) ∈ B).
All omitted proofs can be found in the supplementary materials. This proposition implies that if the supports of the marginal distributions of the source and target domains overlap, and if the distributions are different in the specific region of the support overlap (set A), then we cannot use a fixed transformation to transform the data in that region to an invariant representation in Z space.
There is an additional case in which a single encoder ϕ is not sufficient to learn an optimal latent representation Z. The scenario is illustrated on Figure 2b. In this case, even though both domains have the same distribution P (X) in the region of support overlap between the source and the target domain (labeled in red), the value of Y is opposite between the two domains. The following proposition demonstrates that in this setting, a single encoding function ϕ is inherently sub-optimal:
Proposition 2: Let the true labeling functions in the source and target domain be fS , fT : X → Y , respectively. Let A ⊂ X be a region s.t. fS(a) ̸= fT (a),∀a ∈ A. Let g : X → Y be a composition of a representation learner ϕ : X → Z and a classifier h : Z → Y . If ϕ is the same function across domains, then for a 0-1 loss, the risk over the region A: ϵA(g) = ϵAS (g) + ϵ A T (g) ≥ 1, where ϵAS (g) and ϵAT (g) are the source and target domain risks respectively.
In this scenario, no matter what criterion one uses to find the representation for downstream classification using h, as long as the two domains use the same transformation ϕ for the representation, the points in A cannot be correctly classified in the target domain, if they are correctly classfied in the source domain. This is because the final classifer of the data, g = h ◦ ϕ, is fixed across domains. As a consequence, any point a ∈ A will be mapped to the same class label, no matter which domain it is in. These two scenarios generally occur in low-dimensional datasets, such as scientific and healthcare applications. In very high-dimensional structured data such as images and text, there will
be some domain-specific dimensions which will have very different values in the source and the target domain (such as brightness in images in daytime vs. nighttime). This will result in no overlap between the supports of the source and target domain distributions, and thus the domain-specific dimensions will act as domain specific information. In this case using ϕ which is only a function of X is sufficient. However, even in this scenario, we need to find a principled way to constrain the encoder such that Z contains relevant label-specific information for the prediction in the target domain.
The observations from the perspective of the causal mechanism of generating X from Z and θX provide hints how to tackle the problem of UDA using invariant latent representations in a general way. First of all, we observe that θX needs to be an input to the encoder ϕ, in addition to X. Secondly, in this causal mechanism, we can safely assume that the influence of θX on the relationship between X and Z (and therefore the relationship between X and Y ) is minimal. This allows us to: (1) model latent variable θX as a function of the domain index and provide it is an input to ϕ(X, θX), instead of learning two separate encoders ϕS and ϕT ; (2) mimic the causal mechanism of generating X, given by X = ϕ̃(Z, θX), via a decoder ϕ̃ : Z ×Θ → X , in which we constrain the influence of θX to be minimal. The hope is then that such a decoder would act as a regularizer on the encoder ϕ, forcing it to preserve important semantic information when inferring Z. In this paper, we design a method based on this reasoning in order to address the problem of UDA in a principled manner.

heading: 3 Proposed Method: Domain-Specific Adversarial Training
text: Motivated by the above discussion, we aim to design a domain-adversarial network which can make use of the domain-specific parameters θX . Let θX ∈ {θSX , θTX}, which are its possible values in the source and target domains. The model for the proposed method DSAN (Domain-Specific Adversarial Network) is depicted in Figure 3.
We first consider the domain index of the data: j ∈ {S, T } (which indicates whether a data-point comes from the source or target domain), and parameterize the estimated latent domain-specific parameters θ̂(j)X as a function of the domain index of the i-the data-point ji (0 for source, 1 for target domain). We make use of θ̂(j)X and data encoder ϕ to obtain z (j) i = ϕ(x (j) i , θ̂ (j) X ), where x(j)i and z (j) i represent the the i-th input point and latent representation point respectively, from the j-th domain. We can then use the latent representation z(j)i to obtain a softmax probability for the predicted label label ỹ(j)i = h(z (j) i ), and this predictor can be trained in the source domain. To enforce the condition PS(Z) = P T (Z), we also have an adversarial predictor which predicts the domain index, given by j̃ = ha(z (j) i ). In order to constrain the influence of domain-specific changes θX , our model also needs to make use of (or mimic) the data-generating process depicted on Figure 1, in particular the process of generating X from Z and θX . This can be achieved via a decoder ϕ̃ which reconstructs the features X in both the source and the target domain, from Z and θX , given by x̃ (j) i = ϕ̃(z (j) i , θ̂ (j) X ). As described so far, our model consists of the following loss functions:
Lreconst. = 1
mS +mT mS+mT∑ i=1 ||xi − x̃||22, Lclassif. = − 1 mS C∑ c=1 mS∑ i=1 yiclog(ỹic),
Linv. = − mS+mT∑
i=1 {ji log j̃i + (1− ji) log(1− j̃i)}, Lcent. = − mT∑ i=1 h(zTi ) T log(h(zTi ))
Here, Lreconst., Lclassif., and Linv. are the reconstruction, classification and adversarial loss respectively. Lcent. is the conditional cross-entropy of the predictions in the target domain, and intuitively, it serves to enforce that the decision boundary does not cross data-dense regions [32]. The current loss functions do not involve a constraint on the encoding transformation ϕ which would prevent it from discarding valuable information about Y . We now describe how we can make use of the data-generating process and the assumption of minimal change across domains to enforce such a constraint.

heading: 3.1 Enforcing Minimal Change by Mutual Information Minimization
text: Before we delve into the details of how to enforce minimality of the influence of θX on the generating process of X from Y , we need to find a way to measure it. One way to formally do so is the joint
mutual information I(θX ; (X,Y )). Conveniently, this term is related to a distributional divergence measure that can be minimized, as stated by the following lemma:
Lemma 1: Let X,Y ∼ P (X,Y ; θX), and let θX be a discrete random variable with possible values θSX and θ T X in the source and target domains respectively, with a uniform prior P (θX = θ S X) = 0.5, P (θX = θ T X) = 0.5. Then, I(θX ; (X,Y )) = JSD(PX,Y ;θX=θSX ||PX,Y ;θX=θTX ), where JSD is the Jensen-Shannon Divergence, and PX,Y ;θX=θSX and PX,Y ;θX=θTX are the source and target domain joint distributions respectively.
Therefore, to measure the influence of θX on the generating process as mimicked by our decoder, we need to measure the JSD between the joint distributions as implied by the reconstructed data, given by JSD(PX,Y ;θX=θ̂SX ||PX,Y ;θX=θ̂TX ) (note that here we are using inferred domain-specific variables θ̂SX and θ̂ T X ). In theory, given enough data and expressiveness of the decoder, we can assume that we can learn to reconstruct the data perfectly. However, due to lack of labeled data in the target domain, we can only reconstruct the unlabeled features in the target domain, and have no access to PX,Y ;θX=θ̂TX .
To cope with this, we can use the inferred domain-specific variable θ̂TX to translate the source domain data to the target domain via the decoder: x̃Strans. = ϕ̃(z
S , θ̂TX). This translated data from the source to the target domain is labeled and can serve as an approximation of the joint distribution in the target domain, which we can use to constrain the JSD. A very intuitive way to minimize the JSD of the joint distributions of the source domain reconstructed and translated data, is to first define new random variables V := [X̃S , Y S ]T and Vtrans. := [X̃Strans., Y
S ]T as concatenations of the features and the label. Then, the JSD can be minimized by ensuring that the distributions of this joint feature-label vector is the same between the source reconstructed and translated data: P (V ) = P (Vtrans.). This can be achieved in several different ways, on of which is training an adversarial classifier on V and Vtrans.. However, since Y is usually a scalar variable with a specific semantic meaning and often has a simple parametric form, concatenating it to X would discard this prior knowledge, rendering this approach statistically inefficient.
Fortunately, the following theorem, adopted from [15], gives an upper bound of the JSD which is easy to minimize, and which makes use of the specific parametric form of Y in the case of classification:
Theorem 1 [15]: Let PY X|θX=θS and PY X|θX=θT denote the source and target domain distributions respectively. Let QhcY |X denote the conditional distribution of Y given X specified by the auxiliary classifier hc. We have:
JSD(PXY ;θX=θSX ||PXY ;θX=θTX ) ≤ 2c1 √ JSD(PX|θX=θSX ||PX|θX=θTX )
+ c2 √ KL(PY |X,θX=θSX ||Q hc Y |X) + c2 √ KL(PY |X,θX=θTX ||Q hc Y |X)
, where c1 and c2 are upper bounds of 12 ∫ |PY |X(y|x)|µ(x, y) and 12 ∫ |QX(x)|µ(x) respectively (µ is a σ-finite measure). Here, JSD(PX|θX=θ̂SX ||PX|θX=θ̂TX ) is equal to the JSD between the source and target domain marginal distributions of X and is fixed, we can ignore this term when we minimize the upper bound using our algorithm. Therefore, to minimize the two KL divergences on the right-hand side:
KL(PY |X,θ̂X=θSX ||Q hc Y |X) and KL(PY |X,θX=θ̂TX ||Q hc Y |X),
it suffices to train a joint auxiliary classifier hc on the reconstructed source domain data x̃S and the translated data x̃Ttrans., whose task is to learn to predict the source domain label. Therefore, we introduce a cross-entropy loss Ltrans.inv. for the auxiliary classifier, whose two terms can be used to minimize the two KL divergences respectively:
Ltransinv. = − C∑
j=1 mS∑ i=1 yij log(ỹ c ij)− C∑ j=1 mS∑ i=1 yij log(ỹ c ij,trans.), where ỹ c i = hc(x̃ S i ),
and ỹci,trans. = hc(x̃ S i,trans.) are the softmax predictions of the reconstruction and the translation of the source domain respectively, using predictor hc. Therefore, the total loss function can be formulated in the following manner:
Ltotal = λαLreconst. + λδLclassif. − λγLinv. + λτLtrans.inv. + λκLcent.,
and can be minimized by alternating optimization:
min ϕ,ϕ̃,h,hc
Ltotal
min ha
λγLinv..
This alternating minimization problem can be solved using Gradient-Reversal Layers (GRL), as shown in [13]. Linv. can also be expressed in terms of Maximum Mean Discrepancy (MMD) [16]. Our model is now equipped with a way to ensure that when reconstructing the data, θX cannot play a major role in the decoder ϕ̃. Since θX correspond to minimal changes of the conditional distribution P (X|Y ), this forces Z to retain Y -specific structure, and in turn implicitly regularizes the encoder ϕ, resulting in a conditionally invariant representation. In the following section, we will empirically demonstrate this property of the constraint.

heading: 4 Empirical Evaluation
text: In order to provide intuition about the proposed method’s advantages in the challenging scenarios described above, we first designed a simple but informative 2D simulated example for domain adaptation using mixtures of Gaussian distributions. This simulated dataset is presented on Figure 4(a), and is representative of the example discussed in Figure 2, since there is a significant region of overlap between the two domains, in which the points have opposite labels across domains (details about the dataset can be found in the supplementary materials). In this experiment, we compared our method DSAN, with the following baselines: (1) DSAN-unreg., which is our proposed method but without regularizing the transformation ϕ using the mutual information minimization in our loss; (2) DSN, the method proposed in [4], which makes use of domain-specific private encodings, but which are only used for reconstruction of the data, and not for inferring the shared invariant representation Z; (3) DANN, the classical domain-adversarial method proposed by [13]. We used a 2D hidden
representation Z (more details about implementation and tuning can be found in the supplement).
The results are presented in Figure 4b, demonstrating that our approach greatly outperforms the baselines. To visualize how our algorithm is able to solve this challenging example with much greater success, we also present the 2D invariant representations in the source and target domains on
Figure 6. At the top, we have the hidden representations Z of DSAN-unreg., and on the bottom the ones yielded by our method DSAN (run on a separate random initialization). On the left side we present Z in the source domain, labeled by the true values, in the middle we show this representation in the target domain labeled with the prediction of the algorithm, and on the right-hand side we present Z in the target domain labeled with the true labels. From this figure, one can appreciate that the unregularized version of our approach yields a representation Z which discards a lot of information about Y , as can be seen in parts 5b and 5c. On the other hand, parts 5e and 5f show that our algorithm with the mutual information minimization constraint can successfully preserve label-specific information in the invariant representation.

heading: 4.1 Real Data Experiments
text: To evaluate our method on real datasets, we consider three datasets and respective tasks from various domains of applications: cross-domain Wi-Fi localization, Amazon product reviews and image classification. For detailed descriptions of the experimental design, hyperparameter tuning and neural network architectures, as well as ablation studies, we refer the interested reader to the supplementary materials. We performed evaluation on the following datasets: Wi-Fi localization: introduced in [49], is a dataset in which wireless signal data was collected in a hallway area discretized as a grid. In each grid, data was recorded from 67 access points (dimensions). The task is to predict the location from the signals (which has been converted to a classification problem with 19 classes). There are three domains in this dataset (with 1140 points each), collected in three different time points. This is a low-dimensional dataset, in which we verified via manual inspection that both scenarios of support overlap described in Section 2 hold; Amazon Review: Amazon Review is another benchmark for multi-domain sentiment analysis. It contains positive and negative reviews of four kinds of products: Kitchen appliance (K), DVDs (D), Electronics (E), and Books (B). ImageCLEF: A standard UDA benchmark dataset for image classification, consisting of three domains: Caltech-256(C), ImageNet ILSVRC(I), and Pascal VOC2012(P), consisting of 12 classes. Baselines: We compare DSAN with classical approaches like TCA [28] and GFK [14], as well as with some deep transfer learning models. Three recently proposed methods, DSR [5], DIRT-T [32], MDD [51], BSP [7], MSTN [42] and RSDA [17] are included. For both the real-world and simulated experiments, we ran and tuned the baseline methods ourselves (all baselines for Wi-Fi and simulated, and MDD and DIRT-T for Amazon dataset).
The classification accuracies on the Amazon-Review Dataset for unsupervised domain adaptation are shown in Table 2, which shows that our algorithm model outperforms the baselines in the vast majority of source-target directions. We also perform a Wilcoxon signed-rank test on the experiment result of different random seed, and the p-value is 0.0216. The performance of our method on the Wi-Fi localization dataset is shown on Table 1. In this experiment, we sub-sampled 950 points in each domain to create 10 replicate experiments, and we present the accuracies and standard deviations across all replicates. From the results, one can appreciate that our method outperforms all baselines, for the majority of the pairs. We calculated a Wilcoxon p value p = 0.002 across all replicate experiments combined. The performance of our method on the ImageCLEF dataset is presented on Table 3, in which one can appreciate that we achieve state-of-the-art performance on most transfer directions. For this experiment, certain baselines such as MDD and RSDA use a pre-trained ResNet neural network architecture as the transformation function ϕ. For the proposed method, we instead used features obtained from a pre-trained ResNet architecture, and we incorporated them into the autoencoder neural network described in the previous section (more details on the architecture and hyperparameters can be found in the supplementary information). While we currently obtain state-of-the-art performance on this dataset, we believe that adapting our method to make use of ResNet fine-tuning will further increase the performance.

heading: 5 Conclusion
text: In this paper, we have demonstrated that the existing paradigm of representation learning for domain adaptation using a fixed encoder ϕ of the data, can be sub-optimal when the same points can
have different labels across domains. We have introduced a method which takes domain-specific information in order to provide the data encoder with the necessary flexibility and appropriate constraints, in order to retain valuable information about Y in the latent representation Z. We note that in many real-world applications with high-dimensional datasets, the supports of the source and target domains may not overlap. In this case, domain-specific information is contained in the features X and can be automatically used by the encoder ϕ(X). However, ϕ(X) would then have the same problem with excessive flexibility and can still suffer from finding trivial representations of Z that discard information about Y . Therefore, our principle for constraining domain-specific encoders would still be beneficial in this scenario. For future work, a promising direction is combining this framework motivated by the data-generating process with more powerful encoders, such as the spherical neural network proposed by [17].

heading: 6 Acknowledgements
text: We are very grateful to the anonymous reviewers for their help in improving the paper. This work was supported in part by the National Institutes of Health (NIH) under Contract R01HL159805, by the NSF-Convergence Accelerator Track-D award #2134901, by the United States Air Force under Contract No. FA8650-17-C7715, and by a grant from Apple Inc. GM was supported by Australian Research Council Project DE210101624. Ruichu and Zijian were supported in part by National Science Fund for Excellent Young Scholars (62122022), Natural Science Foundation of China (61876043, 61976052).

Review summaries:
This paper focuses on making use of latent invariant representations for unsupervised domain adaptation. The goal is to address two State-Of-The-Art existing methods issues. The first is there is no principled way to ensure that marginal invariance (P^S(Z) = P^T (Z)) preserves conditional invariance (P^S (Z|Y) = P^T (Z|Y)). The second is there are cases (especially in lower-dimensional datasets) where the encoding function Phi needs domain-specific information, in addition to the features X.  This paper introduces an invariant representation learning method that uses the data-generating process to justify and efficiently use domain-specific information in the encoder Phi and constraint it to ensure that Z contains relevant label-specific information. Results on three real datasets and one synthetic dataset show the proposed Domain-Specific Adversarial Network (DSAN) model’s performances (accuracy and Wilcoxon p-value) compared with existing methods and an unregularized DSAN (DSAN-U) version. 

The paper proposes a unsupervised domain adaptation method through the lens of learning invariant representations. The paper validates theoretically shows some limitations of current methods when learning from distinct source and target distributions. The paper validates experimentally on a large scale image dataset and 2 other datasets.

This paper analyzes the mechanism of domain adaptation and proposes a new strategy to reduce the gap between the two domains. The valuable information to achieve accurate classification can be well preserved. Extensive experiments are conducted on multiple dataset, such as Wi-Fi localization and Amazon Review.

**Summary and contributions:** The paper considers representation learning for unsupervised domain adaptation (UDA). They show that, in some cases, it is necessary to learn encoders that vary across domains, something which is rarely done in practice. They thus propose a new method for incorporating domain-dependent features in the encoding process.  The end model/objective contains many different components: it is essentially a DANN with a decoder that aims to reconstruct the inputs by using domain-dependent and independent features. Furthermore, the decoder is forced to rely as little as possible on the domain-dependent features.