Domain Adaptation with Invariant Representation Learning: What Transformations to Learn?

1 Introduction . Unsupervised domain adaptation ( UDA ) is a common setting for supervised learning , in which the labeled training and unlabeled test data come from different distributions . More formally , given features X ∈ Rd and labels Y ∈ R , we observe labeled source and unlabeled target domain instances , represented by ( xS , yS ) = ( xSk , y S ) mSk=1 and x T k = ( x T ) mTk=1 respectively , where P S ( X , Y ) ̸= P T ( X , Y ) , and mS and mT are the number of observations in the source and target domains , respectively . The main challenge of domain adaptation is to use the given source domain observations for learning a predictor that will perform well in the target domain . To do so , the procedure needs to make use of some similarities between the two domains . One way to formalize this is by making assumptions about how the joint distribution joint P ( X , Y ) changes across domains . For example , in the well-studied setting of covariate shift [ 31 , 44 , 20 , 35 , 2 , 8 ] , the marginal distribution P ( X ) changes while the conditional distribution P ( Y |X ) ( i.e . the optimal predictor ) is shared across domains . However , in many real-world applications , P ( Y |X ) can also change , and this requires making use of further assumptions . One such assumption is that the factorization P ( X , Y ) = P ( Y ) P ( X|Y ) allows for addressing the changes in P ( Y ) and P ( X|Y ) independently in a situation where their ∗These authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . respective changes are simple and easier to capture [ 48 , 30 ] . In this particular setting , the problem is generally broken down into : ( 1 ) Target shift : P ( Y ) changes across domains while P ( X|Y ) stay the same [ 34 , 21 , 29 , 48 ] ( 2 ) Conditional shift : P ( X|Y ) changes across domains but P ( Y ) stays the same [ 48 , 26 , 24 ] . ( 3 ) Conditional-target shift : Both P ( X|Y ) and P ( Y ) change independently across domains - the most general setting under this generating process assumption [ 48 , 6 ] . Then , assumptions about the changes of the factors of the joint distrubution can be made so that the problem is solvable , such as location-scale transformation [ 48 , 26 ] , or that the changing parameters lie on a low-dimensional manifold [ 33 ] , and algorithms can be designed to enforce these constraints and make use of them for prediction in the target domain . Another fruitful view of the problem is through the lens of representation learning , due to wide-spread applicability of neural architectures for many real-world problems . In particular , state of the art deep learning techniques harness the high representational capacity of neural networks to transform the input data into a latent feature representation which is predictive of the target variable Y in the source domain , and has the same distribution across domains . Formally , this means learning a function ( encoder ) ϕ : X → Z from the input space to a latent space Z , such that PS ( Z ) = PT ( Z ) . At the same time , a function h : Z → Y can be learnt to minimize the risk in the labeled source domain [ 1 ] . The hope is then , that the overall function g : = h ◦ ϕ will have low prediction risk in the target domain . However , the above-described theoretical and methodological framework does not guarantee that the learnt representation Z will have any relevant information for predicting Y in the target domain . Namely , one can easily have a situation in which the learnt representation Z is marginally invariant ( PS ( Z ) = PT ( Z ) ) , but not conditionally invariant ( PS ( Z|Y ) ̸= PT ( Z|Y ) ) , as discussed in [ 53 ] . This means that the learnt function g : = h ◦ ϕ can have very good prediction performance in the source domain , but generalize very poorly to the target domain . In many methods , the same encoding function ϕ ( X ) across domains is used to learn invariant latent representations . This enjoys computational benefits and makes the learning procedure relatively simple , and the vast majority of approaches ( [ 12 , 22 , 25 , 23 , 17 ] among many ) employ this technique . However , in certain situations , the same encoding function across domains can not learn a marginally invariant representation that is optimal for classification in the target domain . There are certain studies ( [ 37 , 38 , 4 ] ) which implement domain-specific encoders ϕS and ϕT . Unfortunately , such methods suffer from the following drawbacks : ( 1 ) the exact motivation behind having separate encoders for each domain is not clear ; ( 2 ) including two separate encoders may be inefficient because it greatly increases the number of parameters that we need to learn ; ( 3 ) in the field of UDA based on invariant representation learning , there is still no principled way to guarantee that the learned marginally invariant representation Z has sufficient structural ( semantic ) information , and has the potential to be conditionally invariant . In this paper we assume the setting of conditional shift , and we make use of the data-generating process to : ( i ) justify the use of two separate encoding functions in order to infer the latent representation , ( ii ) implement the two encoding functions more efficiently , and ( iii ) constrain the latent representation Z to have meaningful structure which is useful for prediction in the target domain . In Section 2 , we first motivate the use of two separate encoders to infer Z , via rigorous treatment and an illustrative example . Subsequently , in Chapter 3 , we introduce an efficient way to implement two separate functions for inferring Z . In Chapter 3 we shall also introduce a principled way to ensure that the latent representation Z contains useful information for prediction , and finally , in Chapter 4 we provide empirical evaluation . 1.1 Related Work . As previously mentioned , there is a vast body of work exploring approaches to make use of latent invariant representations for the purposes of unsupervised domain adaptation , using both deep learning and more traditional techniques . Namely , invariant representations were considered as a linear projection in [ 26 ] , where theoretical guarantees were established regarding the conditional invariance of the latent representation and the identifiability of the joint distribution in the target domain . Furthermore , this framework was generalized to nonlinear maps to invariant latent representations , parameterized by neural network termed DANN [ 12 ] . In this framework , the constraint that PS ( Z ) = PT ( Z ) can be enforced in various ways , including adversarial training , Maximum Mean Discrepancy [ 22 ] , Wassertstein Distance [ 9 ] and Margin Disparity Discrepancy [ 51 ] . However , none of these studies have considered using two separate encoders for the two domains . Subsequently , this reasoning has been considered in combination with image translation ( CYCADA ) [ 19 ] . Another assumption frequently made in recent studies is that the input features X have a clustering structure in which each cluster has a different label Y . Therefore , further directions were pursued by incorporating additional information contained in the unlabeled target domain data , such as pseudo-labels provided by the classifier which is initially trained on the labeled source domain data [ 23 , 41 ] ( termed CDAN and SLPP , respectively ) . In addition , one can also make use of this assumption to enforce that the prediction function g does not pass through high-density regions in input space , as performed by the DIRT-T algorithm [ 32 ] . The method by [ 37 ] makes use of two separate transformation functions , but does not regularize their output to preserve semantic information in any way . In addition to these , approaches , the DSN algorithm [ 4 ] takes into models the changing parameters of P ( X , Y ) in the latent representation and enforces that they are different from the invariant representation Z . The only method that we are aware of which takes domain-specific information into account is the study by [ 38 ] , in which domain information is given and continuous , and therefore contains a lot more information than the discrete domain index that is typically considered . In [ 17 ] , a spherical architecture is used for ϕ , however there is only one encoder for both domains . The study by [ 27 ] performs bridging between the source and the target domains using data-augmentation , but this is a technique specific to image data . In addition to domain adaptation methodologies in the setting where the is only one labeled source domain , there have been several efforts to tackle the problem of multiple-source unsupervised domain adaptation . Namely , the method by [ 3 ] tackles the problem by using modified kernel SVMs to operate on both input features X and marginal distributions PX . Furthermore , the method by [ 46 ] assumes that the target domain is a linear mixture of the source domains , and infers the linear coefficients and makes use of them for prediction in the target domain . Furthermore , in [ 47 ] , data-generating process information is discovered from the multiple-domain data , and made use for recovering the changing parameters in the target domain in order to construct a predictor for it . While the above-mentioned methods use various approaches to improve the quality of the learnt invariant latent representation and show competitive performance , there is no guarantee that these approaches are general enough to encompass all scenarios . Namely , the challenge that existing methods face is two-fold : ( 1 ) there is no principled way to ensure that marginal invariance ( PS ( Z ) = PT ( Z ) ) preserves conditional invariance ( PS ( Z|Y ) = PT ( Z|Y ) ) , ( 2 ) there are cases ( especially in lower-dimensional datasets ) where the encoding function ϕ needs domain-specific information , in addition to the features X . The aim of this paper is to address these two issues . In this paper , we introduce the first invariant representation learning method that makes use of the data-generating process to justify and efficiently use domain-specific information in the encoder ϕ , and to properly constrain it to ensure that Z contains relevant label-specific information . 2 What Does it Take to Learn a Useful Invariant Representation ? . To gain insights into what information is necessary for learning an intermediate representation Z via ϕ , let us consider the data-generating process depicted on Figure 1 . In this graphical model , the label Y is generated first from its prior distribution P ( Y ) . Then , the invariant representation Z is generated from Y via the generating mechanism P ( Z|Y ) and X is subsequently generated from P ( X|Z ; θX ) , where θX are the changing parameters of P ( X|Y ) across domains . We consider θX as a parameter vector with a constant value for each point within a domain , and we generally assume that θX contains little information , i.e . the change across domains in P ( X|Y ) is minimal . This generating process , in part or in whole , can be exploited in many classification applications . For example , in object classification in images , Y is the label ( the object the picture was taken of ) , X is the observed pixels , and Z can correspond to latent features with semantic relevance , such as some function of the edges and contours on the image . In this case , θX can correspond to different resolution , illumination conditions , or other environment-specific changes that are not relevant for predicting the class Y . In this graphical representation , we see that generally speaking , Z is conditionally dependent from θX given X , although they may be marginally independent . This implies that in order to recover Z , given that X is considered , the information of θX should also be considered in the transformation , because it is in the Markov Blanket of Z ( the Markov Blanket of a variable of interest consists of all other variables that are either its parents , its children , or the parents of its children in the probabilistic graphical model . The Markov Blanket represents all of the information needed to predict the variable of interest ) . To gain intuition why considering the Markov Blanket is important , consider the following scenario , depicted in Figure 2a . In this setting , we are given the source domain with Uniform distribution , and the target domain with Gaussian distribution . Here , we have a region A ⊂ X ( highlighted in red ) , in which both of the domains ’ distributions have support , so there is support overlap . However , each domain has a different density in this region . The following proposition claims that in this situation , we can not infer an invariant representation Z from X with a single encoder ϕ : Proposition 1 : Let A ⊂ X be a region in input space such that PS ( X ∈ A ) > 0 and P T ( X ∈ A ) > 0 , and PS ( X ∈ A ) ̸= P T ( X ∈ A ) . Furthermore , let ϕ : X → Z be an encoder s.t . A = { a : ϕ ( a ) ∈ B } for some B ⊂ Z . Then , there is no function ϕ s.t . PS ( ϕ ( X ) ∈ B ) = P T ( ϕ ( X ) ∈ B ) . All omitted proofs can be found in the supplementary materials . This proposition implies that if the supports of the marginal distributions of the source and target domains overlap , and if the distributions are different in the specific region of the support overlap ( set A ) , then we can not use a fixed transformation to transform the data in that region to an invariant representation in Z space . There is an additional case in which a single encoder ϕ is not sufficient to learn an optimal latent representation Z . The scenario is illustrated on Figure 2b . In this case , even though both domains have the same distribution P ( X ) in the region of support overlap between the source and the target domain ( labeled in red ) , the value of Y is opposite between the two domains . The following proposition demonstrates that in this setting , a single encoding function ϕ is inherently sub-optimal : Proposition 2 : Let the true labeling functions in the source and target domain be fS , fT : X → Y , respectively . Let A ⊂ X be a region s.t . fS ( a ) ̸= fT ( a ) , ∀a ∈ A . Let g : X → Y be a composition of a representation learner ϕ : X → Z and a classifier h : Z → Y . If ϕ is the same function across domains , then for a 0-1 loss , the risk over the region A : ϵA ( g ) = ϵAS ( g ) + ϵ A T ( g ) ≥ 1 , where ϵAS ( g ) and ϵAT ( g ) are the source and target domain risks respectively . In this scenario , no matter what criterion one uses to find the representation for downstream classification using h , as long as the two domains use the same transformation ϕ for the representation , the points in A can not be correctly classified in the target domain , if they are correctly classfied in the source domain . This is because the final classifer of the data , g = h ◦ ϕ , is fixed across domains . As a consequence , any point a ∈ A will be mapped to the same class label , no matter which domain it is in . These two scenarios generally occur in low-dimensional datasets , such as scientific and healthcare applications . In very high-dimensional structured data such as images and text , there will be some domain-specific dimensions which will have very different values in the source and the target domain ( such as brightness in images in daytime vs. nighttime ) . This will result in no overlap between the supports of the source and target domain distributions , and thus the domain-specific dimensions will act as domain specific information . In this case using ϕ which is only a function of X is sufficient . However , even in this scenario , we need to find a principled way to constrain the encoder such that Z contains relevant label-specific information for the prediction in the target domain . The observations from the perspective of the causal mechanism of generating X from Z and θX provide hints how to tackle the problem of UDA using invariant latent representations in a general way . First of all , we observe that θX needs to be an input to the encoder ϕ , in addition to X. Secondly , in this causal mechanism , we can safely assume that the influence of θX on the relationship between X and Z ( and therefore the relationship between X and Y ) is minimal . This allows us to : ( 1 ) model latent variable θX as a function of the domain index and provide it is an input to ϕ ( X , θX ) , instead of learning two separate encoders ϕS and ϕT ; ( 2 ) mimic the causal mechanism of generating X , given by X = ϕ̃ ( Z , θX ) , via a decoder ϕ̃ : Z ×Θ → X , in which we constrain the influence of θX to be minimal . The hope is then that such a decoder would act as a regularizer on the encoder ϕ , forcing it to preserve important semantic information when inferring Z . In this paper , we design a method based on this reasoning in order to address the problem of UDA in a principled manner .

Summary: This paper focuses on making use of latent invariant representations for unsupervised domain adaptation. The goal is to address two State-Of-The-Art existing methods issues. The first is there is no principled way to ensure that marginal invariance (P^S(Z) = P^T (Z)) preserves conditional invariance (P^S (Z|Y) = P^T (Z|Y)). The second is there are cases (especially in lower-dimensional datasets) where the encoding function Phi needs domain-specific information, in addition to the features X.  This paper introduces an invariant representation learning method that uses the data-generating process to justify and efficiently use domain-specific information in the encoder Phi and constraint it to ensure that Z contains relevant label-specific information. Results on three real datasets and one synthetic dataset show the proposed Domain-Specific Adversarial Network (DSAN) model’s performances (accuracy and Wilcoxon p-value) compared with existing methods and an unregularized DSAN (DSAN-U) version.