Domain Adaptation with Invariant Representation Learning: What Transformations to Learn?

1 Introduction . Unsupervised domain adaptation ( UDA ) is a common setting for supervised learning , in which the labeled training and unlabeled test data come from different distributions . More formally , given features X ∈ Rd and labels Y ∈ R , we observe labeled source and unlabeled target domain instances , represented by ( xS , yS ) = ( xSk , y S ) mSk=1 and x T k = ( x T ) mTk=1 respectively , where P S ( X , Y ) ̸= P T ( X , Y ) , and mS and mT are the number of observations in the source and target domains , respectively . The main challenge of domain adaptation is to use the given source domain observations for learning a predictor that will perform well in the target domain . To do so , the procedure needs to make use of some similarities between the two domains . One way to formalize this is by making assumptions about how the joint distribution joint P ( X , Y ) changes across domains . For example , in the well-studied setting of covariate shift [ 31 , 44 , 20 , 35 , 2 , 8 ] , the marginal distribution P ( X ) changes while the conditional distribution P ( Y |X ) ( i.e . the optimal predictor ) is shared across domains . However , in many real-world applications , P ( Y |X ) can also change , and this requires making use of further assumptions . One such assumption is that the factorization P ( X , Y ) = P ( Y ) P ( X|Y ) allows for addressing the changes in P ( Y ) and P ( X|Y ) independently in a situation where their ∗These authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . respective changes are simple and easier to capture [ 48 , 30 ] . In this particular setting , the problem is generally broken down into : ( 1 ) Target shift : P ( Y ) changes across domains while P ( X|Y ) stay the same [ 34 , 21 , 29 , 48 ] ( 2 ) Conditional shift : P ( X|Y ) changes across domains but P ( Y ) stays the same [ 48 , 26 , 24 ] . ( 3 ) Conditional-target shift : Both P ( X|Y ) and P ( Y ) change independently across domains - the most general setting under this generating process assumption [ 48 , 6 ] . Then , assumptions about the changes of the factors of the joint distrubution can be made so that the problem is solvable , such as location-scale transformation [ 48 , 26 ] , or that the changing parameters lie on a low-dimensional manifold [ 33 ] , and algorithms can be designed to enforce these constraints and make use of them for prediction in the target domain . Another fruitful view of the problem is through the lens of representation learning , due to wide-spread applicability of neural architectures for many real-world problems . In particular , state of the art deep learning techniques harness the high representational capacity of neural networks to transform the input data into a latent feature representation which is predictive of the target variable Y in the source domain , and has the same distribution across domains . Formally , this means learning a function ( encoder ) ϕ : X → Z from the input space to a latent space Z , such that PS ( Z ) = PT ( Z ) . At the same time , a function h : Z → Y can be learnt to minimize the risk in the labeled source domain [ 1 ] . The hope is then , that the overall function g : = h ◦ ϕ will have low prediction risk in the target domain . However , the above-described theoretical and methodological framework does not guarantee that the learnt representation Z will have any relevant information for predicting Y in the target domain . Namely , one can easily have a situation in which the learnt representation Z is marginally invariant ( PS ( Z ) = PT ( Z ) ) , but not conditionally invariant ( PS ( Z|Y ) ̸= PT ( Z|Y ) ) , as discussed in [ 53 ] . This means that the learnt function g : = h ◦ ϕ can have very good prediction performance in the source domain , but generalize very poorly to the target domain . In many methods , the same encoding function ϕ ( X ) across domains is used to learn invariant latent representations . This enjoys computational benefits and makes the learning procedure relatively simple , and the vast majority of approaches ( [ 12 , 22 , 25 , 23 , 17 ] among many ) employ this technique . However , in certain situations , the same encoding function across domains can not learn a marginally invariant representation that is optimal for classification in the target domain . There are certain studies ( [ 37 , 38 , 4 ] ) which implement domain-specific encoders ϕS and ϕT . Unfortunately , such methods suffer from the following drawbacks : ( 1 ) the exact motivation behind having separate encoders for each domain is not clear ; ( 2 ) including two separate encoders may be inefficient because it greatly increases the number of parameters that we need to learn ; ( 3 ) in the field of UDA based on invariant representation learning , there is still no principled way to guarantee that the learned marginally invariant representation Z has sufficient structural ( semantic ) information , and has the potential to be conditionally invariant . In this paper we assume the setting of conditional shift , and we make use of the data-generating process to : ( i ) justify the use of two separate encoding functions in order to infer the latent representation , ( ii ) implement the two encoding functions more efficiently , and ( iii ) constrain the latent representation Z to have meaningful structure which is useful for prediction in the target domain . In Section 2 , we first motivate the use of two separate encoders to infer Z , via rigorous treatment and an illustrative example . Subsequently , in Chapter 3 , we introduce an efficient way to implement two separate functions for inferring Z . In Chapter 3 we shall also introduce a principled way to ensure that the latent representation Z contains useful information for prediction , and finally , in Chapter 4 we provide empirical evaluation . 1.1 Related Work . As previously mentioned , there is a vast body of work exploring approaches to make use of latent invariant representations for the purposes of unsupervised domain adaptation , using both deep learning and more traditional techniques . Namely , invariant representations were considered as a linear projection in [ 26 ] , where theoretical guarantees were established regarding the conditional invariance of the latent representation and the identifiability of the joint distribution in the target domain . Furthermore , this framework was generalized to nonlinear maps to invariant latent representations , parameterized by neural network termed DANN [ 12 ] . In this framework , the constraint that PS ( Z ) = PT ( Z ) can be enforced in various ways , including adversarial training , Maximum Mean Discrepancy [ 22 ] , Wassertstein Distance [ 9 ] and Margin Disparity Discrepancy [ 51 ] . However , none of these studies have considered using two separate encoders for the two domains . Subsequently , this reasoning has been considered in combination with image translation ( CYCADA ) [ 19 ] . Another assumption frequently made in recent studies is that the input features X have a clustering structure in which each cluster has a different label Y . Therefore , further directions were pursued by incorporating additional information contained in the unlabeled target domain data , such as pseudo-labels provided by the classifier which is initially trained on the labeled source domain data [ 23 , 41 ] ( termed CDAN and SLPP , respectively ) . In addition , one can also make use of this assumption to enforce that the prediction function g does not pass through high-density regions in input space , as performed by the DIRT-T algorithm [ 32 ] . The method by [ 37 ] makes use of two separate transformation functions , but does not regularize their output to preserve semantic information in any way . In addition to these , approaches , the DSN algorithm [ 4 ] takes into models the changing parameters of P ( X , Y ) in the latent representation and enforces that they are different from the invariant representation Z . The only method that we are aware of which takes domain-specific information into account is the study by [ 38 ] , in which domain information is given and continuous , and therefore contains a lot more information than the discrete domain index that is typically considered . In [ 17 ] , a spherical architecture is used for ϕ , however there is only one encoder for both domains . The study by [ 27 ] performs bridging between the source and the target domains using data-augmentation , but this is a technique specific to image data . In addition to domain adaptation methodologies in the setting where the is only one labeled source domain , there have been several efforts to tackle the problem of multiple-source unsupervised domain adaptation . Namely , the method by [ 3 ] tackles the problem by using modified kernel SVMs to operate on both input features X and marginal distributions PX . Furthermore , the method by [ 46 ] assumes that the target domain is a linear mixture of the source domains , and infers the linear coefficients and makes use of them for prediction in the target domain . Furthermore , in [ 47 ] , data-generating process information is discovered from the multiple-domain data , and made use for recovering the changing parameters in the target domain in order to construct a predictor for it . While the above-mentioned methods use various approaches to improve the quality of the learnt invariant latent representation and show competitive performance , there is no guarantee that these approaches are general enough to encompass all scenarios . Namely , the challenge that existing methods face is two-fold : ( 1 ) there is no principled way to ensure that marginal invariance ( PS ( Z ) = PT ( Z ) ) preserves conditional invariance ( PS ( Z|Y ) = PT ( Z|Y ) ) , ( 2 ) there are cases ( especially in lower-dimensional datasets ) where the encoding function ϕ needs domain-specific information , in addition to the features X . The aim of this paper is to address these two issues . In this paper , we introduce the first invariant representation learning method that makes use of the data-generating process to justify and efficiently use domain-specific information in the encoder ϕ , and to properly constrain it to ensure that Z contains relevant label-specific information . 2 What Does it Take to Learn a Useful Invariant Representation ? . To gain insights into what information is necessary for learning an intermediate representation Z via ϕ , let us consider the data-generating process depicted on Figure 1 . In this graphical model , the label Y is generated first from its prior distribution P ( Y ) . Then , the invariant representation Z is generated from Y via the generating mechanism P ( Z|Y ) and X is subsequently generated from P ( X|Z ; θX ) , where θX are the changing parameters of P ( X|Y ) across domains . We consider θX as a parameter vector with a constant value for each point within a domain , and we generally assume that θX contains little information , i.e . the change across domains in P ( X|Y ) is minimal . This generating process , in part or in whole , can be exploited in many classification applications . For example , in object classification in images , Y is the label ( the object the picture was taken of ) , X is the observed pixels , and Z can correspond to latent features with semantic relevance , such as some function of the edges and contours on the image . In this case , θX can correspond to different resolution , illumination conditions , or other environment-specific changes that are not relevant for predicting the class Y . In this graphical representation , we see that generally speaking , Z is conditionally dependent from θX given X , although they may be marginally independent . This implies that in order to recover Z , given that X is considered , the information of θX should also be considered in the transformation , because it is in the Markov Blanket of Z ( the Markov Blanket of a variable of interest consists of all other variables that are either its parents , its children , or the parents of its children in the probabilistic graphical model . The Markov Blanket represents all of the information needed to predict the variable of interest ) . To gain intuition why considering the Markov Blanket is important , consider the following scenario , depicted in Figure 2a . In this setting , we are given the source domain with Uniform distribution , and the target domain with Gaussian distribution . Here , we have a region A ⊂ X ( highlighted in red ) , in which both of the domains ’ distributions have support , so there is support overlap . However , each domain has a different density in this region . The following proposition claims that in this situation , we can not infer an invariant representation Z from X with a single encoder ϕ : Proposition 1 : Let A ⊂ X be a region in input space such that PS ( X ∈ A ) > 0 and P T ( X ∈ A ) > 0 , and PS ( X ∈ A ) ̸= P T ( X ∈ A ) . Furthermore , let ϕ : X → Z be an encoder s.t . A = { a : ϕ ( a ) ∈ B } for some B ⊂ Z . Then , there is no function ϕ s.t . PS ( ϕ ( X ) ∈ B ) = P T ( ϕ ( X ) ∈ B ) . All omitted proofs can be found in the supplementary materials . This proposition implies that if the supports of the marginal distributions of the source and target domains overlap , and if the distributions are different in the specific region of the support overlap ( set A ) , then we can not use a fixed transformation to transform the data in that region to an invariant representation in Z space . There is an additional case in which a single encoder ϕ is not sufficient to learn an optimal latent representation Z . The scenario is illustrated on Figure 2b . In this case , even though both domains have the same distribution P ( X ) in the region of support overlap between the source and the target domain ( labeled in red ) , the value of Y is opposite between the two domains . The following proposition demonstrates that in this setting , a single encoding function ϕ is inherently sub-optimal : Proposition 2 : Let the true labeling functions in the source and target domain be fS , fT : X → Y , respectively . Let A ⊂ X be a region s.t . fS ( a ) ̸= fT ( a ) , ∀a ∈ A . Let g : X → Y be a composition of a representation learner ϕ : X → Z and a classifier h : Z → Y . If ϕ is the same function across domains , then for a 0-1 loss , the risk over the region A : ϵA ( g ) = ϵAS ( g ) + ϵ A T ( g ) ≥ 1 , where ϵAS ( g ) and ϵAT ( g ) are the source and target domain risks respectively . In this scenario , no matter what criterion one uses to find the representation for downstream classification using h , as long as the two domains use the same transformation ϕ for the representation , the points in A can not be correctly classified in the target domain , if they are correctly classfied in the source domain . This is because the final classifer of the data , g = h ◦ ϕ , is fixed across domains . As a consequence , any point a ∈ A will be mapped to the same class label , no matter which domain it is in . These two scenarios generally occur in low-dimensional datasets , such as scientific and healthcare applications . In very high-dimensional structured data such as images and text , there will be some domain-specific dimensions which will have very different values in the source and the target domain ( such as brightness in images in daytime vs. nighttime ) . This will result in no overlap between the supports of the source and target domain distributions , and thus the domain-specific dimensions will act as domain specific information . In this case using ϕ which is only a function of X is sufficient . However , even in this scenario , we need to find a principled way to constrain the encoder such that Z contains relevant label-specific information for the prediction in the target domain . The observations from the perspective of the causal mechanism of generating X from Z and θX provide hints how to tackle the problem of UDA using invariant latent representations in a general way . First of all , we observe that θX needs to be an input to the encoder ϕ , in addition to X. Secondly , in this causal mechanism , we can safely assume that the influence of θX on the relationship between X and Z ( and therefore the relationship between X and Y ) is minimal . This allows us to : ( 1 ) model latent variable θX as a function of the domain index and provide it is an input to ϕ ( X , θX ) , instead of learning two separate encoders ϕS and ϕT ; ( 2 ) mimic the causal mechanism of generating X , given by X = ϕ̃ ( Z , θX ) , via a decoder ϕ̃ : Z ×Θ → X , in which we constrain the influence of θX to be minimal . The hope is then that such a decoder would act as a regularizer on the encoder ϕ , forcing it to preserve important semantic information when inferring Z . In this paper , we design a method based on this reasoning in order to address the problem of UDA in a principled manner .

Summary: **Summary and contributions:** The paper considers representation learning for unsupervised domain adaptation (UDA). They show that, in some cases, it is necessary to learn encoders that vary across domains, something which is rarely done in practice. They thus propose a new method for incorporating domain-dependent features in the encoding process.  The end model/objective contains many different components: it is essentially a DANN with a decoder that aims to reconstruct the inputs by using domain-dependent and independent features. Furthermore, the decoder is forced to rely as little as possible on the domain-dependent features.