EMTL: A Generative Domain Adaptation Approach

We propose an unsupervised domain adaptation approach based on generative models . We show that when the source probability density function can be learned , one-step Expectation–Maximization iteration plus an additional marginal density function constraint will produce a proper mediator probability density function to bridge the gap between the source and target domains . The breakthrough is based on modern generative models ( autoregressive mixture density nets ) that are competitive to discriminative models on moderate-dimensional classification problems . By decoupling the source density estimation from the adaption steps , we can design a domain adaptation approach where the source data is locked away after being processed only once , opening the door to transfer when data security or privacy concerns impede the use of traditional domain adaptation . We demonstrate that our approach can achieve state-of-the-art performance on synthetic and real data sets , without accessing the source data at the adaptation phase . 1 INTRODUCTION . In the classical supervised learning paradigm , we assume that the training and test data come from the same distribution . In practice , this assumption often does not hold . When the pipeline includes massive data labeling , models are routinely retrained after each data collecion campaign . However , data labeling costs often make retraining impractical . Without labeled data , it is still possible to train the model by using a training set which is relevant but not identically distributed to the test set . Due to the distribution shift between the training and test sets , the performance usually can not be guaranteed . Domain adaptation ( DA ) is a machine learning subdomain that aims at learning a model from biased training data . It explores the relationship between source ( labeled training data ) and target ( test data ) domains to find the mapping function and fix the bias , so that the model learned on the source data can be applied in target domain . Usually some target data is needed during the training phase to calibrate the model . In unsupervised domain adaptation ( UDA ) only unlabeled target data is needed during training phase . UDA is an appealing learning paradigm since obtaining unlabeled data is usually easy in a lot of applications . UDA allows the model to be deployed in various target domains with different shifts using a single labeled source data set . Due to these appealing operational features , UDA has became a prominent research field with various approaches . Kouw & Loog ( 2019 ) and Zhuang et al . ( 2020 ) surveyed the latest progress on UDA and found that most of the approaches are based on discriminative models , either by reweighting the source instances to approximate the target distribution or learning a feature mapping function to reduce the statistical distance between the source and target domains . After calibrating , a discriminative model is trained on the adjusted source data and used in target domain . In this workflow , the adaptation algorithm usually have to access the source and target data simultaneously . However , accessing the source data during the adaptation phase is not possible when the source data is sensitive ( for example because of security or privacy issues ) . In particular , in our application workflow an industrial company is selling devices to various service companies which can not share their customer data with each other . The industrial company may contract with one of the service companies to access their data during an R & D phase , but this data will not be available when the industrial company sells the device ( and the predictive model ) to other service companies . In this paper we propose EMTL , a generative UDA algorithm for binary classification that does not have to access the source data during the adaptation phase . We use density estimation to estimate the joint source probability function ps ( x , y ) and the marginal target probability function pt ( x ) and use them for domain adaption . To solve the data security issue , EMTL decouples source density estimation from the adaptation steps . In this way , after the source preprocessing we can put away or delete the source data . Our approach is motivated by the theory on domain adaptation ( Ben-David et al. , 2010 ) which claims that the error of a hypothesis h on the target domain can be bounded by three items : the error on the source domain , the distance between source and target distributions , and the expected difference in labeling functions . This theorem motivated us to define a mediator density function pm ( x , y ) i ) whose conditional probability y|x is equal to the conditional probability of the source and ii ) whose marginal density on x is equal to the marginal density of the target . We can then construct a Bayes optimal classifier on the target domain under the assumption of covariate shift ( the distribution y|x is the same in the source and target domains ) . Our approach became practical with the recent advances in ( autoregressive ) neural density estimation ( Uria et al. , 2013 ) . We learn pm ( x , y ) from ps ( x , y ) and pt ( x ) to bridge the gap between the source and target domains . We regard the label on the target data as a latent variable and show that if ps ( x |y = i ) be learned perfectly for i ∈ { 0 , 1 } , then a one-step Expectation–Maximization ( and this is why our algorithm named EMTL ) iteration will produce a density function pm ( x , y ) with the following properties on the target data : i ) minimizing the Kullback–Leibler divergence between pm ( yi|xi ) and ps ( yi|xi ) ; ii ) maximizing the log-likelihood ∑ log pm ( xi ) . Then , by adding an additional marginal constraint on pm ( xi ) to make it close to pt ( xi ) on the target data explicitly , we obtain the final objective function for EMTL . Although this analysis assumes a simple covariate shift , we will experimentally show that EMTL can go beyond this assumption and work well in other distribution shifts . We conduct experiments on synthetic and real data to demonstrate the effectiveness of EMTL . First , we construct a simple two-dimensional data set to visualize the performance of EMTL . Second , we use UCI benchmark data sets and the Amazon reviews data set to show that EMTL is competitive with state-of-the-art UDA algorithms , without accessing the source data at the adaptation phase . To our best knowledge , EMTL is the first work using density estimation for unsupervised domain adaptation . Unlike other existing generative approaches ( Kingma et al. , 2014 ; Karbalayghareh et al. , 2018 ; Sankaranarayanan et al. , 2018 ) , EMTL can decouple the source density estimation process from the adaption phase and thus it can be used in situations where the source data is not available at the adaptation phase due to security or privacy reasons . 2 RELATED WORK . Zhuang et al . ( 2020 ) , Kouw & Loog ( 2019 ) and Pan & Yang ( 2009 ) categorize DA approaches into instance-based and feature-based techniques . Instance-based approaches reweight labeled source samples according to the ratio of between the source and the target densities . Importance weighting methods reweight source samples to reduce the divergence between the source and target densities ( Huang et al. , 2007 ; Gretton et al. , 2007 ; Sugiyama et al. , 2007 ) . In contrast , class importance weighting methods reweight source samples to make the source and target label distribution the same ( Azizzadenesheli et al. , 2019 ; Lipton et al. , 2018 ; Zhang et al. , 2013 ) . Feature-based approaches learn a new representation for the source and the target by minimizing the divergence between the source and target distributions . Subspace mapping methods assume that there is a common subspace between the source and target ( Fernando et al. , 2013 ; Gong et al. , 2012 ) . Courty et al . ( 2017 ) proposed to use optimal transport to constrain the learning process of the transformation function . Other methods aim at learning a representation which is domain-invariant among domains ( Gong et al. , 2016 ; Pan et al. , 2010 ) . Besides these shallow models , deep learning has also been widely applied in domain adaptation ( Tzeng et al. , 2017 ; Ganin et al. , 2016 ; Long et al. , 2015 ) . DANN ( Ganin et al. , 2016 ) learns a representation using a neural network which is discriminative for the source task while can not distinguish the source and target domains from each other . Kingma et al . ( 2014 ) and Belhaj et al . ( 2018 ) proposed a variational inference based semi-supervised learning approach by regarding the missing label as latent variable and then performing posterior inference . 3 NOTATION AND PROBLEM DEFINITION . We consider the unsupervised domain adaptation problem in a binary classification setting ( the setup is trivial to extend to multi-class classification ) . Let p ( x , y ) be a joint density function defined on X × Y , where x ∈ Rp is the feature vector and y ∈ { 0 , 1 } is the label . We denote the conditional probability p ( y = 1|x ) by q ( x ) . A hypothesis or model is a function h : X 7→ [ 0 , 1 ] . We define the error of h as the expected disagreement between h ( x ) and q ( x ) , i.e. , ( h ) = Ex∼p |h ( x ) − q ( x ) | . ( 1 ) We use superscripts s and t to distinguish the source and target domains , that is , ps ( x , y ) and pt ( x , y ) are the joint density functions in the source and target domains respectively . In general , we assume that ps ( x , y ) 6= pt ( x , y ) . Let Ds = { ( xsi , ysi ) } n s i=1 and U t = { xti } n t i=1 be i.i.d . data sets generated from the source distribution ps ( x , y ) and the marginal target distribution pt ( x ) , respectively , where ns and nt are source and target sample sizes . The objective of unsupervised domain adaptation is to learn a model ĥ by using labeled Ds and unlabeled U t , which achieves lowest error in target domain . 4 GENERATIVE APPROACH . Ben-David et al . ( 2010 ) proved that the error of a hypothesis h in the target domain t ( h ) can be bounded by the sum of error in source domain s ( h ) , the distribution distance between the two domains , and the expected L1 distance between two conditional probabilities . Theorem 1 ( Ben-David et al . ( 2010 ) , Theorem 1 ) For a hypothesis h , t ( h ) ≤ s ( h ) + d1 ( ps ( x ) , pt ( x ) ) + min { Ex∼ps |qs ( x ) − qt ( x ) | , Ex∼pt |qs ( x ) − qt ( x ) | } , ( 2 ) where d1 ( ps ( x ) , pt ( x ) ) = 2sup B∈B |Prs ( B ) − Prt ( B ) | is the twice the total variation distance of two domain distributions and qs ( x ) and qt ( x ) are the source and target probabilities of y = 1|x , respectively . In the covariate shift setting , we assume that the conditional probability p ( y|x ) is invariant between the source and the target domains . Thus in the right hand side of Eq . ( 2 ) , the third component will be zero , which means that the target error is bounded by the source error plus the distance between two domains . Many current unsupervised domain adaptation solutions work on how to reduce the distance between the two domain densities . Importance-sampling-based approaches manage to resample the source data to mimic the target data distribution , and feature-mapping-based approaches do that by learning a transformation function φ ( x ) for the source data . However , both approaches need to access source and target data simultaneously . In this paper , we propose a domain adaptation approach based on generative models . First , we learn all multivariate densities using RNADE ( Uria et al. , 2013 ) , an autoregressive version of Bishop ( 1994 ) ’ s mixture density nets . We found RNADE excellent in learning medium-dimensional densities , and in a certain sense it is RNADE that made our approach feasible . Second , we introduce a mediator joint density function pm ( x , y ) that bridges the gap between ps ( x , y ) and pt ( x , y ) . Since the source distribution information is stored in the learned generative model after training , we do not need to access source data in the adaptation phase .

Summary: This paper proposes a novel method for Unsupervised Domain Adaptation (UDA) when the source domain's privacy should be preserved. The authors propose EMTL, which is a generative method using multivariate densities using RNADE (Uria et al., 2013) and a mediator joint density function bridging both source and target domains. EMTL achieves comparable performances to those of DANN (Ganin et al., 2016) on a single dataset.