Knothe-Rosenblatt transport for Unsupervised Domain Adaptation

1 INTRODUCTION . In classical machine learning , we assume that both the training and test data follow the same distribution and we can thus expect to generalize from the training set to the test set . In practice , this assumption does not always hold . For example , data is often collected in asynchronous manner , at different times and locations , and may be labeled by different people , which can affect the efficiency and quality of the standard supervised learning models ( Quionero-Candela et al. , 2009 ; Pan & Yang , 2010 ) . Collecting data from multiple sources may also lead to distribution shift between the collectors . For example , wireless network data would present different properties and patterns depending on time ( such as day , night , week-end ) , or location/ infrastructure ( downtown , countryside , or touristic area ) . Even when the task is common , an efficient approach should take into account the shift . Coping with this problem lead to the development of transfer learning methods that adapt the knowledge from a source domain to a new target domain . Transfer learning , or domain adaptation , is central in vision ( image classification , image segmentation , or activity recognition ) ( Li et al. , 2020b ) and natural language processing ( translation , language generation ) ( Malte & Ratadiya , 2019 ; Ruder et al. , 2019 ) problems . Both of these domains generate very high-dimensional data , and transfer learning usually focuses on fine tuning pre-trained models to specific tasks . In contrast , the principal problem in many industrial applications is not dimensionality , rather class imbalance , probability shift in data collection , and small data ( Zhang et al. , 2019 ) . For example , wireless network data ( 5G and beyond ) , IoT or smart cities are often low dimensional ( less than 100 ) , and highly dependent on the data collection context ( Fu et al. , 2018 ; Arjoune & Faruque , 2020 ; Benzaid & Taleb , 2020 ) . These issues are rarely dealt within the transfer learning literature . Transfer learning on high-dimensional data usually proceeds by mapping the data into a smaller dimensional space and carrying out the transfer in this latent space . In the lower dimensional domain we are targeting , we can use recently developed powerful density estimation techniques and principled transport-based approaches that rely on these precise estimates . Our contribution is Knothe-Rosenblatt Domain Adaptation , or KRDA . We tackle Domain Adaptation ( DA ) which arises when the probability distribution of the source and the target data are different but related . We focus on the more challenging task where we do not have labeled target data . This approach , called Unsupervised Domain Adaptation , is the most difficult case of distribution shift . We estimate the density of both the source and the target data in order to transfer the former to the later . We use RNADE ( Uria et al. , 2013 ) , an autoregressive technique that decomposes the d-dimensional density into d one-dimensional conditional densities , represented by input dependent mixtures of Gaussian ( also known as mixture density nets ( Bishop , 1994 ) ) . Using these explicit representations , KRDA transfers each sample by preserving the conditional quantiles with KnotheRosenblatt transport . Once embedded in the target domain , the source and its labels are learned by a supervised learning algorithm . Although theoretically simple , using autoregressive models in order to perform a transport has not yet been considered in the transfer learning literature . As it will be illustrated , KRDA is particularly well suited for small data ( less than 10000 samples ) in small dimension ( less than 100 ) , where other state-of-the-art methods tend to under-perform , as shown in Section 6 . We can also consider KRDA as an embedding algorithm with a great advantage : all the extra-computational cost of KRDA is spent in the computation of the transfer map . Once the source is transferred , training and testing will have no overhead beyond the cost of the supervised learning algorithm used . The paper is structured as follows . We first introduce KRDA , an algorithm based on density estimation . We review some topics in density estimation in Section 4 . We then introduce KRDA , the core of our paper , in Section 5 , and expose its properties and limitations . Finally we compare our approach against state-of-the-art transfer learning algorithms on several benchmark . A detailed experimental setting and the results are given in Section 6 . 2 RELATED WORK . Transfer learning ( TL ) aims at building algorithms that generalize across different domains with different probability distributions , see for example ( Pan & Yang , 2010 ; Kouw & Loog , 2019 ; Zhuang et al. , 2019 ) for global surveys of the field . Domain adaptation is the specific case when the task is the same across the different domains . DA approaches may be roughly divided in two categories depending on whether we have access to labels in the target space , or not . The first case is known as semi-supervised DA . The usual approach is to find a global transformation that aligns the different domains by preserving the information coming from a few labels ( Saenko et al. , 2010 ) . Many papers embed both domains in the same latent space using different tools such as similarity ( Donahue et al. , 2013 ) , non-linear kernel mapping ( Pan et al. , 2010 ; Gong et al. , 2016 ) , or entropy ( Saito et al. , 2019 ) . In unsupervised domain adaptation , we assume that we have no labels from the target domain . One avenue is to reweight the samples in order to correct the shift between the source and target distributions ( Huang et al. , 2007 ; Gretton et al. , 2007 ) . This method has the advantage of not requiring distribution estimation or specific embedding . As in semi-supervised DA , other unsupervised approaches rely on a common latent space . Both the source and target data are projected into this space , and a classifier is then learned using the labeled source data in the latent space . Another shallow approach , subspace mapping , aims at learning a linear map that aligns source and target ( Gong et al. , 2012 ; Fernando et al. , 2013 ; Sharma et al. , 2012 ) . More recently , deep neural nets became a popular choice in UDA due to the flexibility of these models to learn rich non-linear mappings . Deep Adaptation Network ( Long et al. , 2015 ) adds multiple kernel variants of MMD at the top layers to push the target distribution close to the source . Domain-Adversarial Neural Network ( Ganin et al. , 2016 ) introduces adversarial training to reduce the distance between the source and target feature distributions . Joint Adaptation Networks ( Long et al. , 2017 ) and Conditional Adversarial Domain Adaptation ( Long et al. , 2018 ) aim at aligning the joint or conditional distributions . Instead of learning transferable representations , Saito et al . ( 2018 ) align the source and target distributions by maximizing the discrepancy between the outputs of two classifiers . Using clustering is another approach ( Shu et al. , 2018 ; Liang et al. , 2020 ; Li et al. , 2020a ) . In this paper we use recently developed powerful density estimators to relate the source and target domains . Density estimation is an important problem in statistics in general and machine learning in particular ( Bishop , 1994 ; Wasserman , 2004 ) . Among the plethora of methods ( Salakhutdinov & Hinton , 2009 ; 458 ; Rezende & Mohamed , 2015 ; Ho et al. , 2019 ) , we use autoregressive models for their triangularity that is crucial for our approach ( Larochelle & Murray , 2011 ; Uria et al. , 2013 ) . These algorithms model the joint distribution as product of one-dimensional conditional densities using the probability chain rule . They take advantage of recent developments in recurrent neural networks ( Oord et al. , 2016 ) . A drawback of this approach is the fixed arbitrary ordering of the components , although it seems not to be crucial in many applications ( Kégl et al. , 2021 ) , including ours , arguably explained by the flexibility of the mixtures that can model the potentially complex conditional densities . Optimal transport sees the domain adaptation problem as graph matching ( Courty et al. , 2017 ) and embed the source into the target by minimizing a transportation cost . Knothe-Rosenblatt transport has been independently introduced in ( Rosenblatt , 1952 ; Knothe , 1957 ) , the former for multivariate statistics analysis and the latter to study isoperimetric inequality problems . This approach has been applied for histogram equalization for RGB pictures ( Pitié et al. , 2007 ) . More recently , ( Muzellec & Cuturi , 2019 ) use a generalization of Knothe-Rosenblatt transport as a surrogate to optimal transport in high dimensional spaces : the paper fits a multi-dimensional GMM that is transferred at once . 3 BACKGROUND AND NOTATIONS . We consider the setting of classical transfer learning . The fundamental objective is to use some knowledge acquired during the learning of a specific predictive task in order to perform a similar but different task . More precisely , the domain D of a learning task is a couple composed of a feature space X and a marginal probability distribution p ( X ) . A task is a couple ( Y , f ) composed of a label space Y and a prediction function f : X → Y . In transfer learning , we consider two domains and learning tasks named source ( DS , YS ) and target ( DT , YT ) . In the transfer learning setting ( XS , pS ) 6= ( XT , pT ) and the goal is to transfer some knowledge from the source to the target . There are several sub-cases such as covariate shift , on which KRDA relies , in which pS ( x ) 6= pT ( x ) but the conditional probabilities are invariant : pS ( y|x ) = pT ( y|x ) for every y ∈ Y . In the unsupervised setting we have no access to the labels of the target data . We thus aim at building a transfer map T : XS → XT which associates a vector in the target feature space to every source sample before applying a classifier . Let DS = ( XS , YS ) and DT = ( XT , YT ) be the source and target data sets , respectively . Let pS and pT be the probability density functions ( PDF ) of the source data XS and target data XT , and let p̂S and p̂T be the estimated densities , respectively . We will denote by Fp the cumulative density function ( CDF ) associated with the density p. For a vector function g : Rm → Rn and x ∈ Rm , let gi ( x ) ∈ R be the i-th coefficient of g ( x ) . 4 AUTOREGRESSIVE DENSITY ESTIMATION . In this work , we will focus on autoregressive models . The probability density function is expressed using the probability chain rule : the PDF of a vector x = ( x1 , . . . , xd ) ∈ Rd is the product of one-dimensional conditional densities p ( x ) = d∏ i=1 pi ( xi|x < i ) . ( 1 ) Each conditional factor density will be approximated by a Gaussian mixture distribution . Note that this straightforwardly generalizes to any type of mixture distribution , although we will focus on Gaussian mixtures in this paper for didactic purposes . RNADE ( Uria et al. , 2013 ) is a robust and flexible deep learning method that , following Eq . ( 1 ) , fits one-dimensional conditional Gaussian mixtures ( originally proposed by ( Bishop , 1994 ) under the name of mixture density net ( MDN ) ) for every coefficient of a vector x = ( x1 , . . . , xd ) ∈ Rd . More precisely we associate to each conditional probability pi ( xi|x < i ) a distribution composed of a mixture of N Gaussians ∑N k=1 w i kN ( µik , σik ) . The RNADE algorithm with hidden size H is based on NADE ( Larochelle & Murray , 2011 ) and can be summarized as follows . We first compute from the input x = ( x1 , . . . , xd ) the sequence ai ∈ RH , i = 1 , . . . , d , in an iterative manner : a1 = c ; ai+1 = ai + xiW· , i , ( 2 ) where c ∈ RH and W ∈ RH×d are learned parameters , and W· , i denotes the ith column of the parameter matrix W . We then apply a non-linearity after re-scaling hi = σ ( Ciai ) ( 3 ) to get the parameters of the conditional Gaussian mixture as output of linear layers : wi = Softmax ( Lin1 ( h i ) ) , ( 4 ) µi = Lin2 ( h i ) , ( 5 ) σi = exp ( 0.5× Lin3 ( hi ) ) , ( 6 ) where Lin1 , Lin2 , Lin3 are three linear layers RH → RN with bias . In this work , we use σ = RELU as the non-linearity applied in ( 3 ) . The exact likelihood is thus directly accessible , and the model is trained end-to-end by maximizing the log-likelihood using gradient ascent . Note that this density estimator also makes data generation from the estimated distribution easy ( go through the chain Eq . ( 1 ) and sample from Gaussian mixtures ) . In domain adaptation , we make the assumption of having related distributions for the source and the target data . In order for the density estimation model to use this assumption , we share the parameters c and W in Eq . ( 2 ) for all data sets . The last linear layer of Eqs . ( 4-6 ) are specific to the source and target and will capture the dissimilarities between the domains . The density estimation network is then trained simultaneously on both the source and the target data .

Summary: The paper proposes Knothe-Rosenblatt Domain Adaptation (KRDA) for unsupervised domain adaptation with covariate shift. The key idea of the paper is to use an autoregressive model (RNADE) to estimate the density of source distribution and target distribution, then transport the estimated source density to the estimated target density by Knothe-Rosenblatt transportation. In particular, the transportation maps are constructed on each conditional density that is obtained from fitted autoregressive models. The authors evaluated the proposed model on synthetic data (mixture of Gaussians, Inter-twinning moons), and real data (Hepmass, Amazon dataset).