Knothe-Rosenblatt transport for Unsupervised Domain Adaptation

1 INTRODUCTION . In classical machine learning , we assume that both the training and test data follow the same distribution and we can thus expect to generalize from the training set to the test set . In practice , this assumption does not always hold . For example , data is often collected in asynchronous manner , at different times and locations , and may be labeled by different people , which can affect the efficiency and quality of the standard supervised learning models ( Quionero-Candela et al. , 2009 ; Pan & Yang , 2010 ) . Collecting data from multiple sources may also lead to distribution shift between the collectors . For example , wireless network data would present different properties and patterns depending on time ( such as day , night , week-end ) , or location/ infrastructure ( downtown , countryside , or touristic area ) . Even when the task is common , an efficient approach should take into account the shift . Coping with this problem lead to the development of transfer learning methods that adapt the knowledge from a source domain to a new target domain . Transfer learning , or domain adaptation , is central in vision ( image classification , image segmentation , or activity recognition ) ( Li et al. , 2020b ) and natural language processing ( translation , language generation ) ( Malte & Ratadiya , 2019 ; Ruder et al. , 2019 ) problems . Both of these domains generate very high-dimensional data , and transfer learning usually focuses on fine tuning pre-trained models to specific tasks . In contrast , the principal problem in many industrial applications is not dimensionality , rather class imbalance , probability shift in data collection , and small data ( Zhang et al. , 2019 ) . For example , wireless network data ( 5G and beyond ) , IoT or smart cities are often low dimensional ( less than 100 ) , and highly dependent on the data collection context ( Fu et al. , 2018 ; Arjoune & Faruque , 2020 ; Benzaid & Taleb , 2020 ) . These issues are rarely dealt within the transfer learning literature . Transfer learning on high-dimensional data usually proceeds by mapping the data into a smaller dimensional space and carrying out the transfer in this latent space . In the lower dimensional domain we are targeting , we can use recently developed powerful density estimation techniques and principled transport-based approaches that rely on these precise estimates . Our contribution is Knothe-Rosenblatt Domain Adaptation , or KRDA . We tackle Domain Adaptation ( DA ) which arises when the probability distribution of the source and the target data are different but related . We focus on the more challenging task where we do not have labeled target data . This approach , called Unsupervised Domain Adaptation , is the most difficult case of distribution shift . We estimate the density of both the source and the target data in order to transfer the former to the later . We use RNADE ( Uria et al. , 2013 ) , an autoregressive technique that decomposes the d-dimensional density into d one-dimensional conditional densities , represented by input dependent mixtures of Gaussian ( also known as mixture density nets ( Bishop , 1994 ) ) . Using these explicit representations , KRDA transfers each sample by preserving the conditional quantiles with KnotheRosenblatt transport . Once embedded in the target domain , the source and its labels are learned by a supervised learning algorithm . Although theoretically simple , using autoregressive models in order to perform a transport has not yet been considered in the transfer learning literature . As it will be illustrated , KRDA is particularly well suited for small data ( less than 10000 samples ) in small dimension ( less than 100 ) , where other state-of-the-art methods tend to under-perform , as shown in Section 6 . We can also consider KRDA as an embedding algorithm with a great advantage : all the extra-computational cost of KRDA is spent in the computation of the transfer map . Once the source is transferred , training and testing will have no overhead beyond the cost of the supervised learning algorithm used . The paper is structured as follows . We first introduce KRDA , an algorithm based on density estimation . We review some topics in density estimation in Section 4 . We then introduce KRDA , the core of our paper , in Section 5 , and expose its properties and limitations . Finally we compare our approach against state-of-the-art transfer learning algorithms on several benchmark . A detailed experimental setting and the results are given in Section 6 . 2 RELATED WORK . Transfer learning ( TL ) aims at building algorithms that generalize across different domains with different probability distributions , see for example ( Pan & Yang , 2010 ; Kouw & Loog , 2019 ; Zhuang et al. , 2019 ) for global surveys of the field . Domain adaptation is the specific case when the task is the same across the different domains . DA approaches may be roughly divided in two categories depending on whether we have access to labels in the target space , or not . The first case is known as semi-supervised DA . The usual approach is to find a global transformation that aligns the different domains by preserving the information coming from a few labels ( Saenko et al. , 2010 ) . Many papers embed both domains in the same latent space using different tools such as similarity ( Donahue et al. , 2013 ) , non-linear kernel mapping ( Pan et al. , 2010 ; Gong et al. , 2016 ) , or entropy ( Saito et al. , 2019 ) . In unsupervised domain adaptation , we assume that we have no labels from the target domain . One avenue is to reweight the samples in order to correct the shift between the source and target distributions ( Huang et al. , 2007 ; Gretton et al. , 2007 ) . This method has the advantage of not requiring distribution estimation or specific embedding . As in semi-supervised DA , other unsupervised approaches rely on a common latent space . Both the source and target data are projected into this space , and a classifier is then learned using the labeled source data in the latent space . Another shallow approach , subspace mapping , aims at learning a linear map that aligns source and target ( Gong et al. , 2012 ; Fernando et al. , 2013 ; Sharma et al. , 2012 ) . More recently , deep neural nets became a popular choice in UDA due to the flexibility of these models to learn rich non-linear mappings . Deep Adaptation Network ( Long et al. , 2015 ) adds multiple kernel variants of MMD at the top layers to push the target distribution close to the source . Domain-Adversarial Neural Network ( Ganin et al. , 2016 ) introduces adversarial training to reduce the distance between the source and target feature distributions . Joint Adaptation Networks ( Long et al. , 2017 ) and Conditional Adversarial Domain Adaptation ( Long et al. , 2018 ) aim at aligning the joint or conditional distributions . Instead of learning transferable representations , Saito et al . ( 2018 ) align the source and target distributions by maximizing the discrepancy between the outputs of two classifiers . Using clustering is another approach ( Shu et al. , 2018 ; Liang et al. , 2020 ; Li et al. , 2020a ) . In this paper we use recently developed powerful density estimators to relate the source and target domains . Density estimation is an important problem in statistics in general and machine learning in particular ( Bishop , 1994 ; Wasserman , 2004 ) . Among the plethora of methods ( Salakhutdinov & Hinton , 2009 ; 458 ; Rezende & Mohamed , 2015 ; Ho et al. , 2019 ) , we use autoregressive models for their triangularity that is crucial for our approach ( Larochelle & Murray , 2011 ; Uria et al. , 2013 ) . These algorithms model the joint distribution as product of one-dimensional conditional densities using the probability chain rule . They take advantage of recent developments in recurrent neural networks ( Oord et al. , 2016 ) . A drawback of this approach is the fixed arbitrary ordering of the components , although it seems not to be crucial in many applications ( Kégl et al. , 2021 ) , including ours , arguably explained by the flexibility of the mixtures that can model the potentially complex conditional densities . Optimal transport sees the domain adaptation problem as graph matching ( Courty et al. , 2017 ) and embed the source into the target by minimizing a transportation cost . Knothe-Rosenblatt transport has been independently introduced in ( Rosenblatt , 1952 ; Knothe , 1957 ) , the former for multivariate statistics analysis and the latter to study isoperimetric inequality problems . This approach has been applied for histogram equalization for RGB pictures ( Pitié et al. , 2007 ) . More recently , ( Muzellec & Cuturi , 2019 ) use a generalization of Knothe-Rosenblatt transport as a surrogate to optimal transport in high dimensional spaces : the paper fits a multi-dimensional GMM that is transferred at once . 3 BACKGROUND AND NOTATIONS . We consider the setting of classical transfer learning . The fundamental objective is to use some knowledge acquired during the learning of a specific predictive task in order to perform a similar but different task . More precisely , the domain D of a learning task is a couple composed of a feature space X and a marginal probability distribution p ( X ) . A task is a couple ( Y , f ) composed of a label space Y and a prediction function f : X → Y . In transfer learning , we consider two domains and learning tasks named source ( DS , YS ) and target ( DT , YT ) . In the transfer learning setting ( XS , pS ) 6= ( XT , pT ) and the goal is to transfer some knowledge from the source to the target . There are several sub-cases such as covariate shift , on which KRDA relies , in which pS ( x ) 6= pT ( x ) but the conditional probabilities are invariant : pS ( y|x ) = pT ( y|x ) for every y ∈ Y . In the unsupervised setting we have no access to the labels of the target data . We thus aim at building a transfer map T : XS → XT which associates a vector in the target feature space to every source sample before applying a classifier . Let DS = ( XS , YS ) and DT = ( XT , YT ) be the source and target data sets , respectively . Let pS and pT be the probability density functions ( PDF ) of the source data XS and target data XT , and let p̂S and p̂T be the estimated densities , respectively . We will denote by Fp the cumulative density function ( CDF ) associated with the density p. For a vector function g : Rm → Rn and x ∈ Rm , let gi ( x ) ∈ R be the i-th coefficient of g ( x ) . 4 AUTOREGRESSIVE DENSITY ESTIMATION . In this work , we will focus on autoregressive models . The probability density function is expressed using the probability chain rule : the PDF of a vector x = ( x1 , . . . , xd ) ∈ Rd is the product of one-dimensional conditional densities p ( x ) = d∏ i=1 pi ( xi|x < i ) . ( 1 ) Each conditional factor density will be approximated by a Gaussian mixture distribution . Note that this straightforwardly generalizes to any type of mixture distribution , although we will focus on Gaussian mixtures in this paper for didactic purposes . RNADE ( Uria et al. , 2013 ) is a robust and flexible deep learning method that , following Eq . ( 1 ) , fits one-dimensional conditional Gaussian mixtures ( originally proposed by ( Bishop , 1994 ) under the name of mixture density net ( MDN ) ) for every coefficient of a vector x = ( x1 , . . . , xd ) ∈ Rd . More precisely we associate to each conditional probability pi ( xi|x < i ) a distribution composed of a mixture of N Gaussians ∑N k=1 w i kN ( µik , σik ) . The RNADE algorithm with hidden size H is based on NADE ( Larochelle & Murray , 2011 ) and can be summarized as follows . We first compute from the input x = ( x1 , . . . , xd ) the sequence ai ∈ RH , i = 1 , . . . , d , in an iterative manner : a1 = c ; ai+1 = ai + xiW· , i , ( 2 ) where c ∈ RH and W ∈ RH×d are learned parameters , and W· , i denotes the ith column of the parameter matrix W . We then apply a non-linearity after re-scaling hi = σ ( Ciai ) ( 3 ) to get the parameters of the conditional Gaussian mixture as output of linear layers : wi = Softmax ( Lin1 ( h i ) ) , ( 4 ) µi = Lin2 ( h i ) , ( 5 ) σi = exp ( 0.5× Lin3 ( hi ) ) , ( 6 ) where Lin1 , Lin2 , Lin3 are three linear layers RH → RN with bias . In this work , we use σ = RELU as the non-linearity applied in ( 3 ) . The exact likelihood is thus directly accessible , and the model is trained end-to-end by maximizing the log-likelihood using gradient ascent . Note that this density estimator also makes data generation from the estimated distribution easy ( go through the chain Eq . ( 1 ) and sample from Gaussian mixtures ) . In domain adaptation , we make the assumption of having related distributions for the source and the target data . In order for the density estimation model to use this assumption , we share the parameters c and W in Eq . ( 2 ) for all data sets . The last linear layer of Eqs . ( 4-6 ) are specific to the source and target and will capture the dissimilarities between the domains . The density estimation network is then trained simultaneously on both the source and the target data .

Summary: This paper proposes to tackle unsupervised domain adaptation by following a classic trend aiming at finding an alignment between source and target domains. More specifically, the method uses a Knothe-Rosenblatt transport approach which applies a one-dimensional optimal transport to all conditional marginals of one distribution into another. The method uses an autoregressive density estimation from the RNADE algorithm based on a Gaussian Mixture strategy. The experimental evaluation compares the approach and other classic domain adaptation benchmarks on synthetic data, on the Amazon benchmark and a benchmark in higgh-energy physics.