Year: 2021

Abstract:
Unsupervised domain adaptation (UDA) aims at exploiting related but different data sources to tackle a common task in a target domain. UDA remains a central yet challenging problem in machine learning. In this paper, we present an approach tailored to moderate-dimensional tabular problems which are hugely important in industrial applications and less well-served by the plethora of methods designed for image and language data. Knothe-Rosenblatt Domain Adaptation (KRDA) is based on the Knothe-Rosenblatt transport: we exploit autoregressive density estimation algorithms to accurately model the different sources by an autoregressive model using a mixture of Gaussians. KRDA then takes advantage of the triangularity of the autoregressive models to build an explicit mapping of the source samples into the target domain. We show that the transfer map built by KRDA preserves each component quantiles of the observations, hence aligning the representations of the different data sets in the same target domain. Finally, we show that KRDA has state-of-the-art performance on both synthetic and real world UDA problems.

heading: 1 INTRODUCTION
text: In classical machine learning, we assume that both the training and test data follow the same distribution and we can thus expect to generalize from the training set to the test set. In practice, this assumption does not always hold. For example, data is often collected in asynchronous manner, at different times and locations, and may be labeled by different people, which can affect the efficiency and quality of the standard supervised learning models (Quionero-Candela et al., 2009; Pan & Yang, 2010). Collecting data from multiple sources may also lead to distribution shift between the collectors. For example, wireless network data would present different properties and patterns depending on time (such as day, night, week-end), or location/ infrastructure (downtown, countryside, or touristic area). Even when the task is common, an efficient approach should take into account the shift. Coping with this problem lead to the development of transfer learning methods that adapt the knowledge from a source domain to a new target domain.
Transfer learning, or domain adaptation, is central in vision (image classification, image segmentation, or activity recognition) (Li et al., 2020b) and natural language processing (translation, language generation) (Malte & Ratadiya, 2019; Ruder et al., 2019) problems. Both of these domains generate very high-dimensional data, and transfer learning usually focuses on fine tuning pre-trained models to specific tasks. In contrast, the principal problem in many industrial applications is not dimensionality, rather class imbalance, probability shift in data collection, and small data (Zhang et al., 2019). For example, wireless network data (5G and beyond), IoT or smart cities are often low dimensional (less than 100), and highly dependent on the data collection context (Fu et al., 2018; Arjoune & Faruque, 2020; Benzaid & Taleb, 2020). These issues are rarely dealt within the transfer learning literature.
Transfer learning on high-dimensional data usually proceeds by mapping the data into a smaller dimensional space and carrying out the transfer in this latent space. In the lower dimensional domain we are targeting, we can use recently developed powerful density estimation techniques and principled transport-based approaches that rely on these precise estimates.
Our contribution is Knothe-Rosenblatt Domain Adaptation, or KRDA. We tackle Domain Adaptation (DA) which arises when the probability distribution of the source and the target data are different but related. We focus on the more challenging task where we do not have labeled target data. This approach, called Unsupervised Domain Adaptation, is the most difficult case of distribution shift.
We estimate the density of both the source and the target data in order to transfer the former to the later. We use RNADE (Uria et al., 2013), an autoregressive technique that decomposes the d-dimensional density into d one-dimensional conditional densities, represented by input dependent mixtures of Gaussian (also known as mixture density nets (Bishop, 1994)). Using these explicit representations, KRDA transfers each sample by preserving the conditional quantiles with KnotheRosenblatt transport. Once embedded in the target domain, the source and its labels are learned by a supervised learning algorithm. Although theoretically simple, using autoregressive models in order to perform a transport has not yet been considered in the transfer learning literature.
As it will be illustrated, KRDA is particularly well suited for small data (less than 10000 samples) in small dimension (less than 100), where other state-of-the-art methods tend to under-perform, as shown in Section 6. We can also consider KRDA as an embedding algorithm with a great advantage: all the extra-computational cost of KRDA is spent in the computation of the transfer map. Once the source is transferred, training and testing will have no overhead beyond the cost of the supervised learning algorithm used.
The paper is structured as follows. We first introduce KRDA, an algorithm based on density estimation. We review some topics in density estimation in Section 4. We then introduce KRDA, the core of our paper, in Section 5, and expose its properties and limitations. Finally we compare our approach against state-of-the-art transfer learning algorithms on several benchmark. A detailed experimental setting and the results are given in Section 6.

heading: 2 RELATED WORK
text: Transfer learning (TL) aims at building algorithms that generalize across different domains with different probability distributions, see for example (Pan & Yang, 2010; Kouw & Loog, 2019; Zhuang et al., 2019) for global surveys of the field. Domain adaptation is the specific case when the task is the same across the different domains. DA approaches may be roughly divided in two categories depending on whether we have access to labels in the target space, or not. The first case is known as semi-supervised DA. The usual approach is to find a global transformation that aligns the different domains by preserving the information coming from a few labels (Saenko et al., 2010). Many papers embed both domains in the same latent space using different tools such as similarity (Donahue et al., 2013), non-linear kernel mapping (Pan et al., 2010; Gong et al., 2016), or entropy (Saito et al., 2019).
In unsupervised domain adaptation, we assume that we have no labels from the target domain. One avenue is to reweight the samples in order to correct the shift between the source and target distributions (Huang et al., 2007; Gretton et al., 2007). This method has the advantage of not requiring distribution estimation or specific embedding. As in semi-supervised DA, other unsupervised approaches rely on a common latent space. Both the source and target data are projected into this space, and a classifier is then learned using the labeled source data in the latent space. Another shallow approach, subspace mapping, aims at learning a linear map that aligns source and target (Gong et al., 2012; Fernando et al., 2013; Sharma et al., 2012).
More recently, deep neural nets became a popular choice in UDA due to the flexibility of these models to learn rich non-linear mappings. Deep Adaptation Network (Long et al., 2015) adds multiple kernel variants of MMD at the top layers to push the target distribution close to the source. Domain-Adversarial Neural Network (Ganin et al., 2016) introduces adversarial training to reduce the distance between the source and target feature distributions. Joint Adaptation Networks (Long et al., 2017) and Conditional Adversarial Domain Adaptation (Long et al., 2018) aim at aligning the joint or conditional distributions. Instead of learning transferable representations, Saito et al. (2018) align the source and target distributions by maximizing the discrepancy between the outputs of two classifiers. Using clustering is another approach (Shu et al., 2018; Liang et al., 2020; Li et al., 2020a).
In this paper we use recently developed powerful density estimators to relate the source and target domains. Density estimation is an important problem in statistics in general and machine learning in particular (Bishop, 1994; Wasserman, 2004). Among the plethora of methods (Salakhutdinov & Hinton, 2009; 458; Rezende & Mohamed, 2015; Ho et al., 2019), we use autoregressive models for their triangularity that is crucial for our approach (Larochelle & Murray, 2011; Uria et al., 2013). These algorithms model the joint distribution as product of one-dimensional conditional densities using the probability chain rule. They take advantage of recent developments in recurrent neural
networks (Oord et al., 2016). A drawback of this approach is the fixed arbitrary ordering of the components, although it seems not to be crucial in many applications (Kégl et al., 2021), including ours, arguably explained by the flexibility of the mixtures that can model the potentially complex conditional densities.
Optimal transport sees the domain adaptation problem as graph matching (Courty et al., 2017) and embed the source into the target by minimizing a transportation cost. Knothe-Rosenblatt transport has been independently introduced in (Rosenblatt, 1952; Knothe, 1957), the former for multivariate statistics analysis and the latter to study isoperimetric inequality problems. This approach has been applied for histogram equalization for RGB pictures (Pitié et al., 2007). More recently, (Muzellec & Cuturi, 2019) use a generalization of Knothe-Rosenblatt transport as a surrogate to optimal transport in high dimensional spaces: the paper fits a multi-dimensional GMM that is transferred at once.

heading: 3 BACKGROUND AND NOTATIONS
text: We consider the setting of classical transfer learning. The fundamental objective is to use some knowledge acquired during the learning of a specific predictive task in order to perform a similar but different task. More precisely, the domain D of a learning task is a couple composed of a feature space X and a marginal probability distribution p(X). A task is a couple (Y, f) composed of a label space Y and a prediction function f : X → Y . In transfer learning, we consider two domains and learning tasks named source (DS ,YS) and target (DT ,YT ). In the transfer learning setting (XS , pS) 6= (XT , pT ) and the goal is to transfer some knowledge from the source to the target. There are several sub-cases such as covariate shift, on which KRDA relies, in which pS(x) 6= pT (x) but the conditional probabilities are invariant: pS(y|x) = pT (y|x) for every y ∈ Y . In the unsupervised setting we have no access to the labels of the target data. We thus aim at building a transfer map T : XS → XT which associates a vector in the target feature space to every source sample before applying a classifier.
Let DS = (XS , YS) and DT = (XT , YT ) be the source and target data sets, respectively. Let pS and pT be the probability density functions (PDF) of the source data XS and target data XT , and let p̂S and p̂T be the estimated densities, respectively. We will denote by Fp the cumulative density function (CDF) associated with the density p. For a vector function g : Rm → Rn and x ∈ Rm, let gi(x) ∈ R be the i-th coefficient of g(x).

heading: 4 AUTOREGRESSIVE DENSITY ESTIMATION
text: In this work, we will focus on autoregressive models. The probability density function is expressed using the probability chain rule: the PDF of a vector x = (x1, . . . , xd) ∈ Rd is the product of one-dimensional conditional densities
p(x) = d∏ i=1 pi(xi|x<i). (1)
Each conditional factor density will be approximated by a Gaussian mixture distribution. Note that this straightforwardly generalizes to any type of mixture distribution, although we will focus on Gaussian mixtures in this paper for didactic purposes.
RNADE (Uria et al., 2013) is a robust and flexible deep learning method that, following Eq. (1), fits one-dimensional conditional Gaussian mixtures (originally proposed by (Bishop, 1994) under the name of mixture density net (MDN)) for every coefficient of a vector x = (x1, . . . , xd) ∈ Rd. More precisely we associate to each conditional probability pi(xi|x<i) a distribution composed of a mixture of N Gaussians ∑N k=1 w i kN (µik, σik). The RNADE algorithm with hidden size H is based on NADE (Larochelle & Murray, 2011) and can be summarized as follows. We first compute from the input x = (x1, . . . , xd) the sequence ai ∈ RH , i = 1, . . . , d, in an iterative manner:
a1 = c; ai+1 = ai + xiW·,i, (2)
where c ∈ RH and W ∈ RH×d are learned parameters, and W·,i denotes the ith column of the parameter matrix W . We then apply a non-linearity after re-scaling
hi = σ(Ciai) (3)
to get the parameters of the conditional Gaussian mixture as output of linear layers: wi = Softmax(Lin1(h i)), (4)
µi = Lin2(h i), (5)
σi = exp(0.5× Lin3(hi)) , (6) where Lin1, Lin2, Lin3 are three linear layers RH → RN with bias. In this work, we use σ = RELU as the non-linearity applied in (3). The exact likelihood is thus directly accessible, and the model is trained end-to-end by maximizing the log-likelihood using gradient ascent. Note that this density estimator also makes data generation from the estimated distribution easy (go through the chain Eq. (1) and sample from Gaussian mixtures).
In domain adaptation, we make the assumption of having related distributions for the source and the target data. In order for the density estimation model to use this assumption, we share the parameters c and W in Eq. (2) for all data sets. The last linear layer of Eqs. (4-6) are specific to the source and target and will capture the dissimilarities between the domains. The density estimation network is then trained simultaneously on both the source and the target data.

heading: 5 KNOTHE-ROSENBLATT TRANSPORT
text: Having two densities pµ and pν , there are several ways to built a transport place T such that T]µ = ν. For example, the change of variable formula pµ = pν(T (x)) det(JacxT ) defines a PDE for which T is solution (assuming existence). However this direct approach is not tractable in general. We propose here to use our autoregressive density estimation model in order to build a Knothe-Rosenblatt transport map. We refer to (Villani, 2008; Santambrogio, 2015) for an extensive presentation and study on Optimal Transport (OT) in general and Knothe-Rosenblatt (KR) transport in particular.

heading: 5.1 KNOTHE-ROSENBLATT TRANSPORT
text: Let µ and ν two absolutely continuous measures of R with F (x) = ∫ x −∞ dµ and G(x) = ∫ x −∞ dν their cumulative distribution function (CDF). We define the pseudo inverse of the CDF F as F−1(x) = inf{z ∈ R : F (z) > x} .
The following theorem gives a transportation map (actually optimal) between µ and ν. Theorem 1 ((Santambrogio, 2015, Theorem 2.5)). The map T = G−1 ◦ F verifies T#µ = ν.
Knothe-Rosenblatt transport (Rosenblatt, 1952; Knothe, 1957) is a simple transportation plan that applies one-dimensional optimal transport to all conditional marginals of one distribution into another. For didactic purposes, we give here a definition involving only density functions defined through the Lebesgue measure, i.e. we write µ(A) = ∫ A fdx where f is the density of the probability of µ. Let pS and pT be two density functions on Rd, hence pS = µS dλ(Rd) and pT = νT dλ(Rd).
Consider the first marginals pS(x1) and pT (x1) as one-dimensional random variables. By Theorem 1, we have a transport map T1 : R1 → R such that for x1 ∼ pS(x1), we have T1(x1) ∼ pT (x1). Now consider the conditional marginal pS(x2|x1) and pT (x2|x1), by Theorem 1 we construct again a map T2 : R2 → R such that for x1 ∼ pS(x1) and x2 ∼ pS(x2|x1), we have T2(x1, x2) ∼ pT (x2|x1). By iterating the previous process for all components, we construct a collection of d maps T1, . . . , Td. The Knothe-Rosenblatt transport is the map that sends x ∈ Rd to Rd by applying this construction to all conditional marginals in the following way:
T (x1, . . . , xd) = (T1(x1), T2(x1, x2), . . . , Td(x1, . . . , xd)) . (7) The following theorem assures the correctness of this approach as the density of the source is perfectly mapped on the target in the following sense. Proposition 2 ((Santambrogio, 2015, Proposition 2.18)). The map T satisfies T#µS = νT .
Relationship with Optimal Transport In one dimension, KR transport and OT coincide. Hence, KR transport optimally couples all conditionals. More generally Carlier et al. (2009) show that KR is a limit of optimal transport with quadratic costs lλ(x, y) = ∑ i λi(xi − yi) when λi/λi+1 → 0. By proceeding coefficient after coefficient instead of globally such as OT, KR transport might offer some interesting regularization for the specific case of transfer learning.

heading: 5.2 KNOTHE-ROSENBLATT DOMAIN ADAPTATION
text: In this section, we present the KRDA algorithm (for Knothe-Rosenblatt Domain Adaptation), the main contribution of the paper. We follow the Knothe-Rosenblatt construction of a transportation map based on estimated densities of the source and the target domain. In order to maximally exploit the conditional marginal structure of the map, we are relying on autoregressive density estimation models such as RNADE.
Let XS the source input data set with associated density pS . Modeling the PDF as a conditional mixture of Gaussians for every component, we obtain p̂S . For x ∈ XS we estimate p̂iS(xi) =∑ k w i kN (µik, σik). The CDF F is easily obtained by linearity from the PDF as
F i(xi) = ∑ k wikFN (µik,σik)(x i), (8)
where FN (µi,σi) is the CDF of the Gaussian N (µi, σi). KRDA builds a transfer function T : XS → XT such that for every sample x ∈ XS , we have F̂ iS(xi) = F̂ iT (T (xi)), or T (xi) = F̂ iT −1 ◦ F̂ iS(xi) (F̂ iT
−1 is the generalized inverse distribution function as in Equation 7). Theorem 1 assures that taking T = F̂ iT
−1 ◦ F̂ iS for every scalar component xi maintains the previous property. KRDA relies on Proposition 2 in the following sense. With RNADE, we first estimate the densities p̂S and p̂T of the source and the target data, before transferring all samples from the data set XS to the target domain XT . Let xS = (x1, . . . , xd) ∈ XS . From p̂i(xiS), we compute F̂ i(xiS) and using Proposition 2 we compute T (xi) ∈ XT such that F̂ iS(xi) = F̂ iT (T (xi)). Note that autoregressive models are triangular in the sense that F i(xi) does only depend on coefficients x<i = (xj)j<i. After having estimated the density of both the source and the target data, we thus construct T (x) deterministically, component by component.
1. p1T (x 1 T ) is fully determined by w 1, µ1 and σ1 that depend only on the parameter c, and hence is independent of x1S . We then assign x 1 T = F̂ 1 T −1 ◦ F̂ 1S(x1S) so that we have
F̂ 1T (x 1 T ) = F̂ 1 S(x 1 S).
2. piT (x i T |x<iT ) (or the neural net outputs wi, µi and σi) only depends on the components x <i T .
We then run RNADE on the partially computed xT in order to access the Gaussian mixture. As previously, we assign xiT = F̂ i T −1 ◦ F̂ iS(xiS) and we keep the quantile invariant property by construction.
This procedure terminates once all components of xT have been computed, hence after d steps in order to obtain xT ∈ XT such that
∀i ∈ {1, . . . , d}, F̂ i(xiS) = F̂ i(xiT ) . (9)
Note that in the case where the source and target domains are the same, i.e. pS = pT , we have T = id.
At every step, the computation of the inverse of the CDF is done with a binary search: the CDF is monotonically increasing and F−1(x) is a zero of the function z 7→ F (z) − x. The complete description of the algorithm is given in Algorithm 1, and we provide several visualizations of the transfer map in Section 6.
Once the whole source data XS is transferred into the target domain T (XS) ⊂ XT , a supervised learning algorithm (e.g., an SVM) is trained on the (labelled) data composed of the transferred source samples with their labels ( T (XS), YS ) . At test time, samples from the target domain are directly given to the learning algorithms for prediction, hence giving no overhead cost after the transfer phase.
Implementation details We perform the CDF inversion by exploiting its non-decreasing property. To compute F−1(x) we use a bisection algorithm to find a zero of the function z 7→ F (z)− x. The initial search interval is iteratively determined by the presence of a root in [−2k, 2k] for growing k. For numerical stability, we clip the values of x to [ , 1− ], with = 10−8. The transfer algorithm is parallelizable since each sample can be treated independently.
Algorithm 1 KRDA Learn p̂S on XS Learn p̂T on XT for xS ∈ S do
Limitations KRDA has two main limitations. The first one is the reliance of KRDA on the estimation made by RNADE. Since RNADE does not generalize well to high-dimensional spaces (≥ 100), we cannot expect KRDA to work well in this regime. The second limitation is the computation cost of the transfer: the algorithm iterates on the dataset (parallelizable) and on the components (not parallelizable because of the conditional distributions). Even though KRDA is linear in the number of samples, it may not be suited for very large datasets of millions of samples. As we do not use label information, our approach is suited for the covariate-shift setting.

heading: 6 EXPERIMENTS

heading: 6.1 DOMAIN ADAPTATION
text: In this section, we evaluate KRDA in both synthetic and real data sets. In all KRDA experiments, we model the source and target distributions with mixtures of five Gaussian components. All the
following experiments are performed in the unsupervised domain adaptation setting, when no labels of the target domain are accessible. After transformation, we apply an SVM on the transferred source. We compare our method to various techniques which can be classified into three classes:
• Baseline solutions: For a specific ML model, here Support Vector Machine (SVM), Source only learns this model using source data set with its labels; Target only learns the model by using the labeled target data (note that none of the transfer competitors have access the target data set labels, so this is meant as an optimistic baseline); • Shallow solutions: Subspace Alignment SA, (Fernando et al., 2013) and reweighting
methods Transfer Component Analysis TCA (Pan et al., 2011), Kernel Mean Matching KMM(Gretton et al., 2009) , each followed by a SVM; • Optimal Transport OT (Courty et al., 2017), followed by a SVM; • Deep learning solutions include DAN (Long et al., 2015), DANN (Ganin et al., 2016),
JAN (Long et al., 2017), CDAN (Long et al., 2018) and SHOT (Liang et al., 2020). We implement DAN, DANN, JAN and CDAN using the dalib library (Junguang Jiang, 2020). Instead of using the original image classifier in dalib that would be unsuitable in our benchmark, we use a five-layer MLP for each algorithm. For SHOT, the feature net is a four-layer MLP and the classifier net is a one-layer regression model a total of five layers.
For space reasons, we only show a sub-selection of the benchmark: the complete tables are located in Appendix A.
Hyperparameter Tuning We fix most hyperparameters in our experiments. In deep learning models (DANN, JAN, and CDAN) η = 1. The number of hidden neurons of MLP is set to 250 which will be equivalent to the number of parameters of the KRDA models. All the deep learning models are trained using the Adam optimizer (Kingma & Ba, 2015) with a learning rate set to 10−3.
Metric and cross validation Accuracy is used as the metric to evaluate the performance of different algorithms in the following experiments. All experiments have two classes. We run each experiment five times. In each running, we randomly pick 90% source and 90% target data to train the model. The test is systematically performed on unseen samples from the target data set. In each table, we report the average accuracy of the five experiments as well as 95% confidence intervals. The code used in all our experiments is available on Github1.

heading: 6.2 SYNTHETIC DATA EXPERIMENTS
text: Mixtures of Gaussians We generate a mixture of Gaussians for the source and target data. We use 1000 samples in both data sets, and we are interested in seeing how the domain adaptation tasks are handled by KRDA. For all tasks, RNADE uses N = 5 Gaussian components and a hidden layer of dimension 50. We plot several transfers from mixtures of m Gaussians to mixtures of n Gaussians. The results are presented in Figure 1.
Inter-twinning moons We perform three experiments on the classical inter-twinning moons dataset. In all these experiments, KRDA uses N = 5 Gaussian components and a hidden layer of dimension 50. The inter-twinning moon dataset is composed of two interlacing half-moons with labels 0 and 1.
1. We show the KRDA embedding from the source to a target domain with different components of the Gaussian mixture. We highlight the transfer of the same set of source sample in all figures. Both train and target data are of size 1000, the target distribution is a 40◦ rotation of the source distribution. These visualizations are shown in Figure 2.
2. We use the same experimental setup as in (Germain et al., 2013). We sample 300 samples from the source distribution and 300 in the target domain with various angles between 10◦ and 90◦. The difficulty of the problem increases with the angle. The test set is composed of 1000 samples from the target distribution. We show the performance of KRDA and competitors in Table 4.
3. KRDA is run in six inter-twining moons tasks to investigate its performance in different training sizes. The source and target training data size ranges from 200 to 1, 000. In each task, the target data
1It will be released after the review process.
distribution is rotated with 40◦ from the source. Following previous cross validation setting, in each task we run each algorithm five times with randomly picked 90% source and 90% target data and average the results. Figure 3 shows results and the corresponding 95% confidence interval. We find that although all algorithms tend to converge, KRDA acts excellently in small size cases and it is stabler than other algorithms in the low-data setting.

heading: 6.3 EXPERIMENTS ON REAL DATA
text: Hepmass The HEPMASS data comes from high-energy physics (Baldi et al., 2016). The objective of the task is to learn how to separate exotic particles from background noise using collision data (27 measured features). The data may be split according to the mass of the observed particles (m ∈ {500, 750, 1000, 1250, 1500}). We create different transfer learning tasks: transferring the domain from one mass to another. We build the source and target data by subsampling 1000 and 500 instances from the original data with a given source and target mass, respectively. We also sample 2000 independent instances from the target domain for the test set. KRDA uses N = 5 Gaussian components and dimension 100 for its hidden layer. Each transfer of the source data is followed by a SVM with the hyperparameters shared for all experiments that require it. SA uses 10 components. The results are summarized in Table 5. As shown, KRDA performs similarly and often better than state-of-the-art competitors.
Amazon dataset This data (McAuley et al., 2015) is an aggregation of reviews from four different products (dvd (D), books (B), electronics (E), kitchen (K)) and their given grade by customers. Each product or domain has about 2000 training samples and 4000 test samples. Each sample is presented by 5000 features and is associated to a binary class: 0 for samples ranked less than three stars and 1 otherwise. The goal is to transfer the review-to-grade classification from one product to another. Thus we created twelve transfer tasks using these products. As KRDA is based on RNADE for the density estimation, and RNADE is not designed for high dimensional data, we used a neural net (NN)
to reduce the sample dimensionality from 5000 to 5. The NN has two hidden layers of dimensions 10 and 5 (the encoding dimension). For each task, we train the NN as a classifier on the source data with a cross entropy loss. We then cut the output layer and use the trained NN to encode both the source and target data into 5 dimensions. This dimension reduction is the input data of all the algorithms on the benchmark and we trained all the competitors on this data. The results are shown in Table 6.

heading: B→ D 79.9±0.1 79.4±0.1 79.9±0.1 79.9± 0.1 79.9±0.1 79.7±0.1 79.7±0.6 80.0±0.2

heading: B→ E 69.2±0.2 72.7±0.0 69.9±0.4 73.0± 0.1 71.8±0.1 73.0±0.1 72.6±0.7 73.0±0.1

heading: B→ K 75.7±0.2 76.2±0.1 75.8±0.2 76.1± 0.1 76.3±0.1 76.2±0.1 76.2±0.2 76.2±0.1

heading: D→ B 75.1±0.1 75.6±0.1 75.1±0.2 75.3± 0.1 75.5±0.1 75.2±0.1 75.5±0.1 75.3±0.1

heading: D→ E 71.0±0.3 74.6±0.1 71.1±0.7 74.3± 0.1 72.7±0.2 74.6±0.1 73.8±0.9 74.4±0.2

heading: D→ K 74.9±0.1 77.3±0.1 74.8±0.5 77.5± 0.1 76.2±0.1 77.5±0.1 77.0±0.5 76.3±0.2

heading: E→ B 70.7±0.1 71.5±0.0 70.6±0.4 71.2± 0.2 70.9±0.2 71.1±0.1 71.3±0.3 70.9±0.2

heading: E→ D 72.3±0.1 73.6±0.0 72.1±0.4 73.1± 0.2 73.3±0.4 72.7±0.1 72.8±0.3 72.6±0.5

heading: E→ K 86.1±0.1 86.2±0.1 85.5±0.5 85.8± 0.2 83.5±0.3 86.0±0.1 85.6±0.6 85.2±1.0

heading: K→ B 71.2±0.2 71.4±0.1 71.2±0.3 71.6± 0.1 71.7±0.1 71.8±0.0 71.6±0.1 69.4±0.3

heading: K→ D 70.8±0.1 72.8±0.0 70.7±1.1 72.7± 0.3 70.6±0.0 72.8±0.2 70.6±1.0 72.8±0.7

heading: K→ E 84.0±0.0 84.4±0.0 84.0±0.2 84.3± 0.0 84.2±0.0 84.3±0.0 84.2±0.1 84.1±0.1

heading: 7 CONCLUSION
text: In this paper, we presented a novel transfer learning framework that exploits recent advances in density estimation techniques in order to transfer samples from a source domain to a target domain with a distribution shift using Knothe-Rosenblatt transport. The property that is invariant by the transfer performed by KRDA is the one dimensional conditional quantile distributions from the source in the target domain space, using an autoregressive setup. We showed that KRDA is state of the art on small to moderate dimensional tasks, often outperforming competitors especially in the case where there are few data samples.
Future work includes extending KRDA to the semi-supervised domain adaptation where a few labels are present in the target data. Another interesting research direction is to use other modern density estimators such as normalizing flows or variational autoencoders, in place of autoregressive mixture density nets.

heading: A EXPERIMENTS
text: A.1 INTER-TWINNING MOONS

Review summaries:
This paper proposes to tackle unsupervised domain adaptation by following a classic trend aiming at finding an alignment between source and target domains. More specifically, the method uses a Knothe-Rosenblatt transport approach which applies a one-dimensional optimal transport to all conditional marginals of one distribution into another. The method uses an autoregressive density estimation from the RNADE algorithm based on a Gaussian Mixture strategy. The experimental evaluation compares the approach and other classic domain adaptation benchmarks on synthetic data, on the Amazon benchmark and a benchmark in higgh-energy physics. 

In this paper, the authors propose a Knothe-Rosenblatt Domain Adaptation (KRDA) method. Specifically, it exploits autoregressive model using a mixture of Gaussians to accurately model the different sources, and then takes advantage of the triangularity of the autoregressive models to build an explicit mapping of the source samples into the target domain. Experiments on both synthetic and real-world datasets are done to verify KRDA.

The paper proposes Knothe-Rosenblatt Domain Adaptation (KRDA) for unsupervised domain adaptation with covariate shift. The key idea of the paper is to use an autoregressive model (RNADE) to estimate the density of source distribution and target distribution, then transport the estimated source density to the estimated target density by Knothe-Rosenblatt transportation. In particular, the transportation maps are constructed on each conditional density that is obtained from fitted autoregressive models. The authors evaluated the proposed model on synthetic data (mixture of Gaussians, Inter-twinning moons), and real data (Hepmass, Amazon dataset).