Knothe-Rosenblatt transport for Unsupervised Domain Adaptation
Authors: Aladin Virmaux
Illyyne Saffar
Jianfeng Zhang
Balázs Kégl
Venue: ICLR 2022 Submitted (2022)
Keywords: domain adaptation
transfer learning
deep learning
density estimation
transport

Abstract:     Unsupervised domain adaptation (UDA) aims at exploiting related but different data sources in order to tackle a common task in a target domain. UDA remains a central yet challenging problem in machine learning.
    In this paper, we present an approach based on the Knothe-Rosenblatt transport: we exploit autoregressive density estimation algorithms to accurately model the different sources by an autoregressive model using a mixture of Gaussians.
    Our Knothe-Rosenblatt Domain Adaptation (KRDA) then takes advantage of the triangularity of the autoregressive models to build an explicit mapping of the source samples into the target domain. We show that the transfer map built by KRDA preserves each component quantiles of the observations, hence aligning the representations of the different data sets in the same target domain.
    Finally, we show that KRDA has state-of-the-art performance on both synthetic and real world UDA problems.

Under review as a conference paper at ICLR 2022
KNOTHE-ROSENBLATT TRANSPORT FOR UNSUPER- VISED DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised domain adaptation (UDA) aims exploiting related but different data sources to tackle a common task in a target domain. UDA remains a central yet challenging problem machine learning. In this paper, we present an approach tailored to pndeerateddimesiion tabular problems which are hugely important in industrial applications and less well-served by the plethora of methods designed for image and language data. on the moothe-Rosenble algorithms to accurately model the different sources by an autoregressive model using a mixture of Gaussians. KRDA then takes advantage of the triangularity of the autoregressive models to build an explicit mapping of the source samples into the target domain. We show that the transfer map built by KRDA preserves each component quantiles of the observations, hence aligning the representations of the different data sets in the same target domain. Finally, we show that KRDA has state-of-the- performance both synthetic and real world UDA problems.
Domain Adaptation (KRDA) is based transport: we exploit autoregressive density estimation
INTRODUCTION
In classical machine learning, we assume that both the training and test data follow the same distribution and we can thus expect to generalize from the training set to the test set. In practice, this assumption does not always hold. For example, data is often collected in asynchronous manner, at different times and locations, and may be labeled by different people, which can affect the efficiency and quality of the standard supervised learning models (Quionero-Cancee 2010). Collecting data from multiple sources may also lead to distribution shift between the collectors. For example, wireless network data would present different properties and patterns depending (such as day, night, week-end), or location/ infrastructure (downtown, countryside, or touristic area). Even when the task is common, an efficient approach should take into account the shift. Coping with this problem lead to the development of transfer learning methods that adapt the knowledge from a source domain a new target domain.
et al., 2009; Pan & Yang,
time
Transfer learning, or domain adaptation, is central in vision (image classification, image segmentation, or activity recognition) (Li al., 2020b) and natural language processing (translation, language generation) (Malte & Ratadiya, 2019; Ruder et al., 2019) problems. Both of these domains generate very high -igh-dimensional data, and transfer learning usually focuses on fine tuning pre-trained models to specific tasks. In contrast, the principal problem in many industrial applications is not dimensionality, rather class imbalance, probability shift in data collection, and small data (Zhang et al., 2019). For example, wireless network data (5G and beyond), IoT or smart cities are often low dimensional (less than 100), and highly dependent on the data collection context (Fu et al., 2018; Arjoune Faruque, 2020; Benzaid & Taleb, 2020). These issues are rarely dealt within the transfer learning literature.
Transfer learning on dige-diimnsional data usually proceeds by mapping the data into a smaller dimensional space and carrying out the transfer in this latent space. In the lower dimensional domain we are targeting, we can use recently developed powerful density estimation techniques and principled transport- based approaches that rely these precise estimates.
Our contribution is Roseebl-t Domain Adaptation, or KRDA. We tackle Domain Adaptation (DA) which arises when the probability distribution of the source and the target data different but related. We focus on the more challenging task where we do not have labeled target data. This approach, called Unsupervised Domain Adaptation, is the most difficult case of distribution shift.

---

Under review as a conference paper at ICLR 2022
We estimate the density of both the source and the target data in order to transfer the former to the later. We use RNADE (Uria et al., 2013), an autoregressive technique that decomposes the d-dimensional density into d dieeensinni conditional densities, represented by imput dependent mixtures of Gaussian (also known as mixture density nets (Bishop, 1994)). Using these explicit representations, KRDA transfers each sample by preserving the conditional quantiles with Knothe- Rosenblatt transport. Once embedded in the target domain, the source and its labels are learned by a supervised learning algorithm. Although theoretically simple, using autoregressive models in order to perform a transport has not yet been considered in the transfer learning literature.
As it will be illustrated, KRDA is particularly well suited for small data (less than 10000 samples) in small dimension (less than 100). where other state-of-the shown in Section 6. We can also consider KRDA as an embedding algorithm with great advantage: all the extra-computational cost of KRDA is spent in the computation of the transfer map. Once the source is transferred, training and testing will have no overhead beyond the cost of the supervised learning algorithm used.
methods tend to under- snder-perform, as
The paper is structured as follows. We first introduce KRDA, an algorithm based on density estimation. We review some topics in density estimation in Section 4. We then introduce KRDA, the core of our paper, in Section 5, and expose its properties and limitations. Finally we compare our approach against state-of-the art transfer learning algorithms several benchmark. A detailed experimental setting and the results are given in Section 6.
RELATED WORK
Transfer learning (TL) aims at building algorithms that generalize across different domains with different probability distributions, see for example (Pan & Yang, 2010; Kouw & Loog, 2019; Zhuang al., 2019) for global surveys of the field. Domain adaptation is the specific case when the task is the same across the different domains. DA approaches may be roughly divided in two categories depending on whether we have access to labels in the target space, or not. The first case is known as semi-supervised DA. The usual approach is to find global transformation that aligns the different domains by preserving the information coming from a few labels (Saenko et al., 2010). Many papers embed both domains in the same latent space using different tools such as similarity (Donahue et al., 2013), non-linear kernel mapping (Pan al., 2010; Gong et al., 2016), or entropy (Saito et al., 2019).
In unsupervised domain adaptation, we assume that we have no labels from the target domain. One avenue is to reweight the samples in order to correct the shift between the source and target distributions (Huang et al., 2007: Gretton et al., 2007). This method has the advantage of not requiring distribution estimation or specific embedding. As in semi-supervised DA, other unsupervised approaches rely a common latent space. Both the source and target data are projected into this space, and a classifier is then learned using the labeled source data in the latent space. Another shallow approach, subspace mapping, aims at learning linear map that aligns source and target (Gong et al., 2012; Fernando et al., 2013; Sharma et al., 2012).
More recently, deep neural nets became popular choice in UDA due to the flexibility of these models to learn rich non-linear mappings. Deep Adaptation Network (Long et al., 2015) adds multiple kernel variants of MMD at the top layers to push the target distribution close to the source. Domain Adversaad Neural Network (Ganin et al., 2016) introduces adversarial training to reduce the distance between the source and target feature distributions. Joint Adaptation Networks (Long et al., 2017) and Conditional Adversarial Domain Adaptation (Long et al., 2018) aim at aligning the joint or conditional distributions. Instead of learning transferable representations. Saito et al. (2018) align the source and target distributions by maximizing the discrepancy between the outputs of two classifiers. Using clustering is another approach (Shu et al., 2018; Liang et al., 2020; Li et al., 2020a).
In this paper we use recently developed powerful density estimators to relate the source and target domains. Density estimation is an important problem in statistics in general and machine learning in particular (Bishop, 1994; Wasserman, 2004). Among the plethora of methods (Salakhutdinov & Hinton, 2009; 458; Rezende & Mohamed, 2015; Ho et al., 2019), we use autoregressive models for their triangularity that is crucial for our approach (Larochelle & Murray, 201 1; Uria al., 2013). These algorithms model the joint distribution as product of one-dimensional conditional densities using the probability chain nule. They take advantage of recent developments in recurrent neural
2

---

Under review as a conference paper at ICLR 2022
networks (Oord al., 2016). A drawback of this approach is the fixed arbitrary ordering of the components, although it seems not to be crucial in many applications (Kégl et al., 2021). including ours, arguably explained by the flexibility of the mixtures that can model the potentially complex conditional densities.
Optimal transport sees the domain adaptation problem as graph matching (Courty al., 2017) and embed the source into the target by minimizing a transportation cost. Roseeb-Rossnblatt transport has been independently introduced in (Rosenblatt, 1952; Knothe, 1957), the former for multivariate statistics analysis and the latter to study isoperimetric inequality problems. This approach has been applied for histogram equalization for RGB pictures (Pitié et al., 2007). More recently, (Muzellec Cuturi, 2019) use a generalization of Roseeb--ss transport as a surrogate to optimal transport in high dimensional spaces: the paper fits a multi-dimensional GMM that is transferred at once.
BACKGROUND AND NOTATIONS
We consider the setting of classical transfer learning. The fundamental objective is to use some knowledge acquired during the learning of a specific predictive task in order perform a similar but different task. More precisely, the domain 'D of learning task is a couple composed of feature space X' and a marginal probability distribution p(X). A task is a couple ()) composed of a label space y and a prediction function f X + y. In transfer learning, we consider two domains and learning tasks named source (/s) and target (or.))), In the transfer learning setting (Xs.Ps) + (&y,Pr) and the goal is to transfer some knowledge from the source the target. There are several sub-cases such as covariate shift, on which KRDA relies, in which ps (2( o pT (a) but the conditional probabilities invariant: ps (y(lar) = PT for every y y.
In the unsupervised setting we have no access to the labels of the target data. We thus aim at building transfer map T : X's - Xr which associates vector in the target feature space every source sample before applying a classifier.
Let Ds (Xs, Ys) and DT = (XT, be the source and target data sets, respectively. Let ps and pr be the probability density functions (PDF) of the source data X and target data Xy. and let PS and fT be the estimated densities, respectively. We will denote by Fp the cumulative density function (CDF) associated with the density p. For vector function R + R" and x E R". let gxx) be the i-th coefficient of g(x).
AUTOREGRESSIVE DENSITY ESTIMATION
In this work, we will focus autoregressive models. The probability density function is expressed
using the probability chain rule: the PDF of a vector x (x' ...., rd)) E Rd is the product of one-dimensional conditional densities
p I''(
Each conditional factor density will be approximated by a Gaussian mixture distribution. Note that this straightforwardly generalizes to any type of mixture distribution, although we will focus Gaussian mixtures in this paper for didactic purposes.
RNADE (Uria et al., 2013) robust and flexible deep learning method that, following Eq. (I), fits one-dimensional conditional Gaussian mixtures (originally proposed by (Bishop, 1994) under the name of mixture density net (MDN)) for every coefficient of a vector z (z) xd) E Id More precisely we associate to each conditional probability p' (=*|z<>)) a distribution composed of a mixture of N Gaussians Σn wL. of). The RNADE algorithm with hidden size H is based on NADE (Larochelle & Murray, 2011) and can be summarized as follows. We first compute from the input (x² ............................... x²)) the sequence a² E R.H i == I,..., in iterative manner:
a
a d+ a² z'W...
(2)
where E IR' H and W E #xd learned parameters, and W., denotes the ith column of the parameter matrix W. We then apply a nin-linyarity after re-sciging
h² (C'aa')
(3)
3

---

Under review as a conference paper at ICLR 2022
to get the parameters of the conditional Gaussian mixture output of linear layers:
w' = Softmax(Ling (h')),
(4)
μ² Lin2(h'),
(5)
0' = exp(0.5 x Ling(h'))
(6)
where Ling, Ling, Ling are three linear layers IR - IR with bias. In this work, we use RELU
as the non -on-linear applied in (3). The exact likelihood is thus directly accessible, and the model is trained end-to-end by maximizing the log- ligeelkli using gradient ascent. Note that this density
estimator also makes data generation from the estimated distribution easy (go through the chain Eq. (1) and sample from Gaussian mixtures).
In domain adaptation, we make the assumption of having related distributions for the source and the target data. In order for the density estimation model use this assumption, we share the parameters and W in Eq. (2) for all data sets. The last linear layer of Eqs. (4-6) are specific to the source and target and will capture the dissimilarities between the domains. The density estimation network is then trained simultaneously on both the source and the target data.
5 KNOTHE-ROSENBLANE TRANSPORT
Having two densities Pa and Pv. there are several ways built transport place T such that Tap v. For example, the change of variable formula Рy, (Txr)) det(Jac,T") defines a PDE for which is solution (assuming existence). However this direct approach not tractable in general. We propose here to use our autoregressive density estimation model in order to build a Roseellats transport map. We refer to (Villani, 2008; Santambrogio, 2015) for an extensive presentation and study on Optimal Transport (OT) in general and
(KR) transport in particular.
5.1 KNOTHE-ROSENRL.N TRANSPORT
Let μI and v two absolutely continuous measures of R with F(x) = f du and G(z) = f"oo de their cumulative distribution function (CDF). We define the pseudo inverse of the CDF F Fas
F (z) inf( R F(z) x}.
The following theorem gives a transportation map (actually optimal) between μ and v. Theorem ((Santambrogio, 2015, Theorem 2.5)). The map T G o F verifies T
Knothe-Rosenblan transport (Rosenblatt, 1952; Knothe, 1957) is simple transportation plan that applies one-dimensional optimal transport to all conditional marginals of one distribution into another. For didactic purposes, give here definition involving only density functions defined through the Lebesgue measure, i.e. we write p(A) = fdr where f is the density of the probability of p. Let Ps and pT be two density functions on R2d hence ps μs ((((R)) and pT VT d
Consider the first marginals ps (x1) and py (11) ) as one-dimensional random variables. By Theorem 1, we have transport map T, R - R such that for 21 ~ ps(x1), we have T, (x1) PT (T(r). Now consider the conditional marginal (ps(222az (x1) and pT (#2]21), J. by Theorem we construct again map
T2 2 + I such that for æ (2 and 12 ps( (x1), we have (2 22) PT (22/21)) By iterating the previous process for all components, we construct a collection of maps Tı.... Ta- The Knothe-Rosenble transport is the map that sends Idd to Rd by applying this construction to all conditional marginals the following way:
T(r Ta) (Ti (x1). T2(21, 22), .... Ta(xi,..., xa))
The following theorem assures the correctness of this approach as the density of the source is perfectly mapped on the target the following sense.
Proposition 2 (SSantamboggoo, 2015, Proposition 2.18).. The map T satisfies Tyus = VT.
Relationship with Optimal Transport In one dimension, KR transport and OT coincide. Hence, KR transport optimally couples all conditionals. More generally Carlier et al. (2009) show that KR is limit of optimal transport with quadratic costs (x y) = E, >x - y1) when A4/2441 - 0. By proceeding coefficient after coefficient instead of globally such as OT, KR transport might offer some interesting regularization for the specific case of transfer learning.
4

---

Under review as a conference paper at ICLR 2022
5.2 KNOTHE- ROSENBAAT DOMAIN ADAPTATION
Domain Adaptation), the construction of transportation map based on estimated densities of the source and the target domain. In order to maximally exploit the conditional marginal structure of the map, we are relying on autoregressive density estimation models such RNADE.
In this section, we present the KRDA algorithm (for main contribution of the paper. We follow the
Let Xs the source input data set with associated density ps. Modeling the PDF as a conditional mixture of Gaussians for every component, obtain Ps. For Xs estimate Pas Σn w (1 of). The CDF F is easily obtained by linearity from the PDF as
F'(x")= - Σ wt (f)
(8)
where PN oit is the CDF of the Gaussian (u Ø²). KRDA builds transfer function T' Xs + Ny such that for every sample x E Ag, we have Fx = (F((T=))) or T(x*) F7 (x²) (F+-1 is the generalized inverse distribution function as in Equation 7). Theorem 1 assures that taking T = F7 -1 o F'S for every scalar component x maintains the previous property.
KRDA relies on Proposition 2 in the following sense. With RNADE, we first estimate the densities Ps and py of the source and the target data, before transferring all samples from the data set Xs to the target domain Xy. Let Js (s' ,...,.") E Xs. From 1 (x's), we compute F F'(x's) and using Proposition we compute T(x') E Xy such that Fz"(") = Fi (T(")). Note that autoregressive models are triangular in the sense that (*xx') does only depend on coefficients σ (1)))< After having estimated the density of both the source and the target data, we thus construct (xx) deterministically. component by component.
1. p (x+} is fully determined by w n² and a² that depend only on the parameter c, and hence is independent of z's- We then assign 21 = f7-1 1 (x's) so that we have (++)) = (1's))
Sa 2. ply 127 (or the neural net outputs w", μe and σ²) only depends on the components 27 We then run RNADE on the partially computed xt in order to access the Gaussian mixture. As previously, we assign 2t = Fi- F3 (x's) and we keep the quantile invariant property by construction.
This procedure terminates once all components of xт have been computed, hence after d steps in order to obtain Iy E Xy such that
Vi E (1....,d), (1's)) = (z
Note that in the case where the source and target domains are the same, Lee. ps PT, have T= id.
At every step, the computation of the inverse of the CDF is done with binary search: the CDF is monotonically increasing and F- is a zero of the function 2 + F(2) - x. The complete description of the algorithm is given in Algorithm and we provide several visualizations of the transfer map in Section 6.
Once the whole source data Xs is transferred into the target domain T(Xs) X' supervised learning algorithm (e.g., an SVM) is trained on the (labelled) data composed of the transferred source samples with their labels (T(Xs), Ys). At test time, samples from the target domain are directly given to the learning algorithms for prediction, hence giving no overhead cost after the transfer phase.
We perform the CDF inversion by exploiting its non-decreasing property. To compute F-anteeFFGG (x)) we use a bisection algorithm to find a zero of the function 2 + F F(2) - r. The initial search interval is iteratively determined by the presence of root in [-2 2*1 for growing k. 10-8
Implementation details
For numerical stability, we clip the values æ to [c, 1 - c). with e = 10 8, The transfer algorithm is parallelizable since each sample can be treated independently.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 KRDA
Learn Ps on Xs Learn PT on XT
for Js S do
Initialize 27 = 0 E Rd for i E {1..... d do
Compute F (x's)
Compute the partial CDF T -1 from 8
Compute 2 F7 (2'()
end for
Set ( 2T end for
run learning algorithm (e.g. SVM) on (T(Xs), Ys)
= = 3
3, 2
n
Figure I: m- Gaussian mixture transferred with KRDA to an n-Gaussian mixture; we plot the source (blue), target (green), transferred (orange) and some mappings (red).
components
3 components
5 components
Figure 2: KRDA the inter-twinning moon dataset with different number of components; plot the source (blue), target (green), transferred (orange) and some mappings (red).
Limitations KRDA has two main limitations. The first one is the reliance of KRDA on the estimation made RNADE. Since RNADE does not generalize well to high-dimensional spaces (z 100), we cannot expect KRDA to work well in this regime. The second limitation is the computation of the transfer: the algorithm iterates on the dataset (parallelizable) and on the components (not parallelizable because of the conditional distributions), Even though KRDA is linear in the number samples, may be suited for very large datasets of millions of samples. As we do not use label information, our approach is suited for the covariate shift setting.
EXPERIMENTS
6.1 DOMAIN ADAPTATION
In this section, we evaluate KRDA in both synthetic and real data sets. all KRDA experiments, model the source and target distributions with mixtures of five Gaussian components. All the

---

Under review as a conference paper at ICLR 2022
following experiments are performed in the unsupervised domain adaptation setting, when labels of the target domain are accessible. After transformation, we apply an SVM on the transferred source. We compare method to various techniques which can be classified into three classes:
Baseline solutions: For a specifie ML model, here Support Vector Machine (SVM), Source only learns this model using source data set with its labels; Target only learns the model by using the labeled target data (note that none of the transfer competitors have access the target data set labels, so this is meant as optimistic baseline);
Shallow solutions: Subspace Alignment SA, (Fernando et al., 2013) and reweighting methods Transfer Component Analysis TCA (Pan et al., 2011), Kernel Mean Matching K.M(GGretton et al., 2009), each followed by a SVM;
Optimal Transport OT (Courty et al., 2017), followed by SVM;
Deep learning solutions include DAN (Long et al., 2015), DANN (Ganin et al., 2016),
JAN (Long et al., 2017), CDAN (Long et al., 2018) and SHOT (Liang et al., 2020). We implement DAN, DANN, JAN and CDAN using the da 1 iibb library (Junguang Jiang, 2020). Instead of using the original image classifier in dal ib that would be unsuitable in our benchmark, we use a five-layer MLP for each algorithm. For SHOT, the feature net is four-layer MLP and the classifier net is a one-layer regression model a total of five layers.
For space reasons, we only show sub-selection of the benchmark: the complete tables are located in Appendix A.
Hyperparameter Tuning We fix most hyperparameters in our experiments. In deep learning models (DANN, JAN, and CDAN) 1 = 1. The number of hidden neurons of MLP is set to 250 which will be equivalent the number parameters of the KRDA models. All the deep learning models are trained using the Adam optimizer (Kingma & Ba, 2015) with learning rate set to 10 10-3.
Metric and cross validation Accuracy used as the metric evaluate the performance of different algorithms in the following experiments. All experiments have two classes. We run each experiment five times. In each rumning, we randomly pick 90% source and 90% target data to train the model. The test is systematically performed on unseen samples from the target data set. In each table, we report the average accuracy of the five experiments as well as 95% confidence intervals. The code used in all our experiments is available on Github".
6.2 SYNTHETIC DATA EXPERIMENTS
Mixtures of Gaussians We generate a mixture of Gaussians for the source and target data, We use 1000 samples in both data sets, and are interested in seeing how the domain adaptation tasks are handled by KRDA. For all tasks, RNADE uses = 5 Gaussian components and a hidden layer of dimension 50. We plot several transfers from mixtures of m Gaussians mixtures of n. Gaussians. The results are presented in Figure 1.
Inter-twinning moons In all these experiments, KRDA uses N = 5 Gaussian components and a hidden layer of dimension 50. The inter-twinning moon dataset is composed of two interlacing half-moons with labels 0 and 1.
We perform three experiments the classical inter- twttnigwinning moons dataset.
1. We show the KRDA embedding from the source to a target domain with different components of the Gaussian mixture. We highlight the transfer of the same of source sample in all figures. Both train and target data are of size 1000, the target distribution is a 40" rotation of the source distribution. These visualizations are shown in Figure 2.
2. We use the same experimental setup as in (Germain et al.. 2013). We sample 300 samples from the source distribution and 300 in the target domain with various angles between 10º and 90 The difficulty of the problem increases with the angle. The test set is composed of 1000 samples from the target distribution. We show the performance of KRDA and competitors in Table
3. KRDA is run in six inter-twining moons tasks to investigate its performance in different training sizes. The source and target training data size ranges from 200 to 1, 000. In each task, the target data
It will be released after the review pracess.
7

---

Under review as a conference paper at ICLR 2022
distribution is rotated with 40 from the source. Following previous cross validation setting, in each task we run each algorithm five times with randomly picked 90% source and 90% target data and average the results. Figure 3 shows results and the corresponding 95% confidence interval. We find that although all algorithms tend to converge, KRDA acts excellently in small size cases and is stabler than other algorithms in the low-data setting.
Table 1: Inter-twinning moons unsupervised
Task
Source
Target
DANN
SA
KMM
OT
SHOT
KRDA
100=0.00
100+0
100=0.00
85.3+0.5
50.1+12.0
100.0-:0.0
82.8+10.8
100+00.0
20
99.911011
10010
99.3.0.55
78.510.4
53.0.188
100.011.00
81.413.7
100:10.0
30
96.6110.4
10010
89.417.2
73.410..3
51.4110.9
99.810.22
77.112.1
100.0.00
40"
73.1.2.5
10010
73.8.121.3
69.2.10.2
53.4127.3
89.6.11.2
24.4112.2
98.4-12.2
50
41.5+2.0
100+
48.5+21.9
61.8+0.5
56.347.8
83.8+0.88
22.3+1.6
98.4=1.55
60"
28.7::0.6
100+0
44.2#11.33
54.8=0.4
4788+18.1
78.0+1...
19,941.3
90.5+2.8
70"
23.3=t0.4
100+0
24.3#1.6
49.0=0.4
52.5+9.0
71.6+0.6
17.41122
84,4
80"
20.41:1.70
10040
20.14:2.4
43.2+10.6
48.2499.1
65.8+00.7
15.7+=1.3
81.2++3,4
90
18.1+00.3
100+0
17.7
38.2+0011
54.4+17.99
59.411.22
14.8+11.3
71.14339
1.0
0.8
Accuracy
0.7-
Algo
0.6
JAN
DANN
0.5-
KRDA
0.4-
OT
KMM
200
300
400
500
600
700
800
900
1000
Training Size
Figure 3: The learning curves of different UDA algorithms under different source and target training sizes. KRDA has excellent performance small data sets and shows very stable accuracy over different training sizes.
6.3 EXPERIMENTS ON REAL DATA
Hepmass The HEPMASS data comes from high-energy physics (Baldi al., 2016). The objective of the task is to learn how to separate exotic particles from background noise using collision data (27 measured features). The data may be split according to the mass of the observed particles (m E (500, 750, 1000, 1250, 1500)). We create different transfer learning tasks: transferring the another. We build the source and target data by subsampling 1000 and 500 instances from the original data with a given source and target mass, respectively. We also sample 2000 independent instances from the target domain for the test set. KRDA uses N Gaussian components and dimension 100 for its hidden layer. Each transfer of the source data is followed by SVM with the hyperparameters shared for all experiments that require SA uses 10 components. The results are summarized in Table 5. As shown, KRDA performs similarly and often better than state-of-the-art competitors.
domain from
Amazon dataset This data (McAuley et al., 2015) is aggregation of reviews from four different products (dvd (D), books (B), electronics (E), kitchen (K)) and their given grade by customers. Each product domain has about 2000 training samples and 4000 test samples. Each sample is presented by 5000 features and is associated to a binary class: 0 for samples ranked less than three stars and 1 otherwise. The goal is to transfer the review-to -o-gdde classification from one product another. Thus we created twelve transfer tasks using these products. As KRDA is based on RNADE for the density estimation, and RNADE is not designed for high dimensional data, used neural net (NN)
8

---

Under review as a conference paper at ICLR 2022
Table 2: HEPMASS unsupervised domain adaptation
Task
Source
Target
DANN
SA
KMM
OT
SHOT
KRDA
500
82.2:10.3
7..6::2.3
500 - 1000
67.2.10.1
89.2.10.4
56.71.4.4
7..74:10
50.1
72.1 1.9
85.5 L1
80.3 3.1
500 +1250
6..6:1:0.3
93.3-1-0.3
54.3433,8
71.3444.1
40.1
74.6=2.3
89.8:114
82.9
500 - 1500
58.0.00.2
95.4.1.0.
55.71.3.1
3..0.1.7
61881209
73.61..00
90.8.10.4
60.0.1..1 6.1
750 +0500
52.6.2..1
56.3:10.5
54.02:0.2
16.1::111
53.1
56.9=0.7
53.4::088
55.7=1.1
750 1000
86.5=0.11
89.2+004
82.2+:0.8
83.1+22.6
87.9-0.2
86.2=0.6
87.5:
87.7=0.7
750 +1250
87.7--0.D
95.4:20.4
81.31:2.9
88.2::3.11
89.5:100.5
92.5=0.9
92.0=10.4
92.3
750
82.51.2.8
92.5
1000 ++500
51.9+10.0
56.3:10.5
51.940.4
55.5:10999
51.9-0022
55.0=0.5
53.3:10.
53.2=117
1000 -3750
77.11:0.1
82.2.10.3
7..81:0.6
77.311111
77.7:10.44
80.5
80.6.11.11
80.7 0.9
1000 250
92.6:10.00
93.3-10.3
90.910.55
88.7:1100
91.2:10.4
92.1=0.4
91.7:104
92.5 0.3
1000 -+1500 1500
93.0:10.0
93.4
91.02
88.3:224
91.2--0.3
93.5=0.6
92.0=t0.3
93,9 0.9
51.3=00.0
56.3+:0.5
51.5+0.4
55.7++099
52.1:00.5
53.8=0.5
53.5::0.55
56.5
1250 +750
68.9110.1
82.2:10.3
7.22+:1.2
77.01110
77.0:10.7
0..1-0.3
77.8:10.9
80.2 0.7
-1000
1250 - 500
94.1+0.11
9844004
93.0+0.3
89.5422
92.7=0.11
94.3=0.4
94.0:10.2
9..0=0.7
1500 -5500
50.8.0.11
56.3.10.5
50.6..0.5
56.2.10.6
$1881022
0.5
51.8.004
54.0 2.2
1500 7500
63.3....1
82.2403
6.2
78.21116
75.0-006
80.0=0.2
76.4:10888
8.
1500 - 1000
83.9+0.11
89.2
82.8+0.7
84.7
88.4204
874=0.4
88.9+00.
88.5=0.8
+220
90.6:0055
88.7:2233
93.0:1033
to reduce the sample dimensionality from 5000 to 5. The NN has two hidden layers of dimensions 10 and (the encoding dimension). For each task, we train the NN as a classifier on the source data with cross entropy loss. We then cut the output layer and use the trained NN to encode both the source and target data into dimensions. This dimension reduction is the input data of all the algorithms on the benchmark and we trained all the competitors this data. The results are shown in Table 6.
Table 3: Amazon dataset, unsupervised. Tasks B: books, D: Dvd, E: Electronics, K: kitchen
Task
Source
Target
DANN
SA
KMM
OT
SHOT
KRDA
19.9110.1
79.420.1
79.910.1
79.9 1 0.1
79.9=0.1
79.710.1
7..7=0.6
80.0:10.2
E
69 69.2::022
72.7=0.0
69.9+0.
73.0 = 0.1
71.8=0.1
73.0=20.1
72.6=0.7
73.0::0.1
75.7+-0.2
76.220.1
7584:022
76.3=0.1
76.2::0.1
76.2=002
76.2:10.11
D
+ B
75.1.10.1
75.61.0.1
75.1
75.3 1 0.1
75.5
75.2.10..1
75.5 0.1
75.3.10.1
71.01:033
74.620.1
71.11:077
74.3
72.7-0.2
74.6.10.1
7.8
K 74.910..1
77.310.11
74.81..5
77.5
1 0.1
76.2:1.11
77.5
77.0
76.3:10.2
R 70.7.:0.1
71.5.00.0
70.6:10.4
71.2 1 0.2
70.9-0.2
71.1:10.1
71.3-10.3
70.9:10.2
E 72.3+00.1
73.620..
72.1
73.1 0.2
73.3=0.4
72.7=:0.1
7..8=0.3
72.6:10.5
B
K
86.11:0.1
86.2=0.1
85.5+:0.5
85.8 - 0.2
83.5-0.3
86.0-20.1
85.6=0.6
85.2++1.0
7121002
K
++D
D
70.810.11
72.8.00.0
70.7.1111
72.7 1 0.3
70.6
72..10.2
70.6
72.810.7
E
84.1000
84.420.0
84.0:022
- 0.0
84.2-0.0
84.3-00.0
84.2-0.1
84.1
7 CONCLUSION
In this paper, we presented a novel transfer learning framework that exploits recent advances in density estimation techniques in order transfer samples from source domain a target domain with a distribution shift using Knothe-Rosen transfer performed by KRDA is the one dimensional conditional quantile distributions from the source in the target domain space, using autoregressive setup. We showed that KRDA is state of the art on small to moderate dimensional tasks, often outperforming competitors especially in the case where there are few data samples.
transport. The property that is invariant by the
Future work includes extending KRDA the semi-supervised domain adaptation where a few labels are present in the target data. Another interesting research direction is to use other modern density estimators such as normalizing flows or variational autoencoders, in place of autoregressive mixture density nets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Youness Arjoune and Saleh Faruque. Artificial intelligence for 5g wireless systems: Opportumities. challenges, and future research direction. In 2020 10th Annual Computing and Communication Workshop and Conference (CCWC), pp. 1023-1028. IEFF, 2020.
Pierre Baldi, Kyle Cranmer, Taylor Faucett, Peter Sadowski, and Daniel Whiteson. Parameterized neural networks for high-energy physics. The European Physical Journal C, 76, 05 2016.
Chafika Benzaid and Tarik Taleb, Ai-driven zero touch network and service management in 5g and beyond: Challenges and research directions. IEEE Network, 34(2):186-19,, 2020.
Christopher M. Bishop. Mixture density networks. Technical report, Birmingham, 1994.
Guillaume Carlier, Alfred Galichon, and Filippo Santambrogio. From knothe's transport to brenier's map and continuation method for optimal transport. SIAM J. Math. Anal., 41:2554-25777 2576, 2009.
N. Courty, R. Flamary, D. Tuia, and A. Rakotomamoniy. Optimal transport for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(9): (9ttliiennee33))11855555 1865, Sep. 2017. ISSN 1939-3539.
Donahue, Hoffman, E. Rodner, K. Saenko, and T. Darrell. Semi-supervised domain adaptation with instance constraints. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 668 675, 2013.
Basura Fernando, Amaury Habrard, Mare Sebban, and Tinne Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In Proceedings of the IEEE international conference computer vision, pp. 2960-2967, 2013.
Yu Fu, Sen Wang. Cheng--Xang Wang, Xuemin Hong, and Stephen McLaughlin. Artificial intelli- gence manage network traffic of 5g wireless networks. IEEE Network, 32(6):58-64, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversari training of neural networks. Journal of Machine Learning Research, 17(59):1-35, 2016.
Pascal Germain, Amaury Habrard, François Laviolette, and Emilie Morvant. A pac-bayesian approach for domain adaptation with specialization to linear classifiers. In Sanjoy Dasgupta and David McAllester (eds.), ), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 738-746, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2066-2073. IEEE, 2012.
Mingming Gong, Kun Zhang, Tongliang Liu, Dacheng Tao, Clark Glymour, and Bernhard Schölkopf. Domain adaptation with conditional transferable components. In International conference on machine learning, pp. 2839-2848, 2016.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problem. In Advances in neural information processing systems, pp. 513-520, 2007.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. volume 97 of Proceedings of Machine Learning Research, pp. 2730, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
10

---

Under review as a conference paper at ICLR 2022
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex J Smola. Cor- recting sample selection bias by unlabeled data. In Advances in neural information processing systems. pp. 601-608, 2007.
Mingsheng Long Junguang Jiang, Bo Fu. leanffereeannnn--ibbbaae https://gittubb.co thuml/ Tww.://raanseerreeeee Lew.-eere-eaaainn--eeeee
2020.
Balázs Kégl, Gabriel Hurtado, and Albert Thomas. Model-based micro-data reinforcement learning: what the crucial model properties and which model to choose? In International Conference Learning Representations, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Herbert Knothe. Contributions to the theory of convex bodies. Michigan Mathematical Journal, 4(1): 39 52, 1957.
Wouter Marco Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence, 2019.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. volume 15 of Proceedings of Machine Learning Research, pp. 29-37, Fort Lauderdale, FI., USA, 11-13 Apr 2011. JMLR Workshop and Conference Proceedings.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9641-9650, 2020a.
Xuhong Li, Yves Grandvalet, Franck Davoine, Jingchun Cheng, Yin Cui, Hang Zhang, Serge Belongie, Yi-Hsuan Tsai, and Ming-Hsuan Yang. Transfer learning in computer vision tasks: Remember where you come from. Image and Vision Computing, 93:103853, 2020b. ISSN 0262-8856.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference Machine Learning, pp.
PMLR, 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97-105. PMLR, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 220222082222 2217, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR.
Mingsheng Long, ZHANGJIE CAO, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 31, pp. 1640-1650. Curran Associates, Inc., 2018.
Aditya Malte and Pratik Ratadiya. Evolution of transfer learning in natural language processing, 2019.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton den Hengel. Image- based rec- ommendations on styles and substitutes. SIGIR '15, pp. 43-52, New York, NY, USA, 2015. Association for Computing Machinery.
Boris Muzellec and Marco Cuturi. Subspace detours: Building transport plans that are optimal on subspace projections. In NeurIPS, 2019.
11

---

Under review as a conference paper at ICLR 2022
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. volume 48 of Proceedings of Machine Learning Research, pp. 1747-1756, New York, New York, USA, 20-22/ Jun 2016. PMLR.
S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):135--135) 2010.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210 2010.
Sinno Jialin Pan, Ivor W. Tsang, James T. Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE Transactions on Neural Networks, 22(2):199-210, 2011. doi: 10.1 109/ TNN.2010.2091281.
François Pitié, Anil C. Kokaram, and Rozenn Dahyot. Automated colour grading using colour distribution transfer. Computer Vision and Image Understanding, 107(1):123-137, 2007. ISSN 1077-3142. Special issue on color image processing.
Joaquin Quionero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D. Lawrence, Dataset Shift Machine Learning. The MIT Press, 2009. ISBN 0262170051.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07-09 Jul 2015. PMLR.
Murray Rosenblatt. Remarks on a Multivariate Transformation. The Annals of Mathematical Statistics, 23(3):470 472, 1952.
Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18, 2019.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the I Ith European Conference on Computer Vision: Part IV, ECCV'10, pp. 213-226, Berlin, Heidelberg, 2010. Springer- Springer-Verlag. ISBN 364215560X.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3723-3732, 2018.
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In David van Dyk and Max Welling (eds.), Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics, volume 5 of Proceedings of Machine Learning Reseurch, pp. 448 455, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR.
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94 2015.
A. Sharma, A. Kumar, H. Daume, and D. W. Jacobs. Generalized multiview analysis: discriminative latent space. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2160- 2167, 2012. doi: 10. 00.119/CVPR.2202..44777
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint 1r0ivv1802.08733 2018.
B. Uria. I. Murray, and H. Larochelle. Rnade: The real-valued neural autoregressive density-estimator. In NIPS, 2013.
Cédric Villani. Optimal transport - Old and new, volume 338, pp. xxii+973. 01 2008. doi: 01.1077/978-3-54--7100
12

---

Under review as a conference paper at ICLR 2022
L. Wasserman. All of Statistics: A Concise Course in Statistical Inference. Springer Texts in Statistics. Springer New York, 2004. ISBN 9780387217369.
C. Zhang, H. Zhang, J. Qiao, D. Yuan, and M. Zhang. Deep transfer learning for intelligent cellular traffic prediction based cross-domain big data. IEEE Journal on Selected Areas in Communications, 17(6):1389-1401 2019. doi: 10. 2.110///....211..99900
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. arXiv preprint 19Xiv:9/11.2000 2019.
13

---

Under review as a conference paper at ICLR 2022
Knothe-Rosenblat transport for Unsupervised Domain Adaptation
Supplementary material
A EXPERIMENTS
A.1 INTER-TWINNING MOONS
Table 4: Inter-twinning moons unsupervised
Task
Source
Target
CDAN
DAN
JAN
TCA
KRDA
10º
100:100
100+00
100:10
50.0=0.1
100::0
100.0::0
100-1-
20
99.910.11
10010
99.8=0.2
48.5=33.4
1001:
100.0
100:10
30
96.61:0.4
100:0
97.8=2.1
56.1
99.7:00.7
100.0=:0
100:10
40
73.1+2.50
100+:
80.4
50.0
93.247.9
98.010.8
98.4
50"
41.5++2.0
100+00
47.4+24.5
50.0+0.11
89.8+4..3
80.1+2.00
98.4++1.5
60'
28.7+:0.6
100+:0
30.1=5.0
50.0=0.1
87.4+4.33
44.8+=4.7
90.5+2288
70"
23.3+004
100+0
24.9=117
49,9
62.24:34.9
24.9+00.6
84.4+0.8
80
20.411.77
10010
26.71100.
50.0.1.11
49.91122.6
20.510..
81,2.13,4
90
18.110.3
10010
17.2.0.7
50.0.1..1
27.9114.1
17.810.4
71.1
A.2
HEPMASS
Table 5: HEPMASS unsupervised domain adaptation
solution
Source
Target
CDAN
DAN
JAN
TCA
KRDA
500 -5700
68.740.1
8..2-0.3
56.044.00
51.3
58.2+11.9
67.2400.4
75.6-+2.3
500 -> 1000
67.2+.0.1
89.2-0.4
49.245.6
51.4+-.11
56.143.8
71.7+-0.4
80.34:3.1
500 1250
62.6+00.3
93.3-0.3
53.748.8
48.9+-2.7
54.9++2.3
77.9+0.3
82.9
500 1500
58.010.2
95.4004
56.319.1
52.810.3
54.3
75.910.5
80.016.1
750 5500
52.610.11
56.3-0.5
54,210.5
50.410.8
54.911.99
53.710.2
55.7111.1
750 - 1000
86.510.11
89.2-0.4
82.610.4
49.010.2
81.811.7
84.510.2
87.710.7
750 1250
87.7+0.0
95.4=0.4
82.5+3.1
52.1=7.4
77.8+3.9
85.7+0.1
92.3++1.3
750 1500
87.7+0.00
95.4=0.4
74.8+7.9
56.1
76.6+2.1
85.7+0.1
92,8+1.6
1000 +500
51.9+0.0
56.3=0.5
52.0+0.5
50.0+0.0
51.2+00.
53.1-0.11
53.2+117
1000 +500
77.110.1
82.2-0.3
74.8.11.2
51.510.0
76.2-1111
73.6.10.2
80.7-10.
1000 +1250
92.6.10.0
93.3-0.3
91.210055
$7.2117.3
88.8.10.4
91.1110.1
92.5.10.3
1000 +1500
93.0.10.0
95.4-0.4
90.6.10.6
61.417700
88.6111.3
91.710.11
93.9110.9
1250 +500
51.3+0000
56.3=0.5
$1.310.3
50.020.0
52.120.6
51.7+0.1
1..
1250 +750
68.920.1
82.-=033
69,81009
54.8112.6
768820.7
69.100.2
80,210.7
1250 +1000
87.710.1
892-00.4
85.210.3
69.9119.1
85.310.77
86.510.2
88.5
1250 +1500
94.1+0.1
95.4-0.4
93.2+00.4
72.6+-33.8
93.3+00.7
93.0+-0.1
94.0+-0.7
1500 +500
50.8+0.11
56.3-0.5
50.2+0.4
50.0+0.2
51.3+00.6
51.3+0.1
54.0+2.2
1500 +750
63.3+0.1
82.2-033
62.140.9
53.1+10.3
73.9+0.66
66.0+0.2
80.4+1.2
1500 1000
83,910.1
892--04
82.3114
75.8111.
85.911..
83,410.2
88,510,88
1500 1250
92.910.1
93.3-033
91.0.10.6
84.1118.88
91.311022
91.910.3
92.4-10.8
A.3 AMAZON
14

---

Under review as a conference paper at ICLR 2022
Table 6: Amazon dataset, unsupervised. Tasks are B: books, D: Dvd, E: Electronics, K: kitchen
solution
Source
Target
CDAN
DAN
JAN
TCA
KRDA
B
79.910.1
79.410.1
79.7:00.6
63.1
19.5
79.910.1
80.0
B +
69.2100.2
72.710.0
64.518.5
59.6210.88
70.9:1.44
69.910.2
73.0-10.1
B + K
75.7+002
76.2+001
75.8++0.1
55.2+11.6
70.9+11.5
75.8+0..1
76,2-0.1
D B 75.1+0.1
75.6+0011
75.3+00.2
69.3-9.1
70.2+10.9
75.0++0.0
75.3-0011
D
71.0+0.3
74.6+0.11
66.9+.9.3
68.0-9.9
67.0+9..
71.2+002
7..4=0.2
- K
74.910.1
77.310.1
75.1100.2
60.2114.0
62.1113.0
75.3100.
76.3
E +
70.710.11
71.5110.0
70.7100.4
58.3
58.6.111.5
71.010.1
70.9
E G-D D
72.310.1
73.6.10.0
71.411..
66.5.19.8
63.811126
72.110.1
7..6
E K
86.1
86.2-0.1
85.5+004
83.5=5.4
64.4=11.6
86.0
85.2=110
K
71.210.2
71.410.1
71.2+:0.2
69.7-3.6
62.6111.6
71.2:10.1
69.4=0.3
K D
70.8+0.1
72.8+0.0
70.5+0.4
62.4=11.7
71.0
70.7+0.0
72.8=0.7
K - E
84.0+10.0
84.4+10.0
84.0+L0.2
84.1-10.2
77.24:1.4
83.940..1
84.1-10.1
15


Reviews:
[{"review_id": "j0XkBEE48l_", "reviewer": "Reviewer_HHVi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n- The paper is the first work that applies Knothe-Rosenblatt to the unsupervised domain adaptation problem. So, originality is an advantage of the paper. \n- The paper is well-written, easy to follow. Parts of the paper are well organized.\n- The author cleverly exploits the nature of Knothe-Rosenblatt that is well-fitted to the density estimation tool like an autoregressive model.\n- Literature is cited quite sufficiently.\n**Weaknesses**:\nI think that the paper can become a much stronger paper if the author can address the following weaknesses:\n- The limitation of autoregressive models is also the limitation of KRDA as mentioned by the authors,  so the KRDA is not scalable in terms of the number of dimensions. Hence, I suggest that the authors should propose an additional \"deep\" KRDA method that uses a neural net as a dimensional reduction method and feature extraction method (as authors did on Amazon datasets with the difference that the neural net is also trained with an adaptation loss ), and then apply KRDA on the latent space. Optimal transport losses are mostly utilized on this approach [1] - OT, [2] - unbalanced OT, [3] - partial OT. Also, it is necessary to compare deep KRDA with these models on some famous datasets such as SVHN, MNIST, UPS, Office-home, etc. For me, I consider KRDA is a deep domain adaptation method since it uses RNADE. So, comparison with more deep DA methods is needed.\n\n[1] DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptation\n\n[2] Unbalanced minibatch Optimal Transport; applications to Domain Adaptation\n\n[3] Improving Mini-batch Optimal Transport via Partial Transportation\n\n- I am not sure about the benefit of KRDA over the OT-based DA. For example, JDOT does not need to smooth the source and the target density, namely, JDOT only needs to deal with empirical measures.  Since the performance on the Amazon dataset of KRDA is not superior to JDOT, I suggest the authors should do a speed comparison between methods.  I believe that training RNADE is also expensive.\n\n- It seems that KRDA needs invertible CDFs for the conditional distributions of RNADE (e.g. Mixture of Gaussians in the paper). I wonder if the authors can discuss families of distributions that can be utilized here and compare their performance. Also, the authors should motivate the usage of mixtures of Gaussians here.\n\n- The authors have not published the code yet. So, I could not run any quick verification of the experiments. I encourage the author to publish the code in the discussion phase.\n\n**Comments and Questions:**\n- Could the authors make a discussion between Knothe-Rosenblatt transportation and sliced optimal transport. I know that the difference is that sliced OT is based on random projections while Knothe-Rosenblatt is based on conditional distributions. However, I think a discussion is a good contribution here.\n- RNADE is trained with the maximum likelihood estimator (KL divergence). Could Knothe-Rosenblatt transportation be transformed into a notation of divergence between the source measure and the target measure?", "summary_of_the_paper": "The paper proposes Knothe-Rosenblatt Domain Adaptation (KRDA) for unsupervised domain adaptation with covariate shift. The key idea of the paper is to use an autoregressive model (RNADE) to estimate the density of source distribution and target distribution, then transport the estimated source density to the estimated target density by Knothe-Rosenblatt transportation. In particular, the transportation maps are constructed on each conditional density that is obtained from fitted autoregressive models. The authors evaluated the proposed model on synthetic data (mixture of Gaussians, Inter-twinning moons), and real data (Hepmass, Amazon dataset). ", "summary_of_the_review": "I consider the paper is a methodology paper, so I believe that more experiments are needed e.g. comparison with deep DA methods including OT-based deep DA, computational speed comparison between these methods. Also, some discussion is needed as mentioned in the review part.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cf3OKHgscZL", "reviewer": "Reviewer_KT59", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\nAs far as I can check, one of the main originality of the paper is to use an auto-regressive model to estimate the conditional marginals allowing a relatively easy application of the Knothe-Rosenblatt transport. Another point is the application to a dataset in high-energy physics with interesting results.\n\nWeaknesses\nThe idea of using optimal tranport based approaches is not really novel and many approaches exist. This paper proposes a novel strategy but there is no real strong point that makes this contribution significantly different from others, in particular with respect to other optimal transport strategies.\nThe contribution is mainly algorithmic, there is no theoretical results supporting the method (convergence properties, generalization bounds, ...).\nThe experimental evaluation uses mainly existing competitors which is good, but since the method is in the line of optimal transport (OT) strategies. A deeper comparison with other OT baselines for domain adaptation (different regularization or other strategies: JDOT [Courty et al., NeurIPS 2017], COOT [Vayer et al., NeurIPS 2020], ...) would reinforce the interest of the approach and possibly exhibits interesting properties. The expressiveness with respect to other OT methods should be developped and better compared.\nI do not find the experimental setup sufficiently clear, a detailed paragraph (in supplementary material). It does not seem that the paper tunes the competitors according to their classic usage in their original paper, the architectures seem limited both in terms of number of layers and in terms of the number of neurons per layer, this is not clear that the experimental setting is fair for the other competitors. \nThere is no benchmarks in vision, which limits the comparison with existing results in the State of the Art.\nThe authors claim that their approach is better for small datasets, why not, but this should be better analyzed: using more datasets (going to the few-shot learning setting?), tune other competitors adequately for the considered problems, maybe deep learning solutions are not the best here (even though, they should be considered as well of course.\nRunning time indicators should be useful to justify gains in terms of computational efficiency.\n", "summary_of_the_paper": "This paper proposes to tackle unsupervised domain adaptation by following a classic trend aiming at finding an alignment between source and target domains. More specifically, the method uses a Knothe-Rosenblatt transport approach which applies a one-dimensional optimal transport to all conditional marginals of one distribution into another. The method uses an autoregressive density estimation from the RNADE algorithm based on a Gaussian Mixture strategy. The experimental evaluation compares the approach and other classic domain adaptation benchmarks on synthetic data, on the Amazon benchmark and a benchmark in higgh-energy physics.\n", "summary_of_the_review": "The paper presents a novel idea consisting in using an auto-regressive approach to estimate the condition marginal densities which allows the to use a more efficient application of the Knothe-Rosenblatt optimal transport (OT). This is new and interesting.\nHowever, the comparison both in terms expressiveness and experiments with respect to other OT is rather limited and should be extended.\nThe experimental setup is not completely clear and thus not clearly convincing.\nSome benchmarks in vision are not considered.\nThere is no theoretical results supporting the paper.\n\nOverall, there is the beginning of an interesting idea but that is not enough developed for ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aEpuwE_oSd", "reviewer": "Reviewer_kupY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n(1)\tA new KRDA method is proposed. \n\n(2)\tBoth synthetic and real-world experiments are done. \n\nThe overall idea is estimating the source and target distributions and then learn the matching through transport. The idea is simple and not new. Here are some detailed comments:\n\n(1)\t The related works are out-of-date, especially for subspace based methods. Please consider the more recent works, e.g., [ref1] and [ref2].\n\n[ref1] Visual Domain Adaptation with Manifold Embedded Distribution Alignment\n[ref2] Subdomain adaptation with manifolds discrepancy alignment\n\n(2)\tOnly using the last linear layer to capture the domain difference is some weak. You may consider the structure in DAN [ref3] where the first several layers are generic and thus fixed, the medium layers are fine-tuned, and the last several layers are leaned to be aligned. \n\n[ref3] Learning Transferable Features with Deep Adaptation Networks \n\n(3)\tThe superiority of KRDA on small data with small dimension is highlighted. In this sense, it is more preferred to compare with those subspace transfer methods. However, the current baselines are not state-of-the-art, which makes the empirical results less convincing. Moreover, some benchmark datasets, e.g., office 31 which has small data size, are expected to be tested.\n", "summary_of_the_paper": "In this paper, the authors propose a Knothe-Rosenblatt Domain Adaptation (KRDA) method. Specifically, it exploits autoregressive model using a mixture of Gaussians to accurately model the different sources, and then takes advantage of the triangularity of the autoregressive models to build an explicit mapping of the source samples into the target domain. Experiments on both synthetic and real-world datasets are done to verify KRDA.", "summary_of_the_review": "See comments above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "UZaQnMUc9pk", "reviewer": "Reviewer_paqc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Strengths:\n\nThe usage of density estimation by autoregressive model for Knothe-Rosenblatt transport for unsupervised domain adaptive obtains good performances.\n\n+ Weakness:\n\nIt seems that the contribution is quite incremental: (i) using optimal transport to transfer source to target for domain adaptation is not new (ii) Although the usage of density estimation by autoregressive model for Knothe-Rosenblatt transport is novel, its motivation is unclear -- e.g., about the advantage of the proposed approach compared with other optimal transport-based approaches for domain adaptation.\n\n+ Detailed comments:\n\n1. What are the motivations for the proposed method (e.g., using density estimation by autoregressive model for Knothe-Rosenblatt transport) compared with other optimal transport-based approaches for domain adaptation? (e.g., standard optimal transport, or sliced-optimal transport for its fast computation and its variants)\n\n2. The choice of autoregressive modeling for density estimation is not well-motivated. What's kind of applications suitable for this density estimation? (In which cases, this approach may not be a good choice). I think a discussion about the choice of autogressive modeling for density estimation will help readers.\n\n3. It seems that the proposed approach is closely related with the sliced-Wasserstein. I think it is a good baseline for the proposed approach (simple + fast computation) -- autogressive modeling density estimation is also based on 1d-density estimation. \n\n4. For experiments, why does the proposed approach obtain better results than the OT approach? The authors should discuss the experimental results in detail.\n\n5. For experiments, the setting may be limited when only binary classification is considered for both source and target. As illustrated in Figurer 2, when the source and target have different numbers of classes, is it possible to use the proposed approach?\n\n\n\n", "summary_of_the_paper": "The authors consider the unsupervised domain adaptation (UDA) by using the Knothe-Rosenblatt transport. The authors propose to estimate the density of both source and target data via autoregressive modeling (i.e., decompose the d-dimensional density into d one-dimensional conditional densities), then apply Knothe-Rosenblatt transport to obtain the transfer map from the source to target for UDA. \n\n\n", "summary_of_the_review": "It seems that the contribution is quite incremental. The usage of optimal transport-based approaches for domain adaptation is not new. The proposed OT-based approach is not well-motivated, its advantage is not cleared yet (although it has better performances, there are no discussions why such results are expected)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to address the problem of domain adaption using Knothe-Rosenblatt transport withe the method denoted as KRDA . The main idea is to perform density estimation of the different distributions with mixture of Gaussians and then estimate a  an explicit mapping between the distribution using  Knothe-Rosenblatt. Experiments show that the proposed method works well on toy and real life datasets.

 The paper had low score during the reviews (3,3,3,3). While the reviewers appreciated the idea, they felt that the originality of the method is not well justified compared to a number of existing UDA approaches using OT. Also the reviewers noted several important references missing and that should also be compared during the numerical experiments. A discussion about the limits of the method in high dimension would also be very interesting.

The authors did not provide a reply to the reviewers' comments so their opinion stayed t same during the discussion. The paper is then rejected and the AC strongly suggests that the authors take into account the numerous comments from the reviewers before re-submitting ton a new venue.
Decision: Reject