Learning Dictionaries over Datasets through Wasserstein Barycenters
Authors: Eduardo Fernandes Montesuma
Fred Maurice Ngole Mboula
Antoine Souloumiac
Venue: Submitted to ICLR 2023 (2023)
Keywords: Dictionary Learning
Optimal Transport
Domain Adaptation
Manifold Learning

Abstract: Dictionary learning consists of trying to represent objects in terms of basic elements (atoms) weighted by an importance factor (representation). Non-linear dictionary learning using optimal transport as a metric has been previously studied for normalized non-negative data on a fixed grid. We propose a new framework by using Wasserstein Dictionary Learning on datasets understood as empirical distributions. We leverage Wasserstein barycenters for learning a dictionary of virtual datasets and embeddings in a simplex. We apply our method for unsupervised domain adaptation, improving the state-of-the-art over 1.96% and 2.70%, respectively, and manifold learning of Gaussian distributions and color histograms.

Under review as a conference paper at ICLR 2023
LEARNING DICTIONARIES OVER DATASETS THROUGH WASSERSTEIN BARYCENTERS
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised Domain Adaptation is important machine learning problem that aims at mitigating data distribution shifts, when transferring knowledge from one labeled domain to another similar and unlaheled domain. Optimal transport has been shown in previous works to be a powerful tool for comparing and matching empirical distributions. As such, we propose novel approach for Multi-Source Domain adaptation which consists on learning Wasserstein Dictionary of labeled empirical distributions, as a means of interpolating distributional shift across sev- eral related domains, and inferring labels on the target domain. We evaluate this method on Caltech- Offecce-fffiee/eeeee 10 and Office 31 benchmarks, where we show that our method improves the state-of-the- art of 1.96% and 2.70% respectively. We pro- vide further insight on our dictionary, exploring how interpolations of atoms pro- vide useful predictors for target domain data, and how it can be used to study the geometry of data distributions. Our framework opens interesting perspectives for fitting and generating datasets based on learned probability distributions.
INTRODUCTION
Dictionary Learning (DiL) is representation learning technique that seeks express of vectors in Rd as the linear weighted combinations of elementary elements named atoms. When vectors represent histograms, this problem is known as Nonnegative Matrix Factorization (NMF). Optimal Transport (OT) previously contributed to this case, either through metric over histograms (Rolet et al., 2016) or by defining a non-linear way of aggregating atoms (Schmitz et al., 2018) through Wasserstein barycenters (Agueh & Carlier, 2011).
In paraliel, different problems in Machine Learning (ML) can be analyzed through a probabilistic view, e.g., generative modeling (Goodfellow et al., 2014) and Domain Adaptation (DA) (Pan & Yang, 2009). For instance, Multi-Source Domain Adaptation (MSDA), one wants to adapt data from heterogeneous domains or datasets to Minimization (ERM) principle cannot be correctly applied due to the non-i.i.d. character of the data. However, assume that the domain shifts have regularities that can be leamned and leveraged for MSDA. Thus, in this paper, we take novel approach to MSDA, using distributional DiL: we learn dictionary of empirical distributions. As such, we reconstruct domains using interpolations in the Wasserstein space, also known as Wasserstein barycenters. As we explore in section 3, this offers principled framework for MSDA.
setting. In this case, the celebrated Empirical Risk
We take inspiration from the works of Bonneel et al. (2016) and Schmitz et al. (2018) for defining our novel DiL framework. Indeed, these authors consider DiL. over histograms, while we propose DiL over datasets, understood point clouds, which enables its application to DA. We summarize our contributions as follows,
Dictionary Learning. To the best of our knowledge, we are the first to propose a DiL problem over point clouds.
Empirical Distributions Embedding. As a by-product, we get embeddings of the DiL datasets their barycentric coordinates w.n.t the dictionary. We build this new rep- resentation to define a (semi-)metric called Wasserstein Embedding Distance (WED). We explore the WED theoretically (theorems 3.2 and 3.3) and in experiments (section 4.2).

---

Under review as a conference paper at ICLR 2023
Domain Adaptation. We propose two novel ways for performing MSDA. The first relies on reconstructing labeled samples in the support of the target distribution. The second relies weighting predictors learned each atom, thus defining new classifier that works on the target domain. We offer theoretical justification for both methods (section 3.1). We further explore, in section 4.1.3, general interpolations in the latent space of our dictionary.
Notation. We denote as AN = {a E RY En a, 1} the N probability simplex. We consider up i.idd samples ( = {x, pe E Ropxd from an unknown distribution P: x; (P) P.
The samples X(P) yield an empirical approximation, P of P as a sum of delta Diracs, P E P:O2(0) with ps = 1//p unless stated otherwise. Similarly, Q = E q5 with = I/na. is an empirical approximation of probability distribution Q on R2 Additionally, each x; (P) may have label v! {1,... ,n. or Y An. for its one-hot encoding. In this
case P Ei= pio (x!?) m! is an empirical approximation for the joint P(X,Y)
Paper Structure. Section 2 covers brief introductions OT, DA, and DiL., Section 3 introduces our view on dictionary learning. Section 4 covers experiments in manifold learning of distributions and domain adaptation. Section 5 presents our conclusions.
2
PRELIMINARIES
2.1
OPTIMAL TRANSPORT
In this section, we focus the computational treatment of OT (Peyré et al., 2019), which predom- inantly relies on empirical approximations of distributions. There are two discretization strategies. First, known as Eulerian discretization, one seeks to bin R into fixed grid {x: (P) Ji=1 that p; corresponds to how many samples are assigned to the i-th bin. The second, known as Lagrangian discretization, assume x; Schmitz et al. (2018), we use the Lagrangian discretization.
)
(P) i.i.d. according to P. Henceforth, contrary to Bonneel et al. (2016);
For P, Q in the formulation of Kantorovich (1942) OT seeks a transport plan и RRPxxe, where (P)
(Q) x is required to preserve mass, that Tif represents how much mass transported from x; to x, is, Eie Tij 9j and D TiJ Pt, (p, q). the set of bi-stochastic matrices with marginals p E Anp, E Ang Therefore,
Way no ΣΣ ==
π = OT(P,, q. C) argmin
CisTi.= (C,T)
MED (p.9)
is the OT problem between P5 and Q. In Equation 1, Ci. = c(x (P) (2)) is called ground-cost
matrix. When c is a distance, OT defines a distance between distributions (Peyré et al., 2019) called Wasserstein distance, W.(P, Q) = (C,**), For P = {Px) and a E Ax. the Wasserstein barycenter (Agueh & Carlier, 201 1) is solution
K
B" = B(a; = inf
ag (Pk,
(2)
Henceforth we call B(::)) barycentric operator. For empirical Pe, estimating B** is done through the fixed-point iterations of Álvarez-Esteban al. (2016),
K
(C* k. ||x (PW) x; 1 (pp.,. b. C2); X BB)
d x
(3)
until convergence. this case b E A, (e.g., by = 1/n). We further discuss iterations in equation in appendix A.
2

---

Under review as a conference paper at ICLR 2023
2.2 EMPIRICAL RISK MINIMIZATION AND DOMAIN ADAPTATION
In traditional ML, the classification problem can be formalized through the ERM principle of Vapnik (1991), which consists in finding h", which verifies
ht argmin (P( 1 Σ ((((x))) hEH
where L is loss function and h Rd - {1,.... mej is classifier, chosen among a family H. Under the i.i.d. hypothesis between training and real application data, the classification error expectation quantified as
(p ho) Ex-p (xxxx((((xxx,,...))))))))())>>)) ho()))
where hg is the unknown ground-trutt labeling function. Given that enough samples are available R. and Rp are close with high probability (Redko et al., 2020, Theorem 1).
Nonetheless, the i.i.d. assumption is restrictive since real-world data may be acquired under different regimes. In this case, train and test data may follow different distributions (Quinonero-Camm Candela et al. (2008)). DA (Kouw& & Loog, 2019) is a framework for non non-standard cases.
Following Pan & Yang (2009), a domain D = (X, Q())) is a pair of a feature space X (e.g., Rd) and its distribution (X).. ) In DA, one has Ds * DDr due to different distributions, that is, Qs o Qr- The goal is thus adapting a classifier learned with labeled data from Ds, using unlabeled data from Ne
Dr. When Ng > 1 sources are available, {Ds, ) 1=1 (resp. Qs.), one has MSDA. DA can be further divided into shallow and deep DA. the first case, one leverages feature extractors (e.g., pre-trained convolutional layers). In contrast, in the second, one uses unlabeled target data during training for learning features invariant to distributional shifts.
OT has contributed to DA in various ways. For instance, in the seminal works of Courty et al. (2016), the authors proposed transporting source domain samples using (Q+) diag (ms) - (Qr). In this sense (Qs) Qr, thus their method generates labeled data on the target domain. OT has been applied for MSDA as well. For instance, Montesuma & Mboula (2021a;b) proposes to aggregate Ns {Qs. t=1
) using the Wasserstein barycenter B, then transporting B to Qr. Otherwise, Turrisi et al. (2022) estimate domain importance coefficients a for weighting source domain distributions.
2.3 DICTIONARY LEARNING
DiL is representation learning technique that tries to express collection of N vectors {xx) ) Xg gd through set of K atoms P [p*]K_> P& Rd and N weights A {0t))=11 E R Mathematically, DiL corresponds to,
N Σ
1
(P*.A*) = argmin P.A N
a5) + Aplp(P),
(5)
where L is suitable loss, whereas QA and 2p regularization terms of representations and atoms. OT has previously contributed to Dil., when the vectors are histograms, that is X, E Ag. In this sense, these contributions assume Fulerian discretization paradigm. For instance Rolet al. (2016) considered using the Sinkhorn divergence (Cuturi, 2013) as a loss in 5, and Schmitz et al. (2018) proposed substituting pd ae for the Wasserstein barycenter.
DATASET DICTIONARY LEARNING
We seek to learn dictionary over point clouds. As such, Q= (Q+) N where each Q has support [x] (R)) 1- or ((x(2)) .y; (80)
corresponding to whether Qe is labeled or not. In this setting,
we rewrite equation as,
(P", =armmin L(P,
B(ank,Baan;P))
(6)
P.A
3

---

Under review as a conference paper at ICLR 2023
where p = (Px is our set of atoms, and A {au) with ag AK. aie& denotes how much Qe is composed by Ps in the Wasserstein distance sense. Each Ps: a points cloud parametrized by its support X (Px) and labels ((s) when desired. Hence, use the fixed-point iterations of Álvarez- Esteban et al. (2016), and differentiate (( P) w.r.t. a and x; (P)) using the Envelope theorem (Bert- sekas, 1997). More this matter is discussed in appendix A.
In addition, optimizing 6 over entire datasets might be untractable due the time and memory com- plexity of We. which are, given samples, O(n log n) and On²)), respectively. We thus employ mini-batch OT (Fatras et al., 2021). For M mini-batches of size nig < n, this reduces complexity to ( log m43) and O respectively.
As the Qe can be either marginal over fea- tures, or feature-label joint distributions in equation 6, the atoms P& are joint features- labels distributions, and the ground-cost must take into account both labels and fea- tures, We propose using,
Algorithm 1 Dataset Dictionary Learning (DaDiL) learning loop.
Require: Q = {Â+)/ number iterations Niters num- ber of atoms K, number of batches M, batch size nis
Ci, = |x P) x,
1011
1: Initialize x! (PP) (O L.). a. ~N((,
+ (P) - 11
2: for it = 1... .Nater do batch = 1,... M do
3:
for 3 = max,, ||x (P) x
x 12. This determine (B)
4: for 8=
N do Vk, sample ( {x (P))
ground-cost allows using iterations 3 and to formally justify the label propagation approach of Redko et al. (2019).
5:
ii=l
6:
sample (2r) = {x) (2)) ma
7:
change variables ar softmax(ar)
8:
calculate X (Bp) B(ac; P) end for
9:
K
10:
L L= (1/N) ΣX ΣN We(Br, Qi)
(B)) axdiaa(()) - (Pb),
(8)
II:
Vk, i, update x; using aL/axx (Px) (Pk)
1=1
12:
Vf, update as using /L//a..
13: end for
for labeling the samples in ( Simi- larly Bonneel et al. (2016) and Schmitz et al. (2018), we optimize w.r.t A
14:
end for
Ensure: Dictionary Pd and weights A²"
(ar.)) N then perform change of variables oz softmax(ax). The overall procedure is shown in algorithm 1, which implemented using Pytorch (automatic differentiation) of Paszke al. (2019) and Python Optimal Transport of Flamary et al. (2021).
In the following, we present the derived MSDA approaches.
3.1 DOMAIN ADAPTATION
We assume that Ns > labeled source distributions, Qs..".., Qskg. are available. The goal is improve performance
unlabeled target distribution, Qr- Hence, in eq. 6, Q (Qs((X))) NY, U (Qr(X)} with P = { and A {are J No+1
By definition of Q, NNN+1 corresponds to the weights of the target distribution, i.e., 0tp. We now discuss our two strategies, DaDiL-R and DaDiL-E. An overview shown in figure
DaDiL-R. Our first strategy is based on the reconstruction Br(X, Y) = B(ar: : P) of Qr. Features X and labels Y are generated with equations 3 and 8 respectively. Using (Redko et al., 2017, Theorem 2), we can theoretically bound the risks on these domains using,
Rar (h) S RB, (h) Wi(2r, V2 log(
(V
V
)
+A.
(9)
nQT
As such, a classifier trained with data from a distribution Br close to Qr is likely to perform well on Qr. The other terms in are related to the sample size and the joint error X = argminnewRog (h)+ RB, (h). Since Qr is unlabeled, À cannot be controlled directly. Similarly to Redko et al. (2017).
4

---

Under review as a conference paper at ICLR 2023
Set of Datasets
Atoms
Weights
Q
0
0
1
0.8
0
Datasets
Q:
0.01
0.94
0.04
0.6
-5
51
0.99
0.01
0.4
P3
-0.2
0.18
0.77
0.05
-0.0
-5
-
P1
P2
Pa
Atoms
DaDiL-R
DaDiL-E
Original
5.0
5.0
Reconstruction
0.0
0.0
-2.-
-2.-
-5.0-
-5.0
Figure 1: Illustration of DaDiL for MSDA. (Upper left) set of datasets Q. (Upper middle) learned atoms Ps. and their corresponding classifiers h2 (color map). (Upper right) learned weights. (Bottom left) reconstruction Br of QT Q1. (Bottom middle) Ensembled classifier har (color map) and the true labels of QT (scatter plot).
we hypothesize that the inclusion of labels in the ground-cost and the presence of labels in the other sources help control A.
DaDiL-E. Our second strategy is based on ensembling. Each atom Pr. has labeled support, i.e. ((x!P) Wi ) for which we learn a classifier dyg = argmin,ex: (h). To predict target (xx)) Ek hrr,h (T)).. We theoretically justify this labels, we use the predictor (x) method as follows,
Theorem 3.1. Let [X(P =k=1 and (Qr) be samples of size and nQr from P3 and Qr- Let lig be the minimizer of RR,. Then, for any d' > d and Ç > V2 there exists No. depending on d' such that for any 5 0 and min(n, nQr) > Nomax(6-d -d' 1) and Ak. with probability at least - 6, the following holds,
K Σ k=1
Rox(ha) S Ra(ha) +
an(Wi(Ps QT) + da + +c),
(10)
where ha Ek anhe d minimize Rp (h) + Roz (hi).
NEW
Ra (h) Σ R Rr, n. (h) and c 2 (og(
K
nQr
k=1
The bound in Equation 10 is very similar that of equation 9, but it takes into account how the atom classifiers are weighted to form hig- In addition, ar minimizes the r.h.s. in equation 10. Similarly to DaDiL-R, assume that using equation helps controlling Ag-
In the next subsection, we introduce a proxy for the Wasserstein distance based on the barycentric weights in the learnt dictionary.
5

---

Under review as a conference paper at ICLR 2023
3.2 A PROXY FOR THE EMPIRICAL WASSERSTHIN DISTANCE
We propose approximating distributions by their projection on the set (B Ax).
Thus, introduce the following proxy for W2, called WED,
K
(FD(Q1,
argmin (0U(
Σ
The Wo (NN)::p)): Pka);
(11)
b=1
which relies on the notion of Barycentric Coordinate Regression (BCR) (Bonneel et al., 2016, Defi nition 2), defined as,
a' = argmin Wa(B(a; P). acAx
(12)
Theorem 3.2. Let Q1 and Q2 be empirical distributions, and P {Px) be set of atoms learned by minimizing equation 6. In this case, the WED a pseudo-metric over M (proof in A.3).
Calculating the WED is simpler than Wa since needs (i) compute and store the pairwise Wo(Pk, Pky): (ii) solve small K K OT problem. Especially the distributions Ps have much fewer samples than Qr, calculating the WED is faster than W.. We present theoretical result bounding the WED by the Wasserstein distance of different terms in our dictionary learning problem,
Theorem 3.3. Let Q1 and Q2 be 2 empirical distributions, and P {Px} k be set of learned atoms. For an (resp. az2) the barycentric coordinates (e.g.. equation 12) of Q1 (resp. Q2) and Bi B(ar;; (resp. B₂) the WED is bounded as follows,
VED(@1, 11 Q2)
OL Bi) + a2, B²)
W. (BI, Q1) + We B2 Q2).
This theorem bounds our proxy by the Wasserstein distance, plus two kinds of terms. The first corre- spond to the geometry of the learned dictionary, whereas the second correspond to the approximation error (e.g.. (B1, Q1)), which explicitly minimized in our algorithm.
EXPERIMENTS
4.1 MULTI-SOURCE DOMAIN ADAPTATION
4.1.1 SHALLOW DOMAIN ADAPTATION
We compare MSDA methods with five other (shallow) DA algorithms: (i) Subspace
Table 1: Classification accuracy (in %) of DA methods. Each column represents target domain which we report mean + standard deviation over 5 folds. denote results from Montesuma & Mboula (2021a), while f denotes results from Turrisi et al. (2022).
Alignment (SA) (Fermando al., 2013); (ii) Transfer Component Analysis (TCA) (Pan al., 2010): (ii) Optimal Transport Do- main Adaptation (OTDA) (Courty et al., 2016); (iv) Wasserstein Barycenter Trans- port (WBT) (Montesuma & Mboula, 2021a); (v) Weighted JDOT (WJDOT) (Turrisi al., 2022). (i) and (ii) are standard algorithms in DA, (iii) general the OT baseline, and (iv) and (v) are the State-of-the-Arr (SOTA) in shal- low MSDA. We further consider the baseline case, which corresponds training classifier with the concatenation source domain data, and evaluating target domain data.
Method
A
12
W
Arg
Beseline
92.36 1: 1.21
98.33
1.56
93.26 1 1.38
56.58
1.52
92.63
SA
97.51
OTDA
88.25 1: 1:36
90.41
3.86
3.50
83002
1.67
$7.44
WIDOT!
94.23
0.90
100.00
59.13 2.91
35.93
2.07
92.37
WBT::..
92.74
0.45
95.87
1.43
96.57
1.76
35.01
0.54
92.55
DaDiL
95.26 2: 1.53
57,50
2.43
98.38
0.71
56.65
1.13
94.07
DaDiL
93.89 1: 1.81
06.33
0.83
98.38
0.71
$7.00 = 1.42
$1.53
We experiment on the Caltech-Office 10 benchmark, which is the intersection of the Caltech 256 dataset of Griffin et al. (2007) and the Office 31 dataset of Saenko et al. (2010). This benchmark

---

Under review as a conference paper at ICLR 2023
consists of four domains: Amazon (A), dSLR (D), Webeam (W) and Caltech (C). More information on these can be found in appendix B. A summary of our experiments is presented in table 1.
We focus and WBT, being especially better on Webcam and Caltech domains. W.r.. WIDOT, Turrisi et al. (2022) show in their experiments that their algorithm chooses one source domain (optimal w.r.. domain similarity) for the adaptation. This yields the best performance on domains A and D. On the other hand, ours combine information learned through the atoms for predicting on the target domain, which proves to be better for domains W and C. Note that a similar method to ours, WBT, also performs well on these domains. Nonetheless, as we learn direct reconstruct the target distribution, we manage to outperform their method.
analysis w.r.t the SOTA. Our method improves average performance over WJDOT
4.1.2 DEEP DOMAIN ADAPTATION
For deep DA, we compare six methods from the SOTA; (i) Domain Adversarial Neural Net- work (DANN) (Ganin ct al., 2016); (ii) Wasser- stein Distance Guided Representation Learning (WDGRL) (Shen et al., 2018); (iii) Maximum Classifier Discrepancy (MCD) (Saito al., 2018); (iv) MOST (Nguyen et al., 2021); (v) Moment Matching for MSDA (M3SDA) (Peng et al., 2019) and (vi) WBT (Montesuma & Mboula. 2021a). MOST and WBT consti- tute the OT-based MSDA SOTA. In our ex- periments we consider the Office 31 dataset of Saenko et al. (2010) (see appendix B). As the the backbone, we fine-tune a ResNet-50 (He et al., 2016) on the concatenation of source do- main data, then perform adaptation with target domain data. A summary of our experiments is shown in table 2.
Table 2: Classification accuracy (in %) of DA
methods on the Office 31 benchmark. Each col- umn represents a target domain for which we re- port mean standard deviation over 5 folds.
Method
A
D
W
Avg
Baseline
57.45
98.00
92.45
82.63
DANN
58.69
98.00
93.08
83,25
WDGRL
59.75
98.00
94.34
84.03
MCD
62.23
98.00
94.34
84.85
MOST
64.01
97.00
96.23
85.74
M3SDA
50.35
100.00
98,74
83.03
WBT
65.78
91.00
89.94
82.24
DaDiL.RR
72.52
95.00
98.11
88.54
DaDiL-E
73.58
96.00
96.23
88.60
On average, our method is SOTA, primarily due to its outstanding performance on the Amazon domain, surpassing other methods by a margin of 7.80%. On average, it presents an increase of 2. 2.70% in accuracy. Nonetheless, the transfer towards the dSLR domain is negative. In this domain, deep methods such as Margin Disparity Discrepancy (MDD) and M3SDA perform better. Overall, the main difference between our method and the deep methods in the SOTA is that we cannot update the parameters of convolutional layers, thus decreasing performance on this domain.
4.1.3 ADAPTATION USING ATOM INTERPOLATIONS
Besides using the inferred target domain weights or = aNs+1 for DaDiL-E and DaDiL-R, our method has the potential to generate infinitely muny domains, by interpolating atoms using an arbi- trary a E AR, that is, through Mp = (B(a;P) a E Ax). We thus explore the performance on Caltech-Office 10 and Office 31 benchmarks in terms of ha >k aghy and Ba B(a; P), call these interpolation models. This shown in figure for K k-1
3, which allows for nice visualization of the latent space. First, in some cases, the atoms themselves provide good predictors for the target domain (e.g., column 2, rows 2 and 3 in figure 2). This implies that DaDiL learns distributions that can discriminate between classes. Second, the reconstruction loss is correlated to performance DaDiI.-R, as regions with low reconstruction error tend to give better predictions on target. This remark agrees with our theoretical discussion (c.g., eq. 9). Further experiments on this idea can be found in appendix B.2.2.
To give an overview of the performance of interpolations in the latent space we show in figure 3 box plots corresponding to the adaptation performance of DaDiL-E and DaDiL-R on the Caltech-Offie 10 benchmark. for each target and arbitrary For domains A, C and W. the choice orr (dotted red line) is clearly above average. Surprisingly, for D, this choice is sub-optimal, indicating that other interpolations in the latent space can be useful for adaptation.
7

---

Under review as a conference paper at ICLR 2023
Loss Atom 3
Acc (E)
Acc (R)
Loss Aoom
Acc (E)
Acc (R)
Abom
Abom
Atom
Atom
Amazn
Aum
Atiom
2130,0 2049.7 2769.5
90.771 02.378 92.955
50,004 91/702 92.830
1575.4
1721.9
1868.5
60.452
66.844 68,216
63,856 06,029 20.3112
Atom 3
Atom
Atom
Atom
Atom
Atom
Atam
Auomi
2095.5 2270.3 2064.2
07.812
686 391-77
97,344
1000000
1622.6
1800.8
1065.0
54 S01
$1..30
92,10
S8,05
93.45
Arom 3
Atom
Atom
Arrym
Atom
Atrom
Webcam
Abomi
Abort
Atom
Abo
Mom
Aboin
2661.7
23000.0 2896.1
B1 18l
997 97.416
87.47
9S3 18 88
12119
151017
17295
J00 11 2011
90.01
ML.871
BS, 7L1
Aram 3
Atrom
Atom
Cihech
N/A
NAA
N/A
Ato
Abom
Mom
Abom
7omm
50.503
Figure 2: DA performance of interpolations in the latent space of DaDiL.. Columns represent re- construction loss W Wa(Ba. QT), and classification accuracy (Acc) of DaDiL-E and DaDiL-R, for Caltech-Office 10 (left) and Office 31 (right) benchmarks.
Amazon
Caltech
dSLR
Webcam
100
Accuracy [
87.5
95
95
85.0
90.0
82.5
100
100
Accuracy (R)
87.5
92.5
95
95
85.0
90.0
90
82.5
8 Number of Atoms
Figure 3: Performance analysis latent space interpolations on the Caltech-Office 10 benchmark. The red dotted line corresponds to the results reported in Table I for DaDiL.
4.2 ToY EXAMPLE: LEARNING THE MANIFOLD OF GAUSSIAN DISTRIBUTIONS
In this section we propose to learn a dictionary over the Gaussian distributions. Let Q: 1Q===((p (u I),
where Me E [0, 8]2 R2 is the mean vector. We discretize [0, 8)2 using 25 uniformly spaced points. For each Qc. we sample n = 512 points, generating datasets Qc- In a first moment, we learn a dictionary for K 2, ... 6, which is shown in Figure 4. Except for K 2, all dictionaries provide good reconstructions for the manifold. Furthermore, since μ = [Mr, [2] is a coordinate system for Q, one needs at least three atoms recover faithful reconstructions.
8

---

Under review as a conference paper at ICLR 2023
100
-5
0.0
2.5
5.0
7.5
10.0
(a) Set of distributions Q
(b) Loss per epoch.
(c) Learned Atoms (K 3).
135
m1 66.10
IEESKQ..0 (z)
(d) W2. Figure 4: Overview of DiL on the space Gaussian distributions.
(e) WED.
(f) Correlation analysis.
We focus on K 3 for better visualization. og (2/3, 1/s, 1/s) and calculate and analyze the W2 and the WED on MM = [B(x;P)) : a E As). This is shown in Figures (d) and (e). Since each Qr E Q is Gaussian, so is B ((Agueh & Carlier, 2011, Section 6.3)). Therefore W2 B) (00, og )u which explains the quadratic contours. Moreover, the WED contours are associated with the positions of the learned atoms and the polytope U (ao. a). since the ground-cost matrix .ky, k2 W2(Pr Pky.) does not depend on On or a. Indeed, for each o, the WED is defined by 3 x 3 OT problem, which has less degrees of freedom (w.r.t. a E As) than W2, which is 512 x 512 OT problem. Finally, figure (e) shows that the WED highly correlated W2. This is
mainly due the fact that our dictionary perfectly reconstructs distributions in Q.
5 CONCLUSION
We present a novel probabilistic framework, based on DiL, for learning dictionaries over datasets understood as point clouds, Q = [Q We learn dictionary of labeled points clouds P (Px) and weights A {az ) 2 where each distribution Qe is expressed as Wasserstein barycenter of atoms, i.c. B(a;: P). We propose two novel ways of applying our dictionary for MSDA, by either reconstructing the target domain the combination of atom distributions, or by ensembling classifiers learned on the atoms Pk. We show that interpolations in the latent space of our dictionary provide good predictors for the target distribution well. Finally, we define pseudo-metric for empirical distributions, based on their barycentric coordinates in the dictionary which is valuable approximation of the exact Wasserstein distance.
Limitations and perspectives: first, in DA, our method relies on the quality of feature extractors or embedding functions. In comparison with deep DA, DaDiL is not able to guide representation learning. We plan to pursue this line of work future research. Second, similarly to Bonneel et al. (2016), framework is not able to represent distributions outside Mp. This may lead to inconsistent reconstructions, which could actually be leveraged for novelty detection.

---

Under review as a conference paper at ICLR 2023
REFERENCES
SN Afriat. Theory of maxima and the method of lagrange. SIAM Journal on Applied Mathematiics, 20():343-335 357, 1971.
Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space. SIAM Journal on Mathematical Analysis, 43(2):904 924, 2011.
Pedro C Alvarez--Etteban E Del Barrio, JA Cuesta-Albertos, and C Matrán. A fixed- approach to barycenters in wasserstein space. Journal of Mathematical Analysis and Applications, 42): 744-762, 2016.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1)15111175 2010.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):
1997.
François Bolley, Arnaud Guillin, and Cédric Villani, Quantitative concentration inequalities for empirical measures non-compact spaces. Probability Theory and Related Fields, 137(3-4): 541-593, 2007.
Nicolas Bonneel, Gabriel Peyré, and Marco Cuturi. Wasserstein barycentric coordinates: histogram regression using optimal transport. ACM Trans. Graph., 35(4):71-1, 2016.
Nicolas Courty, Rémi Flamary, Devis Tuia, and Alain Rakotomamonjy. Optimal transport for do- main adaptation. IEEE transactions on pattern analysis and machine intelligence, 39(9):1853- 1865, 2016.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. In International conference on machine learning, pp. 685-693. PMLR, 2014.
Antoine de Mathelin, François Deheeger, Guillaume Richard, Mathilde Mougeot, and Nicolas Vay- atis. Adapt: Awesome domain adaptation python toolbox. arXiv preprint arXiv:2107.03045 2021.
Kilian Fatras, Younes Zine, Szymon Majewski, Rémi Flamary, Rémi Gribonval, and Nicolas Courty. Minibatch optimal transport distances; analysis and applications. arXiv preprint 1rXiv:2101.01192, 2021.
Basura Fernando, Amaury Habrard, Marc Sebban, and Tinne Tuytelaars. Unsupervised visual do- main adaptation using subspace alignment. In Proceedings of the IEEE international conference on computer vision, pp. 2960-2967, 2013.
Sira Ferradans, Nicolas Papadakis, Gabriel Peyré, and Jean-François Aujol. Regularized discrete optimal transport. SIAM Journal on Imaging Sciences, 733:11553 2014.
Jean Feydy, Thibault Séjourné, Fravçois-Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhorn divergences. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2681-2690. PMLR, 2019.
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z Alaya, Aurélie Boisbunon, Stanis- las Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, et al. Pot: Python optimal transport. J. Mach. Learn. Res., 22(78):1-8, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversenn training of neural net- works. The journal of machine learning research, 17(1):2966-000 2016.
10

---

Under review as a conference paper at ICLR 2023
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farte Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Technical report, California Institute of Technology, 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
L Kantorovich. On the transfer of masses (in russian). In Doklady Akademit Nauk, volume 37, pp. 227-229, 1942.
Wouter M Kouw and Marco Loog. A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence, 43(3):766-785, 2019.
Eduardo Fernandes Montesuma and Fred Maurice Ngole Mboula. Wasserstein barycenter for multi- source domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16785-16793, June 2021a.
Eduardo Fernandes Montesuma and Fred Maurice Ngole Mboula. Wasserstein barycenter transport for acoustic adaptation. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3405-3409, 2021b. doi: 10. 10.1109/ICASSP399788
2021 9414199.
Tuan Nguyen, Trung Le, He Zhao, Quan Hung Tran, Truyen Nguyen, and Dinh Phung- Most: Multi-source domain adaptation via optimal transport for student-teacher learning. In Uncertainty in Artificial Intelligence, pp. 225-235. PMLR, 2021.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22(10))1345-133) 2009.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. JEEE transactions on neural networks, 1992):199-210, 2010.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035 Curran Associates, Inc., 2019.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(-6)):355-677 2019.
Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008
levgen Redko, Amaury Habrard, and Mare Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 737-753. Springer, 2017.
levgen Redko, Nicolas Courty, Rémi Flamary, and Devis Tuia. Optimal transport for multi-source domain adaptation under target shift. In The 22nd International Conference on Artificial Intelli- gence and Statistics, pp. 858. PMLR, 2019.

---

Under review as a conference paper at ICLR 2023
levgen Redko, Emilie Morvant, Amaury Habrard, Mare Sebban, and Younès Bennani. A sur- vey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv preprint 1188: 2020.
Antoine Rolet, Marco Cuturi, and Gabriel Peyré. Fast dictionary learning with smoothed wasser- stein loss. In Artificial Intelligence and Statistics, pp. 630-638, PMLR, 2016.
Kate Saenko. Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pp. 213-226 Springer, 2010.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 3723-3732, 2018.
Morgan A Schmitz, Matthieu Heitz, Nicolas Bonneel, Fred Ngole, David Coeurjolly, Marco Cuturi, Gabriel Peyré, and Jean-Luc Starck. Wasserstein dictionary learning: Optimal transport-based unsupervised nonlinear dictionary learning. SIAM Journal on Imaging Sciences, 11(1):643-678 2018.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Proceedings the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Rosanna Turrisi, Rémi Flamary, Alain Rakotomamoniy, et al. Multi-source domain adaptation via weighted joint distributions optimal transport. In The 38th Conference Uncertainty in Artificial Intelligence, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural informa- fion processing systems, 1991.
Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact wasserstein distance. In Uncertainty in artificial intelligence, pp. 433-453. PMLR, 2020.
12

---

Under review as a conference paper at ICLR 2023
A THEORETICAL DETAILS
A.1 DIFFERENTIATION
In this section, we discuss the derivatives of (i) P Wo(P, Q). (ii a + B(a; and (iii) P + ((a;; P). Since we parametrize empirical distributions by their support, taking derivatives of distribution-valuee functionals (e.g., the Wasserstein distance) is equivalent differentiating w.r.t. the support of the free distribution (e.g., P).
Considering the Wasserstein distance, one has two alternatives: (i) using the Sinkhorn algorithm (Cuturi (201))) for calculating म. R x², where is parameter that controls the amount of entropic regularization. Since the Sinkhorn algorithm relies on differentiable operations, one can backprop- agate through its iterations. (ii) Differentiating through the min/inf using the Envelope or Danskin's Theorem Afriat (1971); Bertsekas (1997) optimality.
The Sinkhorn algorithm has mainly two drawbacks. First, it is unstable numerically for low regu- larization values, leading to problematic optimization. As remarked by Xie et al. (2020), possible workaround would be using stabilized versions of the algorithm, but these come with costly expo- nential/logarithmic operations. The second issue is that the computed OT plans are longer sparse, which may lead to degenerated distributions (e.g., take the extreme case when c - oc). For those reasons, decide use exact OT between mini-batches.
A second approach is using the Envelope theorem, which, as advocated by Feydy et al. (2019), can be faster than differentiating through Sinkhorn's 's iterations. For completeness, state the Envelope theorem in the following.
Theorem A.1. (Envelope Theorem Bertsekas (1997)) Z C I a compact set and Φ Rn x Z > IR be continuous and such that ((,,2) IR + R is convex for each z E Z. The function f: R given by,
f(x) max ((xx 2), ZEZ
is convex. When Z(x) = {z 2xx max, (xx, z)} consists on an unique point then 0f dx, Oz;
aΦ(x, Z)
(13)
The application of this theorem for OT is straightforward. In this case Z = U(a [ b) C Rmmaaxnne which compact. Naturally, the variable = т. whereas (C,,T) (C, T).. This a linear function of C, hence convex. Finally, the uniqueness of x depends on the regularity of the ground- cost. This property is particularly true for the Euclidean distance. In this case, the Wasserstein distance's derivative reads as,
W2 Q)
(2)
OW PX
(P.Q)
2 X() x(10) B.
Here, it is essential note that due to equation 13, to be able to evaluate aW/oox"") one needs to compute x* OT(p,q, q. C) at optimality. This calculation poses a problem for the Sinkhorn algorithm since the number of iterations needed to converge is inversely proportional to Thus, in some cases, it is more efficient to compute exact OT.
Now, we may apply the Envelope theorem to the iterations of Álvarez-Esteban et al. (2016) in equation 3. Let my be the OT plan between (Pk) and (B) at convergence. The barycentric operator is thus,
( B(a;; P)
Σ a (PP),
which is linear w.r.t. og and x; (P)) The derivatives are, therefore,
(B Dax
dx! (B)
x (Pa) E Boxd and
andiag((TT)).)) E Rdxd dx (P))
13

---

Under review as a conference paper at ICLR 2023
A.2 WASSERSTEIN BARYCENTERS WITH FIXED- PRINT ITERATIONS
In previous work (Montesuma & Mboula, 202]a;b), authors have used the iterations of Cuturi & Doucet (2014) for calculating the support x of Wasserstein barycenters. A similar approach is using the fixed-point approach of Alvarez-Esteban et al. (2016). Let Tk,it be the barycentric mapping between Ps and B(it), that is, ix (B(r)) -E (TT))
(Pt), where n. is the OT
bj
plan between ((P) and X(B("), Moreover, let Tix(x) > on (x)r, Alvarez-Esteban et al. (2016) introduce the following mapping,
(PP) = T = - Σ ST((x)-
In this setting, Alvarez-Estebrn al. (2016) prove that the Wasserstein barycenter is -redt solution to 15 namely, vb B. Their procedure is then equivalent to B(0++)) = 1/ (B))). This
iteration corresponds exactly to those in Equation 3. These are summarized in Algorithm 2.
In section 3, Equation 7 introduced a supervised ground-cost between pairs of samples (x! (P) y/
())
+ ; - -y 1| where 3 2 O defines how costly it is to transport samples from different classes. For one-hor encoded vectors Y R=x*= the Euclidean distance is equivalent to the 0-1 loss, ||Y! x 1 = dvy (P) yj ()) This makes a direct link between our proposed ground-cost, and the semi-supervised regularization proposed by Courty et al. (2016).
and (x; (2) yi ), Ci. ||x ( x, (
In addition, using an Euclidean cost for the labels allows to interpolate distributions P(X, Y) and Q(X, Y). with respect to features and labels. Indeed, taking derivatives as in the previous section,
W.(P.Q) ΣΣ
(11x () - x + 11)),
2 (P) 2(
aw PW./ox(Pi(PP..
(14)
Up
2/ y(P) 23( 23(
- oWo/orln Q)
(15)
Equating the derivatives in 14 and 15 to 0 yields, respectively, the barycentric mapping of Courty et al. (2016) and the label propagation formula of Redko et al. (2019). This allows us to reformulate the fixed-point iterations for joint distributions. Algorithms 2 and 3 show the proposed iterations for barycenters of features and features-labels distributions respectively.
Algorithm Free-Support Wasserstein Barycenter of Unlabeled Distributions
Algorithm 3 Free-Support Wasserstein Barycenter Labeled Distributions
Require:
{x(P)) a E AK. F > 0.
Require: (X(Fe), ((Px) Ag. n. B
1: X (B initializee ((x (P())) }k=1)
1: x(a(0)) y:n(0)) = iniiitioo ((x) (Pe), P( (
2: while (B(1)) ((((te1))) Ir S converged
2: while |x(1(()) ((B(U-1) || has not converged do
3:
for 1,... K
4:
(Gr)es = 1x (Pk) x (((lt)) a|x
3: for k= 1,. K do
(ifitt)
4: (Ct),, ||x (Pt)
12
5:
OT(ung Mn, Ct)
5: Th = =OT(ung; Un. Ck)
6:
end for
6: end for
7:
((((++1))
EK (Pp)
7: (((*+()) DK T (P.)
((
>K au (
8: end while
9: end while
Ensure: Barycenter support samples (B)
Easure:
Barycenter support samples X(B and labels (B)
Remark 1: in Algorithms and 3 we use the routine initialize to get the initial barycenter sup- port X (B(9) (B()) In our experiments we sub-sample n points from the concatenated matrices (PL) (P)) I==1- Alternatively, one could do as Montesuma & Mboula (2021a:b) and initialize the barycenter support randomly.
14

---

Under review as a conference paper at ICLR 2023
A.3 APPROXIMATION PROPERTIES OF WED
Proof of Theorem 3.2: our proof is done in 4 steps: (i) (2) 2 0, vQ,, Q2. (ii) WED(Q1, Q1) 0, vQ2. (iii) WED(Q), Q2) = WED(Q2, Q1) (iv) WED(Q), Qs) - WED(Q1, Q2) + WED(Q2, Qs).
(i) Since Thi Ao and Wo(Pk) Pko)) are non-negative, vỘ1, Q2 has WED Q2) 2 0.
iii) First, note that Wo (Q Q1) O. Now, since equation 12 is a convex optimization problem, to each Q1 one has single solution a₁ Hence, Ix E U a1), and WED(Q1, Q1) = 0.
(ii) If л E U(a1, 0x2), then n² E U(a2,a1) Due to commutativity the summation, WED(Q1, Q2) = WED(Q2,
(iv) Let x(1, .2(2 and (3) be the solutions WED(Q1,, WED(Q1, Q2) and WED(Q2, Q3), respectively. In this case,
WED Qa) Σ
Σ
W Wc Pka);
T kk
K
We WC(Pk. Pra)
k2
1== k==1
since (2) is sub-optimal for the pair (Q1, Q3). Now,
WED(Q), Q2)
(2) Wo(Pn, Pka).
k1
k==1
K
+ Tk
(3)
Wo Pko)
1
k==1
WED( 22) + WED(Q2, Qs)-
since (3) is non-negative.
Proof Theorem 3.3: our proof relies on the successive application of the triangle inequality for the Wasserstein distance. Let U (a1. a2) be the minimizer in equation 11. Then,
K
K
WED(Q,,
Σ
Σ
kr W WclPki Pko).
bn 1==1
K
The (Wo (2) + Wo(Ppp Q2)),
Ag 1==10 k
Σ Th M (Wa(Pk), Q1) + (2(Q1;12) Wc(P Q2)).
We now break this sum in parts. First, let us consider W2(Pk,, Q:):
K
K
kg W. Q1) = Σ Wc(P,, Q1)
Σ
01 W (P)
mm
K
(W. W.
15

---

Under review as a conference paper at ICLR 2023
where we used the triangle inequality in the last line. As consequence,
Th Wo(Ps, Q1) W. (B( Q1) +
a1 W Wc(Pe. Bi).
The exact same process can be repeated for the term involving W2 Q2). that
Σ k2 Wo(Pes Q2) W. (B2, Q2) Σ a W.W.WWWl (Pn. B2). k=1
The final term follows from the fact that W2(Q1, Q2) is independent of both kn nor l2. Therefore,
Th Wo (o(Q1, W.(21, Q2)
Σ
Wo(Q1,
A.4 THEORETICAL BOUNDS FOR DOMAIN ADAPTATION
In what follows, we consider the theoretical results of Ben-David et al. (2010) and Redko et al. (2017) for giving the theoretical guarantees of both DaDiL-E and DaDiL-R. For completeness, re-state Lemma and Theorem
Redko et al. (2017), and prove Theorem 3.1.
Lemma A.1. (Due to Redko al. (2017)) Let P and Q be two probability distributions over Rd Assume that the cost function c(x(1", x)) = [1x(((()) (xx(())ll He where Hs is reproducing kernel Hilbert space equipped with kernel ④ Rd x Rd + R induced by Φ Rd - H₂ and (Px(P), x())) = (d(x(P)), ()) Assume that the loss function In, x + I(h(x), hx))) is convex, symmetric, bounded, obeys the triangular incquality and has the parametric form [x(x) (x for some q > 0. Assume that the kernel & E Hg is square-root integrable w.r.t. both / and Q and 0 s (x(()) x()) < M, (x(P),
x(() Rd, Then the following holds,
Ro(h, M) S Rr(h, R.) + (P,QQ).
(16)
Lemma A.I bounds the risk (recall equation 4) of h with respect h' under distribution Q by the risk under distribution P, plus term depending on the distance between those 2 distributions. Intuitively, P is close to Q, its samples are similar and thus the risk are similar. We are now interested in acquiring bounds for the empirical risks R, and R& in terms of W1 (P,, Q). which are quantities we can estimate from data. We start by stating Theorem 1.1 of Bolley et al. (2007),
Lemma A.2. (Due to Bolley et al. (2007)) Let P be probability distribution over IR,, so that for
some >> 0 we have that fge= eal x1I dP < oo and P be its associated empirical approximation with support {x (P) j drawn independently from P. Then, for any d' > d and Ç < v22 there is constant no depending on d' and some square exponential moment of P such that for any > 0 and n (c
exp (-5ne)
where d' and Ç' can be calculated explicitly.
Lemma A.2 states the conditions for which P and P are close in the sense of Wasserstein. This last bound is on the form Piquantity > c] < 8, that is, with high probability quantity S c. These types bounds are ubiquitous the theoretical analysis of learning algorithms. We can express e explicitly in terms of 6,
2
log(!) nc
(17)
which will be useful in the following discussion. These results allowed Redko et al. (2017) to provide theoretical guarantees for the OTDA framework of Courty et al. (2016),
16

---

Under review as a conference paper at ICLR 2023
Theorem A.2. (Due to Redko al. (2017)) Under the assumptions of Lemma A.I, let ( and ( he 2 samples of size np and nq. drawn i.i.d. from P and Q. Let P and Q be the respective empirical approximations. Then for any d' > d and Ç < v2 there exists some constant No. depend- ing on d' such that for any 8 0 and min(n no) 2 Nomax(6 - 1) with probability least 1 - 5 for all h E H. then,
Rk(1) S Rp(1) + (P(,))) + 22og(Voffc (Viup + Vi/na) + A.
This last theorem effectively states that, besides constant terms that depend on the number of samples np and nQ, there are 2 factors that determine whether the risk under Q is similar to that under P: (i) the distance in distribution between P and Q, (ii) how well can classifier in work on both domains. practice, what OTDA does is minimizing Wi Q) while keeping À constant (e.g. by enforcing class-sparsity).
Now, let us discuss how these concepts apply DaDiL-R. Note that our strategy consists learning with samples obtained by the barycentric distribution By B(orr; P)) since it approximates the target distribution Qr. In this sense,
Ro, (h) S RB, (h) + W1 (Q(Qr. Br) + = 2log(Vo0/ V4/Tr VFn +
In this last equation, the term Wi (Qr, Br) the reconstruction error for the target domain, which we directly minimize in our objective function (equation 6). The remaining term is X, which we cannot directly optimise since target domain labels are not available during training. Nonetheless our algorithm leverages label information from source domains. If we suppose that their class structure similar to that of the target domain, the regularization term ensures that À remains bounded, since transport plans will not mix classes.
We now focus on Theorem 3.1, which presents a theoretical guarantee for DaDiL-E.
Proof of Theorem 3.1 This proof relies the triangle inequality for the risk. Let h7 Den (h) + Rp, (h) and ha Ex R. (ht) for hū argmin Rp, (h). Then,
(ha) s Ror (hi + Ror (h) ha).
(Ror (hi ha) - Rp ( ha)) + (Ror x) + Rp, (hi u ha)); s (Rer ha) Rr, (hi ha)) )) + (Ror (hir,)) + Rr, (1) RR (ha),
SW,(Pt.Q4 Qr) (Lemma A.
S (ha) Wi (P),QT)) + Au-
Summing over k. weighted by a,
Roz (haa = Rax <
Σ [halpp(aa))
(Pn (T) Ax).
k
k
= Ra(haa + Σ (PK Qr) + Ao).
s Ra(haa) + Σ (W, Qr) + d + +c),
This theorem is similar to previous theoretical guarantees for MSDA, such as those of Ben-Davd et al. (2010), Redko et al. (2017), and WBT (Montesuma & Mboula (2021a)). In particular, DaDiL-E uses ay for weighting ht. Furthermore this choice of a minimizes the r.h.s. of the last equation.
17

---

Under review as a conference paper at ICLR 2023
B TECHNICAL DETAILS
B. DATA PREPARATION
Caltech-Office 10: we use this dataset for shallow DA. As follows, we use the experimental setting of Montesuma & Mboula (2021a), namely, the fold cross-validation partitions and the features (DeCAF 7th layer activations).
Office 31: we use this dataset for deep DA. that end, we download the raw images from its public repository'. We train the feature extractor ourselves, which consists ResNet50 (He et al. (2016)). The preprocessing steps taken are: (i) resizing each image to (224, 224, 3), and (ii) apply- ing sot.fkke keras appp iiaiiiiionn tonsioon.... resnet preprocess s.sisssnttuttt..ee function. We initialize its parameters with the weights trained on ImageNet, then fine-tune for 51 epochs on each combination of source domains (e.g., A, D). Fine-tuning was performed using the ADAPT library of de Mathelin et al. (2021). We understand each epoch as a full pass through the entire dataset. We then use the fine-tuned network to extract features (vectors
R2018) from the target domain only.
Remark 2: we use a ResNet backbone since we were not able to reproduce previous results on the standard AlexNet backbone (e.g., Nguyen al. (2021)).
we give the specifics for the Caltech-Office and Office 31 datasets in Table 3. As follows, train our algorithms with all source domain data (train and test), and train target domain data. For instance, when training a specific algorithm on Office 31 dataset with the setting A, D - W, at training time the algorithm has 2817 labeled samples from A. 498 labeled samples from D and 636 unlabeled samples from W available. For evaluation, we use the target test set. In the context of our example, 159 unlabeled samples from W.
Remark 3:
Table 3: Details about the datasets considered for domain adaptation.
Dataset
Classes
Domain
Training Samples
Test Samples
Amazon
748
210
dSLR
108
49
Caltech-Off Offfiee 10
10
Webcam
224
71
Caltech
956
224
Amazon
2253
564
dSL.R
398
100
Office 31
31
Webcam
636
1559
Artworks dataset: select sub-set of 50 artworks from the Best Artworks of All Time?, from authors Monet, Delacroix, Magritte, Caravaggio and Van Gogh, with 10 images from each. The artworks were selected so that they group around 3 major color palettes (red, blue and somber).
B.2 ADDITIONAL EXPERIMENTS
B.2.1 TOY EXAMPLE: DICTIONARIES OVER COLOR PALETTES
One of the standard applications of OT is the analysis of color histograms (e.g., Ferradans et al. (2014)). This analysis done by considering RGB images I E hhxux3 as point clouds of = haw samples in R Thus, to each /,, one has Qe with support X (Q.) E Rhwxx3, In this experiment, we want to learn dictionary over @ = N N = 50 artworks of Monet, Delacroix, Magritte, Caravaggio, and Van Gogh, based on their color histograms. We further investigate the geometry induced by the latent codes A E (Ax)* N and the WED compared to the Wasserstein distance. Here images train our dictionary, leaving the rest Nr, = 15 for test. We can map these images into the latent space using equation 12. We show the result of DiL in figure
we discuss our results for K 3, using N₁
hhttps://facul:: gatech. wwwp://ddddddaaaaaaaae hntp://pp/maaaapp//////
hhtpps://ww. kaggl. com/ com//atasets/ wwwpaaattts/ikkkas7aaaa /ttp://///////tttttrrrrrrrs-----
18

---

Under review as a conference paper at ICLR 2023
Caravaggo
Delacrox
Magritte
Monet
Van Gogh
Atom
Atom 2
Atom
(a) Atoms learned through dictionary learning.
(b) Latent codes in the simplex A
(c) Wasserstein distance emheddings
(d) WED emheddings.
Figure 5: Summary of DaDiL for palette learning. Our algorithm learns palettes that reflect different palette clusters (a). These allow for an useful visualization on the latent space (b). The latent codes can be used to approximate the Wasserstein geometry (c) through the proposed WED (d).
We compare the Wasserstein distance with DaDiL, and the WED (i) running time; (ii) their induced geometry. For large-scale images, calculating W2(Qe, Q2,) is unfeasible due to the number of pixels. We thus down-sample the support matrices by selecting
2048 points randomly, hence ( E 20088x3 This step is not necessary for DaDiL since we perform optimization in mini- batches. Concerning running time, the 1225 pairwise W2 distances took approximately 26 minutes For DaDil., training takes an overall of 15 minutes. We remark that, for new distributions, we only need to solve 12 over Nts X K variables. This is usually faster than calculating pairwise distances.
In Figure 5 (a) we show the learned atoms, which correspond to the predominant colors in the train- ing images. Figure (b) shows the latent codes. The embeddings lead to an intuitive description of images, they express each image combination of blue, yellow, and gray palettes. In Fig- ure 5 (c) and (d), we compare the Wasserstein and WED embeddings using t-distributed Stochastic Neighbor Embeddings (t-SNE) (Van der Maaten & Hinton (2008)), which shows that are able to capture the Wasserstein geometry faithfully.
B.2.2 SENSITIVITY ANALYSIS
We study how the number of components K, the batch size b, and the number of samples in the atoms support influence the performance of DiL. Our results are summarized in Figures 6 and 7. We want to verify whether the Dil. loss T(P,, A) is good predictor for the test score.
exxperiments were performed on Intel Xcon CPU 2.20 GHz with 12GB of RAM
19

---

Under review as a conference paper at ICLR 2023
On one hand, for the Caltech-Office dataset, Figure 6 shows that increasing the number of samples and batch sizes globally leads to better approximation of the distributions manifold, as L(P, A) shrinks. This, however, is not necessarily true for the test score, as b 128 and 2048 leads to better adaptation results. Figures (d) and (e) shows that the DiL a good proxy for D-DiL and DaDiL.-R, it is strongly correlated with the test score. We further remark that I.PP, A) is more correlated with the test score of DaDiL-R, since this method directly relies on the goodness of reconstructions.
On the other hand, for the Office 31 dataset, Figure 7 (a) shows that the dependence on these pa- rameters much more complex, showing that general more samples or components do not lead to better approximations. Moreover, Figures (d) and (c) show that L(P, A)) is not correlated with neither DaDiL.-R nor DaDiL.-E. We believe that since this dataset encompasses more classes than the Caltech-Office, one needs more atoms to model the particularities of each domain.
We now focus our attention on t-SNE embeddings of reconstructions BT. Figures and 9 shows the embeddings for Caltech-Office and Office 31 datasets. For this first dataset, our reconstructions are correctly aligned with the dataset's classes, indicating that the atoms have correctly captured the characteristics of the domains. For the second dataset, as the number of classes is large, we only show comparison between original and reconstructed samples.
20

---

Under review as a conference paper at ICLR 2023
150 31
1-138-311 302
14.3
230 11
1350 240
94.
1200
230
kedWy -000
11000
Va
Toder Conpeeds
Sarder Conportd
(a) Loss.
(b) DaDiL-R validation accuracy.
DaDiL-E validation accuracy.
100:100
Tex Steee
(d) Correlation between DiL loss and DaDiL-R classification performance.
(c) Correlation between DiL loss DaDiL-E classification performance.
Figure 6: Study of how the batch size, number of samples and number of components impact DiL., DaDiL.-R and E performances on Caltech-Officc 10 dataset.
1700
L-050
1-250
1224 5-512
k-107
IEN bud07
10000
2240
2248
1000
Cansenera
Tumoe Cangoss
Namber
(a) LOss.
(b) DaDiL-R validation accuracy.
(c) DaDiL-E validation accuracy.
DUDKU Scre
Scure
md 59x
-.7
A.P..4
(d) Correlation between DiT. loss and DaDil. -R classification performance.
(c) Correlation between Dil. loss and DaDiL- classification performance.
Figure 7: Study of how the batch size, number of samples and number of components impact DiL, DaDiL-R and E performances Office 31 dataset.
21

---

Under review as a conference paper at ICLR 2023
Clos
backpack
backpock
bike
colrulator
coleulatio
heacphones
hoadphones
layybard
layybaard
lophop
laptop
moritor
monitor
mag
mug
projector
projector
(a) Amazon.
(b) dSLR.
Class
Class
backpock
bike
like
Calculatox
Calccllatoo
leyhoorrt
leyboard
loptop
laptop
(c) Webcam,
(d) Caltech.
Figure 8: I-SNE visualization of DaDiL-R reconstructions of each target domain. Semi-transparent circles correspond to points in the target domain, whereas triangles correspond to points in Br support.
22

---

Under review as a conference paper at ICLR 2023
Amazon
Reconstruction
amazon
dslr
webcam
(a) All domains.
(b) Amazon.
Webcam
dSLR Reconstruction
Reconstruction
(c) Webcam.
(d) dSLR
Figure 9: t-SNE visualization of DaDiL-R reconstructions of each domain the Office 31 dataset. Semi-tran temi-transparent circles correspond to points in the target domain, whereas triangles correspond to points in Br support.
23


Reviews:
[{"review_id": "o0Eo0MCfQeK", "reviewer": "Reviewer_fvrm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an optimal transport-based method to learn dictionaries that generalize over datasets (this is a domain adaptation problem). They propose two approaches: one to reconstruct the samples based on the learned barycenters, and another based on an ensemble of classifiers that are learned based on the dictionary atoms. They provide a toy example of how the method is able to learn a set of dictionaries based on a group of Gaussian-distributed data. They apply their method to learning dictionaries across several datasets and evaluate domain adaptation. They show the outperformance of their method in multi-source domain adaptation and how it performs better than the baselines in both shallow and deep feature generators. At last, they visualize the embedding of the method and how it better encodes similar data close to one another compared to latent based on Wasserstein distance.", "summary_of_the_review": "The paper proposes an OT-based approach to learning dictionaries across datasets that offer better domain adaptation compared to the baselines. The experimental results are detailed and nicely done. My reservations are coming from clarification on how their method differs from other OT frameworks and the inclusion of domain adaption as motivation into the abstract and intro which is missing in this version. I recommend acceptance of this paper as marginally above.\n\n\n----- after the authors' response\n\nDue to the raised concerns by another reviewer, I have reduced my score.\n", "strength_and_weaknesses": "Strengths: The paper proposes a novel OT-based framework to enable domain adaptation. The approach has advantages over prior works. I recommend its acceptance.\n\nWeaknesses: The paper lacks clear motivation in the abstract and introduction. The organization of the paper makes it hard to distinguish between what is prior work and their work. For example, section 2 says preliminaries, but the authors mention \"we propose ...\" several times within this section. Please clarify.\n\nHere are some questions that answering may improve the paper and are key to my review:\n\n- How this work differs from the two prior works mentioned in the contribution paragraph?\n- The paper lacks a clear motivation. Abstract has no information on what problem this framework may solve and their motivation for pursuing such a model. Why someone should use their method? For example, based on the preliminaries subsection 2.2, is their model trying to address iid assumption made by the majority of the models that may not be valid on a certain application or during the test? This comment of mine is mainly to improve the clarity of the paper. \n- Are WJDOT and WBT SOTA? or they are OT-based SOTA? Please clarify.\n- What is the complexity of the proposed method compared to other OT methods? In addition, how about a speed comparison?\n- Please include C in Table 1 (right). Why is it not included?\n- How is the performance compared to autoencoders for Figure 7?\n- The ensemble method is not very clear on how it is used for reconstruction. Please elaborate.\n\nHere are some terms that may improve the paper:\n\n- Is there a constraint on the dictionary in optimization formulation (1)?\n- I like the toy example. To highlight the advantages of the proposed method, I suggest including a baseline of form OT and dictionary learning in Figure 5.\n- What are the dictionaries in Figure 5 left? Possible to visualize them on the same figure?\n\n\nMinor points:\n\n- I suggest removing the red box around the words.\n- There is an extra () for year of the citations. It should be (name, year). You can use \\citep instead of \\citet.\n- The third line after (1) is not a complete sentence. Please rephrase.\n- The second line, Section 3, please cite previous works.\n- Is the method in Figure 3b DaDiL-E and R?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HkW03vcIpcn", "reviewer": "Reviewer_6SsD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work develops a novel method for applying Wasserstein barycenters to learn a dictionary for a collection of point clouds. Their method works by optimizing an objective over the support of the dictionary distributions and the dictionary weights with a Wasserstein barycenter loss. In particular, the cost function in the labeled data case forces points with the same labels to be mapped to one another. They apply this method to the problem of multi-class domain adaptation where they use the learned dictionary to transfer labels from a target dataset to an un-labeled dataset, and achieve some state-of-the-art results.", "summary_of_the_review": "This work studies an important problem and proposes a fairly natural and yet still novel solution. The empirical performance of their solution is strong. However, their theoretical results have some correctness issues (particularly Theorem 2), and otherwise some of their presentation (particularly the statements of their theoretical results, as well as the definitions of their classification procedures in section 3.2) are opaque. For these reasons, I think their work is just below the criteria of acceptance.", "strength_and_weaknesses": "Strengths:\n- Studies an important problem, that of domain adaptation with optimal transport\n- Proposes a new method based on using Wasserstein barycenters to form dictionaries of a larger set of point distributions\n- Achieves some state of the art empirical results for multi-class domain adaptation\n- Many parts are well-written, despite the fact that much mathematical notation is not defined before it is used\n\nNeutral:\n-While there is some mathematical justification given for their method, it doesn't require novel theory to accomplish. Also, there are some issues with correctness\n\nWeaknesses:\n- The barycentric coordinates definition is equivalent to 1NN: the minimizer in equation 11 is simply just a point mass on the $\\hat{P}_k$ which is closest to $\\hat{Q}$. The definition in Bonneel et al is distinct from yours, because it also includes a minimization over $\\hat{Q}$\n- The proof of Theorem 3.2 is incorrect: the last inequality on page 16 goes in the opposite direction, so the proof of Theorem 3.2 as written is incorrect\n- Figure 3 doesn't make sense because it is not learning a dictionary since $K = N$ there\n- It isn't clearly explained how the $alpha_T$ weights for the target distribution $\\hat{Q}_T$ are computed\n\nBonneel, N., Peyré, G., & Cuturi, M. (2016). Wasserstein barycentric coordinates: histogram regression using optimal transport. ACM Trans. Graph., 35(4), 71-1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9HO0U-4HPj", "reviewer": "Reviewer_LdfT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes using Wasserstein barycenters as a dictionary to recover multiple datasets and uses it for domain adaptation.\nIt shows that their approximation with the learned dictionary is bounded. Empirical results show that the proposed method marginally improved the DA classification accuracy.", "summary_of_the_review": "I recommend rejection at this point. The paper does not offer new insights into understanding the connection between dictionary learning and OT and it's hard to read.", "strength_and_weaknesses": "Strength:\n\nThe bounds in Theorem 3.1, 3.2, and 3.3 are novel to me.\n\nExploring Wasserstein dictionary learning for multi-source domain adaptation is a direction worth to explore.\n\n\nWeaknesses\n\nThe paper does not offer much beyond what Schmitz et al. did in their Wasserstein Dictionary Learning paper. At least it's not clear to me the contribution of this paper.\n\n\nThe paper is hard to read and understand. \n\n* 1. Introduction is basically a historic review of OT which is not informative and mostly irrelevant to the scope of the paper.\n* 2. Preliminaries are mathematical basics plus some history for dictionary learning, domain adaptation, and OT. They are a collection of previous knowledge in these areas and authors presented them without a flow in between. Some history and citations are stalling readers and are redundant for readers to understand the rest of the paper.\n\n* 3.1 Wasserstein Embedding distance as a proxy for the empirical Wasserstein distance. I appreciate authors' effort on Theorem 1 & 2 but I don't follow the intension of this proxy. I didn't find it being used in the rest of the paper and it doesn't seem to be a theoretical foundation for 3.2 either. \n\n* 3.2 is not clear. Authors jumped too quickly from problem statement to justifying their solution. No reasoning is given linking the problem and the solution. And it's not clear at least to me that how Theorem 3.3 which is yet another bound can justify \"this strategy\". \"We want to learn a dictionary of labeled distributions\": how do we use the dictionary for tasks on the target domain?\n\n* 4.2 Because it's not clear in 3.2 how a dictionary is used for domain adaptation, it's hard to understand 4.2 except a bare table showing high numbers.\n\n* 4.3 \"In this experiment, we want to learn a dictionary over 50 artworks\": I don't quite follow the reason to learn dictionaries of artworks as RGB images. \"which shows that we are able to capture the Wasserstein geometry faithfully\". This seems like a toy example for validation of the learned embeddings. I don't find it informative given we already have 4.1. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject