Approximating 1-Wasserstein Distance with Trees
Authors: Makoto Yamada
Yuki Takezawa
Ryoma Sato
Han Bao
Zornitsa Kozareva
Sujith Ravi
Venue: Accepted by TMLR ()

Abstract: The Wasserstein distance, which measures the discrepancy between distributions, shows efficacy in various types of natural language processing and computer vision applications. One of the challenges in estimating the Wasserstein distance is that it is computationally expensive and does not scale well for many distribution-comparison tasks. In this study, we aim to approximate the 1-Wasserstein distance by the tree-Wasserstein distance (TWD), where the TWD is a 1-Wasserstein distance with tree-based embedding that can be computed in linear time with respect to the number of nodes on a tree. More specifically, we propose a simple yet efficient L1-regularized approach for learning the weights of edges in a tree. To this end, we first demonstrate that the 1-Wasserstein approximation problem can be formulated as a distance approximation problem using the shortest path distance on a tree. We then show that the shortest path distance can be represented by a linear model and formulated as a Lasso-based regression problem. Owing to the convex formulation, we can efficiently obtain a globally optimal solution. We also propose a tree-sliced variant of these methods. Through experiments, we demonstrate that the TWD can accurately approximate the original 1-Wasserstein distance by using the weight estimation technique.  Our code can be found in the GitHub repository.

Published in Transactions on Machine Learning Research (MM/YYYY)
Approximating 1-Wasserstein Distance with Trees
Makoto Yamada
makoto. hatioloo.ymmad@ooss....
Okinawa Institute of Science and Technology
Kyoto University
RIKEN ATP
Yuki Takezawa
Kyoto University
RIKEN AIP
Ryoma Sato
r.soto moato@mm.ccm ist. wtt..s......tto...........
Kyoto University
RIKEN AIP
Han Bao
bao bto&ikkotoou..c.jj
Kyoto University
Zornitsa Kozareva
zormitsu@kozareww.co
Meta A1
Sujith Ravi
srani@sravi.org
SliceX AI
Reviewed on OpenReview:
/ttps://oppnreeet net/ forum? id= 198218720
Abstract
The Wasserstein distance, which measures the discrepancy between distributions, shows efficacy in various types of natural language processing and computer vision applications. One of the challenges in estimating the Wasserstein distance is that it is computationally expensive and does not scale well for many distributtion-compariii aim to approximate the L-Wasserstein distance by the tree-Wassee Wasserstein distance (TWD), where the TWD is a 1-Wasserstein distance with tree-based embedding that can be com- puted in linear time with respect to the number of nodes on a tree. More specifically, we propose a simple yet. efficient L.lregularized approach for learning the weights of edges a tree. To this end, we first demonstrate that the I-Wasserstein approximation problem can be formulated as distance approximation problem using the shortest path distance tree. We then show that the shortest path distance can be represented by linear model and formulated as Lasso-based regression problem. Owing to the convex formulation, we can efficiently obtain a globally optimal solution. We also propose a tree-sliced variant of these methods. Through experiments, we demonstrate that the TWD can accurately approximate the original I-Wasserstein distance by using the weight estimation technique. Our code can be found in /tttps://gitae cww/coo///o//eee oist//reetttt
tasks. In this study, we
1 Introduction
The Wasserstein distance, which is an optimal transport (OT))based distance, measures the discrepancy between two distributions and widely used in natural language processing (NLP) and computer vision (CV) applications. For example, measuring the similarity between documents is fundamental natural language processing task that be used for semantic textual similarity (STS) tasks (Yokoi al. 2020

---

Published in Transactions on Machine Learning Research (MM/YYYL
For CV tasks, because matches between samples using OT can be obtained, it is used to determine the correspondence between two sets of local features (Sarlin et al. 2020
One of the most widely used applications of the Wasserstein distance is the word mover's distance (WMD) Kusner et al. 2015), which measures the similarity between documents by solving the optimal transport problem. Recently, Yokoi et al. (2020) proposed a more effective similarity measure based on the word rotator's distance (WRD). The WMD and WRD are distance-based optimal transport metrics that can be estimated by linear programming with ((n²) order of computation, where is the number of words in a document. Thus, using the WMD and WRD on large number of documents is challenging.
A well-known speedup technique for OT problems is the use of the Sinkhorn algorithm (Cuturi, 2013), which solves the entropic regularized optimal transport problem. Using the Sinkhorn algorithm, we can estimate the WMD and WRD with a computational cost of O(n²). However, this is slow for most NLP and CV tasks. Instead of using lincar programming and the Sinkhorn algorithm, we can speed up the computation by projecting word vectors into a 1D space and solving the OT problem. This approach gives the sliced Wasserstein distance (Rabin et al. 2011 Kolouri et al. 2016) and its computational complexity is O(n log n). Note that the result of the sliced Wasserstein distance can be different if we use another projection matrix. Thus, the performance of the sliced Wasserstein distance is highly dependent on the projection matrix. To alleviate this problem, we averaged sliced Wasserstein distances using different, projection matrices.
The tree- Wasserstein distance (TWD) [Evans& & Matsen 2012 Le et al. 2019) computes the optimal trans- port on a tree, and can be computed in O(N), where N is the number of nodes in tree. Specifically, we first embedded word vectors into a tree and computed the optimal transport based on tree metric. Because we can obtaín an analytical solution to the OT problem a tree, it can be efficiently solved. Morcover, the TWD includes the sliced Wasserstein a special case and empirically outperforms the sliced Wasserstein distance (Le et al., 2019). The performance of TWD is highly dependent on the tree construction. One of the widely used tree construction methods is based on QuadTrees (Indyk & Thaper, 2003). Another approach uses a clustering- based tree algorithm (Le et al. 2019). Recently, Takezawa et al. (2021) proposed
a continuous tree-construction approach. Although tree-based methods improve computational complexity by a large margin, existing tree- based methods do not satisfactorily approximate the Wasserstein distance with arbitrary metric.
In this study, we aimed to accurately approximate the 1-Wasserstein distance using the TWD. Specifically, we propose a simple yet efficient learssellerninggbeeeee approach to train the edge weights of tree. To this end, we first demonstrate that the 1-Wasserstein approximation problem can be formulated as a distance approximation problem by using the shortest path distance on Iree. We then show that the shortest path distance can be represented by a linear model when we fix the tree structure. The weight estimation problem is then formulated as non-negative Lasso-based optimization problem. Owing to the convex formulation, we can efficiently obtain globally optimal solution. Moreover, propose tree-sliced variant of the proposed method. One of the key ad advantages of our formulation is that the optimization can be efficiently solved using an off-the-shelf solver. Through experiments, we evaluated the proposed algorithm using Twitter, BBCSport, and Amazon datasets and showed that the 1-Wasserstein distance computed by linear programming can be accurately approximated by the TWD, whereas the QuadTree (Indyk & Thaper 2003) and ClusterTree Le
ct al. 2019 have high approximation errors.
Contribution: The contributions of our paper are summarized as follows:
A Lasso-based weight estimation is proposed for the TWD.
An estimation of the weights for the tree-sliced Wasserstein distance is proposed.
We empirically demonstrate that the proposed method is capable of approximating the 1- Wasserstein distance accurately.
2

---

Published in Transactions on Machine Learning Research (MM/YYY)
2 Related Studies
Optimal Transport and Wasserstein distance: Measuring the distance between sets is an important research topic in machine learning, computer vision, and natural language processing. In early studies, the earth mover's distance (EMD), which can be computed using linear programming, was proposed as an optimal transport-based distance, equivalent to the Wassorstein distance, for cases where distance was employed to compute the cost function. The EMD has been particularly studied in the computer vision community because it can be used obtain matching between samples. For example, [Sarlin et al. (2020) proposed the Super Glue method, which determines the correspondence between two sets of local features by using optimal transport. Liu et al. (2020) used the optimal transport method for semantic correspondence. For NLP tasks, Kusner et al. (2015) proposed the word mover's distance (WMD)and were the first to use the Wasserstein distance for textual similarity tasks; this distance is widely used in NLP tasks, including text generation evaluation (Zhao et al., 2019). Recently, [Sato et al. (2022) re-evaluated the WMD in various experimental setups and summarized its advantages and disad vantages. Another promising approach is basod on the word rotater's distance (WRD) (Yokoi ct al., 2020), which normalizes the word vectors and solves the WMD with an adjusted probability mass. It has been reported that its performance is significantly better than that of the WMD. However, the computational cost of these approaches is very high. Therefore, they cannot be used for large-scale distribution-compariie tasks.
To speed up the EMD and Wasserstein distance computation, Cuturi 2013 proposed the Sinkhorn algo- rithm, which solves the entropic regularized optimization problem and computes the Wasserstein distance in quadratic order (i.c., (n))). Kusner et al. (2015) proposed a relaxed Wasserstein Mover's distance (RWMD), which only considers the sum-to-one constraint. Because the RWMD computation is simple, the RWMD can be efficiently computed. Ataas & Mittelholzer (2019) proposed the approximate constrained transfer method, which iteratively adds constraints to the RWMD and improves its performance while maintaining the computational speed. To solve large-scale OT problems, Genevay et al. (2016) formulated the optimal transport problem as a concave maximization problem using a c-transform, and then solved the problem with the stochastic gradient descent (SGD) and the stochastic average gradient (SAG). Mensch & Peyré (2020) proposed the online Sinkhorn algorithm, which solves OT problems using a stream of samples.
Another approach is based on the sliced Wasserstein distance (SWD) (Rabin et al. 2011 Kolouri et al,, 2016), which solves the optimal transport problem in projected one-dimensional subspacc. Because the one- dimensional optimal transport problem can be solved using sorting, the SWD can be computed in O(n log n). Several extensions of the SWD have been proposed. The generalized sliced Wasserstein distance extends the 1D sliced Wasserstein distance for multidimensional cases (Kolouri et al.) 2019). The max-sliced Wasserstein distance determines the 1D subspace that maximizes the discrepancy between two distributions and then computes the optimal transport in the subspace (Mueller & Jaakkola,
2015 Deshpande el al.] 2019)) The subspace robust Wasserstein distance (SRWD) is a general extension of the max-sliced Wasserstein distance for multi-dimennsonal subspaces (Paty & Cuturi 2019 Lin ct al. 2020).
The tree- Wasserstein distance (TWD) (Evans & Matsen 2012 Te et al. 2019 uses tree-based embedding, whereas the SWD uses 1D embedding. Because a chain of trees can represent ID embedding, the TWD includes the SWD as a special case. Le et al. (2019) reported that the TWD can empirically outperform the SWD. The TWD has also been studied in the theoretical computer science community and can be computed using the QuadTree algorithm (Indyk & Thaper 2003). Extensions of the TWD have also been studied; these include the unbalanced TWD (Sato et al.) 2020 Le Nguyen 2021 supervised Wasser- stein training (Takezawa et al.) 2021), and tree barycenters (Tkeezawa et al. 2022 These approaches mainly focus approximating the 1-Wasserstein distance with tree construction, where the weights of the edges are set by a constant number. Recently, [Backrrs et al. (2020) proposed FlowTree that combines the QuadTTree method and the cost matrix computed from vectors. They then showed that the FlowTree outperformed QuadTree- based approaches. Moreover, they theoretically guarantee that the Quad'Tree and FlowTree methods can approximate the nearest neighbor using the 1-Wasserstein distance. [oy & Zhang (2022) proposed an L1-embedding approach for approximating the 1-Wasserstein distance for the persistence diagram. However, there no learning-bass based approaches for approximating the 1- 1-Wasserstein distance in
3

---

Published in Transactions on Machine Learning Research (MM/YYYY)
general setup. Thus, focused estimating the weight parameter of the TWD from data to approximate the 1-Wasserstein distance.
The optimal transport has been applied to various types of tasks including barycenter estimation Cuttui & Doucet, 2014 Benamou et al. 2015), generative modeling
Applications of the Optimal transport:
(Arjovsky et al. 2017 Gulrajani et al. 2017 Kolouri et al. 2018 domain adaptation (Courty et al.
2017
Shen et al.j 2018|, inverse optimal transport (Li al. 2019p, differentiable sorting (Cuturi et al. 2019
Blondel et al.) 2020 Xie et al.) 2020), and modeling of population dynamics (Bunne al. 2022). Moreover,
using the Sinkhorn alg,rithm, we can solve the Wasserstein distance minimization problem in an end-lo-end manner (Genevay et al. 2018 In this research direction, solving the Wasserstein distance minimization problem in the mini-batch sctup is an important research topic (Nguyen et al. 2022).
Many OT-based applications were formulated as one of two minimization problems of the Wasserstein dis- tance (Arjovsky et al. 2017), a minimization problem using the Sinkhorn algorithm (Genevay et al. 2018 or a minimization problem with 1D slicing (Kolouri al. 2018). The tree- Wasserstein distance, which includes sliced Wasserstein distance a special case, has been applied to document classification (Le et al., 2019
topological data analysis (TDA) (Te et al. 2019). neural architecture search (NAS) (Nguyen et al.
2021
and barycenter estimation (Takeaawa et al. 2022). Moreover, the TWD is known as the UniFraac distance [Lozupone & Knight] 2005), which is phylogenetic method for comparing microbial communities. Thus, our proposed weighting technique for the TWD can be used for microbial community comparison.
3
Preliminary
In this section, we introduce the optimal transport, Wasserstein distances, and Free-Wassersten distances.
3.1 Optimal transport and Wasserstein distance
Optimal transport: We computed the distance between two datasets {( a:)] and ((af, bo)) where E X' R4, I' E X' Rd, and Σ a, 1 and Σj= by are the probability masses. For example, in the text similarity setup, æ and x' are precomputed word-embedding vectors, and is the frequency of word o; in a document. Let D = { denote the entire set of vectors where N' denotes the number of vectors. This study aims to measure the similarity bet ween the two datasets: ((m,,,;)) and ((a4,,,))3
In the Kantorovich relaxation of the OT, admissible couplings are defined by the set of transport plans between two discrete measures: / =E= a:5m; and Σ b50g
U(H.v) (II E R III w a, II 1, b}.
(1)
where dz, is the Dirac at position Hi, 1n is n-dimensional vector whose elemets 1, and (01, .2,,..,aaa) an) E R and b = (b,, b2...................... ) E Ry are the probability masses of distributions μ and respectively.
Then, the OT problem between two discrete measures, μ and v, is given as
min (EU().))
Ticll, 25)),
(2)
where Fyj is the i.j-th element of II, and c(æ, x²) is cost function (e.g., x²) | x x'1l)) Eq.
2
is equivalent to the earth mover's distance, and it can be solved by using linear programming with O(n computation (n = n'). To accelerate the optimal transport computation, Cuturi 2013)) proposed an entropic regularized optimal transport problem:
n'
min (eU().))
Tzc( (2;, (II
(3)
i=1
4

---

Published in Transactions on Machine Learning Research (MM/YYY)
where e the regularization parameter and (II) - Σ - 1) the entropic reg- ularization. The entropic regularized OT is equivalent to the EMD when 6= 0. The entropic regularization problem can be efficiently solved using the Sinkhorn algorithm with ( cost. Moreover, the Sinkhorn algorithm can be efficiently implemented on GPU, leading to a super-efficient matrix computation. Hence, the Sinkhorn algorithm is one of the most widely used OT solvers in ML.
Wasserstein distance: we use distance function d(æ, x²) for the cost c(x, x') and p 2 1, the p- Wasserstein distance of two discrete measures between two probability measures and v is defined as
1/8
(reinf inf (Ell(p.))
/
Wp(u,v)
d(æ, x' w'ddd'dddaaa (z, xxxx"
(4)
where U denotes the set of joint probability distributions Peyré et al. 2019 (Remark 2.13).
u(u,v) = { E M, x X'): PxyT = μ., Px =/),
(5)
and M (X') is the set of probability measures, Pxt and Pxrt are push-forward operators (sce Remark 2.2 in Peyré et al. 2019 The p- --ssssesstein distance of the two discrete measures μ 2i=1 0:0a; and
ا== dz is defined as:
1/p
n
Wp(n,
min (EU()M))
Tadd( 3
(6)
From the definition of the p- p-Wasscrstein distance, the 1- 1-Wassorstein distance of the two discrete measures is given as
Wi Wi v) = min NEU(U.))
mydd 2z)),
which is equivalent to Eq. 2 when c(x, d(æ x²). For the distance function, the Euclidean distance d(x, æ²)= = || =''l2 and Manhattan distance d(x, x²) = - - x'Il1 are common choices. Note that Cuturi (2013) proposed the entropic regularized Wasserstein distance, although the entropic regularized formulation does not admit the triangle inequality; therefore, it is not a distance. Sinkhorn divergence was proposed to satisfy the metric axioms of the regularized entropic formulation (Genevay et al. 2018).
3.2 Wresessasrrn distance (TWD)
Although the Sinkhorn algorithm has a quadratic time complexity, it is still prohibitive when dealing with large vocabulary sets. The TWD has been attracting attention owing its light computation and is thus used to approximate the W-Wasserstein distance (Indyk & Thaper 2003 Evans & Matsen 2012 Le el al.
2019).
We define T= (V. E), where V and E are sets of nodes and edges, respectively. In particular, for given tree T, the EMD with tree metric dr (x, x') is called the TWD. The TWD admits the following closed-form expression:
(T((v)) = Σ weln(T(oe)) ((T(v.))
(8)
cEE
where e is an edge index, we E R4 is the edge weight of edge is the eth node index, and p(T(ve)) is the lotal mass of the subtree with rool e.. The restriction on a tree metric advantageous in terms of computational complexity; the TWD can be computed as O(N), where N the mumber of nodes in a tree. It has been reported that the TWD comparcs favorably with the vamilla. Wasscrstein distance and is computationally efficient. Note that the sliced Wasserstein distance can be regarded a special case of the TWD (Takezawa et al. 2022
5

---

Published in Transactions on Machine Learning Research (MM/YYY)
0
W1
W2
W1
W2
2
1
2
W3
W4
3
W3
W4
3
4
3
4
Ø
22
I 1
L2 2
(a) Illustration of tree embedding.
(b) Shortest path between an and æ3.
Figure 1: Illustaaionn of tree embedding and the shortest path between æ1 and æy of the tree. All the imput vectors are associated with the leaf nodes. The bi, b2, and by of the tree are given as b, (1,1,0,1,0) b2 01,1,0,0,1) and by (1, 0, 0, respectively. In this case, dr (x1, æz) W3 + w1 + 022
The TWD can be represented in matrix form as (Takezawa et al. 2021
W (p, v) = (diag(w) B(a b)ll1
(9)
where w E N. is the weight vector, B E {0, 1 is a tree parameter, [B], = node is the ancestor the total number of nodes of tree, Nieaf is the number of
node of leaf node j and zero otherwise, leaf nodes, and diag(w) RR- is a diagonal matrix whose diagonal elements are w. Figure la shows an example of tree embedding. In this tree, the total number of nodes is N and the number of leaf nodes is Nisaf 3. Note that wa is the weight of the root node. Because and b probability vectors (i.c., a 1=1=11 1) and the elements of the first row of B are all 1, the weight of the root node is ignored in W1,,,
To compute the TWD, the choice of the tree structure and weight is crucial for approximating the Wasserstein distance. Indyk & Thaper (2003) proposed the QuadTree algorithm, which constructs a tree by recursively splitting a space into four regions (quad) and setting the weight parameter as (-((e) (f(e) is the depth an edge e). For the QuadTree, the TWD is defined as
WT(uvv) = ((-(e)(((())))) -(((())))(((())))))
(10)
eE E
Le al. (2019) proposed clustering- based tree construction. More recently, Takezawa al. (2021) proposed training tree using continuous optimization in a supervised learning setup.
Previous research related to the TWD has focused on tree construction. However, learning edge weights in the TWD have not been thoroughly examined. Thus, in this study, we propose weight estimation procedure for the TWD to approximate the Wasserstein distance.
4 Proposed method
In this section, we propose a tree weight estimation method using non-negative Lasso to accurately approx- imate the W-sssserstein distance.
4.1 Tree weight estimation using Lasso
Because the TWD is a I-Wasserstein distance with a tree metric, is natural to estimate the weight that can approximate with an arbitrary distance d(x,,²) Thus, we propose estimating the weight w by minimizing the error between the 1-Wasserstein distance and the distance d(x, x') and TWD. First, we show that the TWD approximate the 1-Wasserstein distance arbitrarily with some trees.

---

Published in Transactions on Machine Learning Research (MM/YYY)
Algorithm Sliced weight estimation with trees
1: Input: The matrix X, the regularization parameter X 2 0, and set of indices f2
2: for = 1....... do
3: random.seed(1)
B, = QuadTree(X) or B, = ChesterTree(X)
5: Compute z from B, and ((2;,,,), (i,5) E $2
6: W. =
argmin WER E(.neeld(a;; - w =²z())) =1,)2 ? + A|/w|11
7: end for
return {(B,, 01)
Proposition 1 For any measure μ μ== Ei= azdz, andv= E byónt and the distance function d: X X E, there erists tree T such that Wi(u, v) = W)(,vv) and II (u, [I v/) holds, where II (v. v) is an optimal solution to Eq. ( with cost d, and (1, an optimal solution with the cost function being the shortest path distance dy on tree
The proof is provided in section A This proposition indicates that the TWD has sufficient expressive power
to model the 1-Wasserstein distance.
Proposition 2
Le et al. 2019
/ da x' (dx((x
(T, inf
(11)
=El((u.)) xxxx
where (x x') is the length of the (unique) path between and x' on tree (h.e, the shortest path on the tree), and U is set of joint probability distributions.
This proposition indicates that the 1-Wasserstein distance with an arbitrary distance metric and the TWD are equivalent (i.e., (i(, = WT (p, v)) if d(xx x') = dr æ'). Thus, we aim to fit d(x, x') using dy (a, x'), where the tree structure and odge weights are the potential learning parameters. Figure [bb illustrates the shortest distance between æ1 and øz. In this case, (2 23) = EbePath(a ax)) wk w3 + w1 + Wa thus, a closed-form expression for the shortest path distance be obtained as follows:
Proposition 3
We denote E (0, (0,1)NxNaa = [b,, b...... b and b, E 10.11N. The shortest path distance between leaves i and can be represented as
dr(x,, w (b, + b, 26, o bj).
(12)
(Proof) The shortest distance between leaves i and can be uritten as
dz(x;, kt)) = Σ
wh
kEPath(m,..;)
Σ Wk Σ wP Σ Wk
KES.
WES,
kES,nS,
+ by 2w (b: o b,)
w T (b. + by - 2b, o bb),
where S, includes the set of ancestor node indices of leaf i and node inder of leaf i.
Given the closed-form expression in Proposition 3 we formulated the edgg-weight estimation problem
follows. First, we assumed that the tree obtained using tree construction algorithm, such the

---

Published in Transactions on Machine Learning Research (MM/YYYY
QuadTree [Indyk & Thaper 2003) and ClusterTree (Le et al. 2019 algorithms, and we fixed the tree structure (i.e., fixed B). We then propose the following optimization for learning w:
i argmin Σ (d(z;, w 20'2),)22 N|/w|12
(13)
WERN
where Zz, = b, 2b, by E R2,, |llla is the Ll-norm, and À 2 a regularization parameter. This optimization problem convex with respect to un We can easily solve the problem using an off-the-shelf non-negative Lasso solver. Zi sparse vector with only 2// < N elements, where H the tree depth. Thus, store all of in sparse format and efficiently solve the Lasso problem. Moreover, owing to the Lilregularization we can make most of the weights zero, which corresponds to merging the nodes of the tree. Thus, with this optimization, we are not only able to estimate the weight of the tree but also compress its size.
We also propose subsampled variant of the regression method.
W = argmin Σ (d(a,, w TTx ==,))² + A||2011,,
(14)
WE (i,))e
where S denotes a set of indices. In this study, we randomly sub-sampled indices.
Although our proposed method is specifically designed for the 1-Wasserstein distance approximation, it can also be used to approximate any distance by using the shortest path distance on a tree.
4.2 Tree-slice extension
Thus far, we have focused on estimating distance using a single tree. However, the approximation perfor- mance is highly dependent on the tree that is fixed before weight estimation. To address this issue, we can use the tree-sliced Wasserstein distance (Le al. 2019)
1
Σ (n, WT(,,) T
(15)
where WT, (H, v) is the TWD with the 1-th generated tree and T is the total number of trees. In previous studies, the TWD outperformed approaches based on a single tree. The matrix version the tree-sliced Wasserstein distance can be written as
WT(H,V) v) = T Σ dddag w. ( b)ll
(16)
where we and B, are the weight and parameters of the 1-th tree, respectively.
In this study, we extended the single-tree weight estimation to multiple trees. To this end, we soparately estimated w; from each tree and then averaged the estimates.
W=(,,v)
1 T
1
Σ
diag @. B, b)ll
(17)
where
w argmin Σ (d(a,, T 2213) A||w]|1, WERNN (iJER
(18)
(1) E RN is computed from B., where N₂ is the number of nodes in the fth tree. We call the proposed 2.
methods with the QuadTree and ClusterTree algorithms, the sliced qTWD and slicod-eTWD, respoctively.
Algorithm D the pseudocode for the weight estimation procedure that uses trees. If we T 1, this corresponds to the single-tree case.
8

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table I: Results of the I-Wasserstein distance approximation for the Twitter dataset. We report the averaged mean absolute error (MAE), averaged Pearson's 's correlation coefficients (PCC), and averaged number of nodes used for computation, respectively. For the tree-sliced variants, we empirically fixed the number of sliced T 3. For the MAE and PCC computation, computed the MAE and PCC between the 1-Wasserstein distance computed by EMD with Euclidean distance and tree counterparts, and we ran the algorithm 10 times by changing the random seed.
Twitter
Methods
MAE
PCC
Nodes
Quad'Tree
0,486 + 0.060
0.702 = 0.077
5097.2 = 56.3
gTWD (A = 10 3)
0.053 1 0.003
0.880 + 0.031
4976.4 = 103.6
gTWD (A = 10-2)
0.053 + 0.003
0.879 + 0.030
4542.2 = 56.9
TTWD (A = 10 1)
0.051 + 0.002
0.873 = 0.030
4328.4 = 42.0
ClusterTree
8.028 + 0.262
0.674 0.056
5717.5 = 47.1
cTWD (A = 10 10-3)
0.039 + 0.002
0.885 = 0.025
5674.1 = 50.0
cTWD (A = 2)
0.038 + 0.002
0.881 = 0.026
5481.0 = 31.6
cTWD (A = 1
0.036 1: 0.002
0.867 + 0.029
4195.2 - 63.1
Sliced- Srredd-uaaTTree
0.463 + 0.045
0.796 = 0.048
15164.6 1 141.4
Sliced-QTWWD (A =10-3
0.053 + 0.003
0.890 = 0.032
14702.7 + 256.5
SlDDed-qIWD (A 10
0.052 + 0.002
0.892 + 0.031
13581.1 + 82.8
Sliced-qTWD (A = 1
0.050 + 0.002
0.892 = 0.028
13026.3 + 82.6
Sliced-ClusterTT Tree
7.989 + 0.139
0.804 0.035
17129.5 + 58.9
SlDDed-eTWD (A 10 10-3)
0.035 = 0.002
0.929 = 0.017
17002.6 = 61.9
SlDD (A = 10
0.034 + 0.001
0.929 1 0.017
16414.7 1 61.8
Sliced-cTWW (A 10-1)
0.029 + 0.001
0.930 = 0.019
12565.3 + 104.0
4.3 Computational cost
The computational cost of the proposed algorithm consists of three parts: computing tree, estimating the weight, and computing the TWD. In many Wasserstein distance applications (e.g., document retrieval), it is necessary to compute the tree and weight only once. In this study, we employed the QuadTree and ClusterTree algorithms, which are efficient methods for tree construction. The computational complexity of the QuadTree is linear with respect the number of samples (dN log 1/c), where the number of input dimensions and > 0 is the accuracy. For the Cluster (Le et. al. 2019). we employed farthest-point clustering [Gonzalez) 1985). and the computational complexity of O(dH Niatt log K), where H is the depth of the tree and K is the number of clusters. For weight estimation, thresholding algorithm (FISTA) [Beck & Teboulle 2009), which is the ISTA with Nesterov's accelerated gradient descent (Nesterov 1983 FISTA converges to the optimal value as O(1kk2,, where & is the number of the iteration. For example, the dominant computation of FISTA on the Lasso problem yyy - Aw | + A//l is the matrix-vector vector multiplication of A and AT. In our case, the size of A |S2] x N, which includes [ multiplications and summations for each iteration. In practice, the computational cost of matrix-vector multiplication is small (see Table 5). Overall, when the number of vectors Nieat is 10, 000 and 30, 000, tree construction and weight estination can be performed efficiently with a single CPU. For inference, the TWD can be efficiently computed using our tree-based approach because its computational complexity is O(N), where is the total number of nodes of a tree. For the trec-sliced variant case, the computational cost of the treo-sliced Wasserstein is times larger than that of single-tree case.
employed the fast iterative shrinkage
5 Experiments
We evaluated our proposed methods using the Twitter, BBCSport, and Amazon datasets.

---

Published in Transactions on Machine Learning Research (MM/YYY)
Table 2: Results of the 1-Wasserstein distance approximation for the BBCSports dataset. We report the averaged mean absolute error (MAE), averaged Pearson's correlation coefficients (PCC), and averaged mum- ber of nodes used for computation, respectively. For tree-sliced variants, we empirically fixed the number of sliced as T = 3. For the MAE and PCC computation, computed the MAE and PCC between the 1- 1-Wasserstein distance computed by EMD with Euclidean distance and the tree counterparts. Further, we ran the algorithm 10 times by changing the random seed.
BBCSport
Methods
MAE
PCC
Nodes
Quad'Tree
0.512 + 0.023
0.851 = 0.031
11463.1 + 153.6
gTWD (A = 10 30-3)
0.098 + 0.005
0.923 + 0.016
11016.4 1 21.6
gTWD (A = 10 10-2)
0.095 + 0.007
0.921 + 0.017
9968.5 = 47.1
TTWD (A = 10 1)
0.073 + 0.022
0.878 = 0.046
8393.8 = 304.9
cTrre
5.842 + 0.171
0.799 = 0.041
12217.7 + 57.4
cTWD (A 10-3)
0.028 + 0.005
0.902 = 0.020
12145.3 + 56.5
cTWD (A = 2
0.023 + 0.004
0.894 = 0.022
11031.8 = 99.2
cTWD (A= = 10
0.023 1 0.003
0.866 = 0.027
6680.7 = 97.8
Sliced- hiieed--uaTTreee
0.498 + 0.020
0.901 = 0.022
34205.7 1 284.6
Sliced- (A =10-3
0.100 + 0.003
0.945 + 0.010
32447.4 + 586.0
Slieed-QTWD (A 10
0.096 + 0.004
0.945 + 0.011
29966.5 + 75.0
Sliced-qTWD (A = 1
0.073 + 0.014
0.925 = 0.020
25647.7 + 649.4
Sliced-ClusterTT Tree
5.830 + 0.080
0.863 0.026
36644.2 + 75.1
SlDDed-CTWD (A 10 10-3)
0.026 = 0.002
0.948 = 0.009
36405.2 + 80.5
SliDed-eTWD (A = =10- 10-2)
0.021 + 0.002
0.943 1 0.009
33024.7 1 112.9
Sliced-cTWD (A == =10-1)
0.018 + 0.004
0.924 = 0.013
19947.2 + 148.3
5.1 Setup
The approximation ability of the proposed method was evaluated. More specifically, measured the error and correlation between the 1-Wasserstein distance using Euclidean norm and tree-based methods. For the tree-based methods, we evaluated QuadTree (nndyk & Thaper 2003) and ClusterTree (e et al. 2019). For the ClusterTree, we set the number of clusters K = for all experiments. For the 1-Wasserstein distance, we used the Python optimal transport (POT) package ! For the tree methods, we first constructed a tree using an entire word embedding vector X and then computed the Wasserstein distance with the tree. For the proposed methods, we selected the regularization parameter from {10-3, 10-2, 100 The qTWD and cTWD are both proposed methods, and TWD and cTWD employ QuadTree and ClusterTree, respectively. We compared all methods using the Twitter, BBCSport, and Amazon datasets We further evaluated the group-sliced TWD using Qued and ClusterTree. We randomly subsampled 100 document pairs and computed the mean absolute error (MAE) and Pearson's correlation coefficient (PCC) between the 1- Wasserstein distance computed by EMD with the Fuclidean distance and its tree counterparts. Moreover, we reported the number of nonzero tree weights. In this experiment, we set the number of slices to 3 and the regularization parameters À {10 10 1001). We used SPAMS to solve the Lasso problem For all methods, using Xeon CPU E5-2690 v4 (2.60 GHzz and Xeon CPU E7-8890 v4 (2.20 GHz). Note that because the number of vectors a; tends to be large for real-world applications, evaluated only the subsampled training method.
the algorithm 10 times by changing the random seed. We evaluated all the met
https://pytthnnt..itttut wwt.hu.i.iiiides......... html
http://hohh..nriappes.tttt htte:///io....aaapes///////tta/a
10

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 3: Results the 1-Wasserstein distance approximation for the Amazon dataset. We report the aver- aged mean absolute error (MAE), averaged Pearson's 's correlation coefficients (PCC), and averagod number of nodes used for computation. For the tree-sliced variants, we empirically fixed the number of sliced T For the MAE and PCC computation, computed the MAE and PCC between the 1-Wasserstein distance computed by EMD with Euclidean distance and the tree counterparts. Further, we ran the algorithm 10 times by changing the random seed.
Amazon
Methods
MAE
PCC
Nodes
Quad'Tree
0.573 + 0.034
0.596
0.070
34203.0 = 672.6
qTWD (A 10 10-3)
0.072 1 0.002
0.789 + 0.040
32132.7 1 956.4
TTWD (A = 10-2)
0.067 + 0.005
0.783 + 0.035
28443.3 + 312.7
gTWD (A = 10 1)
0.051 + 0.009
0.732 = 0.064
16983.9 = 2202.4
Cluster Tree
8.074 + 0.073
0.759
0.031
33430.0 = 51.8
cTWD (A = 10
0.036 + 0.002
0.785
0.046
32642.3 + 59.7
cTWD (A = 10 1)
0.033 = 0.002
0.778 = 0.047
28701.8 = 134.8
cTWD (A= = 11101
0.028 +: 0.002
0.768 + 0.047
9879.8 347.0
Sliced- QuadTree
0.552 +: 0.026
0.759
0.047
102274.2 + 992.5
SlicedgTTWD (A = 10-3
0.072 + 0.002
0.838 + 0.024
94737.4 + 1756.5
Sliced-TTWD (A = 10
0.067 +: 0.003
0.849 + 0.023
85604.91 514.1
Sliced-gTWD (A = 10-1)
0.047 + 0.009
0.838 + 0.039
53689.9# 4423.5
Sliced-Cluster Tree
8.071 + 0.066
0.822 0.021
100,251.3 89.6
Sliced-cTIWD (A 10 10-3)
0.035 + 0.002
0.888 0.020
97891.5 5 = 89.8
Slieed-IIWD (A =10-2)
0.031 1 0.002
0.884 + 0.019
86068.8 1 202.1
Sliced-cTTWT (A = =10-1)
0.022 + 0.001
0.870 + 0.012
29540.0 + 519.6
5.2 Results of the approximation of the 1-Wasserstein distance
Tables 12 and 3 present the experimental results for the Twitter, BBCSport, and Amazon datascts, re- spectively. Evidently, the qTWD and cTWD can obtain a small MAE and accurately approximate the original 1- Wassersttin distance. In contrast, the Quad'Tree and ClusterTTre had larger MAE values than the proposed methods. Although the QuadTree has a theoretical guarantee, its MAE is one order of magnitude larger than that the proposed method. Becanse ClusterTree Tree is based on clustering, and the tree construc- tion independent of the scaling of vectors, it cannot guarantee a good approximation of the 1-Wasserstein distance. Our proposed method can approximate the 1-Wasserstein distance even if Cluster'rree used, For the PCC, the qTWD and cTWD outperformed the vanilla Quad and Cluster when we set small A. Moreover, even when using half of the nodes (i.e., X = 10=1), the PCCs of TTWD and cTWD were comparable to those of QuadTree and Cluster Figure 2 shows scatter plots for each method. We can observe that all methods correlate with the J-Wasserstein distance. However, it is evident that the proposed method has a scale similar to that the original WD, whereas QuadTree and ClusterTree have larger values than the original WD. Figure 3 shows the effect of the training sample size used for training in Eq. (14). In these experiments, the number of samples was varied as 25, 000, 50, 000, 75, 000, and 100, 000. We found that the PCC could be improved by using more samples. In contrast, the MAE was small, even when we used a relatively small number of samples.
For the trec-sliced variants, as reported in (Le et al. 2019 the sliced QuadTree and sliced ClusterTre oulperformed their non-sliced counterparts. Although there was no gain in MAE, our proposed methods, Sliced-qTWD and Sliced-cTWD,, can significantly improve the PCC values. Furthermore, because we use the distance to retrieve similar documents, we can obtain similar results using the Wasserstein distance. More interestingly, for the Amazon dataset, the sliced-eTWD (A 10-1) had PCC value of 0.870 with 29,540 nodes, whereas the cTWD (A 10-3) had PCC value of 0.785 with 32,642.3 nodes. Thus, the
11

---

Published in Transactions on Machine Learning Research (MM/ (YYY)
1.4
1.30
1.4
1.3
QuadTTee
QuadTree
1.3
1.0
1.15
1.2
0.9
1.10
1.1
0.4
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8
WD
(a) Twitter, QuadTree 0.781).
(b) BBCSport, QuadTree (p 0.904).
(c)
Amazon, QuadTree 0.669)
0.8
0.825
0.85
0,800
3TWD
0.7
0.775
0.750
@TWD
0.6
0.725
0.75
0.700
0.5
0.675
0.70
0.650
0.625
0.65
0.4
0.5
0.6
0.7
0.8
0.5
0.6
0.7
0.8
WD
WD
WD
(d)
Twitter, gTWD (p 0.930).
(e) BBCSport, gTWD (p 0.929).
Amazon, qTWD (p 0.848).
11-
8.0
10-
10.5
7.5
10.0
ClusterTree
9
Cluste/Tree
7.0
ClusterTree
9.5
8-
9.0
7-
8.5
6.0
8.0
6-
5
5.5
7.0
5.0
6.5
0.4
0.5
0.6
0.7
0.8
0.5
0.8
WD
WD
WD
(g)
Twitter, Cluster
(p
(h) BBCSport, Cluster Tree
(p
(i)
Amazon, ClusterTree (p
0.651).
0.828).
0.747)
0.8
0.75
0.85
0.7
0.70
0.80
CTWD
0.6
0.65
0.5
0.60
0.4
0.7
0.8
WD
WD
WD
(j) Twitter, cTWD (p 0.942).
(k) BBCSport, <TWD (p 0.898).
(I)
Amazon, cTWD (p = 0.841).
Figure 2: Scatter plots of the Twitter, BBCSport, and Amazon datasets.
sliced-cTWD significantly outperformed cTWD. Thus, to obtain low approximation error, one strategy is to use the sliced version and to prune unimportant nodes using Lasso.
5.3 Results of document classification
We evaluated the document classification experiment tasks with the 1-Wasccrstein distance (Sinkhorn algo- rithm), QuadTree, ClusterTree, qTWD, cTWD, and their sliced counterparts. For the proposed method, we set the regularization parameter to 10-3, Tn this experiment, used 90% of the samples for training
12

---

Published in Transactions on Machine Learning Research (MM/YYYY)
1.0
1.0
1.0
QuadTree TWWD
QuadTree
QradTree
0.8
0.8
qTWD
0.8
TWWD
0.6
0.6
0.6
MAE
MAE
MAE
0.4
0.4
0.4
0.2
0.2
0.2.5
0.2.
0.555
5
7.5
10
7.5
10
5
7.5
10
Number of samples (x10*)
Number of samples (x10*)
Number of samples (x10%)
(a) Twitter, q(TWDD
(b) BBCSport, qTTWD.
(c) Amazon, qTWD.
1.0
1.0
1.0
ClusterTree
ClusterTree
TlusterTree
0.8-
cTWD
0.8
cTWD
0.8-
cTWD
MAE
0.6
MAE
0.6
MAE
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.2.5
0,2
0.0.5
5
7.5
10
7.5
10
2.5
5
7.5
10
Number of samples (x10*)
Number samples (x10*)
Number of samples (x10%)
(d) Twitter, cTWD.
(e) BBCSport, cTWD.
(f) Amazon, cTWD.
1.0
1.0
1.0
QuadTree qTWD
0.9
0.9
0.9
0.8
0.8
0.8
PCC
0.7
0.7
0.7
0.6
QuadTree
0.6
QuadTree
0.6
qTWD
qTWD
0.2.5
2.5.5
0.3.5
5
7.5
10
5
7.5
10
5
7.5
10
Number of samples (x10*)
Number of samples (x10*)
Number of samples (x10*)
(g) Twitter, qTWD.
(h) BBCSport, qTWD.
(i) Amazon, gTWD.
1.0
1.0
1.0
ClusterTree
0.9
0.9
0.9
cTWD
0.8
PCC
0.8
PCC
0.8
0.7
0.7-
0.7
0.6
ClusterTree
0.6
ClusterTree
0.6
CTWD
cTWD
0.5,
5
7.5
10
0.5
2
7.5
10
0.5
2.5
5
7.5
10
Number of samples (x10*)
Number of samples (x10*)
Number of samples (x10*)
(i) Twitter, cTWD.
(k) BBCSport, cTWD.
(1) Amazon, cTWD.
Figure 3: MAE and PCC with respect to the number of training samples. In this experiments, we computed the MAE and PCC for m E (25, 000, 50, 000, 75, 000, 100, 0003. Note that the MAEs of the Cluster are larger than 1, and they do not appear in (d).(e), and (f).
and the remaining samples for the test. We then used the ten nearest neighbor classification methods. We ran the algorithm five times by changing the random seed and report the average mean accuracy.
13

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 4: Document classification results for the Twitter, BBCSport, and Amazon datasets. We used the ten nearest neighbor classifiers. For the Sinkhorn algorithm, we the regularization parameter with 10-2 and the maximum iteration with 100, respectively. We ran the algorithms five times by changing the random soed and report the averaged classification accuracy. For the proposed method, set the regularization parameter À = 10-3
Methods
Twitter
BBCSport
Amazon
WMD (Sinkhorn)
0.675 + 0.033
0.973 + 0.015
0.903 + 0.006
QuadTree
0.701 + 0.027
0.970
0.016
0.865 + 0.001
gTWD
0.691 + 0.028
0.967 + 0.014
0.847 = 0.036
Biced-QuaddTerr
0.694 + 0.017
0.970
0.020
0.877 = 0.001
Slieed-qTWD
0.697 + 0.027
0.967 0.014
0.887 + 0.011
ClusterTree
0.683 +: 0.019
0.901 + 0.056
0.873 + 0.010
cTWD
0.699 + 0.082
0.962 + 0.016
0.878 = 0.006
ShreddClusterTree
0.694 + 0.010
0.929
0.037
0.900 0.011
SlDD
0.700 + 0.021
0.970 0.018
0.905 = 0.010
Table 5: Time comparisons for the Twitter, BBCSport, and Amazon datasets. For training, we computed the time using Xeon CPU E5-2690 v4 (2.60 GHlz). For the test, we ran all the methods with an A6000 GPU, We ran the algorithms five times by changing the random seed and report the averaged computational time. We computed the average time consumption for comparing entire documents in the training set with one document for the test. The number in the parenthesis of each datasct is the number of the word vectors used in this study.
Training (CPU)
Test (GPU)
Dalaset
Methods
Cost matrix
Tree const.
Training set gen.
Lasso
Sinkhorn
3.0
n/a
n/a
n/a
0.159
QuadTreee
u/a
8.0
u a
n/a a
9.1 x10
Twitter (Neat = 4, 489)
Cluster
n/a
6.0
n
n/a
8.8 x10
TTWD (A = 10
n/a
8.0
39.4
4.1
8.8 x10-6
cWWD 10
n/a
6.0
49.8
11.7
9.0 x10-
Sinkhorn QuadTree
14.7
n/a
n/a
n/a
0.125
n/a
24.9
n/a
n/a
8.7 x10
BBCsport (Neat 10, 103)
Cluster Tree
n/a
21.3
n/a
n/a
x10-5
TWWD (A 10 cTWD (A = 30
n/a
24.9
95.6
4.5
8.7 x10
n/a
21.3
109.7
12.4
8.7 x10
Sinkhoru QuadTree
183.4
n/a
u/a
n/a a
2.24
n/a
162.2
n/a
n/a
x10
Amazon
ChusterTres
n/a
126.3
n/a
n/a
1.1 x10
(Niad 30, 249)
qT'WD 10
n/a
162.2
368.8
5.6
x10
cTWD (A 10
n/a 126.3
366.8
12.9
1.2 x10
Table 4 shows the classification accuracies for each method. The proposed weighting-based approaches compare favorably with the WMD (Sinkhorn). In particular, the proposed method can significantly improve the Cluster cases, whereas the improvements with the vanilla QuadTree method are modest.
5.4 Computation and memory costs
Table summarizes the computational costs for training and testing. For training, we measured the average computational cost using Xeon CPU E5-2690 v4 (2.60 GHz) for computing the cost matrix and constructing
14

---

Published in Transactions on Machine Learning Research (MM/YYY)
Table 6: Memory usage of each method for the test time. Note that we report the sizes of dense matrices because dense matrices are more suited for the GPU.
Twitter (Niest = 4, 489)
BBCSport (Nieat
10, 103)
Amazon (Nieat = 30, 249)
Methods
Cost matrix
Tree matrix
Cost matrix
Tree matrix
Cost matrix
Tree matrix
Sinkhorn
161.2 MB
n/a
816.6 MB
n/a
7320.0 MB
n/a
QuadTree
n/a
184.8 MB
n/a
929.7 MB
n/a
8149.3 MB
Cluster Tree
n/a
203.4 MB
n/a
982.6 MB
n/a
8089.1 MB
3 qTWD (A 10
n/a 184.8 MB
n/a
929.7 MB
n/a
8149.3 MB
cTWD (A 10
n/a
203.4 MB
n/a
982.6 MB
n/a
8089.1 MB
the tree training set generation for weight estimation, and the Lasso optimization. The training set generation process includes the calculation of the cost matrix and vectors. For the test, we computed document the average time consumption for comparing the entire document in the training set with for the test. As can be observed, training set generation is the most computationally expensive operation for our proposed method. The computational costs of computing the cost matrix and tree construction were comparable. In contrast, the computational cost of the Lasso optimization was small. Overall, the the test,
computational cost of training increased when the mumber of training vectors Nusf increased. the computational speed of the proposed method was several orders of magnitude faster with GPUs. It is important to note that the proposed approximation approach cannot be used to compute the exact 1- Wasserstein distance; the proposed approach may not be suitable for tasks in which the 1-Wasserstein distance should be computed exactly.
Memory cost: Table shows the required memory space for each method. For the Sinkhorn algorithm, it needs to store the Nicat x Meaf dimensional cost matrix. Tree methods need to store the N x Niat matrix, where N (N > Neeat) is the number of entire nodes. Thus, if we store the matrices in a dense format, the required memory size of the tree mothod is slightly larger than that of the Sinkhorn algorithm. Note that we report the size of dense matrices because they are more suited for GPU computation. However, for CPUs, because the tree matrix B can be efficiently stored in sparse format, the memory space required for B is negligible. For sparse matrix, with ClusterTree, the sizes of B for the Twitter, BBCsport, and Amazon datasets 94.4 KB, 226.2KB. and 681.1KB, respectively. Thus, if compute the TWD with CPU, the required memory space is much lower than that of the Sinkhorn-based algorithms.
6 Conclusion
In this study, we considered approximating the 1-Wasserstein distance with trees. More specifically, we first revealed that the 1-Wasserstein distance approximation can be formulated as distance approximation problem. Then, proposed non-negative Lasso-based optimization technique for learning the weights of a tree. As the proposed method is convex, globally optimal solution can be obtained. Moreover, we proposed weight estimation procedure for a tree-sliced variant of the Wasserstein distance. Through experiments, we show that the proposed weight estimation method can significantly improve the approximation performance of the L-Wasserstein distance for both the QuadTree and ClusterTree cases. Owing to Li-regularization, we can compress the tree size without losing the approximation performance.
A potential limitation of our proposal that the tree-based approach may not be suitable for tasks that require exact. evaluation of the 1-Wasserstein distance. Moreover, because tree must be constructed from data, the proposed approach may not be suitable for the end-to-end training of deep neural network models. Another limitation of the tree-based approach is that it may not be suitable for matching problems, such as a semantic correspondence delection lask (Liu et al. 2020). In contrast, the tree-based approach is very fast, with a reasonable approximation performance for the original L-Wasserstein distance. Thus, the
Due our implementation of construction, the computational complexities of Quad and Cluster (dNNoatt and O( respectively.
15

---

Published in Transactions on Machine Learning Research (MM/YYYY)
proposed tree-based approach is more suited for computing large number of 1-Wasserstein distances, such as document retrieval and document classification tasks (i.e., k-nearest neighbor classifier).
Acknowledgement
M.Y. supported by MEXT KAKENHI Grant Number 20H04243. R.S. supported by JSPS KAK-
ENHI Grant Number 21JJ24990.
References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial nelworks. ICML, 2017.
Kubilay Atasu and Thomas Mittelholzer. Tinear-complexity data-parallel earth mover's distance approxi- mations. In ICML, 2019.
Arturs Backurs, Yihe Dong, Piotr Indyk, llya Razenshteyn, and Tal Wagner. Scalable nearest neighbor search for optimal transport. In ICML, 2020.
Amir Beck and Mare Teboulle. A last iteralive shrinkagg- algorithm for linear inverse problems. STAM journal on imaging sciences, 2(1):183-220) 2009.
Jean-David Benamou, Guillaume Carlier, Marco Cuturi, Luca Nenna, and Gabriel Peyré. Iterative bregman projections for regularized transportation problems. SIAM Journal on Scientific Computing, 3722):A1111- A1138, 2015.
Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking, In ICML, 2020.
Charlotte Bunne, Laetitia Papaxanthos, Andreas Krause, and Marco Cuturi. Proximal optimal transport modeling of population dynamics. In AISTATS, 2022.
Nicolas Courty, Rémi Flamary, Amaur Habrard, and Alain Rakotomamonjy, Joint distribution optimal transportation for domain adaptation. NewrIPS, 2017.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In NIPS, 2013.
Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. In ICML, 2014.
Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. NewrIPS, 2019.
Ishan Deshpande, Yuan-Ting Hu, Ruoyu Sun, Ayis Pyrros, Nasir Siddiqui, Sanmi Koyejo, Zhizhen Zhao, David Forsyth, and Alexander G Schwing. Max-sliced Wasserstein distance and its use for GANs. In CVPR, 2019.
Tamal K Dey and Simon Zhang. Approximating 1-wasscrstein distance between persistence diagrams by graph sparsification. In ALENEX, 2022.
Steven N Evans and Frederick A Matsen. The phylogenetic ceutorovich-rubinstee metric for environmental sequence samples. Journal of the Royal Statistical Society: Series B (Statisticol Methodology), 74(3): 569-592, 2012.
Aude Genevay, Marco Cuturi, Gabriel Peyré, and Francis Bach. Stochastic optimization for large-scale optimal transport. NIPS, 2016.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. AISTATS, 2018.
16

---

Published in Transactions on Machine Learning Research (MM/YYY)
Teofilo F Gonzalez. Clustering to minimize the maximum intercluster distance. Theoretical computer science, 38:293- -009000-----0 1985.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans, NeurIPS, 2017.
Piotr Indyk and Nitin Thaper. Fast image retrieval via embeddings. In 3rd international workshop on statistical and computational theories of vision, volume 2, pp. 5. Nice, France, 2003.
Soheil Kolouri, Yang Zou, and Gustavo K Rohde. Sliced wasserstein kernels for probability distributions. CVPR, 2016.
Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced wasserstein auto-encoders. In ICLR, 2018.
Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo Rohde. wasserstein distances. In NeurIPS, 2019.
Generalized sliced
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document dis- tances. In ICML, 2015.
Tam Le and Truyen Nguyen. Entropy partial transport with tree metries: Theory and practice. In AISTATS, 2021.
Tam Le, Makoto Yamada, Kenji Fukumizu, and Marco Cuturi. Tree-sliced approximation of wasserstein distances. NewrIPS, 2019.
Ruilin Li, Xiaojing Ye, Haomin Zhou, and Hongyuan Zha. Learning to match via inverse optimal transport. Journal of machine learning research, 20, 2019.
Tianyi Lin, Chenyou Fan, Nhat Ho, Marco Cuturi, and Michael Jordan. Projection robust wasserstein distance and riemannian optimization. NearIPS, 2020.
Yanbin Liu, Linchao Zhu, Makoto Yamada, and Yi Yang. Semantic correspondence as an optimal transport problem. In CVPR, 2020.
Catherine Lozupone and Rob Knight. Unifrac: new phylogenetic method for comparing microbial com- munities. Applied and environmental microbiology, 71(12):8228 8235, 2005.
Arthur Mensch and Gabriel Peyré. Online sinkhorm: Optimal transport distances from sample streams. NewrIPS, 2020.
Jonas W Mueller and Tommi Jaakkola. Principal differences analysis: Interpretable characterization of differences between distributions. NIPS, 2015.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate (1/k" 2). Dokl. akad. nauk SasT, volume 269, pp. 543-547, 1983.
Khai Nguyen, Dang Nguyen, Tung Pham, Nhat Ho, et al. Improving mini-batch optimal transport via partial transportation. In ICML, 2022.
Vu Nguyen, Tam Le, Makoto Yamada, and Michael A Osborne. Optimal transport kernels for sequential and parallel neural architecture search. In ICML, 2021.
François-Pierre Paty and Marco Cuturi. Subspace robust wasserstein distances. In ICML, 2019.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and TrendsB in Machine Learning, 11(66)):355 607, 2019.
Julien Rabin, Gabriel Pcyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and application texture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision, pp. 435-446, Springer, 2011.
17

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Paul-Edouard Sarlin, Daniel Denone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In CVPR, 2020.
Ryoma Sato, Makoto Yamada, and Hisashi Kashima, Fast unbalanced optimal transport on tree. In NeurIPS, 2020.
Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Re-evaluating word mover's distance. ICML, 2022.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAT, 2018.
Yuki Takezawa, Ryoma Sato, and Makoto Yamada. Supervised tree- wasserstein distance. In ICML, 2021.
Yuki Takezawa, Ryoma Sato, Zornitsa Kozareva, Sujith Ravi, and Makoto Yamada, Fixed support tree-sliced wasserstein barycenter. AISTATS, 2022.
Yujia Xie, Hanjun Dai, Minshuo Chen, Bo Dai, Tuo Zhao, Hongyuan Zha, Wei Wei, and Tomas Pfister. Differentiable top-k with optimal transport. NeurIPS, 2020.
Sho Yokoi, Ryo Takahashi, Reina Akama, Jun Suzuki, and Kentaro Inui. Word rotator's distance. EMNLP, 2020.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. EMNLP-IJCNLP, 2019.
18

---

Published in Transactions on Machine Learning Research (MM/YYN)
A Proof of Proposition
Let II' = IIT (14, v). If there exists (i,j, k) such that Rij > 0 and jjk > 0,
Tij min( Tyk)
(p: g) = (i,j)
Tjk min(Tayy Tykk)
(p: (5,
T. pq
(19)
Tik min Tyk)
(p. = (i,
otherwise
is no worse than II because d(i.j) + +d]j k)) 2 d(i, k) is due to the triangle inequality. We assume no tuples (i,j, k) exist such that Tij > 0 and jt > 0 without loss of gemerality. If there exist ii, i2, "1 '2n d2++1 = 0
such that P, > O, Pigio > 0, Pigee > 0, ..... Paiian > 0, either
Tigh - (p. 9) (i2k+1, (2x++2
Tiae4aian- +E (p.g) = (i2R+3, 22k+2)
(20)
"pp
Tpq
otherwise
or
122 + (p, 9) = (i28+1, (28+2)
Tpg
Tigu
(p,9) (izx+3, (2x+2)
(21)
TTp
otherwise
is than II, where min ( (2i k 0,......... 1} U (Tia = 0, .....nn--- 1}, and both supports of og and Tig do not have this cycle. Therefore, we assume the support of p has cycle without loss of generality; that is, the graph is tree. Let T be the graph induced by the support of II and let the weight of edge = (i,j) be We = d(w,, wy). As the shortest path distance between u and v on tree T is no less than d(u, v) by definition, Wi(,v)) S (w,,v) holds. However, any pair (i.j) on the support of II is directly connected in T. Therefore,
Wi (1( = min (eU(u.)
T Tyd(a, x'))),))))
n
T dl az 2.²))
T d (z 2/))
min neU(uw)
Tzy ( 2/))
j=1
WT(H.v).
Furthermore, ui(nv)= = WT(, and P an optimal solution for both (a, and WT(, v).
19


Reviews:
[{"review_id": "Y6gU9sntTx", "reviewer": "Reviewer_GEEH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "While the formulation of the problem is interesting, and has potential, I think that in the current form the paper is missing key experiments showing the execution time benefits other regular solvers (EMD, sinkhorn or sliced versions). It is definitely not clear if the combination of the time needed to construct the tree, solve its weights with the Lasso solver, and compute the corresponding tree Wasserstein metric is faster on the considered problems. \nPerformances should also be reported for down stream tasks (in a context of word mover distances or other, I would suggest domain adaptation or more simply color adaptation). Notably, is there a direct correlation between low MAE and high performances ? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an extension of Tree-Sliced Wasserstein and supervised Tree Wasserstein distances that propose to compute an approximation of Wasserstein distances by finding tree metrics related to the two distributions to be compared. The novelty in this paper is that Authors propose to compute the weights of the tree branches by solving a lasso problem which enables\nto (theoretically) get as close as possible to the original Wasserstein distance. Experiments on document comparisons are proposed, showing that the proposed distance behaves similarly to the true Wasserstein distance. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Minor remarks/suggestion of improvments\n  - instead of (Kolouri et al., 2016), [1] should be credited for the first paper on sliced Wasserstein distance. Also, what do you mean by ‘performances are highly dependent on the projection matrix’ ? This point should really be clarified\n - Section 3.1 and 3.2 are redundant. Please merge them. \n - after Prop 2., the sentence starting with ‘This proposition indicates …’ misses a verb\n- Computational complexity: this part should definitely be enhanced with O() complexity for each step of the corresponding procedures- \n- Related works: there are many other applications in ML of Wasserstein distances. I suggest authors complete this aspects of the paper with references (including for instance, generative modeling, domain adaptation or zero shot learning, inverse optimal transport, etc.). Also, Numerous papers use the concept of mini-batches to lower the computational burden of solving a full optimal transport problem by averaging over smaller (and more tractable) problems. This line of works should also definitely be cited and compared to. \n\n\n[1] Julien Rabin, Gabriel Peyré, Julie Delon, and Marc Bernot. Wasserstein barycenter and its application to\ntexture mixing. In International Conference on Scale Space and Variational Methods in Computer Vision,\npp. 435–446. Springer, 2011", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3xRD5zgSiR", "reviewer": "Reviewer_JU6a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper is easy to read and follow. \nThe problem that they try to address is important to the community. The contribution is simple and straightforward.\n\nWeakness:\nThe experimental studies are limited and do not show the effectiveness of their approach. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Wasserstein distance is an important metric for many tasks such as document classification or retrieval. However, the computation of Wasserstein distance is computationally expensive. The prior works have discussed using tree-Wasserstein which can be evaluated faster than Wasserstein. \nThis paper proposed a method for learning the edge weights of the tree-Wasserstein such that it closely mimics the 1-Wasserstein distance. \nThe proposed method is based on Proposition 3 in the paper which shows \n$$\nd_\\mathcal{T} (x_i, x_j) = \\mathbf{w}^T (b_i + b_j -2b_i \\circ b_j),\n$$\nwhere $b_i$ indicated the ancestors of vector $i$ in the tree.\nThen they regress the edge weights $w$ such that  $d_\\mathcal{T} (x_i, x_j)$ matches L2 distance between two vectors. \nThe closeness of the resulting $\\mathcal{W}_{\\mathcal{T}}$ to 1-Wasserstein is based on the previous work.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1- ClusterTree information is missing from Figure 3.d, e, c. \n2- The training time for your algorithm should be presented in the paper.\n3- The advantage of the proposed method in end tasks such as classification or document retrieval is unclear. The author should also study the performance of their method in, for example, document classification and compare the performance (accuracy) to just using QuadTree/ClusterTree and 1-Wasserstein. Is the difference significant enough to justify using the proposed method?\n4- The author should also discuss the limitation of their work\n\n\nPresentation:\n1- Section 3.2: m and n' have been used interchangeably. Please use n' only to avoid confusion. \n2- Section 3.3: if the node i is the parents node of the leaf node j -> if node i is the ancestor of leaf node j\n3- Please add an equation number for the equations.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dGSCI862p4", "reviewer": "Reviewer_gte7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is tackling a high impact problem (efficient computation of 1-Wasserstein distance), the contribution is original as far as I know, and the resulting algorithm is easily implemented (via off-the-shelf solver) and provably attains a global optimum. The paper is well-written overall with clear explanations/figures. The experiments seem thorough and correctly executed, with convincing results as to the practical utility of the proposed method.\n\nI think this is a strong paper overall, the minor weaknesses I found are listed under requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper correctly points out that existing tree-based approximations of Wasserstein distance, while computationally efficient, fail to satisfactorily approximate the vanilla Wasserstein distance. The authors propose to train the edge weights of the tree via a sparse lasso algorithm via an efficient algorithm that enables the resulting tree Wasserstein distance to closely approximate its vanilla counterpart (at least for the L1 norm).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Given the purpose of the paper is to introduce efficient approximation algorithm, I would like to see runtimes of the methods whose performance is reported in the Experiments.\n\nI'd also like to see the authors discuss the extra memory / space requirement of maintaining the tree structure and weighted edges, which is not present in the vanilla Wasserstein distance calculation.\n\nWhen listing the computational complexity of the proposed algorithm, clarify how is this affected by the dimensionality d of the datapoints x.\n\nGiven the popularity of the Sinkhorn algorithm, I would like to see it included in the empirical comparisons. It will help readers understand the pros/cons of the proposed approach vs the most popular approximate-Wasserstein algorithm that I'm aware of.\n\n\nHere is another paper that could be discussed which efficiently approximates Wasserstein distance in 1D like max-sliced Wasserstein distance: \"Principal Differences Analysis: Interpretable Characterization of Differences between Distributions\"\n\nAuthors should comment why the focus is on 1-Wasserstein distance. I presume it is because trees are particularly well-suited for it but would be helpful to explicitly describe the intuition here.\n", "broader_impact_concerns": "None that I can think of.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to approximate the 1-Wasserstein distance by the tree-Wasserstein distance by computing tree embeddings of the distributions involved and showing how a simple Lasso can retrieve the edge weights of the trees, which are needed to compute the distance approximation via a shortest path problem reduction.

The reviewers recognised the importance of approximations of this kind and the main results in the paper. During the review and discussion phase some concerns were raised about the time and space complexity of the proposed approximations as well as the limited experiments run. Authors are addressed these concerns in a satisfactory way, adding one more baseline, two downstream tasks and reporting the time and memory taken to build the trees and compute the distance sub-routines. 

Even if in some scenarios the proposed methods might be slower than a baseline or require additional memory, I deem this work to be interesting enough to inspire follow up works that can close the gap for faster approximations. The paper is therefore accepted with minor revision. Authors are encouraged to incorporate the following minor and very minor changes in the camera ready.

Minor changes:
  - discuss the time complexity of all sub-routines. Stating that FISTA has an "efficient solver" is not clear. Discuss the 
  - discuss the space complexity of the method and its competitors (e.g. Sinkhorn).
  - report error bars for metrics in Table 1 and 2

Very minor changes:
  - increase font sizes of axis labels in plots (e.g. Fig3)
  - draw tables without vertical bars and boxes (one can use the `booktabs`package, see eg, [1])

[1] https://nhigham.com/2019/11/19/better-latex-tables-with-booktabs/

  
Decision: Accept (poster)