Heterogeneous Wasserstein Discrepancy for Incomparable Distributions

1 INTRODUCTION . Optimal Transport-based data analysis has recently found widespread interest in machine learning community , since its significant usefulness to achieve many tasks arising from designing loss functions in supervised learning ( Frogner et al. , 2015 ) , unsupervised learning ( Arjovsky et al. , 2017 ) , text classification ( Kusner et al. , 2015 ) , domain adaptation ( Courty et al. , 2017 ) , generative models ( Arjovsky et al. , 2017 ; Salimans et al. , 2018 ) , computer vision ( Bonneel et al. , 2011 ; Solomon et al. , 2015 ) among many more applications ( Kolouri et al. , 2017 ; Peyré & Cuturi , 2019 ) . Optimal Transport ( OT ) attempts to match real-world entities through computing distances between distributions , and for that it exploits prior geometric knowledge on the base spaces in which the distributions are valued . Computing OT distance equals to finding the most cost-efficiency way to transport mass from source distribution to target distribution , and it is often referred to as the Monge-Kantorovich or Wasserstein distance ( Monge , 1781 ; Kantorovich , 1942 ; Villani , 2009 ) . Matching distributions using Wasserstein distance relies on the assumption that their base spaces must be the same , or that at least a meaningful pairwise distance between the supports of these distributions can be computed . A variant of Wasserstein distance dealing with heterogeneous distributions and overcoming the lack of intrinsic correspondence between their base spaces is Gromov-Wasserstein ( GW ) distance ( Sturm , 2006 ; Mémoli , 2011 ) . GW distance allows to learn an optimal transport-like plan by measuring how the similarity distances between pairs of supports within each ground space are closed . It is increasingly finding applications for learning problems in shape matching ( Mémoli , 2011 ) , graph partitioning and matching ( Xu et al. , 2019 ) , matching of vocabulary sets between different languages ( Alvarez-Melis & Jaakkola , 2018 ) , generative models ( Bunne et al. , 2019 ) , or matching weighted networks ( Chowdhury & Mémoli , 2018 ) . Due to the heterogeneity of the distributions , GW distance uses only the relational aspects in each domain , such as the pairwise relationships to compare the two distributions . As a consequence , the main disadvantage of GW distance is its computational cost as the associated optimization problem is a non-convex quadratic program ( Peyré & Cuturi , 2019 ) , and as few as thousand samples can be computationally challenging . Based on the approach of regularized OT ( Cuturi , 2013 ) , in which an entropic penalty is added to the original objective function defining the Wasserstein OT problem , Peyré et al . ( 2016 ) propose an entropic version called entropic GW discrepancy , that leads to approximate GW distance . Another approach for scaling up the GW distance is Sliced Gromov-Wasserstein ( SGW ) discrepancy ( Vayer et al. , 2019 ) , which leverages on random projections on 1D and on a closed-form solution of the 1D-Gromov-Wasserstein . In this paper , we take a different approach for measuring the discrepancy between two heterogeneous distributions . Unlike GW distance that compares pairwise distances of elements from each distribution , we consider a method that embeds the metric measure spaces into a one-dimensional space and computes a Wasserstein distance between the two 1D-projected distributions . The key element of our approach is to learn two mappings that transform vectors from the unit-sphere of a latent space to the unit-sphere of the metric space underlying the two distributions of interest , see Figure 1 . In a nutshell , we learn to transform a random direction , sampled under an optimal ( learned ) distribution ( optimality being made clear later ) , from a d-dimensional space to a random direction into the desired spaces . This approach has the benefit of avoiding an ad-hoc padding strategy ( completion of 0 of the smaller dimension distributions to fit the high-dimensional one ) as in SGW method ( Vayer et al. , 2019 ) . Another relevant feature of our approach is that the two resulting 1D distributions are now compared through Wasserstein distance . This point , in conjunction , with other key aspect of the method , will lead to a relevant discrepancy between two distributions , called heterogeneous Wasserstein discrepancy ( HWD ) . Although we lose some properties of a distance , we show that HWD is rotation-invariant , that it is robust enough to be considered as a loss for learning generative models between heterogeneous spaces . We also establish that HWD boils down to the recent distributional sliced Wasserstein distance ( Nguyen et al. , 2020 ) if the two distributions live in the same space and if some mild constraints are imposed on the mappings . In summary , our contributions are as follows : • we propose HWD , a novel slicing-based discrepancy for comparing two distributions living in different spaces . Our chosen formulation is based on comparing 1D random-projected versions of the two distributions using a Wasserstein distance ; • The projection operations are materialized by optimally mapping from one common space to the two spaces of interest . We provide a theoretical analysis of the resulting discrepancy and exhibit its relevant properties ; • Since the discrepancy involves several mappings that need to be optimized , we depict an alternate optimization algorithm for learning them ; • Numerically , we validate the benefits of HWD in terms of comparison between heterogeneous distributions . We show that it can be used as a loss for generative models or shape objects retrieval with better performance and robustness than SGW on those tasks . 2 BACKGROUND OF OT DISTANCES . For the reader ’ s convenience , we provide here a brief review of the notations and definitions , that will be frequently used throughout the paper . We start by introducing Wasserstein and Gromov-Wasserstein distances with their sliced versions SW and SGW , where we consider these distances in the specific case of Euclidean base spaces ( Rp , ‖ · ‖ ) and ( Rq , ‖ · ‖ ) . We denote P ( X ) and P ( Y ) the respective sets of probability measures whose supports are contained on compact sets X ⊆ Rp and Y ⊆ Rq . For r ≥ 1 , we denote Pr ( X ) the subset of measures in P ( X ) with finite r-th moment ( r ≥ 1 ) , i.e. , Pr ( X ) = { η ∈ P ( X ) : ´ X ‖x‖ rdη ( x ) < ∞ } . For µ ∈ P ( X ) and ν ∈ P ( Y ) , we write Π ( µ , ν ) ⊂ P ( X × Y ) for the collection of joint probability distributions with marginals µ and ν , known as couplings , Π ( µ , ν ) = { γ ∈P ( X × Y ) : ∀A ⊂ X , B ⊂ Y , γ ( A× Y ) = µ ( A ) , γ ( X ×B ) = ν ( B ) } . 2.1 OT DISTANCES FOR HOMOGENEOUS DOMAINS . We here assume that the distributions µ and ν lie in the same base space , for instance p = q . Taking this into account , we can define the Wasserstein distance and its sliced variant . Wasserstein distance The r-th Wasserstein distance is defined on Pr ( X ) by Wr ( µ , ν ) = ( inf γ∈Π ( µ , ν ) ˆ X×Y ‖x− y‖rdγ ( x , y ) ) 1 r . ( 1 ) The quantityWr ( µ , ν ) describes the least amount effort to transform one distribution µ into another one ν . Since the cost distance used between sample supports is the Euclidean one , the infimum in ( 1 ) is attained ( Villani , 2009 ) , and any probability γ which realizes the minimum is called an optimal transport plan . In a finite discrete setting , Problem ( 1 ) can be formulated as a linear program , that is challenging to solve algorithmically as its computational cost is of order O ( n5/2 log n ) ( Lee & Sidford , 2014 ) , where n is the number of sample supports . Contrastingly , for the 1D case ( i.e . p = 1 ) of continuous probability measures , the r-th Wasserstein distance has a closed-form solution ( Rachev & Rüschendorf , 1998 ) , namely , Wr ( µ , ν ) = ( ´ 1 0 |F−1µ ( u ) − F−1ν ( u ) |rdt ) 1 r where F−1µ and F −1 ν are the quantile functions of µ and ν . For empirical distributions , the 1D-Wasserstein distance is simply calculated by sorting the supports of the distributions on the real line , resulting to a complexity of order O ( n log n ) . This nice computational property motivates the use of sliced-Wasserstein ( SW ) distance ( Rabin et al. , 2012 ; Bonneel et al. , 2015 ) , where one calculates an ( infinity ) of 1D-Wasserstein distances between linear projection pushforwards of distributions in question and then computes their average . To precisely define SW distance , we consider the following notation . Let Sp−1 : = { u ∈ Rp : ‖u‖ = 1 } be the unit sphere in p dimension in ` 2-norm , and for any vector θ in Sp−1 , we define Pθ the orthogonal projection onto the real line Rθ = { αθ : α ∈ R } , that is Pθ ( x ) = 〈θ , x〉 , where 〈· , ·〉 stands for the Euclidean inner-product . Let µθ = Pθ # µ the measure on the real line called pushforward of µ by Pθ , that is µθ ( A ) = µ ( P−1θ ( A ) ) for all Borel set A ⊆ R. We may now define the SW distance . Sliced Wasserstein distance The r-th order sliced Wasserstein distance between two probability distributions µ , ν ∈Pr ( X ) is given by SWr ( µ , ν ) = ( 1 Ap ˆ Sp−1 Wrr ( µθ , νθ ) dθ ) 1 r , ( 2 ) where Ap is the area of the surface of Sp−1 , i.e. , Ap = 2π p/2 Γ ( p/2 ) with Γ : R → R , the Gamma function given as Γ ( u ) = ´∞ 0 tu−1e−tdt . Thanks to its computational benefits and its valid metric property ( Bonnotte , 2013 ) , the SW distance has recently been used for OT-based deep generative modeling ( Kolouri et al. , 2019 ; Deshpande et al. , 2019 ; Wu et al. , 2019 ) . Note that the normalized integral in ( 2 ) can be seen as the expectation for θ ∼ σp−1 , the uniform surface measure on Sp−1 , that is SWr ( µ , ν ) = ( Eθ∼σp−1 [ Wrr ( µθ , νθ ) ] ) 1 r . Therefore , the SW distance can be easily approximated via a Monte Carlo sampling scheme by drawing uniform random samples from Sp−1 : SWrr ( µ , ν ) ≈ 1K ∑K k=1Wrr ( µθk , νθk ) where θ1 , . . . , θK i.i.d.∼ σp−1 and K is the number of random projections . 2.2 OT DISTANCES FOR HETEROGENEOUS DOMAINS . To get benefit from the advantages of OT in many machine learning applications involving heterogeneous and incomparable domains ( p 6= q ) , the Gromov-Wasserstein distance ( Mémoli , 2011 ) stands for the basic OT distance dealing with this setting . Gromov-Wasserstein distance The r-th Gromov-Wasserstein distance between two probability distributions µ ∈Pr ( X ) and ν ∈Pr ( Y ) is defined by GWr ( µ , ν ) = inf γ∈Π ( µ , ν ) Jr ( γ ) def . = 1 2 ( ¨ X 2×Y2 |‖x− x′‖ − ‖y − y′‖|rdγ ( x , y ) dγ ( x′ , y′ ) ) 1 r . ( 3 ) Note that GWr ( µ , ν ) is a valid metric endowing the collection of all isomorphism classes metric measure spaces of Pr ( X ) ×Pr ( Y ) , see Theorem 5 in ( Mémoli , 2011 ) . The GW distance learns an optimal transport-like plan which transports samples from a source metric space X into a target metric space Y , by measuring how the similarity distances between pairs of samples within each space are close . Furthermore , GW distance enjoys several geometric properties , particularly translation and rotation invariance . However , its major bottleneck consists in an expensive computational cost , since problem ( 3 ) is non-convex and quadratic . A remedy to such a heavy computational burden lies in an entropic regularized GW discrepancy ( Peyré et al. , 2016 ) , using Sinkhorn iterations algorithm ( Cuturi , 2013 ) . This latter needs a large regularization parameter to guarantee a fast computation , which , unfortunately , entails a poor approximation of the true GW distance value . Another approach to scale up the computation of GW distance is sliced-GW discrepancy ( Vayer et al. , 2019 ) . The definition of SGW shows 1D-GW distances between projected pushforward of an artifact zero padding of µ or ν distribution . We detail this representation in the following paragraph . Sliced Gromov-Wasserstein discrepancy Assume that p < q and let ∆ be an artifact zero padding from X onto Y , i.e . ∆ ( x ) = ( x1 , . . . , xp , 0 , . . . , 0 ) ∈ Rq . The r-th order sliced Gromov-Wasserstein discrepancy between two probability distributions µ ∈Pr ( X ) and ν ∈Pr ( Y ) is given by SGW∆ , r ( µ , ν ) = ( Eθ∼σq−1 [ GWrr ( ( ∆ # µ ) θ , νθ ) ] ) 1r . ( 4 ) It is worthy to note that SGW∆ , r is depending on the ad-hoc operator ∆ , hence the rotation invariance is lost . Vayer et al . ( 2019 ) propose a variant of SGW that does not depend on the choice of ∆ , called Rotation Invariant SGW ( RI-SGW ) for p = q , defined as the minimizer of SGW∆ , r over the Stiefel manifold , see ( Vayer et al. , 2019 , Equation 6 ) . In this work , we are interested in calculating an OT-based discrepancy between distributions over distinct domains using the slicing technique . Our approach is different from the SGW one in many points , specifically ( and most importantly ) we use a 1D-Wasserstein distance between the projected pushforward distributions and not a 1D-GW distance . In the next section , we detail the setup of our approach .

Summary: In the paper, the authors proposed heterogeneous Wasserstein discrepancy (HWD) to compare incomparable distributions. The main idea of HWD is via distributional slicing, which had been employed in the work of Nguyen et al. [2020].