Heterogeneous Wasserstein Discrepancy for Incomparable Distributions

1 INTRODUCTION . Optimal Transport-based data analysis has recently found widespread interest in machine learning community , since its significant usefulness to achieve many tasks arising from designing loss functions in supervised learning ( Frogner et al. , 2015 ) , unsupervised learning ( Arjovsky et al. , 2017 ) , text classification ( Kusner et al. , 2015 ) , domain adaptation ( Courty et al. , 2017 ) , generative models ( Arjovsky et al. , 2017 ; Salimans et al. , 2018 ) , computer vision ( Bonneel et al. , 2011 ; Solomon et al. , 2015 ) among many more applications ( Kolouri et al. , 2017 ; Peyré & Cuturi , 2019 ) . Optimal Transport ( OT ) attempts to match real-world entities through computing distances between distributions , and for that it exploits prior geometric knowledge on the base spaces in which the distributions are valued . Computing OT distance equals to finding the most cost-efficiency way to transport mass from source distribution to target distribution , and it is often referred to as the Monge-Kantorovich or Wasserstein distance ( Monge , 1781 ; Kantorovich , 1942 ; Villani , 2009 ) . Matching distributions using Wasserstein distance relies on the assumption that their base spaces must be the same , or that at least a meaningful pairwise distance between the supports of these distributions can be computed . A variant of Wasserstein distance dealing with heterogeneous distributions and overcoming the lack of intrinsic correspondence between their base spaces is Gromov-Wasserstein ( GW ) distance ( Sturm , 2006 ; Mémoli , 2011 ) . GW distance allows to learn an optimal transport-like plan by measuring how the similarity distances between pairs of supports within each ground space are closed . It is increasingly finding applications for learning problems in shape matching ( Mémoli , 2011 ) , graph partitioning and matching ( Xu et al. , 2019 ) , matching of vocabulary sets between different languages ( Alvarez-Melis & Jaakkola , 2018 ) , generative models ( Bunne et al. , 2019 ) , or matching weighted networks ( Chowdhury & Mémoli , 2018 ) . Due to the heterogeneity of the distributions , GW distance uses only the relational aspects in each domain , such as the pairwise relationships to compare the two distributions . As a consequence , the main disadvantage of GW distance is its computational cost as the associated optimization problem is a non-convex quadratic program ( Peyré & Cuturi , 2019 ) , and as few as thousand samples can be computationally challenging . Based on the approach of regularized OT ( Cuturi , 2013 ) , in which an entropic penalty is added to the original objective function defining the Wasserstein OT problem , Peyré et al . ( 2016 ) propose an entropic version called entropic GW discrepancy , that leads to approximate GW distance . Another approach for scaling up the GW distance is Sliced Gromov-Wasserstein ( SGW ) discrepancy ( Vayer et al. , 2019 ) , which leverages on random projections on 1D and on a closed-form solution of the 1D-Gromov-Wasserstein . In this paper , we take a different approach for measuring the discrepancy between two heterogeneous distributions . Unlike GW distance that compares pairwise distances of elements from each distribution , we consider a method that embeds the metric measure spaces into a one-dimensional space and computes a Wasserstein distance between the two 1D-projected distributions . The key element of our approach is to learn two mappings that transform vectors from the unit-sphere of a latent space to the unit-sphere of the metric space underlying the two distributions of interest , see Figure 1 . In a nutshell , we learn to transform a random direction , sampled under an optimal ( learned ) distribution ( optimality being made clear later ) , from a d-dimensional space to a random direction into the desired spaces . This approach has the benefit of avoiding an ad-hoc padding strategy ( completion of 0 of the smaller dimension distributions to fit the high-dimensional one ) as in SGW method ( Vayer et al. , 2019 ) . Another relevant feature of our approach is that the two resulting 1D distributions are now compared through Wasserstein distance . This point , in conjunction , with other key aspect of the method , will lead to a relevant discrepancy between two distributions , called heterogeneous Wasserstein discrepancy ( HWD ) . Although we lose some properties of a distance , we show that HWD is rotation-invariant , that it is robust enough to be considered as a loss for learning generative models between heterogeneous spaces . We also establish that HWD boils down to the recent distributional sliced Wasserstein distance ( Nguyen et al. , 2020 ) if the two distributions live in the same space and if some mild constraints are imposed on the mappings . In summary , our contributions are as follows : • we propose HWD , a novel slicing-based discrepancy for comparing two distributions living in different spaces . Our chosen formulation is based on comparing 1D random-projected versions of the two distributions using a Wasserstein distance ; • The projection operations are materialized by optimally mapping from one common space to the two spaces of interest . We provide a theoretical analysis of the resulting discrepancy and exhibit its relevant properties ; • Since the discrepancy involves several mappings that need to be optimized , we depict an alternate optimization algorithm for learning them ; • Numerically , we validate the benefits of HWD in terms of comparison between heterogeneous distributions . We show that it can be used as a loss for generative models or shape objects retrieval with better performance and robustness than SGW on those tasks . 2 BACKGROUND OF OT DISTANCES . For the reader ’ s convenience , we provide here a brief review of the notations and definitions , that will be frequently used throughout the paper . We start by introducing Wasserstein and Gromov-Wasserstein distances with their sliced versions SW and SGW , where we consider these distances in the specific case of Euclidean base spaces ( Rp , ‖ · ‖ ) and ( Rq , ‖ · ‖ ) . We denote P ( X ) and P ( Y ) the respective sets of probability measures whose supports are contained on compact sets X ⊆ Rp and Y ⊆ Rq . For r ≥ 1 , we denote Pr ( X ) the subset of measures in P ( X ) with finite r-th moment ( r ≥ 1 ) , i.e. , Pr ( X ) = { η ∈ P ( X ) : ´ X ‖x‖ rdη ( x ) < ∞ } . For µ ∈ P ( X ) and ν ∈ P ( Y ) , we write Π ( µ , ν ) ⊂ P ( X × Y ) for the collection of joint probability distributions with marginals µ and ν , known as couplings , Π ( µ , ν ) = { γ ∈P ( X × Y ) : ∀A ⊂ X , B ⊂ Y , γ ( A× Y ) = µ ( A ) , γ ( X ×B ) = ν ( B ) } . 2.1 OT DISTANCES FOR HOMOGENEOUS DOMAINS . We here assume that the distributions µ and ν lie in the same base space , for instance p = q . Taking this into account , we can define the Wasserstein distance and its sliced variant . Wasserstein distance The r-th Wasserstein distance is defined on Pr ( X ) by Wr ( µ , ν ) = ( inf γ∈Π ( µ , ν ) ˆ X×Y ‖x− y‖rdγ ( x , y ) ) 1 r . ( 1 ) The quantityWr ( µ , ν ) describes the least amount effort to transform one distribution µ into another one ν . Since the cost distance used between sample supports is the Euclidean one , the infimum in ( 1 ) is attained ( Villani , 2009 ) , and any probability γ which realizes the minimum is called an optimal transport plan . In a finite discrete setting , Problem ( 1 ) can be formulated as a linear program , that is challenging to solve algorithmically as its computational cost is of order O ( n5/2 log n ) ( Lee & Sidford , 2014 ) , where n is the number of sample supports . Contrastingly , for the 1D case ( i.e . p = 1 ) of continuous probability measures , the r-th Wasserstein distance has a closed-form solution ( Rachev & Rüschendorf , 1998 ) , namely , Wr ( µ , ν ) = ( ´ 1 0 |F−1µ ( u ) − F−1ν ( u ) |rdt ) 1 r where F−1µ and F −1 ν are the quantile functions of µ and ν . For empirical distributions , the 1D-Wasserstein distance is simply calculated by sorting the supports of the distributions on the real line , resulting to a complexity of order O ( n log n ) . This nice computational property motivates the use of sliced-Wasserstein ( SW ) distance ( Rabin et al. , 2012 ; Bonneel et al. , 2015 ) , where one calculates an ( infinity ) of 1D-Wasserstein distances between linear projection pushforwards of distributions in question and then computes their average . To precisely define SW distance , we consider the following notation . Let Sp−1 : = { u ∈ Rp : ‖u‖ = 1 } be the unit sphere in p dimension in ` 2-norm , and for any vector θ in Sp−1 , we define Pθ the orthogonal projection onto the real line Rθ = { αθ : α ∈ R } , that is Pθ ( x ) = 〈θ , x〉 , where 〈· , ·〉 stands for the Euclidean inner-product . Let µθ = Pθ # µ the measure on the real line called pushforward of µ by Pθ , that is µθ ( A ) = µ ( P−1θ ( A ) ) for all Borel set A ⊆ R. We may now define the SW distance . Sliced Wasserstein distance The r-th order sliced Wasserstein distance between two probability distributions µ , ν ∈Pr ( X ) is given by SWr ( µ , ν ) = ( 1 Ap ˆ Sp−1 Wrr ( µθ , νθ ) dθ ) 1 r , ( 2 ) where Ap is the area of the surface of Sp−1 , i.e. , Ap = 2π p/2 Γ ( p/2 ) with Γ : R → R , the Gamma function given as Γ ( u ) = ´∞ 0 tu−1e−tdt . Thanks to its computational benefits and its valid metric property ( Bonnotte , 2013 ) , the SW distance has recently been used for OT-based deep generative modeling ( Kolouri et al. , 2019 ; Deshpande et al. , 2019 ; Wu et al. , 2019 ) . Note that the normalized integral in ( 2 ) can be seen as the expectation for θ ∼ σp−1 , the uniform surface measure on Sp−1 , that is SWr ( µ , ν ) = ( Eθ∼σp−1 [ Wrr ( µθ , νθ ) ] ) 1 r . Therefore , the SW distance can be easily approximated via a Monte Carlo sampling scheme by drawing uniform random samples from Sp−1 : SWrr ( µ , ν ) ≈ 1K ∑K k=1Wrr ( µθk , νθk ) where θ1 , . . . , θK i.i.d.∼ σp−1 and K is the number of random projections . 2.2 OT DISTANCES FOR HETEROGENEOUS DOMAINS . To get benefit from the advantages of OT in many machine learning applications involving heterogeneous and incomparable domains ( p 6= q ) , the Gromov-Wasserstein distance ( Mémoli , 2011 ) stands for the basic OT distance dealing with this setting . Gromov-Wasserstein distance The r-th Gromov-Wasserstein distance between two probability distributions µ ∈Pr ( X ) and ν ∈Pr ( Y ) is defined by GWr ( µ , ν ) = inf γ∈Π ( µ , ν ) Jr ( γ ) def . = 1 2 ( ¨ X 2×Y2 |‖x− x′‖ − ‖y − y′‖|rdγ ( x , y ) dγ ( x′ , y′ ) ) 1 r . ( 3 ) Note that GWr ( µ , ν ) is a valid metric endowing the collection of all isomorphism classes metric measure spaces of Pr ( X ) ×Pr ( Y ) , see Theorem 5 in ( Mémoli , 2011 ) . The GW distance learns an optimal transport-like plan which transports samples from a source metric space X into a target metric space Y , by measuring how the similarity distances between pairs of samples within each space are close . Furthermore , GW distance enjoys several geometric properties , particularly translation and rotation invariance . However , its major bottleneck consists in an expensive computational cost , since problem ( 3 ) is non-convex and quadratic . A remedy to such a heavy computational burden lies in an entropic regularized GW discrepancy ( Peyré et al. , 2016 ) , using Sinkhorn iterations algorithm ( Cuturi , 2013 ) . This latter needs a large regularization parameter to guarantee a fast computation , which , unfortunately , entails a poor approximation of the true GW distance value . Another approach to scale up the computation of GW distance is sliced-GW discrepancy ( Vayer et al. , 2019 ) . The definition of SGW shows 1D-GW distances between projected pushforward of an artifact zero padding of µ or ν distribution . We detail this representation in the following paragraph . Sliced Gromov-Wasserstein discrepancy Assume that p < q and let ∆ be an artifact zero padding from X onto Y , i.e . ∆ ( x ) = ( x1 , . . . , xp , 0 , . . . , 0 ) ∈ Rq . The r-th order sliced Gromov-Wasserstein discrepancy between two probability distributions µ ∈Pr ( X ) and ν ∈Pr ( Y ) is given by SGW∆ , r ( µ , ν ) = ( Eθ∼σq−1 [ GWrr ( ( ∆ # µ ) θ , νθ ) ] ) 1r . ( 4 ) It is worthy to note that SGW∆ , r is depending on the ad-hoc operator ∆ , hence the rotation invariance is lost . Vayer et al . ( 2019 ) propose a variant of SGW that does not depend on the choice of ∆ , called Rotation Invariant SGW ( RI-SGW ) for p = q , defined as the minimizer of SGW∆ , r over the Stiefel manifold , see ( Vayer et al. , 2019 , Equation 6 ) . In this work , we are interested in calculating an OT-based discrepancy between distributions over distinct domains using the slicing technique . Our approach is different from the SGW one in many points , specifically ( and most importantly ) we use a 1D-Wasserstein distance between the projected pushforward distributions and not a 1D-GW distance . In the next section , we detail the setup of our approach .

Summary: The authors propose a variant of Wasserstein for distributions supported on different Euclidean spaces (e.g., different dimensions). The main ideas are to combine the embedding (for distributions supported in different spaces -- Alaya et al. 2020) and distributional sliced Wasserstein (Nguyen et al., 2020) to generate random slicing projections into 1d-space, and leverage the 1d-Wasserstein for the comparison. The authors apply the proposed method for generating modeling and in query framework.