How Well Do WGANs Estimate the Wasserstein Metric?

1 INTRODUCTION . Recently , optimal transport ( OT ) has become increasingly prevalent in computer vision and machine learning , as it allows for robust comparison of structured data that can be cast as probability measures , e.g. , images , point clouds and empirical distributions ( Rubner et al. , 2000 ; Lai & Zhao , 2014 ) , or more general measures ( Gangbo et al. , 2019 ) . Key properties of OT include its non-singular behavior , when comparing measures with disjoint supports , and the fact that OT inspired objectives can be seen as lifting similarity measures between samples to similarity measures between probability measures . This is in stark contrast to the more traditional information theoretical divergences , which rely on only comparing the difference in mass assignment . Additionally , when the cost function c is related to a distance function d by c ( x , y ) = dp ( x , y ) , p ≥ 1 , the OT formulation defines the so called Wasserstein metric , which is a distance on the space of probability measures , i.e . a symmetric and positive definite function that satisfies the triangle inequality . Despite its success , scaling OT to big data applications has not been without challenges , since it suffers from the curse of dimensionality ( Dudley , 1969 ; Weed & Bach , 2017 ) . However , significant computational advancements have been made recently , for which a summary is given by Peyré et al . ( 2019 ) . Notably , the entropic regularization of OT introduced by Cuturi ( 2013 ) preserves the geometrical structure endowed by the Wasserstein spaces and provides an efficient way to approximate optimal transport between measures . Generative modelling , where generators are trained for sampling from given data distributions , is a popular application of OT . In this field , generative adversarial networks ( GANs ) by Goodfellow et al . ( 2014 ) have attracted substantial interest , particularly due to their success in generating photorealistic images ( Karras et al. , 2017 ) . The original GAN formulation minimizes the Jensen-Shannon divergence between a model distribution and the data distribution , which suffers from unstable training . Wasserstein GANs ( WGANs ) ( Arjovsky et al. , 2017 ) minimize the 1-Wasserstein distance over the l2-metric , instead , resulting in more robust training . The main challenge in WGANs is estimating the Wasserstein metric , consisting of estimating expected values of the discriminator from samples ( drawn from a model distribution and a given data distribution ) , and optimizing the discriminator to maximize an expression of these expected values . The discriminators are functions from the sample space to the real line , that have different interpretations in different GAN variations . The main technical issue is that the discriminators have to satisfy specific conditions , such as being 1-Lipschitz in the 1-Wasserstein case . In the original paper , this was enforced by clipping the weights of the discriminator to lie inside some small box , which , however , proved to be inefficient . The Gradient penalty WGAN ( WGAN-GP ) ( Gulrajani et al. , 2017 ) was more successful at this , by enforcing the constraint through a gradient norm penalization . Another notable improvement was given by the consistency term WGAN ( CT-WGAN ) ( Wei et al. , 2018 ) , which penalizes diverging from 1-Lipschitzness directly . Other derivative work of the WGAN include different OT inspired similarity measures between distributions , such as the sliced Wasserstein distance ( Deshpande et al. , 2018 ) , the Sinkhorn divergence ( Genevay et al. , 2018 ) and the Wasserstein divergence ( Wu et al. , 2018 ) . Another line of work studies how to incorporate more general ground cost functions than the l2-metric ( Adler & Lunz , 2018 ; Mallasto et al. , 2019 ; Dukler et al. , 2019 ) . Recent works have studied the convergence of estimates of the Wasserstein distance between two probability distributions , both in the case of continuous ( Klein et al. , 2017 ) and finite ( Sommerfeld & Munk , 2018 ; Sommerfeld , 2017 ) sample spaces . The decay rate of the approximation error of estimating the true distance with minibatches of size N is of order O ( N−1/d ) for the Wasserstein distances , where d is the dimension of the sample space ( Weed & Bach , 2017 ) . Entropic regularized optimal transport has more favorable sample complexity of order O ( 1/ √ N ) for suitable choices of regularization strength ( see Genevay et al . 2019 and also Feydy et al . 2018 ; Mena & Weed 2019 ) . For this reason , entropic relaxation of the 1-Wasserstein distance is also considered in this work . Contribution . In this work , we study the efficiency and stability of computing the Wasserstein metric through its dual formulation under different schemes presented in the WGAN literature . We present a detailed discussion on how the different approaches arise and differ from each other qualitatively , and finally measure the differences in performance quantitatively . This is done by quantifying how much the estimates differ from accurately computed ground truth values between subsets of commonly used datasets . Finally , we measure how well the distance is approximated during the training of a generative model . This results in a surprising observation ; the method best approximating the Wasserstein distance does not produce the best looking images in the generative setting . 2 OPTIMAL TRANSPORT . In this section , we recall essential formulations of optimal transport to fix notation . Probabilistic Notions . Let ( X , dX ) and ( Y , dY ) be Polish spaces , i.e. , complete and separable metric spaces , denote by P ( X ) the set of probability measures on X , and let f : X → Y be a measurable map . Then , given a probability measure µ ∈ P ( X ) , we write f # µ for the push-forward of µ with respect to f , given by f # µ ( A ) = µ ( f−1 ( A ) ) for any measurable A ⊆ Y . Intuitively speaking , if ξ is a random variable with law µ , then f ( ξ ) has law f # µ . Then , given ν ∈ P ( Y ) , we define ADM ( µ , ν ) = { γ ∈ P ( X × Y ) | ( π1 ) # γ = µ , ( π2 ) # γ = ν } , ( 1 ) where πi denotes the projection onto the ith marginal . An element γ ∈ ADM ( µ , ν ) is called an admissible plan and defines a joint probability between µ and ν. Optimal Transport Problem . Given a continuous and lower-bounded cost function c : X ×Y → R , the optimal transport problem between probability measures µ ∈ P ( X ) and ν ∈ P ( Y ) is defined as OTc ( µ , ν ) : = min γ∈ADM ( µ , ν ) Eγ [ c ] , ( 2 ) where Eµ [ f ] = ∫ X f ( x ) dµ ( x ) is the expectation of a measurable function f with respect to µ . Note that ( 2 ) defines a constrained linear program , and thus admits a dual formulation . From the perspective of WGANs , the dual is more important than the primal formulation , as it can be approximated using discriminator neural networks . Denote by L1 ( µ ) = { f : X → R | Eµ [ f ] < ∞ } the set of measurable functions of µ that have finite expectations under µ , and by ADM ( c ) the set of admissible pairs ( ϕ , ψ ) that satisfy ϕ ( x ) + ψ ( y ) ≤ c ( x , y ) , ∀ ( x , y ) ∈ X × Y , ϕ ∈ L1 ( µ ) , ψ ∈ L1 ( ν ) . ( 3 ) Then , the following duality holds ( Peyré et al. , 2019 , Sec . 4 ) OTc ( µ , ν ) = sup ( ϕ , ψ ) ∈ADM ( c ) { Eµ [ ϕ ] + Eν [ ψ ] } . ( 4 ) When the supremum is attained , the optimal ϕ∗ , ψ∗ in ( 4 ) are called Kantorovich potentials , which , in particular , satisfy ϕ∗ ( x ) + ψ∗ ( y ) = c ( x , y ) for any ( x , y ) ∈ Supp ( γ∗ ) , where γ∗ solves ( 2 ) . Given ϕ , we can obtain an admissable ψ satisfying ( 4 ) through the c-transform of ϕ , ϕc : Y → R , y 7→ inf x∈X { c ( x , y ) − ϕ ( x ) } , ( 5 ) so that ( ϕ , ϕc ) ∈ ADM ( c ) for any ϕ ∈ L1 ( µ ) . Moreover , the Kantorovich potentials satisfy ψ = ϕc , and therefore ( 4 ) can be written as ( Villani , 2008 , Thm . 5.9 ) OTc ( µ , ν ) = max ( ϕ , ϕc ) ∈ADM ( c ) { Eµ [ ϕ ] + Eν [ ϕc ] } . ( 6 ) In other words , the ADM ( c ) constraint can be enforced with the c-transform , and reduces the optimization in ( 6 ) to be carried out over a single function . Wasserstein Metric . Consider the case when X = Y and c ( x , y ) = dpX ( x , y ) , p ≥ 1 , where we refer to dX as the ground metric . Then , the optimal transport problem ( 2 ) defines the p-Wasserstein metric Wp ( µ , ν ) : = OTdpX ( µ , ν ) 1 p on the space PpdX ( X ) = { µ ∈ P ( X ) ∣∣∣∣ ∫ dpX ( x0 , x ) dµ ( x ) < ∞ } , for some x0 ∈ X , ( 7 ) of probability measures with finite p-moments . It can be shown that ( PpdX ( X ) , Wp ) forms a complete , separable metric space ( Villani , 2008 , Sec . 6 , Thm 6.16 ) . Entropy Relaxed Optimal Transport . We can relax ( 2 ) by imposing entropic penalization introduced by Cuturi ( 2013 ) . Recall the definition of the Kullback-Leibler ( KL ) divergence from ν to µ as KL ( µ‖ν ) = ∫ X log ( pµ pν ) pµdχ , ( 8 ) where we assume that µ , ν are absolutely continuous with respect to the Lebesgue measure χ on X with densities pµ , pν , respectively . Using the KL-divergence as penalization , the entropy relaxed optimal transport is defined as OT c ( µ , ν ) : = min γ∈ADM ( µ , ν ) { Eγ [ c ] + KL ( γ‖µ⊗ ν ) } , ( 9 ) where > 0 defines the magnitude of the penalization , and µ ⊗ ν denotes the independent joint distribution of µ and ν . We remark that when → 0 , any minimizing sequence ( γ ) > 0 solving ( 9 ) converges to a minimizer of ( 2 ) , and in particular , OT c ( µ , ν ) → OTc ( µ , ν ) . Analogously to ( 4 ) , the entropy relaxed optimal transport admits the following dual formulation ( Peyré et al. , 2019 ; Feydy et al. , 2018 ; Di Marino & Gerolin , 2019 ) OT c ( µ , ν ) = sup ϕ∈L1 ( µ ) , ψ∈L1 ( ν ) { Eµ [ ϕ ] + Eν [ ψ ] − Eµ⊗ν [ exp ( −c+ ( ϕ⊕ ψ ) ) − 1 ] } , ( 10 ) where ( ϕ⊕ψ ) ( x , y ) = ϕ ( x ) +ψ ( y ) . In contrast to ( 4 ) , this is an unconstrained optimization problem , where the entropic penalization can be seen as a smooth relaxation of the constraint . As shown by Feydy et al . ( 2018 ) ; Di Marino & Gerolin ( 2019 ) , a similar approach to ( 6 ) for computing the Kantorovich potentials can be taken in the entropic case , by generalizing the c-transform . Let Lexp ( µ ) : = { g : X → R | Eµ [ exp ( g/ ) ] < ∞ } , and consider the ( c , ) -transform of ϕ ∈ Lexp ( µ ) , ϕ ( c , ) ( y ) = − log ( ∫ X exp ( −c ( x , y ) − ϕ ( x ) ) dµ ( x ) ) . ( 11 ) As → 0 , ϕ ( c , ) ( y ) → ϕc ( y ) , making the ( c , ) -transform consistent with the c-transform . Analogously to ( 6 ) , one can show under mild assumptions on the cost c ( Di Marino & Gerolin , 2019 ) , that OT c ( µ , ν ) = max ϕ∈Lexp ( µ ) { Eµ [ ϕ ] + Eν [ ϕ ( c , ) ] } . ( 12 ) Sinkhorn Divergence . Since the functional OT c fails to be positive-definite ( e.g . OT c ( µ , µ ) 6= 0 ) , it is convenient to introduce the ( p , ) -Sinkhorn divergence S p with parameter > 0 , given by S p ( µ , ν ) = OT dpX ( µ , ν ) 1 p − 1 2 ( OT dpX ( µ , µ ) 1 p + OT dpX ( ν , ν ) 1 p ) , ( 13 ) where the terms OT dpX ( µ , µ ) 1 p and OT dpX ( ν , ν ) 1 p are added to avoid bias , as in general OT dpX ( µ , µ ) 1 p 6= 0 . The Sinkhorn divergence was introduced by Genevay et al . ( 2018 ) , and has the following properties : ( i ) it metrizes weak convergence in the space of probability measures ; ( ii ) it interpolates between maximum mean discrepancy ( MMD ) , as →∞ , and the p-Wasserstein metric , as → 0 . For more about the Sinkhorn divergence , see Feydy et al . ( 2018 ) .

Summary: This paper provides an empirical evaluation of commonly used discriminator training strategies in estimating the Wasserstein distance between distributions. The paper finds that methods motivated from optimal transport theory, e.g. c-transform and (c,\eps)-transform, perform better in evaluating the Wasserstein distance than methods commonly used in WGAN practice such as weight clipping and gradient penalty. However, when deployed in WGANs as the discriminator training strategy, these methods do not generate images as high-quality as the gradient penalty method.