Potential Flow Generator with $L_2$ Optimal Transport Regularity for Generative Models

1 INTRODUCTION . Many of the generative models , for example , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ; Salimans et al. , 2018 ) and normalizing flow models ( Rezende & Mohamed , 2015 ; Kingma & Dhariwal , 2018 ; Chen et al. , 2018 ) , aim to find a generator that could map the input distribution to the target distribution . In many cases , especially when the input distributions are purely noises , the specific maps between input and output are of little importance as long as the generated distributions match the target ones . However , in other cases like imageto-image translations , where both input and target distributions are distributions of images , the generators are required to have additional regularity such that the input individuals are mapped to the “ corresponding ” outputs in some sense . If paired input-output samples are provided , Lp penalty could be hybridized into generators loss functions to encourage the output individuals to fit the ground truth ( Isola et al. , 2017 ) . For the cases without paired data , a popular approach is to introduce another generator and encourage the two generators to be the inverse maps of each other , as in CycleGAN ( Zhu et al. , 2017 ) , DualGAN ( Yi et al. , 2017 ) and DiscoGAN ( Kim et al. , 2017 ) , etc . However , such a pair of generators is not unique and lacks clear mathematical interpretation about its effectiveness . In this paper we introduce a special generator , i.e. , the potential flow generator , with L2 optimal transport regularity . By applying such generator , not only are we trying to find a map from the input distribution to the target one , but we also aim to find the optimal transport map that minimizes the squared Euclidean transport distance . In Figure 1 we provide a schematic comparison between generators with and without optimal transport regularity . While both generators provide a scheme to map from the input distribution to the output distribution , the total squared transport distances in the left generator is larger than that in the right generator . Note that the generator with optimal transport regularity has the characteristic of “ proximity ” in that the inputs tend to be mapped to nearby outputs . As we will show later , this “ proximity ” characteristic of L2 optimal transport regularity could be utilized in image translation tasks . Compared with other approaches like CycleGAN , the L2 optimal transport regularity has a much clearer mathematical interpretation . There have been other approaches to learn the optimal transport map in generative models . For example , Seguy et al . ( 2017 ) proposed to first learn the regularized optimal transport plan and then the optimal transport map , based on the dual form of regularized optimal transport problem . Also , Yang & Uhler ( 2018 ) proposed to learn the unbalanced optimal transport plan in an adversarial way derived from a convex conjugate representation of divergences . In the W2GAN model proposed by Leygonie et al . ( 2019 ) , the discriminator ’ s objective is the 2-Wasserstein metric so that the generator is supposed to recover the L2 optimal transport map . All the above approaches need to introduce , and are limited to , specific loss functions to train the generators . Our proposed potential flow generator takes a different approach in that with only a slight augmentation to the original generator loss functions , our generator could be integrated into a wide range of generative models with various generator loss functions , including different versions of GANs and normalizing flow models . This simple modification makes our method easy to adopt on various tasks considering the existing rich literature and the future developments of generative models . In Section 2 we present a formal definition of optimal transport map and the motivation to apply L2 optimal transport regularity to generators . In Section 3 we give a detailed formulation of the potential flow generator and the augmentation to the original loss functions . Results are then provided in Section 4 . We include the discussion and conclusions in Section 5 . 2 GENERATIVE MODELS AND OPTIMAL TRANSPORT MAP . First , we introduce the concept of push forward , which will be used extensively in the paper . Definition 1 Given two Polish space X and Y , B ( X ) and B ( Y ) the Borel σ-algebra on X and Y , and P ( X ) , P ( Y ) the set of probability measures on B ( X ) and B ( Y ) . Let f : X→ Y be a Borel map , and µ ∈ P ( X ) . We define f # µ ∈ P ( Y ) , the push forward of µ through f , by f # µ ( A ) = µ ( f−1 ( A ) ) , ∀A ∈ B ( Y ) . ( 1 ) With the concept of push forward , we can formulate the goal of GANs and normalizing flow models as to train the generator G such that G # µ is equal to or at least close to ν in some sense , where µ and ν are the input and target distribution , respectively . Usually , the loss functions for training the generators are metrics of closeness that vary for different models . For example , in continuous normalizing flows ( Chen et al. , 2018 ) , such metric of closeness is DKL ( G # µ||ν ) or DKL ( ν||G # µ ) . In Wasserstein GANs ( WGANs ) ( Arjovsky et al. , 2017 ) , the metric of closeness is the Wasserstein-1 distance between G # µ and ν , which is estimated in a variational form with the discriminator neural network . As a result , the generator and discriminator neural networks are trained in an adversarial way : min G max D is 1-Lipschitz Ex∼νD ( x ) − Ez∼µD ( G ( z ) ) , ( 2 ) where D is the discriminator neural network and the Lipschitz constraint could be imposed via the gradient penalty ( Gulrajani et al. , 2017 ) , spectral normalization ( Miyato et al. , 2018 ) , etc . Now we introduce the concept of optimal transport map as follows : Definition 2 Given a cost function c : X×Y→ R , and µ ∈ P ( X ) , ν ∈ P ( Y ) , we let T be the set of all transport maps from µ to ν , i.e . T : = { f : f # µ = ν } . Monge ’ s optimal transport problem is to minimize the cost functional C ( f ) among T , where C ( f ) = Ex∼µc ( x , f ( x ) ) ( 3 ) and the minimizer f∗ ∈ T is called the optimal transport map . In this paper , we are concerned mostly with the case where X = Y = Rd with L2 transport cost , i.e. , the transport c ( x , y ) = ‖x− y‖2 . We assume that µ and ν are absolute continuous w.r.t . Lebesgue measure , i.e . they have probability density functions . In general , Monge ’ s problem could be ill-posed in that T could be empty set or there is no minimizer in T. Also , the optimal transport map could be non-unique . However , for the special case we consider , there exists a unique solution to Monge ’ s problem ( Brenier , 1991 ; Gangbo & McCann , 1996 ) . Informally speaking , with L2 transport cost the optimal transport map has the characteristic of “ proximity ” , i.e . the inputs tend to be mapped to nearby outputs . In image translation tasks , such “ proximity ” characteristic would be helpful if we could properly embed the images into Euclidean space such that our preferred input-output pairs are close to each other . A similar idea is also proposed in Yang & Uhler ( 2018 ) for unbalanced optimal transport . Apart from image translations , the L2 optimal transport problem is important in many other aspects . For example , it is closely related to gradient flow ( Ambrosio et al. , 2008 ) , Fokker-Planck equations ( Santambrogio , 2017 ) , flow in porous medium ( Otto , 1997 ) , etc . 3 POTENTIAL FLOW GENERATOR . 3.1 POTENTIAL FLOW FORMULATION OF OPTIMAL TRANSPORT MAP . We assume that µ and ν have probability density ρµ and ρν , respectively , and consider all smooth enough density fields ρ ( t , x ) and velocity fields v ( t , x ) , where t ∈ [ 0 , T ] , subject to the continuity equation as well as initial and final conditions ∂tρ+∇ · ( ρv ) = 0 , ρ ( 0 , · ) = ρµ , ρ ( T , · ) = ρν . ( 4 ) The above equation states that such velocity field will induce a transport map : we can construct an ordinary differential equation ( ODE ) du dt = v ( t , u ) , ( 5 ) and the map from the initial point to the final point gives the transport map from µ to ν . As is proposed by Benamou & Brenier ( 2000 ) , for the transport cost function c ( x , y ) = ‖x− y‖2 , the minimal transport cost is equal to the infimum of T ∫ Rd ∫ T 0 ρ ( t , x ) |v ( t , x ) |2dxdt ( 6 ) among all ( ρ , v ) satisfying equation ( 4 ) . The optimality condition is given by v ( t , x ) = ∇φ ( t , x ) , ∂tφ+ 1 2 |∇φ|2 = 0 . ( 7 ) In other words , the optimal velocity field is actually induced from a flow with time-dependent potential φ ( t , x ) . The use of this formulation is well-known in optimal transport community ( Trigila & Tabak , 2016 ; Peyré et al. , 2019 ) . In this paper we integrate this formulation in the deep generative models . Instead of solving Monge ’ s problem and find the exact L2 optimal transport map , which is unrealistic due to the limited families of neural network functions as well as the errors arising from training the neural networks , our goal is to regularize the generators in a wide range of generative models , so that the generator maps could approximate the L2 optimal transport map at least in low dimensional problems . The maps would also be endowed with the characteristics of “ proximity ” so that we can apply them to engineering problems . 3.2 POTENTIAL FLOW GENERATOR . The potential φ ( t , x ) is the key function to estimate , since the velocity field could be obtained by taking the gradient of the potential and consequently the transport map could be obtained from Equation 5 . There are two strategies to use neural networks to represent φ . One can take advantage of the fact that the time-dependent potential field φ is actually uniquely determined by its initial condition from Equation 7 , and use a neural network to represent the initial condition of φ , i.e . φ ( 0 , x ) , while approximating φ ( t , x ) via time discretization schemes . Alternatively , one can use a neural network to represent φ ( t , x ) directly and later apply the PDE regularity for φ ( t , x ) in Equation 7 . We name the generators defined in the above two approaches as discrete potential flow generator and continuous potential flow generator , respectively , and give a detailed formulation as follows .

Summary: The paper proposes a ‘potential flow generator’ that can be seen as a regularizer for traditional GAN losses. It is based on the idea that samples flowing from one distribution to another should follow a minimum travel cost path. This regularization is expressed as an optimal transport problem with a squared Euclidean cost. Authors rely on the dynamic formulation of OT proposed by Benamou and Brenier, 2000. They propose to learn a time-dependent potential field which gradient defines the velocity fields used to drive samples from a source distribution toward a target one. Experiments on a simple 1D case (where the optimal transport map is known), and on images with an MNIST / CelebA qualitative example.