Potential Flow Generator with $L_2$ Optimal Transport Regularity for Generative Models

1 INTRODUCTION . Many of the generative models , for example , generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Arjovsky et al. , 2017 ; Salimans et al. , 2018 ) and normalizing flow models ( Rezende & Mohamed , 2015 ; Kingma & Dhariwal , 2018 ; Chen et al. , 2018 ) , aim to find a generator that could map the input distribution to the target distribution . In many cases , especially when the input distributions are purely noises , the specific maps between input and output are of little importance as long as the generated distributions match the target ones . However , in other cases like imageto-image translations , where both input and target distributions are distributions of images , the generators are required to have additional regularity such that the input individuals are mapped to the “ corresponding ” outputs in some sense . If paired input-output samples are provided , Lp penalty could be hybridized into generators loss functions to encourage the output individuals to fit the ground truth ( Isola et al. , 2017 ) . For the cases without paired data , a popular approach is to introduce another generator and encourage the two generators to be the inverse maps of each other , as in CycleGAN ( Zhu et al. , 2017 ) , DualGAN ( Yi et al. , 2017 ) and DiscoGAN ( Kim et al. , 2017 ) , etc . However , such a pair of generators is not unique and lacks clear mathematical interpretation about its effectiveness . In this paper we introduce a special generator , i.e. , the potential flow generator , with L2 optimal transport regularity . By applying such generator , not only are we trying to find a map from the input distribution to the target one , but we also aim to find the optimal transport map that minimizes the squared Euclidean transport distance . In Figure 1 we provide a schematic comparison between generators with and without optimal transport regularity . While both generators provide a scheme to map from the input distribution to the output distribution , the total squared transport distances in the left generator is larger than that in the right generator . Note that the generator with optimal transport regularity has the characteristic of “ proximity ” in that the inputs tend to be mapped to nearby outputs . As we will show later , this “ proximity ” characteristic of L2 optimal transport regularity could be utilized in image translation tasks . Compared with other approaches like CycleGAN , the L2 optimal transport regularity has a much clearer mathematical interpretation . There have been other approaches to learn the optimal transport map in generative models . For example , Seguy et al . ( 2017 ) proposed to first learn the regularized optimal transport plan and then the optimal transport map , based on the dual form of regularized optimal transport problem . Also , Yang & Uhler ( 2018 ) proposed to learn the unbalanced optimal transport plan in an adversarial way derived from a convex conjugate representation of divergences . In the W2GAN model proposed by Leygonie et al . ( 2019 ) , the discriminator ’ s objective is the 2-Wasserstein metric so that the generator is supposed to recover the L2 optimal transport map . All the above approaches need to introduce , and are limited to , specific loss functions to train the generators . Our proposed potential flow generator takes a different approach in that with only a slight augmentation to the original generator loss functions , our generator could be integrated into a wide range of generative models with various generator loss functions , including different versions of GANs and normalizing flow models . This simple modification makes our method easy to adopt on various tasks considering the existing rich literature and the future developments of generative models . In Section 2 we present a formal definition of optimal transport map and the motivation to apply L2 optimal transport regularity to generators . In Section 3 we give a detailed formulation of the potential flow generator and the augmentation to the original loss functions . Results are then provided in Section 4 . We include the discussion and conclusions in Section 5 . 2 GENERATIVE MODELS AND OPTIMAL TRANSPORT MAP . First , we introduce the concept of push forward , which will be used extensively in the paper . Definition 1 Given two Polish space X and Y , B ( X ) and B ( Y ) the Borel σ-algebra on X and Y , and P ( X ) , P ( Y ) the set of probability measures on B ( X ) and B ( Y ) . Let f : X→ Y be a Borel map , and µ ∈ P ( X ) . We define f # µ ∈ P ( Y ) , the push forward of µ through f , by f # µ ( A ) = µ ( f−1 ( A ) ) , ∀A ∈ B ( Y ) . ( 1 ) With the concept of push forward , we can formulate the goal of GANs and normalizing flow models as to train the generator G such that G # µ is equal to or at least close to ν in some sense , where µ and ν are the input and target distribution , respectively . Usually , the loss functions for training the generators are metrics of closeness that vary for different models . For example , in continuous normalizing flows ( Chen et al. , 2018 ) , such metric of closeness is DKL ( G # µ||ν ) or DKL ( ν||G # µ ) . In Wasserstein GANs ( WGANs ) ( Arjovsky et al. , 2017 ) , the metric of closeness is the Wasserstein-1 distance between G # µ and ν , which is estimated in a variational form with the discriminator neural network . As a result , the generator and discriminator neural networks are trained in an adversarial way : min G max D is 1-Lipschitz Ex∼νD ( x ) − Ez∼µD ( G ( z ) ) , ( 2 ) where D is the discriminator neural network and the Lipschitz constraint could be imposed via the gradient penalty ( Gulrajani et al. , 2017 ) , spectral normalization ( Miyato et al. , 2018 ) , etc . Now we introduce the concept of optimal transport map as follows : Definition 2 Given a cost function c : X×Y→ R , and µ ∈ P ( X ) , ν ∈ P ( Y ) , we let T be the set of all transport maps from µ to ν , i.e . T : = { f : f # µ = ν } . Monge ’ s optimal transport problem is to minimize the cost functional C ( f ) among T , where C ( f ) = Ex∼µc ( x , f ( x ) ) ( 3 ) and the minimizer f∗ ∈ T is called the optimal transport map . In this paper , we are concerned mostly with the case where X = Y = Rd with L2 transport cost , i.e. , the transport c ( x , y ) = ‖x− y‖2 . We assume that µ and ν are absolute continuous w.r.t . Lebesgue measure , i.e . they have probability density functions . In general , Monge ’ s problem could be ill-posed in that T could be empty set or there is no minimizer in T. Also , the optimal transport map could be non-unique . However , for the special case we consider , there exists a unique solution to Monge ’ s problem ( Brenier , 1991 ; Gangbo & McCann , 1996 ) . Informally speaking , with L2 transport cost the optimal transport map has the characteristic of “ proximity ” , i.e . the inputs tend to be mapped to nearby outputs . In image translation tasks , such “ proximity ” characteristic would be helpful if we could properly embed the images into Euclidean space such that our preferred input-output pairs are close to each other . A similar idea is also proposed in Yang & Uhler ( 2018 ) for unbalanced optimal transport . Apart from image translations , the L2 optimal transport problem is important in many other aspects . For example , it is closely related to gradient flow ( Ambrosio et al. , 2008 ) , Fokker-Planck equations ( Santambrogio , 2017 ) , flow in porous medium ( Otto , 1997 ) , etc . 3 POTENTIAL FLOW GENERATOR . 3.1 POTENTIAL FLOW FORMULATION OF OPTIMAL TRANSPORT MAP . We assume that µ and ν have probability density ρµ and ρν , respectively , and consider all smooth enough density fields ρ ( t , x ) and velocity fields v ( t , x ) , where t ∈ [ 0 , T ] , subject to the continuity equation as well as initial and final conditions ∂tρ+∇ · ( ρv ) = 0 , ρ ( 0 , · ) = ρµ , ρ ( T , · ) = ρν . ( 4 ) The above equation states that such velocity field will induce a transport map : we can construct an ordinary differential equation ( ODE ) du dt = v ( t , u ) , ( 5 ) and the map from the initial point to the final point gives the transport map from µ to ν . As is proposed by Benamou & Brenier ( 2000 ) , for the transport cost function c ( x , y ) = ‖x− y‖2 , the minimal transport cost is equal to the infimum of T ∫ Rd ∫ T 0 ρ ( t , x ) |v ( t , x ) |2dxdt ( 6 ) among all ( ρ , v ) satisfying equation ( 4 ) . The optimality condition is given by v ( t , x ) = ∇φ ( t , x ) , ∂tφ+ 1 2 |∇φ|2 = 0 . ( 7 ) In other words , the optimal velocity field is actually induced from a flow with time-dependent potential φ ( t , x ) . The use of this formulation is well-known in optimal transport community ( Trigila & Tabak , 2016 ; Peyré et al. , 2019 ) . In this paper we integrate this formulation in the deep generative models . Instead of solving Monge ’ s problem and find the exact L2 optimal transport map , which is unrealistic due to the limited families of neural network functions as well as the errors arising from training the neural networks , our goal is to regularize the generators in a wide range of generative models , so that the generator maps could approximate the L2 optimal transport map at least in low dimensional problems . The maps would also be endowed with the characteristics of “ proximity ” so that we can apply them to engineering problems . 3.2 POTENTIAL FLOW GENERATOR . The potential φ ( t , x ) is the key function to estimate , since the velocity field could be obtained by taking the gradient of the potential and consequently the transport map could be obtained from Equation 5 . There are two strategies to use neural networks to represent φ . One can take advantage of the fact that the time-dependent potential field φ is actually uniquely determined by its initial condition from Equation 7 , and use a neural network to represent the initial condition of φ , i.e . φ ( 0 , x ) , while approximating φ ( t , x ) via time discretization schemes . Alternatively , one can use a neural network to represent φ ( t , x ) directly and later apply the PDE regularity for φ ( t , x ) in Equation 7 . We name the generators defined in the above two approaches as discrete potential flow generator and continuous potential flow generator , respectively , and give a detailed formulation as follows .

Summary: This is a great paper using optimal transport theory for generative and implicit models. Instead of using general vector fields, the authors apply the potential vector fields in optimal transport theory to design neural networks. The mathematics is correct with convincing examples. This brings an important mathematical connection between fluid dynamics and GANs or implicit models.