Year: 2019

Abstract:
We propose a potential flow generator with L2 optimal transport regularity, which can be easily integrated into a wide range of generative models including different versions of GANs and normalizing flow models. With only a slight augmentation to the original generator loss functions, our generator not only tries to transport the input distribution to the target one, but also aims to find the one with minimum L2 transport cost. We show the effectiveness of our method in several 2D problems, and illustrate the concept of “proximity” due to the L2 optimal transport regularity. Subsequently, we demonstrate the effectiveness of the potential flow generator in image translation tasks with unpaired training data from the MNIST dataset and the CelebA dataset with a comparison against vanilla WGAN-GP and CycleGAN.

heading: 1 INTRODUCTION
text: Many of the generative models, for example, generative adversarial networks (GANs) (Goodfellow et al., 2014; Arjovsky et al., 2017; Salimans et al., 2018) and normalizing flow models (Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018; Chen et al., 2018), aim to find a generator that could map the input distribution to the target distribution.
In many cases, especially when the input distributions are purely noises, the specific maps between input and output are of little importance as long as the generated distributions match the target ones. However, in other cases like imageto-image translations, where both input and target distributions are distributions of images, the generators are required to have additional regularity such that the input individuals are mapped to the “corresponding” outputs in some sense. If paired input-output samples are provided, Lp penalty could be hybridized into generators loss functions to encourage the output individuals to fit the ground truth (Isola et al., 2017). For the cases without paired data, a popular approach is to introduce another generator and encourage the two generators to be the inverse maps of each other, as in CycleGAN (Zhu et al., 2017), DualGAN (Yi et al., 2017) and DiscoGAN (Kim et al., 2017), etc. However, such a pair of generators is not unique and lacks clear mathematical interpretation about its effectiveness.
In this paper we introduce a special generator, i.e., the potential flow generator, with L2 optimal transport regularity. By applying such generator, not only are we trying to find a map from the input distribution to the target one, but we also aim to find the optimal transport map that minimizes the squared Euclidean transport distance. In Figure 1 we provide a schematic comparison between generators with and without optimal transport regularity. While both generators provide a scheme to map from the input distribution to the output distribution, the total squared transport distances in the
left generator is larger than that in the right generator. Note that the generator with optimal transport regularity has the characteristic of “proximity”in that the inputs tend to be mapped to nearby outputs. As we will show later, this “proximity” characteristic of L2 optimal transport regularity could be utilized in image translation tasks. Compared with other approaches like CycleGAN, the L2 optimal transport regularity has a much clearer mathematical interpretation.
There have been other approaches to learn the optimal transport map in generative models. For example, Seguy et al. (2017) proposed to first learn the regularized optimal transport plan and then the optimal transport map, based on the dual form of regularized optimal transport problem. Also, Yang & Uhler (2018) proposed to learn the unbalanced optimal transport plan in an adversarial way derived from a convex conjugate representation of divergences. In the W2GAN model proposed by Leygonie et al. (2019), the discriminator’s objective is the 2-Wasserstein metric so that the generator is supposed to recover the L2 optimal transport map. All the above approaches need to introduce, and are limited to, specific loss functions to train the generators.
Our proposed potential flow generator takes a different approach in that with only a slight augmentation to the original generator loss functions, our generator could be integrated into a wide range of generative models with various generator loss functions, including different versions of GANs and normalizing flow models. This simple modification makes our method easy to adopt on various tasks considering the existing rich literature and the future developments of generative models.
In Section 2 we present a formal definition of optimal transport map and the motivation to apply L2 optimal transport regularity to generators. In Section 3 we give a detailed formulation of the potential flow generator and the augmentation to the original loss functions. Results are then provided in Section 4. We include the discussion and conclusions in Section 5.

heading: 2 GENERATIVE MODELS AND OPTIMAL TRANSPORT MAP
text: First, we introduce the concept of push forward, which will be used extensively in the paper.
Definition 1 Given two Polish space X and Y, B(X) and B(Y) the Borel σ-algebra on X and Y, and P(X),P(Y) the set of probability measures on B(X) and B(Y). Let f : X→ Y be a Borel map, and µ ∈ P(X). We define f#µ ∈ P(Y), the push forward of µ through f , by
f#µ(A) = µ(f−1(A)),∀A ∈ B(Y). (1)
With the concept of push forward, we can formulate the goal of GANs and normalizing flow models as to train the generator G such that G#µ is equal to or at least close to ν in some sense, where µ and ν are the input and target distribution, respectively. Usually, the loss functions for training the generators are metrics of closeness that vary for different models. For example, in continuous normalizing flows (Chen et al., 2018), such metric of closeness is DKL(G#µ||ν) or DKL(ν||G#µ). In Wasserstein GANs (WGANs) (Arjovsky et al., 2017), the metric of closeness is the Wasserstein-1 distance between G#µ and ν, which is estimated in a variational form with the discriminator neural network. As a result, the generator and discriminator neural networks are trained in an adversarial way:
min G max D is 1-Lipschitz
Ex∼νD(x)− Ez∼µD(G(z)), (2)
where D is the discriminator neural network and the Lipschitz constraint could be imposed via the gradient penalty (Gulrajani et al., 2017), spectral normalization (Miyato et al., 2018), etc.
Now we introduce the concept of optimal transport map as follows:
Definition 2 Given a cost function c : X×Y→ R, and µ ∈ P(X), ν ∈ P(Y), we let T be the set of all transport maps from µ to ν, i.e. T := {f : f#µ = ν}. Monge’s optimal transport problem is to minimize the cost functional C(f) among T, where
C(f) = Ex∼µc(x, f(x)) (3) and the minimizer f∗ ∈ T is called the optimal transport map.
In this paper, we are concerned mostly with the case where X = Y = Rd with L2 transport cost, i.e., the transport c(x,y) = ‖x− y‖2. We assume that µ and ν are absolute continuous w.r.t. Lebesgue
measure, i.e. they have probability density functions. In general, Monge’s problem could be ill-posed in that T could be empty set or there is no minimizer in T. Also, the optimal transport map could be non-unique. However, for the special case we consider, there exists a unique solution to Monge’s problem (Brenier, 1991; Gangbo & McCann, 1996).
Informally speaking, with L2 transport cost the optimal transport map has the characteristic of “proximity”, i.e. the inputs tend to be mapped to nearby outputs. In image translation tasks, such “proximity” characteristic would be helpful if we could properly embed the images into Euclidean space such that our preferred input-output pairs are close to each other. A similar idea is also proposed in Yang & Uhler (2018) for unbalanced optimal transport. Apart from image translations, the L2 optimal transport problem is important in many other aspects. For example, it is closely related to gradient flow (Ambrosio et al., 2008), Fokker-Planck equations (Santambrogio, 2017), flow in porous medium (Otto, 1997), etc.

heading: 3 POTENTIAL FLOW GENERATOR

heading: 3.1 POTENTIAL FLOW FORMULATION OF OPTIMAL TRANSPORT MAP
text: We assume that µ and ν have probability density ρµ and ρν , respectively, and consider all smooth enough density fields ρ(t,x) and velocity fields v(t,x), where t ∈ [0, T ], subject to the continuity equation as well as initial and final conditions
∂tρ+∇ · (ρv) = 0, ρ(0, ·) = ρµ, ρ(T, ·) = ρν .
(4)
The above equation states that such velocity field will induce a transport map: we can construct an ordinary differential equation (ODE)
du dt = v(t,u), (5)
and the map from the initial point to the final point gives the transport map from µ to ν.
As is proposed by Benamou & Brenier (2000), for the transport cost function c(x,y) = ‖x− y‖2, the minimal transport cost is equal to the infimum of
T ∫ Rd ∫ T 0 ρ(t,x)|v(t,x)|2dxdt (6)
among all (ρ,v) satisfying equation (4). The optimality condition is given by
v(t,x) = ∇φ(t,x), ∂tφ+ 1
2 |∇φ|2 = 0. (7)
In other words, the optimal velocity field is actually induced from a flow with time-dependent potential φ(t,x). The use of this formulation is well-known in optimal transport community (Trigila & Tabak, 2016; Peyré et al., 2019). In this paper we integrate this formulation in the deep generative models. Instead of solving Monge’s problem and find the exact L2 optimal transport map, which is unrealistic due to the limited families of neural network functions as well as the errors arising from training the neural networks, our goal is to regularize the generators in a wide range of generative models, so that the generator maps could approximate the L2 optimal transport map at least in low dimensional problems. The maps would also be endowed with the characteristics of “proximity” so that we can apply them to engineering problems.

heading: 3.2 POTENTIAL FLOW GENERATOR
text: The potential φ(t,x) is the key function to estimate, since the velocity field could be obtained by taking the gradient of the potential and consequently the transport map could be obtained from Equation 5. There are two strategies to use neural networks to represent φ. One can take advantage of the fact that the time-dependent potential field φ is actually uniquely determined by its initial condition from Equation 7, and use a neural network to represent the initial condition of φ, i.e. φ(0,x), while approximating φ(t,x) via time discretization schemes. Alternatively, one can use a neural network to represent φ(t,x) directly and later apply the PDE regularity for φ(t,x) in Equation 7. We name the generators defined in the above two approaches as discrete potential flow generator and continuous potential flow generator, respectively, and give a detailed formulation as follows.

heading: 3.2.1 DISCRETE POTENTIAL FLOW GENERATOR
text: In the discrete potential flow generator, we use the neural network φ̃0(x) : Rd → R to represent the initial condition of φ(t,x), i.e. φ(0,x). The potential field φ(t,x) as well as the velocity field v(t,x) could then be approximated by different time discretization schemes. As an example, here we use the first-order forward Eular scheme for the simplicity of implementation. To be specific, suppose the time discretization step is ∆t and the number of total steps is n with n∆t = T , then for i = 0, 1...n, φ(i∆t,x) could be represented by φ̃i(x), where
φ̃i+1(x) = φ̃i(x)− ∆t
2 |∇φ̃i(x)|2, for i = 0, 1, 2..., n− 1. (8)
Consequently, the velocity field v(i∆t,x) could be represented by ṽi(x), where
ṽi(x) = ∇φ̃i(x), for i = 0, 1...n. (9)
Finally, we can build the transport map from Equation 5:
ũ0(x) = x,
ũi+1(x) = ũi(x)+∆tṽi(ũi(x)), for i = 0, 1, 2...n− 1, (10)
with G(·) = ũn(·) be our transport map. The discrete potential flow generator has built-in optimal transport regularity since the optimal condition (Equation 7) is encoded in the time discretization (Equation 8). However, such discretization also introduces nested gradients, which dramatically increases the computational cost when the number of total steps n is increased. In our tests, we found that even n = 5 is almost intractable.

heading: 3.2.2 CONTINUOUS POTENTIAL FLOW GENERATOR
text: In the continuous potential flow generator, we use the neural network φ̃(t,x) : R × Rd → R to represent φ(t,x). Consequently, the velocity field v(t,x) can be represented by ṽ(t,x), where
ṽ(t,x) = ∇φ̃(t,x). (11)
With the velocity field we could estimate the transport map by solving the ODE (Equation 5) using any numerical ODE solver. As an example, we can use the first-order forward Eular scheme, i.e.
ũ(0,x) = x,
ũ((i+ 1)∆t,x) = ũ(i∆t,x)+∆tṽ(i∆t, ũ(i∆t,x)), for i = 0, 1, 2...n− 1, (12)
with G(·) = ũ(T, ·) be the transport map, where ∆t is the time discretization step and n is the number of total steps with n∆t = T .
In the continuous potential flow generator, increasing the number of total steps would not introduce high order differentiations, therefore we could have large n, for a better precision of the ODE solver. Different from the discrete potential flow generator, the optimal condition (Equation 7) is not encoded in the continuous potential flow generator, therefore we need to penalize Equation 7 in the loss function, as we will discuss in the next subsection.
One may come up with another strategy of imposing the L2 optimal transport regularity: to use a vanilla generator, which is a neural network directly mapping from inputs to outputs, and penalize the L2 transport cost, i.e., the loss function is
Lvanilla = Loriginal + αEx∼µ‖G(x)− x‖2, (13)
where Loriginal is the original loss function for the generator, and α is the weight for the transport penalty. We emphasize that such strategy is much inferior to penalizing Equation 7 in the continuous potential flow generator. When training the vanilla generator with L2 transport penalty, no matter how we weight the L2 transport cost penalty, we always have to make a trade off between “matching the generated distribution with the target one” and “reducing the transport cost” since there is always a conflict between them, and consequently G#µ will be biased towards µ. On the other hand, there is no conflict between matching the distributions and penalizing Equation 7 in the continuous potential flow generator. As a consequence, the continuous potential flow generator is robust with respect to different weights for the PDE penalty. We will show this in Section 4.

heading: 3.3 TRAINING THE POTENTIAL FLOW GENERATOR
text: While the optimal condition (Equation 7) has been considered in the above two generators, the constraints of initial and final conditions have so far been neglected. However, the constraint of initial and final conditions provides the principle to train the neural network: we need to tune the parameter in the neural network φ̃ so that G#µ matches ν. This could be done in the fashion of GANs or normalizing flow models.

heading: 3.3.1 LOSS IN GAN MODELS
text: For the discrete potential flow generator, since the optimal transport regularity is already built in, the loss for training G is simply the GAN loss for the generator, i.e.
LD−PFG = LGAN , (14) where LGAN actually depends on the specific version of GANs. For example, if we use WGAN-GP, then LGAN = −Ez∼µD(G(z)), where D is the discriminator neural network.
For the continuous potential flow generator, as mentioned above, we also need to make φ̃ satisify Equation 7 for the optimal transport regularity. Inspired by the applications of neural networks in solving forward and backward problems of PDEs (Lagaris et al., 1998; Raissi et al., 2017a;b; Sirignano & Spiliopoulos, 2018), we penalize the squared residual of the PDE on the so-called “residual” points. In particular, the loss for continuous potential flow generator would be
LC−PFG = LGAN + λ 1
N N∑ i=1 [∂tφ̃(ti,xi) + 1 2 |∇φ̃(ti,xi)|2]2, (15)
where {(ti,xi)}Ni=1 are the residual points for estimating the residual of the PDE (Equation 7), and λ is the weight for the PDE penalty. In this paper we set them as the points on “trajectories” of input samples, i.e.
{(ti,xi)}Ni=1 = n⋃ i=0 ⋃ xj∈B {(i∆t, ũ(i∆t,xj))}, (16)
where B is the set of batch samples from µ. Note that the coordinates of the residual points involve ũ, but this should not be taken into consideration when calculating the gradient of the loss function with respect to the generator parameters.
We point out that the residual points should cover the whole spatial-temporal domain. Theoretically, only penalizing the squared residual of the PDE on “trajectories” could lead to failure in approximating the L2 optimal transport map. However, in our numerical experiments, this flawed sampling strategy still works. As an improvement, in each training iteration we can perturb the trajectory points with Gaussian noise in space and uniform noise in time as residual points, so that in principle they are sampled from the whole spatial-temporal domain.

heading: 3.3.2 LOSS IN NORMALIZING FLOW MODELS
text: Our continuous potential flow generator could be viewed as a further development of Neural ODE (Chen et al., 2018) applied to generative models, i.e. the continuous normalizing flow. The difference mainly lies in that we set the velocity as the gradient of a time-dependent potential function, and an augmented PDE loss is required for the generator. While both density matching and maximum likelihood training could be applied, here we take the latter as an example: we assume that the density of µ and samples from ν are available, and we maximize Ey∼ν [log pG#µ(y)], where pG#µ is the density of G#µ. Then, the loss for the continuous potential flow generator would be:
LC−PFG = −Ey∼ν [log pG#µ(y)] + λ 1
N N∑ i=1 [∂tφ̃(ti,xi) + 1 2 |∇φ̃(ti,xi)|2]2, (17)
where as in the GAN model, {(ti,xi)}Ni=1 are the residual points for estimating the residual of PDE (Equation 7), and λ is the weight for the PDE penalty. Ey∼ν [log pG#µ(y)] could be estimated via the approach introduced in Chen et al. (2018) and we give a detailed description in Appendix B.
The discrete potential flow cannot be trivially applied in normalizing flow models since we found that the time step size is too large to calculate the density accurately.

heading: 4 RESULTS

heading: 4.1 2D PROBLEMS
text: In this subsection, we apply the potential flow generators to several two-dimensional problems. We mainly study the following two problems where we know analytical solutions for the optimal transport maps. In problem 1 we assume that both µ and ν are Gaussian distributions with µ = N ([0; 0], [0.25, 0; 0, 1]) and ν = N ([0; 0], [1, 0; 0, 0.25]). In this case the optimal transport map is f((x, y)) = (2x, 0.5y). In problem 2 we assume that µ and ν are concentrated on concentric rings. In polar coordinates, suppose µ has (r, θ) uniformly distributed on [0.5, 1]× [0, 2π), while ν has (r, θ) uniformly distributed on [2, 2.5]× [0, 2π), where r and θ are radius and angular, respectively. In this case the optimal transport map is f((r, θ)) = (r+ 1.5, θ) in polar coordinates. We present the proofs in Appendix A. Samples from µ and ν as well as the optimal transport map in both problems are illustrated in Figure 2. We prepared 40000 samples for the input distribution and target distribution in each problem as training data for Section 4.1.1 and 4.1.2, and 1000 samples for Section 4.1.3.

heading: 4.1.1 VANILLA GENERATOR VERSUS POTENTIAL FLOW GENERATOR
text: For the above two problems we compare the following methods: (a) vanilla generator with L2 transport cost penalty, i.e., using the loss function in Equation 13 with GAN loss as Loriginal, (b) discrete potential flow generator, and (c) continuous potential flow generator with PDE penalty. For the vanilla generator and the continuous potential flow generator, we test different weights for the penalty in order to compare the influence of penalty weights in both generators. As for the GAN loss for generators we use the sliced Wasserstein distance1, due to its relatively low computational cost, robustness, and clear mathematical interpretation in low dimensional problems (Deshpande et al., 2018). In Figure 2 we illustrate the maps of different generators. A more systematic and quantitative comparison from three independent runs for each case is provided in Table 1, where the best results are marked as bold. The statistics come from 100,000 testing data.
1Strictly speaking, there is no “adversarial” training when we use sliced Wasserstein loss since the distance is estimated explicitly rather than represented by an other neural network. However, the idea of computing the distance between fake data and real data coincides with other GANs, especially WGANs. Therefore, in this paper we view the sliced Wasserstein distance as a special version of GAN loss.
As we already mentioned, vanilla generators with L2 transport penalty would make G#µ biased towards µ to reduce the transport cost from µ to G#µ. This is clearly shown in both problems with the penalty weight α = 0.1. In fact, we observed more significant biases with larger penalty weights. For the cases with smaller penalty weights α = 0.01, 0.001, in some of the runs, while G#µ are close to ν, the maps of generators are far from the optimal ones, which shows that the L2 transport penalty cannot provide sufficient regularity if the penalty weight is too small. These numerical results are consistent with our earlier discussion about the intrinsic limitation of the L2 transport penalty.
On the other hand, the potential flow generators give better matching between G#µ and ν, as well as smaller errors between the estimated transport maps and the analytical optimal transport maps. Notably, in both problems the continuous potential flow generators give good results with a wide range of PDE penalty weights ranging from 0.1 to 10, which shows the superiority of PDE penalty in the continuous potential flow generators compared with the transport penalty in vanilla generators. We also report that while in the first problem the discrete potential flow generator achieves a comparable result with the continuous potential flow generators, in the second problem we encountered “NAN” problems during training of the discrete potential flow generator in some of the runs. This indicates that the discrete potential flow generator is not as robust as the continuous one, which could be attributed to the high order differentiations and small total time steps n in the discrete potential flow generators.

heading: 4.1.2 CYCLEGAN VERSUS POTENTIAL FLOW GENERATOR
text: We also apply CycleGAN on the above two problems, with different random seeds. Here, we use feedforward networks as generators and discriminators, with WGAN-GP for the GAN loss function, and L1 loss for cycle-consistency loss with weight 5. In Figure 2 we illustrate the maps of G (red arrows) and F (blue arrows), i.e. the two generators in CycleGAN. The red and blue arrows overlap with opposite directions, which indicates that G and F are approximately the inverse map of each other, as we expected from the cycle-consistency loss. However, the maps are totally different in the three runs with different random seeds, which agrees with our discussion in Section 1 that the generator pair in CycleGAN is not unique. Moreover, the generator maps are less “regular” than the maps from the potential flow generator. Specifically, we can hardly interpret the generator maps given by CycleGAN.

heading: 4.1.3 DISCRETE REGULARIZED OPTIMAL TRANSPORT SOLVER VERSUS POTENTIAL FLOW GENERATOR
text: Finally, we compare the continuous potential flow generator in SWG and WGAN-GP with the discrete regularized optimal transport solver2 introduced by Seguy et al. (2017) on the above two problems. The results of the output distributions, as well as the errors between estimated transport maps and the
2We used the code from https://github.com/vivienseguy/Large-Scale-OT, with the author’s permission. All the setups are kept as default, except the training data and batch size. Here we use the same training dataset of size 1000 for all the methods, which is of the same magnitude as their original training dataset in the test code. Their default batch size is 50. Note that their solver is actually looking for the regularized optimal transport, but when the weight for regularization is small, e.g. 0.02 in their code, we expect the results to be close to the exact optimal transport.
analytical optimal transport maps are shown in Table 2, where the best results for different batch size setups are marked as bold. The statistics come from 100,000 testing data.
We first set the batch size as 50; in this case, the errors of maps are similar for all three methods, while the output distributions of the continuous potential flow generator in WGAN-GP match the best with the target ones. As is well known, the gradients of the sample Wasserstein loss are biased (Bellemare et al., 2017), thus the output distributions are biased for GANs based on the Wasserstein loss. This problem could be serious when the batch size is small. Therefore, we increased the batch size to 1000. In this case, we encounterred “NAN” problems when learning the barycentric mapping in the discrete regularized OT solver. SWG and WGAN-GP with continuous potential flow generators are stable, and we can see an improvement in the output distributions and error of maps, as we expected.

heading: 4.1.4 MORE PROBLEMS IN GAN MODEL AND NORMALIZING FLOW MODEL
text: Apart from the previous two problems, we also apply WGAN-GP and continuous normalizing flow models with continuous potential flow generators to several more complicated distributions. The results are illustrated in Figure 3. We can see the match between G#µ and ν in each of the problems, as well as that the samples of µ tend to be mapped to nearby positions. This shows the effectiveness of the continuous potential flow generator in various generative models, as well as the characteristics of “proximity” in the potential flow generator maps due to the L2 optimal transport regularity.

heading: 4.2 IMAGE TRANSLATION TASKS
text: In this section, we aim to show the capability of continuous potential flow generator in dealing with high dimensional problems, and also to show its advantage in image translations tasks with unpaired training data. We use WGAN-GP for the GAN loss. Before feeding the images into the generators,
we embed them into a Euclidean space, where the L2 distances between embedding vectors should quantify the similarities between images. In this paper we apply the principal component analysis (PCA) (Jolliffe, 2011), a simple but highly interpretable approach to conduct the image embedding.

heading: 4.2.1 THE MNIST DATASET
text: should be close in the L2 distance.

heading: 4.2.2 THE CELEBA DATASET
text: Original
Projection
C-PFG
(ours)
Vanilla
CycleGAN
FNN, L1
CycleGAN
FNN, L2
CycleGAN
Official
CycleGAN
FNN, L1
CycleGAN
FNN, L2
CycleGAN
Official
Figure 5: Comparison between our method, vanilla WGAN-GP, CycleGAN with WGAN-GP as GAN loss and L1 or L2 loss as cycle-consistency loss, as well as official CycleGAN. The first two rows are the original images and their projections on the 700-dimensional Euclidean space induced by PCA, which are similar. The next five rows are the corresponding outputs. The last three rows are the reconstructed images from different CycleGANs.
In this section we test the translation task between the CelebA images (Liu et al., 2015). We randomly pick 60000 images from the CelebA training dataset and divide them into two clusters: (a) images with attribute “smiling” labeled as false, and (b) images with attribute “smiling” labeled as true. The images are cropped so that only faces remain on the images. We view the two clusters as samples of µ and ν, respectively. The total number of components in PCA is set to 700. Note that our goal is to generate images of smiling faces belonging to the same people as in input images. The difficulty lies in that the training data are unpaired and actually we do not have the ground truth as a reference.
We compare our method against vanilla WGAN-GP, and CycleGAN with WGAN-GP for GAN loss. In CycleGAN we use L1 or L2 loss as cycle-consistency loss, with weight = 10 as in Zhu et al. (2017). To make a fair comparison, for each generator and discriminator, we use a feedforward neural network (FNN) with the same size of hidden layers (5× 256). We also test the official CycleGAN3 on this problem, where we feed the cropped face images (with resizing but without random jitter cropping) instead of embedding vectors into the model. The results of images randomly picked from the test dataset are shown in Figure 5.
For most of the images, our method could successfully translate the no-smiling faces to smiling faces belonging to the same people. Some of the output images are blurred, since it is difficult to learn the high order modes of PCA with FNN. Vanilla WGAN-GP and CycleGAN with FNN totally failed, in that the input and output images come from different people. This comparison clearly showed the necessity of additional regularity for the generators in translation tasks with only unpaired data, and that GAN loss + cycle-consistency loss cannot provide sufficient regularity.
The official CycleGAN is decent in performance, generating images less blurred than our method, but failed to change the countenance for some images. Note that the number of parameters in official CycleGAN is about 30 times more than that in our method, and the total training time is more than two times of our method on a single NVIDIA Tesla V100 GPU.

heading: 5 DISCUSSION AND CONCLUSIONS
text: In this paper we propose potential flow generators with L2 optimal transport regularity as plug-andplay generator modules that could be easiy integrated in a wide range of generative models. In particular, we propose two versions: the discrete one and the continuous one. For the discrete version, the L2 optimal transport regularity is directly encoded in, while for the continuous version we only need a slight augmentation to the original generator loss functions to impose the L2 optimal transport regularity.
We firstly show that the potential flow generators are able to approximate the L2 optimal transport maps in 2D problems. The continuous potential flow generator outperforms the discrete one in robustness. The continuous potential flow generator is also applied to WGAN-GP and continuous normalizing flow models, where we illustrate the characteristic of “proximity” for the potential flow generator due to the L2 optimal transport regularity. Consequently, we show the effectiveness of our method in image translation tasks using unpaired training data from the MNIST dataset and the CelebA dataset. We can see that our method significantly outperforms the vanilla WGAN-GP and CycleGAN using FNN with the same size of hidden layers.
We think that the results of our method in translation tasks are impressive considering that we only use PCA, a linear embedding method, with only feedforward neural networks. Such a naive strategy actually leads to the blurred patterns in the output images, which is also the case (even more severe) for vanilla WGAN-GP and CycleGAN using the same strategy. A possible improvement is to integrate the potential flow generator with other embedding techniques like autoencoders with convolutional neural networks. Apart from image-to-image translations, it is also possible to apply the potential flow generator to other translation tasks, if the translation objects could be properly embedded into Euclidean space. Moreover, it was perceived that the training of ODE-based model is slow, but the training of our method could be accelerated by applying methods related to optimal transport, e.g. the Wasserstein natural gradient method (Tong Lin et al., 2018; Li & Montúfar, 2018). We leave these possible improvements to future work.
3We used the code from https://www.tensorflow.org/tutorials/generative/cyclegan. The size of training dataset is reduced to 1000 for official CycleGAN, which is similar to the horse-to-zebra dataset. The number of total epochs is set back to 200 as in the original CycleGAN paper.

heading: A PROOF OF THE OPTIMAL MAPS IN SECTION 4.1
text: In problem 1, for Gaussian distributions µ and ν with mean m1 and m2, as well as covariance matrices Σ1 and Σ2, from Gelbrich (1990) we know that the minimum transport cost from µ to ν with cost function c(x,y) = ‖x− y‖2 is
‖m1 −m2‖2 + Tr(Σ1 + Σ2 − 2(Σ1/21 Σ2Σ 1/2 1 ) 1/2), (18)
which is known as the squared Wasserstein-2 distance between two Gaussian distributions. In particular, the minimum transport cost is 0.5 in our problem.
For the map f((x, y)) = (2x, 0.5y), f#µ is Gaussian since f is linear. By checking the mean and covariance we have f#µ = ν. Also, the transport map of f is
Ex∼N (0,0.25)(2x− x)2 + Ey∼N (0,1)(0.5y − y)2 = 0.25 + 0.25 = 0.5 (19)
which is exactly the minimum transport cost, thus f is the optimal transport map. We complete the proof of the optimal transport map in problem 1.
In problem 2, denote X = [0.5, 1], Y = [2, 2.5], O = [0, 2π), and m1 = U(X), m2 = U(Y), mθ = U(O), where we use U(A) to represent the uniform probability measure on set A. For f((r, θ)) = (r + 1.5, θ), µ = U(X) × U(O), ν = U(Y) × U(O), we have f#µ = ν. For any transport map from µ to ν, denote as g((r, θ)) = (gr(r, θ), gθ(r, θ)) in polar coordinates, we only need to show that the transport cost of g is no less than the cost of f .
Let h((r, θ)) = (gr(r, θ), θ), then the transport cost of g is no less than the cost of h since the transport cost is reduced for any point (r, θ).4
Actually we could view gr(r, θ) as a multivalued function in r so that gr(r, θ) induces a transport plan H from m1 to m2. More formally, define measure H : B(X× Y)→ R by
H(A) = ∫ X ∫ Y 1(x,y)∈AM(x, dy)dm1(x) = ∫ A M(x, dy)dm1(x), (20)
where M(·, ·) : X× B(Y)→ R is defined by M(x,A) = ∫ O 1gr(x,θ)∈Admθ(θ). (21)
To see H is a transport plan from m1 to m2, we need to check:
1. ∀A ∈ B(X), H(A× Y) = m1(A). This is true since
H(A× Y) = ∫ A ∫ Y M(x, dy)dm1(x) = ∫ A M(x,Y)dm1(x) = m1(A), (22)
where we utilize that M(x,Y) = 1. 2. ∀A ∈ B(Y), H(X× A) = m2(A).
Note that g#µ = ν, thus µ(g−1(A×O)) = ν(A×O) = m2(A). Also, µ(g−1(A×O)) = ∫ X ∫ O 1g(x,θ)∈A×Odmθ(θ)dm1(x)
= ∫ X ∫ O 1gr(x,θ)∈Admθ(θ)dm1(x)
= ∫ X M(x,A)dm1(x) = H(X× A).
(23)
Therefore H(X× A) = m2(A). 4It’s not necessary that h#µ = ν.
We also claim that the L2 transport cost of h equals to that of H . The transport cost of h and H are
C(h) = ∫ X ∫ O (gr(x, θ)− x)2dmθ(θ)dm1(x),
C(H) = ∫ X×Y (y − x)2dH = ∫ X ∫ Y (y − x)2M(x, dy)dm1(x),
(24)
respectively. By the definition of M , we have M(x, ·) = gr(x, ·)#mθ for any x ∈ X, thus∫ Y (y − x)2M(x, dy) = ∫ O (gr(x, θ)− x)2dmθ(θ) (25)
for any x ∈ X . Therefore C(h) = C(H). Let F (x) = x + 1.5 be another transport plan from m1 to m2, clearly the L2 transport cost of f equals to that of F . Note that the transport cost of H is no less than that of F , since the latter one is the optimal transport plan from m1 to m2. This complete the proof of claim that the transport cost of g is no less than that of f , and thus the proof of the optimal transport map in problem 2.

heading: B DETAILS OF LOSS FUNCTIONS IN CONTINUOUS NORMALIZING FLOWS
text: To estimate log pG#µ(y), we have the ODE that connects the probability density at inputs and outputs of the generator:
d dt log(p(ũ(t,x))) = −∇ũ · ṽ(t, ũ(t,x)) = −∆ũφ̃(t, ũ(t,x)), (26)
for all x in the support of µ, where the initial probability density p(ũ(0,x)) = pµ(x) is the density of µ at input x, while the terminal probability density p(ũ(T,x)) = pG#µ(G(x)) is the density of G#µ at output G(x).
Also, we estimate x = G−1(y) by solving the ODE
dw
dt = −ṽ(T − t,w) (27)
with initial condition w(0) as y = G(x) and w(T ) as the corresponding x = G−1(y).
For each y, we can use Equation 27 to estimate the corresponding x = G−1(y), and consequently log(pµ(x)) since we have the density of µ. Then we apply Equation 26 to estimate log pG#µ(y). By sampling y ∼ ν, we can estimate Ey∼ν [log pG#µ(y)]. In practice, we also need to discretize Equations 26 and 27 properly. For example, we use the first-order Euler scheme in our practice. Note that when applying maximum likelihood training, the density of µ could be unnormalized, since multiplications with pµ would merely lead to a constant difference in the loss function.

heading: C NETWORKS AND HYPERPARAMETERS
text: Except in the regulized discrete optimal transport solver and official CycleGAN, all the neural networks are feedforward neural networks of 5 hidden layers, each of width 128 for 2D problems, or 256 in image translation tasks. For potential flow generator, the activation function is tanh for the smoothness of φ̃. The activation function in vanilla generator in Figure 2 is also tanh. All the other activation functions are leaky ReLU (Maas et al., 2013) with negative slope 0.2, except in the regulized discrete optimal transport solver and official CycleGAN.
The batch size is set as 1000 for all the cases except in Table 2 and in official CycleGAN. We use 1000 random projection directions to estimate the sliced Wasserstein distances. In WGAN-GP model the coefficient for gradient penalty is 0.1, and we do 5 discriminator updates per generator update.
In potential flow generators, the time span T is 1.0. We set the number of total time steps n = 4 in discrete potential flow generators, while n = 100 in continuous potential flow generators for 2D problems and n = 10 in image translation tasks. The PDE penalty weight λ for continuous potential
flow generator is set as 1.0 by default, except those in the 2D problems where we compare different generators.
We use the Adam optimizer (Kingma & Ba, 2014) for all the problems. In the 2D problems, the learning rate is set as l = 10−5, β1 = 0.5, β2 = 0.999 for sliced Wasserstein distance, while set as l = 10−5, β1 = 0.5, β2 = 0.9 in WGAN-GP. We train the generators for 100, 000 iterations in Figure 2 and Figure 3a, and 20, 000 iterations in Table 2. In the normalizing flow model we set l = 10−4, β1 = 0.9, β2 = 0.999, and train the generator for 10, 000 iterations. In image translation tasks we set l = 10−4, β1 = 0.5, β2 = 0.9, and train the generator for 100, 000 iterations for our method and CycleGANs with FNN, while 200, 000 iterations for vanilla GAN.
D MORE RESULTS ON THE MNSIT AND CELEBA DATASET

Review summaries:
The paper proposes a ‘potential flow generator’ that can be seen as a regularizer for traditional GAN losses. It is based on the idea that samples flowing from one distribution to another should follow a minimum travel cost path. This regularization is expressed as an optimal transport problem with a squared Euclidean cost. Authors rely on the dynamic formulation of OT proposed by Benamou and Brenier, 2000. They propose to learn a time-dependent potential field which gradient defines the velocity fields used to drive samples from a source distribution toward a target one. Experiments on a simple 1D case (where the optimal transport map is known), and on images with an MNIST / CelebA qualitative example.

This is a great paper using optimal transport theory for generative and implicit models. Instead of using general vector fields, the authors apply the potential vector fields in optimal transport theory to design neural networks. The mathematics is correct with convincing examples. This brings an important mathematical connection between fluid dynamics and GANs or implicit models.