Potential Flow Generator with $L_2$ Optimal Transport Regularity for Generative Models
Authors: Liu Yang
George Em Karniadakis
Venue:  (2020)
Keywords: generative models
optimal transport
GANs
flow-based models

Abstract: We propose a potential flow generator with $L_2$ optimal transport regularity, which can be easily integrated into a wide range of generative models including different versions of GANs and flow-based models. With up to a slight augmentation of the original generator loss functions, our generator is not only a transport map from the input distribution to the target one, but also the one with minimum $L_2$ transport cost. We show the correctness and robustness of the potential flow generator in several 2D problems, and illustrate the concept of ``proximity'' due to the $L_2$ optimal transport regularity. Subsequently, we demonstrate the effectiveness of the potential flow generator in image translation tasks with unpaired training data from the MNIST dataset and the CelebA dataset. 

Under review as a conference paper at ICLR 2020
POTENTIAL FLOW GENERATOR WITH L2 OPTIMAL TRANSPORT REGULARITY FOR GENERATIVE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose potential flow generator with L2 optimal transport regularity, which can be easily integrated into a wide range of generative models including different versions of GANs and normalizing flow models. With only slight augmentation to the original generator loss functions, our generator not only tries to transport the input distribution to the target one, but also aims to find the with minimum L2 transport cost. We show the effectiveness of our method in several 2D problems, and illustrate the concept of "proximity" due to the L2 optimal transport regularity. Subsequently, we demonstrate the effectiveness of the potential flow generator image translation tasks with unpaired training data from the MNIST dataset and the CelebA dataset with a comparison against vanilla WGAN-GP and CycleGAN.
INTRODUCTION
Many of the generative models, for example, generative adversarial networks (GANs) (Goodfellow et al. 2014: Arjovssy al. 2017 Salimans et al. 2018) and normalizing flow models (Rezende & Mohamed 2015; Kingma & Dhariwal 2018 Chen et al. 2018), aim to find a generator that could map the input distribution to the target distribution.
In many cases, especially when the input dis- tributions are purely noises, the specific maps between input and output are of little importance as long as the generated distributions match the target ones. However, in other cases like image- to-image translations, where both input and tar- get distributions are distributions of images, the generators are required have additional regu- larity such that the input individuals mapped to the "corresponding" outputs some sense. If paired input-output samples are provided, Lp penalty could be hybridized into generators loss functions encourage the output individuals to fit the ground truth (Isola et al. 2017). For the cases without paired data, popular approach is to introduce another generator and encourage the two generators
Generator without L2 Optimal Transport Regularity
Generator with Lg Optimal Transport Regularity
Input distribution
Output distribution
be the inverse maps of each other, as in CycleGAN (Zhu et al. 2017). DualGAN (Yi et al. 2017) and DiscoGAN (Kim et al. 2017), etc. However, such a pair of gener- ators not unique and lacks clear mathematical interpretation about its effectiveness.
Figure 1: Schematic of generator without and with L2 optimal transport regularity.
In this paper we introduce a special generator, i.e., the potential flow generator, with Le optimal transport regularity. By applying such generator, not only are we trying to find map from the input distribution to the target one, but we also aim to find the optimal transport map that minimizes the squared Euclidean transport distance. In Figure we provide schematic comparison between generators with and without optimal transport regularity. While both generators provide a scheme to map from the input distribution the output distribution, the total squared transport distances in the

---

Under review as a conference paper at ICLR 2020
left generator is larger than that in the right generator. Note that the generator with optimal transport regularity has the characteristic of "proximity" 'in that the imputs tend to be mapped to nearby outputs. As we will show later, this "proximity" characteristic of L2 optimal transport regularity could be utilized in image translation tasks. Compared with other approaches like CycleGAN, the L- optimal transport regularity has much clearer mathematical interpretation.
There have been other approaches to learn the optimal transport map in generative models. For example, [Seguy et al. (2017) proposed first learn the regularized optimal transport plan and then the optimal transport map, based on the dual form of regularized optimal transport problem. Also, Yang & Uhler (2018) proposed learn the unbalanced optimal transport plan in an adversarial way derived from convex conjugate representation of divergences. In the W2GAN model proposed by Leygonie et al. 2019), the discriminator's objective the 2- 2-Wasserstein metric so that the generator supposed to recover the T22 optimal transport map. All the above approaches need to introduce, and are limited to, specific loss functions to train the generators.
xxample,[Seguy
Our proposed potential flow generator takes different approach in that with only a slight augmenta- tion to the original generator loss functions, our generator could be integrated into a wide range of generative models with various generator loss functions, including different versions of GANs and normalizing flow models. This simple modification makes our method easy to adopt on various tasks considering the existing rich literature and the future developments of generative models.
In Section we present a formal definition optimal transport map and the motivation to apply L2 optimal transport regularity to generators. In Section 3 we give detailed formulation of the potential flow generator and the augmentation to the original loss functions. Results are then provided in Section 4 We include the discussion and conclusions in Section
2 GENERATIVE MODELS AND OPTIMAL TRANSPORT MAP
First, we introduce the concept of push forward, which will be used extensively in the paper.
Definition Given two Polish space X and Y. B and B( the Borel o-algebra on and Y, and P(X), P(Y) the set of probability measures (X) and B(Y). Let f X + Y be Borel map, and E P(X). We define [y E PYY), the forward of y through J, by
f*u(A)= - (( (A)))(((A))))))))))))( VA E B(Y).
With the concept of push forward, we can formulate the goal of GANs and normalizing flow models as train the generator G such that G is equal or at least close to in some sense, where and v are the input and target distribution, respectively. Usually, the loss functions for training the generators are metrics of closeness that vary for different models. For example, in continuous normalizing flows (Chen et al. 2018), such metric of closeness is DKL (G)plpp) or DKL (vil In Wasserstein GANs (WGANs) (Arjovsky et al. 2017), the metric of closeness is the Wasserstein-1 distance between G and which is estimated in variational form with the discriminator neural network. As a result, the generator and discriminator neural networks are trained in an adversarial way:
min G
max D I-I Lipschitz
(((=))(((())
where D is the discriminator neural network and the Lipschitz constraint could be imposed via the gradient penalty (Gulrajani et al. 2017). spectral normalization (Miyato et al. 2018), etc.
Now introduce the concept of optimal transport map follows:
Definition 2
Given a cost function e: X x V. - RR and E P(X), E P(V), we let T be the set of all transport maps from y to v, i.e. T {f f#p =v. Monge's 's optimal transport problem is to minimize the cost functional C(f) among T, where
C(ff) c(2, (x))
and the minimizer J E T. is called the optimal transport map.
In this paper, we are concerned mostly with the case where = Y= Rd with Lg transport cost, i.c., the transport c(æ, y) = || 2 We assume that μ and absolute continuous w.r.t. Lebesgue
2

---

Under review as a conference paper at ICLR 2020
measure, i.e they have probability density functions. In general, Monge's problem could be ill ill-posed in that could be empty set or there is no minimizer in Also, the optimal transport map could be non-unique. However, for the special case consider, there exists unique solution to Monge's problem (Brenier 1991 Gangho & McCann 1996).
Informally speaking, with L2 transport cost the optimal transport map has the characteristic of "proximity". i.e. the inputs tend to be mapped to nearby outputs. In image translation tasks, such "proximity"" characteristic would be helpful if could properly embed the images into Euclidean space such that our preferred input-output pairs are close to each other. A similar idea is also proposed in Yang & Uhler (2088) for unbalanced optimal transport. Apart from image translations, the Le optimal transport problem is important in many other aspects. For example. it is closely related to gradient flow (Ambrosio al. 2008), Bakker-Plancee equations (Santambrogio) 2017), flow in porous medium (Otto 1997), etc.
3 POTENTIAL FLOW GENERATOR
3.1 POTENTIAL FLOW FORMULATION OF OPTIMAL TRANSPORT MAP
We assume that f and v have probability density Pu and Pv. respectively, and consider all smooth enough density fields p(t, æ) and velocity fields (t, æ), where i E [0, 7), subject to the continuity equation as well initial and final conditions
O.p + V (pv) = 0, p().-) = Pp: p(T.-) = The above equation states that such velocity field will induce transport map: construct an ordinary differential equation (ODE)
du
dt vt, u),
(5)
and the map from the initial point to the final point gives the transport map from ja to v.
As is proposed by Benamou & Brenier (2000), for the transport cost function c(x: y) = ||x - m02l the minimal transport cost is equal to the infimum of
T f
Rd [ p(t, x))v(t,)))) x) dx
(6)
among all (p. v) satisfying equation 44. The optimality condition is given by
v(t, x) x)), 0 } = =.
In other words. the optimal velocity field is actually induced from flow with time-dependent potential oft, ..). The use this formulation is well-known in optimal transport community (Trigila & Tabak 2016 Peyré et al. 2019). In this paper integrate this formulation in the deep generative models. Instead of solving Monge's 's problem and find the exact L2 optimal transport map. which is unrealistic due the limited families of neural network functions well as the errors arising from training the neural networks, our goal is to regularize the generators a wide range of generative models, so that the generator maps could approximate the L2 optimal transport map least low dimensional problems. The maps would also be endowed with the characteristics of "proximity" so that we can apply them to engineering problems.
3.2 POTENTIAL FLOW GENERATON
The potential ø(t, x) is the key function to estimate, since the velocity field could be obtained by taking the gradient of the potential and consequently the transport map could be obtained from Equation There are two strategies to use neural networks to represent á. One can take advantage of the fact that the time-dependent potential field ő is actually uniquely determined by its initial condition from Equation 7 and use neural network to represent the initial condition of c, i.e. 6(0, x). while approximating (t. x>) via time discretization schemes. Alternatively, one can use a neural network to represent ott, x) directly and later apply the PDE regularity for @(t, x) in Equation 7 We name the generators defined in the above two approaches as discrete potential flow generator and continuous potential flow generator, respectively, and give detailed formulation follows.
3

---

Under review as a conference paper at ICLR 2020
3.2.1 DISCRETE POTENTIAL FLOW GENERATOR
In the discrete potential flow generator, we use the neural network (0(x)) IR R to represent the initial condition of (tt, æ), i.. o(0, The potential field ((t, as well as the velocity field v(t, æ) could then be approximated by different time discretization schemes. As an example, here we use the first-order forward Eular scheme for the simplicity of implementation. To be specific, suppose the time discretization step is A/ and the number of total steps n with nÂI T, then for
i = 0, I. (iitt, æ) could be represented by
Φ (2:), where
At
( = (x)
Vo)(2))
for i = 0,1,2...
(8)
Consequently, the velocity field miii, a)) could be represented by (a)), where
ö (a) (œ) for i = 0, 1....
(9)
Finally, we can build the transport map from Equation [3]
( E; i (a:) = (x) Ato, (x)))) for i = 0,,, 1,
(10)
with G(.) = (.) be our transport map.
The discrete potential flow generator has built-in optimal transport regularity since the optimal condition (Equation[] 7 encoded the time discretization (Equation8) 8). However, such discretization also introduces nested gradients, which dramatically increases the computational cost when the number total steps n is increased. In our tests, we found that even n 5 is almost intractable.
3.2.2 CONTINUOUS POTENTIAL FLOW GENERATOR
In the continuous potential flow generator, we use the neural network o(t, æ:) : x R2d - to represent @(t, æ)). Consequently, the velocity field vtt, æ) can be represented by ö(t, æ), where
vt,,x) Vt(t
(11)
With the velocity field we could estimate the transport map by solving the ODE (Equation 5 using any numerical ODE solver. As example, we can use the first-order forward Eular scheme, i.e.
ū(O, x) a, ū( + 11 x) a(ist, Att a(ist, x)), for 0, 2.... 1,
(12)
with G(-) -) be the transport map, where At is the time discretization step and n is the number of total steps with nAt = T.
In the continuous potential flow generator, increasing the number of total steps would not introduce high order differentiations, therefore we could have large n, for a better precision the ODE solver. Different from the discrete potential flow generator, the optimal condition (Equation / is not encoded in the continuous potential flow generator, therefore we need to penalize Equation in the loss function, as we will discuss in the next subsection.
One may come up with another strategy of imposing the L2 optimal transport regularity: to use a vanilla generator, which is neural network directly mapping from inputs outputs, and penalize the L2 transport cost, i.e., the loss function is
Lvanilla Loriginal + a. || G((z) =x|
(13)
where Loriginal is the original loss function for the generator, and a is the weight for the transport penalty. We emphasize that such strategy much inferior to penalizing Equation 7 in the continuous potential flow generator. When training the vanilla generator with L2 transport penalty, no matter how we weight the L2 transport cost penalty, we always have to make a trade off between "matching the generated distribution with the target one" and "reducing the transport cost" since there is always a conflict between them, and consequently Gy will be biased towards fr. On the other hand, there is no conflict between matching the distributions and penalizing Equation 7 in the continuous potential flow generator. As a consequence, the continuous potential flow generator robust with respect to different weights for the PDE penalty. We will show this in Section
4

---

Under review as a conference paper at ICLR 2020
3.3 TRAINING THE POTENTIAL FLOW GENERATOR
While the optimal condition (Equation 7 has been considered in the above two generators, the constraints of initial and final conditions have so far been neglected. However, the constraint of initial and final conditions provides the principle to train the neural network: we need to tune the parameter in the neural network o so that Gyp matches v. This could be done in the fashion of GANs or normalizing flow models.
3.3.1 LOss IN GAN MODELS
For the discrete potential flow generator, since the optimal transport regularity is already built in, the loss for training G is simply the GAN loss for the generator, i.e.
LD-PFG = LGAN,
(14)
where LGAN actually depends on the specific version of GANs. For example, if we use WGAN-GP, then LGAN Dxzz) where D is the discriminator neural network.
For the continuous potential flow generator, mentioned above, we also need to make Ø satisify
Equation 7 for the optimal transport regularity. Inspired by the applications of neural networks in solving forward and backward problems of PDEs (Lagaris et al. 1998; Raissi et al., 2017a b Sirignaan & Spiliopoulos, 2018), we penalize the squared residual of the PDE on the so-called "residual" points. In particular, the loss for continuous potential flow generator would be
1 Ejo,ōtt. 2.) ))+ 2
LC-PFG = LGAN X
(15)
where {(4,, ) are the residual points for estimating the residual of the PDE (Fquation 7. and is the weight for the PDE penalty. In this paper we set them as the points on "trajectories" of input samples, i.c.
{(t. ( = U U (iAtt ((iAt,
(16)
==
where IB is the set of batch samples from Ja. Note that the coordinates of the residual points involve ū, but this should not be taken into consideration when calculating the gradient of the loss function with respect to the generator parameters.
We point out that the residual points should cover the whole ppatial-temporda domain. Theoretically, only penalizing the squared residual of the PDE on "trajectories" could lead to failure in approximating the L2 optimal transport map. However, in our numerical experiments, this flawed sampling strategy still works. As an improvement, in each training iteration we can perturb the trajectory points with Gaussian noise space and uniform noise in time as residual points, so that in principle they are sampled from the whole patial-temporal domain.
3.3.2 Loss NORMALIZING FLOW MODELS
Our continuous potential flow generator could be viewed as a further development of Neural ODE (Chen et al. 2018) applied to generative models, i.e. the continuous normalizing flow. The difference mainly lies in that we set the velocity as the gradient of a time-dependent potential function, and an augmented PDE loss is required for the generator. While both density matching and maximum likelihood training could be applied, here we take the latter as an example: we assume that the density of μ and samples from v are available, and we maximize Ey-ollog PG (y)), where pG is the density of G# Then, the loss for the continuous potential flow generator would be:
N
LC-PFG -Ey (oo (3)) X N
10000t x.) + (o(l..
(17)
where in the GAN model, {t;, x; J.N. |== the residual points for estimating the residual of PDE (Equation 7). and À is the weight for the PDE penalty.
PG (y)] could be estimated via the
approach introduced in Chen et al. (2088) and give a detailed description in Appendix B
The discrete potential flow cannot be trivially applied in normalizing flow models since we found that the time step size is too large to calculate the density accurately.
5

---

Under review as a conference paper at ICLR 2020
4 RESULTS
4.1 2D PROBLEMS
In this subsection, we apply the potential flow generators to several two-dimensional problems.
We mainly study the following two problems where we know analytical solutions for the optimal transport maps. In problem I we assume that both μ and are Gaussian distributions with = [[(0;)), 25, 0; 0, 11) and v = [V{ 0), [1, O; 0, 0.25]). ) In this case the optimal transport map (((x. (2x, 0.5y). In problem 2 we assume that μr and v are concentrated on concentric rings. In polar coordinates. suppose μ has (r, 8) uniformly distributed on [0.5, 1] x [0, 2x). while y/ has (r, 0) uniformly distributed on [2, 2.5] x 2T)). where and 0 are radius and angular, respectively. In this case the optimal transport map is [(r 8)) = -(r + 1.5, 8) in polar coordinates. We present the proofs in Appendix A Samples from μ and illustrated in F We prepared 40000 samples for the input distribution and target distribution in
well as the optimal transport map in both problems
each problem as training data for Section 4.1.I and 4.1.2 and 1000 samples for Section 4.1.3
Samples
Weight
Van Weight
Samples
Vanilla Weight
-1
Figure 2: Comparison of different methods: vanilla generator with L2 transport cost penalty, discrete potential flow generator (D-PFG), continuous potential flow generator (C-PFG), and CycleGAN in problem 1 (left) and (right). The red and blue arrows represent the map of generators, the black arrows represent the analytical optimal transport map.
4.1.1 VANILLA GENERATOR VERSUS POTENTIAL FLOW GENERATOR
For the above two problems we compare the following methods: (a) vanilla generator with La transport penalty, i.e., using the loss function in Equation 13 with GAN loss as Loriginat- (b) discrete potential flow generator, and (c) continuous potential flow generator with PDE penalty. For the vanilla generator and the continuous potential flow generator, penalty in order to compare the influence of penalty weights in both generators. As for the GAN loss for generators we use the sliced Wasserstein distance due to its relatively low computational cost, robustness, and clear mathematical interpretation in low dimensional problems (Deshpande et al.) 2018). In Figure 2 we illustrate the maps of different generators. more systematic and quantitative comparison from three independent runs for each case is provided in Table I where the best results are marked as bold. The statistics come from 100,000 testing data.
different weights for the
Sttricty speaking, there no "adversarial" training when we use sliced Wasserstein loss since the distance estimated explicitly rather than represented by other neural network. However. the idea of computing the distance between fake data and real data coincides with other GANs, especially WGANs. Therefore, in this paper view the sliced Wasserstein distance special version of GAN loss.

---

Under review as a conference paper at ICLR 2020
Table 1: Comparison between different generators on two problems
Problem 1
Problem 2
Std in x-axis
Std in y-axis
Error of map
Mean of norm
Error of map
Reference
1.000
0.500
2.250
Vanilla (a=0.1)
0.592:+0.003 0.919=0.004
0.100=00.022
2.107=0.002
0.146++0.003
Vanilla (a=0.01)
0.985=0.005
0.499:10.006
0.439=0.5887
2.227=0.004
0.973
Vanilla (a=0.001)
0.992=100099
0.493:10.001
0,462:10.593
2.243==0.002
1.000
D-PFG
0.993=0.001
0.498=0.0002
0.018=0.006
Fail sometimes
C-PFG (A=10.0)
0.991=0.001
0.502=#00001
0.018=0.006
2.243=0.001
0.024=1.004
C-PFG (A=1.0)
0.902++00001
0.499
0.019+000077
2.245 + 0.000
0.029++0.002
C-PFG (A=0.1)
0.990+0.002
0.503++0.003
0.025++0.008
2.245 + 0.001
0.031+000004
As we already mentioned, vanilla generators with L2 transport penalty would make Gap biased towards j to reduce the transport cost from μ to Gy p.. This is clearly shown in both problems with the penalty weight For the cases with smaller penalty weights a = 0.01, 0.001, in some of the runs, while G are close v, the maps of generators are far from the optimal ones, which shows that the L-2 transport penalty cannot provide sufficient regularity the penalty weight is too small. These numerical results consistent with earlier discussion about the intrinsic limitation of the L2 transport penalty.
0.1. In fact, observed more significant biases with larger penalty weights.
On the other hand, the potential flow generators give better matching between G f and v/ as well as smaller errors between the estimated transport maps and the analytical optimal transport maps. Notably, in both problems the continuous potential flow generators give good results with a wide range of PDE penalty weights ranging from 0.1 10, which shows the superiority of PDE penalty in the continuous potential flow generators compared with the transport penalty in vanilla generators. We also report that while in the first problem the discrete potential flow generator achieves a comparable result with the continuous potential flow generators, in the second problem we encountered "NAN" problems during training of the discrete potential flow generator in some of the runs. This indicates that the discrete potential flow generator is not as robust as the continuous one, which could be attributed to the high order differentiations and small total time steps n in the discrete potential flow generators.
4.1.2 CYCLEGAN VERSUS POTENTIAL FLOW GENERATOR
We also apply CycleGAN on the above two problems, with different random seeds. Here, we use feedforward networks as generators and discriminators, with WGAN-GP for the GAN loss function, and L1 loss for cycle-consistency loss with weight 5. In Figure 2 we illustrate the maps of G (red arrows) and F' (blue arrows), i.e. the two generators in CycleGAN. The red and blue arrows overlap with opposite directions, which indicates that G and F are approximately the inverse map of each other, as we expected from the cycle-consistency loss. However, the maps are totally different in the three runs with different random seeds, which agrees with our discussion in Section T that the generator pair in CycleGAN is not unique. Moreover, the generator maps are less "regular" than the maps from the potential flow generator. Specifically, we can hardly interpret the generator maps given by CycleGAN.
4.1.3 DISCRETE REGULARIZED OPTIMAL TRANSPORT SOLVER VERSUS POTENTIAL FLOW GENERATOR
Finally, we compare the continuous potential flow generator in SWG and WGAN-GP with the discrete regularized optimal transport solver) introduuedd by Seguy et al. (2017) on the above two problems. The results of the output distributions, well the errors between estimated transport maps and the
We used the code from https://ithub..co// wwwm//iiiienuuullaggge-See the setups are kept as default, except the training data and batch size. Here we use the same training dataset of size 1000 for all the methods, which is of the same magnitude as their original training dataset in the test code. Their default batch size is 50. Note that their solver is actually looking for the regularized optimal transport, but when the weight for regularization is small, e.g. 0..2 in their code, we expect the results to be close to the exact optimal transport.
with the author's permission. All
7

---

Under review as a conference paper at ICLR 2020
Table Comparison between discrete regularized OT solver and C-PFG on two problems
Problem
Problem 2
Std in x-axis
Std in y-axis
Error of map
Mean of norm Error of map
Reference
1.000
0.500
2.250
RegOT, bs=50
0.908
0.444
0.105
2.185
0.169
C-PFG, bs=50, SWG
0.921
0.600
0.117
2.119
0.158
C-PFG, bs=50, WGAN
1.017
0.519
0.102
2.201
0.257
RegOT, bs=1000
Fail
Fail
C-PFG, b5=0000, SWG
1.019
0.509
0.079
2.237
0.116
C-PFG, bs=1000, WGAN
1.022
0.502
0.097
2.239
0.129
analytical optimal transport maps shown in Table 2 where the best results for different batch size setups are marked as bold. The statistics come from 100,000 testing data.
We first set the batch size as 50; in this case, the errors of maps similar for all three methods, while the output distributions of the continuous potential flow generator in WGAN-GG match the best with the target ones. As is well known, the gradients of the sample Wasserstein loss biased (Bellemare et al. 2017), thus the output distributions are biased for GANs based on the Wasserstein loss. This problem could be serious when the batch size is small. Therefore, increased the batch size to 1000. In this case, we encounterred "NAN" problems when learning the barycentric mapping in the discrete regularized OT solver. SWG and WGAN-GP with continuous potential flow generators are stable, and we can see an improvement in the output distributions and error of maps, as we expected.
4.1.4 MORE PROBLEMS IN GAN MODEL AND NORMALIZING FLOW MODEL
(a) WGAN-GP
(b) Continuous normalizing flow
Figure 3: Potential flow generator in (a) WGAN-GP and (b) continuous normalizing flow for different problems. Each column shows the setup and results one problem. The top row shows the samples or the unnormalized density functions of μ (purple) and (orange), the bottom row shows the map estimated by potential flow generator and samples G.
Apart from the previous problems, also apply WGAN-GP and continuous normalizing flow models with continuous potential flow generators to several more complicated distributions. The results illustrated in Figure We can the match between G and each of the problems, as well as that the samples of tend to be mapped to nearby positions. This shows the effectiveness of the continuous potential flow generator various generative models, as well as the characteristies of "proximity" in the potential flow generator maps due to the L2 optimal transport regularity.
4.2 IMAGE TRANSLATION TASKS
In this section, aim show the capability of continuous potential flow generator in dealing with high dimensional problems, and also to show its advantage image translations tasks with unpaired training data. We WGAN-GP for the GAN loss. Before feeding the images into the generators,
8

---

Under review as a conference paper at ICLR 2020
embed them into Euclidean space, where the L2 distances between embedding vectors should quantify the similarities between images. this paper we apply the principal component analysis (PCA) (Joliiffe 201T), simple but highly interpretable approach to conduct the image embedding.
4.2.1
THE MNIST DATASET
We firstly test the problem of translation be- tween the MNIST images (LeCun al. (0110). We divide the MNIST training dataset into two clusters: (a) images of digits 0 to 4, and (b) im- ages of digits 5 to 9. We view the two clusters of images as samples of μ and v, respectively, i.e., we want to find the optimal transport map from images of digits 0 to to images of digits
0
0
O
S
G
S
6
5
6
6
G
5 5
1
/
I
I 7 2 8
/
7
7
7
9
7
7
7
5
7
6
7 2 4 3
3
2:
2
2
2
2
2
2
2
2
2
2
2
9
8
8
8
6
6
6
6
6
$
7
5 to 9. The dimensionality of Euclidean space, i.c. the total components in PCA, set as 100.
3
3
3
3
3
3
3
3
3
3
3
3
3
3
5
3
5
5
7
5
6
5
8
8
8
6
In Figure 4 we randomly pick images from the test dataset and show the corresponding inputs and outputs (more images in Appendix DD. As we can see, the potential flow generator tends to translate images of digit 0 to digit 6, digit 1 to digit 7, digit 3 to digit 5 or 8, and digit 4 to digit 9. This is consistent with our previous discus- sion about the characteristics of "proximity" in that the input digits and output digits "look sim- ilar", and the corresponding embedding vectors
4
4
4
4
4
4
/
4
4
4
4 9
4
4 9
4 9
4
9
4
9
7
9
9
f
9
9
7
9
Figure 4: Potential flow generator on the MNIST. In each raw, the top images the inputs, while the bottom images are the corresponding outputs.
should be close in the L2 distance.
4.2.2
THE CELEBA DATASET
Original
Projection
C-PFG (ours)
Vanila
CycleGAN FNN, L1
CycleGAN FNN, L2
CyclGGAN Official
CycleGAN FN
CycleGAN
FNN,
CycleGAN
Official
Figure 5: Comparison between our method, vanilla WGAN-GP. CycleGAN with WGAN-GP as GAN loss and L, or Tog loss as cycle-consistency loss, as well as official CycleGAN. The first two the original images and their projections the 700-dimensional Euclidean space induced by PCA, which are similar. The next five rows are the corresponding outputs. The last three rows are the reconstructed images from different CycleGANs.

---

Under review as a conference paper at ICLR 2020
In this section we test the translation task between the CelebA images (Liu et al. 2013). We randomly pick 60000 images from the CelebA training dataset and divide them into two clusters: (a) images with attribute "smilinn" labeled as false, and (b) images with attribute "smiling" labeled true. The images are cropped so that only faces remain on the images. We view the two clusters as samples of and v, respectively. The total number of components in PCA is set 700. Note that our goal is to generate images of smiling faces belonging to the same people as in input images. The difficulty lies in that the training data are unpaired and actually we do not have the ground truth as reference.
We compare our method against vanilla WGAN-GP, and CycleGAN with WGAN-GP for GAN loss. In CycleGAN we use L1 or L2 loss as cycle-consistency loss, with weight 10 as in Zhu et al. (2017). To make a fair comparison, for each generator and discriminator, we use a feedforward neural network (FNN) with the same size of hidden layers (5 x 256). We also test the official CycleGAN! on this problem, where we feed the cropped face images (with resizing but without random jitter cropping) instead of embedding vectors into the model. The results of images randomly picked from the test dataset are shown in Figure 5
For most of the images, our method could successfully translate the no-smiling faces to smiling faces belonging to the same people. Some of the output images are blurred, since it is difficult to learn the high order modes of PCA with FNN. Vanilla WGAN-GP and CycleGAN with FNN totally failed, in that the input and output images come from different people. This comparison clearly showed the necessity of additional regularity for the generators in translation tasks with only unpaired data, and that GAN loss cycle-consistency loss cannot provide sufficient regularity.
The official CycleGAN is decent in performance, generating images less blurred than our method, but failed change the countenance for some images. Note that the number of parameters in official CycleGAN is about 30 times more than that in our method, and the total training time is more than two times of our method on a single NVIDIA Tesla V100 GPU.
DISCUSSION AND CONCLUSIONS
In this paper we propose potential flow generators with L2 optimal transport regularity as plug-and- play generator modules that could be easiy integrated in a wide range of generative models. In particular, we propose two versions: the discrete one and the continuous one. For the discrete version, the L2 optimal transport regularity is directly encoded in, while for the continuous version we only need a slight augmentation to the original generator loss functions to impose the L2 optimal transport regularity.
We firstly show that the potential flow generators are able to approximate the L2 optimal transport maps in 2D problems. The continuous potential flow generator outperforms the discrete one in robustness. The continuous potential flow generator is also applied to WGAN-GP and continuous normalizing flow models, where we illustrate the characteristic of "proximity" for the potential flow generator due to the L2 optimal transport regularity. Consequently, we show the effectiveness of method in image translation tasks using unpaired training data from the MNIST dataset and the CelebA dataset. We can see that our method significantly outperforms the vanilla WGAN-GP and CycleGAN using FNN with the same size of hidden layers.
We think that the results of our method in translation tasks are impressive considering that we only use PCA, linear embedding method, with only feedforward neural networks. Such naive strategy actually leads to the blurred patterns in the output images, which is also the case (even more severe) for vanilla WGAN-GP and CycleGAN using the same strategy. A possible improvement is to integrate the potential flow generator with other embedding techniques like autoencoders with convolutional neural networks. Apart from torage-to-image translations, it is also possible to apply the potential flow generator to other translation tasks, if the translation objects could be properly embedded into Euclidean space. Moreover, it was perceived that the training of ODE- based model is slow, but the training of our method could be accelerated by applying methods related to optimal transport, e.g. the Wasserstein natural gradient method (Tong Lin et al. 2018 Li & Montúfar 2018). We leave these possible improvements to future work.
We used the code from http::/wwww.enssrrooow..... wwtp://wwooooalleeeeerrvrvveeeee The size of training dataset is reduced to 1000 for official CycleGAN, which is similar to the horse-to-7ebra dataset. The number of total epochs is set back to 200 in the original CycleGAN paper.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2008.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning pp. 211-2212222 2017.
Mare G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos. The Cramer distance as a solution biased Wasserstein gradients. arXiv preprint arXiv: / 700077003.. 2017.
Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the monge- kantorovich mass transfer problem. Numerische Mathematik, 84(3):375-393, 2000.
Yann Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Commu- nications pure and applied mathematics, 44(4):375-417 1991.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. In Advances in Neural Information Processing Systems, pp. 6571-6583, 2018.
Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced Wasserstein distance. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3483-3491, 2018.
Wilfrid Gangbo and Robert J McCann. The geometry of optimal transportation. Acta Mathematica. 177(2)113--6111 1996.
Matthias Gelbrich. On a formula for the L2 Wasserstein metric between measures on Euclidean and Hilbert spaces. Mathematische Nachrichten, 147(1):185-203, 1990.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 1 122--1134, 2017.
Ian Jolliffe. Principal component analysis. Springer, 2011.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to dis- cover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1857-1865. JMLR. org, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1et..412.669.. 6,800, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 150098771 1998.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yaam. lecun. com/exdb/mnist, 2, 2010.
Jacob Leygonie, Jennifer She, Amjad Almahairi, Sai Rajeswar, and Aaron Courville. Adversarial computation of optimal transport maps. arXiv preprint arXiv: /906: 1906. 096619991, 2019.
11

---

Under review as a conference paper at ICLR 2020
Wuchen Li and Guido Montúfar. Natural gradient via optimal transport. Information Geometry, 1(2): 181-214, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.
Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 2018.
Felix Otto. Viscous fingering: optimal bound the growth rate of the mixing zone. SIAM Journal on Applied Mathematics, 57(4))982-990, 1997.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport. Foundations and Trends in Machine Learning, 156-6))355-677, 2019.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 1rXiv:1711.105611 2017a.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations. arXiv preprint arXiv:1711.10566
2017b.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:1505.055 05770, 2015.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving GANs using optimal transport. arXiv preprint arXiv: /0tv::/800.0000 005.05573, 2018.
Filippo Santambrogio. {Buclidean, metric, and Wasserstein) gradient flows: overview. Bulletin of Mathematical Sciences, 7(1):87-154, 2017.
Vivien Seguy. Bharath Bhushan Damodaran, Rémi Flamary, Nicolas Courty, Antoine Rolet, and Mathieu Blondel. Large-scale optimal transport and mapping estimation. arXiv preprint
arXiv:171..0228 2017.
Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1366 2018.
Alex Tong Lin, Wuchen Li, Stanley Osher, and Guido Montúfar. Wasserstein proximal of GANs. 2018.
Giulio Trigila and Esteban G Tabak. Data-driven optimal transport. Communications on Pure and Applied Mathematics, 6944)6613-64,, 2016.
Karren D Yang and Caroline Uhler. Scalable unbalanced optimal transport using generative adversarial networks. arXiv preprint 10Xiv.1810.11444 2018.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. DualGAN: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2849-2857, 2017.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2020
A PROOF OF THE OPTIMAL MAPS IN SECTION 4.1
In problem 1, for Gaussian distributions j and v with mean my and m2, as well as covariance matrices Σ1 and Eg. from Gelbrich (1900) we know that the minimum transport cost from μ with cost function c( y) = x y is
m m2 Tr(E, + Σ - 2( Σ /
(18)
which known as the squared Wasserstein-2 distance between two Gaussian distributions, In particular, the minimum transport cost is 0.5 in our problem.
For the map f((,, v)) (2x, 0.5y), # is Gaussian since is linear. By checking the mean and covariance we have f# v. Also, the transport map of f is
Ex~A11..0.0) (2x x²² + 10 (0.5y (0.5) y)² 0.25 + 0.25 0.5
(19)
which is exactly the minimum transport cost, thus f is the optimal transport map. We complete the proof of the optimal transport map in problem 1.
In problem 2, denote X = [0.5, 1). Y [2, 2.5). O = [0, 2m). and m. = 4(), m2 = u(V), mg = OO), where we use (A) represent the uniform probability measure on set A.
U(Y) x U(O), we have fu v. For any transport map from μ to v, denote as g 0)) = (r, (r, 0)) in polar coordinates, we only need to show that the transport cost of y is no less than the cost of f.
For f((r, 8)) (r + 1.5, 0), μ = ((x) U()).
Let (((r, 0)) = (g, 0), then the transport cost of g is no less than the cost of since the transport cost is reduced for any point (r, [
Actually we could view (r, 0) as multivalued function so that (r, 0) induces a transport plan If from m to m2. More formally, define measure H. B(X x Y) + R by
(A) = [ (a M dy)dm (x) = =/ Mxx dy)dm1
(20)
where M(..): X x B(V) + R is defined by
M(x, A) = [ 1g.(e.0)eadmm(())
(21)
To see H is a transport plan from my to m2, we need to check:
1. VA E H( x Y! This true since
(A).
x(AXV= S.Je M(x, dy)d / M(x, W)dmi(x) m1 mA),,
(22)
where we utilize that Mx, Y) =
2. VA E B(Y), HX x A) m2 (A))
Note that 9# = v, thus pág -1 x = =) mA) Also,
p (A 0)) [ 1gl=:)eaxxadmmo()dmm (x)
[ dm) (x)
(23)
M (dd
H x A).
Therefore H(X x A. -2(A)..
'I'ss not necessary that h
13

---

Under review as a conference paper at ICLR 2020
We also claim that the L2 transport cost of h equals to that of H. The transport cost of h and H are
C(h) = / (gr 0) - (m))dmm (0)dm)
(24)
C(H) = / (y x)dd dH - / (y x)2 M
xxXY
respectively. By the definition of M, have (x, =gr(x, .) me for any a E X, thus
1 () dy) / (gr (z, )²² dms
x)²
(25)
for any x E X. Therefore C(h) C(H).
Let F(x) = + 1.5 be another transport plan from mi to m2g clearly the Lg transport cost of equals to that of F. Note that the transport cost of H is no less than that of F, since the latter one is the optimal transport plan from mi to m2. This complete the proof of claim that the transport cost of g is no less than that of f, and thus the proof of the optimal transport map in problem 2.
B DETAILS OF LOSS FUNCTIONS IN CONTINUOUS NORMALIZING FLOWS
To estimate log p (v), we have the ODE that connects the probability density at inputs and outputs of the generator:
dt ltt(p(((t, a))) V a ((t, Aanott a(t, a)),
(26)
for æ in the support of μ. where the initial probability density p(ū x)) = (x)))) the density of μ at input æ, while the terminal probability density (T) PG (G(x)) is the density of Gyju at output ((x).
Also, we estimate x = - (zy by solving the ODE
du
dt = Ü(T - t, w)
(27)
with initial condition (0) as y G(w)) and w(T) as the corresponding x Gy)
For each y. we can use Equation 27 to estimate the corresponding x = G and consequently (og(p,(<)) since have the density of μ.. Then apply Equation 26 to estimate log PG (y). By sampling y v, Equations 26 and 27 properly. For example, we use the first-order Euler scheme in our practice. Note that when applying maximum likelihood training, the density of could be unnormalized, since multiplications with Рμ would merely lead to a constant difference in the loss function.
estimate Ey-r (log PG (y)]. In practice, we also need to discretize
C NETWORKS AND HYPERPARAMETERS
Except in the regulized discrete optimal transport solver and official CycleGAN, all the neural networks are feedforward neural networks of 5 hidden layers, each of width 128 for 2D problems, or 256 in image translation tasks. For potential flow generator, the activation function is tanh for the smoothness of b. The activation function in vanilla generator in Figure 2 is also tanh. All the other activation functions are leaky ReLU (Maas et al. 2013) with negative slope 0.2, except in the regulized discrete optimal transport solver and official CycleGAN.
The batch size is set as 1000 for all the cases except in Table 2 and in official CycleGAN, We use 1000 random projection directions to estimate the sliced Wasserstein distances. In WGAN-GP model the coefficient for gradient penalty is 0.1, and we do 5 discriminator updates per generator update.
In potential flow generators, the time span T is 1.0. We set the number of total time steps 4 in discrete potential flow generators, while n = 100 in continuous potential flow generators for 2D problems and 10 in image translation tasks. The PDE penalty weight À for continuous potential
14

---

Under review as a conference paper at ICLR 2020
flow generator is set 1.0 by default, except those in the 2D problems where compare different generators.
We use the Adam optimizer (Kingma & Ba 2014) for all the problems. In the 2D problems, the learning rate is set as / = 10 D1 0.5, Bg 0.999 for sliced Wasserstein distance, while set as 10-5 B1 0.5, 82 0.9 in WGAN-GP. We train the generators for 100, 000 iterations in
Figure 2 and Figure Sal and 20, 000 iterations Table 2 In the normalizing flow model we set - 10 B 0.9, 32 0.999, and train the generator for 10, 000 iterations. In image translation
tasks we set 10-1, 31 = 0.5, 32 0.9, and train the generator for 100, 000 iterations for our
method and CycleGANs with FNN, while 200, 000 iterations for vanilla GAN.
D MORE RESULTS ON THE MNSIT AND CELEBA DATASET
0
0
0
0
6
7
6
6
6
6
5
6
6
6
6
I
/
1
/
/
1
5
8
6
1
6
7
7
7
7
5
1
7
7
9
2
9
2
2
2
2
2
2
2
2
2
2
2
2
6
6
6
7
6
8
8
0
6
6
9
3
3
8
3
3
3
3
3
3
3
3
3
3 5
3
3 6
5
8
8
5
5
8
8
5
5
5
5
8
5
4
4
4
4
4
N
4
4
4
4
4
4
4
4
4 9
9
9
9
9
9
9
9
4
9
9
9
9
1
9
(a) Training dataset
0
0
b
0 6
6
6
6
S
6
6
7
6
1
|
/
{
/
I
I
8
7
7
7
8
9
7
7
8
7
9 2 2
7
7
6
2
2
2
2
2
2
2
2
2
2
2
2
2
7
3
8
8
8
6
6
7
7
7
8
4
6
6
6
3
3
3
3
3
3
3
3
3
3
3
3
3
3
$
5
8
5
5
8
S
8
5
8
8
8
9
5
8
4
4
4
4
4
9
4
4
4
4
4
4
4
4
4
4
6
9
9
9
9
9
9
9
4
7
9
9
6
(b) Testing dataset
Figure 6: Potential flow generator the MNIST (a) training and (b) testing dataset. In each raw, the top images are reconstructed from the input vectors, while the bottom images are reconstructed from the corresponding output vectors.
15

---

Under review as a conference paper at ICLR 2020
(a) Training dataset
(b) Testing dataset
Figure 7: Potential flow generator on the CelebA (a) training and (b) testing dataset. In each raw, the top images are reconstructed from the input vectors, while the bottom images are reconstructed from the corresponding output vectors.
16


Reviews:
[{"review_id": "SkxDSfz8FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a ‘potential flow generator’ that can be seen as a regularizer for traditional GAN losses. It is based on the idea that samples flowing from one distribution to another should follow a minimum travel cost path. This regularization is expressed as an optimal transport problem with a squared Euclidean cost. Authors rely on the dynamic formulation of OT proposed by Benamou and Brenier, 2000. They propose to learn a time-dependent potential field which gradient defines the velocity fields used to drive samples from a source distribution toward a target one. Experiments on a simple 1D case (where the optimal transport map is known), and on images with an MNIST / CelebA qualitative example.\nThe use of this dynamic formulation is well known in the OT community. See as a good examples:\nTrigila, G., & Tabak, E. G. (2016). Data‐driven optimal transport. Communications on Pure and Applied Mathematics, 69(4), 613-648.\nand more generally Chapter 7 of the ‘Computational Optimal Transport’ book by Peyré and Cuturi.\n\nThe novelty arises from the use of neural networks to represent the potentials. However, the claim that the obtained map is the optimal transport map seems wrong to me, because:\nThe class of potential functions over which the optimization is performed is not the whole class of functions, leading to approximations;\nThe optimality conditions (a.k.a continuity or preservation of mass equations) are only enforced on sampled trajectories, not on the entire space. \nWhile this claim should definitely be lowered, it is nonetheless still acceptable provided that the proposed model is performing good. On this part, the paper strength could be improved provided that comparisons with existing methods computing a Monge map could be given. Notably, a comparison with the approach from Seguy et al. 2018 is missing. On a same level, a qualitative comparison with cycleGAN  in Figure 4 is missing. \nThere are some unclear elements in the paper. The final, total, optimization problem is never clearly expressed. I believe a general algorithm presentation could help in understanding the general picture of the method. Notably, for instance, It is still not clear if the generator G is disconnected from the potential definition (following Eq. 10 I assume not). How are the trajectories sampled ? Is the discriminator trained on the same sampled trajectories or different ones ?  In the potential generator, it seems that the time is considered the same way as the feature space. Can you comment on this point ? Is it possible to evaluate the flow of different time stamps that the ones used for training ? \n\nAs a summary,\nPros:\nAn interesting way to represent time-dependent potentials with a network for regularizing generative models\nCons:\nNot much theoretical novelties in the paper, nor a good analysis on the source of errors of the model (e.g. impact of discretization on the problem)\nThere are some unclear aspects in the paper (see comments)\nThe potential benefits of the approach over the state-of-the-art should be more clearly discussed. \n\n\nMinor comment:\n P3. Uniqueness of Monge problem for the squared Euclidean cost should be attributed to Brenier 91 and his polar factorization theorem. McCann generalized it to Riemannian manifold.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJedfWUaYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "######## Updated Review ############\n\nThe author(s) have presented a sincere rebuttal, which I really appreciate. Although I still don't quite agree with all the points made by the author(s), I have changed my mind to be more or less borderline about this submission, given that the author(s) have gone through great length to clarify and improve their manuscript. \n\n\n#################################\n\nThis paper proposed a generative modeling framework called potential flow generator. Instead of deriving new matching criteria between distributions, the authors considered redefining the generative process via simulating a continuous flow that is constrained by the optimality conditions on the flow potential field derived based on L2 optimal transport. This is certainly an interesting direction to explore, however, while the points made are valid, they are not well justified. My major criticism is that too much compromise needs to be made in order to construct such a flow generator. In practical terms, it's computationally costly and sacrifices too much of the network's flexibility. My overall evaluation for this work is a straightforward/brute-force application of well-known (but less practical) results, without proposing any remedies to the real challenges that underlie. My detailed comments are listed below:\n\n1. It is assumed that the input distribution should have the same ambient dimensionality as the target distribution, and the continuity constraints mean that the deforming to the target can take an excruciatingly slow pace, the main obstacle faced by all flow-based constructions. This point is partly evidenced by the experiment section where none of the input distributions is far from the target. It's questionable whether this framework can efficiently perform \"generative modeling\", in which a simple noise distribution is pushed to a more sophisticated target distribution. \n\n2. Relations to the Neural ODE literature is not sufficiently discussed, which I believe is closest to this work. A major drawback of Neural ODE is slow computations. \n\n3. While the author(s) have criticized an intuitive construction of L2 transport penalty in Eqn (13), their objective Eqn (17) suffers a similar issue. \n\n4. The experiments are weak and not convincing. First, ss mentioned in earlier comments, 2D toy transport and image translation are fairly easy tasks. Second, only qualitative results are reported, and there is no baseline model to compare with. Third, without ablation study, We can hardly verify the fact the gains are actually coming from the flow part, as vanilla GANs can also perform a similar task. \n\nMinors: Additionally, language issues can be spotted here and there. The author(s) should more carefully proofread this manuscript. And I find it confusing that Fig 3 (a) and (b) uses different examples for WGAN and CNF. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gXPSXhYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a great paper using optimal transport theory for generative and implicit models. Instead of using general vector fields, the authors apply the potential vector fields in optimal transport theory to design neural networks. The mathematics is correct with convincing examples. This brings an important mathematical connection between fluid dynamics and GANs or implicit models. \n\nI suggest the acceptance of this paper after addressing the following minor questions. \n\n1. Would the authors provide slightly more details about the design of networks? \n\n2. In the literature, the author may need to cite \n\n\"A. Lin, W. Li, S. Osher, G. Montufar, Wasserstein proximal of GANs, 2018.\"\n\"W. Li, G. Montufar, Natural gradient via optimal transport, 2018\"\n\nThe Wasserstein natural gradient method there may improve the computational speed of the proposed models. \n\nIn all, this is an exciting paper with many potentials in future neural network designs. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes applying potential flow generators in conjunction with L2 optimal transport regularity to favor solutions that "move" input points as little as possible to output points drawn from the target distribution.  The resulting pipeline can be effective in dealing with, among other things, image-to-image translation tasks with unpaired data.  Overall, one of the appeals of this methodology is that it can be integrated within a number of existing generative modeling paradigms (e.g., GANs, etc.).

After the rebuttal and discussion period, two reviewers maintained weak reject scores while one favored strong acceptance.  With these borderline/mixed scores, this paper was discussed at the meta-review level and the final decision was to side with the majority, noting that a revision which fully addresses reviewer comments could likely be successful at a future venue.  As one important lingering issue, R1 pointed out that the optimality conditions of the proposed approach are only enforced on sampled trajectories, not actually on the entire space.  The rebuttal concedes this point, but suggests that the method still seems to work.  But as an improvement, the suggestion is made that randomly perturbed trajectories could help to mitigate this issue.  However, no experiments were conducted using this modification, which could be helpful in building confidence in the reliability of the overall methodology.

Additionally, from my perspective the empirical validation could also be improved to help solidify the contribution in a revision.  For example, the image-to-image translation experiments with CelebA were based on a linear (PCA) embedding and feedforward networks.  It would have been nice to have seen a more sophisticated setup for this purpose (as discussed in Section 5), especially for a non-theoretical paper with an ostensibly practically-relevant algorithmic proposal.  And consistent with reviewer comments, the paper definitely needs another pass to clean up a number of small grammatical mistakes.
Decision: Reject