Sample and Map from a Single Convex Potential: Generation using Conjugate Moment Measures
Authors: Nina Vesseron
Louis Béthune
marco cuturi
Venue: NeurIPS 2025 poster ()
Keywords: optimal transport
generative modelling
input convex neural networks
Primary area: deep_learning

Abstract: The canonical approach in generative modeling is to split model fitting into two blocks: define first how to sample noise (e.g. Gaussian) and choose next what to do with it (e.g. using a single map or flows). We explore in this work an alternative route that ties sampling and mapping. We find inspiration in moment measures, a result that states that for any measure $\rho$, there exists a unique convex potential $u$ such that $\rho=\nabla u \sharp e^{-u}$. While this does seem to tie effectively sampling (from log-concave distribution $e^{-u}$) and action (pushing particles through $\nabla u$), we observe on simple examples (e.g., Gaussians or 1D distributions) that this choice is ill-suited for practical tasks. We study an alternative factorization, where $\rho$ is factorized as $\nabla w^* \sharp e^{-w}$, where $w^\ast$ is the convex conjugate of a convex potential $w$. We call this approach conjugate moment measures, and show far more intuitive results on these examples. Because $\nabla w^*$ is the Monge map between the log-concave distribution $e^{-w}$ and $\rho$, we rely on optimal transport solvers to propose an algorithm to recover $w$ from samples of $\rho$, and parameterize $w$ as an input-convex neural network. We also address the common sampling scenario in which the density of $\rho$ is known only up to a normalizing constant, and propose an algorithm to learn $w$ in this setting.

Sample and Map from a Single Convex Potential: Generation using Conjugate Moment Measures
Nina Vesseron CREST-ENSAE, IP Paris nina. www..veeeeerroessee.....
Louis Béthune Apple 1_ b_bethuneaapple.. com
Marco Cuturi CREST-ENSAE, Apple
cuturiSapple.com com
Abstract
The canonical approach in generative modeling is split model fitting into two blocks: define first how to sample noise (e.g. Gaussian) and choose next what do with it (e.g. using a single map or flows). We explore this work an alternative route that ries sampling and mapping. We find inspiration in moment measures
Eraussuui and Klartag, 2015]. result that states that for any measure there exists unique convex potential such that Vute-", While this does seem to tie effectively sampling (from log-concave distribution e=*) and action (pushing particles through Vu), observe simple examples (e.g.. Gaussians or ID distributions) that this choice ill-suited for practical tasks. We study alternative factorization, where P is factorized as Vw te-w, where w" is the convex conjugate convex potential w. call this approach conjugate moment measures, and show far more intuitive results these examples. Because Vw is the Monge map between the log-concave distribution e and p. we rely optimal transport solvers to propose algorithm to recover w from samples p. and parameterize w as an input-convex neural network. We also address the common sampling scenario in which the density of p is known only up normalizing constant, and propose an algorithm to learn w in this setting.
1 Introduction
decade after the introduction of GANs Goodffllow et al. 2014] and VAEs (Kingma and Welling 2014), the field of generative modeling has grown into of the most important areas of research in machine learning. Both canonical approaches follow the template learning a transformation that can map random codes to meaningful data. These transformations can be learned in supervised manner, as in the dimensionality reduction pipeline advocated in VAEs, or distributionally as advocated the purely generative literature. In the latter, the variety of such transforms has gained remarkably in both complexity, using increasingly creative inductive biases, from Neural-ODEs((hhen et al. 2018 Grathwohl al. 2018), diffusions (Song al. 2020 Ho al. 2020). optimal transport Korotin
et al. 2020 to flow-matching [Lipman et al. 2023 Tong et al. 2023 Pooladian et al. 2023
Sample first, move next. All of these approaches grounded, however, choosing standard Gaussian multivariate distribution to sample noise/codes. For both GANs and VAEs, that choice is usually made because of its simplicity, but also, in the case of diffusion models, because Gaussian Cannstiin-Uhlembede
distributions can be recovered quickly with suitable stochastic processes Other works have considered optimizing prior noise distributions [Tomezak and Welling 2018 Lee et al. 2021 Liang et al. 2022 but still do so a two step approach where mappings are estimated
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

independently. We explore in this work a new generative paradigm where sampling and transforms are not treated separately, but seen as two facets of the same single convex potential.
Tying sampling and action. Our work looks to moment measures Frautrr-r and Klartag 2015 [Santambrogio 2016] to find inspiration for factorization that ties both sampling and action. Cordero-Erausquin and Klartag (2015) proved that for any sufficiently regular probability measure p. one can find an essentially-contiuuuus convex potential such that p = Vu f "; essentially, any probability distribution can be recovered by first sampling points from a log-concave distribution e e" and moving them with Vu where, remarkably. the same single convex potential u is used twice.
Our Contributions. We build on the work of Cordero- -aallruu-r and Klartag 2015 to propose an alternative path to build generative models, that we call conjugate moment measures.
After recalling in detail the contribution of Cordero- Cordero-Frauuquin and Klartag 2015 in 2 we argue in $3.1 that the moment measure factorization may not be suitable for practical tasks. For instance, in the case of Gaussian distributions, we show that p is Gaussian with variance Σ, the corresponding log-concave distribution e is Gaussian with variance V-1, amplifying beyond necessary any minor degeneracy in Σ. Similarly, for univariate p. we found that the spread of e is inversely proportional to that of p. While these results may not be surprising from theoretical perspective, they call for a different strategy to estimate a tied sample/action factorization.
We propose instead in $3.2 new factorization that is conjugate to that of Cordero Erausquin and Klartag 2015): We show that any absolutely continuous probability distribution supported on a compact convex can be written as p Vw pe- where w" is the convex conjugate of w. Importantly, in our factorization the convex potential w is used to map P to e e- since one has cquivalently that Vw = e " Note that evaluating w", the convex conjugate of w. only requires solving a convex optimization problem, since w is assumed to be convex.
-CCouueno-Erausquin
We explore how to infer the potential w from the conjugate factorization using either samples from p or its associated energy function when the density of p is only known up to a normalizing constant, leveraging optimal transport (OT) theory (Santambrogio 2015). Indeed, since the condition linking e to p, we can parameterize w as an input-convex neural network wy whose gradient is an OT map. We provide illustrations on simple generative examples.
0 V w" e is equivalent to stating that w² is the Brenier potential Preenicrpotentiall
2 Background
Optimal Transport. Let P(Rd)) denote the set of Borel probability measures on RRd For μt, v E P(RR), let 11(,, v) denote the set of couplings between je and We consider the primal OT problem,
(F, v) = inf [
2 +
f/ (x,
xEll(p,e) "d"
with squared- Euclidean cost. This problem admits a dual formulation:
[
[ gdv, s.t. f(x) + g(v) < 3 1| - yl12 ((I,)) E Rd x Rd
sup (fa))et) (I) J fdp +
(2)
The functions f and g that maximize the expression in ( are referred to as the Kantorovich potentials. Equation (1) can itself be seen as relaxation of the Monge formulation of the OT problem,
inf T -
2
1 - ((=) p(dz)
(3)
gd
Tiu=r
where is the pushforward operator. An important simplification of (2) comes from the fact that solutions f g" for equation (2) must be such that
f is convex. Using this parameterization, let C(RR)) be the space of convex functions Rd Brenier proved that in that case, a T solving
problem B exists, and is the gradient of the convex function 11 1|2 -f", namely Id - Vf*. Making
change of variables, u = - 1|2 ff and using the machinery of cost-concavity (Santambrogio 2015 1..4, one can show that solving (2) is equivalent to computing the minimizer of:
Jpd u(x) du(x)) + 1fc u" (x) dv(x)
min WEC =(0)=

---

where u² == max (z, u(x) is the convex conjuaco of u and noting that we lift any ambiguity on u by selecting the unique potential such that 2()) = 0. By denoting B(H, 2) the Brenier potential solving problem (4) above, this results in v = Bi, v) ye.
Neural OT solvers. The goal of neural OT solvers is estimate (( v) using samples drawn from the source μ and the target distribution v. Makkuva et al. 2020]. Korotin et al. [2020] proposed methods that leverage input convex neural networks (ICNN), originally introduced by Amos et al. [2017]. to parameterize the potential u as an ICNN. A key challenge in these approaches is handling the Legendre transform u². To overcome this, surrogate network can approximate u" with recent
advances by Amos 2023 improving these implementations through amortized optimization.
Moment Measures. For given convex function u. the moment measure of is defined as the pushforward measure Vut n. where P. is the log-concave (Gibbs) probability measure with density
Pz) := Jud e- The moment measure of is well defined when the quantity fg- e e=((=)dz is positive and finite. While any measure supported on a compact set K' is guaranteed to be the moment measure of some convex potential u (Santambrogio) 2016), Kordero-Erausquun and Klartag
e-u(=)
showed that u is often discontinuous at the boundary ØK. By studying the variational problem
u''pp In (ʃ )
min (EC(R4)
(5)
Cordero- Eaalerr-Erruuuunn and Klartag 2015], proved that if p is a mcasure with a finite first moment, a barycenter at 0 and is not supported hyperplane, then it is the moment measure of an essentially continuous potential (in the sense of -aaderr-rrauquin and Klartag (2055).
From Moment Measures to Conjugate Moment Measures
For measure P. we call the convex function u that satisfies Vurf, = p the moment potential of p. Additionally, we refer to the Gibbs distribution associated with such potential, Bu. as the Gibbs factor of p. We show in this section that moment measures are not well suited for generative modeling purposes, as they often produce a Gibbs factor that differs significantly from the target distribution This limitation is particularly evident in the case of Gaussian distributions, which motivates the introduction of an alternative representation: the conjugate moment measure.
3.1 Limitations of Moment Measures
By writing the KI. divergence between and F.- as kl((pl In(pddp + Tu'dy
In (J e e-"")) and applying this in the variational problem studied by Cordero-Eraana Erausquin and Klartag as referenced (5), we obtain that the moment potentials of p minimize the following problem:
min (L(P)19..-) - In (fo" c-")
(6)
"EC(R")
Equation 6 reveals that the Gibbs distributions associated with the Legendre transform of p's moment potentials, Bu.. are the log-concave distributions closest p. up to the regularization term This quantity has been studied by [Blll (1966) and Artstein-Avidan et al. [2004), who showed that for convex functions u whose corresponding Gibbs factor F. has its barycenter at the origin, the term - In (ʃ e { Te-"") attains its minimum value of -d (n2r)) when P. is Gaussian. A direct consequence of (6) is that the Gibbs factor of p can differ significantly from particularly when p is Gaussian, as formalized in the following proposition:
-In (fe-"
Proposition 1. Let N (0ga E). If Σ is non degenerate, the moment potentials of p are un (2) {(r -lr-me 3)mmm)2 "[ - m., with m E Rd. The associated Gibbsffacton of p is Tu.. = =N(m, 1-1).
Proof. The optimization problem in equation (6) is translation invariant: replacing u with x + u(x + a) for any a E Rd leaves the objective unchanged. We can thus restrict to convex functions u such that F. has barycenter at zero. In this setting, choosing u(x) = f T Σx yields P. = (o 1ー and T.. (O, E). which minimize the first term in (6). while the second term reaches its minimum value of (2 [Artstein-Auie Avidan et al., 2004]. Hence, this choice minimizes the full objective, and translations of u span the set of moment potentials of p.
3

---

Intuitively this results can be interpreted as follows. When a peaked distribution p is centered around the origin (e.g. Σ R 0), its corresponding moment poten- tial is such that the image set Vu on the support of Pu is necessarily tightly concentrated around O. This has the implication that u is a slowly (almost constant) varying potential on the entire support of T... As result, one has the (perhaps) counter-intuitive result that the more peaky p. the more spread-out Fu must be. From this simple observation-vaaliatee experimen- tally in Figure 2 we draw the intuition that change is needed to reverse this relationship, while still retaining the interest of measure factorization result.
Conj. Gibbs Moment Y
Gibbs Moment P
P
Figure 1: Gibbs factor and conjugate
2
1.8
Gibbs factor of p N 0.
1.8
2
3.2 The Conjugate Moment Measure factorization
Our main result, which establishes the conjugate moment measure factorization, stated below: Theorem Let E P(RR) be an absolutely continuous probability measure supported on a compact, convex set. Then, there exists convex function w such that V 1
Accordingly, now refer to p the conjugate moment measure of w, the conjugate moment
potential of p. and T... as the conjugate Gibbs factor of p. Our proof strategy differs significantly
from that used in Goardero-Frausuii and Klartag [015) and (Santambrogio (2016) since we use Schauder's 's fixed point theorem to show that the fol lowiinng map admits a fixed point:
(7) where ((Rd) is the set of functions mapping TRd to R U (+x),, and G, assigns to a potential w the Brenier potential transporting p to P.. Fixed points of Gp correspond precisely to the conjugate moment potentials of p. and their existence guarantees a solution the moment measure factorization. A sketch of the proof is provided in Appendix A. with full details in Appendix AA2 When is Gaussian, the conjugate moment measure factorization admits an explicit solution, as given in Proposition
Gp : ((R)) + L(R"), Gp(w) := := Pw.))
2 Recall that we saw in $3.1 Gibbs factors of ( Σ) were multivariate normal distributions with covariance matrix Σ I
Proposition 2. When D is non-degenerate, = (m,, is the conjugate moment measure of r), whose Gibbs distribution is Fu N(r, 11/3) where r (La / The function w is the unique conjugate moment potential of p whose Gibbs distribution remains Gaussian.
w(x) = (x --)''r-//((
The proof is provided in Appendix A.4 and relies on the fact that the OT map between two Gaussians is known in closed form (Peyré et al.) 2019]. While this potential w may not be the only conjugate moment potential, the Gibbs distribution associated with it, (r, 2/33), appears to be better suited to the target distribution P. This is particularly evident in the example shown in Figure [ Additionally, it is worth noting that for the Gaussian case, factorization still works when p not centered, whereas the approach of Cordero- -rautruur and Klartag 2015 requires to be mean 0.
3.3 Monge-Ampère Equation for Conjugate Moment Measures
From the moment measure factorization, = Vuffu.. which holds at the level of probability distributions, one can derive the corresponding equality between probability density functions and obtain the following Monge-Ampère equation (Cordero-Erausqui and Klartag 2015):
(8) where u² is the convex conjugate of u, H.- (ar) denotes the Hessian of u" ær, and C. = f-= is the normalizing constant ensuring that B. e ""/C, is probability distribution. Similarly, Theorem I guarantees the existence a convex function w that satisfies the following Monge-Amper cquation for any absolutely continuous probability measure p supported
p(z) e - (x) with Ey, = u(Vu" (x)) - In(det H. (x)) + In(C.))
compact, convex set:
p(x) = e (x)(x) where (x) = w (w(z)) In(det Ha (z)) + Il((W,)
(9)
with (x)) denoting the Hessian of w at x and C = e- being the normalizing constant of P... The potentials that solve (9) are precisely the conjugate moment potentials of p.

---

Estimating Conjugate Moment Potentials in Practice
We first explain how to sample from p when one of its conjugate moment potential is known, using the Langevin Monte Carlo (LMC) algorithm and a conjugate solver. We then describe a method to estimate a conjugate moment factorization of p using i.i.d samples (21,..., Fn) p. In this approach, the conjugate moment potential of is parameterized using an input convex neural network (ICNN) Wa following the architecture proposed in Vesseron and Cuturi| [002]. The conjugate moment potential is then estimated using algorithm inspired by the fixed-point method associated to Theorem 1 Finally, we address the case where the density of p is known up to a normalizing constant, common scenario in sampling; we use an ICNN to parameterize the potential and estimate it via regression using the Monge-Ampère equation
4.1 Sampling from p using its Conjugate Moment Factorization
In this paragraph, we suppose that we know a conjugate moment potential w of p. i.e. Vw" 1 Knowing w, drawing samples from p can be done by first sampling P' and then applying Vw to those points as Vw (I) ~ Vw" p = p. The LMC algorithm is a widely used method for generating samples from a smooth, log-concave density like T [Roberts and Tweedie 1996 Cheng and Bartlett 2018 Dalalyan and Karagulyan 2019). Starting from an initial point (0)), the LMC
algorithm iterates according to the lollowing update rule:
x (k+11 = (kk) 7 [u)(x(t)) + (2r2(k) 2(h)
(O Ta);
where 7 is the step size. As for the gradient of the convex conjugate Vw", can be efficiently estimated from By applying Danskin's envelope theorem [1966]. it follows that V (v) is the solution to the following concave maximization problem: V W (w) = arg sup, (a. y) w(r) This optimization problem can be solved using algorithms such as gradient ascent, (L)BFGS [Liu and Nocedal 1989]. or ADAM [Kingma and Ba 2014]. Thus, having access to a conjugate moment potential of p enables to efficiently draw samples from it. Note that the gradient steps of the conjugate solver can be interpreted as denoising procedure applied to samples drawn from Fa.
4.2 Learning the Conjugate Moment Factorization from Samples: CMFGen
We consider the case where p (( only be accessed through samples, in empirical distribution approximation Pn := 1 E òz,- The fixed points of the map Gp. as defined in (7). correspond exactly to the conjugate moment potentials of p. This observation motivates the following fixed-point iteration scheme to compute a conjugate moment potential of p:
Wg ill 11:: 2 1, W++1 Gwz),,
(10)
Starting from wig ! 11 which corresponds to Bwo (0, Ta), we iteratively compute the 2
Brenier potential 1 between the distribution p and Q Bw. at iteration t 2+ During the next iteration t + 2, the updated distribution Tw becomes the target distribution to compute the next Brenier potential starting from the source p. This process is repeated until the algorithm converges.
Gibbs Factor T.
Conj. Gibbs Factor T.
Gibbs Factor T.
Con Gibbs Factor P.
(0, 1)
.f (0,1)
.f (0. 1
y 1)
Pi
Pi
P2
demslty
density
densitt
femsity
10
10
0
4
-5
5
0
(a)
(b)
(d)
Figure 2: Comparison between the Gibbs factor B. and the conjugate Gibbs factor F. for two mixtures of ID Gaussian distributions, Pi and p2. The density plots overlay the (conjugate) Gibbs factor with p and a standard Gaussian A (0, 1) for reference. Gibbs factors spread inversely p ((a), (c)) while conjugate Gibbs factors show more suitable alignment ((b), (d)).
5

---

Univariate distributions In the ID case, the OT map between probability density functions j and has a closed-form expression given by Peyré al. (2019):
VB v) CT C₂
where C₁ R - (0, 1] the cumulative distribution function associated to μt, defined as: (x(x)) = [ du. The quantile function Cj [0, 1] [of- the pseudoinverse (r) := min
RU{ ( : (x(x)) 2 r). For given distribution p and an initial distribution Bu. the analytical form of the Brenier map enables the execution of the algorithm in (IO). To compare the conjugate moment measure factorization with the moment measure factorization in 1D, we also propose a similar 1D algorithm for estimating moment measure potential of distribution p, described in Appendix B.1 as no method currently exists for solving the moment measure factorization. The code for both algorithms is also provided in Appendix B.]
Higher dimensional case In higher dimensions, where the OT map is not available in closed form, we estimate using ICNNs with the architecture used in Vesseron and Cuturi] [024] and the neural OT solver proposed in Amos [003). We found that performing single optimization step to estimate the OT map at each iteration was sufficient. As a result, our final methodology relies a single ICNN, wg, which serves two purposes at each iteration: generating samples from Bw, and being optimized by single step of the neural OT solver to approximate the OT map between and Bwn- Algorithm details the steps of the procedure, where z(y.) is the estimation of Vug (v,), computed as described in Amos [2023]. The LMC sampling step in step 4 of Algorithm D requires selecting the step size hyperparameter т, which depends wa- Since wi evolves during the algorithm, the step size must be dynamically adjusted. Fol- lowing Proposition in Dalalyan (2016), we set T Me where Ma is the largest eigen- value of the Hessian of wa, estimated from the current minibatch (s1, Ta. To ac- celerate convergence, the LMC algorithm is initialized using particles from the previous iteration.
Algorithm CMFGen algorithm
1: Initialize we such that wp = -/1l-11
2: while not converged do
3: Draw n i.i.d samples I; ~
4: Draw M1...... Yn Bwo using LMC
5: La + 1/ -i=1 Wo(x;) - +/2 Ei wo (i(:))
6: Update wy with VLg
7:
end while
4.3 Learning the Conjugate Moment Factorization from an Energy: CMFMA
We now consider the case where the density of p is known only up to normalizing constant, which is the typical setting in sampling-baas based frameworks. Specifically, we assume access to an energy function & such that & e To learn conjugate moment potential of p. we propose to leverage the Monge-Ampère formulation defined in (9), and parameterize the potential using an ICNN we trained via regression the following objective:
IF Exar [1]((x) wo (V)u()))) + In(ett Han (x))
In this objective, P can be taken the uniform distribution the sampling space when the dimension is small and the domain is bounded. In higher-dimensional settings, IP can instead be chosen as p. with samples obtained via the Langevin Monte Carlo (LMC) algorithm. At the end of training, the learned potential wp satisfies wo(Vwn (x)) + In(det. H (x)) ss E(x) and the pushforward Vug provides a close approximation of the target p. $4.1 details how to sample from V VugdF we using only the learned potential wy. Note that, in contrast, learning the
moment potential via regression seems particularly challenging due to the presence of the Legendre transform u² in multiple terms of the Monge-Amper equation (8)
5 Experiments
We begin with preliminary experiments using the CMFGen and CMFMA algorithms introduced in Section 4.2 and 4.3 starting with univariate distributions p for which the Monge map is available in closed form. We then estimate the conjugate moment potential for several 2D distributions, either from samples (using CMFGen) or from an energy function (using CMFMA). Finally, we demonstrate the applicability of CMFGen to higher-dimensional datasets such as MNIST [eCun et al., 2010 and

---

the Cartoon dataset from Royer et al. (2018). At the moment, CMFGen is not comparable to state-of- the-art generative models such as flow matching [Lipman et al. 2023 Albergo and Canden-Eijndes 2023). and we believe this is primarily due to CMFGin's ss reliance on ICNNs, which, while useful to parameterize convex functions, are known to be challenging to train [Korotin et al. 2021]. For this reason, we restrict our comparisons to generative ICNN trained to transport the Gaussian AO, 7) to the distribution of data p. We use the Sinkhorn divergence as a metric for the 2D experiments, and provide images generated by both our approach CMFGen and generative ICNNs.
5.1 Univariate distributions
We compute the conjugate moment potential for two univariate Gaussian mixtures, P1 and shown in Figure 2 using both CMFGen and CMFMA. CMFGen uses i.i.d. samples from the mixture, while CMFMA leverages the log-density. The distribution #1 is sharply concentrated around zero, whereas P2 exhibits heavier tails. For comparison, we also compute the standard moment potentials using the fixed- fixed-point method described in Appendix [.I]] The pushforward densities Vw" w and Vutfu, (Figures II and () closely match the target distributions, confirming that the three algorithms successfully recover the (conjugate) moment potentials for /1 and P2- The recovered conjugate potentials are shown in Figures 9 and 10 As illustrated in Figure (a), the Gibbs factor associated with P1 has heavier tails than the standard Gaussian, consistent with the theoretical insight that concentrated distributions yield broader Gibbs measures. Conversely, the broader P2 induces more concentrated Gibbs factor (panel (c)). In contrast, the conjugate Gibbs factors (panels (b) and (d)) more closely match the target distributions.
5.2 2D Experiments
We consider several 2D distributions defined either through samples-(a) Circles, (b) S-curve, (c) Checkerboard, (d) Scaled-Rotated S-curve, and (e) Dlag-Checkerboard (Figure - through known energy functions &,, Eg, and &g (Figures 12 Our first step is estimate a conjugate moment potential for these distributions using either the CMFGen or the CMFMA algorithm. Following this, we generate new samples based on the learned conjugate potential we using the methodology detailed in | The potential we is implemented as an ICNN with five hidden layers of size 128
and quadratic input connections, based on the architecture of Vesseron and Cuturi [2024). Detailed hyperparameters for both 2D and high-dimensional experiments are provided in Appendix
&
wa w (wa) Hg)
W
F
A
2.5
5
00
0
25
5.
4
-5.0
2.5 0.0
2.5
5.0
0.0
25
0.0
255
5.0
3.0
-2.5
0.0
10
25
0.0
2.5
3.0
0.0
0
62
Wo (wo) In((etHgg)
Vwin
10
550 00
0.0
25
80
010
2.5
2b0
25
5D
5.0
25 SU
00 255
SU
Figure 3: Learning the conjugate moment potential from energy. E, and Eg are learned by regression with CMFMA. The second column shows the learned energy; the third displays the corresponding conjugate moment potential; the fourth shows samples (in red) drawn from Vug wn.
7

---

Mcasure p
12
S
84
S
60
93
00
00
potential 188
15
-15
S
8
48
15
0.8
Fwy # 93
-15
-04
86
-15
02
18
-38
38
68
-15
00
18
-16
18
(a)
(b)
(c)
(d)
(e)
Figure 4: Samples from p (top), level sets of we (middle) and samples from Vug i we (bottom).
The energy functions &,. Eg, and &a are standard 2D benchmarks for evaluating optimization algorithms; their analytical forms are provided in Appendix C.2 For all three, the ICNNs wy successfully learn the corresponding energy landscape and permit draw new samples from the target distributions, as demonstrated in Figure aand 12
CMFMA.
CMFGen. Our method accurately estimates the conjugate moment potentials, the associated measures Vwğ + F closely align with the target distributions in Figure The second row further illustrates that the conjugate Gibbs factor follows the shape of the distribution p each case. For comparison, we train an ICNN to map Gaussian directly to the target distribution using the solver of[Amos [023]. The boxplots in Figure [3] which show the Sinkhorn divergence [Randas al.) 2017] between generated and target data, demonstrate that CMFGen consistently produces samples of equal or superior quality. Note that CMFGen introduces no additional hyperparameters compared to the generative ICNN, aside from the number of L.C steps used when sampling from B.n.
5.3 High-dimensional experiments.
Image generation. We evaluate CMFGen MNIST [LeCun al. 2010 and the Cartoon
dataset [Royer et al. 2018). As illustrated in Figures 5 and 6 (see also 14 for additional gen- erated cartoons), CMFGen successfully generates visually coherent images of digits and cartoon faces. Interestingly. the noise sampled from the log-concave distribution T already exhibits fea- tures of the generated distribution: digits and faces emerge directly in the noise. To the best of our
&
5
1
6
3
X
7
6
0
2
6
2
3
9
9
2
88
91
7
8
4
7 9 9
3
2
5
+
2
7
3
4
4
6
8
5
3
6
2
8
3
X
2
2
3
3
2
8
9
9
44
5
4
7
8
2
4
0
8
9
7
3
4
Figure 5: MNIST Generation using CMFGen. Samples from the Gibbs noise distribution Bas
(left); digits generated from TughT (middle); and digits generated by an ICNN trained to dircctly transport Gaussian noise to MNIST (right).
8

---

Gibbs P w. obtained from Langevin dynamics
R2
Gibbs Q ws obtained from Vweip
Conjugate (w) = arg sup y) m((x)
Transportation map Vwg (Vuz))
Generated images Vugi 9 we
Data distribution p
Figure 6: Cartoon Generation using CMFGen. The conjugate potential wy is parameterized as an ICNN following Vesseron and Cuturi 2022). with five hidden layers of size 512 and four quadratic input connections, each with two additional layers of size 512. After training with CMFGen, the generative map Vwe transforms structured data p into the log-concave distribution Ban. To sample from we first draw a sample from F using the LMC algorithm, and then apply conjugate solver to iteratively invert the map Vwg. The strict convexity of wg ensures both (i) invertibility of Vwy. and (ii) correctness of Langevin dynamics for sampling from Bwo.
knowledge, this is the first instance where the MNIST distribution has been successfully generated using an ICNN. For comparison, Figures [5] and [5 show samples generated by ICNNs trained to transport a Gaussian distribution to the MNIST and Cartoon data. For both datasets, CMFGen generates samples of higher quality.
Image reconstruction. Similar to normalizing flows Rezende and Mohamed 2015 we have access to the (unnormalized) probability density of the distribution VwgffW,, generated by CM- FGen and CMFMA: that density is proportional (det Hw, (z)) (see (3.3). After training wa, this enables downstream tasks such as image in- painting. To evaluate this, we mask half of the pixels in test samples from MNIST and the Cartoon
e S.,, (x) where Ewp (x) wo( (wy()))
Figure 7: Image inpainting task on Cartoon. The learned uh trained on the Cartoon dataset is used for post-processing task to recover the masked pixels. Gradient ascent is performed on the masked pixels to maximize the loo-brobyyylity of the full image. Top: Masked images; Bottom: Reconstructed images.

---

dataset, and perform gradient ascent on masked pixels to maximize the log-pro probbabbiyy of the full
image. As seen in Figures 16 and 17
our method effectively reconstructs missing regions.
Conclusion
We borrowed inspiration from Cordero-Erauspeee and Klartag [2055] to define new representation for probability measures. For a given p. we prove the existence of a convex potential w such that Vw" & p. We show that this representation is better suited generative modeling, because the Gibbs factor F follows more closely the original measure p. in contrast to Cordero-Erausquue p. which results in Gibbs factor Pa whose spread is inversely
and Klartag's s approach, Vut proportional to that of p. Our conjugate measure factorization uses w to sample noises (using LMC on log-concave distribution) and transforms these codes in final step, using Vw*. We propose to parameterize the conjugate potential w as an ICNN wa. and estimate it using the OT toolbox in two settings: when the target distribution accessible via samples, and when it known up to normalizing constant. We validate both approaches on generative modeling tasks. In the future, we wish to explore the suitability of replacing (0, with our pre-trained Gibbs factor Fy, in generative modeling pipelines, and retrain maps on top of it. Interestingly. one can draw parallels between CMFGen and flow matching in the sense that CMFGen does simple noisy gradient flow to generate codes (the LMC algorithm), concluded by one step generation step Vwz. which can compared the iterated application of time-varying (non-conservative) velocity field generate data.
Acknowledgements
This work was performed using HPC resources from GENCI-IDRIS (Grant 2023-103245). This work was partially supported by Hi! Paris through the PhD funding of Nina Vesseron. The authors would like to thank Nicolas Papadakis for his valuable advice on leveraging the conjugate moment measure factorization for image restoration.
References
Michael S Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In 1 1th International Conference on Learning Representations, ICLR 2023, 2023.
Brandon Amos. On amortizing convex conjugates for optimal transport. In The Eleventh International Conference on Learning Representations, 2023.
Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. International Conference Machine Learning, pages 146-155 PMLR, 2017.
Shiri Artstein-Avidan, Bo' az Klartag, and Vitali Milman. The santaló point of function, and functional form of the santaló inequality. Mathematika, 51(1-2):33-48, 2004.
Ball. Isometric problems in Ep and sections of convex sets. Thesis (Ph.D.), University of Cambridge, 1986.
Yann Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Commu- nications on pure and applied mathematics, 44(4):375-417, 1991.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Xiang Cheng and Peter Bartlett. Convergence of langevin meme in kl-divergence. In Algorithmic Learning Theory, pages 186-211. PMLR, 2018.
Dario Cordero-Erausquin and Bo' az Klartag. Moment measures. Journal of Functional Analysis, 268(12):383--866, 2015.
Arnak Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities, 2016. URL https::tt/s/tect http:://aaavvv.o/bbbbeaa /a1bo.///11.... 7392
10

---

Arnak S Dalalyan and Avetik Karagulyan. User-friendly guarantees for the langevin monte carlo with inaccurate gradient. Stochastic Processes and their Applications, 129(12):5278-53 2019,
John Danskin. The theory max-min, with applications. SIAM Journal on Applied Mathematics, 14(4):641-66, 1966.
Matthias Gelbrich. On formula for the 12 wasserstein metric between measures euclidean and hilbert spaces. Mathematische Nachrichten, 14 147(1):185-203, 1990. doi: https///doi.org/10.1000 mana. 1990147111. URL http:p::s///olllllllbbrrry wwley..ww....................e.. wttple...coo/oooa///0ooo.. 19901470121
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato. R. Hadsell. M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates. iewrrrre neurips wttpccpppppeeeeeeeeeeeee filefilllppplee paapperfileeeppper//OO2222 /iles///aperr222//Eiiel Inc., 2020. URL wat.aa1110111111111111 0att033a0000077111100000000aaba2 #ccbcfecc584af (ww.66544a0d9667168
pdf
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Alexander Korotin, Vage Egiazarian, Arip Asadulaev, Alexander Safin, and Evgeny Burnaev. Wasserstein-2 generative networks. In International Conference on Learning Representations, 2020.
Alexander Korotin, Lingxiao Li, Aude Genevay, Justin M Solomon, Alexander Filippov, and Evgeny Burnaev. Do neural optimal transport solvers work? continuous wasserstein-2 benchmark. Advances in neural information processing systems, 1114453-116000 2021.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://annn.l wwt.yaannlecuu.c.m www.n. cow.exdb/n/isstttt..p...mc///a 2, 2010.
Dong Bok Lee, Dongchan Min, Seanie Lee, and Sung Ju Hwang. Meta-GMVAE: Mixture of gaussian
VAE for unsupervised meta-learning. In International Conference on Learning Representations, 2021. URL htttps: wtt.:///oorrurttdsssffi/..r
Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Gmmseg: Gaussian mixture based generative semantic segmentation models. Advances in Neural Information Processing Systems, 35:31366 31375, 2022.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Dong C Liu and Jorge Nocedal. On the limited memory bigs method for large scale optimization. Mathematical programming, 45(1-3):503-528, 1989.
Ashok Makkuva, Amirhossein Taghvaei, Sewoong Oh, and Jason Lee. Optimal transport mapping via input convex neural networks. In International Conference on Machine Learning, pages PMLR, 2020.
Gaspard Monge. Mémoire sur théorie des déblais et des remblais. Histoire de I' l'Académie Royale des Sciences, pages 666-704, 1781.

---

Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ Machine Learning. (166)))355-607,, 2019.
Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, and Ricky TQ Chen. Multisample flow matching: Straightening flows with minibatch couplings. arXiv preprint arXiv:2304. 14772, 2023.
Aaditya Ramdas, Nicolás Gareía Trillos, and Marco Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47, 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows.
In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volure 37, pages 1530-1538, 2015.
Gareth Roberts and Richard Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli, 2(4):341 - 363,, 1996.
Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, and Kevin Murphy, Xgan: Unsupervised image-to-ing -mage translation for many-to-many mappings, 2018. https://: //axiiv org/ wwtp///s.....///.. 05139
Filippo Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 55(58-63):94, 2015.
Filippo Santambrogio. Dealing with moment measures via entropy and optimal transport. Jour- nal of Functional Analysis, 271(2):488 436, 2016. ISSN 0022-1236. doi: https://doi.org/10 20.1...... URL https: /wwp///wwww..../ www.wceennedrrrct.ooom wtt.//cciinc//arrilleeeoooo
$0022123616305007
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Jakub Tomezak and Max Welling. Vae with a vampprior. In International conference on artificial intelligence and statistics, pages 1214-1223. PMLR, 2018.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brook Kilian Fatras, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. arXiv preprint arXiv:2302.00445 2023.
Nina Vesseron and Marco Cuturi. On neural implementation of brenier's 's polar factorization. In International Conference Machine Learning, pages 49434-49454, PMLR, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction claim that we introduce new factorization of probability measures-the the conjugate moment measure factorization-ann propose an algorithm that leverages this factorization for generative modeling tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of approach discussed in the paper. Theorem clearly states that our proposed factorization is valid only for absolutely continuous probability mea- sures supported compact convex sets. Additionally, in the introduction, we acknowledge that our experiments are restricted to relatively simple generative modeling tasks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
13

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For our main theorem and the two propositions, all assumptions are stated explicitly the time of their formulation. The proof of the main theorem is provided in Appendix and Appendix [..2] The proof of Proposition is given directly following its statement, while the proof of Proposition is provided in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The CMFGen and CMFMA algorithms are detailed in Sections 4.2 and 4.3
respectively. The data generation process using our methodology is described in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
14

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Because the main innovation is methodological, we plan to release the code gradually. The implementation made of OSS bricks available elsewhere.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ctipsscc/ppullc//uuii
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameters used in our experiments, with the procedures used to select them, are detailed in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We present boxplots in Figure 13 to compare our method with a baseline using the Sinkhorn divergence.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details on the computational resources used for our experiments are provided in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted in the paper fully conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
16

---

We do not envision immediate societal impacts beyond those commonly associated with the proposal of new methods to understand and train generative models. Guidelines:
Justification:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This question is not relevant to our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The authors produced the code for the experiments. The real-world datasets used to illustrate our method are explicitly referenced: the MNIST dataset [LCunn et al. 2010) and the Cartoon dataset from Royer al. (2018).
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
17

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: All details regarding our algorithms are provided in the paper. Additionally, the code will be made publicly available upon completion of the double-blind review process.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
18

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http::////eepppp.ooooooooorrrrrr /2tp://pp//.cc/ooooeeeeee/////// for what should or should be described.
19

---

A Appendix/ Supplementary Material
A.1 Sketch of Proof for Theorem
Proof Sketch, The complete proof, along with the theorems used in the proof, can be found in Appendix A.2 We rely on the identity Vw Vw id that holds for strictly convex functions w and show the existence of strictly convex potential w that verifies i Bw- To proceed, we first introduce some notations. Let ((IR)) be the set of functions mapping I to R U [+00). Given negligibee set Q Rd and continuous function v - R such that fga e e="(=) is finite, we denote by P the probability measure with density
e (()
We then define, for probability p supported on non-negligible set Set the map Cp L(RR) L(R)) which to a potential associates the Brenier potential, defined in (4) from p to ?
G (v) == B(p, (p)
The fixed points of Gp are precisely the convex functions such that Votp = F. from which we can construct w defined as w(x) v(x) for x E Ωt and w(x) = +oo elsewhere. This function w is convex and verifies Vw p = Pu (since p is supported on Ω2. We rely on Schauder's fixed point theorem to show that Go admits a fixed-point Given a Banach space (X, II-II) and a compact, convex and nonempty set . C X, Schauder's theorem states that any continuous operator A .M + M has at least one fixed point. In this proof, we consider the Banach space of continuous functions over compact set Ω2 denoted 8 (52), equipped with the supremum norm, and establish the following theorem that directly implies Theorem
Lemma 1. Let be an absolutely continuous probability measure supported on compact, conver set Ω C Rd. Then GB admits a fixed point in
M = (S (S.) such that Vx, y E Ω I5(x) f(w)] < R| 2112 and f(00e) = 0} where R is the radius of an euclidean ball that contains Ω ie.. P B(0, R) {a E R. |=11a S R).
We first show that the set M is non-empty, compact, convex set of X (66)), || Ilox) by relying Arzela-Ascoli's 's theorem for the compactness. We then use Brenier's theorem that, given the absolute continuity of p. garantees the existence of the Brenier potential B(p, P?) and the fact that G is
therefore well-defined on the set M C E(2). Moreover, the gradient of the obtained potential (w) transports / on F8 which is compactly supported on Ω C B(0, R). For this reason, VG (v) is bounded by Rt Q and the obtained potential is Lipschitz continuous with constant R. This permits to show that GR C M. To prove the continuity of G on M, one can remark that GØ = Hp o YO, with (v) = F. and H₂(μ) = B(p, 1)), both viewed as functions from E(2) to ((.). To show that 1² is continuous, we use the definition of continuity in X (E()), lloo) while we rely on Theorem 1.52 from Santambrogio (2015) to prove that H, is continuous. We conclude by applying Schauder's ' theorem.
A.2 Proof Lemma 1
Let p an absolutely continuous probability measure supported on compact, convex set Ω C Rdd Ω being compact, it is bounded and there exists a real number R such that Ω be included in the ball B(0, R). We begin by recalling Schauder's fixed point theorem, which is central to our proof for showing that Ch. defined as:
(0))::Bl 2??)
where T the probability measure with density
e v(x) 5-tx Loen
_ee r())
admits fixed-pore point.
20

---

Theorem (Schauder's ' fixed point Theorem). Let (X, II-II) be a Banach space and M C X is compact, convex, and nonempry. Any continuous operator M - -M has least one fixed point.
We consider the Banach space X = =(6(2), |l (lx)) to study the continuity of Gg on the set
M ( E (92) such that Væ, y E $2, [f f(w)l - R|i vll2 and f00gs = 0}.
We prove the two following lemmas in order to use Schauder's theorem.
Lemma 2. The set M is non-empty, compact, convex of X. Lemma 3. Gp is well defined and continuous on M and G% C.M
By applying these two lemmas along with Schauder's 's theorem, we conclude that Gp has a fixed point Vop in M C (2)). Moreover, the absolute continuity of Fe ensures that the optimal solution is strictly convex on the support of p. Ω. Additionally, the Popt whose gradient transports p onto cupe
gradient of its Legendre transform Vuggi is the OT map from T to p.
Vags
A.2.1 Proof of Lemma 2
M is non-empty 0 is included in MM which is therefore not empty.
M is convex Let f and g functions of .M and À [0. 1]. We have that Xf + (1- (1-1)g- ((2) and that Af(() + (1- (1-))g(0) = 0. Let z E Ω and Ω2 then
((A((>) + (1 -))()))))))))))))))))))))))))- (AS(v) + \)g(1\) S (1f(x))
+ (1- g(v)
I XR 2 - vll2 + +1 -) داا = R|| - دlla
where, for the first inequality, we rely the fact that the absolute value verifies the triangular inequality and for the second inequality, we use that ʃ and g belong to M and are therefore R-
lipschtiz.
M is compact We rely on Arzela-Ascol theorem which recalled below to show that the set M is compact. We also recall the definitions of equicontinuity and equiboundness.
Definition. A family of functions C (S2) is equibounded if there exists a constant Co such that for all x E Ω and f E F, we have (x() S Co-
Definition. A family of functions C 66 (9t is equicontinous for all > O, there exists such that for x. S2
-E= - yll < - Ef [f()) <E
Theorem (Arzela-Ascolij) Let (K, d) a compact space, a family of functions F C ((( is relatively compact if and only if F is equibounded and equicontinous.
The family of functions M is equibounded. In fact, for f E M and Ω, we have that:
[f(z) = 1f(z) - f(0) S R|| Oli2 S R²
The first equality comes from the fact that f E M, therefore f(0 = O, the first inequality uses the fact that is Lipschitz continuous with constant R while the last one is due to the fact that x E C B(0, R).
The family of functions M is equicontinuous. For > 0, we define 5 = &. Let ʃ E M and and a, E Ω such that | - y|I2 < å then because f is Lipschitz on Ω we have that
[f(x) f(v)] S R|| vlla < R8 =-
Then, according to Arzela-Ascoli theorem, M is relatively compact i.e. its closure is compact.
M is closed in X (8)) oloo) Let (fx) be sequence of functions in M that converges uniformly to a limiting function f. We have that f E ((()) which is closed. Moreover, because (Jn) converges uniformly to it also converges pointwise and in particular (o) 0 = f(0). Finally, because Vn, Jn is Lipschitz continuous, we have
(x) fn(v)) < R| 2ll2 Vn
21

---

By taking the limit when n + too we get that:
(f(x) f(y)] S R||ar y/1
and f is Lipschitz continuous with constant and f E M.. This proves that M is closed and that M is therefore compact.
A.2.2 Proof of Lemma
Gp is well-defined on M For a given v E M, the fact that v E (S2)) and that Q2 is non-negligible (because is an absolutely continuous probability measure supported on !?) ensure the measure D be well-defined. Using Brenier's theorem, the absolute continuity of p garantees the existence of the Brenier potential pp,, F..) and the fact that Gp is therefore well-defined on the set M c (S2).
(MM) c M For given M. the Brenier potential Gz(vv) verifies ()) 0 by definition. As convex function which takes real values on 52 Gp (v) is continuous on 52 and (e) E (22). Moreover, the gradient of the obtained potential VG (v) transports p on P The probability measures T and are both compactly supported on Ω C BO, Rt) which implies (v)) that the support of the optimal transport plan 7 = (id, (V) # is included in Ω Q. P being
(p(()
the support of p. we have that for each x E Q (v)(a) E Ω C B(0, R) and in particular [VVg (u)))ll I R. Ω being convex and GP (v) being differentiable in the interior of Q, we can apply the mean value theorem to deduce that:
Yx, y E Ω C)
vll2 S R|| yl12 (v)(())) S sup [V)G
=E
(e) is therefore Lipschitz continuous with constant R and it belongs to the set M.
Gg is continuous on M with
To prove the continuity of G on M, one can remark that C Hp F
pn. M - N
and
Hp: N - (62))
P B(p,H) p1)
with V (90. {()) 6(2). We show that both function F and Hp are continuous
in X = (2), Ilso). Let v E '()) and (e,) familly of functions of ((() that converges uniformly to e. We will show that the family (F"().) converges uniformly to F (v). Let x in Ω and n E N, we have:
(v.(x)
-v((2)
-F)))())=(()((0)))
SA e-on(()d
Sn e-
(e-va()) In E ((()))) (E-v(() Sg (2)=)(())dz)
Jo e u Jn e- -="(=)dz
We will use the following lemma:
Lemma 4. If (wn) converges uniformly in (92) with Ω compact then th converges uniformly to e in 6 (22).
Proof of Lemma Let us denote by f the real function f - c= is continuous on R. Because V E (32) and Ω compact, we have, by Heine's ' theorem, that v( C R is compact and therefore bounded. There exists and such that vS2) C [a, b). Moreover because (v,) converges uniformly to there exists N such that n 2 N = un ( C la 1, + 1]. being continuous on R. it is uniformly continuous on the interval that (v()) converges uniformly to fe)
1, + according to Heine's theorem. We deduce
22

---

On the other hand, the function g g:t - e e- is continuous on the compact ! as composition of continuous functions. By Heine's theorem, the image of Ω by 9 is a compact and there exists f E IR+* such that g(S2) C (c, f with
0. Because e converges uniformly to e in 8 ( there exists No such that n 2 No = e (x) z 2 Vz E Ω. As consequence, we have for 2 No:
( c= J2
/ Lalz
odz = e²
2 |
e
l2
with [S2] the lebesgue measure of the compact Ω. Ω being non-negligible, we have ||] > 0. Let Co Tp we have shown that for No had:
<o
Ja e mz) So e e(2
Furthermore, one notes that:
(-pa(() / c --((tddz)
(-v(())
/ emmlltldz)
vn()) / c- (-u(s)dz) -
/
(onre0) +
/ (-o(s)dz) (e-ute) / e e-muleldz)
d2 (-v(s)
8 vm - E -0666)
o(a) dz
3 (agn (al (18 e (u(()) - e e-n((t)dz)
+
52
$ 1/ (m) -0(aa) |2/l +
6
12
12
$ |le Un e 1
1/22 +
c -v()) 1 - d2
221si//e - 1 -
Until now, we have shown that for 2 No we had, for x E S::
(vw)(t)) - (v)( Co241111) -
Let > let us define Eo T-20 O. Because ve converges uniformly e" there exists Ni such that 2 N1 = |ee -Pn - - Eo. Let us denote N max(No. N1), we have that for n 2 N,
(vN)(z)) F())
Because this is true for any Q we have [(.) - F (v) S We just proved that FO is continuous at v for any E '6( ie lo is continuous on (S?) and in particular it is continuous on M E(2).
We will now show that H, is continuous on N. Let μ E N and let (Pa) family densities from N that converges uniformly μ. For E N, denote by P(μn) and P(μ) the probability measures associated to the respective densities 14, and μ. Because (pa) converges uniformly to μ. we have that ((,,)) converges in distribution to P(u). Then according to Santambrogio 2015 (Theorem 1.52), the family of Kantorovich potentials from to P(pn) with value 0 in 0 that we denote (f²) converges uniformly the Kantorovich potential f from p P(μ) whose value is 0 in 0. Therefore, the family of Brenier potentials (H,()n) f - [[11] - ft) converges uniformly to
(p) I11 1|2 fr. We conclude that H, is continuous on N and by composition of continuous functions Go is continuous on M.
23

---

A.3 Additional Proof for Proposition]
Let us observe that p the moment measure of if and only if Vu is the Monge map from F. to p. One can then consider quadratic functions (r,((z) ( r) A A-1(x r) whose Gibbs
distribution is F. = [r, A) and use the expression [Gelbrich] 1990 of the OT map between two Gaussian distributions (mm,, Ea) and EB)
T Ammg Fg x mn M(x ma)
where M M= - (xx En 1/2 1 1/2 to solve the equation
Vu, = (.) Vo E Rd
One get:
A (x (r) M(x - with M = A 1/2 "A
which leads to A Σ- It means that the functions in (u, 14.11.111 r. R fixed-poing point of G, with p = (0, E). According to Cordero-Erausquin and Klartag 2013), the moment potential unique up to translation, which implies that there are other convex functions whose moment measure is p.
By identification, we obtain that A M A 1/2 -
A.4 Proof of Proposition 2
The proof is similar to the additional proof of Proposition D provided Section A.3 Given
(m,, Σ), we are looking for such that V # p and F (r, A). This means that
w(x) = 2 () r' (z r) and w" (z) x t r. Moreover Vw" x - Ar +- must be the OT map between F w = (r, A) and p = (m,, E). The OT map between these two gaussians known in closed form and is given by I m+ M (x r) with M A 1/2 /A//2/////////// 1/2
Solving A A 1/ /2) // 1/2 gives A / while solving = m Mr gives (La /3)3 -'mmm-mmm--- m. We conelude that p the conjugate moment measure of the unique potential w among those whose Gibbs distribution is Gaussian.
B Experiments in the 1D case where p is known from samples
B.1 Proposed algorithm for moment measures
For probability measure p E P( we define the map Jp L(Rd) C(RR) which to potential associates the |renierr potential from F. to
Ju(u) = B)
The fixed point of Jp correspond exactly to the moment potentials of This observation motivates the following fixed-point iteration scheme compute moment potential of p:
"o = 11
VI 2
# := (ue):
(11)
In the ID case, the OT map between density functions μ and is known in closed form (see $4.2) which enables the exact application of the fixed-poin algorithm.
B.2 Details the 1D experiments
We conducted two experiments, with defined as a mixture of Gaussian distributions. In the first experiment, / is mixture of four Gaussians: (-10.1 0.07). 0. 0.1). 0.1). 0.15) with mixture weights $, 3 2 respectively. In the second experiment, is a mixture of the three 7: Gaussians (--0.8, (1.55, 0.6), ./(3,, 0.5), with weights 1 tures, we used 400,000 samples and estimated their histograms using 100,000 bins. This allowed us to compute the cumulative distribution and quantile functions using numpy's 's cumsum and quantile functions, which were essential for estimating the OT map at each iteration of the fixed-point algorithm. The algorithm converged after 300 iterations.
2: 3 6. To approximate these mix-

---

B.3
Code for the ID experiments
Code for estimating the moment potential in ID
Listing 1: Code for estimating the moment potential in ID
@jax.
def compute_oo wtt.oootttmommmaaee_e source,
source, fruqes samples_target)
cdf jup. cumsum (treq axis-0) cdf cdf / cdf [-1]
quantile_fr fn jax vmap(1ambda jup- quantile (samples x)) inverse_cdf cdf quantile, (cdff)
return (positions_source, inverse_cdf, cdf, freq_source)
def compute_next. (p
freq_source, samples,target).
positions_gouree source, inverse_cdi _df, source
freq_source, samples_target)
integrate.cumtrapz (inverse_ positions_soure _surce, initial=0) integrate cumtrapz
weights jup. array (scipy special. softmax(-u)) jup.sum ww..sumffeggsurree return (positions_ source, weights), (u, inverse_cdf)
# hyperparameters nb_ bins 100000
bound_min -11
bound _ax 11
nb_ ab_points 400000
nb_ ab_bins_plos = 1000
# generate source and target distributions rng jax. Bandom.PPNGKee (0)
rog_source, rag_target, rng = jax random. (pniom,, 3)
mu_ target jup.array ar-ay([-O.66 -0.1, 0.3, 0.7]) 0.8
parget = jmp. array 150.15, 15, 0.07, 0.1, 0.15) 0.8
targe_target jup arr.a
sampler, target multivariatt__aussiian
taau-mu_target. aagma=sigga_targee alpha=
(ipha_target)
aamples_target sampler target. www.ttaeeeggggraaee...mmm wwweeateeeeaplleeeeeeeeeeeeeeeee target, nb nb_points)
samples_targer = samples_targe jnp. mean ptamples_targets
samples_source = jax. random. normal (rng_sources
construct bins histogram
freq_source, edges_source jnp www.hhooraaccmmpessssorrrc nb_bing, Lange=(bound min, bound_max))
positions_source = (edges_source [: + cdles_soource 2.0
treget, edges_targes jup. www.hissttgammmcm (samples_targeel nb_ ab_bins, range= (bound min, bound_max)
positions_target = (edges_ target[:-1] edges_target(ic 2.0
for in range
sesurces_ssrsee,,,ee_eeee (positions_ source, freq_source), (u, grad_u) _u compute_next new.sneasuteeeeeeeeeeeeeeeee ceritions_souree, source, camples_target)
Code for estimating the conjugate moment potential in 1D: CMFGen
25

---

Listing 2: Code for estimating the conjugate moment potential in ID
@jax. jit
def compute_ot_ _tt___ott____arr____eee cestarrpppoiiosstarraeeee __arget, farget, samples_source):
cdf jup cumuum(fre__targee axis=0) cdf edf /dff/cd[[- [-1]
quantile_ff jax. vmap (1abda x: jup. wwt..quanntlleeee (smpleessssampeessssrrrro x)) inverse_cdf quantile_fn(cdff
return (positions_targee target, cdff treq_target)
def caapute__eet__meauueeppppppooeee (positioonoss _arget, freq_target, samples_
positions_targer inverse_cdf, freq_target = compute_ot_map_
(ttpppsstttoosiiiosstttaree (positions_target
freq_ samples_souree
integrate cuntrapz
positions, target, initial-0)
weights (sciy return (positions_tarre target, weights), (u, inverse_cdf)
(scipy. seseippyspecaa..... www.cal..softaa....c.c.......... softmax(-u)
(free.
hyperparameters
ab_bins 100000
bound_min -4 bound_max nb_ nb_points 400000
ab_bins_plot 1000
# generate source and target distributions rng jax. www..ranw....nee...aeeeaaa.
rng sourse rng_target, rng jax. sandoom.sllitrrp.. 3)
target jup. ar-.y -0.1, 0.3, 0.7]) 0.8
target jnp. 1[0.15, 0.07, 0.1, 0.15) 0.8
alpha_target array ar (1,,,,,,))
sampler_targe target
target, targget alpha= caatttteggussiiaa___ee
taphatt
samples_targe sampler_target generate generate_samples (rng target, nb ub_points)
samples_target ptrgles_target jnp. mean (samples_target)
samples_soure www.daando.oooo (rng. source, shape= (nbpe=(nb_pae (ablsinnnpppioee (ub__ponnts,))
# construct bins and histogram
freq_source, edges_source = jup. histogran(samples_ooo nb_bit bbns,n,, Lange=(bouud_nng mmin, bound_max)) positions_source sodges_soure([--11 edges_source (1:]) 2.0
treq_source_plos plott, edges_ logesssourre_jg _plt = jnp. histogram(samples _ource, nb_ nb_bins)
positions ceattionsssourcepplee (edgea_soureee source_plost [:-1] edgea_source, pouurceppootti [1:1) / 2.0
freq_target, edges_target = jup. www.hhisssaasaaamamm (iamlgrrantammleesttareeee nb_bins, range= (bound_min
bound_max))
trattions_taree.. tedges_target.:.
2.0
for range (300):
rng., rng jax. www.oon.sno.tttee....... 2)
(positions_souut source,, freq_source), (u_star, grtarr - compute__net__eesuue positions_targee target,, treq_target, samples_source)
26

---

samples_source jax. random. choice (rag_, positions_source, shape=(nb.points, p= freq_source)
samples_ source samples_source jip.meannaamllees_sourr
B.4
Other figures
Conj. Gibbs Factor Pr
Vwgg
Gibbs Factor P.
VauP. pi
A (0,
pi
(0, 1]
Pj
Pi
density
density
densily
deasity
Conf Gibbs Factor F.,
Gibbs Factor M (0.1)
Vus?3 pa
pz
Pa
p2
densitv
deasity
density
density
0.0
2.5
Figure 8: Comparison between the Gibbs factor T. and the conjugate Gibbs factor B. for two mixtures of ID Gaussian distributions, Pi and p2. Pushforward densities Vw"FB and VutT closely match the target distributions, illustrating that the exed-point algorithms converged and succefully estimated the (conjugate) potentials.
C CMFMA experiments
C.1 1D experiments: Figures
Epy
vo(Vwo) In(det Hwg))
W
10
10
0
Figure 9: Energy associated to the mixture of gaussians Pi (left), learned energy (middle), associated potential (right).
10
Ep
wo( ve(Vwo) In(det
w
10
0
2.5
0.0
2.5
5.0
2.5
0.0
2.5
5.0
2.5
0.0
2.5
5.0
Figure 10: Energy associated to the mixture gaussians P2 (left), learned energy (middle), associated potential (right).
27

---

V Fwa
density
density
1.0
-0.5
0.0
0.5
1.0
Figure 11: The pushforward densities Vugifww closely match the target distributions P1 and p2. illustrating that CMFMA succefully estimated the (conjugate) potentials.
C.2 2D experiments
&
Wo (wg) ln(
V
5.
8.1
5.0 25 0. 2.
su-55
0.0
50 25 0.0
s.0
5.0
25
0.
25
8
we ww(Wwa) In(dtt
we
-00
5.0
25 50-50
83
We() In He)
7.5
50
50
-50
00
25
SU
25
00
30
25
00
25
5.0
0.0
50 25 0.
2.5 50
00
Figure 12: Learning the conjugate moment potential from an energy. E, &g and Eg are learned by regression with CMFSamp. The second column shows the learned energy, the third column displays the corresponding conjugate moment potential, and the fourth column presents samples (in red) drawn from VwgrBBme me-
The expressions of the three energies used for the 2D experiments with CMFMA are the following:
(x v) -6- sin =1)==² (i0.))
Ex(x, y) = (x. + y - 11)2 + (x + 8² 7)2
100
Ex 3) = =3-cos
2n
+
cos 10 y +
28

---

D CMnGen experiments
D.1 2D experiments: comparison with generative ICNNs
In this experiment, we compare our method, CMFGen, with the standard ICNN-based generative pipeline (see Amos [203]) that maps a standard gaussian noise to data. The two approaches are evaluated using the Sinkhorn divergence between generated samples and reference samples from the target distribution, computed over batches size N 2048. We repeat this evaluation across 100 random seeds applied to both generated and target distributions, and the resulting divergences are summarized by the boxplots Figure T3
a baseline, the leftmost column reports the 2-Sinkhorn divergence between two independent samples drawn from the target distribution. The parameter used to compute the Sinkhorn divergence is adapted to the scale of the data. As in Vesseron and Cuturi [024], set 0.05 2'pp'<p/2 z²12, where the expectation is estimated from two batches of 2048 samples each, drawn independently from the target distribution p.
Diag- Checkkrarcd
Checkerboard
0.014
Sinkhorn divergence
0.025
Sinkhorn divergence
0.012
0.020
0.010
0.015
0.008
0.006
0.010
0.004
0.005
0.002
2 Samples Target
ICNN
CMFGen (ours)
Samples Target
ICNN
CMFGen (ours)
Circles
S-curve
Sinkhom divergence
Sinkhorn divergence
0.08
0.07
0.06
0.004
0.05
0.04
0.003
0.03
0.002
0.02
0.01
0.001
0.00
2 Samples Target
ICNN
CMFGen (ours)
Samples Target
ICNN
CMFGen (ours)
Saale-Rotated S-curve
Sinkhorn divergence
0.05
0.04
0.03
0.02
0.01
0.00
2 Samples Target
ICNN
CMFGen (ours)
Figure 13: Comparison with generative ICNN across various datasets: Diagonal Checkerboard, Checkerboard, Circles, S-curve, and Scale-Rotated S-curve. Each boxplot shows the 2-Sinkhorn divergences between generated and target samples, computed over N 2048 samples and averaged over 100 random seeds. For each dataset, the first boxplot shows the divergence between two independent samples from the target, the second for the generative ICNN, and the third for our method, CMFGen.
29

---

D.2 Cartoon dataset: comparison with generative ICNNS
Figure 14: Additional images for the cartoon experiment using CMFGen, Top left: Images from the cartoon dataset; top right: Transported images using the map Vw; bottom left: Samples from the Gibbs distribution Buy bottom right: Samples from the distribution VugtFuw.
Figure 15: Images produced with generative ICNN on the cartoon dataset. Left: Gaussian noise; Right: Generated samples by the ICNN with the same architecture as used by CMFGen, i.c., an ICNN with five hidden layers of size 512 and four quadratic input connections.
30

---

D.3 Image Reconstruction: MNIST and Cartoon dataset
4
6
40
8
$
585
h
1 Y
22 8 0 8
9
D
&
7 8
7 N
9 0
3 3
0
In
5
8
5
B
Figure 16: The learned wo trained on the MNIST dataset is used for post-processing task to recover the masked pixels. Gradient ascent is performed on the masked pixels to maximize the log pog-pobaaiility of the full image. Left: Masked images; Right: Reconstructed images.
Figure 17: The learned wy trained on the Cartoon dataset is used for post- post-processing task to recover the masked pixels. Gradient ascent is performed on the masked pixels to maximize the log-probability of the full image. Top: Masked images; Bottom: Reconstructed images.
E Hyperparameters and infrastructure
All runs have been made on single GPU V100-32GB.
Noisy MNIST and Cartoon Datasets For the MNIST experiment, We convolve the MNIST images with Gaussian noise of standard deviation 0.3 to ensure the existence of the OT map from the MNIST distribution to the log-concave distribution B... as guaranteed when the starting distribution is absolutely continuous (Santambrogio 2015). Similarly, for the Cartoon dataset, we apply Gaussian noise with standard deviation of 0.2 to avoid significantly altering the images.
E.1 Hyperparameters for CMFGen
Our algorithm, CMFGen, introduces no additional hyperparameters compared the generative ICNN trained with the solver from Amos 2023 except for the number of LMC steps used when sampling
31

---

from Be For a fair comparison, we adopt the same ICNN architecture for both CMFGen and the generative ICNN baseline shown in Figures B and T3 We follow the standard ICNN design
from Vesseron and Cuturi (204). which is known to perform well in generative settings. All the 2D experiments in Figure were generated using common set of hyperparameters, detailed in Table [ Only the number of particles used (i.e., batch size) varies across experiments. For the 2D distributions Circles, S-curve, Scaled -oteeddd S-curve and Diag-Checkerbong Checkerboard, a larger number of particles (4096) was used to ensure the stability of the algorithm. The hyperparameters used for the MNIST and Cartoon experiments are given in Table 2 Moreover, adopt the approach of Amos 2023 where the cost of computing the conjugate each training step is amortized by initializing the conjugate solver with the predictions of an MLP. This MLP is trained via regression layers with 128 units each for the 2D experiments and three hidden layers with 256 units each for the MNIST and Cartoon experiment. It is trained using the Adam optimizer with default parameters, and a fixed learning rate of le-4 for the 2D experiments and 5e-4 for the MNIST experiment, As explained in Amos 2023). the predictions of the MLP are then refined with conjugate solver whose hyperparameters are given Table 3 and are kept consistent across all experiments with both CMFGen and the generative ICNN. It is important refers to the steps taken to sample from P starting from uniform noise, as used to generate the figures. During training, however, we reuse the particles sampled from the previous gradient step and apply 200 LMC steps to these particles to form the new batch. The number of LMC steps during training reflects trade-off between computational efficiency and performance: using significantly fewer steps (e.g., 50) results in degraded performance.
we used 1024 particles, while for
the outputs of the conjugate solver. The MLP consists of three hidden
that the number of LMC steps listed in the tables
Hy Hyperaarameter
Value
dense
Wy architecture
2 128 128 128 128 128
ELU activation functions
Adam
step size 0.0001
optimizer
31 0.5
82 0.5
Langevin sampling number of gradient steps
number of steps 10000 (=200 during training)
1,000,000
Table 1: Hyperparameters for CMFGen and the generative ICNN in 2D experiments. The only additional hyperparameter used by CMFGen, compared to the generative ICNN, denoted orange.
Hyperparameter
Value dense
We architecture
2 - 512 - 512 + 512 - 512 - 512
ELU activation functions
Adam
step size 0.0005
We optimizer
31 0.5
82 = 0.5
Langevin sampling
number of steps = 10000 (=200 during training)
number of gradient steps batch size
50,000
512
Table 2: Hyperparameters for CMFGen and the generative ICNN in the MNIST and Cartoon experiments. The only additional hyperparameter used by CMFGen, compared to the generative ICNN, is denoted in orange.
32

---

Hyperparameter
Value
Adam
step size with cosine decay schedule
Conjugate solver wg
init value = 0.1
alpha le-4
31 = 0.9
82 0.999 100 iterations
Table 3: Hyperparameters for the conjugate solver used in CMFGen and for the generative ICNN (the same hyperparameters are used all experiments).
E.2 Hyperparameters for CMFMA
The hyperparameters for our CMFMA algorithm listed in Tables 4 and 5 We follow standard ICNN architectures, except for the 1D experiments, where we observed discontinuities the learned potential when using ELU activations. This issue arises because the second derivative of ELU is not continuous, and the CMFMA regression loss involves the Hessian of the network wa. To address this, use Softplus activations with beta coefficient of 10.
Hyperparameter
Value
dense
Wy architecture
- 128 - 128 - 128 - 128 - 128
Softplus(B 10.0) activation functions
Adam
step size 0.0001
wy optimizer
81 0.5
3g = 0.5
number of gradient steps batch size
100,000
1024
Table 4: Hyperparameters for CMFMA in 1D experiments.
Hyperparameter
Value
dense
We architecture
2 - 128 - 128 + 128 128 + 128
ELU activation functions
Adam
step size 0.0001
we optimizer
31 0.5
8g 0.5
number of gradient steps batch size
500,000
1024
Table 5: Hyperparameters for CMFMA in 2D experiments.
33


Reviews:
[{"review_id": "Q3NNvEENQF", "reviewer": "Reviewer_EwEU", "summary": "This paper develops a new, mathematically-motivated method for generative modeling. The approach is based on a so-called moment measure factorization of Cordero-Erausquin and Klartag [2015] that provides a factorization of any measure $\\rho$ as $\\rho = \\nabla u \\sharp e^{-u}$ where $u$ is a convex potential function. The authors observe that this factorization can be ill-conditioned, and instead propose to leverage a novel, related factorization $\\rho = \\nabla \\omega^* \\sharp e^{-\\omega}$ where $\\omega$ is convex and $\\omega^*$ denotes its convex conjugate. The approach is demonstrated on several simple numerical examples, and some theory is proven demonstrating the validity of the approach.", "questions": "1. In the introduction, the authors comment that $\\omega^*$ is the Brenier potential linking $e^{-\\omega}$ to $\\rho$. Can the authors describe, then, why the method is different from optimal transport? Is it simply because we also learn the source distribution? We know that the optimal transport map can be hard to learn, and so many generative modeling approaches are intentionally based on sub-optimal transport to simplify the estimation problem. Can the authors explain clearly why it is a \"good idea\" to try to estimate the optimal transport map if the goal is \"only\" generative modeling?\n\n2. Can the authors explain the computational efficiency of their approach? The iterative optimal transport computation in Sec. 4.2 seems very expensive.\n\n3. Can the authors comment on the feasibility of their sampling method from Sec. 4.3? The sampling method requires Langevin samples from the target measure $\\rho$ to learn. This seems to be a chicken and an egg problem, because the method's goal is to generate samples from $\\rho$. From this perspective, I'm having trouble seeing if the method will really work in practice. Is there a way to make it iterative and sample from the current estimate of $\\rho$, using the fact that $\\omega$ is convex? Will this suffer from mode collapse?\n\n4. Can the authors get their method to work better in the numerical examples? I'm a firm believer that we shouldn't require SoTA image results in every paper on generative modeling, but I do think if the method doesn't really work on the 2d checker, it indicates a problem. For publication, I think the authors need to have compelling results on some simple examples. They should also try to quantify their results in some way, such as FID on images and metrics such as KL or Sinkhorn in low-d. The authors should also tone down the claims in the text that the method works well, which are sort of contradicted by their figures.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The paper is very well-written and is a pleasure to read.\n- The idea is novel and interesting, and leverages deep mathematical results that have not yet been applied in the generative modeling space, implying a significant level of originality.\n- The method is mathematically very well-motivated and grounded, which is refreshing, particularly in a space where engineering advancements can overpower theoretical ones.\n\n\n**Weaknesses**\n- The notation can at times be difficult to read -- for example $*$ is used both to denote the optimizer and the convex conjugate, and there are several usages of related letters such as $\\mathcal{B}$ and $\\mathfrak{B}$ for distinct concepts.\n- The method seems to be very expensive numerically, particularly given its relation to optimal transport. Moreover, the sampling variant requires the ability to sample from the target, which feels circular.\n- Most significantly, despite its elegance, the method doesn't work very well: even on the two-dimensional checker example, for which we can now obtain very good results using flow matching or diffusion models in a matter of minutes, the results are pretty poor. The results on higher-dimensional datasets like images are unfortunately quite bad, even for simple cases such as MNIST.\n- Very minor comment, but the authors miss a few of the major initial references on flow matching, namely https://arxiv.org/abs/2209.15571, https://arxiv.org/abs/2303.08797, and https://arxiv.org/abs/2209.03003", "was_revised": true, "final_justification": "After reading all reviews and rebuttals, I have been convinced that the conceptual innovation is strong enough to outweigh the poor performance on the numerical examples.", "rebuttal": "Many thanks you for your time and for your kind comments on the writing, novelty, and theoretical grounding of our work. We address your questions and concerns below.\n\n\n> **The notation can at times be difficult to read -- for example $*$ is used both to denote the optimizer and the convex conjugate, and there are several usages of related letters such as $\\mathcal{B}$ and $\\mathfrak{B}$ for distinct concepts.** \n\nWe agree that the $\\mathcal{B}$ - \\mathcal{B} notation for the Brenier potential and $\\mathfrak{P}$ - \\mathfrak{P} for Gibbs distribution look similar. Following your comment, we have changed the notation of the Gibbs potential to $\\mathfrak{G}$ - \\mathfrak{G}.\n\nRegarding the star notation, we make a distinction between the optimizer denoted $\\star$ and the convex-conjugate operator, denoted *. While we agree that the symbols look similar (less so in the paper than in markdown), these are standard notations, and we prefer to keep them as they are.\n\n>**The results on higher-dimensional datasets like images are unfortunately quite bad, even for simple cases such as MNIST**\n\nWe acknowledge that our results are far below from the state-of-the-art in generative modeling. \nHowever, compared to the existing OT approaches that use ICNNs to generate data from Gaussian (e.g. [Amos’23], [Makkuva et al. 2020], [Korotin et al. 2020]) our method improves the results on the 2D datasets, MNIST and Cartoon, while also relying exclusively only on ICNNs. The improvement is particularly visible on MNIST (see Figure 5 for comparison).\n\n>**Very minor comment, but the authors miss a few of the major initial references on flow matching** \n\nThank you for these additional references, we have added them. Please feel free to suggest any others you believe could be relevant.\n\n>**1.  In the introduction, the authors comment that $\\omega^*$ is the Brenier potential linking $e^{-\\omega}$ to $\\rho$. Can the authors describe, then, why the method is different from optimal transport? Is it simply because we also learn the source distribution?**\n\nYes, exactly, in the classical optimal transport (OT) problem the source and target distribution are fixed and the objective is to estimate the OT map between these distributions (gradient of a convex potential). In the (conjugate) moment factorizations, **both source distribution and transport map** are uniquely parameterized by the same convex potential (L.37, L.206)\n\n>**We know that the optimal transport map can be hard to learn, and so many generative modeling approaches are intentionally based on sub-optimal transport to simplify the estimation problem. Can the authors explain clearly why it is a \"good idea\" to try to estimate the optimal transport map if the goal is \"only\" generative modeling?** \n\nThis is a great point. In this paper, our hands are tied with the “ideal” OT parameterizations of transport, by a convex potential, because this is what the factorization theorem dictates. \n\nWhile having a convex potential helps with LMC sampling, it is indeed a burden when computing the map. We leave the door open on using our method as a first pre-processing step, in order to learn first a “good” prior, before using a more advanced generative tool.\n\n>**2. Can the authors explain the computational efficiency of their approach? The iterative optimal transport computation in Sec. 4.2 seems very expensive.** \n\nThe proposed method inherits the efficiency (and limitations...) of neural OT solvers. \n\nAlgorithm 1 mirrors the standard procedure for estimating the OT map between two distributions [Amos’23]. The only difference is that, in our case, the (noise) source distribution is re-updated at each iteration. Specifically, in line 4 of Algorithm 1, instead of drawing a minibatch from a fixed distribution, we sample from the evolving distribution using the Langevin Monte Carlo (LMC) algorithm. Re-ampling from this moving, log-concave distribution is the only overhead compared to a standard OT solver. \nTo mitigate this cost, we speed up LMC by reusing particles from the previous iteration as a warm start (L.219 in the paper). Furthermore, to compute $\\tilde{x}$ at line 5, we leverage the latest OT solver by  [Amos’23], which warm-starts the conjugate solver using predictions from a trained MLP, significantly accelerating the overall algorithm, as noted in Appendix (L.693).\n\n> **3.  Can the authors comment on the feasibility of their sampling method from Sec. 4.3? The sampling method requires Langevin samples from the target measure $\\rho$ to learn. This seems to be a chicken and an egg problem, because the method's goal is to generate samples from $\\rho$. From this perspective, I'm having trouble seeing if the method will really work in practice. Is there a way to make it iterative and sample from the current estimate of $\\rho$, using the fact that $\\omega$ is convex? Will this suffer from mode collapse?**\n\nIn the variational inference framework used in Section 4.3, we do not require samples from $\\rho$. Instead, we assume access to the unnormalized densities associated to $\\rho$ i.e. for a given sample $x$, we can require access to $q(x)$ where $q(x) \\propto \\rho(x)$. The algorithm does not train on samples from $\\rho$ but rather on couples $(x, q(x))$ where $x$ is sampled in a more or less clever way. In our experiments, $x$ is sampled uniformly over the domain, and Section 4.3 focuses on learning the conjugate moment potential in this context. To make the method practical,  an idea would be to combine this algorithm with MCMC methods to draw samples $x$ smartly in the same spirit as the methods explored in [Gabrié et al. 2022]. We will include these clarifications in the revised version.\n\n>**4. Can the authors get their method to work better in the numerical examples? I'm a firm believer that we shouldn't require SoTA image results in every paper on generative modeling, but I do think if the method doesn't really work on the 2d checker, it indicates a problem. For publication, I think the authors need to have compelling results on some simple examples.**\n\nWe agree that CMFGen does not perform perfectly on the 2D checkerboard dataset. However, note that the generative ICNNs also fail in this case (see Figure 13). More generally, the box plots on Figure 13 shows that in low-dimensional settings, our method performs as well as generative ICNNs, and outperforms them on higher-dimensional datasets such as MNIST and Cartoon (see Figures 5, 14, and 15). Since ICNNs are known to be challenging to train [Korotin et al. 2021], we believe the failure on the checkerboard dataset is primarily due to the limitations of the ICNN itself rather than the methodology we propose. In response to your comment, we experimented with larger ICNN architectures—specifically, models with 6 hidden layers of size 128 and 5 hidden layers of size 512—to improve performance. For now the gains were not significant, but we plan to continue exploring this.\n\n>**They should also try to quantify their results in some way, such as FID on images and metrics such as KL or Sinkhorn in low-d. The authors should also tone down the claims in the text that the method works well, which are sort of contradicted by their figures.**\n\nNote that we use the Sinkhorn divergence to compare our method to generative ICNNs (see Figure 13), where boxplots provide a visual summary of the results. From our perspective, the method performs well—this is, to our knowledge, the first time MNIST samples have been successfully generated using ICNNs (see Figure 5 for a comparison between our results and samples generated by a generative ICNN). That said, we acknowledge that, when compared to state-of-the-art generative methods, the results are not impressive. We will make this distinction clearer in the revised version."}, {"review_id": "kl3rmbO4YI", "reviewer": "Reviewer_CFgx", "summary": "The paper proposes to a novel approach to generative modeling, using a variant of optimal transport. Rather than relying on a heuristically chosen initial distribution, which is possibly the biggest hyperparameter in generative modelling, they propose a method which constructs the base distribution along with the pushforward map to the data distribution. They build upon known theoretical tools like the brenier potential and moment measure to eliminate the limitations of typical optimal transport based methods which are prone to create very ill-conditioned maps. They demonstrate the results on some synthetic low dimensional datasets as well as MNIST and Cartoon dataset.", "questions": "High level questions:\n1. Can you please provide a complete and formal statement of Breiner's thorem that is being used either in the main paper or in Appendix?\n2. From what I understand, although the theory holds for convex potentials, none of the methods are actually using convex potentials. Is that correct? If not, how are you ensuring that $w_\\theta$ are convex? If yes, how is convex conjugate / legendre transform interpreted for a non-convex function?\n3. Would it be possible to compare performance with other energy-based modelling methods (like say score matching) in a synthetic setting for example over an exponential family?\n4. Line 214: How is $w_\\theta$ initialized to be roughly $\\lVert \\cdot \\rVert^2$? It does not feel necessary to have this initialization for the proposed method, can you explain choice of this initialization?\n\nQuestions/comments about the proofs:\n1. Line 404: $w$ is convex only if $\\Omega$ is convex. Is this assumed?\n2. Line 405: How is $\\nabla w$ defined? $w$ seems to be discontinuous on boundary of $\\Omega$.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The paper developes a critical idea to eliminate one of the more important hyper parameters in generative modelling, the initial distribution. The known approaches to picking this distribution are mainly heuristic, and this proposed alternative provides a good approach to this problem. The main result of the paper is the existence of conjugate moment potential, which is proved via a fixed point theorem applied to the Brenier potential map. I found the proof to be fairly involved.\n\nPaper presents a fairly convincing argument about why conjugate moment potentials are preferred over moment potentials. The argument is limited to a Gaussian distribution, but suffice to convince the ill-conditioned nature of moment potentials.\n\nThe paper has limited experimental coverage with focus on low-dimensional synthetic datasets along with some experiments on high dimensional datasets like MNIST and Cartoon. It would be interesting to see how the experiments scale, but I think current experiments suffice for the purpose of this paper whose main goal is to explore a novel modelling idea.", "was_revised": true, "final_justification": "I would like to thank the authors for response to my concerns, all the ones which can be answered without increasing the scope of the paper drastically and unrealistically have been answered. I would like to keep my original score and would urge the AC to lean towards acceptance if the paper is right on the border.", "rebuttal": "Many thanks you for your encouraging comments. We appreciate that you have recognized our contribution, which proposes to reassess the link between noise distribution and transport.\n\n>**Can you please provide a complete and formal statement of Brenier's theorem that is being used either in the main paper or in Appendix?**\n\nWe will include a formal description of the Brenier theorem in our appendix. \n\n>**From what I understand, although the theory holds for convex potentials, none of the methods are actually using convex potentials. Is that correct? If not, how are you ensuring that $w_\\theta$ are convex? If yes, how is convex conjugate / legendre transform interpreted for a non-convex function?**\n\nThank you for the question. In our case, the potentials $w_\\theta$ are indeed convex with respect to their input by construction, as we rely on Input Convex Neural Networks (ICNNs) [Amos et all. 2017], this is mentioned L.170. We will add a more detailed explanation of the ICNN construction in the appendix for clarity. Regarding the convex conjugate: since our potentials are convex by design, the Legendre transform is well-defined and its gradient can be reliably computed by maximizing the concave problem defined L.181 using a solver. We use an off-the-shelf L-BFGS implementation (L.182).\n\n>**Would it be possible to compare performance with other energy-based modelling methods (like say score matching) in a synthetic setting for example over an exponential family?**\n\nOur method does not currently outperform state-of-the-art generative models, and we expect that advanced score-matching-based models will achieve better performance than ours. We believe this is primarily due to our reliance on ICNNs, which, while advantageous for parameterizing convex functions, are known to be challenging to train ([Korotin et al. 2021]). For this reason, we compare our method to other generative ICNNs using the Sinkhorn divergence as a metric (see boxplots in Figure 13). Make the method scalable will require improving further the training of ICNN architectures, an important research topic.\n\n>**Line 214: How is $w_\\theta$ initialized to be roughly $\\lVert \\cdot \\rVert^2$? It does not feel necessary to have this initialization for the proposed method, can you explain choice of this initialization?**\n\nThis initialization was proposed in [Bunne et al. 2022] and is akin to initializing with the identity map, but you are correct, one can consider other more data informed initializations, and this topic has been the subject of great research in [Hoedt, Klambauer, 23] but this is not yet useful for OT maps.\n[ Hoedt, Klambauer, Principled Weight Initialisation for Input-Convex Neural Networks, Neurips 23]\n\n>**Line 404: $w$ is convex only if $\\Omega$ is convex. Is this assumed?**\n\nYes, $\\Omega$ is assumed to be convex, as stated in Theorem 1. We will also clarify this in the appendix when presenting the proof.\n\n>**Line 405: How is $\\nabla w$ defined? $w$ seems to be discontinuous on boundary of $\\Omega$.**\n\nThe function $w$ is differentiable in the interior of $\\Omega$, so taking its gradient there is well-defined. On the boundary, $w$ is discontinuous by definition, as it takes infinite values outside $\\Omega$, and thus its gradient is not defined there. However, note that the boundary of the bounded convex set $\\Omega$ has Lebesgue measure zero, so this does not affect the proof.\n\n>**Formatting issues** \n\nThanks for spotting these typos — we will take them into account in the revised version.\n\n>**Lines 91: Does not supported on hyperplane mean that support is not contained in any hyperplane?**\n\nIt means that the support does not **strictly** include any hyperplane (or, more generally, does not put positive mass on a subset of lower dimension than the ambient space, or, more loosely, that it has a density w.r.t. the Lebesgue measure) This is a classic assumption to obtain results in OT.\n\n>**Isn't boundary a measure zero set, implying that if $u$ is only discontinuous at the boundary then it is still essentially continuous? I do not see how this is relevant to rest of the paper.**\n\nThe boundary of the bounded, convex set $\\Omega$ is a null set for the $n$-dimensional Hausdorff measure, but the definition of essential continuity in this context requires the discontinuity set to have zero $(n-1)$-dimensional Hausdorff measure. Since the boundary has positive $(n-1)$-dimensional measure, $u$ is not essentially continuous. This result is not directly relevant to our contributions; it was intended to introduce [Cordero-Erausquin and Klartag, 2015], and the problem they studied, as they made a significant contribution to the moment measure factorization with this result.\n\n>**From what I understand LMC only ever provides an approximation, these lines seem to suggest that it matches exactly to the final distribution after a warm-up period.**\n\nYou are right, the LMC algorithm is the Euler discretisation of the Langevin diffusion process and this discretization induces a bias that can be controlled for smooth log concave distributions [Dalalyan 2017]. We will add a note on that bias in the revised version. \n\n>**Line 428: Is $\\mathscr{C}(\\Omega)$ the set of convex functions on $\\Omega$? $\\mathcal{C}(\\Omega)$ is used for same purpose on line 71**\n\nIt is actually the Banach space of continuous functions over a compact set, so no convexity assumption is involved. While we used a different font, we will introduce explicit notation such as \\(\\operatorname{cvx}(\\Omega)\\) to clarify these distinctions."}, {"review_id": "uGz952Wkh6", "reviewer": "Reviewer_dw2D", "summary": "The standard approach in generative modeling separates noise sampling (e.g., Gaussian) from mapping (e.g., flows). The authors propose an alternative that links sampling and mapping, inspired by moment measures, which connect log-concave distributions and convex potentials. However, this method is impractical for common tasks. Instead,  ρ is factorized as ∇w* ♯ e^−w, with w* the convex conjugate of w.  This is called the conjugate moment measure approach. Using optimal transport and input-convex neural networks, w is recovered from samples, even when ρ’s density is only known up to a constant. This yields more intuitive results in practice.", "questions": "I have the following questions:\n\n(1) Is the conjugate moment measure factorization given in Theorem 1 unique? \n\n(2) What is the  computational complexity the proposed algorithm?\n\n(3) How scalable the algorithm is?\n\n(4)  I wonder if the authors can comment on the statistical errors of the generated distribution \n      based on the proposed method.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Stengthes: This paper presents a novel approach to generative modeling by proposing an alternative to the traditional two-step process of noise sampling and mapping. Drawing inspiration from the theory of moment measures, the authors explore a framework that directly couples the sampling and mapping steps.  Specifically,  the paper introduces the idea of \"conjugate moment measures,\" where the target distribution is represented via the Monge map associated with the convex conjugate of a potential function. The authors use optimal transport solvers and parameterize the potential using input-convex neural networks.  \n\nWeakness: The main weakness of this work is that the computational cost is not dsicussed in detail. For sampling \nsampling from ρ using its Conjugate Moment Factorization, it requires concave maximization problem:∇w∗(y) = arg supx ⟨x, y⟩ − w(x) in each iteration. It is not how expensive this is computaitonally, especially in high-dimensional settings. Second, there is no discussion on the advantages/disadvantages of the proposed method relative to the existing generative methods, such as diffuion and continuous normalizing flows. Finally, the experimental results are not convicining. While the results show some promise of the method, more experiments with higher-dimensional image data will  significantly strengthen the paper.", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for taking the time to review our paper and for your valuable feedback.\n\n>**The main weakness of this work is that the computational cost is not dsicussed in detail. For sampling sampling from ρ using its Conjugate Moment Factorization, it requires concave maximization problem:∇w∗(y) = arg supx ⟨x, y⟩ − w(x) in each iteration. It is not how expensive this is computaitonally, especially in high-dimensional settings.**\n\nWe agree that this aspect is not discussed in the paper and we will add a paragraph in the revised version. \n\nAs you noted, the problem is a well-posed concave maximization that must be solved at inference time but also during training at each gradient step — not by differentiating through the solver, but by using the optimal solution to get the gradient.\n\nThis challenge is inherent to neural OT solvers ([Makkuva 2020],  [Amos 2023]). The recent solver we use from [Amos 2023]  amortizes the cost of computing $\\nabla w^\\*$ \nby training an MLP that, given $y$, predicts $\\nabla w^\\*(y)$ **L.687** in order to initialize the solver. This prediction is then refined by running 30 optimization steps (as proposed in [Amos 2023]) to improve accuracy. This refined value is used both in the loss (step 5 of Algorithm 1), applying `jax.lax.stop_gradient` to treat it as constant, and to train the MLP for better approximations. At inference, we can again amortize the cost by using the MLP prediction refined with these optimization steps to get a fast and accurate approximation of $\\nabla w^*$.\n\n\n>**Second, there is no discussion on the advantages/disadvantages of the proposed method relative to the existing generative methods, such as diffuion and continuous normalizing flows.**\n\nAt this moment, we do not see the conjugate moment factorization as an immediate competitor in the field of generative modeling, and do not wish to present it that way, for the obvious reason that our performance is lagging behind. \nWhile we believe this is very likely due to the limitations arising when training ICNNs. Our paper simply asks whether one can propose a generative model that is able to include **both** noise distribution **and** transport in a single parameterization. At the moment we see no other approach than the conjugate moments approaches presented in the paper. While this might be a curiosity, we hope the originality of our idea can trigger new ideas in the field.\n\n>**Finally, the experimental results are not convicining. While the results show some promise of the method, more experiments with higher-dimensional image data will significantly strengthen the paper.**\n\nWe agree that the current method does not yet compete with state-of-the-art generative models. However, relying solely on ICNNs, it improves upon results generated by generative ICNNs. See Figure 5 for comparisons on MNIST, and Figures 14 and 15 for results on the Cartoon dataset. For a discussion on the limited expressiveness of ICNNs in generative modeling, see [Korotin et al., 2021].\n\n>**(1) Is the conjugate moment measure factorization given in Theorem 1 unique?**\n\nWe did not manage to answer this question in the general case. However, for the Gaussian case, we proved the existence of a unique fixed point within the class of quadratic functions (whose Gibbs distributions are Gaussian). This is a challenging theoretical question that we are currently investigating.\n\n>**(2) What is the computational complexity the proposed algorithm?**\n\nFor CMFGen, the algorithm complexity per gradient step is `O(d PB(T_{LMC}+T_{conj}))` with `B` the batch size, `P` the number of parameters, `T_{LMC}` the number of steps used for `LMC`, and `T_{conj}` the number of steps used for computing the convex conjugate `d` dimension. For CMFMA the complexity is `O(d PB(T_{LMC}+T_{conj}+P))` due to hessian computations.  For comparison, a supervised training of a NN is typically `O(PB)`.\n\n>**(3) How scalable the algorithm is?** \n\nThis question can be interpreted in multiple ways. \n\nPerhaps the most obvious bottleneck in our implementation lies in data dimensionality, for two reasons:\n- $w_{\\theta}$ is parameterized as an ICNN. In high $d$, ICNNs are known to be hard to train. While they encode desirable convexity properties, the constraints in their parameters prevent established optimizers (e.g. ADAM) from converging quickly. While we do obtain reasonable results (our paper is, to the best of our knowledge, the first to train ICNNs in the pixel space of MNIST) there is still a gap w.r.t non-constrained models.\n- Our approach requires evaluating conjugate functions at all points in a mini-batch. While the function itself is convex and the optimization problem is therefore well posed, increase in dimension will typically mean that more effort is needed to evaluate these conjugate functions $w_{\\theta}^\\*$.\n\n\n>**(4) I wonder if the authors can comment on the statistical errors of the generated distribution based on the proposed method.**\n\nPractically speaking, the box plots on the Sinkhorn divergence (Figure 13) demonstrate that the generated distribution was close to the target, indicating good empirical performance.\nTheoretically, the statistical error depends on how well the method approximates the conjugate moment potential, which in turn hinges on whether the fixed point is found—something we have not yet proven. In theory, at least, the use of ICNNs is not a limitation here, as they are universal approximators of convex functions."}, {"review_id": "zengJFlTwP", "reviewer": "Reviewer_CXvw", "summary": "This paper introduces conjugate moment measures for sampling. Drawing inspiration from moment measures, which enable the expression of a measure $\\rho \\in \\mathcal P(\\mathbb{R}^d)$ as $\\rho = \\nabla u_\\sharp e^{-u}$ for some convex $u: \\mathbb{R}^d \\to \\mathbb{R}$, the paper shows that any measure $\\rho \\in \\mathcal P(\\mathbb{R}^d)$ that is absolutely continuous with compact, convex support can also be written $\\rho = \\nabla w^{\\ast}_{\\sharp} e^{-w}$, where $w: \\mathbb{R}^d \\to \\mathbb{R}$ is a convex potential and $w^{\\ast}$ is the convex conjugate of $w$. The paper additionally shows the existence of such a conjugate moment decomposition for Gaussian measures with non-degenerate covariance matrices and argues, based on examples for two-dimensional Gaussian and one-dimensional Gaussian mixture distributions, that conjugate moment measures are better suited for sampling than moment measures.\n\nThe paper then proposes computational strategies for sampling from $\\rho$ via its conjugate moment measure decomposition under three different information scenarios: (1) when $w$ is known, (2) when existing samples from $\\rho$ are known, and (3) when the unnormalized density of $\\rho$ is known. Estimation of $w$ from samples involves a fixed-point algorithm wherein a Brenier potential is computed on each step, either in closed form for 1D distributions or by optimization of an ICNN using neural optimal transport. Estimation of $w$ from a density entails training an ICNN on a collocation loss corresponding to the Monge-Ampere equation (a-la PINNs). In either case, once $w$ is in hand sampling is performed by first using Langevin Monte Carlo (LMC) to sample from $e^{-w}$ and then using $w^*$, identified via gradient-based optimization, to push-forward these samples to $\\rho$. The numerical examples in the paper demonstrate proof-of-concept.", "questions": "* Regarding the increased computational cost relative to other methods: if we have the density of $\\rho$, in order to sample from $\\rho$ using a conjugate moment potential we have to (1) train a PINN (which specifically must be an ICNN) to obtain $w$, (2) run LMC to get samples of $e^{-w}$, and then (3) compute $\\nabla w^*(x)$ for every sample $x$ generated by LMC. The pipeline for generative modeling with conjugate moment measures is similar. There are many sampling methods that enable the user to generate new samples directly after the training of a neural network (neural ODEs, flow-matching, using PINNs to solve continuity equations, etc.), without the post-processing steps involved in the proposed method. In the density driven-setting, one could run LMC on $\\rho$ directly. How does one rationalize the extra costs associated with using conjugate moment measures to sample? Have you compared the quality of samples generated by the conjugate moment measure with those generated by a trained neural network (for dynamic transport) directly or by LMC? \n\n* Theorem 1 guarantees existence of a conjugate moment factorization for absolutely continuous measures $\\rho$ with compact, convex support. Many of the distributions in the numerical examples do not satisfy these assumptions. Is this a problem? How do you know, for example, that the fixed-point iteration in CMFGen will converge? Have you tried any examples where it didn't converge? \n\n* It appears that noise is added to the generative modeling training images to guarantee existence of an OT map. Computationally, does the pipeline break down if noise is not added?\n\n* The face image inpainting example seems relatively easy because the mask covers one vertical half of the the image and faces are symmetric about the vertical axis. I could inpaint the images by simply reflecting the unmasked half across the vertical axis. What happens if you mask the top or bottom half of the image instead of a left or right half? Is the quality of the in-painting as good, or does it get degraded?", "limitations": "It appears that in the generative modeling setting noise must be added to images before training to ensure the existence of an OT map. This requirement seems like a major limitation for generative modeling, as it would preclude the generation of noiseless images, but it is not discussed outside of one line in the appendix. Do the authors see a way that this step could be avoided? Or do they see a way that their conjugate moment factorization could play an intermediate role in generative modeling pipelines which ultimately produce noiseless images? Further discusson of these limitations and possibilities in the conclusion would help contextualize the results of the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths \n* This paper is clearly written and easy-to-follow. \n\n* The result showing existence of conjugate moment measures is interesting and new and follows from a nice fixed-point argument. \n\n* The results and figures concerning the moment potentials and conjugate moment potentials of Gaussians are helpful for building intuition.\n\n* The experimental details are clearly detailed in the appendix and would facilitate reproduction of these results by others.\n\n## Weaknesses\n* The argument that moment potentials are unsuitable for practical sampling tasks and that conjugate moment potentials offer a better alternative is primarily based on a relatively narrow set of examples -- Gaussians and Gaussian mixtures in one and two dimensions.  I am not convinced that the behavior in these examples generalizes to non-Gaussian (mixture) and higher-dimensional distributions based on the arguments given in the paper. \n\n* Moreover, moment measures/potentials are not typically used for practical sampling tasks, so from that standpoint the idea that we need an alternative to the moment potential factorization isn't credible. \n\n* The computational effort required to sample via a conjugate moment measure is high compared to other sampling methods, and the extra computation has not been adequately justified.  \n\n* The numerical results include no comparisons to widely used or state-of-the art sampling methods. The only comparisons are to ICNNs trained to push forward a Gaussian to $\\rho$ directly, whereas state-of-the-art neural network transport approaches usually invoke some kind of dynamics, rather than being \"one-shot\".", "was_revised": true, "final_justification": "After discussion with the authors, I agree with them that the primary contribution of this paper is not a new, state-of-the-art method for sampling or generative modeling, but rather a new paradigm in which to approach generative modeling. The conjugate moment potential factorization and the fixed point algorithm they devise to find it are mathematically interesting, and I think that aspects of this work have the potential to inform future generative modeling pipelines. I therefore am increasing my score to a (4), as the numerical results are still somewhat weak and limited.", "rebuttal": "We are very grateful for your time spent reading our paper, as well as for your detailed review. We did our best to address your concerns and questions below.\n\n>**The argument that moment potentials are unsuitable for practical sampling tasks and that conjugate moment potentials offer a better alternative is primarily based on a relatively narrow set of examples -- Gaussians and Gaussian mixtures in one and two dimensions. I am not convinced that the behavior in these examples generalizes to non-Gaussian (mixture) and higher-dimensional distributions based on the arguments given in the paper.**\n\nIf you were not convinced by our Gaussian experiments, we suggest that you read the argument put forward in L.119-131, which helped us understand better the limitations. In the moment measure factorization, $\\nabla u$ must take values, by construction, on the support of the measure 𝜌 to be factorized. If 𝜌 is narrow and concentrated, so must $\\nabla u$ be. If $\\nabla u$ moves slowly, $u$ looks like a linear potential.  If 𝜌 is narrow and concentrated around 0, then $u$ is almost constant. In both cases, $u$ would be fairly degenerated and extremely spread on its support.\n\n>**Moreover, moment measures/potentials are not typically used for practical sampling tasks, so from that standpoint the idea that we need an alternative to the moment potential factorization isn't credible.**\n\nWe agree that moment measures were never highlighted as a means to generate data better.\nGiven the importance of generative models in society as a whole, we believed that exploring original ideas to define new generative models was a worthwhile avenue. We looked for models that include, holistically, a parameterized noise description, and naturally found inspiration in moment factorizations (L3-4).\nMoment measures are not widely known within the machine learning community, and one of our contributions is to introduce them to Neurips, and propose an alternative factorization.\n\n>**The numerical results include no comparisons to widely used or state-of-the art sampling methods. The only comparisons are to ICNNs trained to push forward a Gaussian to $\\rho$ directly, whereas state-of-the-art neural network transport approaches usually invoke some kind of dynamics, rather than being \"one-shot\".**\n\nIt is true that our algorithm does not match the performance of progressive generative models. However, our approach addresses a different aspect of the problem—namely, finding a suitable prior that facilitates the transport by the transport map. We will clarify.\n\n>**The pipeline for generative modeling with conjugate moment measures is similar. There are many sampling methods that enable the user to generate new samples directly after the training of a neural network (neural ODEs, flow-matching, using PINNs to solve continuity equations, etc.), without the post-processing steps involved in the proposed method.**\n\nYour parallel with Neural ODE is interesting! \n\nIn that sense the LMC effort is indeed comparable to the ODE solver effort (parameterized by # steps), with the nuance that LMC uses the gradient of a time-independent *potential*, vs. a time parameterized velocity field for ODE.\n\nFor that reason, the convergence of LMC is conceptually simpler than the integration of a vector field with a rotational component.\n\n>**In the density driven-setting, one could run LMC on $\\rho$ directly. [...] Have you compared the quality of samples generated by the conjugate moment measure with those generated by a trained neural network (for dynamic transport) directly or by LMC?**\n\nIn the case of non–log-concave distributions such as $\\rho$, the LMC algorithm lacks theoretical guarantees [Roberts and Tweedie, 1996; Cheng and Bartlett, 2018; Dalalyan and Karagulyan, 2019]. In particular, when the probability density function has multiple local maxima, the generated samples tend to be highly correlated, as particles from the LMC algorithm often become trapped in local basins [Pompe et al. 2020] [Gabrié et al.\n2022].\n\nIn the variational inference framework used in Section 4.3, our focus is on learning the conjugate moment potential, rather than designing a practical algorithm to sample from unnormalized densities. For this reason, we did not compare to other sampling algorithms. To make the method practical, a possible direction would be to combine our approach with MCMC methods to improve the current uniform sampling over the optimization space, in the spirit of the algorithm proposed by [Gabrié et al., 2022].\n\n>**Theorem 1 guarantees existence of a conjugate moment factorization for absolutely continuous measures $\\rho$ with compact, convex support. Many of the distributions in the numerical examples do not satisfy these assumptions. Is this a problem?**\n\nWhile the existence is guaranteed for densities, the neural networks _must_ be trained on a stream of i.i.d. samples. This is the same type of result that allows for instance the Brenier theorem to guarantee the existence of an optimal map, and numerical methods to try recovering them. While degeneracies might happen if e.g. the samples do clearly lie on a thin submanifold, as is the case for images if the background is always the same, they can be corrected by adding noise.\n\n>**How do you know, for example, that the fixed-point iteration in CMFGen will converge? Have you tried any examples where it didn't converge?**\n\nWe did not manage to theoretically prove the convergence of the fixed-point iteration in CMFGen. Note that in practice (except in 1D), we run a smoothed version where we do not compute the exact OT map at each iteration but instead take a single gradient step. With this strategy, the algorithm always converged for all the experiments. In 1D, where the OT map is fully estimated at each iteration, we encountered convergence issues in some experiments (distinct from those presented in the paper). However, these were easily resolved using a simple heuristic: averaging the OT maps computed at iterations $n-1$ and $n-2$. This is a failure case that we omitted and will add to the Appendix. It may in fact suggest that using momentum based corrections could be beneficial.\n\n>**It appears that noise is added to the generative modeling training images to guarantee existence of an OT map. Computationally, does the pipeline break down if noise is not added?** \n\nFollowing your question, we tested our algorithm without adding noise on the MNIST and Cartoon datasets. While the pipeline remained stable and did not break, the visual results were slightly degraded. These cases are interesting because they are not covered by Theorem 1, which assumes absolutely continuous probability measures.\n\n>**The face image inpainting example seems relatively easy because the mask covers one vertical half of the the image and faces are symmetric about the vertical axis. [...] Is the quality of the in-painting as good, or does it get degraded?**\n\nIt is not clear whether the experiments mentioned are more challenging for the algorithm. We conducted the two experiments you suggested—masking the top half and masking the bottom half of the image—and in both cases, the algorithm consistently filled in the masked regions. The results were comparable in quality to those obtained with horizontal masking, as shown in Figure 17. When masking the bottom part of the image, the algorithm tends to produce samples that closely match the original. However, when masking the top part, features such as hair color can become ambiguous—especially in cases where the cartoon character has very short hair—leading the algorithm to generate cartoons with different hair colors or hairstyles. Unfortunately we cannot share these images due to the rebuttal process in place this year.\n\n>**Do the authors see a way that this step could be avoided? Or do they see a way that their conjugate moment factorization could play an intermediate role in generative modeling pipelines which ultimately produce noiseless images?**\n\nWe currently do not see a way to remove the added noise without degrading performance, but this is intrisically due to the degeneracy (uniform background) of the data. A possible direction to improve scalability would be to use our method to estimate the conjugate moment potential of the distribution $\\rho$, and then retain only the corresponding Gibbs distribution as a learned log-concave prior. Flow Matching combined with mini batch Optimal Transport ([Tong et al. 2023], [Pooladian et al. 2023]) could then be used to learn a generative model between this prior and the target distribution. This approach would approximate our current method and may offer better scalability. This however would involve two neural networks (one for learning the prior, one for transport), and thus move slightly away from the original theory of conjugate moment measure factorization, which relies on a single convex potential."}, {"review_id": "6ZVsTmJJxy", "reviewer": "Reviewer_Cdjx", "summary": "This paper proposes a new method for generative modeling based on optimal transport and moment measures. The method links the sampling phase and the mapping phase, which are typically treated as completely separate in most generative modeling approaches. First, the authors recall a result on moment measures stating that for any measure ρρ (satisfying certain conditions), there exists a unique convex function $ u $ such that  $ \\rho = \\nabla u $ # $  e^{−u} $. They then observe, using a simple Gaussian example, that this factorization is not well suited for generative modeling. The authors subsequently propose an alternative factorization: there exists a unique convex function $ w $ such that  $ \\rho = \\nabla w^* $ # $  e^{−w} $ where $ w^* $ is the convex conjugate of $ w $. They propose learning the convex potential $ w$  using Input-Convex Neural Networks (ICNNs), leveraging recent advances in neural optimal transport solvers, and then sampling from $ e^{-w} $ using a Langevin dynamics scheme. Finally, the authors demonstrate the effectiveness of their method for generative modeling on low-dimensional toy examples and small synthetic image datasets.", "questions": "- 1/ If $ \\rho = \\nabla u$  #  $ \\mathfrak{B}_u = \\nabla w^* $ #  $ \\mathfrak{B}_w $, do we have $ \\mathfrak{B}_u^*= \\mathfrak{B}_w $? \n\n- 2/ What could be the impact of the fact that your results only hold for absolutely continuous measures, given the manifold hypothesis, which suggests that real data distributions are highly degenerate?\n\n- 3/ Is $ \\nabla w^* $ still an OT map? \n\n- 4/ From the experiments, it appears that the results are often noisy. Do you have an intuition to explain this?", "limitations": "I think the limitations of the method could be a bit more discussed, relatively to the potential discussion that better situates the contribution already mentioned in section 'Strengths and Weaknesses'.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This is a good theoretical paper that is well written. The maths, as far as I can tell, seem to be correct, and the method appears to be original and novel. Although the experimental results suggest that the method cannot yet compete with state-of-the-art generative modeling approaches—even on relatively simple tasks such as 2D toy examples or MNIST generation—I still consider it a promising new direction with solid theoretical foundations and a strong connection to optimal transport.\n\nThat being said, I believe the experimental section could be slightly improved. In my opinion, it lacks commentary on failure cases, such as the checkerboard patterns in 2D, for instance. Additionally, the ICNN architecture used for the MNIST experiment seems overly simple compared to standard models typically applied to this dataset. It would have been helpful to test a more expressive architecture—at least a convolutional neural network (CNN)—to better assess the potential of the method for generative modeling.\n\nLastly, I think the paper would benefit from a brief discussion that situates the contribution within the broader field of generative modeling. In particular, it would be useful to outline the potential advantages of this method over existing alternatives.", "was_revised": true, "final_justification": "I think this is an interesting theoretical paper with a nice new idea. However, the heavy dependence on Input Convex Neural Networks (ICNNs) for the practical realization of the method seems to be a serious limitation. For this reason, I will keep my rating (borderline accept).", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to review our paper. We did our best to answer some of your concerns / questions below.\n>**That being said, I believe the experimental section could be slightly improved. In my opinion, it lacks commentary on failure cases, such as the checkerboard patterns in 2D, for instance.**\n\nWe agree that CMFGen does not perform perfectly on the 2D checkerboard dataset and that we could comment more on that. However, note that the generative ICNNs also fail in this experiment (see Figure 13). More generally, the box plots on Figure 13 shows that in low-dimensional settings, our method performs as well as generative ICNNs, and outperforms them on higher-dimensional datasets such as MNIST and Cartoon (see Figures 5, 14, and 15). Since ICNNs are known to be challenging to train [Korotin et al. 2021], we believe the failure on the checkerboard dataset is primarily due to the limitations of the ICNN itself rather than the methodology we propose. We will clarify this briefly in the main paper and provide details in the appendix.\n\n\n>**Additionally, the ICNN architecture used for the MNIST experiment seems overly simple compared to standard models typically applied to this dataset. It would have been helpful to test a more expressive architecture—at least a convolutional neural network (CNN)—to better assess the potential of the method for generative modeling.**\n\nThank you for the suggestion. We tested convolutional ICNNs, which have been used, for example, in [Korotin et al. 2021] to generate clean images from degraded inputs. However, to our knowledge, they have not been applied in generative modeling settings where the starting distribution is a noise prior. In our experiments, convolutional ICNNs did not outperform plain MLP-based ICNNs. We believe our method could benefit from improved integration of convolutional layers into ICNN architectures.\n\n>**Lastly, I think the paper would benefit from a brief discussion that situates the contribution within the broader field of generative modeling. In particular, it would be useful to outline the potential advantages of this method over existing alternatives.**\n\nAt this moment, we do not see the conjugate moment factorization as a direct competitor to current state-of-the-art generative models, as its performance lags behind. The main contribution of our approach lies in linking the sampling process of the prior distribution with the transport of this prior to the data distribution. At the moment we see no other approach than the (conjugate) moments approaches presented in the paper. While this may seem like a conceptual curiosity, we hope the originality of our idea can trigger new ideas in the field.\n\n Compared to standard generative models like diffusion models, our method offers the following advantages: \n- it does not require arbitrarily choosing a prior distribution\n- it provides a fast access to the generated log-probability, enabling applications such as image completion.\n\nThe main limitation is that CMFGen relies on Input Convex Neural Networks (ICNNs), which require further improvements for the method to compete with state-of-the-art generative models.\n\n>**1/ If $ \\rho = \\nabla u$ # $ \\mathfrak{B}_u = \\nabla w^* $ # $ \\mathfrak{B}_w $, do we have $ \\mathfrak{B}_u^*= \\mathfrak{B}_w $?**\n\nNo, this is not the case. Note that in both formulations, the potential appears both in the transport map—$\\nabla u$ for the moment measure factorization and $\\nabla w^* $ for the conjugate moment measure—and in the associated Gibbs distribution, namely $\\mathfrak{B}_u$ and $\\mathfrak{B}_w$ respectively. Therefore, the first equation does not imply the second. For instance, in the Gaussian case (see Propositions 1 and 2), we have $\\mathfrak{B}_u^* = \\mathcal{N}(0, \\Sigma)$, while $\\mathfrak{B}_w = \\mathcal{N}(0, \\Sigma^{1/3})$.\nWe did not manage to establish any theoretical relationship between $u$ and $w$, or between $\\mathfrak{B}_u$ and $\\mathfrak{B}_w$.\n\n\n>**2/ What could be the impact of the fact that your results only hold for absolutely continuous measures, given the manifold hypothesis, which suggests that real data distributions are highly degenerate?**\n\nWhile the existence is guaranteed for densities, the neural networks must be trained on a stream of i.i.d. samples. This is the same type of result that allows for instance the Brenier theorem to guarantee the existence of an optimal map, and numerical methods to try recovering them. While degeneracies might happen if e.g. the samples do clearly lie on a thin submanifold, as is the case for images if the background is always the same, they can be corrected by adding noise.\n\nFor these reasons, we do not consider this assumption to be particularly limiting for practical applications.\n\n>**3/ Is $ \\nabla w^*$ still an OT map?**\n\nYes, this follows from Brenier's theorem: for any convex function $f$ and any probability measure $P$, the gradient map $ \\nabla f $ is the optimal transport map from $P$ to the pushforward measure $\\nabla f$#$P$ under the squared Euclidean cost. Since $w*$ is convex, it follows that $\\nabla w^*$ is indeed an optimal transport map.\n\n>**4/ From the experiments, it appears that the results are often noisy. Do you have an intuition to explain this?** \n\nWe added low-variance Gaussian noise to the dataset to ensure absolute continuity of the target probability measures. This explains why the generated samples also appear a bit noisy. Moreover, since our networks do not use convolutional layers, this may further amplify the impression of noise in the generated samples."}]

Author rebuttal:

Meta-review: This work studied two factorizations of a density $\rho$ through a convex function $u$ or $w$: 
- moment measure due to Cordero-Erausquin and Klartag [2015]: $\rho = (\nabla u)_{\sharp} e^{-u}$
- conjugate moment measure (proposed): $\rho = (\nabla w^*)_{\sharp} e^{-w}$

Existence of the conjugate moment measure is established under reasonable assumptions and a fixed-point algorithm is proposed to estimate $w$, when the density $\rho$ is available either in form or in samples or in score. Numerical experiments were conducted to show the feasibility of the proposed factorization as a generative modelling tool. 

The reviewers all praised the originality of the idea, the presentation and clarity of the paper and its potential to inspire new developments down the road. The reviewers also expressed concerns on the sub-par empirical performance when applied to generative modelling. Nevertheless, the consensus is that the originality of the idea outweighs the limited empirical results, which I concur. In the end all reviewers recommended (weak) acceptance, and we believe it'll serve the community much good by making the ideas  in this work more widely known. 

Some extra comments for the authors to consider incorporating: 

- The argument against moment measures in terms of the support is somewhat weak and hypothetical. What is wrong when $u$ is spread out or near constant? The authors also did not exclude the possibility of other examples that would demonstrate similar issues for the conjugate moment measure (at least from the same arguments in Lines 119-131 it follows that $w$ would have concentrated support, which is not necessarily easy to capture through ICNNs).

- Is generative modelling really the right application for the two factorizations? Learning the optimal transport map with two fixed densities is already non-trivial, let alone learning the OT map together with a tied source distribution? 

- Related to above, the following issue is somewhat neglected: the conjugate moment measure approach is more expensive during inference time, since we only learn $w$ while $\nabla w^*$ is required for each generation. In this respect, the moment measure approach seems to be more economical. Perhaps the authors should explicitly discuss this tradeoff. 

- The argument against a fixed Gaussian source distribution is weak. There is nothing wrong by itself to fix the source distribution to be Gaussian, unless one can compellingly show otherwise (not the case from the authors experiments). In this regard, the following work that learns a source distribution to better capture heavy-tails might be useful: Tails of Lipschitz Triangular Flows (https://arxiv.org/abs/1907.04481).
Decision: Accept (poster)