Understanding DDPM Latent Codes Through Optimal Transport
Authors: Valentin Khrulkov
Gleb Ryzhakov
Andrei Chertkov
Ivan Oseledets
Venue: ICLR 2023 poster (2023)
Keywords: diffusion models
ddpm
optimal transport
theory

Abstract: Diffusion models have recently outperformed alternative approaches to model the distribution of natural images. Such diffusion models allow for deterministic sampling via the probability flow ODE, giving rise to a latent space and an encoder map. While having important practical applications, such as the estimation of the likelihood, the theoretical properties of this map are not yet fully understood. In the present work, we partially address this question for the popular case of the VP-SDE (DDPM) approach. We show that, perhaps surprisingly, the DDPM encoder map coincides with the optimal transport map for common distributions; we support this claim by extensive numerical experiments using advanced tensor train solver for multidimensional Fokker-Planck equation. We provide additional theoretical evidence for the case of multivariate normal distributions.

Published as a conference paper at ICLR 2023
UNDERSTANDING DDPM LATENT CODES THROUGH OPTIMAL TRANSPORT
Valentin Khrulkov
Gleb Ryzhakov & Andrei Chertkov
Yandex
Skolkovo Institute of Science and Technology Moscow, Russia
Moscow, Russia
khrulkov. Chrulkov.v@@mal 1l. com
{a. chertkov, 9. ryw.haaorresooottee.... ru
Ivan Oseledets
Skolkovo Institute of Science and Technology and AIRI
Moscow, Russia
oseleeele wew.eeettteelleeeooe.e..ee tecc. ru
ABSTRACT
Diffusion models have recently outperformed alternative approaches model the distribution of natural images. Such diffusion models allow for deterministic sampling via the probability flow ODE, giving rise to latent space and an encoder map. While having important practical applications, such as the estimation of the likelihood, the theoretical properties of this map are not yet fully understood. In the present work, we partially address this question for the popular case of the VP-SDE (DDPM) approach. We show that, perhaps surprisingly, the DDPM encoder map coincides with the optimal transport map for common distributions; support this hypothesis by extensive numerical experiments using advanced tensor train solver for multidimensional Fokker-Plane equation. We provide additional theoretical evidence for the case of multivariate normal distributions.
INTRODUCTION
Denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020) have recently outperformed alternative approaches model the distribution of natural images both in the realism of individual samples and their diversity (Dhariwal & Nichol, 2021). These advantages of diffusion models are successfully exploited applications, such as colorization (Song et al., 2021b), inpainting (Song et al., 2021b), super- resolution (Saharia et al., 2021; Li et al., 2022), and semantic editing (Meng et al., 2021), where DDPM often achieve more impressive results compared to GANs.
One crucial feature of diffusion models is the existence of deterministic invertible mapping from the data distribution to the limiting distribution of the diffusion process, commonly being a standard normal distribution. This approach termed denoising diffusion implicit model (DDIM) (Song et al., 2021a), or the probability flow ODE in the continuous model (Song et al., 2021b), allows invert real images easily, perform data manipulations, obtain a uniquely identifiable encoding, as well as to compute exact likelihoods. Despite these appealing features, not much is known about the actual mathematical properties of the encoder map and corresponding latent codes, which is the question we address in this work. Concretely, in this paper, we show that for the DDPM diffusion process, based on the Variance Preserving (VP) SDE (Song et al., 2021b), this encoder map with a large numerical accuracy coincides with the Monge optimal transport map between the data distribution and the standard normal distribution. We provide extensive empirical evidence on controlled synthetic examples and real datasets and give proof of equality for the case of multivariate normal distributions. Our findings suggest a complete description of the encoder map and an intuitive approach understanding the 'structure' of latent code for DDPMs trained visual data. In this case, the pixel- based Euclidean distance corresponds to high-level texture and color-level similarity, directly observed on real DDPMs. To summarize, the contributions of our paper are:

---

Published as a conference paper at ICLR 2023
1. We theoretically verify that for the case of multivariate normal distributions the Monge optimal transport map coincides with the DDPM encoder map.
2. We study the DDPM encoder map by numerically solving the Bokker-Plancee equation on a large class of synthetic distributions and show that the equality holds up negligible errors.
3. We provide additional qualitative empirical evidence supporting our hypothesis real image datasets.
2 REMINDER ON DIFFUSION MODELS
We start by recalling various concepts from the theory of diffusion models and stochastic differential equations (SDEs).
2.1 DENOISING DIFFUSION PROBABILISTIC MODELS
Denoising diffuusion probabilistic models (DDPMs) a class of generative models recently shown to obtain excellent performance on the task of image synthesis (Dhariwal & Nichol, 2021; Ho et al., 2020: Song et al., 2021b). We start with forward (non pora-pmmmttric) diffusion which gradually adds noise to data, transforming it into Gaussian distribution. Formally, we specify the transitions probabilities as
g( == N(z; / B,
(1)
for some fixed variance schedule BI,..., Bt. Importantly, noisy sample can be obtained directly from the clean sample Tg as Xt = vā,z0 + vI - ā,z, with ~ NO, I) and ar == Br. ag = II Cg.. The generative model then learns to reverse this process and thus gradually produce realistic samples from noise. Specifically, DDPM learns parameterized Gaussian transitions:
po(2)_1(2)) = N (2( o1).
(2)
In practice, rather than predicting the mean of the distribution in Equation (2), the noise predictor network ea (æt, t) predicts the noise component from the sample x, and the step f; the mean is then linear combination of this noise component and Σt. The covariances d be either fixed or learned as well; the latter was shown to improve the quality of models (Nichol & Dhariwal, 2021). 1). Interestingly, the noise predictor c0,,,, is tightly related to the score function (Stein, 1972; Liu et al., 2016; Gorham, 2017) of the intermediate distributions; specifically by defining sa(s,, () as
s0 t) = co co( t)
(3)
de
we obtain that Sp- (I+.t) ~ V, lo (x,), with p. (x,) being the density of the target distribution after steps of the diffusion process and 0* are the parameters at convergence.
Stochastic and deterministic sampling. pressed as
Given the DDPM model, the generative process is ex-
1
ai co (Izz () + 0.2, V1
x--1 Vac
with the initial sample or (). and (). The sampling procedure is stochastic, and no single 'latent space" exists. The authors of Song et al. (2021a) proposed deterministic approach to produce samples from the target distribution, termed DDIM (denoising diffusion implicit model). Importantly, this approach does not require retraining DDPM and only changes the sampling algorithm; the obtained marginal probability distributions Pt are equal those produced by the stochastic sampling. It takes the following form:
Cree Crea (1),1) (reo(s)..)) + V1 ā, co(2,, t).
24-1 0,-1
(4)
Va
By utilizing DDIM, obtain concept of latent space and encoder for diffusion models, since the only input to the generative model now is Fr ~ (O I) for sufficiently large T. In the next section, will see how it is defined in the continuous setup.
2

---

Published as a conference paper at ICLR 2023
2.2 SDE VIEW ON DIFFUSION MODELS
The authors of Meng et al. (2021) proposed to view diffusion models as a discretization of certain stochastic differential equations. SDEs generalize standard ordinary differential equations (ODEs) by injecting random noise into dynamies. Specifically, the diffusion process specified by Equation (1) is a discretization of the following SDE:
dx 3(t)xdt + v B(t)dw.
(5)
where da is an increment of x over the infinitesimal time step dt. Here, w represents a Brownian motion process, du can be intuitively understood DDPMs, an initial distribution with density po evolves into a standard normal distribution under this SDE. Various DDPM algorithms can be seen as discretizations of SDE-basee algorithms. For instance, the continuous analogue of DDIM sampling (4) is constructed in the following way. The following (deterministic) ODE, termed probability flow ODE (Song al., 2021b)
infinitesimal Gaussian noise. Similar to
3(t) 2
dr
Jxr V. loggt())) dl,
(6)
results in the same marginal distributions Pt as given by the SDE (5). Here, the score function Ve lolgogP(( (x()) is similarly approximated via neural network during training. If reverse this ODE, we obtain a mapping from the limiting distribution to the data distribution. It turns out that the DDIM sampling is a discretization of this continuous approach (Song al., 2021b).
The well-developed theory of SDEs provides us with rigorous mathematical apparatus allowing us study the properties of diffusion models at hand. We will utilize the following equation termed the forward Kolmogorov, or Fokker-Planck equation (Risken, 1996). This equation determines how a density po evolves under a (general) SDE. For (5) takes the following form:
Öp:
3(t)
[V, (xpp) Vipi]
Öt
Note, that (()) can be removed from (7) by change of variables dt = 2 dt, resulting in the PDE of the form
2
Op: V. (xpp) + V
(8)
Ōt
Without loss of generality, we will work with this simplified equation since the specific form of (()) does not affect the results. In this case, the probability flow ODE takes the following form:
dx = - ( + V log (x)) dt.
Encoder map. As mentioned above, by considering the probability flow ODE (6), can obtain latent codes for samples from the original distribution. For a given distribution Po and timestep t, let us denote the flow generated by this vector field as Exo (t, .). Le., a point x ~ μi mapped to Fo (f. 2) when transported along the vector field for a time t. The "final" encoding map is obtained when 0c, i.e.
Eno (x) = lim (t,
t-sac
Note that ETP, implicitly depends on all the intermediate densities μt obtained from the diffusion process (or the Plakkr equation).
DDPM ENCODER AND OPTIMAL TRANSPORT
By construction and properties of the phase flow, the map F,,. transforms the original distribution No into the standard normal distribution π == Moo NN0 7). Our further analysis investigates perhaps surprising hypothesis that this map is very close to the optimal transport map. How to arrive at this idea? A priori, it seems nontrivial that the theory of optimal transportation is related to the Fokker-Planck equation and diffusion processes. Some intuition may be obtained from the Otto calculus (Otto, 1996; 2001; Villani, 2009). It presents alternative view of the solutions of the BokkerrPlanck-Ae PDEs and diffusion equations in the following manner. It can shown that the trajectory (m, (2 obtained by solving the Fokker-Planck equation (or equivalent diffusion SDE),
3

---

Published as a conference paper at ICLR 2023
Figure I: A toy example of our approach on 2d multivariate normal distribution μ.. The first four plots visualize the diffusion process. The fifth plot demonstrates the trajectories of the probability flow ODE. In this case, the optimal transport map is known analytically and exactly coincides with the mapping Egs introduced in Section 3.2. Note that the trajectories of the probability flow ODE not straight lines even in this simple case.
is, in fact, the gradient flow of a certain functional in the Wasserstein space of probability measures. Moreover, locally, optimal transport maps between two 'consecutive' densities (i.e., separated by an infinitesimal time interval) along this trajectory are given precisely by the flow of the ODE (6). Thus, this trajectory is generally a sequence of infinitesimal optimal transports, which may "global' optimal transport map. However, our experiments suggested that in the case when the target density (0, I) as DDPMs, the map is practically indistinguishable from the optimal transport map. We start with toy example and then proceed to more general cases.
result in the
3.1 TOY EXAMPLE
One of the few cases when both optimal transport and diffusion equations can be solved explicitly the case of multivariate Gaussian distributions. For simplicity, in this example, we consider 840 (a, I) with E R" and I E R=x* being the identity matrix. Rather than utilizing
Equation (8) to obtain intermediate density values, we can do directly from transition formulas
for SDE (5) found in Song al. (2021b); Särkkä & Solin (2019); the corresponding probabilities then take the form:
14 ~ N.ae
(10)
The probability flow ODE takes the form de -te Solving this ODE amounts to simply integrating the function in the right-hand side, and the limit > do we obtain Epo a. Note that this is exactly the optimal transport map between Ho and I).
de
3.2 REMINDER ON OPTIMAL TRANSPORT
We now briefly recall the main definitions of the optimal transport theory. For thorough review of optimal transport, we refer the reader to classical texts such as Thorpe (2018); Villani (2009); Ambrosio et al. (2005).
Kantorovich and Monge formulation. Suppose have two random variables with densities μ and y supported on spaces X and Y respectively (in our work, we only consider distributions in Fuclidean spaces). We consider the optimal transport problem between Jı and v. There several possible formulations this task, namely the Kantorovich formulation and Monge formulation. Under quite general assumptions, these two formulations are equivalent. We will start from the Kantorovich formulation. In this formulation, we consider densities 7 E P(X x Y) such that their marginal densities are μ and v respectively. Such set is denoted by I1(,,,)) v) and its elements are termed transport plans. The Kantorovich formulation of the optimal transport problem is finding the optimal transport plan:
I c(x (1),,)) (o.dH,y
min
(11)
7EH1(..)))
where c(x, y) is a cost function. Under quite general assumptions on μ. V. there exists a unique solution to the Kantorovich problem. Rather than working with transport plans, it is more convenient to work with transport maps, which provide the means to actually move one distribution to another.
In this formulation (termed the Monge formulation), goal is to find transport map S X Y
4

---

Published as a conference paper at ICLR 2023
such that
/ clz, ((z))u(z)dd + inf,
(12)
over all ju-meauuaalle transport maps S such that Stft. Here, S4 y is the push-forward measure, i.e. (A) - ( for all measurable sets A.
Importantly, for non-atomic measures, these two definitions are equivalent (Villani, 2009). Given an optimal transport map. it can be straightforwardly converted to the optimal transport plan. Such a plan is deterministic, as we know precisely where to move each point. In this work, we consider the quadratic cost function, i.e., c(x, v) || vll as commonly done in the study of diffusion equations and gradient flows in Wasserstein spaces Villani (2009). In this case, the distance between distributions is denoted (2(-..) :) and is given by the square root of the optimal transport cost (11). In the appendix, we also discuss the Benamou- BBrenier formulation of optimal transport.
3.3 MULTIVARIATE NORMAL DISTRIBUTION CASE
We start by showing theoretically that for arbitrary multivariate normal distributions, the DDPM encoder map coincides with the optimal transport map. This map is known analytically for normal distributions in the case of the quadratic cost function (Givens & Shortt, 1984).
Theorem 3.1. Let Po ~ ((a(0), E(O)) be a multivariate normal distribution. Then E is the Monge optimal transport map between po and N(O 1), i.e.,
Epp (x) = E(O) ((()))))
See the appendix for the proof.
Discussion. We show that for multivariate normal distributions, the probability flow induced by the DDPM diffusion reduces to the optimal transport map in the limit. Despite the apparent simplicity of this case, the proof requires a non-trivial argument about the probability flow ODE, Thus, it is hard to attribute this result to mere coincidence. We hypothesize that the statement of Theorem 3.1 is valid, perhaps, approximately, for arbitrary distributions (or at least a very large class of distributions), but defer precise theoretical analysis of this statement future work. For now, thoroughly evaluate this hypothesis on a large class of synthetic and real distributions and show that holds with high numerical precision. Very recently, in Lavenant & Santambrogio (2022) appeared counterexample to the above statement with the two maps not being exactly equal (at single point). However, our further analysis suggests that when given finite number of samples in higher dimensions, the difference is still extremely low.
3.4 NUMERICAL EXPERIMENTS ON SYNTHETIC DATA
In this section, we numerically evaluate the aforementioned optimal transport map hypothesis. The main computational difficulty is the solution of the multidimensional Fokker-Planck equation (8). To maintain accuracy in traditional discretization-base numerical methods, the number degrees of freedom of the approximation, i.e., the number of unknowns, grows exponentially the di- mensionality of the underlying state-space increases. recent years, low-rank (low (lw-parametric) tensor approximations have become especially popular for solving multidimensional problems in various fields of knowledge (Cichocki et al., 2016; 2017). Recently proposed approach (Chertkov & Oseledets, 2021) for the solution of the Fokker-Planee equation utilizing low-rank tensor train (TT) format (Oseledets, 2011), I), Chebyshev interpolation, splitting and spectral differentiation techniques, allowing us consider sufficiently fine grids.
Method. We start by numerically solving the Pokker-Planck equation for an initial density Po(z) = p(z, 0). According to approach (Chertkov Oseledets, 2021), Equation (8) is discretized on tensor- product Chebyshev grid with N nodes for each dimension, and the density at each time step represented as a d-dimensional tensor (array) P. This tensor approximated in the low-rank TT-format
R
Pln, n2, nd]
Σ
ra, nd, 1}, i
(13)
5

---

Published as a conference paper at ICLR 2023
where n4 1.2....,N NN (k = 1.2....... ..,.....)) represent the multi-index, three-dimensional tensors Gk E RR 1xNx are named TT-cores, and integers Ro, R1 Ri..... ......,,R Ra (with convention Rg = Rg = 1) are named TT-ranks. Storage of the TT-cores G1, .2.....,.. Ga requires less equal than x N x maxi Ri memory cells instead N cells for the uncompressed tensor, and hence the ITTdecomposiiion is free from the curse of dimensionality if the TT-ranks are bounded. Thus, each time step m = 1, 2, fman, we obtain discrete values of the density p(z, mh), represented in the compact TT-format in Equation (13). In the appendix, we describe the algorithm in more details.
....,
A
Next, we solve the probability flow ODE from Equation (9). We sample a set of points, termed Xu, from the original density (=( (since the initial density is presented in the TT-format, we use a specialized method for sampling from (Dolgov & Scheichl, 2019)). Then we numerically solve the probability flow ODE with the Runge-Kull method (De Vries & Wolf, 1994) for each point (we calculate the density logarithm gradient using the spectral differentiation methods in the TT-format from (Chertkov Oseledets, 2021)). The resulting set denoted as X1-
Given these two point clouds, Xo and X1, we numerically compute the optimal transport cost with the Python Opt imal Transport (POT) library (Flamary et al., 2021). We then compute the transport cost for the map sending a point z; E Xo to its counterpart Ex(x2) E X11 Our hypothesis suggests that these two costs will be equal to a reasonable precision.
Distributions. We consider d-dimensional distributions specified by the following set of parameters: a1 E RR, 42 E R2, Q1 E Raxd Q2 E Rdxd, with Q1 and Q2 being symmetric positive definite. Given these parameters, we set the density p(z) (with x E RR)) by the following formula
1 z exp (z) (2))))())))))
p(x)
@ (1:) (x ai) ) (x (2(x) ((x ((x)) Q2(x a2)²
02)
where the square in 92 is understood elementwise (so do is a polynomial of degree 4) and Z is normalizing constant obtained by numerical integration. We then consider uniform mixtures of up to 5 of such randomly generated distributions, i.c.,
K
1 K P((), K~ X~Uniform(55)
plz)
(14)
with Pt being as above. Some examples for the 2-dimensional case (i.e., d = 2) are visualized at Figure 2. We see that the distributions are quite diverse and lack any symmetry.
We use the distribution from Equation (14) for the and 3-dimensional case. If d > 3, the number of parameters in the TT-representation of this distribution turns out to be too large, so we use a density of a simpler form
exp exp ||0:|
p(x) 7 7g(=)= (2T) 4
(15)
where Z is a normalizing constant obtained by numerical integration in the TT-format and g(x) random function in the TT-format with positive TT-cores (each element of the TT-cores was sampled from (1,0 1)) and of rank 2. Since the Gaussian density has rank 1 with variables being completely separated, the above formula provides us with a low-rank fast decaying density suitable for experiments.
Experiment and results. We consider three dimension numbers d = 2, 3, 7. For the case d = and d = 3 we use the density from Equation (14) and for the case d = 7 we use the density from Equation (15). As the spatial domain, we consider the square [-8, 8]d, and densities were scaled beforehand decay sufficiently fast towards the boundary. For the time dimension, the range [0, 5]. We construct 100 random densities for each d and compute two transport costs as described above. Some examples of the obtained encoder maps for the 2-dimensional case are given at Figure 3. The trajectories of the probability flow ODE are highly nonlinear, despite providing the optimal transport map in the limit.
consider
The obtained results are presented at Table 1. Here, the error is defined as
(p) = Cost(E,) - COs)
Cost(OT)

---

Published as a conference paper at ICLR 2023
x- Eg10 x10.11
Figure 2: example 2- dimensional distributions considered for numerical comparison the Monge optimal transport map against the DDPM encoder map.
Figure 3: An example of the trajectories of the proba- bility flow ODE and the limiting encoder map for two 2-dimensional distributions studied Section 3.4.
Table 1: Numerical results for synthetic data different dimensions.
Dimensionality d
Spatial grid size
Temporal grid size
Maximum emror (rat(p)
2
250
250
5.7 10
3
100
100
2.2 10-15
7
50
2.1 10-15
and in the table we report the obtained maximum value of the enror for 100 random densities. For all three considered settings, we have a very high (up to machine precision) accuracy, and we conclude that this experiment fully supported our hypothesis. Note that these calculations were carried out on a regular laptop and run for one random density took about 100 seconds on average for each model this experiment for the counterexample provided in Lavenant &
problem. We have additionally Santambrogio (2022), again obtaining a machine precision level difference, which is not surprising since the difference between maps is shown to hold only (0, 0). This suggests that constructing a 'numerical' counterexample is nontrivial task. The code is available in the supplementary material.
We now attempt to verify it on more qualitative level for high-dimensional distributions, where solving the Fokker-Plaan equation directly is infeasible. Namely, we consider DDPMs trained on high-resolution image data.
3.5 EXPERIMENTS OF IMAGE DATASETS
Our experiments in this section are organized follows. Suppose that are given two datasets and train DDPM for each dataset. Then, by the hypothesis, each DDPM encoder performs optimal transport from respective distributions to (0, Since the optimal transport in setting is performed the L2 sense, for images, this will be reflected by similarity the pixel space, i.c., high- level texture similarity. We can perform the following experiment see if this indeed holds. Take latent code from (00, I), and compute a corresponding image from each distribution respectively by reversing the encoder map. We now compare if these images share texture and a high level semantic structure. Indeed, if each of those images is L2-close 'on average" to the same latent code, expect them to also be similar the pixel level. Additionally, on datasets of relatively small sizes, we again can compare the directly computed OT map in the pixel space with the DDPM encoder map.
We consider the AFHQ animal dataset (Choi et al., 2020). It consists of 15000 images split into 3 categories: cat, dog, and wild. This a common benchmark for image-to-ing to-iiage methods. We also verify our theory the FFHQ dataset of 70.000 human faces (Karras et al., 2019) and the
Datasets.
7

---

Published as a conference paper at ICLR 2023
MetFaces dataset (Karras et al., 2020) consisting of 1000 human portraits. Finally, consider conditional DDPM on the ImageNet dataset (Deng et al., 2009). By changing the conditioning label, control what distribution is being produced, and the argument above still holds.
Models. We consider guided-diftusion, state-of-the-arl DDPMs (Dhariwal & Nichol, 2021). We the official implementation available at github For each of the datasets we train a separate DDPM model with the same config utilized for the LSUN datasets in Dhariwal & Nichol (2021) (with dropout); we use default 1000 timesteps for sampling. The AFHQ models were trained for 3 10 steps, the FFHQ model was trained for 10t steps; for the MetFaces model, we finetune the FFHQ checkpoint for 25 10 steps similar Choi et al. (2021), 1). All models were trained on 256 x 256 resolution. For the ImageNet experiment, we utilize the 256 resolution checkpoint for the conditional model available at the GitHub link above.
Algorithm. We follow the aforementioned experimental setup: sample number of latent codes from (0, f) in the pixel space (I2256 algorithm. For the ImageNet, we have single DDPM model but vary the conditioning label.
256 x 256x3) and decode them by different DDPMs with the DDIM
Figure 4: Examples of synthetic samples produced with DDIM sampling from the same latent codes. (Top) Three independent DDPMs trained on AFHQ Dog/ Cat/Wild. Each row translates into each row most cases preserving high-level semanties such pose and texture, thus supporting the claim of our hypothesis. (Bottom) Two DDPMs trained on FFHQ/MetFaces; high-level features such as gender and texture/color are transferred. 'Noisy' samples seem also be shared across models.
Qualitative results. We fix a number of latent codes and produce images by all the AFHQ models. The obtained samples are visualized on Figure (top). We observe that samples from different rows indeed share high-level features such as texture and pose. Similarly, we visualize samples from FFHQ and MetFaces on Figure (bottom). We note that samples are aligned this case as well; notably, in most cases, pose, gender, and various features such as hairstyle and mustache are transferred. Interestingly, artifacts of models such as slight noisiness also seem to be shared.
Results obtained with the ImageNet model are provided at Figure 5. We selected a number of diverse classes, namely, rooster, pizza, crab, corgi. We sampled a number of latent codes and produced the respective label. We see that in this case, high-level textures
images by conditioning transferred as well, sometimes in quite amusing ways (e.g.. rooster/pizza rows). As a sanity check, we performed similar experiment with the conditional BigGAN model (Brock al., 2019) pretrained
'https://g" thub. com/openai /gui ded-di ouustion
8

---

Published as a conference paper at ICLR 2023
on ImageNet. We did not observe similar behavior, suggesting that the texture alignment is artifact specific to DDPM and not happening simply due to the conditioning mechanism.
Numerical results. We now apply the evaluation scheme of Section 3.4 to the AFHQ dataset. We take the validation subset of each part of the dataset (Cat/Dog/Will)) containing 500 images and invert them with DDIM. We then compute the optimal trans- port matrix between the images and their respective codes (understood as vectors POT. We then compared the obtained OT cost with the cost for the ODE induced map. In all three cases, they matched up to machine precision. Given the high dimensionality of the pixel space and low number of samples, the exact coincidence with the hypothesis is not surprising, since for the majority of synthetic experiments, there
x2256x266x with
exact match. Intuitively, this should become easier in high dimen- sions as we may trade some inaccuracies in the model with there being 'more space'.
To conclude, experiments on real image datasets sup- port our hypothesis and provide interesting examples of how latent codes can utilized to transfer images from one distribution to another via a "proxy' normal distribution.
Figure 5: Examples of synthetic samples pro- duced with DDIM sampling from same latent codes for different classes with a conditional ImageNet model. We observe that samples share texture and pixel- based similarity.
4 RELATED WORK
OT is a classical area mathematics tightly related to diffusion processes and PDEs. With hundreds of papers this topic, can not provide thorough review of all the literature and refer the reader to great texts such as Villani (2009); Ambrosio et al. (2005); Thorpe (2018). Computational approaches to OT have been extensively studied by the machine learning communities (Cuturi, 2013; Altschuler et al., 2017; Lin et al., 2019). We also note the work (Onken et al., 2021) in which the relationship between OT and normalizing flows was considered, as well the work (Mokrov et al., 2021), where a method based on the Wasserstein gradient flows was proposed for solution of the Fokker-Plann equation. A recently introduced class of crore-based models termed diffusion Schrödinger bridges (Chen et al., 2016; De Bortoli et al., 2021; Gushchin et al., 2022) is tightly related the entropy-regulariin OT problem, providing continuous analog of the Sinkhorn algorithm. In our paper, we show connection between DDIM and regulaegulrrized) OT formulations, not covered by the Schrödinger bridge theory. As mentioned Section 3.2, a similar question was studied in Lavenant & Santambrogio (2022) providing a theoretical counterexample, while our analysis is mostly numerical and is motivated by the practical goal of understanding the DDPM encoder.
CONCLUSION
In this work, we provided theoretical and experimental evidence that due to the nature DDPMs, the encoder map turns out to be the optimal transport map. We believe that this result will be interesting to both optimal transport and diffusion models communities. We hope that will inspire other researchers to obtain general proof of this hypothesis. One observation about the obtained results is that the quadratic cost in the pixel space seems suboptimal when working with visual data. Perhaps, more elaborate diffusion process leading feature-based cost may be constructed. We leave this analysis to future work.
ACKNOWLEDGMENTS
The work was supported by the Analytical center under the RF Government (subsidy agreement 000000D73032 IP5Q0002, Grant No. 70-2021-00144 02. 11.201..

---

Published as a conference paper at ICLR 2023
REFERENCES
Jason Altschuler, Jonathan Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. Advances in neural information processing systems, 30, 2017. 9
Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures. Springer Science & Business Media, 2005. 4, 4
Jean-Davis Benamou and Yann Brenier. A computational fluid mechanics solution the Monge- Kantorovich mass transfer problem. Numerische Mathemarik, 84(3):375-393, 2000. 13
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
Yongxin Chen, Tryphon T Georgiou, and Michele Pavon. On the relation between optimal transport and schrödinger bridges: A stochastic control viewpoint. Journal of Optimization Theory and Applications, 169(2):671-6911 2016.
Andrei Chertkov and Ivan Oseledets. Solution of the Fokker-Planck Equation by Cross Approx- imation Method in the Tensor Train Format. Frontiers in Artificial Intelligence. 4, 2021. doi:
10338///rai.221166688888 5, 6, 15
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Conditioning method for denoising diffusion probabilistic models. Proceedings of the IEEE international conference on computer vision, 2021.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8188-8199 2020. 7
Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, and Danilo Mandic. Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions. Foundations and Trends in Machine Learning, 9(4-5):249 2016.
Andrzej Cichocki, Anh Phan, Qibin Zhao, Namgil Lee, Ivan Oseledets, Masashi Sugiyama, and Danilo Mandic. Tensor networks for dimensionality reduction and large-scale optimization: Part applications and future perspectives. Foundations and Trends in Machine Learning, (6):431-673, 2017. 5
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34, 2021. 9
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Paul L DeVries and Robert Wolf. first course in computational physics. Computers in Physics, (2):178-179, 1994. 6
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusson models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. 1,
Sergey Dolgov and Robert Scheichl. A hybrid alternating least -ttcross algorithm for parametric pdes. SIAM/ASA Journal on Uncertainty Quantification, 7(1):260-291, 2019. 6
DC Dowson and BV666017 Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 2(3)):50-455 1982. 14
10

---

Published as a conference paper at ICLR 2023
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalic T.H. Gayraud, Hicham Janati, Alain Rakotomamoniy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. POT: Python Optimal Transport. Journal of Machine Learning Research, 22(78): 1-8, 2021. 6
Clark Givens and Rae Shortt. A class of Wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31(2):3312440, 1984. 5
Roland Glowinski, Stanley J Osher, and Wotao Yin. Splitting methods in communication, imaging,
science, and engineering, Springer, 2017. 15
Jackson Gorham, Measuring sample quality with Stein's method. Stanford University, 2017.
Nikita Gushchin, Alexander Kolesov, Alexander Korotin, Dmitry Vetrov, and Eygeny Burnaev. Entropic neural optimal transport via diffusion processes. arXiv preprint arXiv:2211.01 150,,,,,, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017. 14
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 1840-6851, Curran Associates, Inc., 2020. 1,
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401 1401-4410, 2019. 7
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NeurIPS, 2020. 8
Hugo Lavenant and Filippo Santambrogio. The flow map of the fokker fokker-planck equation does not provide optimal transport. Applied Mathematics Letters, 133:108225, 2022. ISSN 0893-9659. doi: https://doi.org/ http:///dooorr/10... 10. 101 101 a/j.amm..m.22. 2022. 108225. 5, 9.
Haoying Li, Yifan Yang. Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. SRDiff: Single Image Super-resolution with Diffusion Probabilistic Models. Neurocomputing, 2022. ISSN 0225-2312. doi: https://doi.org/10 n0006/..eucom.m022.00 2022.01..22222
Tianyi Lin, Nhat Ho, and Michael Jordan. On efficient optimal transport: An analysis of greedy and accelerated mirror descent algorithms. International Conference on Machine Learning, pp- 3982-3991. PMLR, 2019. 9
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized Stein discrepancy for goodness-of-fit tests. In International conference machine learning, pp. 276-284, PMLR, 2016. 2
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXhv:2108.01073 2021. 1,3 3
Petr Mokrov, Alexander Korotin, Lingxiao Li, Aude Genevay, Justin M Solomon, and Evgeny Burnaey. Large-scale wasserstein gradient flows. Advances in Neural Information Processing Systems, 14:15233-15556, 2021, 9
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. ICML, 2021.
Ingram Olkin and Friedrich Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications, 48:257-263, 1982. 14

---

Published as a conference paper at ICLR 2023
Derek Onken, Samy Wu Fung, Xingjian Li, and Lars Ruthotto. Ot-flow: Fast and accurate continuous normalizing flows via optimal transport. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9223-92322 2021. 9
Ivan Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295- 2317, 20
Ivan Oseledets and Eugene Tyrtyshnikov. Tt-cross approximation for multidimensional arrays. Linear Algebra and its Applications, 432(1):70-88) 2010. 15
Felix Otto. Double degenerate diffusion equations as steepest descent. Citeseer, 1996.
Felix Otto. The geometry of dissipative evolution equations: the porous medium equation. 2001. 3
Hannes Risken. Fokker-Planck equation. In The Fokker-Planck Equation. Springer, 1996. 3
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. 2021. 1
Simo Särkkä and Arn Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019. 4, 13
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 1256-2265 PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference Learning Representations, 2021a. 1,2
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021b. 2, 3, 13
Charles Stein. A bound for the error in the normal approximation to the distribution of sum of dependent random variables. In Proceedings of the sixth Berkeley symposium on mathematical statistics and probability, volume 2: Probability theory, volume 6, pp. 583-603. University of California Press, 1972. 2
Matthew Thorpe. Introduction to optimal transport, 2018. 4,
Lloyd N Trefethen. Spectral methods in MATLAB, volume 10. Siam, 2000. 15
Cedric Villani. Optimal transportation, dissipative pde's and functional inequalities. In Optimal transportation and applications, pp. 53-89. Springer, 2003. 13
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009. 3, 4, 5,
12

---

Published as a conference paper at ICLR 2023
A BENAMOU-BRENIER FORMULATION OF OPTIMAL TRANSPORT
It will be instructive to consider another formulation of the optimal transport, originating in spirit from the fluid dynamies (Benamou & Brenier, 2000; Villani, 2003). Assume that 0 we are given set of "particle'' from the density Po which move in such way that at t = their state described by the density Pl. Moreover, these particles move in such that they perform the least amount of work. Formally, they minimize the following action:
A= I Σ [1E()) dt,
with varying in the set of particles. In the continuous limit, we obtain the following formulation (with 24 being the velocity field):
+ Opr + V. (peve) 0f S (a((s) (o(()))n(6)) dodl; P.u.
inf
(16)
at
where the infimum is taken over all time-dependent probability densities {pt Hi= which agree with Pn and #1 at respective times i 0 and 1, and overall time-dependent velocity fields {vr (r))
which convect Pt. as expressed by the continuity equation in the right-hand side of Equation (16).
It can be shown that the infimum in Equation (16) is given by W2(Po. p1) Moreover, given optimal transport map the corresponding admissible trajectory for the Benamou-Brentn formulation can be easily constructed. Importantly, trajectories generated by this vector field are straight lines, which does not hold for the probability flow ODE (see, e.g., Figure 1). There is no contradiction here: do not claim that the probability flow ODE the solution of the manamou-Brenice problem, but only that the flow map induced by this ODE (in the limit t - oc) solves the Monge problem. So, in general, the vector field solving (16) and the vector field of the probability flow ODE will be different.
B PROOF OF THEOREM 3.1
The proof of Theorem 3.1 based on the fact that in the case of μ4 being normal distribution, all the intermediate densities μę are normal as well, i.e, H4 ~ (a((t), E(t)). Moreover, we can easily obtain ODEs describing how their means and covariances transform (Song al., 2021b; Särkkä & Solin, 2019). These ODEs take the following form:
dE(t) dt
da(t) -a(1)
-()); = 2(T - E(I);
dt
with the solutions
((t = 1 (2(00) 1); a(I) Ea()
Let us start with the homogeneous case (0)) 0, so at) 0 (we provide the proof of the general case below). The probability flow ODE then takes the following form:
dr = - di
(17)
We start by noticing that all the matrices [t) commute with each other. We then can write the solution of Equation (20) as:
(t)) = exp (-
with exp being the matrix exponential. Let consider the cigendecomposition of (0)), i.c., assume that E() = T AU with = diag(Xi,...,, being a diagonal matrix of positive eigenvalues and
UTU I. We obtain that
(1) = U diag(J 1(2, 1(aa ()),,
(18)
13

---

Published as a conference paper at ICLR 2023
where f(A,t) is function defined as
f(A,t) exp (-fo (1
where now all the functions operate on real numbers. Direct integration shows that:
f(A,t) exp
- log (A - 1 c²²))
log A
2
2
(19)
1 + 22t 21 ( 1) +1
de
X
We see that lim [(A, 1) = X-1 Then, by definition of the matrix square root, Eno (2) = 1/0 This is exactly the optimal transport map between and N(0, I).
For general case, we obtain the following ODE.
dx dt -I
(20)
This is an inhomogeneous ODE. We can obtain its solution via the following standard approach. Let (t) be the fundamental solution of the homogeneous part (see Equation (18)), i.e,
(t) U diag(f (A2, f.l..........ffff...
with
( 1
f(2,t)
X
We find that a particular solution (with 0 initial condition) of the inhomogeneous problem is given by:
a(t) -tt) / (T)))=)(())(())
Recall that
1 - 1); a(t) = a(()). (t) = 1 +e 222(0)
We obtain:
a(t) = (Z(() 1'' (T) (T)E dd
dr a(0).
Again, it suffices to work on the level of the spectrum. L.e.
a(t) U ()) 1). .... 9( ())(
with
E g(A,t) t((,t) Jo (1+ 1)){(,)) dr
f(A,t) L e-
(3(2TT) dr
(21)
-V S' (e-2r(X (A - 1) 1) dr.
6-
Direct integration shows that:
lim g(A,t) = Vx ++00
Thus, we obtain that
lim a(1) - -')a)))))(
By combining this piece with the limit of general solution obtained above, we find that
E (z) = (2(00)-/) - =(00->/aa(0) = =-²2200 a()),
which is exactly the desired optimal transport map between (4y and AO 1) (Dowson & Landau, 1982; Olkin & Pukelsheim, 1982). This expression is also familiar from the commonly used Frechét Inception Distance metric, where the OT distance is used to evaluate generative models (Housel et al., 2017).
14

---

Published as a conference paper at ICLR 2023
C TENSOR-TRAIN SOLVER FOR THE BORKER-PLANCE EQUATION
Let us briefly summarize the algorithm from (Chertkov & Oseledets, 2021) for solving the Fokker- Planck equation. If we introduce diffusion and convection operators from Equation (8)
Vp(æ, t) = V2p(z, t), Wp(z, t) = V (.p(z, f)),
then on each time step m (m 1, ..,, M) the standard second order operator splitting technique (Glowinski et al., 2017) may be formulated
Pm+1 (V++W) Pm e hW A
Pms
where h is the size of the temporal grid and pm(()) p(x, mh) is a solution on the m-th time step. It equivalent the sequential solution of the following equations
dyp ViPD (1)
(1)
(. (,,,m))= = pm)());
(22)
Ōt
Ōpc (x(pc)) po(.,tm) = =pC 0 (, tm + h 2
(23)
0t
0rp (2)
= V.rD
PD (2)
(,, =Po( mm + h),
(24)
Øt
with the final approximation of the solution Pm++1 = ( Im , -)
To solve the diffusion part (see Equations (22) and (24)), we discretize the Laplace operator using the second order Chebyshev differential matrices (Trefethen, 2000), and convolve the matrix exponential with the TT-cores of pm. The convection part (see Equation (23)) is solved by efficient interpolation in the TT-format with the TT-cross method (Oseledets & Tyrtyshnikov, 2010). Finally, at each time step m, we obtain discrete values of the density p(z, mh), represented in the low rank TT-format as in Equation (13). Then we can solve the probability flow ODE, using these tensors, described in the main text.
15


Reviews:
[{"review_id": "ztkXcaa86T", "reviewer": "Reviewer_6Sgh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides theoretical and experimental support to the following hypothesis: maps induced by denoising diffusion processes are optimal transport maps between the base and target distributions. For the case of multidimensional Gaussians, the paper provides a complete analytical result. For the general case, it shows this directly for synthetic experiments with mixtures of Gaussian and indirectly, through consequences such as texture preservation, for image data.", "summary_of_the_review": "The paper gives compelling evidence that links denoising diffusion processes to optimal transport maps from base to target distributions. This is done clearly and methodically. This may form the basis of a better understanding of the remarkable performance of this approach to generating natural images and of furthering its use in novel applications. Apart from a couple of clarifications, the paper seems worthwhile to share with the community.", "strength_and_weaknesses": "### Strengths\n* Theoretical understanding of denoising diffusion processes is important to bring their impressive qualitative performance with images to other applications. As such, the characterization put forth by this paper can give a good starting point to understanding the nature of the resulting maps.\n* The paper is written well, with good overview of the background material, a clear motivation, and directly appreciable results.\n\n### Weaknesses\n* The analytical result is very limited, as it pertains to mapping from Gaussians to Gaussians. At some level it is also unsurprising, considering that Gauss-Markov processes do have energy-optimality properties. This doesn’t mean that the result is trivial, but it begs for additional complementation with empirical results to support the hypothesis.\n* There is a little bit of ambiguity when moving from the discrete processes to their corresponding continuous versions. For example, the difference equation (4) is a backward process, but the corresponding differential equation (6) appears to be a forward one. It is worth saying something about this to ease reading for those who are not immersed in this area.\n* The indirect evidence based on texture similarity in the image data experiments is a very compelling approach. However, it is worth discussing why other reasonable explanations may not work. For example, if the neural network is regularized in all cases to be a Lipschitz map, then the texture similarity could be simply by virtue of that limited distortive ability instead of some global form of transport optimality (though, admittedly, these two concepts could be related.)\n* The numerical results paragraph in the image data experiments (p.9) is unclear. To have the analogue of Section 3.4, I would have expected computing the cost of the Monge map for these images, but it doesn’t seem that’s what is done. A clarification here is necessary.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mo6KEB2F95w", "reviewer": "Reviewer_R2Hy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper claims that diffusion-based models actually implement transportation plans between target and Gaussian distributions. It comes with a very pedagogical introduction to both DDPM and optimal transport, and then the rest of the paper is spent doing experiments to show how this claim is verified on many examples. ", "summary_of_the_review": "The paper is a nice read, but I think it is a bit borderline, although I like it.\nI believe it could be improved in the following ways:\n* more references about DDPM and OT and their connections. (For instance you could mention diffusion based OT systems based on Wasserstein flows). \n* This improved state of the art may be done at the expense of putting the details of the  \"method\" part in appendix.\n* a much clearer (title + text) presentation of this paper as stating a conjecture or an hypothesis that DDPM actually construct transportation maps. (plans ?) ", "strength_and_weaknesses": "As a definite strength, the paper is very readable and a pleasure to read. Its claim is clearly stated, although I am not 100% sure it is actually novel. Still, the experiments are nice and quite convincing.\n\nAs a weakness, maybe I could say that the paper is a bit weak on the theory, which is not a no-go for me. What I feel could be done in a clearer way is to frame the paper as clearly stating an \"hypothesis\" or a \"conjecture\". This could help better understand its contribution, which I believe is pretty exciting and simple to grasp.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "avwztJVIfn", "reviewer": "Reviewer_CjpS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work hypothesizes that the limit maps produced by   DDPMs  coincide with the optimal transport maps, or they are very close to each other with high similarity. The theoretical proofs are given for the cases of the normal source distributions, and experimental results are given for more general distributions, such as the mixtures of uniform distributions. The experiments on real image data sets show that the images sharing the same latent code obtained by diffusion have the similar high level texture features, postures and semantics. Also, the L2 distance seems not suboptimal for visual data. The work bridges the optimal transport theory and the diffusion model. \n\nThe contributions are:\n\n1. theoretically verify that for the case of multivariate normal distributions the Monge optimal transport map coincides with the DDPM encoder map.\n\n2. Numerically verification of the similarity between the optimal transport maps and the DDPM encoder map by solving the Fokker-Planck equation on more general synthetic distributions.\n\n3. Additional qualitative empirical evidence supporting the hypothesis on real image datasets are provided.", "summary_of_the_review": "This work hypothesizes that the limit maps produced by   DDPMs  coincide with the optimal transport maps, or they are very close to each other with high similarity. The work theoretically verifies the hypothesis for the case of multivariate normal distributions the Monge optimal transport map coincides with the DDPM encoder map, gives the numerical verification of the similarity between the optimal transport maps and the DDPM encoder map by solving the Fokker-Planck equation on more general synthetic distributions, and offers additional qualitative empirical evidence supporting the hypothesis on real image datasets are provided.\n\nThe idea of bridging optimal transport theory and DDPM is inspiring and important. The mathematical proof is rigorous and elegant, the empirical experimental results are encouraging. The hypothesis can be further formulated to be more quantitively  precise. The mathematical proofs can be directly generalized to other well known distributions. More complicated synthesized distributions can be tested for the hypothesis. ", "strength_and_weaknesses": "The work has many strengths:\n\n1. It raised an interesting hypothesis which bridges DDPM and optimal transport theory, the conjecture is sensible, elegant and inspirational.\n2. Solid theoretical proofs are given for multivariate Gaussian distribution as the source input. Numerical experiments on synthetic data and empirical validations on real images are supporting.\n3. The writing is clear and easy to follow, the mathematical formulations are rigorous and succinct.\n\nThe weaknesses are \n1. The theoretic proofs can be further improved, in appendix between equation (19) and (20), the authors claims the solution is exaction the optimal transport map, the detailed deduction for this claim is missing. I expected to see more details to show this solution satisfies the Brenier Monge-Ampere equation.\n\n2. In the numerical experiment design, the authors simply used the POT library, which algorithm was applied ? Some algorithms are intrinsically approximation and will introduce approximation errors. This need to be carefully analyzed as well to ensure the precision of the experiments.\n\n3. The hypothesis is kind of vague, the DDPM solution and the optimal transport map are close with small deviation. More accurate quantitively estimate will be more convincing. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ot72eMiiYo", "reviewer": "Reviewer_qZHA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a hypothesis that the DDPM encoder map coincides with the optimal transport map for common distributions, and it is shown for Gaussians this is indeed the case. Empirical evaluations are done to support this hypothesis.\n", "summary_of_the_review": "Overall I think the paper lacks novelty and there are some evaluation choices that could be improved. Hence I'm leaning toward rejection.\n", "strength_and_weaknesses": "## Strengths:\n* Finding meaningful ways to interpret popular diffusion models like DDPM is a meaningful research direction\n* The paper presents a rigorous proof of the hypothesis for the Gaussian case\n* Experiments were done to compute the exact error between transport costs for low-dimensional synthetic examples as well as qualitatively for image generation tasks.\n\n\n## Weaknesses:\n* I found the novelty of this paper lacking as it does not introduce a new method or significantly new theoretical results.\n* For the Gaussian case, the proof becomes a lot simpler because all intermediate densities are also normal. And it looks like the authors further assume $a(0) = 0$ and I have not found a proof of a more general case in the supplementary materials contrary to what is mentioned in the appendix proof.\n* In Section 3.4, when computing the error between the optimal transport and the one induced by solving the Fokker-Planck equation numerically, I think it might be more useful to compare the difference of transport maps (that is, $\\sum_i ||T(x_i)-T^*(x_i)||^2$ where $T$ is the ODE induced map, and $T^*$ is the optimal transport map). It is possible that the error is very small but the transport maps can still be different.\n* I found the rationale of Section 3.5 a bit questionable, namely we can invert the encoder and just compare the inverted latent codes visually. To me, such evaluation only makes sense if the transport map is just translated by a constant vector. Have the authors tried with different random seeds (and hence different randomization of DDPM networks) when training models for a different dataset? Will the results still be consistent?\n* It was very briefly mentioned that recently Lavenant & Santambrogio found a counterexample to the central hypothesis. In my opinion, the paper needs to do a lot more in explaining whether such a counterexample is a singular case or not, other than mentioning the two maps disagree at a single point (how can two maps disagree at only one point if both maps are continuous enough?) and somehow \"finite number of samples in higher dimensions\" this is not a problem.\n* The numerical solver for the Fokker-Planck equation uses a low-rank TT format which I think might be foreign to the machine-learning community and deserves more explanations, at least in the appendix. For instance, the authors ought to elaborate on \"the standard second-order operator splitting technique\" that involves some exponentiation in time-stepping that I do not follow.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster