Non-Denoising Forward-Time Diffusions
Authors: Stefano Peluchetti
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
diffusion
SDE
generative modelling
DDPM

Abstract: The scope of this paper is generative modeling through diffusion processes.
An approach falling within this paradigm is the work of Song et al. (2021), which relies on a time-reversal argument to construct a diffusion process targeting the desired data distribution.
We show that the time-reversal argument, common to all denoising diffusion probabilistic modeling proposals, is not necessary.
We obtain diffusion processes targeting the desired data distribution by taking appropriate mixtures of diffusion bridges.
The resulting transport is exact by construction, allows for greater flexibility in choosing the dynamics of the underlying diffusion, and can be approximated by means of a neural network via novel training objectives.
We develop an unifying view of the drift adjustments corresponding to our and to time-reversal approaches and make use of this representation to inspect the inner workings of diffusion-based generative models.
Finally, we leverage on scalable simulation and inference techniques common in spatial statistics to move beyond fully factorial distributions in the underlying diffusion dynamics.
The methodological advances contained in this work contribute toward establishing a general framework for generative modeling based on diffusion processes.

Under review as a conference paper at ICLR 2022
NON-DENOISING FORWARD-TIME DIFFUSIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
The scope of this paper is generative modeling through diffusion processes. An approach falling within this paradigm is the work of Song et al. (2021), which relies on a time-reversal argument to construct a diffusion process targeting the desired data distribution. We show that the time-reversal argument, common all denoising diffusion probabilistic modeling proposals, is not necessary. We obtain diffusion processes targeting the desired data distribution by taking appropriate mixtures of diffusion bridges. The resulting transport is exact by construction, allows for greater flexibility in choosing the dynamics of the underlying diffusion, and can be approximated by means of neural network via novel training objectives. We develop unifying view of the drift adjustments corresponding to our and to time-reversal approaches and make use of this representation to inspect the inner workings of diffusion-based generative models. Finally, we leverage on scalable simulation and inference techniques common in spatial statistics to move beyond fully factorial distributions in the underlying diffusion dynamics. The methodological advances contained in this work contribute toward establishing a general framework for generative modeling based on diffusion processes.
INTRODUCTION
Denoising diffusion probabilistic modeling (DDPM) (Sohl-Dickstein et al., 2015; Ho al., 2020; Song et al., 2021) is recent generative modeling paradigm exhibiting strong empirical performance. Consider a dataset of N samples D = ) with empirical distribution Pp. The unifying key steps underlying DDPM approaches are: (i) the definition of stochastic process with initial distribution PD. whose forward- -immedd--meeea (noising) dynamics progressively transform PD toward simple iata-independent distribution Pz: (ii) the derivation of the backward-time (denoising / sampling) dynamics transforming Pz toward Pd; (iii) the approximation of the backward-time transitions by means of neural network. Following the training step (iii), sample whose distribution approximates PD is drawn by (iv) simulating from the approximated backward-time transitions starting with sample from Pz. Both discrete-time (Ho et al., 2020) and continuous-time (Song et al., 2021) formulations of DDPM have been pursued. This work focuses on the latter case, to which refer as diffusion Limereversal transport (DTRT). As in DTRT, dynamics are specified through diffusion processes, i.e. solutions to stochastic differential equations (SDH) with associated drift ʃ(.) and diffusion g(.) coefficients. number of approximations are involved in the aforementioned steps. Firstly, as the dynamics are defined on a finite time interval, a dependency from PD is retained through the noising process. Hence, starting with a sample from the data-independent distribution Pz in (iv) introduces an approximation. Secondly, while the backward-time dynamics of (ii) are directly available for diffusions, they are approximated by means of neural network in (iii). Thirdly, sampling in (iv) achieved through discretization on time-grid, which introduces a discretization enror. De Bortoli et al. (2021, Theorem 1) links these approximations to the total variation distance between the distribution of the generated samples from (iv) and Po.
In our first methodological contribution we develop a procedure for constructing diffusion processes targeting Pp without relying on time-reversal arguments. The proposed transport (coupling) between Pz and Pp is achieved by: (1) specifying diffusion process X on [0, 7] starting from a generic no: (2) conditioning X on hitting a generic I, at time т, thus obtaining a diffusion bridge; (3) taking a bivariate mixture Io, of diffusion bridges over (Io. xr) with marginals II, Pz and II, Pp. obtaining a mixture process M; (4) matching the marginal distribution of M over [0, 7] with a diffusion process, resulting in diffusion with initial distribution Pz and terminal distribution Pp.

---

Under review as a conference paper at ICLR 2022
The realized diffusion bridge mixture transport (DBMT) between Pz and PD is exact by construction. We thus sidestep the approximation common to all DDPM approaches due to the dependency from PD retained through the noising process. Moreover, the DBMT can be realized for almost arbitrary Pz. f(.) and g(.). This increased flexibility is a departure from the DTRT where f(.) and g( need to be chosen obtain convergence toward simple distribution Pz-
Similarly to the DTRT, achieving the DBMT requires the computation of a drift adjustment term which depends on D. For SDE class of interest, we develop a unified and interpretable representation of DTRT and DBMT drift adjustments as simple transformations of conditional expectations over D. This novel result provides insights on the target mapping that we aim to approximate and on the quality of approximation achieved by the trained score models of Song et al. (2021). Having defined for the DBMT a Fisher divergence objective similarly to Song et al. (2021), we leverage on this unified representation to define two additional training objectives featuring appealing properties.
In our last methodological contribution we extend the class of SDEs that can be realistically employed in computer vision applications. Specifically, computational considerations have so far restricted the transitions of the stochastic processes employed in DDPM to be fully factorials. We view images at given resolution as taking values over 2D lattice which discretizes the continuous coordinate system [0, 1]² representing heights and widths. Diffusion processes are viewed as processes with spatial support [0, 1 Doing so, it is possible to leverage on scalable simulation and inference techniques from spatial statistics and consider more realistic diffusion transitions.
This paper is structured follows. In Section 2 we review the DTRT of Song et al. (2021) and in Section 3 we introduce the DBMT. In order to implement the DTRT and the DBMT it is necessary to specify the underlying SDE, i.e. the coefficients f(-) and g(-). We study class of interest in Section 4. The unified view of drift adjustments is introduced in Section 5, Section develops the training objectives and Section 7 reviews the obtained results and finalizes the DBMT construction. In Section 8 we establish the connection with spatio-temporal processes. We conclude in Section 9, Appendices to D contain the theoretical framework, assumptions, proofs, and additional material.
Notation and conventions: we use uppercase notation for probability distributions (measures, laws) and lowercase notation for densities; each probability distribution, and corresponding density, is uniquely identified by its associated letter not by its arguments (which are muted); for example P(dx) is distribution, p(x) is its corresponding density; random elements are always uppercase (an exception is made for times, always lowercase for typographical reasons); if P is the distribution of stochastic process, we use subscript notation to refer to its finite dimensional distributions (densities with p), conditional or not, for some collection of times; for example Perjs denotes a transition density, which is understood to be function of four arguments (y/k)) distribution at x and @ is used for product distributions; we refer directly to given SDE instead of referring to the diffusion process satisfying such SDE when no ambiguity arises; we use [a]; and [A]k, for vector and matrix indexing, A for matrix transposition.
f(t, t', x, v); d. is the delta
DIFFUSION TIME- TRANSPORT
The starting point of Song et al. (2021) is a diffusion process Y satisfying a generic D-dimensional time-inhomogenous SDE with initial distribution Yo ~ Pp
dY, = J(r,, r)dr + r),,, over noising time E [) Thorough this paper we denote with Q the law of the diffusion solving
(1) and with g the corresponding densities. Thus, let gr'l (x/r)) 0 S r < r' S r, be the transition density of (I), and let (v). < r S T, be the marginal density of (1), As Yo ~ Po. we have
1 N N
(v( =
Σ (1)
(2)
n=1
The dynamics of (1) over the reversed, i.e. sampling, time t = - r, E [0, are given (Anderson, 1982; Haussmann & Pardoux, 1986; Millet et al., 1989)
by
d.X, | (x, + V G(X,,r) G(X,,)) Vx, In (X0) dt + r),,, T and the D-dmennsional vector V G(x,,) r) is defined by (V G(x, r)): E V., G(x (((x,))) Jag That is the processes X
(3)
where r p= - t is the remaining sampling time, G(x, r) =
2

---

Under review as a conference paper at ICLR 2022
and Y, = = have the same distribution. Approximating the terminal distribution Q+ of (1), i.e. the initial distribution of (3), with Pz. Xo is sampled from Pz and (3) is discretized and integrated over to produce a sample X, approximately distributed as PD.
The computation of the multiplicative drift adjustment V, In (x) entering (3), i.c. the score of the marginal density (2), requires in principle (NN) operations. Let sy, r) be neural network for which we would like 8 (y r) 2 Vyy In gr Ev(. It remains to find suitable training objective for which unbiased gradients with respect to Φ can be obtained at (1) cost with respect to the dataset size N. As gr (y) has a mixture representation, the identity of Vincent (2011) for Fisher divergences provides us with the desired objective for fixed 70
EE V, [I] Vy, (X,) so(Yr, [19]
LFD.DTRT (0, r)
E [I Vr, In qo s +11]
(4)
The key point is that unbiased, (1)) with respect to N, mini-batch Monte Carlo (MC) estimator for the expectation (4) can be trivially obtained by sampling a batch Yo ~ Pp. Y, ~ Qr (dy,) Yo), and evaluating the average loss over the batch. In order achieve global approximation over the whole time interval (0, TJ. Song et al. (2021) proposes uniform sampling of time r
LFD. (Ø) (6..0..)..%%Y..)......
E
R2 Vy, In gojo (Y) Yo) -s 1|
(5)
where R2 = Vy, In gr (Y, ()))))
is a regularization term. A MC estimator for (5) is con- structed by augmenting the MC estimator for (4) with the additional sampling step r ~ U(0, r].
DIFFUSION BRIDGE MIXTURE TRANSPORT
Our starting point is generic D-dimensional time-inhomoege inhomogenous SDE which, in contrast to Song et al. (2021), ,, is directly defined on the sampling time t E [0 dX,; f(1,, t)dt g)X,,))dW,
(6)
We reserve P.jo( (10) to denote the law of the diffusion solving (6) for given starting value xo and p- (o( (ato) to denote the corresponding densities.
3.1 DIFFUSION BRIDGES
Diffusion bridges are central the proposed methodology, in this Section we cover their basic theory. A diffusion bridge is a diffusion process starting from a given value which is conditioned on hitting terminal value. It is a deep result, and consequence of Doob h-transforms (Särkkä Solin (2019, Chapter 7.9), Rogers & Williams (2000, Chapter IV.6.39)), that diffusion processes pinned down on both ends is still a diffusion process. In particular the Markov property is preserved. More precisely, (6) with initial value øy conditioned on hitting terminal value I, at time is characterized the following SDE on [0, r] with initial value Ty (Särkkä & Solin, 2019, Theorem 7.11)
(7) where G(x,, () g(x, ()g(x, T) The multiplicative adjustment factor Va, In Prje(ar (1zz) forces the process to hit xr at time and the diffusion process solving (7) is known as the diffusion bridge from (zo, 0) to (I,, r). As previously noted, (2) (xz) in (7) refers to the transition density of (6).
d.X,- = [f(X,,t) t) + G(X,,t) Vx, In (mpp(((r((X)) dt + ((e,,))WW
3.2 DIFFUSION MIXTURES
The proposed transport construction relies on a representation result for diffusion mixtures. We present here informal version and report the precise statement, the required assumptions, and the proof in Appendix A.
Theorem (Difffusion mixture representation - informal) Let {X*}, J, X E A be a collection of diffusions with associated SDEs } and marginal densities {m²}. Let L be a mixing distribution on A, Te be the L-mixture of { }. Then there exists diffusion process X with marginal ni X follows a SDE whose drift and diffusion coefficients are weighted averages of the corresponding coefficients in (dXX), where the weights are proportional (m²) and to the mixing density.
3

---

Under review as a conference paper at ICLR 2022
Theorem 1 is first established in Brigo (2002, Corollary 1.3) limitedly finite mixtures and 1- dimensional diffusions. The proof of Theorem in Appendix A is more direct and extends the result to the required multivariate setting. In Section 3.1 we introduced diffusion bridges mapping arbitrary initial values Tp to arbitrary final values xz- Let r denote a generic bivariate distribution on RD x R with marginals ITg, II We define the diffusion mixture M as the mixture of diffusion bridges corresponding (Xu, X,) ~ Tu That is, apply Theorem to the collection of diffusion bridges (7) indexed by their initial and terminal values, À = (æo, z,), A R x R with mixing distribution L(dA)= = To (dæn, dx+). By Theorem the following SDE on [0, r] with initial distribution No has the same marginal distribution as M, in particular its terminal distribution is IT,
dx, = (Xe,, Dddt + 9(X,, l)dW,,
1(x,,t) = (mx, 1) + G( /Vn, Ve, In Prie(((t+ Pejo,- (2el TE,)
(x(e) TTo (daro, dx-);
A(a,,)
(e(xt)= = (1e Ir, Hor (dro, dær).
In (8), A(r,, t) gives the multiplicative drift adjustment factor for (6). case of particular interest occurs when II, puts all the mass on single value zo. In the following we refer to A(r,, t,) in stance of A(z,, f), and to (are((re (ar)) in stance of Ti (Zz), when it is necessary to distinguish this specific case. We also extend the scope of II to indicate the law of M. Indeed, we already denoted with oo its initial-terminal distribution, and with N. its marginal density. Accordingly, t Ex, (d(d [x,) V2, In Prie (X,, [x)]].
The transport from Pz to PD is then achieved by 11, = Pp and Pz = 1o (Pz can be arbitrarily defined). As Pp is an empirical distribution, the integral in (8) with respect to x;, reduces to averages over D. In summary, the diffusion X solution of (8) realizes the proposed transport from Pz to Pd by matching the marginal distribution of M.
4 SDE CLASS
The starting point of the proposed transport is the unconstrained SDE (6). In this Section we define SDEs which are realized through a time-change of simpler SDEs and which are general enough to subsume the SDEs introduced in Song et al. (2021). Consider the D-dimensional SDEs
dZ,- = 1//2WW,
(9)
d2, = a + //W
(10)
where at * is scalar function and G(X,, t) = T introduces an arbitrary covariance structure. (9)
is the SDE of correlated and scaled Brownian motion and (10) is the SDE of an process driven by a correlated and scaled Brownian motion. The transition densities of (9) and (10)
are Gaussian (Appendix B). We denote both with porj. informally (9) is a special case of (10) with
Oz = 0. SDE (10) in the time-homogenous case -1/2 has stationary distribution N (0, T). We now introduce the time-change. Let Br > 0 be continuous function on r). Then by f du defines monotonically (strictly) increasing function b, |0, r] + [o, b. The following SDEs on
[0, 7] represent the class of dynamics for (1) and (6) on which we focus on the rest of this paper
dX, 1/2
dX, = 3xdt + /Br.'/2dW.
(12)
and G(x,t)= = B.T. The standard time-change result for diffusions (Øksendal, 2003, Theorem 8.5.1) establishes that the processes X, respectively from (11) and (12) are equivalent in law to their time- scaled counterparts Zu, from (9) and (10). That is, SDEs (11) and (12) correspond the evolution of the simpler SDEs (9) and (10) under a non-linear time wrapping where time flows with instantaneous
intensity 3r. For both (11) and (12) the time-change argument yields P=|((y/z) = Ph- b. (3)2z) for the transition density of (6), and equivalently for the transition density Tz of (I). We thus obtain
(Z (xx) = ND t,a(t, I'((t
(13)
for appropriate scalar functions alt, T), u(,, 7) with v(4, T) > 0 (Appendix B). By direct computation
Vg, In In (x, (2a)) = r-1
Ir - 2z u(t,
at(,,)
(14)
a(t,T)
4

---

Under review as a conference paper at ICLR 2022
Va, In (=)) p-1 - xra((, (x-())(()))) 2t/ v((,T)
(15)
From Bayes theorem and the Markov property we have
(zjo,- xx) ND (#t; Tußbr (0, 7) + 2rr (0, r). (0,
(16)
where once again ahe (0, f). r). The T, and Ubr (0, > 0 are scalar functions given in Ap- pendix B. Finally, by direct computation
Vx, In p (Z 2+) r Toohr (0, 0, + 0br (,,
(17)
Ubr (0,
These results provide all the analytical formulas required for the computation of the adjustment factors A f), A(x,, L, mo) and of the training objectives used approximate them (Section 6).
4.1
INTERPRETATION DENOISING TIME-REVERSED SDES
Song et al. (2021) introduces two specifications of (1), named VESDE and VPSDE, which are respectively given by
dY, BBve, dY, - - Y,dr V Bwp
(18)
1
(19)
See Appendix B for the functional form of Bve.r and Bup.r. We thus recover (18) and (19) from (11) and (12) with T = and a, = 1/2.. That is, VESDE and VPSDE correspond to a time change of the much simpler SDEs for the standard Brownian motion and for the standard Langevin SDE
dzp dWr.
dZ, Zr dW,.
5 UNIFIED VIEW OF DRIFT ADJUSTMENTS
The linearity of SDEs (11) and (12), underlying our and Song et al. (2021) works, has the important consequence that (14) and (15) are linear in Ft. This in turn allow representation for the drift adjustment in (8). Indeed, substituting (14) in (8) gives (Appendix A)
derive an alternative
a² (1,
G( (),, = 8, (a(t,)) x, (,, (x)
E
[x,]
-x
(20)
vtt,
Similarly, for the time-reversal drift adjustment term in (3) we have (Appendix A)
G(I,r) Va gn Br (a(0,)) X.-
E
[X
(21)
v(0,r)
The relations (20) and (21) provide a unified view of the inner workings the DTRT and of the DBMT targeting PD. In the following always refer sampling time Remember that is the remaining sampling time. For ease of exposition we assume 3r = 1. as shown in Section 4 the term 3, corresponds to a time- warping. The terms a(t, r). a00,r) r) are "integrated scalings". They are equal to for (11) and the same holds for (12) as r + 0. The terms utt, r), v00, r) are "integrated v They are equal to r for (11) and the same holds for (12) as r - 0. We commonly refer to E[,,la,, for expectation terms in (20) and (21). Both drift adjustments (20) and (21) are thus essentially of the form (E[X, Ix, t] I r).;"" where the term v, diverges as 0.
The expectations E|X+ Ja,, are convex linear combinations of the samples (x) from D. Explicitly, Ex,, t] = E (ax, t) (m), where the weights w(r, (m) are the probabilities, under the distributions Q (time-reversal sampling process (3)) and II (mixture of diffusions process M from Section 3.2), of reaching each state (=() at terminal time from at time By construction, the initial weights entering expectation (20) are all equal to I/N when starts from a fixed value Zo- and are average when Xo is stochastie. The initial weights entering expectation (21)
5

---

Under review as a conference paper at ICLR 2022
on average approximately equal to I/N, depending on the quality of the approximation Pz Q Thus, [Xo, 0j is an averaging of many samples z(a), As time progresses, changes in X; correspond to changes in [X,, t] through changes in the weights w(z, t)(m), Eventually all mass concentrates on single weight wxx ()(*) corresponding to a dataset sample x(*), Ultimately, the attractor dynamics implied by (E[X, |z, t| - z)U,'' drive X, to x(()) We provide inspection in Figure 1, where D(CIFAR) stands for the training portion the CIFARIO dataset, and (uler(T) corresponds to the Euler scheme (Kloeden & Platen, 1992) applied with T discretization steps.
In the VESDE and VPSDE of Song et al. (2021) we have G(x, r) = I and reversing (21) gives v(0,r) V In (z) +
E
[X]
(22)
X, Qo (d
a
where Vy, In (1) the true score. We can thus take trained score model s,(2,r) s Vz In (r)) plug it in (22), and verity the extent to which E[X, | |x, t] has been approximated, see Figure 1,
64-
62-
Figure 1: VPSDE model 2nd cells' row: evolution of trajectory of X over sampling time (its terminal value X, is the generated sample) via the Euler( 1000) discretization of (3) using the true score V, In g, (y) for D(CIFAR); line-plot: weights' evolution w(X,t) t)(m) for all (a) in D(CIFAR) they remain close to 0); 1st cells' row: |Xx,|X,,tt evolution for the same X; 3rd and 4th cells' rows: same as 1s and 2nd cells' rows for another trajectory X, using the trained score model; 5th and 6th cells' rows: same as 3rd and 4th cells" rows for another trajectory X, using Fuler(1 (100).
for the same X (cyclical palette, many weights cannot be distinguished
We pause for moment to review the findings of Figure 1 (see Appendix D for additional related and of the associated weights in three
plots). Firstly, we can classify the dynamics of E[X, [X,, t] stages. In the 1st stage the weights do not move much. During the 2nd stage, roughly & [0.4, 0.6], the weights' mass gets distributed over a limited number of samples. Interestingly, the weights' dynamics are monotonic. As time progresses the weights' mass shifts between different objects from different classes. From the beginning of the 3rd stage all mass gets allocated to single weight, the terminal image decided well in advance of the terminal time. These dynamics are suboptimal. We would like to shorten the 1s stage, but it is associated with large values of 3r (i.c. quick time passing) which are required to decouple Q, from Pp. This is an intrinsic limitation of tive-seversal approaches. It is also dubious that (partially) sampling multiple objects over t is beneficial for efficient generative modeling when we make use only of the terminal sample. This issue applies to trained models as well, as the 3dd row of Figure shows. An interesting open question is how to obtain more suitable dynamics, where perhaps class transitions happen rarely. Secondly. E[X, [X,, t provides denoised representation of X, across the whole 3rd stage. An alternative the noise removal step applied to X, in Song et al. (2021) is to consider E E[X, t] as the sampling process instead. Thirdly, Figure 1 makes it clear that lowering the number of discretization steps affects generative

---

Under review as a conference paper at ICLR 2022
sampling in multiple ways. On the one hand the terminal sample X, is more noisy. This is not very surprising: close to the drift adjustment is approximately (xx(*) - X Xe), which is the drift of a Brownian bridge. Bridge sampling is notoriously problematic (Bladt et al., 2016). On the other hand larger discretization errors also significantly affect the dynamics of E [X,, resulting in less coherent samples. We remark that none of these insights could have been gained by observing X2 alone, L.e. the even cells' rows of Figure 1. To conclude, (20) and (21) give additional meaning to "denoising". Neural network approximators need to map from a noisy input X to an adjustment toward smoother superimposition of samples. The desire minimize the discrepancy between the smoothness properties of X, and that of EE] X Xe, 4] motivates the developments of Section 8.
6 TRANSPORTS APPROXIMATION
As in Song et al. (2021), computing the multiplicative drift adjustment A () requires ((N) operations. In this Section introduce three training objectives for which unbiased and scalable, i.e. (L) with respect to N, MC estimators can be immediately derived.
The first training objective applies only to A(Te, (o). It relies on the identity (Appendix A)
A(x,, 200) Vz, In ( Vz, In pego(ta))roo.
(23)
It is advantageous to consider the right-hand side of (23) because from (8) we know that (2, has mixture representation. As in Song et al. (2021), we can rely on Vincent (2011) to obtain scalable objective to train neural network approximator sp( () as Va, In Tgo(a) i.e.
LED.DEMT(0) (.
E
Jell Vx, In (x so(X,, +11?
Tu Vx, In Pn (x X,) 80 (N,, (X,, 011]] (24)
(x
where J. E[I Vx, In pn (X, X,) is regularization term.
The remaining training objectives rely on the identities (20) and (21). The goal directly approximate the expectations of (20) and (21) which, as in Section 5, we denote with generic E[X, Jr, t). That is, we aim to train neural network approximator so s, t) [1x t). As conditional expectations are mean squared error minimizers, suitable objectives for the expectation terms of (20) and (21) are
LCE. =
E tx, Iur
|||, so (X(,,, 11
(25)
LCE,DTRT()) (V.
E.
[1 so (V. r1]]]
(26)
In Table we summarize the operations needed to implement the plain MC estimators for the four objectives considered in this work. We reference where to find the required quantities for SDEs (11) and (12). The MC estimators for the Fisher divergence losses LFD,+ involve multiplications by r- (by (15) and (17)). Moreover, computing the drift adjustment at generation time requires multiplications by T. In Section 8 discuss how to manage the computational burden. An appealing property of LCE, is that computing the drift adjustment only requires the application of simple scalar functions (see (20) and (21)). and that their MC estimators only requires sampling operations. A further advantage of LCE,* is that no regularization is required. In contrast, in the absence of regularization terms, LED.. are divergent for R T due to the term v, (Section 5).
L
Sampling (r=201..)..--2((0.7))
Evaluation
LTD.DTRT
Yo-Po, Q (Y)
Vy, In (X.
LFD, DBMT
X,>Po, (Xo=zo), X, (n Yo-Po, (o) X,~Po, (dzo X, (ds,
Vx, In PX. (X.
LEE.DRT
LEE
Table 1: Sampling and evaluation operations required to implement the proposed MC estimators.
7

---

Under review as a conference paper at ICLR 2022
7 DBMT OVERVIEW AND NUMERICAL EXPERIMENT
In this section we finalize the DBMT construction, putting together the results of Sections 3, and 6. The unconstrained SDE follows (1I) (12). It remains to choose the mixing distribution Lo,r The marginal 11, needs to match Pp. but there is flexibility in the choice of 1o) Song et al. (2021) derived random ordinary differential equation (RODE) matching the marginal distribution of generative SDE, leading to faster sampling and to likelihood evaluation. RODE- matching requires II have density. A natural implementation is given by the factorial distribution ll, Pz & PD with Pz = ND (0, r) and the unconstrained SDE following (12) with 0z = 1/2 which preserves Pz. If instead the DBMT starts from fixed value zo. i.c. Ho, 8zo @ PD. we can choose To = 1/N Σn= (0, r)= to remove the drift adjustment at t = (see (20)) and reduce the work required to transport znn Pp. Finally, the use of non-factorial distributions can lead more efficient implementation, by linking the initial distribution to PD.
The training steps for the simplest objective (25) of Section 6 are reported in Algorithm 1. Batch size is assumed to be I to case the description. It is also assumed that Ho,r is factorial, otherwise the obvious modification applies line 2 (Algorithm is unaffected) where the endpoints are sampled. At line 3 a random central time and the corresponding state are sampled. The function optimi sn implements step of stochastic gradient descent update based on the loss The corresponding sampling algorithm is reported in Algorithm 2 where the Euler(T) discretization is assumed line Pijo,- ( Xo, X²), a(l,T), r), ((, r), 3e are defined in Section Section 8 shows how to sample efficiently from Pejo.- (dz, (dzz//oo (Xo. X-) and (0, in computer vision applications.
Algorithm 1 DBMT training (LMF)
Algorithm 2 DBMT sampling (LCF))
Input: Pp. Pz, SDE (11) or (12), NN st(x,t) Output: trained sq (2)
Input: Pz. SDE (11) or (12), trained ss(2, t)
Output: Discretized path Xa.r
I: repeat
I: Xo ~ Pz
2: X, PD. Xo ~ Pz
for s 1..... T do
3: L~ (( X, ~ Pijor (dr:) Xo, X7)
3: l t (s - 1)
I 4- X,-1
4: L || so Xx: 1)
4: Ug + B, ( S0 -x)
(2(r.r
o(1,7]
5: ő optimii zat iotiiiinssppppcc
5: E, ~ (N(,)
6: until convergence
6: X, t x (f(x, + tg
end for
We consider toy numerical example with Pz Pp 1/1(6_ + 62). = The unconstrained SDE follows the standard Brownian motion. We consider two mixing distributions: independent mixing 11,h1 where X, and X2 are independent and fully dependent mixing 11,11 where X₂ = X1. The results are reported in Figure 2. For both couplings the correct terminal distribution PD is recovered, can be seen by taking the row-wise sum of the transition matrices. The initial- terminal distribution of X solving (8), which realizes the DBMT, is different from the corresponding mixing distribution Io,1 which realized by the mixture process M. Ii results in transition
matrix of equal entries 1/9, Ug results in diagonal transition matrix of equal diagonal entries 1/3.
5:44
1:14
1:24
E:mx
6:28
0.0
6:25
0:00
6:25
0.30
24
Figure 2: (1x (IT 0. ). 2nd (15,)) plots): marginal density of the diffusion mixture M in yellow, which matches the marginal density of X solving (8), 5 sample paths of X started 0 in black; (3rd (TT 4th (1 plots): transition matrix of X from t to f estimated from 2000 samples.
8

---

Under review as a conference paper at ICLR 2022
8 NON-DENOISING DIFFUSIONS
In computer vision applications, images of resolution HxW corresponds to = 3HW. The use of an arbitrary covariance matrix I' in (11) and (12) requires its Cholesky (or equivalent) decomposition with cost (DD)). As the resolution increases the computational burden gets intractable very quickly, Indeed, to the best of the authors' knowledge, all prior DDPM literature only considers independent transitions, that is r = I. We suggest to view SDEs (11) and (12) as corresponding to the space discretization on an HxW grid of a spatio-temporal process defined over the spatial domain [0, 1J2. Consider the Euler discretization (12): X1+ =
AtE, where & ~ (O, the idea is to adopt a functional perspective: ((t At, = (t, s) + aa, s), V s) where & [0, 1) defines space coordinates. That is, both (1) and E()) at each time random processes over [0, 1]². We assume the innovations (t) to be a Gaussian process (GP) for each L.
As the GPs (t) are defined spatial statistics. As an example, we consider the circulant embedding method (CEM) (Wood & Chan, 1994; Dietrich & Newsam, 1997) which exploits a connection with the fast Fourier transform (FFT). See Appendix C for cursory review of the CEM. Consider an HxW uniform grid S of size S S= HW discretizing [0, 1|2, i.e. the support of images. For a stationary covariance function the CEM samples {t) on S with cost (nDin()) (D).. This is close to the O(D) cost of sampling from pure white-noise process, and compares very favorably to the ((D)) cost of Cholesky decomposition. One limitation of CEM is that generated samples, while always Gaussian, might not have the correct covariances. Whether this happens, and that case the quality of the approximation, depends on the covariance function. In Appendix C we select and fit an isotropic GP to the microscale properties of ((CIFAR). For this estimated GP sampling is exact. Figure shows samples from a pure white-noise GP,, i.e. I = I, (1st row) and from the fitted GP using CEM (2nd row). As noted in Section 6, sampling is enough to implement the MC estimators for LCF,,, but the MC estimators and drift adjustments for LCE,* involve additional matrix multiplications by and r The CEM allows to compute these at the same OD In (D)) cost if we define the GP ε(t) on a 2D torus (Rue & Held, 2005, Chapter 2.1). This corresponds to introducing dependencies between "opposing" boundaries of [0, 1|2 Figure (3nd row) shows some samples, in the highlighted patch the apposing-boundante dependency is evident. Either way, all samples from the 2nd and 3rd rows of Figure match the smoothness properties of D(CIFAR).
2D domain can leverage scalable inference techniques from
Figure 3: Spatial GP samples, see the main text for the description.
CONCLUSIONS
The DBMT construction of Section 3 is exact. The SDE class of Section is tractable as it results in linear diffusion bridges. The time-space factorization of the diffusion coefficient g(x, () 1/, separates modeling concerns: 3, corresponds to a time-wrapping I can be efficiently modeled by fitting the microscale properties of Pp- Availability of GPU-accelerated FFT implementations motivates our focus on the CEM. Alternative scalable approaches abound, from Gaussian Markov Random Fields (Rue & Tjelmeland, 2002; Rue, 2001) to Karhunen-Loève expansions (Betz et al., 2014). It remains to apply the results of this work to perform an empirical benchmarking. Section 6 develops three novel training objectives, two of which with desirable properties compared to the objective of Song et al. (2021), especially for non- non-factee con-factorial transitions. We remark the simplicity of the proposed DBMT approach (Algorithms and 2) compared to alternatives grounded the Schrödinger bridge problem (De Bortoli et al., 2021; Wang al., 2021; Vargas et al., 2021). The understanding of the target mappings ((20) and (21)) can guide the development of neural networks more closely matching the target structure compared to the U-Net default choice. https: h////ttttll com/ links to the code accompanying this paper which is made available under the MIT license.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Brian D.O. Anderson. Reverse-Time Diffusion Equation Models. Stochastie Processes and their Applications, 12(3)311-326, May 1982.
Wolfgang Betz, lason Papaioannou, and Daniel Straub. Numerical Methods for the Discretization of Random Fields by Means of the Karhunen Loève Expansion. Computer Methods in Applied Mechanics and Engineering, 271:109-129, April 2014.
Mogens Bladt, Samuel Finch, and Michael Sørensen. Simulation Multivariate Diffusion Bridges. Journal of the Royal Statistical Society. Series B (Stutistical Methodology), 78(2):343-369, 2016.
Damiano Brigo. The General Miittue--Diffusio SDE and Its Relationship with an Uncertain- Uncertain-Volaiiiiy Option Model with Volatility- Asset Decorrelation, December 2002.
Noel Cressie. Statistics for Spatial Data. John Wiley & Sons, 1993.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion Schrödinger Bridge with Applications to Ccore-Based Generative Modeling, June 2021.
C. R. Dietrich and G. N. Newsam. Fast and Exact Simulation of Stationary Gaussian Processes through Circulant Embedding of the Covariance Matrix. SIAM Journal on Scientific Computing, 18(4):1088-1107 July 1997.
U. G. Haussmann and E. Pardoux. Time Reversal of Diffusions. The Annals Probability, 14(4): 1188-1205, October 1986.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In H. Larochelle. M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Ioannis Karatzas and Steven E. Shreve. Brownian Motion and Stochastic Calculus. Number 113 in Graduate Texts in Mathematics. Springer, New York, 2nd edition, 1996. ISBN 978-0-387- 97655-6 778-3-540--7655-4
Peter E. Kloeden and Eckhard Platen. Numerical Solution of Stochastic Differential Equations. Springer Berlin Heidelberg, Berlin, Heidelberg, 1992. ISBN 978-3-642-08107-11 778-3-662-12266- 5.
NV Krylov. Introduction the Theory of Diffusion Processes, volume 142. Providence, 1995.
Annie Millet, David Nualart, and Marta Sanz. Integration by Parts and Time Reversal for Diffusion Processes. The Annals of Probability, pp. 208-238, 1989.
Chris G Rogers and David Williams. Diffusions, Markov Processes and Martingales: Volume 2:
Itô Calculus, volume 2. Cambridge university press, 2000.
Havard Rue. Fast Sampling of Gaussian Markov Random Fields. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 63(2):325-338, 2001.
Håvard Rue and Leonhard Held. Gaussian Markov Random Fields: Theory and Applications. Number 104 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton, 2005. ISBN 978-1-58488-432-3
Háävard Rue and Hääkon Tjelmeland. Fitting Gaussian Markov Random Fields to Gaussian Fields. Scandinavion Journal of Statistics, 29(1):31 2002.
Jascha Sohi-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 Proceedings of Machine Learning Research, pp-
2256-2265. PMLR, 2015.
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- Based Generative Modeling through Stochastic Differential Equations. In International Conference Learning Representations, 2021.
10

---

Under review as a conference paper at ICLR 2022
Simo Särkkä and Amo Solin. Applied Stochastic Differential Equations. Cambridge University Press, first edition, April 2019. ISBN 101-108--11773-. 978-1-316-51008-7 978-1-316-6494488 64946-66..
Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving Schrödinger
Bridges via Maximum Likelihood. Entropy, 23(9): 134, September 2021.
Pascal Vincent. Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7):1661-1674 July 2011.
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, and Can Yang. Deep Generative Learning via Schrödinger Bridge, June 2021.
Andrew T. A. Wood and Grace Chan. Simulation of Stationary Gaussian Processes in [0,1 ld. Journal of Computational and Graphical Statisfics, 3(4):409-432, 1994.
B. K. Øksendal. Stochastic Differential Equations: An Introduction with Applications. Universitext. Springer, Berlin; New York, 6th ed. edition, 2003. ISBN 978-3-540-04758-2.

---

Under review as a conference paper at ICLR 2022
A THEORETICAL FRAMEWORK
A.1 ASSUMPTIONS
Assumption (SDE solution). A given D-dimensional SDE(J. g) with associated initial distribution Vo and integration interval [0, 7] admits unique strong solution [0, r].
Assumption 1 can be checked through the application of the standard existence and uniqueness theorems for SDE solutions. Of particular relevance to our setting is the formulation of Krylov (1995, Chapter 5, Theorem 1) that limits the monotonic requirement to, informally speaking, drifts that pull the process toward infinities.
Assumption (SDE density), A given D-dimensional SDE(f.9) g) with associated initial distribution Vo and integration interval [0, r] admits marginal / transition density (0,T) with respect the D-dimensional Lebesgue measure that uniquely satisfies the Fokker-Plank Kolmogorov-forward partial differential equation (PDE).
We refer to Särkkä & Solin (2019, Chapter 5) and to Karatzas & Shreve (1996, Chapter 5.7) for connections between SDEs and PDEs.
All theoretical results of this work rely simple algebraic manipulations and re-arrangements of quantities of interest. The main complication stems from the need to justify differentiation and integration exchanges, i.e. exchange of limits.
Assumption 3 (exchange of limits). We assume that limits exchanges are justified in the steps marked with (+) and (+*).
Similarly, various steps in the derivations involve considering fractional quantities with densities appearing in the denominators.
Assumption 4 (positivity). For given stochastic process, all fiite-diimeni densities, condi- tional or not, are strictly positive.
Assumption 4 is easy verify. We resorted to the practical but somewhat unsatisfactory formulation of Assumption 3 because in full generality it is complicated to give easy to check conditions. We just note that when II = PD, the limit exchange marked with (x) is always justified. So are the limits exchanges marked with (**) when in addition II puts all the mass to a fixed initial value, or (by direct verification) when II is Gaussian for the SDE class of Section 4. Thorough this paper, both in the main text and in the proofs that follow, it is supposed that Assumptions 1, 2 and 4 are satisfied by SDEs (1), (6) and (7). This is the case for the SDE class of Section 4, i.e. (11) and (12), for any To with finite variance.
Remark: For ease of exposition it is assumed thorough this paper that all diffusions take values in the state space RD_ There is no impediment in extending the presented results to the case of diffusions taking values in a subset X' C RDD_ The obvious changes to Assumptions to 4 apply, the proofs carry over without substantial modifications. This extension could be of practical interest as images are often represented as floating point values in [0, 1].
A.2 STATEMENT AND PROOF OF DIFFUSION MIXTURE REPRESENTATIO THEOREM
Theorem 2 (Diffusiion mixture representation). Consider the family of D-dimensional SDEs on t [0, r] indexed by A
ax² = (x,,, + o (X),))WW, x3 V/
(27)
where the initial distributions Va and the BMs W are all independens. Let v² t E (0,T) denote the marginal density of X². For a generic mixing distribution L on A, define the mixture marginal density vi for t E (0, and the mixture initial distribution Vo by
vx(x)) = / (())) Voldx)) = / (dx))
(28)
12

---

Under review as a conference paper at ICLR 2022
Consider the D D-dimensional SDE on E [0, r] defined by
p(z, t) = JA n xz. t)
(e(x)
oxx 1) o (a, (1)
(29)
(1(3)
dX. -M(Xe, f)dt + o (X2, ()dW,,
Yo ~ Vo.
It is assumed that all diffusion processes X* and the diffusion process X solving (29) satisfy the regularity assumptions Assumptions 1, 2 and 4 and that Assumption 3 holds. Then the marginal distribution of the diffision X is Vt.
We start by establishing that the law of X is indeed given by the solution of (29). In this proof we make use of the following notation: for ccalar-valued (f), d for de vector-valued (a), Di La, for A matrix-valued (A),.: -i.j-1 dx,da, A. This notation allows for compact representation of PDEs reminiscent of the I-dimensional setting. Then, for 0 < < we have that
Proof of Theorem 2.
(1xx,)), = (L (A
((a)))
(**)
S (x, (((())))) ((a))
/ (a 1) v(att) v(r,t)
1
o (Z, v(z,t) L(d1) (le
+
ax
- (a, - 8) clax)(a,)) + +2 ( (a,ttt)) c(a))((;,t) v(2,t)
-
(**)
v(x,t)
The second line is an exchange of limits, the third line the application of the Fokker-Plank PDEs for the collection of processes X the fourth line is rewriting in terms of v(y, t), the last line is another exchange of limits. The result follows by noticing that the last line gives the Fokker-Plank representation of (29).
A.3 DRIFT ADJUSTMENTS
Limitedly to this section, we lighten the notation by removing subscripts from probability measures and densities. The missing time points can be inferred without ambiguity from the variables.
A.3.1 DRIFT ADJUSTMENT IDENTITIES FOR CONSTANT INITIAL VALUE
First identity:
plzy|ad (2)) I(dr,) p (xt (ro)
Vx, In
Ve,P(a (xt)
p(x+100))
(x0) p(a (1o
pl wo) pl p ((o))
/ V., In p(x, (2)) pl xr, ze( ((dz))) / /
pl pla,, zx (z0)
II [(dx))
p(x)(20)
p(ar(oo)
/ 12p(z {))x)))))))) ( m(0o/2o)
13

---

Under review as a conference paper at ICLR 2022
Second identity:
Vx, In /
p(ar)oz) [((de))
p 200
vx, In / pl (1s)
((xc) vx, Inp(a)/())
pl 2o)
Vx, (n Vz, Inp(2z/
A.3.2 DRIFT ADJUSTMENTS AS EXPECTATIONS
To establish (20) notice that from (14) we have
G( A)((z), t)
= B /
Ir
(t,,,)) p(ar//00, Ir)
Itt v(t,T) (T(2))
[To (dro, dx,))
a(t,T)
= B (a(tT)) f
T (20, [T))
(L,
11.,- (dxo, dxz)) -x1
((x))
T(l,T)
at((,T)
Be (a(l,)) r) x,,II(da,
E
[X+]
T((,T)
To establish (21) note that from (15) we have
G(yr.r) Vy. In q(v=)) =I Vg. In In ((vrl00)
Po(dyo)
((br)
3,IT / [a(0,r)00 Ur
(80)
PD(d)o)
000r)
((v)))
Br (a(0.r) f
(yrr(uo) ((1))
Po(dyo) yr v(0,r) r)
3r ((00,))
E
YYl yr
Yo Qo=Q(azolu.)
v(0,r)
B SDES CLASS FORMULAS
The transition densities of (9) and (10) are given respectively by
Pem,r (2)) ND (27; 24, I(r -))),
1 20 200z
(=) ND Zr; zee (T))) r
2a,
Here we used the notation Fo. = - S: fudu, i.c Fer is the average value of function fa on the interval [t, r]. The time- hime-homogenous case of (10), where Tee = is thus immediately recovered.
The scalar functions abm (1, T), (l, Vom (t, 7) and you vou are given by
Abbu (t, T) = =,
Vbm St, T) = by b,,
aou(t (t, T) abiier (br (br-
Vou (l,(t,]] 7) 20th.
20 -be)
20th.
The scalar functions ubr (0, t, T), abe a0, t,,t, r) and abr (0,t, are given by v(0,t)u(t,
Ubr (0, ()
v0, (0,1))2(1,77) v(l,7)
((,,7)a(0,1)
abr (,) v (0,t)aa u(t,
v00 ()
Toe (0, (0,,t)a u(t,
14

---

Under review as a conference paper at ICLR 2022
The scalar functions Bve,r and Bup.r given by
2r
Bve, omin
O max
log
mmax
Byp, (Buin r (Bmax 3mii))
Omin
Omin
The constants min, max: Amin: Bmax depend in part on the dataset considered, but are consistently chosen to have 3verr: Bupir small for r ze and large for r R3 T.
The approximating distributions Pz in Song al. (2021) are Pz (0, for VESDE, PZ = (O, I) for VPSDE.
C ADDITIONAL MATERIAL
C.I CLOSELY RELATED WORK
work closely related the present paper is that of Wang et al. (2021) it similarly avoids the time-reversal construction. Wang et al. (2021) construct a 2-stages diffusion process from a constant initial value zy to PD by relying on the theory of Schrödinger bridges. The most notable differences with respect to the DBMT transport are: (i) the dynamics considered in Wang et al. (2021) are less general, our notation they correspond to f()) = 0, g()) = a for fixed scalar σ: (ii) the transport proposed in Wang et al. (2021) necessarily starts from æo. the general result of (8) allows for (almost) arbitrary initial distributions and initial-terminal dependencies. For an initial Wi. i.e. for case of A(z,, 0)) in Section 3.2 and for the more limited dynamics considered in Wang et al. (2021), the achieved transport is the same. In this sense the DBMT generalizes the first stage diffusion of Wang al. (2021). It is interesting note that in the case of constant z0 the DBMT can also be obtained by an application of Doob h-transforms as we show in the following section.
We now review additional works grounded in the Schrödinger bridge problem: De Bortoli et al. (2021); Vargas et al. (2021). Both works rely on the Iterative Proportional Fitting (IPF) procedure solve the (dynamic) Schrödinger bridge problem. Both works leverage on time-reversal results to carry out the alternated Schrödinger half-bridge IPF iterations. The main difference between the two works is that De Bortoli al. (2021) estimates the optimal SDE drifts via neural network approximations and score-matching maximum likelihood fitting. The work of De Bortoli al. (2021) can be seen as an extension of Song et al. (2021), and similarly to our work allows the use of shorter time intervals. Compared to our proposal, it solves harder problem but also presents additional difficulties. Training more involved as all the neural network approximations, one for each IPF iterate, need to converge. Moreover, there limited guidance on how to optimally choose the number integration steps over the number of IPF iterates.
while Vargas et al. (2021) relies on Gaussian Processes and
C.2 CONNECTION WITH DOOB II-TRANSFORMS
The previously established identity
(=(o(Er((z)) II, (dx:):
A(w,, (,)) Vz, In Prio (x+ (20))
shows that the drift adjustment can be equivalently expressed as
(rie(())(1)) II p+10(==(<o)
u(a,, () fxx,, G(v,, 1) V., (,r,, (), h( () =
I1 (dxr),
as Fo is a constant. It can be verified that the h function satisfies the required space-time regularity property (Särkkä & Solin, 2019, Eq. (7.73)). As such, P (Zy (z2) = Peje (zv (zx ( ( t2)/h(z,, t) is the transition density of the DBMT transport from 6n, to II, follows by direct computation.
genuine Doob h-transform. That
C.3 GP MODELLING ON CIFAR10
For simplicity, we assume a factorial distribution over the channels and an isotopic stationary covariance function. We rely on the semivariogram approach (Cressie, 1993) to compare how
15

---

Under review as a conference paper at ICLR 2022
different covariance functions fit D(CIFAR). A semivariogram is measure of dependency across space. In the case of an isotropic stationary covariance it simplifies to a scalar function of the Euclidean distance between points: Asl) rate of decrease of r((] (sl) toward 0 as || - gives a measure of the infinitesimal spatial dependency, i.e. the smoothness of the spatial process. Semivariograms corresponding to different covariance functions are here fitted to their empirical counterparts via weighted minimum-least squares procedure (Cressie, 1993). Figure 4 illustrates the exponential and RBF semivariogram fits for two images of D(CIFAR). The exponential covariance, which corresponds to rougher paths, provides much better fit to the shown samples. This result is consistent across ((CIFAR). We remark that r = I corresponds to pure white-noise process with perfectly flat semivariogram which would clearly result in a very poor fit to the empirical semivariograms shown in Figure Based on these findings, model the innovations of each image channel as GP with exponential covariance function with length-scale match the marginal variance to that of (CIFAR)) o² 0.063.
B[(Az,))1/2 with Ar. Is+As Zg. The
0.205, the median estimated value (Figure (right).. We
4000
0.5
331
330
211
00
Figure 4: Empirical semivariograms (dots) and fitted exponential (solid lines) and RBF (doted lines) semivariograms for the 1st (left) and 2nd (center) image of D(CIFAR); histograms (bins) and medians (lines) of the distributions of the length-scale parameters in the exponential variogram model over D(CIFAR) (right); colors represent the RGB channels.
C.4 CIRCULANT EMBEDDING METHOD
Figure 5: Circulant embedding covariance matrices, see the appendix's 's main text for the description.
We start by providing cursory explanation leading to efficient sampling in the ID case, before giving the intuition behind the extension to the 2D case. We refer to Wood & Chan (1994); Dietrich & Newsam (1997) for complete explanation and Rue & Held (2005) for the results underlying efficient density (likelihood) computation. Let [0, 1] be the spatial domain of interest. Let {s:)A be a uniform grid (regular lattice) discretizing [0, 1]. where the points 8; are assumed be ordered. The first key observation is that for a stationary covariance function +( the covariance matrix C' with entries C. p(3j, sy) is symmetric and Toeplitz, i.c. with constant-diagonall See Figure 5 (leftmost) for an example where M = 9. A property of symmetric Toeplitz matrices is that they can always be embedded in larger symmetric circulant matrices. circulant matrix of size M' M''M' is defined by the property that all its rows (and columns) are obtained by cycling through the same M' dimeennsioo vector. Circulant matrices correspond to covariance matrices GPs defined on (here ID) torus (Rue & Held, 2005, Chapter 1). The circulant embedding matrix just introduced corresponds to an artificial enlargement of the spatial domain [0, 1] to a larger interval leading to torus. See Figure 5 (2nd from left) for circulant embedding of C. The second key observation is that a circulant matrix is diagonalized by the ID FFT matrix. Having obtained the eigenvalues of C, efficient sampling on the enlarged domain is achieved by the ID FFT applied to
16

---

Under review as a conference paper at ICLR 2022
complex standard random numbers multiplied by the (square root of the) eigenvalues. The real and imaginary part of the generated samples are independent. The main issue with the CEM is that the circulant matrix embedding might fail to be positive definite. The issue can be avoided by considering progressively larger embeddings, see the theoretical and empirical findings of Dietrich & Newsam (1997). Otherwise, level of approximation can be accepted by modifying the covariance function by truncating the eigenvalues to be positive.
The development of the 2D CEM follows very similar steps. The domain of interest is now [0, 1J2 the uniform grid is The stationarity of the covariance functions results in a symmetric block-Toeplitz covariance matrix, Toeplitz matrices can be embedded in symmetric block-circulant matrices, as exemplified by Figure 5 (rightmost, zooming might be required to see the block structure). Block-circulant matrices can be shown to be diagonalized by the 2D FFT matrix, and efficient sampling follows from similar steps the ones seen the 1D case.
(tr. Ji.== and the points 8i,j are assumed to be lexicographically ordered.
as shown in Figure 5 (3rd from left). Again, symmetric block-Toeplit
D ADDITIONAL FIGURES
36
64-
62-
Figure 6: Same as Figure for VESDE model.
17

---

Under review as a conference paper at ICLR 2022
Figure 7: Additional samples from the trained VPSDE model of Song et al. (2021), EX,,IX,, t] and X, (interleaved rows) over sampling time 1, Euler(1000) (top 16 rows) and Euler(100) (bottom 16 rows).
18

---

Under review as a conference paper at ICLR 2022
Figure 8: Additional samples from the trained VESDE model of Song et al. (2021), E[X, t] and X, (interleaved rows) over sampling time 1, Euler(1000) (top 16 rows) and Euler( (100) (bottom 16 rows).
19


Reviews:
[{"review_id": "Y-aSfiBc0TW", "reviewer": "Reviewer_oeyA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n\n-The paper is well-written. The mixture of diffusion bridges is clearly\nmotivated and the theoretical results seem to be correct.\n\n-The idea of using diffusion bridges (and mixture of diffusion\nbridges) to build a diffusion-based generative modelling is original and\ninteresting. One of the aspect of this work that I find promising is that in\nthis setting is that both the terminal and initial distributions are pinned down\ncontrary to existing works.\n\n-Related to this idea, remarking that a mixture of diffusion remains a diffusion\nwhose coefficients can be computed is an interesting point to raise and might be\nuseful in other settings. The fact that the resulting drift can be approximated\nusing score-matching techniques is also interesting.\n\nWEAKNESSES:\n\n-There is not enough experimental evidence. In particular, even though comparing\nthe approach of the authors with classical time-reversed methods might be\ncompute consuming on image synthesis tasks the authors should have provided\ncomparison on toy examples. As of now it is not clear at all that diffusion\nbridge based generative modelling is superior or comparable to time-reversed\ndiffusion based generative modelling.\n\n-Also, the authors should compared their methods with other works which also pin\ndown the initial and terminal distributions such as [2,3]\n\n-There is no theoretical justification why diffusion bridge based generative\nmodelling might be better than the classical time-reversed one. I understand\nthat the fact that the initial and final distributions are pinned down might help but it\nis not clear to me what kind of theoretical result could be obtained.\n\n-Overall I think that the presentation of the paper could be improved if the\nauthors focused on their core contribution which is the use of mixture of\ndiffusion bridges for generative modelling. I found Section 4 \"SDE Class\" and\nSection 6 \"Transports approximation\" to be a bit disconnected from the rest of\nthe paper. I would have preferred to see more experiments/comparisons on\ndiffusion bridge models or a more detailed theoretical analysis of these models.\n\nCOMMENTS:\n\n-The authors claim that every approach for score-based generative modeling is\n based on time-reversal. I slightly disagree in the sense that [5] never\n explicitly writes down the time-reversal of the forward process but rather find a parametric expression of some backward kernel which is only loosely connected to time-reversal.\n\n-I understand that the authors don't want to clutter the paper with too many\ntechnical results on the existence/uniqueness of SDE and the\nexistence/positivity of their densities but I think that it would be worth to\ncheck that these assumptions are satisfied in some generative modelling setting.\n\n-I found the experiment presented in Figure 1 to be really interesting but I am\nnot sure I fully understand what is done here. How do the authors compute the\nconditional expectation in Equation (20) and (21)? Is the term in (20) and (21)\ncomputed using score-matching as in [1]? If so, I'm a bit confused at the\nfinding obtained at the authors. It would seems that the process concentrates on\na single point and therefore that there is no innovation in the generative\nmodelling process? However it has been observed that score-based generative\nmodelling is innovative, see Figure 7 in [4] for an ImageNet experiment. Also,\ncould the authors precise how they compute the weights displayed in Figure 1?\n\n-I think Equation 8 could be slightly simplified to really illustrate the fact that $A(x_t, t)$ is a conditional expectation and therefore can be approximated using a neural network similarly to score-based generative modeling approaches. As of now, the only time this conditional expectation is written is in A.3.2.\n\n[1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole -- Score-based Generative Modeling through Stochastic Differential Equations\n\n[2] De Bortoli, Thornton, Heng, Doucet -- Diffusion Schrodinger Bridge with Applications to Score-Based Generative Modeling \n\n[3] Vargas, Thodoroff, Lawrence, Lamacraft -- Solving Schrodinger Bridges via Maximum Likelihood\n\n[4] Dhariwal, Nichol -- Diffusion Models Beat GANs on Image Synthesis\n", "summary_of_the_paper": "In this paper the authors introduce a new model for diffusion-based generative\nmodelling.  Instead of relying on time-reversed processes as in existing works,\nthey propose an approach based on the mixing of diffusion bridges. They show\nthat a mixing of diffusion bridges remains a diffusion with closed-form drift\nand volatility. Similarly to existing score-matching based generative modelling\nworks the drift in this new formulation can be expressed as a conditional\nexpectation and therefore can be seen as the minimizer of some loss function,\nallowing the use of neural-network based approximations. In addition, the\nauthors propose some unification of the SDE classes used in [1] and to use\nnon-identity volatility matrices.\n\n[1] Song, Sohl-Dickstein, Kingma, Kumar, Ermon, Poole -- Score-based Generative Modeling through Stochastic Differential Equations", "summary_of_the_review": "The main idea of the paper is sound and original. However, I think that this\nwork might not be mature enough. I would like to see more theoretical and\nexperimental comparisons with existing works to assess the efficiency of the\nproposed method. For these reasons I recommend the rejection of the paper but am\nready to raise my score if the authors provide compelling justifications for\nyour method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XcQFv91yGea", "reviewer": "Reviewer_FUzA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The methods presented are clear improvements to existing literature relaxing the constraints of time-reversal diffusions and moving beyond fully factorial noise models. The flow of the writing is clear, supporting arguments are properly presented, and representative results are shown appropriately. In addition, a code package is provided by the authors in the comments for reproducibility purposes which is clean and well organized. Overall, I did not find specific issues in this paper and hence I do recommend this paper for publication. That said, I'm not an expert in this field and hence I'll leave some room for my lack of knowledge about the state-of-the-art and significance of the work.", "summary_of_the_paper": "The paper introduces a methodological framework for generative modeling through diffusion processes without time-reversal arguments. By utilizing diffusion bridges the authors consider processes that start and end in pre-determined points $x_0$ and $a_{\\tau}$. Then by extending this to mixtures of diffusion bridges they show how to transport a distribution $\\Pi_0 = P_D$ to $\\Pi_{\\tau} = P_Z$ without time-reversal of the diffusion process. For specific classes of SDEs by drawing connections to the paradigm introduced by Song et al 2021 and provide a unified view on the drift adjustment in forward and backward SDEs. This allows for defining a time and state-dependent class probability function (given by conditional expectation) providing further insight into the inner-workings of DTRT and DBMT. The authors then introduce two other training objectives $L_{\\text{FD}},L_{\\text{CE}}$ and discuss their favorable properties. Finally, methods for moving beyond fully factorial noise models are presented by extending the process and the noise term from functions of time to be functions of both time and space. This allows for incorporating \"priors\" with better spatial dependency models more suitable for specific datasets. ", "summary_of_the_review": "Writing is clear, problem is significant, and proposed methods are novel. I did not find specific issues in this paper (although I did not go through the details of appendix A).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Jm6-sL5yEF", "reviewer": "Reviewer_QAeW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-response update. The authors' addressed my technical comments, and revised the manuscript \nwith clarifying visualisations and algorithm boxes. The three main major issues with the paper nevertheless still stand: the presentation is overly complex; the motivation is not convincing enough; and experiments are anecdotal. \n\n------------\n\nThis is a theory paper with only minimal experiments. The paper analyses Song et al ’21 bi-directional diffusion model, and proposes to replace and/or generalise it with Brownian bridge formulation, which only moves in generative direction. The problem is somewhat poorly motivated and it is not entirely clear how significant are the problems with the bidirectional approach, or what are the advantages of the bridge. One would expect that the bridge formulation now has a major problem with the (x0,xtau) pairing: surely a poor pairing would lead to a poor model. \n\nThe paper derives the bridge formulation at length. The derivation of the method takes almost the whole paper, and I found it to be difficult to follow. The paper does a poor job of describing what is the final model (algorithm box would help). The paper is also filled with tons of math, but has little exposition or motivation on the higher level concepts or intuition. There are also almost no illustrations of the ideas. Ultimately, after several hours on this paper, I only could understand around half of the material. The reader is overburdened by notation. This is going to limit the paper to narrower audience and impact. The scope and contributions are not sufficiently clear.\n\nThe paper shows that Song’21 method can be seen as special cases of covariant Brownian and OU processes. This seems like a side-result and not particularly useful, but is nevertheless interesting. The paper also presents some discussion of spatialization of the generative process with Gaussian processes. This is again interesting, but feels incomplete or more of a sketch. There are only very tentative results regarding this. \n\nThere are no large-scale or exhaustive experiments. THis is somewhat surprising: the proposed method seems like a variant of Song’s method, and I wonder why no comparative experiments were done. Perhaps the authors run out of time. I feel that the theoretical results alone are not sufficient for publication: from SDE perspective changing a reversible SDE to Brownian Bridge is rather incremental, and thus experimental demonstration would have been necessary to make the paper strong enough for publication.\n\n\nTechnical comments\n* It seems that “terminal distribution” is at time 0, while “initial distribution” is at time tau. Yet, in (i) in 1st paragraph the initial time is 0, and this is for observed objects. In (1) in 3rd paragraph the time 0 for latent base object. These are confusing, and needs clarification. A figure would be helpful.\n* Sec 1 “Firstly..”. “Q_tau retains dependency on P_D”. This could be non-true: if forward process mixes, then it will become oblivious of the initial distribution. This is the goal of density destructors: to lose all information of the original distribution. “The exact reverse-time simulation..”. I can’t understand this sentence. Why are we starting from Q_tau, and not Q_0? I also don’t see what approximation does this “firstly” subparagraph talk about, what was the problem again? Try to clarify\n* Secondly: why is it a problem that neural network approximates diffusion process? Is the problem that it introduces bias, or that it has some residual? Don’t we always need to approximate the diffusion if we don’t have access to the true one? Text also claims we have known reverse time dynamics known, surely this can’t be true? If we know the reverse time dynamics exactly, then why are we learning it? Try to clarify the problem and motivation of this point. \n* Thirdly: this is very clear\n* What is x0? This has not been defined, so I don’t understand if this is supposed to be a real object, a random noise object from base distribution, or something in else. It also seems that in 3rd paragraph the forward direction is now generative direction, while previously it was destructive direction. Please clarify to reader. \n* I generally can’t follow the 3rd paragraph. The text talks about DDPM, DBMT, and redefines DDPM as DTRT. There is lots of detail, but not enough background to understand. It seems the main problem is that Q_tau depends on P_D, but it is unclear why is this a problem. \n* In 4th paragraph the “drift adjustment” is unexplained. It is also implied that it is bad that the adjustment depends on data. Surely everything depends on data? I fail to see what is the argument here\n* The f/g are discussed, but not explained\n* Sec 2. “u” is often used in PDEs to denote solutions, but here its time. This can introduce unnecessary confusion for some readers.\n* I can’t follow what transition density q_u|u’(y|x) means. y is undefined (so is actually x as well). What does transition density mean? Is it a conditional marginal distribution or something else? Please define\n* Before eq 2 we first define q_u to be marginal density, while then eq2 redefines the density as a finite mixture. Surely marginal density should refer to \"fully\" continuous density, while eq2 should then be an approximation of the marginal density, or the N-empirical marginal density?\n* In eq 3 we define X_0 ~ P_Z, while in eq 1 we define X_0 ~ P_D. This is confusing\n* Eq 4 is confusing. The first line seems to be a mixture density, but second line Q_0,u is undefined. Does htis equation just state that one can do MC to sample from a mixture density? What is dX_u?\n* Eq 8: how does one do the pairing between x0 and xtau^{(n)}, or is there any pairing?\n* eq 14: \\Gamma looks like a function here, is it?\n* eq 21: weights \\omega are introduced, and it seems that they are central to the method. Yet, they are not part of any of the equations. They need introduction and needs to be connected to the presented theory\n* Sec 4. I got lost here. There is tons of math and theory here, which looks simple enough to follow, but little of this is motivated and the reader gets lost on where are we going with this. The presentation remains on a very technical level, not exposing the underlying concepts and intuition. Not all math needs to be understandable by a reader, but even a casual reader should understand why the math is there or what it achieves. Here I don’t see why the Song’s method is reinterpreted as the two processes 9+10. Does this relate to the rest of the paper or the proposed method? The presentation is also difficult to follow since a lot is left unexplained intentionally (eqs 15..17). The authors need to describe the motivation and role of the math, and also need to conceptualise the theory by helpful descriptions (or figures). \n* Sec 5. I’m again lost here. So eq 14 is applied to 8 to get 20. But 14 applies to Brownian and OU processes, while eq 8 is a bridge process. Why would the result hold? Furthermore, eq 8 gives a quite complex mixture model, which again seems very different from eq 9 or 10. It seems odd that to derive a mixture model 8, we take a detour to a different (and very simple) model family. Can’t we derive the eq 14 directly from eq 8? \n* Furthermore, the full method proposed in the paper is difficult to follow, since its description is scattered in many places. It’s difficult to go back-and-forth between eq 8 and 20. Later one needs to go back-and-forth between eq 23, 20 and 8 again. The description is also not complete yet, since \\Pi, v(), a(),etc are not defined. Because of these challenges, I then couldn’t follow the paragraphs after eq 21. \n* Where is the \\omega coming from? What is Y? What is E[Y]?}) It seems that method becomes some kind of weight-tuning algorithm, but weights are completely absent from any equations. Including an algorithm block would be helpful. \n* It seems that E[Y_t|x,t] becomes a central term later in the paper, but they are not properly defined. In which way they refer to the expectation terms in 20+21? Please define. \n* It would be very helpful to give the \\sum_n=1^N version of eq 8, since it seems that the proposed theory in the paper is based on the mixture view (or is it?)\n* Figure1: Generally I don’t understand what this experiment is doing. I can’t understand what happens on top. What is “all samples x^{(n)} in CIFAR”? This should be 50k points I assume, but it seems we only visualise around 10 curves. What does “true score” mean? How does one get the “true” score, and what does the score mean? What does Euler(1000) mean? What are the colors? Also at bottom I can’t understand what is \"true score”, and how come the score is an image, it was said to be a “single weight” above, so one expects it to be a scalar? The caption says that we track scores, but these look like images instead. Second and third rows are trained models, but what is then first row (is it untrained?). If we follow here E[Y], then should these images be some kind of average trajectories? The text describes three stages, but I can’t see where these come from (or what they are). I’m again having lots of trouble understanding the fig1 explanations in sec 5 since it’s difficult to follow what the figure shows.\n* Eq 25: why would matching s≈X result in s≈E[Y]? These seem like very different targets. \n", "summary_of_the_paper": "The paper studies the Brownian bridge formulation for diffusion-based generative models, derives theory for it, and shows connections to earlier works. Some spatial developments are also discussed. This is a theoretical paper that only has rudimentary experiments. The overall idea of Brownian Bridge diffusion is interesting, but the idea’s merits are not shown. The paper suffers from high technical complexity making it difficult to digest. \n", "summary_of_the_review": "While the theoretical derivations and the idea of Brownian bridge are interesting, and potentially a breakthrough, I feel that the theoretical results alone are not sufficient for publication. Changing the reversible SDE to BB is still somewhat incremental proposal, and thus the experimental demonstration would have been necessary to make the paper strong enough for publication. Furthermore, the presentation of the paper not good enough for a publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "A7YcqVNibHs", "reviewer": "Reviewer_e93q", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper is written in an obscure way for the non-specialists of this field (despite I have some limited background in SDEs). In many places, the motivations and constructions are not clear and not written precise enough for the ICLR readers.\n\nMy comments are as follows:\n\n1) The authors need to derive Eq. (7) within the paper for completeness. This result is deferred to several other theorems but it is hard to check whether assumptions made in these works to prove these theorems hold in this setting. What kind  of assumptions underlie the original result? Do they hold?\n\n2) I wonder why authors move from Eq. (7) to mixtures, as it is not made clear. Why is it not enough to use Eq. (7) for the generative modelling?\n\n3) In Eq. (8) where authors introduce the SDE, $g(X_t, t)$ is not specified.\n\n4) I found Section 4 quite confusing. The authors seem to define in this section an SDE class which are realised through a time-change of simpler SDEs (as they put it), I really couldn’t see how this relates to the rest of the paper. The authors should clarify what they mean by “the following SDEs represent the class of dynamics for (1) and (6)”. If this means that these SDEs, via a change of time argument, can represent (6), this notion has to be made clearer.\n\nAlso, these results need to be proved clearly instead of referring to the Oksendal’s book as done in the paper.", "summary_of_the_paper": "This paper proposes a diffusion bridge approach — based on conditioning on diffusions on both ends. This alleviates the need to use the reverse construction used by the earlier papers in this field.", "summary_of_the_review": "I found the paper hard to read - but found the ideas interesting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method for diffusion-based generative modeling through a Brownian bridge formulation, where the data and latent variable can be coupled. They extend their method to mixtures of diffusion bridges and spatially correlated processes that go beyond the factorial diffusion processes used in prior work.

We thank the authors for engaging with the reviewers and addressing many of their detailed concerns. While reviewers agreed that the proposed theory and methodology were novel and interesting, there are no small or large scale experiments or empirical comparisons to the relevant prior work. In the absence of theoretical justification (bound or proof) as to why the proposed diffusion bridge mixture transport method would result in better performance, more empirical comparisons and evaluations are needed. Additionally, several reviewers found the presentation confusing and overly complex, including the notation, writing, and figures. Given the lack of experimental results and concerns over presentation, I’m inclined to reject this paper.
Decision: Reject