Denoising Diffusion Bridge Models
Authors: Linqi Zhou
Aaron Lou
Samar Khanna
Stefano Ermon
Venue: ICLR 2024 poster (2024)
Keywords: Generative Modeling
Diffusion Models
Primary area: generative models

Abstract: Diffusion models are powerful generative models that map noise to data using stochastic processes. However, for many applications such as image editing, the model input comes from a distribution that is not random noise. As such, diffusion models must rely on cumbersome methods like guidance or projected sampling to incorporate this information in the generative process. In our work, we propose Denoising Diffusion Bridge Models (DDBMs), a natural alternative to this paradigm based on *diffusion bridges*, a family of processes that interpolate between two paired distributions given as endpoints. Our method learns the score of the diffusion bridge from data and maps from one endpoint distribution to the other by solving a (stochastic) differential equation based on the learned score. Our method naturally unifies several classes of generative models, such as score-based diffusion models and OT-Flow-Matching, allowing us to adapt existing design and architectural choices to our more general problem. Empirically, we apply DDBMs to challenging image datasets in both pixel and latent space. On standard image translation problems, DDBMs achieve significant improvement over baseline methods, and, when we reduce the problem to image generation by setting the source distribution to random noise, DDBMs achieve comparable FID scores to state-of-the-art methods despite being built for a more general task.

Published as a conference paper at ICLR 2024
DENOISING DIFFUSION BRIDGE MODELS
Linqi Zhou
Aaron Lou
Samar Khanna
Stefano Ermon
Department of Computer Science, Stanford University zhou, aaronlou, samar. kkanna, ermon] ) @stanford.edu
ABSTRACT
Diffusion models are powerful generative models that map noise to data using stochastic processes. However, for many applications such image editing, the model input comes from a distribution that is not random noise. As such, diffusion models must rely on cumbersome methods like guidance or projected sampling to incorporate this information in the generative process. In our work, we propose Denoising Diffusion Bridge Models (DDBMs), natural alternative to this paradigm based on diffussin bridges, a family of processes that interpolate between two paired distributions given as endpoints. Our method learns the score of the diffusion bridge from data and maps from one endpoint distribution to the other by solving a (stochastic) differential equation based on the learned score. Our method naturally unifies several classes of generative models, such score-based diffusion models and OT-Flow-Matching allowing us to adapt existing design more general problem. Empirically, DDBMs to challenging image datasets in both pixel and latent space. On standard image translation problems, DDBMs achieve significant improvement over baseline methods, and, when we reduce the problem to image generation by setting the source distribution random noise, DDBMs achieve comparable FID scores bate-of-the-an methods despite being built for more general task.
and architectural choices
apply
INTRODUCTION
Diffusion models are powerful class of generative models which learn to reverse diffusion process
mapping data to noise (Sohl-Dickstein et al.) 2015) [Song and Ermon 2019: Ho et al., 2020 Song et al. 2020b). For image generation tasks, they have surpassed based methods (Goodfellow ct al. 2014) and achieved a new state-of-the-ar for perceptual quality (Dhariwal and Nichol 2021) l).
Furthermore, these capabilities have spurred the development of modern text-to-image generative Al systemssRamesh et al. 2022).
Despite these impressive results, standard diffusion models are ill-suited for other tasks. In particular, the diffusion framework assumes that the prior distribution is random noise, which makes it difficult to adapt to tasks such as image translation, where the goal is map between pairs of images. As such, one resorts to cumbersome techniques, such as conditioning the model (Ho and Salimans) [022] Saharia al. 202I) or manually altering the sampling procedure (Meng al. [0022] [ong al. 2020b). These methods are not theoretically principled and map in one direction (typically from corrupted to clean images), losing the cycle consistency condition (Zhu et al. 2017).
Instead, we consider methods which directly model a transport between two arbitrary probability distributions. This framework naturally captures the desiderata of image translation, but existing methods fall short empirically. For instance, ODE based flow-matching methods (Lipman et al. 2023
Albergo and Vanden-Hijnden, 2023 Liu et al. 2022a), which learn a deterministic path between two arbitrary probability distributions, have mainly been applied to image generation problems and have not been investigated for image translation. Furthermore, on image generation, ODE methods have not achieved the same empirical success diffusion models. Schrödinger Bridge and models (De Bortoli et al. 2021) are another type of model which instead learn an entropic optimal transport between two probability distributions. However, these rely on expensive iterative approximation methods and have also found limited empirical use. More recent extensions including Diffusion Bridge Matching (Shi et al. 2023] Peluchetti 2023) similarly require expensive iterative calculations.

---

Published as a conference paper at ICLR 2024
In work, we seek scalable alternative that unifies diffusion- unconditional generation methods and transport- distribution translation methods, and we name our general framework Denoising Diffusion Bridge Models (DDBMs). We consider reverse-time perspective of diffusion bridges, diffusion process conditioned on given endpoints, and use this perspective to establish a general framework for distribution translation. We then note that this framework subsumes existing generative modeling paradigms such as score matching diffusion models (Song et al., 2020b) and
flow matching optimal transport paths (Albergo and Vanden- Eijneenn 2023 Lipman et al. [003]] Liu et al. 2022a). This allows us to reapply many design choices to our more general task. In particular, we use this to generalize and improve the architecture pre-conditioning. noise schedule, and model sampler, minimizing input sensitívity and stabilizing performance. We then apply DDBMs to high-dimensional images using both pixel and latent space based models. For standard image translation tasks, we achieve better image quality (as measured by FID (Heusel et al.) (017)) and significantly better translation faithfulness (as measured by LPIPS (Zhang et al. (0118) and MSE). Furthermore, when reduce our problem to image generation, we match standard diffusion model performance.
PRELIMINARIES
Recent advances in generative models have relied on the classical notion of transporting data distribution Idata (x) gradually to a prior distribution Pprior (x) (Villani 2008). By learning to reverse this process, can sample from the prior and generate realistic samples.
2.1 GENERATIVE MODELING WITH DIFFUSION MODELS
Diffusion process. We are interested in modeling the distribution Idata (x), for x E RRd We do this by constructing diffusion process, which is represented by set of time-indexed variables (x, IT such that X ~ Po(x) = Qdata (x) and XT ~ (x))): := Pprior (x). Here [dssa(X] is the initial "data" distribution and Pprior (x) the final "prior" distribution. The process can be modeled the solution to the following SDE
dx, f(x,, t)dt g(1)dw:
where f: Rd x [0, 7] Rd is vector-valued drift function, g [0, 7] IR. is scalar-valued diffusion coefficient, and w, is a Wiener process. Following this diffusion process forward in time constrains the final variable Xx follow distribution Pprior (x). The reverse of this process is given by
dx, f(x,, t) 2 Vx, log )(x:))dt (())dw,
where p(x:) := p(x,, t) is the marginal distribution of x2 at time t. Furthermore, one can derive an equivalent deterministic process called the probability flow ODE (Song al. 2020b), which has the same marginal distributions:
dx, = [((x;,))
-
1 (1) log log)(x.)
(3)
2
In particular, one can draw XT ~ (data(y) and sample gdata by solving either the above reverse SDE ODE backward in time.
Denoising score-matching. The score, Vx, log ((x;), can be learned by the score-matching loss
((0)= EE (xxx (oo (00,7)
[ t) Vx, log p(x; xo))l3
such that the minimizer s (x,, f) of the above loss approximates the true score. Crucially, the above loss tractable because the transition kernel p(x; Xo). which depends on specific choices of drift and diffusion functions, is designed to be Gaussian
aç89 + o,,6 where a, and are functions of time and e ~ [00 I). It is also common to view the diffusion process in terms of the x, s signal-to-noise ratio (SNR), defined as af o2.
2.2 DIFFUSION PROCESS WITH FIXED ENDPOINTS
Diffusion models are limited because they can only transport complex data distributions to a standard Gaussian distribution and cannot be naturally adapted to translating between two arbitrary distribu- tions, e.g. in the case of image-to-ine -omiage translation. Luckily, classical results have shown that one condition diffusion process fixed known endpoint via the famous Doob's h-traasform:
2

---

Published as a conference paper at ICLR 2024
Forward SDE
Vx, log p(x xz)
X0
dxt [(xe, 8) + +²)2(t))ii (t)(t)((x, 1. v. dt +(())dw,
XT
[((x,,) t) - g (x,, h(x,, 4, M))
X0
dx,+
XT
Probability Flow ODE
xx log ((xe XT
Figure 1:
schematic for Denoising Diffusion Bridge Models. DDBM uses diffusion process guided by drift adjustment (in blue) towards an endpoint XT = y. They lears to reverse such bridge process by matching the denoising bridge score (in orange), which allows one reverse from XT to Xy for any
y ~ 4data(y). The forward SDE process shown on the top is unidirectional while the probability flow ODE shown at the bottom deterministic and bidirectional. White nodes stochastic while grey nodes are deterministic.
Stochastic bridges via h-transform. Specifically, diffusion process defined in Eq. can be driven to arrive at a particular point of interest E IR almost surely via Doob's h--rranformm Doob Rogers and Williams
and Doob 1984
2000),
dx.
f(x:, d) + (()) h(x,),
(()dwr, Xo Qdata
where h,, Xx, logp(x>)
xx, x is the gradient of the log transition kernel of from & to T' generated by the original SDE, evaluated at points x, = 2 and x7 = y, and each x, now xx) satisfies the Kolmogorov backward
explicitly depends equation (specified in Appendix (A). With specific drift and diffusion choices, e.g. f(x,, t) = 0, is tractable due to the tractable (Gaussian) transition kernel of the underlying diffusion process.
time T. Furthermore, p(xy
When the initial point Xg is fixed, the process is often called a diffusion bridge Särkkä and Solin 2019 Heng et al. [202 Delyon and Hu 2006 Schauer et al.) [017 |Peluchetti] Liu et al. al.] [022b)) and its ability to connect any given X to a given value of Xy is promising for image-to-imag translation. Furthermore, the transition kernel may be tractable, which serves further motivation.
DENOISING DIFFUSION BRIDGE MODELS
Assuming that the endpoints of diffusion bridge both exist in Rdd and come from an arbitrary and unknown joint distribution, i.e. (Xu, xx) (x.y) ~ qdaata [x, y). we wish to devise a process that learns to approximately sample from Griata (x y) by reversing the diffusion bridge with boundary distribution Gdata (x, y). given a training set of paired samples drawn from @data (x.
3.1 REVERSED SDE AND PROBABILITY FLOW ODE
Inspired by diffusion bridges, construct the stochastic process {x, ) with marginal distribution q(x,) such that g(Xa, Xy) approximates Idata (xaaaaXx.. (Xg, Xy). Reversing the process amounts to sampling from q(x, x.). Note that distribution q(-) is different from p(-), i.e. the diffusion marginal distribution, that the endpoint distributions are now Idata (Xy. (Xp xT) = (xata(X y) instead of the distribution of diffusion p(Xa, xT) p(xT Xo Adaaa (Xo), which defines Gaussian Xy given Xg. We can construct the time-reversed SDE/probability flow ODE of g(x, xr) via the following theorem.
Theorem
The evolution of conditionnl probability (xx, xT) has time-reversed SDE of the form v, - dx, = f(x,,t) (t) ( s(x,, t,,, h(x,,
dt + g(t)dw, XT =
(6)
with an associated probability flow ODE
dx. -(),))))((1)))) ()xx,))))((((((xx))))))
h(x,, ((x,,4,,,7)) xT=
3

---

Published as a conference paper at ICLR 2024
Figure 2: VE bridge (left) and VP bridge (right) with their SDE (top) and ODE (bottom) visualization.
f(x(,t) t)
(²)
p(x1 xo)
SNR,
Vx, log p(TT x,)
VP
doog as
di i
d
2 do or
dt
of
Maxxo of
af/oo
(ar /xx/ -op--x, Xr of (SNR, /SNR, -I)
VE
0
0
N(x0, oi
1/07
X o7
dt
Table 1: VP and VE instantiations of diffusion bridges.
on t < T - for any e > O, where we denotes Wiener process, s(x, () = V log g(x: =x,xx,=- and h is as defined in Eq. (5)
Xr)
A schematic of the bridge process is shown in Figure Note that this process is defined up to T e. To recover the initial distribution in the SDE case, make an approximation that Xy-c y for some small e simulate SDE backward in time. For the ODE case, since we need to sample from p(x--c) which cannot be Dirac delta, we cannot approximate Xy- with a single y. Instead, we can first approximate XT- where > é' > O, and then take an and Eq. (7) can be used afterward. toy visualization of VE bridge and VP bridges are shown in Figure The top and bottom shows the respective SDE and ODE paths for VE and VP bridges.
step to Xr-c-
3.2 MARGINAL DISTRIBUTIONS AND DENOISING BRIDGE SCORE MATCHING
The sampling process in Theorem I requires approximation of the score sz, t, y, y, = Vx, log ((x I Xy) x =x where qxx, xT) = Jxx,, g(x; Xg, Xy [ddaaa )x) However, as the true score is not known in closed-form, we take inspiration from denoising score-matching (Song et al. 2020b) and use a neural network to approximate the true score by matching against tractable quantity. This usually results in closed-form marginal sampling of x, given data (e.g. Xy in the case of diffusion models and (xo: xT) in our case), and given X,, the model is trained to match against the closed-form denoising score objective. We are motivated to follow a similar approach because (1) tractable marginal sampling of x; and (2) closed-form objectives enable a simple and scalable algorithm. We specify how to design the marginal sampling distribution and the tractable score objective below to approximate the ground-truth conditional score Xx, log (xx: XT).
Sampling distribution. Fortunately, for the former condition, we can design our sampling distribution q(.) such that x(x; Xo, XT) := p(x; Xo; XT), where p(-) the diffusion distribution pinned at both endpoints as in Eq. 5 For diffusion processes with Gaussian transition kernels, e.g. VE,
(Song et al. 2020b), our sampling distribution a Gaussian distribution of the form
q(x, XT) = Nir,, o²
where
Ar SNRT at XT + 0x SNR, oT
SNRT SNR,
(8)
SNRT
67 = of
SNR;
where au and are pre-defined signal and noise schedules and SNR, = a./0? is the signal-to- noise ratio at time t. For VE schedule, we assume a, 1 and derivation details are provided in Appendix A. Notably, the mean of this distribution is linear interpolation between the (scaled) endpoints, and the distribution approaches Dirae distribution when nearing either end. For concreteness, we present the bridge processes generated by both VP and VE diffusion in Table and recommend choosing and specified therein.
Training objective. For the latter condition, diffusion bridges benefit from similar setup in diffusion models, since a pre-defined signal/noise schedule gives rise to a closed-form conditional score Vx. x, log q X.). We show in the following theorem that with x, g(xx, Xp. xy), neural network (x,, Xr,, t) that matches against this closed-form score approximates the true score.
4

---

Published as a conference paper at ICLR 2024
Theorem 2 (Denoising Bridge Score Matching). Let (xi, X.) ~ qdata y), q(xx xo, xT), t ~ p(t) for any non-zero time sampling distribution p(t) in [0, and w(t) be a non-zero loss weighting term of any choice. Minimum of the following objective:
[())- Xxxxx0x [o()]|so(xs xT, - Vx, log xo: xT)
satisfies So (xx XT, () Vx, log (xx xT).
In short, we establish a tractable diffusion bridge over two endpoints and, by matching the conditional score of the Gaussian bridge, learn the score of the new distribution q(x, xxy) that satisfies the boundary distribution (x.
4 GENERALIZED PARAMETERIZATION FOR DISTRIBUTION TRANSLATION
Building the bridge process upon diffusion process allows us to further adapt many recent advance- ments in the score network parameterization So (x,,, Xr, L) (Ho et al.) 2020: [ong et al.) 2020b
Salimans and Ho 2022 Ho et al. 2022 Karras et al.] 2022), different noise schedules, and efficient ODE sampling (Song et al. 2020aa Karras et al. 2022 Lu et al. 2022a b Zhang and Chen 2022) to
our more general framework. Among these works, EDM (Karras et al. 2022) proposes to parameter- ize the model output to be Dy (x,, t) Cskip
Cski(())x, (t)x, Cour (Ci Cnoise where F& is neural network with parameter that predicts Xg. In a similar spirit, we adopt this pred-x parameterization and additionally derive set of scaling functions for distribution translation, which we show is strict superset.
Score reparameterization. Following the sampling distribution proposed in X pred-x model can predict bridge score by
X
SNRR SNR, or
XT + ar DgDo0Xx, XT, ()
SNR SNR,
Vx, log ((; xT) RM
(10)
oi(1
SNRT SNR.
Scaling functions and loss weighting, Following SNRT/SNKS b, a;(1 SNRT WORR/SSRR, /SNR,), a of SNRT /SNR), ). the scaling functions and weighting function w(t) can be derived to be
Karras et al. 2022). and let a, = =a/ar
(tn(t)
Cou (t) = af(0oo00 - -Grr + ozee cm((t (11)
Votoz + +og + 2a.bioor + ce)
(ttip(t) (biof + +(ooo) (t), w(t) Cour (t) Cnoise (t) 4
1 log (t)
(12)
where oū o². and our denote the variance of Xo. variance of xr, and the covariance of the two, respectively. The only additional hyperparameters compared EDM are oT and apr, which characterize the distribution xr and its correlation with Xg. One can notice that in the case of EDM, 0, = t, of of + T because Xy = X T' for some Gaussian noise e, OoT of and SNRT /SRR = 12 /T2. One can show that the scaling functions then reduce to those in EDM. We leave details Appendix(A.5
Generalized time-reversal. Due to the probability flow ODE's resemblance with classifier- guidance (Dhariwal and Nichol 202T: Ho and Salimans (022). we can introduce an additional parameter to set the "strength" of drift adjustment as below.
dx, [(xx,,)) - ()xx))
(²(() ((x (, - wh(x,, L. TT) di,
y
(13)
which allows for a strictly wider class of marginal density of x, generated by the resulting probability flow ODE. We examine the effect of this parameter in our ablation studies.
5 STOCHASTIC SAMPLING FOR DENOISING DIFFUSION BRIDGES
Although the probability flow ODE allows for one to use fast integration techniques to accelerate
the sampling process (Zhang and Chen 2022 Song al. 2020al Karras al. 2022), purely
5

---

Published as a conference paper at ICLR 2024
following ODE path is problematic because diffusion bridges have fixed starting points given as data Xy = ~ (yata(y), and following the probability flow ODE backward time generates a deterministic "expected" path. This can result in "averaged" blunry outputs given initial conditions. Thus, we are motivated to introduce noise into our sampling process to improve the sampling quality and diversity.
Higher-order hybrid sampler. Our sampler is built upon prior higher-order ODE sampler in (Karras) al. 2022), which discretizes the sampling steps into iN 1N-1 ... to with decreasing intervals (see Appendix A.6 for details). Inspired by the predictor-corrector sampler introduced by Song et al. (2220)), we additionally introduce scheduled Buler-Maruyee backward SDE in between higher-orde ODE steps. This ensures that the marginal distribution at each step approximately stays the same. We introduce additional scaling hyperparameter 8, which define step ratio in between 4-11 and 4, such that the interval [t s(t; - 1;-1), [.] is used for
step which follows the
Malermma steps and [t,-1, L2 - s(4 41-1)] ) used for Heun steps, as described in Algorithm
RELATED WORKS AND SPECIAL CASES
Diffusion models. Diffusion models' advancements have boosted image generation, focusing on
network design, noise schedules, samplers, and guidance methods. The advancements in diffusion
models SSokl-Dickstein et al. 2015 Ho et al. 2020 Song et al. 2020b) have improved state-of-the
in image generation with improved network design (Song al. [2220b] Karras et al. 2022] Nichol
and Dhariwal 2021 Hoogeboom et al. 2023 Peebles and Xie 2023). noise-schedules (Nichol and
Dhariwal [022]] Karras al. 2022 Peebles and Xiel 2023), better samplers (Song al. |020a]]
Lu et al. 2022a bj Zhang and Chen 2022). and guidance methods (Dhariwal and Nichol] [221]] Ho
and Salimans| (022). Following these successful design choices, we seek construct our bridge
formulation to allow for seamless integration with this literature. As such, we adopt a time-reversal
perspective to directly extend these methods.
Diffusion bridges, Schödinger bridges, and Doob's 's h-transform. Diffusion bridges (Särkkä and
Solin 2019) have been actively studied in recent years in the context of generative modeling (Liu
et al. 2022b [Somnth et al. 2023 De Bortoli et al. 2021 Peluchetti] 2023) Heng et al. (02
Liu et al. (2022b) use Doob's h-transform to bridge between two dointssisstributions by simulating bridge paths. Other works (Somnath et al. 2023 Peluchetti Delbracio and Milanfar 2023) propose simulation- alternatives for forward-time generation. Another approach De Bortoli ct al. (20TT)
proposes Iterative Proportional Fitting (IPF) to tractably solve Schödinger Bridge (SB) problems in translating between different distributions. Liu et al.] (2023) is built on a tractable class of SB with a simulation-free algorithm and demonstrates strong performance in image translation tasks. Extending SB with IPF, Bridge-Matching (Shi et al.] (023) proposes to use Iterative Markovian Fitting solve the SB problem. A similar algorithm also developed by PPeluchetii 2023) for distribution translation.
Flow and Optimal Transport Works based on fow-Matchiig (Lipman al. 2023 Tong et al. 20236 Pooladian et al. 2023 Tong et al., 2023a Liu et al.) 2022a) learn an ODE- transport map to bridge two distributions. [ipmmnn al. (2023); | Liu et (2022a) has demonstrated that by matching the velocity field of predefined transport maps, one can create powerful generative models competitive with the diffusion counterparts. Improving this approach, Tong et al. (2023b); Pooladian et al. (2023) exploit potential couplings between distributions using minibatch simulation-free OT. Stochastic interpolants (Albergo and Vanden- Eijnneen 2023 Albergo et al., 2023) build flow models and directly avoid the use of Doob's h-functions and provide an easy way to construct interpolation maps between distributions. Separate from these methods, our model uses a different denoising bridge ccore-matching loss and the construction allows integration with existing designs of diffusion models to push ofte--ofthhearr further for image translation while retaining strong performance for unconditional generation.
6.1 SPECIAL CASES OF DENOISING DIFFUSION BRIDGE MODELS
Case 1: Unconditional diffusion process (Song et al. 2020b). We can show that the marginal p(x;) when p(xa) (data(X) (x) exactly matches that of regular diffusion process when X Qdata

---

Published as a conference paper at ICLR 2024
x) (( By taking expectation over Xy in Eq. (8) have
p(x: xo) = .N (a;xo, o,I)
(14)
One can further show that during sampling, Eq. 6 and reduce to those of diffusion process when Xy is sampled from a Gaussian (see Appendix A.4
Case 2: OT-Flow-Matching (Lipman al. 2023 Tong et al., 2023b) and Rectified Flow (Liu
et al. 2022a). These works learn to match deterministic dynamics defined through ODEs instead of SDES. In this particular case, they work with "straight line" paths defined by XT Xo-
To see that our framework generalizes this, first let us define a family of diffusion bridges with variance scaled by c E (0, 1) such that p(x; Xo, XT) C( where ce and o, are as defined in Eq. (8) One can therefore show that with a VE diffusion where of = c²t, given some fixed Xy and Xt, i.e. T 1, and X, sampled from Eq.
lim f(xx,t) (g)(t) - (2Vm,. log p(x, Xo. x1) Vx, p(x) xx))]] x - X0 (15) e-20
where inside the bracket is the drift of probability flow ODE Eq. 7 given Xp and x1, and the right hand side is exactly the straight line path term. In other words, these methods learn to match the drift in the bridge probability flow ODE (with a specific VE schedule) in the noiseless limit. The score model can then be matched against Xy Xo. with some additional caveat to handle additional input xy, our framework exactly reduces to that OT-Flow-Matching and Rectified Flow (details in Appendix A.4)
EXPERIMENTS
In this section verify the generative capability of DDBM, and we want to answer the following questions: (1) How well does DDBM perform image-to-inee -imag translation in pixel space? (2) Can DDBM perform well in unconditional generation when one side of the bridge reduces Gaussian distribution? (3) How does the additional design choices introduced affect the final performance? Unless noted otherwise, we use the same VE diffusion schedule in EDM for bridge model by default. We leave further experiment details to Appendix B
7.1 IMAGE-TO-IMACE TRANSLATION
that DDBM can deliver competitive results in general image-to-image translation tasks. We evaluate on datasets with different image resolutions to demonstrate its applicability on variety of scales. We choose
(Isola et al. 2017) scaled to 64 64 pixels, which contains image pairs for translating from edge maps to colored handbags, and DIODE-Outdoor (Vasiljevic et al.) 2019) scaled to 256 x 256, which contains normal maps and RGB images of real-world outdoor scenes. For evaluation metrics,
Fréchet Inception Distance (FID) (Heusel et al. (017]) and Inception Scores (IS) (Barratt and Sharma, 2018) evaluated on all training samples translation quality, and use LPIPS (Zhang al.] 2018) and MSE (in [-1, scale) measure perceptual similarity and translation faithfulness.
We compare with Pix2Pix (Isola et al. 2017). SDEdit (Meng et 2022). DDIB (Su al. 2022). Rectified Flow (Liu et al., 2022a). and i'SB (Liu et al. 2023) as they are built for image-to-image translation. For SDEdit we train unconditional EDM on the target domain, e.g. colored images, and initialize the translation by noising source image, e.g. sketches, and generate by EDM sampler given the noisy image. The other baseline methods are run with their respective repo while using the same network architecture as ours. Diffusion and transport- based methods are evaluated with the same number of function evaluations (N = 40, which the default for EDM sampler for 64 x 64 images) to demonstrate sampler's effectiveness in the regime when the number of sampling steps are low. Results are shown in Table 2 and additional settings are specified in Appendix B
We observe that our model can perform translation with both high generation quality and faithfulness, and we find that VP bridges outperform VE bridges in some cases. In contrast, Rectified- -elli as an OT-based method struggles to perform well when the two domains share little low-level similarities (e.g. color, hue). DDIB also fails to produce coherent translation due to the wide differences pixel- space distribution between the paired data. IPSB comes closest to method, but falls short when
7

---

Published as a conference paper at ICLR 2024
Rectified
DDBM (VE). DDBM (VE).
Condition
Pix2Pix [16
SDEdit 27
23
SB
22
Flow
ODE sampler hybrid sampler
Figure 3: Qualitative comparison with the most relevant baselines.
Edges Handdags-644x644
DIODE-256x255
FID
IS f
LPIPS
MSE
FID
IS f
LPIPS 4
MSE $
Pix2Pix (Isola al.
2017)
74.8
4.24
0.356
0.209
82.4
4.22
0.556
0.133
DDIB (Su et al.
2022
186.84
2.04
0.869
1.05
242.3
4.22
0.798
0.794
SDEdit (Meng et al.
2022
26.5
3.58
0.271
0.510
31.14
5.70
0.714
0.534
Rectified Flow Liu
2022a
25.3
2.80
0.241
0.088
77.18
5.87
0.534
0.157
PSB (Liu et al.
2023
7.43
3,40
0.244
0.191
9.34
5.77
0.373
0.145
DDBM (VE)
2.93
3.58
0.131
0.013
8.51
6.03
0.226
0.0107
DDBM (VP)
1.83
3.73
0.142
0.0402
4.43
6.21
0.244
0.0839
Table 2: Quantitative evaluation of pixel-space amage-to-imagg translation.
limited by computational constraints, i.e. NFE is low. We additionally show qualitative comparison with the most performant baselines in Figure 3 More visual results can be found in Appendix ??.
We demonstrate
7.2 ABLATION STUDIES
We now study the effect of our preconditioning and hybrid samplers generation quality in the context of both VE and VP bridge (see Appendix B for VP bridge parameterization). In the left column of Figure we fix the guidance scale w at T and vary the Fuler step size from 0 to 0.9 to introduce stochasticity. We see a significant decrease in FID score as we increase which produces Figure
the best performance some value between and (e.g. s 0.3 for Edges->Handbage also shows that the ODE sampler (i.e. s 0) produces blurry images while our hybrid sampler produces considerably sharper results. On the right column, we study the effect of w (from 0 to with fixed We observe that VE bridges are not affected by the change in w whereas VP bridges heavily rely on setting w We hypothesize that this due to the fact that VP bridges follow "curved paths" and destroy signals in between, so it is reliant on Doob's h-function for further guidance towards correct probability distribution.
We also study the effect of our preconditioning in Table 3 and our sampler is a simple model that directly matches output of the neural network to the training target and generates using EDM (Karras et al. 2022] sampler. We see that each introduced component further boosts the generation performance. Therefore, we can conclude that the introduced practical components are essential for the success of our DDBM.
Our baseline without preconditioning
7.3 UNCONDITIONAL GENERATION
When one side of the distribution becomes Gaussian distribution, our framework exactly reduces to that of diffusion models. Specifically, during training when the end point Xy ~ (ary of I). intermediate bridge samples x; follows the distribution x, ~ [aaxx of We empirically verify that using our bridge sampling and the pred-x x objective inspired by EDM, we can recover its performance by using more generalized parameterization.
8

---

Published as a conference paper at ICLR 2024
(a) Edges + Handbags
Our precond.
Our sampler
:11-64x64
VE
VE
VP
VP
X
x
14.02
11.76
126.3
96.93
/
x
13.26
11.19
79.25
91.07
1.83
Euler rabio
Guidence scale
x
/
13.11
29.91
91.31
21.92
VE VP
/
2.93
1.83
8.51
4.43
Table 3: Ablation study on the effect of sam- pler and preconditioning on FID. Cross mark on preconditioning means no output reparam- eterization and directly use network output to match training target. Cross mark our sampler means we reuse the ODE sampler from EDM with the same setting. E-H is short-hand for
50
1.5
8.51
92
Guidance
(b) DIODE
Figure 4: Ablation studies on Euler step ratio and guidance scale w: = 1 for all ablation on and is set to the best-performing value for each dataset for ablation on w.
CIFAR-10
FFHQ-61 x
NFE 4
FID.
NFE
FID
DDPM 13
1000
3.17
1000
3.52
DDIM 42
50
4.67
50
5.18
DDPM++
1000
3.01
1000
3.39
NCSN +
1000
3.77
1000
25.95
Rectified Flow
23
127
2.58
152
4.45
EDM 18
35
2.04
79
2.53
DDBM
35
2.06
2.44
Table 4: Evaluation of unconditional genera- tion.
Figure 5: Generation on CIFAR- 10 and FFHQ-64X 64.
We evaluate our method on CIFAR-10 10 (Krizhevsky al. 2009) and FFHQ-64 x 64 (Karras et al. 2019) which are processed according to Karras et al. (2022). We use FID score for quantitative evaluation using 50K generated images and use number of function evaluations (NFE) for generation efficiency. We compare our generation results against diffusion frffussion-based and optimal transport- based models including DDPM (Ho et al. 2020), DDIM (Song et al. 2020a), DDPM++ (Song et al. 2020b), NCSN++ (Song al.) 2020b), Rectified Flow (Liu et al. (022a), EDM (Karras et al. 2022). Quantitative results are presented in Table and
and generated samples are shown in Figure 5
We observe that model is able to match EDM performance with negligible degradation in FID scores for CIFAR-10 10 and marginal improvement for FFHQ-64 x 64. This corroborates our claim that our method can benefit from advances in diffusion models and generalize many of the advanced parameterization techniques such as those introduced in EDM.
8 CONCLUSION
In this work, we introduce Denoising Diffusion Bridge Models, novel class of models that builds stochastic bridge between paired samples with tractable marginal distributions in between. The model learned by matching the conditional score of a tractable bridge distribution, which allows one to transport from one distribution to another via new reverse SDE or probability flow ODE. Ad- ditionally, this generalized framework shares many similarities with diffusion models, thus allowing us reuse and generalize many designs of diffusion models. We believe that DDBM is significant contribution towards a general framework for distribution translation. In the era of generative AI, DDBM has further role to play.

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGEMENT
This research is supported by HAI, NSF(#1651665, ARO (N91112-1))-)012,,, ONR (N00014-23- 1-2159), CZ Biohub. We additionally thank support from and helpful discussions with friends and colleagues at Stanford University.
REFERENCES
Michael S A Albergo, Nicholas M Boffi, and Eric Vanden- Fijnden. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint WrXiv:2033.05797, 2023.
Michael Samuel Albergo and Eric Vanden-Eijndet Building normalizing flows with stochastic interpolants. In The Eleventh International Conference Learning Representations, 2023. ULLppppp////ppereeeeee
net/ Dorm/ id= /qeBbCRit
Shane Barratt and Rishi Sharma. note the inception score. arXiv preprint arXiv:1801.0197. 2018.
Valentin De Bortoli, James Thorton,, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score-based generative modeling. Advances in Neural Information Processing Systems, 34: 17695-17709, 2021.
Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303. 1435, 2023.
Bernard Delyon and Ying Hu. Simulation of conditioned diffusion and application to parameter estimation. Stochastic Processes and their Applications, 116(11):6600-165 2006.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 014-8700-8794 2021.
Joseph L Doob and JI Doob. Classical potential theory and its probabilistic counterpart, volume 262. Springer, 1984.
Ian J. Goodfellow, Jean Songet-Abadic Mehdi Mirza, Bing Xu, David Warde-Farle Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Jeremy Heng, Valentin De Bortoli, Arnaud Doucet, and James Thornton. Simulating diffusion bridges with score matching, arXiv preprint arXiv:21 1.07243, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 18510 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederík P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint ar arXiv:2210.02300 2022.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint 233.v 1093, 2023.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEFE conference computer vision and pattern recognàtion, pages 1134,114, 2017.
Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 4401 4410, 2019.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models, arXiv preprint (.XXv:22006003644 2022.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 34:21696-217077 2021.
10

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben- Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. The Eleventh International Conference on Learning Representations, 2023. URL
https::/cpenreet ew net/ Forum?1 PQMMMRDC
Guan-Homg Liu, Arash Vahdat, An-An Huang, Evangelos Theodorou, Weili Nic, and Anima Anandkumar. sb:: Image-to-image schrödinger bridge. arXiv, 2023.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint (rXxv:2209003003 2022a.
Xingchao Liu. Lemeng Wu. Mao Ye, and Qiang Liu. Let us build bridges: Understanding and extending diffussin generative models. arXiv preprint arXiv:2208. 14699, 2022b.
Cheng Lu. Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dom-solver: A fast ode solver for diffusion probabilistic model sampling around steps. arXiv preprint WXXi:2220.00092 2022a.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01099 2022b.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Jan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
Stefano Peluchetti. Non-denoising forward-ting diffusions.
Stefano Peluchetti. Diffusion bridge mixture transports, schr\)" odinger bridge problems and generative modeling. arXiv preprint VVXiv:2304.00117 2023.
Aram-Alexandee Pooladian, Heli Ben- Ban-Hamu, Carles Domingo-Enridh Enrich, Brandon Amos, Yaron Lipman, and Ricky Chen. Multisample flow matching: Straightening flows with minibatch couplings. arXiv preprint 1307::3304.1772,, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. ArXiv, abs/2204.06125, 2022.
L Chris G Rogers and David Williams. Diffusions, Markov procesxes and martingales: Volume Itô calculus, volume 2. Cambridge university press, 2000.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Salimans, David Fleet. and Mohammad Norouzi. Palette: image-to-image diffusion models. ACM SIGGRAPH 2022 Conference Proceedings, 2021.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512. 2022.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
Moritz Schauer, Frank Van Der Meulen, and Harry Van Zanten. Guided proposals for simulating multi- dimensional diffusion bridges. 2017.
Yuyang Shi, Valentin Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schr'," matching. arXiv preprint arXiv:2303.16953 2023.
odinger bridge
Jascha Sohl--Dckssten Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, In International Conference on Machine Learning, pages 2256-2266 PMLR, 2015,
Vignesh Ram Somnath, Matteo Pariset, Ya-Ping Hsieh, Maria Rodriguez Martinez, Andreas Krause, and Charlotte Bunne. Aligned diffusion schr'," odinger bridges. arXiv preprint arXiv:2302.1144 /14119922.1111, 2023.
11

---

Published as a conference paper at ICLR 2024
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010002002, 2020a.
Yang Song and Stefano Ermon. Generative modeling estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.3446 2020b.
Xuan Su, Jiaming Song, Chenlin Meng, Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In The Eleventh International Conference on Learning Representations, 2022.
Juraj Szavits-NNsssn and Martin Evans. Inequivalence of nonequilibrium path ensembles: example of stochastic bridges. Journal of Statistical Mechanics: Theory and Experiment, 2015(12):P12008 2015.
Alexander Tong. Nikolay Malkin, Kilian Fatras, Lazar Atanackovic, Yanlei Zhang, Guillaume Huguet, Guy Wolf, and Yoshua Bengio. Simulation free schr'," odinger bridges via score and flow matching. arXiv preprint arXiv:2307. 03672, 2023a.
Alexander Tong. Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-bes now-based generative models with minibatch optimal transport. In JCML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023b.
Igor Vasiljevic,
Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang. Falcon Z. Dai. Andrea F. Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R. Walter, and Gregory Shakhnarovich. DIODE: A Dense Indoor and Outdoor DEpth Dataset. CoRR, abs/1908.0044 2019. (tt.:ttt:://axxiii
abbs/1908 63
Cédric Villani. Optimal transport: Old and new. 2008.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks, 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242-2251, 2017.
12


Reviews:
[{"review_id": "mvN3tHh4pn", "reviewer": "Reviewer_zUzA", "summary": "This paper proposed a novel formulation on diffusion-based generative model, i.e., denoising diffusion bridge models (DDBMs). This formulation further inspire applications in important tasks such as image-to-image translation. Detailed theoretic derivation and empirical results are provided.", "questions": "1. Is there any advantage of DDBMs over some previous image-to-image translation methods such as controlnet?\n2. It would be better to show the variation ability of DDBMs in terms of the generation results. For example, when translating edges into an image, there exists various solutions. Are DDBMs able to generate these various results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method and formulation is novel and can potentially empower some pivotal applications such as image-to-image translation.\n2. The paper is well-written and easy-to-follow.\n3. Sufficient qualitative and quantitative results are provided to validate the effectiveness of the proposed method.", "weaknesses": "1. More results on larger-scale generation would be better (e.g., DDBM with Stable Diffusion).\n2. It would be better if the author could compare the DDBMs with some more advanced image-to-image translation algorithm such as controlnet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pMMAmW6S3g", "reviewer": "Reviewer_H8Nr", "summary": "This paper proposes a means of adapting standard denoising diffusion models, which parameterize the score in the Ornstein Uhlenbeck process to map a Gaussian to a data distribution, so that it can be more properly used to connect arbitrary densities, e.g. for image-to-image translation.\n\nThe main tool at play is Doob's h-transform, which they use to derive an SDE that has a learnable score conditioned on some endpoint $x_T$. They write a variant of the score matching loss that accommodates this. \n\nFollowing this, they address the usual topics of a SBDM paper, which is how to re-write the score in a way that allows you to predict $x_t$, how to consider the signal to noise ratio in the diffusion path (which is not a very meaningful name in the case of the ODE), how to write the probability flow, and what sampler to use.\n\nThey benchmark the method on various image-to-image translation tasks, quantifying image quality metrics with a consideration for the number of function calls necessary to achieve said quality.\n\n\n\n--generalized time reversal is not proven to converge to the correct distribution. Equation 13 should be shown to be justified. \n\n\n\n-- can only do this given a pair", "questions": "- The authors write down the doob-$h$ transformed SDE for arbitrary coupling, but it's not clear in the experiments that they consider any coupling besides $p(x_0, x_1) = p(x_0)p(x_1)$. Can the authors consider any experiments which make use of a more interesting coupling?\n- An important ablation should be based on the sampler the authors used. Did the authors retrain the models they compared against and fixed e.g. the sampling strategy? They discuss a higher-order Euler-Maruyama sampler, but it is unclear if all the comparison to other methods used the same integrator. Many of the methods listed could use a different integrator for the same trained model, and it's unclear to the reviewer what was held fixed across comparisons and what wasn't. The reviewer points this out to stress that a paper is not a method -- the method presented in the paper should be recognizably connected to the conclusions of the experiments. If an auxiliary factor influenced the outcome, e.g. like choosing a better sampler, this diminishes the message about the method itself.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In the specific context of denoising (score-based) diffusion models (DDPM),  this paper introduces the useful perspective from the bridge literature of how to alleviate the limitation of DDPM as a generative model in that it could only connect a data distribution to a Gaussian. \n\nThe experimentation is thorough which shows that the method is performant on the image tasks presented. The ablation study is useful (even if a bit contrary to their claims in the abstract of wanting to move away from \"cumbersome methods like guidance\"). The authors usefully summarize how many of the tricks that make diffusions highly performant fit into their extension.", "weaknesses": "Let me begin by saying, thank you for the effort you put into this. The experiments are very thorough, and the doob-h formulation is sound.\n\nUnfortunately, this paper is written in a way that highlights some misconceptions. It is unclear what the authors see as the contribution of this paper. It has become clear from the wider literature than just the narrow perspective of the same score-based diffusion equation how to do image-to-image translation. In fact, the works that this paper already cite do this, even for both the ODE and SDE between arbitrary densities. \n\nA quick summary of related work that should be properly addressed in this paper, and the authors should highlight what actually sets this method apart:\n- [1,2,3] propose a means of learning an ODE to do connect any two densities, e.g. for your image-to-image translation in finite time $ t \\in [0,1]$ without bias. In fact [1] does this image-to-image translation experiment there directly.\n- [4] shows how to do this for **both** an ODE and an SDE (with the score), also in finite time. It also shows how to avoid the added complexity of doob, and how SBDM is a subcase.\n- [5, 6] shows the influence of changing the coupling $p(x_0, x_1)$ between the densities $p(x_0)$, $p(x_1)$\n- [7] also shows how to use either and ODE or SDE to connect the densities under varying coupling. \n\nThe related worked section, in addition to the intro, should be thoroughly reworked to not overlook these clear contributions. While it is certainly beneficial to have a clear connection to how to do this with SBDM using doob-h and experiments of it, the paper is currently written so as to suggest that there is a clear need to come up with tools to do this as if many works have not addressed it. The reviewer is sympathetic to the fact that this field moves very fast, but also believes that overselling a concept by overlooking other work is detrimental to the field. \n\nWith regards to the statement that the ODE based methods [1,2,3] \"tend to severely underperform when compared to diffusion models,\" there is no evidence of this presented, and in fact the purpose of those papers is to present evidence of the contrary, e.g. see [3]. If the authors would like to make this statement, they should demonstrate it in experiments. It is of the reviewers mindset that the differentiating factor between most simulation-free transport generative models is just whether or not the model was conditionally trained, e.g. as $s(x,y,t)$ vs $s(x,t)$. Many papers report \"unconditional sampling\" FIDs by training a conditional model and using a null-token e.g. $y=-1$ to sample unconditionally, which improves their score.\n\nThere is perhaps a misinterpretation of what optimal transport is, and what role it may play in generative models that are constructed from continuous-time transport plans. The authors refer to \"VE (OT)\" bridges without providing a clear definition of what they mean by OT. Here are the two ways to learn an optimal transport between distributions:\n\n- 1. Choose the optimal coupling $p(x_0, x_1)$, with \"schedule\" $x_t = (1-t)x_0 + tx_1$. This is hard to do, and why people rely, e.g. on Sinkhorn.\n- 2. For independent coupling $p(x_0)p(x_1)$, learn a process $x_t = f_{\\theta}(t,x_0,x_1)$ which gives a time dependent velocity/score and density $v_t, p_t(x)$ that induces a minimum action (or least transport cost) in terms of Benamou-Brenier transport cost.\n\n- In the abstract, the authors reference \"OT-Flow-Matching\", though the work [3] does not propose to solve OT. At most ,they use a relationship about McCann displacement maps (which is related to OT from a Gaussian to a Gaussian) to motivate choosing a straighter conditional probability path -- they are not doing OT. \n\n\n-Throughout the text, the authors need to make sure all acronyms are defined.\n\n- The introduction of the weighting factor in equation (13) is not shown in the text to preserve an exact/unbiased transport. The authors should justify this or state that it introduces a bias, even if a beneficial one.\n\n\n*The reviewer is not opposed to adjusting their score, but there are many organizational and presentational aspects of this work that suggest it is not fit for contribution to the larger body of work at ICLR at this stage. Please find questions below that could help better demonstrate the utility of the method (in controlled experiments or new ones that e.g. exploit an interesting coupling) as well as the remarks above that need addressing.*\n\n[1] Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. *Xingchao Liu, Chengyue Gong, Qiang Liu*, Sept 2022.\n\n[2] Building Normalizing Flows with Stochastic Interpolants. *Michael S. Albergo and Eric Vanden-Eijnden*, Sept 2022.\n\n[3] Flow Matching for Generative Modeling. *Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, Matt Le*, Oct 2022.\n\n[4] Stochastic Interpolants: A Unifying Framework for Flows and Diffusions. *Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden*, March 2023.\n\n[5] Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport. *Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, Yoshua Bengio*, Feb 2023.\n\n[6] Multisample Flow Matching: Straightening Flows with Minibatch Couplings. *Aram-Alexandre Pooladian, Heli Ben-Hamu, Carles Domingo-Enrich, Brandon Amos, Yaron Lipman, Ricky T. Q. Chen*, April 2023.\n\n[7] Simulation-free Schrödinger bridges via score and flow matching. *Alexander Tong, Nikolay Malkin, Kilian Fatras, Lazar Atanackovic, Yanlei Zhang, Guillaume Huguet, Guy Wolf, Yoshua Bengio*, July 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "imq0RfrzwR", "reviewer": "Reviewer_2Uac", "summary": "This paper presents a new class of generative models, Denoising Diffusion Bridge Models (DDBM), that can define a generative path between any 2 paired distributions. This is in contrast to current diffusion models where the path is always from Gaussian to a data distributions. DDBMs are a more generic representation, and they allow solving tasks like image-to-image translation (which is not trivial to do with regular diffusion models). DDBMs work by building a stochastic bridge between a paired samples. DDBMs share several attributes with diffusion models, which allows reusing many of the techniques already available in this field. These new models can be trained in a similar fashion as score matching, and the authors present extensive theoretical backing to allow for a loss formulation and transport between distributions via a reverse SDE and flow ODE. The authors show results in image-to-image tasks with reasonable results. The authors also show competitive results when running tasks currently done by diffusion models, mainly text-to-image generation.", "questions": "- Why didn't the authors compare against ControlNet?\n- What is the advantage of DDBMs when compared to FM given that both can in theory model flow between arbitrary distributions?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors tackle an important and challenging problem, mainly how do we use our current generative pipelines (diffusion) to solve more general generative problems. The authors do a good job at presenting a sounds alternative. I highlight the following strengths:\n- The formulation of DDBMs seems solid. The authors present a good comparison between this method and score matching and flow matching. In fact, the authors show that DDBMs are a generalization of these methods. \n- The authors formulate the model in a way that shares many commonalities with current diffusion models. I appreciate this effort for two reasons. First, it makes understanding and adopting them easier given the familiarity they have with current methods. Second, it reuses many of the formulations already developed in things like score matching, which should lead to better training. \n- The authors go to good length to present and explain all relevant mathematical formulations. These look sound, although I did not fully explore all the details.\n- The authors show favorable results in image-to-image tasks and comparable results in text-to-image generation. Given that these come from the same model formulation (DDBMs), this combination of results is a strength of the formulation.\n- The paper is well written and structured.", "weaknesses": "I have 2 concerns I believe are minor, but would like to hear from the authors:\n- The authors only validate the work with image-to-image translation. In this domain, ControlNet is the dominant solution. I value that the 2 models might not be competing solutions. Can the authors clarify why they didn't compare against ControlNet in the image-to-image task?\n- With generalized flow matching models we can in theory move between any distributions. What is the advantage of DDBMs compared to flow matching? I found some reasoning in the paper but didn't fully understand, therefore I appreciate a brief explanation from the authors.\n\nOverall I believe this are minor issues and I hope the authors provide answers during rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fFrFuCqNzV", "reviewer": "Reviewer_BgL3", "summary": "The authors describe diffusion generative model as a time reversal of a Brownian bridge forward process, where the Brownian bridge end points are paired samples from a coupling of two marginal distributions. The generative model is learnt as a time reversal of a Brownian bridge, amortized across pairs by learning the score conditioned on an end point.\n\nThe authors show strong empirical results and adapt parameterization of networks from current state of the art diffusion models.\n\nThis is closely related to existing diffusion model, diffusion bridge models and bridge matching methods.\n\nEssentially this is a regular denoising diffusion model where the forward and backward process is conditioned on the terminal point. This is also a conditional bridge matching approach where the reverse bridge is matched.", "questions": "How does this work relate to bridge matching? Can the same result not be achieved through conditional bridge matching?\n\nIs it true this work is a continuous time version of [2]?\n\nAm I right in thinking that Theorem 1 equation (6) is the same as that derived in Section 2.1 of [1]?\n\n[1] Heng et al. Simulating Diffusion Bridges with Score Matching, 2021\n[2] Li et al BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models, 2022", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "- The authors show excellent empirical performance of the method and adapt network parameterization from current state of the art diffusion models to bridges.\n- The authors extend the time reversal approach of [1] to the generative setting by amortizing the time reversal across multiple pairs of points from the marginal distributions. Whereas [1] uses general and in particular nonlinear forward process, this work uses a Brownian bridge where the bridge conditioned on end points can now be sampled in closed form and the forward h-transform is also known. Although this is not useful for the applications considered in [1], it is useful for generative modelling where one does not care about the reference process.\n- The work is well explained and is essentially a continuous time version of [2], which does not detail the time reversal interpretation very well and does not condition on the end marginal in the network, but does in the h-transform.\n- Ultimately this is a conditional bridge matching approach which seems to empirically have a better coupling than non conditioned bridges.\n\n[1] Heng et al. Simulating Diffusion Bridges with Score Matching, 2021 \\\n[2] Li et al BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models, 2022", "weaknesses": "**Novelty**\nUltimately this work trains a conditional diffusion model through score matching, and conditioning on the terminal state x_T from a given coupling. The difference to regular diffusion models is simply conditioning the forward process as well as the backward process with additional information from a coupling. This can also be viewed as conditional bridge matching, by matching the reverse bridge.\n\nAlthough I very much like the connection to bridge matching, time reversal and diffusion bridges, this is not discussed to a satisfactory standard.\n\n**Lack of discussion to prior work**\n\nThe authors' work severely lacks a detailed related works section. Many related works and contributions have been briefly commented in passing. Without proper acknowledgement prior work, the authors' contributions appear inflated. \n\n1) Lacking reference [2], this appears to be the same idea but for discrete interpretation of diffusion models / bridge, derived through a variational approach similar to DDPM rather than time reversal - but which are well known to be equivalent. [2] does not condition on the end marginal in the network, but does in the h-transform.\n\n2) Lack of discussion to bridge matching prior work\n\nThere is a lack of discussion to [1,4,5,6,9], which the authors are aware of given these papers are cited. Although phrased differently they appear highly related to this work. [1,4,6,9] similarly sample from a coupling and train a network for the drift of the of the backward or forward diffusion process (which also involves a h transform), but not conditioned on a one of the marginal points. This work considers learning the the drift of the backward process by learning the score (the score and h-transform (forward drift) make up the reverse drift, as shown in [5]), but instead conditioned on the starting point, x_T. I imagine this work is essentially a conditional bridge matching approach.\n\nAs discussed above, [5] is highly related in that the high level approach is the same: train a network between two points by reversing a diffusion bridge. The differences lie in [5] focusing on general / nonlinear forward process for sampling bridges rather than generative modelling. [5] requires simulating the forward diffusion as nonlinear forward bridges cannot be sampled in closed form. [5] is for a fixed pair rather than amortized across pairs. Indeed, to further illustrate the similarities **Equation 6 within Theorem 1 of this work appears to coincide with the equation written below Equation (4) in Section 2.1 of [5].**. \n\n3) Misrepresenting prior work\n\nThe authors claim \"A related work (Somnath et al.,2023) similarly establishes a diffusion bridge that translates between two distributions and has seen success in protein differentiable domain, but the training objective requires sampling an entire SDE trajectory for computation\".\n\nAs far as I am aware this is not true. I do not understand how the authors came to this conclusion.\n\nMinor:\n-  is it not clear why [3] is being cited for the Schrodinger bridge interpretation of diffusion models. The approach of [3] is not a Schrodinger bridge between marginals but two diffusion models back to back with a Gaussian between, there are significantly more relevant works. If anything this is misleading the reader.\n\n- \"A recent work (Liu et al., 2023) considers a special case of SB\" . I2I SB [9] does not actually result in a Schrodinger Bridge, neither does the same method detailed in Aligned SB [1]. They both perform bridge matching with respect to Brownian bridge reference diffusion but for a data driven coupling. Given the data driven coupling does not correspond to the coupling from Brownian motion, the bridge matching procedure returns a Markovian projection and hence breaks the coupling. See discussion in [7] \"the Schrödinger Bridge is the unique path measure which satisfies the initial and terminal conditions, is Markov and is in the reciprocal class of Q, see (Léonard, 2014b).\" One needs to iterate on the coupling and drift in order to obtain a SB. \n\nThis is a tangential remark as one does not need an optimal diffusion in order to perform conditional generation if it can be supervised with given paired samples from a coupling. However there appears to be many errors in papers being published in this area and I feel this should be addressed.\n\n**Claim of generalizing flow matching/ bridge matching**\n\nThe connection of diffusion bridges to flow matching has been established and detailed in [7], detailing the limit as the noise coefficient goes to 0. The original flow matching paper [8] is even derived using result from [4] on bridge matching, so this is widely known. Note: [7] was public well before submission but not published.\n\nHowever, given the score function in this work is conditioned on x_T, the learnt backward diffusion is non-Markovian, hence retains the initial marginal coupling and would not recover rectified flow / schrodinger bridge by iterative Markovian fitting as in [7] or in rectified flow. So I would argue this work does not subsume flow matching / drift matching.\n\n**Summary**\nWhilst I like the core idea of this paper and believe there is a contribution, I believe \"contextualization relative to prior work\" is lacking.\n\n[1] Somnath et al 2023, Aligned Diffusion Schrödinger Bridges \\\n[2] Li et al 2022, BBDM: Image-to-Image Translation with Brownian Bridge Diffusion Models \\\n[3]  Su et al. 2022, Dual Diffusion Implicit Bridges for Image-to-Image Translation \\\n[4] Peluchetti 2022, Non-Denoising Forward-Time Diffusions \\\n[5] Heng et al. Simulating Diffusion Bridges with Score Matching \\\n[6] Liu et al., 2022, Let us Build Bridges: Understanding and Extending Diffusion Generative Models \\\n[7] Shi et al Diffusion Schrödinger Bridge Matching 2023 \\\n[8] Lipman et al 2022, Flow Matching for Generative Modeling \\\n[9] Liu et al 2023, I2SB: Image-to-Image Schro ̈dinger Bridge", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)