Consistency Diffusion Bridge Models
Authors: Guande He
Kaiwen Zheng
Jianfei Chen
Fan Bao
Jun Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Bridges
Consistency Models
Image Translation
Primary area: diffusion_based_models

Abstract: Diffusion models (DMs) have become the dominant paradigm of generative modeling in a variety of domains by learning stochastic processes from noise to data. Recently, diffusion denoising bridge models (DDBMs), a new formulation of generative modeling that builds stochastic processes between fixed data endpoints based on a reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation. However, DDBM's sampling process typically requires hundreds of network evaluations to achieve decent performance, which may impede their practical deployment due to high computational demands. In this work, inspired by the recent advance of consistency models in DMs, we tackle this problem by learning the consistency function of the probability-flow ordinary differential equation (PF-ODE) of DDBMs, which directly predicts the solution at a starting step given any point on the ODE trajectory. Based on a dedicated general-form ODE solver, we propose two paradigms: consistency bridge distillation and consistency bridge training, which is flexible to apply on DDBMs with broad design choices. Experimental results show that our proposed method could sample $4\times$ to $50\times$ faster than the base DDBM and produce better visual quality given the same step in various tasks with pixel resolution ranging from $64 \times 64$ to $256 \times 256$, as well as supporting downstream tasks such as semantic interpolation in the data space.

Consistency Diffusion Bridge Models
Guande He
Kaiwen Zheng"'; Jianfei Chen', , Fan Bao Jun Zhu 123 Dept.. of Comp. Sei. Tech., Institute for AI, BNRist Center, THBI Lab 'Tsinghua- Bosch Joint ML Center, Tsinghua University, Beijing, China
Shengshu Technology, Beijing Paazou Lab (Huangpu), Guangzhou, China
guande.he178r www.hoetooooo. com; zkwthu@gmail com;
fan. bww@shenghhu....oooooooooooo...m ai; {jianfeic, dcszj)8tsinghua edu.
Abstract
Diffusion models (DMs) have become the dominant paradigm of generative mod- cling in variety of domains by learning stochastic processes from noise to data. Recently, diffusion denoising bridge models (DDBMs), new formulation of gener- ative modeling that builds stochastic processes between fixed data endpoints based reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation. However, DDBM's sampling process typically requires hundreds of network evaluations to achieve decent performance, which may impede their practical deployment due to high computational demands. In this work, inspired by the recent advance of consistency models in DMs, we tackle this problem by learning the consistency function the probability-flow ordinary differential equation (PF-ODE) of DDBMs, which directly predicts the solution at a starting step given any point on the ODE trajec- tory. Based on dedicated general-form ODE solver, propose two paradigms: consistency bridge distillation and consistency bridge training, which is flexible to apply on DDBMs with broad design choices. Experimental results show that our proposed method could sample 4x to 50x faster than the base DDBM and produce better visual quality given the same step various tasks with pixel resolution ranging from 64 x 64 to 256 x 256, as well supporting downstream tasks such as semantic interpolation the data space.
Introduction
Diffusion models (DMs) 53 21 60
have reached unprecedented levels family of generative
models in various areas, including image generation [10] (5 48). audio synthesis [5) 45). video
generation 200 well as image editing (2). solving inverse problems (25 [56. and density
estimation [59 28 37 7 the era Al-generated content, the stable training, scalability &
state-of-the-art generation performance of DMs successfully make them serve as the fundamental
component of large-scale, high- nigh-pperfrrance text-to-image [4] and text-to-video
18
models.
A critical characteristic of diffusion models is their iterative sampling procedure, which progressively drives random noise into the data space. Although this paradigm yields sample quality that stands out from other generation models, such as VAEs [29 46 GANs [77. and Normalizing 3D it also results a notoriously lower sampling efficiency compared to other arts. In response to this, consistency models (58) have emerged as attractive family of generative models ordinary differential equation (PF-ODE) at certain starting timestep given any points in the ODE trajectory, designed to be a one-step generator that directly maps noise to data. Consistency models can be
Flows
by learning a consistency function that directly predicts the solution of a probability-
Work done during internship at Shengshu; 'Equal contribution;
The corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

he her T)
Diffusion ODE
by ~ (N(00,3))
ho(
Noise Level
xx
ho ho 8, xT)
he r wr)
Bridge ODE
Pdata
PT
Time
Figure 1: Illustration of consistency models (CMs) on PF-ODEs of diffusion models and our proposed consistency diffusion bridge models (CDBMs) building on PF-ODEs of diffusion bridges. Different from diffusion models, the PF-ODE of diffusion bridge is only well defined in < T due to the singularity induced by the fixed terminal endpoint. To this end, valid input for CDBMs some a, for < T. which typically obtained by one-step posterior sampling with a coarse estimation of 2o with an initial network evaluation.
naturally integrated with diffusion models by adapting the score estimator of DMs to a consistency function of their PF-ODE via distillation (58] 26) or fine-tuning [[S] showing promising performance
for few-step generation in various applications like latent space 40) and video [64].
Despite the remarkable achievements in generation quality and better sampling efficiency, fun- damental limitation of diffusion models is that their prior distribution is usually restricted to a non-informative Gaussian noise, due to the nature of their underlying data to noise stochastic pro- cess. This characteristic may not always be desirable when adopting diffusion models in some scenarios with an informative non-Gaussian prior, such as image-to-image translation. Alterna- tively, an emergent family of generative models focuses on leveraging diffusion bridges, a series of altered diffusion processes conditioned on given endpoints, to model transport between two arbitrary distributions [4] 36
33 54] 72, 77. Among them, denoising diffusion bridge models (DDBMs) (72) study the reverse-time diffusion bridge conditioned on the terminal endpoint, and employ simulation-free, non-iterative training techniques for it, showing superior performance in application with coupled data pairs such as distribution translation compared to diffusion models. However, DDBMs generally require hundreds of network evaluations to produce samples with decent quality, even using an advanced high-order hybrid sampler, potentially hindering their deployments in real-world applications.
In this work, inspired by recent advances in consistency models with diffusion ODEs 58 57 15)..
we introduce consistency diffusion bridge models (CDBMs) and develop systematical techniques to learn the consistency function of the PF-ODEs in DDBMs for improved sampling efficiency. Firstly, to facilitate flexible integration of consistency models in DDBMs, we present a unified perspective on their design spaces, including noise schedule, prediction target, and network parameterizations, termed the same as in diffusion models [28, 24]. Additionally, we derive first-order ODE solver based on the general-form noise schedule. This universal framework largely decouples the formulation of DDBMs and the corresponding consistency models from highly practical design spaces, allowing us to reuse the successful empirical choices of various diffusion bridges for CDBMs regardless of their different theoretical premises. On top of this, we then propose two paradigms for training CDBMs: consistency bridge distillation and consistency bridge training. This approach is free of dependence restricted form of noise schedule and the corresponding Euler ODE solver in previous work (58), thus enhancing the practical versatility and extensibility of the CDBM framework.
We verify the effectiveness of CDBMs in two applications: image translation and image inpainting by distilling or fine-tuning DDBMs with various design spaces. Experimental results demonstrate that our approach can improve the sampling speed of DDBMs from 4x to 50x, in terms of the Fréchet inception distance [99 (FID) evaluated with two-step generation. Meanwhile, given the same computational budget, CDBMs have better performance trade-offs compared to DDBMs, both quantitatively and qualitatively. CDBMs also retain the desirable properties of generative modeling, such sample diversity and the ability to perform semantic interpolation in the data space.

---

2 Preliminaries
2.1 Diffusion Models
Given the data distribution Pdata (2), R", diffusion models [53 21 60] specity a forward-time diffusion process from an initial data distribution finite time horizon E [0, T". defined by a stochastic differential equation (SDE);
Pdata to a terminal distribution pr within a
dæ. (a t)dt + g(t)dw:
2o po:
where w, is standard Wiener process, I x [0, R and g: [0, T] - R are drift and diffusion coefficients, respectively. The terminal distribution pr is usually designed to approximate tractable prior Pprior (e.g., standard Gaussian) with the appropriate choice of f and g. The corresponding reverse SDE and the probability flow ordinary differential equation (PF-ODE) of the forward SDE in Eqn. ① is given by 60]:
dæ, = (f g² (t) log)) + g(t) ET ~ PT = Pprior:
(2)
dæ; = f(xr,))) () 2 (1)
de, oT ~ pr Rs Pprior,
(3)
where W2 is a reverse-time standard Wiener process and (((x)) is the marginal distribution of 2t. Both the reverse SDE and PF-ODE can act as a generative model by sampling ær Pprior and simulating the trajectory from DT to Zp- The major difficulty here is that the score function V lo Pt (ax) remains unknown, which can be approximated by a neural network so sa( () with denoising score matching 63):
E (m (mo) [A(t)]ss(Zz,, - V Pooo
(4)
where 4(0, T) is uniform distribution, X(t) > is a weighting function, and Pr. o a, xo) is the transition kernel from xo to xt. common practice is to use linear drift ft) such that Pajo ( (z (20)
an analytic Gaussian distribution N ((a I) where a =elo f(r)dr of af fo g 2 dr is defined as the noise schedule (28). The resulting score predictor se t) can replace the true score function in Eqn. (2 and (3) obtain the empirical diffusion SDE and ODE, which can be simulated
of
by various SDE or ODE solvers [55 38 39 16 70
2.2 Consistency Models
Given a trajectory ]T= with a fixed starting timestep e of a PF-ODE, consistency models [58] aim learn the solution of the PF-ODE f c, also known as the consistency function, defined as h : (æt, t) + x., The optimization process for consistency models contains the online network he and reference target network hg-, where A refers with operation stopgrad, i.e., 0 stopgrad ()) The networks are hand-designed to satisfy the boundary condition he (æz, which can be typically achieved with proper parameterization on the neural network. For PF-ODE
C..
taking the form in Eqn. (3) with a linear drift (()),; the overall learning objective of consistency models can be described as:
E(EH(()))) Epo (a (m. (ht)ddho (ho (xx, f), ha- (œr,, r']],
(5)
where r(t) is a function that specifies another timestep r (usually with t > r), denotes some metric function with Ya, y d(æ, y) > 0 and d(æ y) 0 iff.
y. Here é, is a function that estimates x, + fr dr dr= which can be done by simulating the empirical diffusion ODE with pre-trained score predictor s4(x,,t) t) empirical score estimator of
=.-a. The corresponding
learning paradigms are named consistency distillation and consistency training, respectively.
2.3 Denoising Diffussoo Bridge Models
Given data pair sampled from an arbitrary unknown joint distribution (æz, y) ~ qdata (2r, y), x. y E Imm and let Zo x, denoising diffusion bridge models (DDBMs) [72] specify stochastic process
3

---

on a reference diffusion that ensures y almost surly via applying Doob's 's h-transform 13 47 process in Eqn. ( dæ, - [ 1) + (t) de log Pr) = y/ dt + (()) (æo, .T) = (x: y) ~ Idata=
(6)
where Pri( - y xxx) is the transition kernel of the reference diffusion process from / to T. evaluated at ay y. Denoting the marginal distribution of Eqn. (6) as {9 it can be shown that the forward bridge SDE in Eqn. ( is characterized by the diffusion distribution conditioned on both endpoints, that is, gejor (x; Fo. 2T) Pajor (Zo 2r), which is an analytic Gaussian distribution. A generative model can be obtained by modeling 9:/T (at prT = y), whose reverse SDE and PF-ODE are given by:
dæ; = [f(xe.t) - (²( (Va, log /1)(21//0 = y) Væ, log prgprl = y))x)))] dt + (t)
[ 2 V.,, log (ur y) Vn, logPr = ,/,,) dt. (8)
dx; f( (t))
The only unknown term remains the score function Va, log qn] ( = y). which can be estimated with a neural network s0(21 1, y) via denoising bridge score matching (DBSM):
mr [A(t)] se (x),
V log (1jor(x;
By
1
Replacing V.,, m, log le ( /zy = y) in Eqn. 77) and ( with the learned score predictor sp t,
would yield the empirical bridge SDE and ODE that could be solved for generation purposes.
3 Consistency Diffusion Bridge Models
In this section, we introduce consistency diffusion bridge models, extending the techniques of consistency models to DDBMs to further boost their performance and sample efficiency. Define the consistency function of the bridge ODE in Eqn. (8) as h (I), t, y) + x, with a given starting
timestep goal is to learn the consistency function using a neural network he( (, with the
following high-level objective similar to Equ. (5)
Enet((e,)),r=>(t: (t) qama (2. de or (r (n m. [A(t)d (ho t, y), he r.)]
(10)
To begin with, we first present a unified view of the design spaces such as noise schedule, network parameterization & precondition,
well
general ODE solver for DDBMs. This allows us
to: (1) decouple the successful practical designs of previous diffusion bridges from their different theoretical premises; (2) decouple the framework of consistency models from certain design choices of the corresponding PF-ODE, such as the reliance on VE schedule with Euler ODE solver of the original derivation of consistency models (58). This would largely facilitate the development of consistency models that utilize the rich design spaces of existing diffusion bridges on DDBMs in universal way. Then, we elaborate on two ways to train he based on different choices of i,,. consistency bridge distillation, and consistency bridge training, with the proposed unified design spaces.
3.1 A Unified View on Design Spaces of DDBMs
Noise Schedule
We consider the linear drift ft)) and define:
pf R²= / --(""(())r
0t = el (())dr
0e = (())dr
dr,
& R=
dr,
(11)
a²
o
which aligns with the common notation of noise schedules used in diffusion models by denoting ot axpt. Then we could express the analytic conditional distributions of DDBMs as follows:
uujor (ee No, ØT) Pujor
(xee Lo, ØT) = - (a:2T + b,00, I.
a
aupt
aPPiPi
(12)
where
by
PT
P7.
The form lt or is consistent with the original formulation of DDBM (72). Here, inspired by [6)
we opt to adopt a more neat set of notations for enhanced compatibility. As shown in Table [ with
such notations, we could easily unify the design choices for diffusion bridges [3 72 6 that have shown effectiveness in various tasks and expeditiously employ consistency models on top of them.

---

Table 1: Specifications of design spaces in different diffusion bridges. The details of network parameterization are in Appendix due to space limit.
Brownian Bridge
12SB (3 default
DDBM
72 VE
Bridge-TII
6
defauit
VP:
grnax
vP
Schedule
1
500 (1)
0
o²
(nn 10/21 11)²
Pu + Pd
3o + Bdl
1
6 e- goue
of
oPt
fo (r))d
But B.l
Bor- A.at?
6g
1
aPt
To (r)dr pi P
But B8af
Pr
o 0 ()
A Aie
T 12
Pi PE
p Pi
Parameters
10-
= 80
0.01 Ba 49.99
0.01
91 VaTiv
Aa = 19.99
0.1
81 = 0.3/1.0
Parameterization by Network F) Data Predictor Dependent on Training
Fo
Casp(()az + (o
Fg
Though 12SB is built on discrete time schedule for 7 1000 timesteps, it can be converted to a continuous time schedule
on E [0, approximately mapping f/(T" 1).
The authors chunge to the sume VP schedule as Bridge-TTS with parumeters 3 0.1, 84
revised version of their pupet.
Network Parameterization & Precondition In practice, the neural network Fa in DBMs does not always directly regress to the target score function; instead, it can predict other equivalent quantities, such as the data predictor 9:J07. Meanwhile, the inputs and outputs of the network Fa could be rescaled for better-behaved optimization process, known as the network precondition. As shown Table we could consistently use 2o as the prediction target with different choices of network precondition to unify the previous practical designs for DBMs.
2c-a.er for Gaussian ((aa2r + by cf like b.
PF-ODE and ODE Solver The validity of a consistency model relies on an underlying PF-ODE that shares the same marginal distribution with the forward process. the original DDBM paper (72. the marginal preserving property of the proposed ODE is justified following an analogous logic from the derivation of the PF-ODE of diffusion models [60] with Kolmogorov forward equation. However, its validity suffers from doubts there is singularity at the deterministic starting point TT. Here, we provide a simple example to show that the ODE can indeed maintain the marginal distribution as long as we use a valid stochastic step to skip the singular point and start from T - y for any >> 0.
Example 3.1. Assume (20,21):
and consider a simple Brownian Bridge between two fixed points
dr. = I1 8t dl. + dw,, 1-t
(13)
with marginal distribution quoo (z, (oo æ1) = =N((1= t)zo + tx t(1 6(1-t)) t)). The ground-truth reverse
SDE and PF-ODE are given by:
3D
dx.
dt + dw
(14)
1- 2t
dr, 26(1 t)
21
1) 01
2t 20 dt.
(15)
2(1
Then first simulating the reverse SDE in Eqn.
14
from 1 = 1 to t = 1 for some E (0,1) and
then starting to simulate the PF-ODE in Eqn.
75
will preserve the marginal distribution.
The detailed derivation can be found in Appendix. model based the bridge ODE needs to be set E [c, >] for some pre-specified c, > 0. Additionally, the marginal preservation of the bridge ODE for more general diffusion bridges can be strictly justified by considering non-Markovian variants, as done in DBIM 669
B
Therefore, the time horizon of the consistency
Another crucial element for developing consistency models is the ODE solver, as solver with lower local error would yield lower error for consistency distillation, well as the corresponding
5

---

consistency training objectives [7]. Inspired by the successful practice of advanced ODE solvers based on the Exponential Integrator (EI) 22 in diffusion models, we present a first-order bridge ODE solver in similar fashion:
Proposition 3.1. Given an initial value at time t > O. the first-order solver of the bridge ODE in Eqn. ( from t to E [0, 8) with the noise schedule defined in Eqn. (17) is:
Or De + Or
[
PePePr Pt
xo 1, + (0² PePePr
y
(16)
OcPtPt
PT
or
We provide detailed derivation in the Appendix B.1 Typically, an El-based solver enjoys lower discretization error and therefore has better empirical performance [16] 38 39
67 70]. Another
notable advantage of this general form solver, as we will show in Section 3.3 is that it could naturally
establish the connection between consistency training and consistency distillation for any noise schedules that take the form in Eqn. (IT), eliminating the dependence of the VE schedule and the corresponding Euler ODE solver in the common derivation [58]
3.2 Consistency Bridge Distillation
Analogous to consistency distillation with the empirical diffusion ODE, we could leverage a pre- trained score predictor (0 L, y) R Vz, log q8/ (ær (TT = y) to solve the empirical bridge ODE to obtain u,. i.e., &, Le (It,, L, T; y), where Ti is the update function one-step ODE solver with fixed SΦ. We define the consistency bridge distillation (CBD) loss as:
LEDD B y) (1) E pawa (o. -o.olo/o=.r-_ (A(t)d (ho y), 2),hh,. hg (& (z,, t, r))),
(17)
where is sampled from the uniform distribution over [e, T - 7], r(t) is a function specifies another timestep r such that e S < t with Atmax := max, {t - r(t)} and At := min, r(t)), X(t) is positive weighting function. d is some distance metric function with Vax, y d(x, y) 2 0 and d(x, y) iff. x = y. and A- stopgrad Similarly to the case of consistency distillation in empirical diffusion ODEs, we have the following asymptotic analysis of the CBD objective:
Proposition 3.2. Given Atmax max, {t r()} and let hal....)) be the consistency function of lipschitzfunction, the empirical bridge ODE taking the form Eqn. (8). Assume he a Lipschitz function, i.e., there
exists L>>000000000 0, such that for all E T [-) 7] 31, I2 y, we have | (x1, t,y) ho ( f, y)l12 < I.| - 22l12- Meanwhile, assume that for all t,r [,,T y ~ qdata (3) == E. [gaata (2., y)]. the ODE solver Lo(.,t, y) has local error uniformly bounded by O((t- r)tt1) with p 2 1. Then,
if CBD O, we have: suPt, [lho (ar, I. y) h I. y) || (((Amaxx)) max )P)
The vast majority of the analysis can be done by directly following the proof in 58 with minor differences between the overlapped timestep intervals (1, r(1) for i [c, T- used Eqn. (77 and the fixed timestep intervals (f, J.y used in (58). We include it in Appendix B.5 for completeness.
In this work, unless otherwise stated, we use the first-order ODE solver in Eqn. (16) as 2.
3.3 Consistency Bridge Training
In addition to distilling from pre-trained score predictor 8g. consistency models be trained [58 57 or fine-tuned [15] by maintaining only one set of parameters 0. To accomplish this, we could leverage the unbiased score estimator:
VR, log (x = y) Exo Ve, log Gejor (x [Fo. xT) Dt. BT y).
(18)
that is, with a single sample (at, yy) ~ Adaaa and æt ~ Qujor (we (20 x, ØT = y). the score Vz, log qe estimation of & into the one-step ODE solver के in Eqn. (17) with the transformation between data and score predictor 2 m tewr we can obtain alternative क, that does not rely on the pre-trained sg for any noise schedule taking the form in Eqn. 11] as follows (detail in Appendix B.3
= y) can be estimated with Væ, log quor (xy Do. 2T). Substituting such an
b,
f = 2(xx, t.r, x, 4) a + b, + c,
(19)

---

where Cr, br: Or are defined as in Eqn. (11 and 2 = 22 ---att-baa b, ~ (O,, Based on this
instantiation of Lr,, we define the consistency bridge training (CBT) loss as:
CBT
(20)
BnEM(GTT-)))-m0 ()) (2.1) [A(t)d (holary - b, + c, y), hg (a, + b, + c, y))),
where i, (r(-),((),, 0-1 are defined the same in Eqn. 17 and (0, I) is shared Gaussian noise used in both he and h We have the following proposition demonstrating the connection between Calmas and LCED with the first-order one-step ODE solver:
Proposition 3.3. Given ms max r(t)} and assume d, f.9 are twice continu- ously differentiable with bounded second derivatives, the weighting function X(-) is bounded, and BIIV æ, log 91fr (x+ < Meanwhile, assume that LCmn employs the one-step ODE solver in Eqn. (76) with ground truth pre-trained score model, i.e., Vt E [c, - 7 y ~ 9data 86 (1 f, y) = Ve, m, log (xp(( = y). Then, we have: L AAlma ol (m)a)))
CBBT
The core part of our analysis also follows (58). except the connection between the CBD & CBT objective relies on the proposed first-order ODE solver and the estimated & in Eqn. 199 with the general noise schedule for DDBM. We include the details in Appendix B.6
3.4 Network Precondition and Sampling
Network Precondition First, focus on enforcing the boundary condition (a c, y) x. of our consistency bridge model, which can be done by designing a proper network precondition. Usually, variable substitution f = t could work in most cases. For example, for the precondition
for 1SSB in Table we have x. + Fg x. V g²(r) Te. Also, the common "EDM" (24] style precondition used in DDBM also satisfies (E) and cou((E) = 0. We also give universal precondition to satisfy the boundary conditions based on the form of the ODE solver in Eqn. (16) in Appendix B.4 cope with the case where the variable substitution is not applicable.
Sampling As explained in Section 3.1 the PF-ODE only well well-defined within the time horizon (0, T). Hence, the sampling of CDBMs should start with Ty-- ~ y). which can be obtained by simulating the reverse SDE in Equ. 7 from - 7. Here we opt to use one first-order stochastic step, which is equivalent to performing posterior sampling, i.e., TT-y 4TT-/OT (1T--11 = h T. y) Dr = y). This sampling approach defaults to two NFEs (Number of Function Evaluations). which is aligned with the practical guideline that employing two-step sampling in CM allows for a better trade-off between quality and computation compared to other treatments such scaling up models [15.. We could also alternate a forward noising step and a backward consistency step multiple times to further improve sample quality consistency models do.
0 S t S T y y for some Gr /T-2/T (TT-)/2)
4 Experiments
4.1 Experimental Setup
Task, Datasets, and Metries In this work, we conduct experiments for CDBM on image-to-image translation and image inpainting tasks with various image resolutions and scales of the data set. For image-to-image translation, we use the EdgessHaandbags 23 with 64 x 64 pixel resolution and DIODE-Outdoor [62] with 256 x 256 pixel resolution. For image inpainting, we choose ImageNet [9] 256 256 with center mask of size 128 x 128. Regarding the evaluation metrics, we report the Fréchet inception distance (FID) 19 for all datasets. Furthermore, following previous works [33 72) we measure Inception Scores (IS) 3 LPIPS 68) and Mean Square Error (MSE) for image-to-image translation and Classifier Accuracy (CA) of pre-trained ResNet50 for image-inpaintion The metrics are computed using the complete training set for Edges Handbaas and Outtor, and a validation subset of 10,000 images for ImageNet.
Training Configurations We train CDBM two ways: distill pre-trained DDBM with CBD or fine- taiig DDBM with CBT. We keep the noise schedule and prediction target of the pre-trained DDBM
7

---

unchanged and modify the network precondition to satisfy the boundary condition. Specifically, we adopt the design space of DDBM- VP and IB in Table on image-to-image translation and image inpainting, respectively. We specify complete training details in Appendix
Specification of Design Choices We illustrate the specific design choices for CDBM. In this work, we use & E [e, - Y and set = 0. 0.0001, 0.001 and sample & uniformly during training. We employ two different sets of the timestep function r(t) and the loss weighting ((t), also named the training schedule for CDBM. The first, following [58], specifies constant quantity for At = t- r(t) The constant gap At treated as hyperparamettr and we search it among {1/9, 1/18, 1/36, 1/60, 1/80, 1/ /20).. The other employs r(t) that gradually shrinks r(t) during the training process and a loss weighting of X(t) r-r(z} which enjoys
with simple loss weighting of A(t)
better trade-off between faster convergence and performance [58] 57 15 Following [[5], we use a sigmoid-style function r(() = ((1 q quieawss] (1 + I+ ). where iters is the number of training
iterations, 9, 8, k b are hyperparameters. We use 2, k = 8, and tune b E {1, 2,5, 10, 20, 50} and E (5000, 10000}.
Table Quantitative Results on the
Translation Task
Edges Handbags (64 x 64)
DIODE-Outdoor (256 256)
FID
LPIPS
MSE
FID |
1
LPIPS J
MSE
Pix2Pix [23
74.8
4.24
0.356
0.209
82.4
4.22
0.556
0.133
DDIB 61
186.84
2.04
0.869
1.05
242.3
4.22
0.798
0.794
SDEdit 41
26.5
3.58
0.271
0.510
31.14
5.70
0.714
0.534
Rectified Flow 35
25.3
2.80
0.241
0.088
77.18
5.87
0.534
0.157
PSB 33
7.43
3.40
0,244
0.191
9.34
5.77
0.373
0.145
DDBM [72
(NFE=115)
1.83
3.73
0.142
0.0402
4.43
6.21
0,244
0.0839
DDBM (ODE-I, (.EE22) NFE=2)
6.70
3.71
0.0968
0.0037
73.08
6.67
0.318
0.0118
DDBM (ODE-I, NFE=50)
1.14
3.62
0.0979
0.0054
3.20
6.08
0.198
0.0179
DDBM (ODE-1, 1, NFE=100)
0.89
3.62
0.0995
0.0056
2.57
6.06
0.198
0.0183
CBD (Ours, NFE=2)
1.30
3.62
0.128
0.0124
3.66
6.02
0.224
0.0216
CBT (Ours, NFE=2)
0.80
3.65
0.106
0.0068
2.93
6.06
0.205
0.0181
Table Quantitative Results on the Image Inpainting Task
DDBM (DDE-1:
FID
8
CBT
ImageNet (256 x 256)
FID
CA 1
Center mask 128 x 128
NFE
DDRM 25
24.4
62.1
Figure 2: NFE-FID plot of CDBM and DDBM on
HIGM
56
7.3
72.6
ImageNet 256 x 256
DDNM
65
15.1
55.9
Palette
49
6.1
63.0
14
41-1.
2t-1.00
CDSB
52
50.5
49.6
13
23-156
85-1022
SB
33
4.9
66.1
DDBM (ODE-I, NFE=2)
17.17
59.6
10
DDBM (ODE-1, NFE=10)
4.81
70.7
CBD (Ours, NFE=2)
5.65
69.6
Training Iterations (x1000)
Training Inerations (x1000)
CBD (Ours, NFE=4)
5.34
69.6
(a) CBD
(b) CBT
CBT (Ours, NFE=2)
5.34
69.8
CBT (Ours, NFE=4)
4.77
70.3
Figure 3: Ablation for hyperparameters of CDBM
4.2 Results for Few-step Generation
We present the quantitative results of CDBM on image-to-image translation and image inpainting tasks in Table 2 with the first-order ODE solver Eqn. (T6) as our main baseline (i.e., "DDBM (ODE-1)"). We report the performance of the baseline DDBM under different Number of Function Evaluations (NFE) as
and and Table 3 We adopt DDBM the same noise schedule and network architecture,

---

LOWA
Condition
DDBM, NFE-100
Condition
DDBM, NFE-100
Condition
DDBM, NFE-8
DDRM, NFE-10
DDBM, NFE=2
CDBM, NFE=2
DDBM, NFE=2
CDBM NFE=2
DDBM, NFE=2
CDBM, NFE=2
CDBM. NFE=10
Figure 4: Qualitative demonstration between DDBM and CDBM.
Figure 5: Example semantic interpolation result with CDBMs
reference for the sampling acceleration ratio (Reduction factor NFE to achieve the same FID) of CDBM. Following 772 33). report the result of other baselines with NFE > 40, which consists of diffusion- methods, diffusion bridges with different formulations, or samplers. We mainly focus on the two-step generation scenario for CDBM, which is the minimal NFEs required for CDBM using the sampling procedure described in Section 3.4
For image-to- immima translation, as shown in Table. 2 we first observed that our proposed first- order ODE solver has superior performance compared DDBM (72). On top of that, CDBM's FID at NFE = 2 is close to even better than DDBM's at NFE around 100 with the advanced ODE solver, achieving sampling speed-up around 50x. This can be corroborated by the qualitative demonstration in Fig. | where CDBMs drastically reduce the blurring effect on DDBMs under few-step generation settings while enjoying realistic and faithful translation performance.
the hybrid high-order sampler used in
For image inpainting, as shown in Table. the baseline ODE solver for DDBM achieves decent sample quality at NFE 10. For CDBM, as shown in Fig. 2 the acceleration ratio is relatively modest in such large-scale and challenging dataset, achieving close a 4x increase in sampling speed. Notably, CBT's FID at NFE matches DDBM at NFE = 10. Moreover, we find that CDBMs have better visual quality than DDBM given the same computation budget, as shown in Fig. 4 and Appendix D which illustrates that CDBM yields a better quality-effccinncy trade-off.
Meanwhile, we observe that fine-tuning DDBMs with CBT generally produces better results than CBD) in all three data sets, demonstrating fine-tuning pre-trained score model to a consistency function is more promising solution with less computational and memory cost compared distillation, which is consistent with recent findings [1].. We also conducted an ablation study for CBD and CBT under different training schedules (i.e., the combination of the timestep function r(t) and the loss weighting X(t)) on ImageNet 256 x 256. As shown in Fig. 3 for a small timestep interval - r(t), e.g.,
small At Fig. 3a or large b in Fig. 3b (detail in Appendix C.2). the performance is generally better but also suffers from training instability, indicated by the sharp increase in FID during training when AL- = 1 1/120 and 50. While for large timestep interval, the performance at convergence is usually worse. In practice, we found that adopting the training schedule that gradually shrinks r(t) - with b = or 50 with CBT could work across all tasks, whereas CBD generally needs meticulous design for At or b to ensure stable training and satisfactory performance.
4.3 Semantic Interpolation
We show that CDBMs support performing downstream tasks, such as semantic interpolation, similar to diffusion models (55]. Recall that the sampling process for CDBM alternates between consistency

---

function evaluation and forward sampling, we could track all noises and the corresponding timesteps re- re-generate the same sample. By interpolating the noises of two sampling trajectories, we can obtain a series of samples lying between the semantics of two source samples, as shown in Fig. 5 which demonstrates that CDBMs have a wide range of generative modeling capabilities, such as sample diversity and semantic interpolation.
5 Conclusion
In this work, we introduce consistency diffusion bridge models (CDBMs) to address the sampling inefficiency of DDBMs and present two frameworks, consistency bridge distillation and consistency bridge training, to learn the consistency function of the DDBM's PF-ODE. Building on unified view of design spaces and the corresponding general-form ODE solver, CDBM exhibits significant flexibility and adaptability, allowing for straightforward integration with previously established successful designs for diffusion bridges. Experimental evaluations across three datasets show that CDBM can effectively boost the sampling speed of DDBM by 4x to 50x. Furthermore, it achieves the saturated performance of DDBMs with less than five NFEs and possesses the broad capacity of generative models, such sample diversity and semantic interpolation.
Limitations and Broader Impact While significantly improving the sampling efficiency in the datasets we used, it remains to be explored how the proposed CDBM, along with the DDBM formulation, performs datasets with larger-scale or more complex characteristics. Furthermore, consistency model paradigm typically suffers from numerical instability and it would be a promising research direction to keep improving CDBM's performance from an optimization perspective. With enhanced sampling efficiency, CDBMs could contribute more energy meegy--fffccent deployment of generative models, aligning with broader goals of sustainable AI development. However, it could also lower the cost associated with the potential misuse for creating deceptive content. We hope that our work will be enforced with certain ethical guidelines to prevent any form of harm.
Acknowledgments and Disclosure of Funding
This work was supported by the National Science and Technology Major Project (2021ZD01 10502). NSFC Projects (Nos. 62350080, 62106122, 92248303, 92370124, 62350080, 62276149, U2341228, 62076147), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J. Zhu was also supported by the XPlorer Prize.
References
[1] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):3133326, 1982.
[2] Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233 2024.
[3] Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv: 2018.
[4] Mari Paz Calvo and César Palencia. A class of explicit multistep exponential integrators for semilinear problems. Numerische Mathematik, 102:367-381, 2006.
[5] Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference Learning Representations, 2021.
[6] Zehua Chen, Guande He, Kaiwen Zheng, Xu Tan, and Jun Zhu. Schrodinger bridges beat diffusion models on tex--to-peecch synthesis. arXiv preprint arXiv:2312.03491, 2023.
[7] Valentin De Bortoli, Guan-Horng Liu, Tianrong Chen, Evangelos A Theodorou, and Weilie Nie. Augmented bridge matching. arXiv preprint arXiv:2311.0697. 2023.
10

---

[8] Valentin De Bortoli, James Thortoo,, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score- generative modeling, Advances in Neural Information Processing Systems, 34:1 34:17695-17709 2021.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. IEFE, 2009.
[10] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pages 8780- 8794, 2021.
[11] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
[12] Laurent Dinh, Jascha Sohl-Dicksteii Dickkseinn and Samy Bengio. Density estimation using real nvp. In International Conference Learning Representations, 2016.
[13] Joseph L Doob and Doob. Classical potential theory and its probabilistic counterpart, volume 262. Springer, 1984.
[14] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. arXiv preprint arXiv:2403. 033000 2024.
[15] Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and Zico Kolter. Consistency models made easy. arXiv preprint arXiv:2406. 14548, 2024.
[16] Martin Gonzalez, Nelson Fernandez, Thuy Tran, Elies Gherbi, Hatem Hajri, and Nader Mas- moudi. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models. arXiv preprint arXiv:2305. /4267, 2023.
[17] Ian J. Goodfellow, Jean Pouget-Ahadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, pages 2672-2680, 2014.
[18] Agrim Gupta, Lijun Yu. Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06661 2023.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 6626-6633 2017.
[20] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230: 2022.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pages 6840-6851, 2020.
[22] Marlis Hochbruck, Alexander Ostermann, and Julia Schweitzer. Exponential rosenbrock- methods. SIAM Journal on Numerical Analysis, 47(1):786 2009.
[23] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
[24] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative models. In Advances Neural Information Processing Systems, 2022.
11

---

[25] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models, In Advances Neural Information Processing Systems, 2022.
[26] Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Representations, 2024.
[27] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
[28] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, 2021.
[29] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[30] Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
[31] Liangchen Li and Jiajun He. Bidirectional consistency models. arXiv preprint arXiv:2403.18033 2024.
[32] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Representations, 2022.
[33] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. 12sb: Image-to-imane schrödinger bridge. In International Conference on Machine Learning, pages 22042-22063 PMLR, 2023.
[34] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference Learning Representations, 2019.
[35] Xingehao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. The Eleventh International Conference on Learning Representations, 2022.
[36] Xingchao Liu, Lemeng Wu, Mao Ye, and qiang liu. Learning diffusion bridges on constrained domains. In The Eleventh International Conference on Learning Representations, 2023.
[37] Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, pages 14429-14440 PMLR, 2022.
(38] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022.
[39] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01005, 2022.
[40] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04338. 2023.
[41] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
12

---

[42] Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image genera- tion and editing with text-guided diffusion models. In International Conference Machine Learning, pages 16784-16804. PMIR, 2022.
[43] Stefano Peluchetti. Diffusion bridge mixture transports, schrödinger bridge problems and generative modeling. Journal of Machine Learning Research, 24(374):1-51, 2023.
[44] Stefano Peluchetti. Non-denoising forward-time diffusions. arXiv preprint arXiv:2312.14589, 2023.
[45] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei. Diffusion-based voice conversion with fast maximum likelihood sampling scheme. In International Conference on Learning Representations. 2022.
[46] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[47] L. Chris G Rogers and David Williams. Diffusions, Markov processes and martingales: Volume
2, Itô calculus, volume 2. Cambridge university press, 2000.
[48] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[49] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-ing ---aaaae diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[50] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems, 2022.
[51] Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Amaud Doucet. Diffusion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36, 2024.
[52] Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Conditional simulation using diffusion schrödinger bridges. In Uncertainty in Artificial Intelligence, pages 1792-1802. PMLR, 2022.
[53] Jascha Sohl- Sich-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[54] Vignesh Ram Somnath, Matteo Pariset, Ya-Ping Hsieh, Maria Rodriguez Martinez, Andreas Krause, and Charlotte Bunne. Aligned diffusion schrödinger bridges. In Uncertainty in Artificial Intelligence, pages 1985-1992 1995. PMLR, 2023.
[55] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. In International Conference on Learning Representations, 2021.
[56] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guicee diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
[57] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024.
[58] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference Machine Learning, pages 32211-32252. PMLR, 2023.
[59] Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Advances Neural Information Processing Systems, volume 34, pages 1415-1428, 2021.
13

---

[60] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. ccore-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[61] Xuan Su, Jiaming Song. Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. arXiv preprint arXiv:2203.08388 2022.
[62] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv: 0908,0046,, 2019.
[63] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation,, 23(7):1661-1674 2011.
[64] Xiang Wang. Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprini arXiv:2312.09109 2023.
[65] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023.
[66] Yuji Wang, Zehua Chen, Xiaoyu Chen, Jun Zhu, and Jianfei Chen. Framebridge: Improving image-to-video generation with bridge models. arXiv preprint urXiv:2410.1533 15371, 2024.
[67] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations, 2023.
[68] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[69] Kaiwen Zheng, Guande He, Jianfci Chen, Fan Bao, and Jun Zhu. Diffusion bridge implicit models. arXiv preprint arXhv:2405.15883 2024.
[70] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[71] Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning. pages 42363 42389, PMLR, 2023.
[72] Linqi Zhou, Aaron Lou, Samar Khanna, and Stefano Ermon. Denoising diffusion bridge models. arXiv preprint arXiv:2309. 16948, 2023.
14

---

A Related Works
Diffuusion Bridges Diffusion bridges [44 36 33 54
are an emerging class of gen- erative models with attractive flexibility in modeling the stochastic process between two arbitrary distributions. The flow matching 32]. and its stochastic counterpart, bridge matching 44 assume the access of a joint distribution and an interpolation, or a forward process, between the samples, then, another SDE/ODE is learned to estimate the dynamics of the pre-defined interpolation, which can be
51
72
used for generative modeling from non-Gaussian priors [33 6
69 [66 In particular, the forward 72,
Among them. DDBM [72] focuses on learning the reverse-time diffusion bridge conditioned on particular terminal endpoint with denoising score matching, which has been shown to be equivalent to conducting a conditioned bridge matching that preserves the initial joint distribution 77. Other works tackle solving the diffuion Schrödinger Bridge problem, such as using iterative algorithms [8] [1 43]. In this work, we use a unified view of design spaces on existing diffusion bridges, in particular, bridge matching methods, to decouple empirical choices from their different theoretical premises and properties and focus on developing the techniques of learning the consistency function of DDBM's PF-ODE with various established design choices for diffusion bridges.
process can be constructed via Doob's 's fi-transfrrm 44 36
Consistency Models Recent studies have continued to explore the effectiveness of consistency models [58). For example, CTM [26) proposes to augment the prediction target from the starting point to the intermediate points along the PF-ODE trajectory from the input to this starting point. BCM [3 additionally expands the model to allow direct mapping at the PF-ODE trajectory points in both forward and reverse time. Beyond different formulations, several works aim to improve the performance of consistency training with theoretical and practical insights. iCT [57] systematically examines the design choices of consistency training and presents improved training schedule, loss weighting, distance metrics, etc. ECT [15] further leverages the insights to propose novel practical designs and show fine-tuning pre-trained diffusion models for learning consistency models yields decent performance with much lower computation compared to distillation. Unlike these works, we focus on constructing consistency models top of the formulation of DDBMs with specialized design spaces and a sophisticated ODE solver for them.
B
Additional Details for CDBM Formulation, CBD, and CBT
B.1 Derivation of First-Order Bridge ODE Solver
We first review the first-order ODE solver in Section
Proposition 3.1. Given an initial value x; at time >> O. the first-order solver of the bridge ODE in Eqn. 88 from t to E [0 with the noise schedule defined in Eqn. (11) is:
arPrPr xt + Or FT,
[
PePrPr Pt
xo(x,,, t, y)
PiP (
y
(16)
a:PeP:
p
Pt
aT
Recall the PF-ODE of DDBM in Eqn.
8 with a linear drift f())a)
dæ:- = [(()a= 0²(() V/ log (ar = y) Vn, logPru(<T = y y) dt. (21)
Also recall the noise schedule in Eqn. ITT and the analytic form of P:jo and pri in diffuusion models:
Pejo (x,, an) = N (a Prio(ar (ax) = N
ar ( ar, of - n2)1)
-
uejor (xt (zo, aT)
Pttorr (m, (zo, 2T) = (a.ar + b. 2
(22)
G b
af ae
where
a. P
ot
We thus have the corresponding score functions and the score-data transformation for so that predicts V ae log o
0c 049
log (TT = y)xx))
(23)
appp
15

---

Va, log q//or (2t (20, ar - y) 3t (a p/30 - 3/04
(24)
0 OPPPEE/ 14
so (xt, y) Dt (an ro 1r, 1, y) + C+ ar
(25)
aipi pi Pr
We use the data parameterization zo (2t, t. y) in following discussions. For PF-ODE in Eqn. (21)
substituting Va, log 4: ( y) with Eqn. (25) and substituting Pri: ( (x)) in with Eqn. 23
we have the following after some simplification:
(() 2t o.y dæ. = f())z= 2
20)))
St. a. a
dt.
(26)
af at
2
which shares the same form as the ODE in Bridge-TTS [6]. In the next discussions, we present
an overview of deriving the first-order ODE solver and refer the reader to Appendix A.2 in [6] for details.
We begin by reviewing exponential integrators 44] 22 a key technique for developing advanced diffusion ODE solvers [16
38 39 70 Consider the following ODE:
dæ [a(1)æ; + ((1) Fo ()]d,,
(27)
where Fe is n-th differentiable parameterized function. By leveraging the "variation-of-cosse formula, we could obtain a specific form of the solution of the ODE in Eqn. 27) (assume r < t)):
x = efi a(r)dr + J. el (())),
( r)d7,
(28)
The integral in Eqn. (28) only involves the function Fg. which helps reduce discretization errors.
With such a key methodology, we could derive the first-order solver for Eqn. 26). First, collecting the coefficients for æt, y, zo. we have:
dar (10)=
(²(() 207 Pt
(((()
()))),
(2(1)
20
24
20
y
20,
Do (r 4.
dc (29)
By setting:
9 (t)
g (t)
(t) g" (t)
(²(
a(t) = f(t)
b2(t)
20 Pi
20 pt
20
20,
with correspondence to Eqn.
28 the exponential terms could be analytically given by:
eli a(r)dr 0,0,ōr
ef a(e)ds a,0,ō, a,0,ō,
(30)
0,
The exact solution for Eqn.
29 thus given by:
I,, OrPrPr
ArPePr 2
S.
(T) ydr OrprPr 2
L: g2(r) xo x0( azptP.
(31)
a:ppt
app.p?
The integrals in Eqn. (without considering xa) can be calculated as:
2
Pr
Pt Pe)
[r
(T( dr
Pt
Pr
Q2ppp?
02020
PI
Pt
Pr.
Then, with the first order approximation (o(w,, solver in Eqn. [6.
wo(#, 8), we could obtain the first order
B.2 An Illustration Example of the Validity of the Bridge ODE
Recall the provided example in Section 3.1
Example 3.1. Assume T 1 and consider simple Brownian Bridge between two fixed points
(No, 21):
dx, 31 3t dt + dw
(13)
1
16

---

with marginal distribution 9410 To. 21) = (((1t)a + tx1, (1(1-()), The ground-truth reverse SDE and PF-ODE are given by:
24 To dt + duūs,
dx.
(14)
1 - 2t
1
dr,-
t) 21
28 2020 dt.
(15)
2f(1 t)
2(1
Then first simulating the reverse SDE in Equ. then starting to simulate the PF-ODE in Equ.
14
from t = 1 to t = y for some y E (0, and will preserve the marginal distribution.
75
and Proof. We first demonstrate the effect of the initial SDE step, according to Table T the expression of the relevant score terms in Eqn. 23) and Equ. (5), the ground-trun reverse SDE can be derived as:
It
TD
dx
dt + dıū,
Then, the analytic solution of the reverse SDE Eqn.
from time to time < t can be derived as:
1
1
dx
7
aedt
v0
dw;
1
1
d Ft
+
di
t
1
1
1
1
-
t Dt
To
s 76
(0011.
8
Let ( = 1, we have:
2. = (1 - 8 3) + 801 + V s(1 s)e,
i..., Fg has the same marginal as the forward process at time Similarly, the ground-truth PF-ODE can be derived as:
dr. 1 - 21
1
2f t) 2t +
2(1 - t) 21
2t 20 dt,
whose analytic solution from time to time a < can be derived as:
1 2
1
1
dæ
zt
2 dt
zodt
2t(1 - t)
2(1 t)
2t
1
t
1 t
d
Tidt
2/t(1 t)) Todt
VI(( 1)
2/t(1 - 1))3/2
1
1
1 1-8
1 1-t
/ /s(1 - s) 3z
20 +
10
Vt(1 - t)
Vs(1 -s)
Vt(1 - f)
/s(1 -
vt(1 - t)
F. Va(1 - 8)
vs(1 -5)
2 + 1- vs(1
Er +
(1 -t) xq.
Vt(1 t)
vt(1 - t)
vt(1 t)
When Z2 ~ N((1 - t)20 + tI t(1 - -t)), we have:
2, Vs(1- - 8)
/s(1 8) -
(1-t) t)zo + tx + vt(1 - v((1-t)c) +
Vi(1 - t) 11
Vt(1-1 -t)
+ 1 - 8 Vs(1 - 3) (1 - t)
vI(1 - 1)
(1 - 2)ze + s + / Vs(1 s)e.
Hence, once the singularity is skipped by a stochastic step, following the PF-ODE reversely will preserve the marginals in this case.
17

---

B.3 Derivation of the CBT Objective
(2) 2. I, 24 y) and an estimate of æ, 2 ( t. t, y) based on the pre-trained score predictor s with the first-order ODE solver in Eqn. (T6), our goal is derive the alternative estimation of क, 2(æ,, æ. y) a, b, + C, used in CBT, where R ~ (O, I) and ar, br, c. are defined in Eqn. 11 We begin with the estimator with pre-trained score model and first-order ODE solver:
Given (z, y) Idata (2: y), 24 ~ q 07 2t
drPoPr OsPiPt
Cry T
[0=
PePrPr Pt.
I xo( + p²
PePrPr
y
2r
Dt
(32)
PT
Pt
OTT
where a is the equivalent data predictor of the score predictor sΦ- By the transformation between data and score predictor EΦ 2.-a,2t-678 and substituting the score predictor sΦ with the score estimator az 9:jor (1
be
x0 = 2, 2T = y), we have:
I, OrPrPr
PT (
ar
PePePr
(pt
PePrPr Pt.
y
2t
x x
(33)
apPiP
aT
By expressing 2; aty b, czz, could derive the corresponding coefficients for 2, y, the right-hand side.
For y:
OrPrPr a:PIP1 at + Or
PtPrPr
OrPrPr aipe
+ or
PePtPr
Pr
arpp
Pt
asPiP: Pq.
aTPY
@rrPrPr Pt + ar
2
PtPrPr
Pt
ārp²
o
= ar,
(34)
where (1) due to the fact For ac:
āi
a
or
arPrPr br + Or asPePt
PePrPr
ayPrPr appi
Gr
(
PePrPr
a,Pr P7
Pr
bp. (35)
Po.
Pt
0sPiP: P
pr
Pt.
For z:
Pr
a = arPrPr a:P:Pt
arprPr Cr-
(36)
acPePt
apP:P: Pr
PT
Hence, we have the alternative model-fre freee estimator 2, = =((x, t, 3) = a, b, + C, where z ~ (O, I) the same Gaussian noise used in sampling æt = a.y + b,a: + O. Substituting L (,, l.r, r,y) y) in the CBD objective in Eqn. (17) with d(v,, r, x. y) gives the CBT objective in (m,,
Eqn. 20)
B.4 Network Parameterization
First, we show the detailed network parameterization for DDBM in Table. network as Fo, the data predictor æe (x. y) is given by:
Denote the neural
Co( 1 y) = Cskip (1)az + Coal (1) (1)ar,, Caoise (1), y),
(37)
where
(in(t) Vatoz bzo 2a,broor Ct.
Cour(t) = v o2(0700 o + oofeeintt),
(38)
(1kip()) = (bro0 + (x
Craise (1) = loog L.
and
a, P.P.PPP
a,
b,
C
of Var[xo), of Var[ær], our Cov(2g. x7
PT
Pf
(39)
18

---

It can be verified that, with the variable substitution & we have = 0, ba = 1, = 0 and thus have Cakip (c) and (é) O.
Meanwhile, we could generally parameterize the data predictor Do with the one-step first-order solver from / to 6, i.e.:
fe(2,, y)
OcpPPe a:piPt It +
Oke 2
[
pe
PePePe
xo t.y) + (0² PePePe y
(40)
PT
Pt
Pt
QT
which naturally satisfies f(x,, c, y)
B.5 Asymptotic Analysis of CBD
Proposition 3.2. Given Almax max, r(t]} and let hol:s-)) be the consistency function of the empirical bridge ODE taking the form in Eqn. (8). Assume hg is a Lipschitz function,, i.e., there exists L 0, such that for all [e, -7], 11, 22, y, we have [ t, y) he (22, y2l2 S L| 22112. Meanwhile, assume that for all r E [e,T
qdata (3) == Ex [ddata (2, y), the ODE solver E.(-,t, r, y) has local error uniformly bounded by O((t r)p+)))))) 1) with p 2 Then,
if CCDD = O, we have: supt,a.y [ t, y) - (2. t, y)ll2 0 ((())a)))))
Most of the proof directly follows the original consistency models analysis (58). with minor differ-
ences in the discrete timestep intervals (i.c., non-overlapped [58) and overlapped in ours) and the form of marginal distribution between (a(x)) for the diffusion ODE and 9t/r( (z: ær y) for the bridge ODE.
Proof. Given Laaan O, we have:
Eqdaua (aa)) mn w,ar v) Ba. [A(t)d (ho(xe,,,,) hg (# r.))) (41)
Since A(t) > O, and for E [c, 7.. (2 = x, yo = y) takes the form of
Naaxa + be with C > 0, which entails for any Fr, l E [e, T - >]. qur b = y)
Em (9yor (a (a0o = F, AT = y)] 0. Hence, Eqn. (41) implies that for all E [e, 7). (a. y)
(data (2. y), 3z qejor ( xo æ, æт y). have:
(ho 1, v) he (á (xx, I, r(), y), r()).)) = 0,
(42)
By the nature of the distance metric function d and the stopgrad operator, then have:
(T L y) = hg (, y), r(), = (,)) Define the error term timestep [e, 7 7) as:
(43)
et ho t, ho(
(44)
We have:
e, ho( 1, 4) 1o,
ho( t,) r(t), h (o) r(),,)
ho( 1, (()) r00, r(). ho(
+ he ( (t). ho(2
ho L, (())))))))))))))))))))))))))))))
(x) r(1): y) + ex(0)-
Since he is Lipschitz with constant L and the ODE solver aे ( (,, L, y) is bounded by O(( p))t1)
with p 2 1, we have:
||el|2 < [|e,() lla + [1
2,
= ||e,()]le l1 + O (())(t))
= [ee,))l2 0 (((t-r(t)ptt)
From the boundary condition of the consistency function, we have:
e, ho( e, y) (ho( E, = H. x. =
19

---

Denote Tm (t) applying on t for m times, since Atmin = min r(t) exists, there exists N
such that rn (t) for N. We thus have:
|eell2 eellz
Σ h=1
[(((r=-(()-ra(t))
N
((1=--(()-ra(pppp)))
(t) -m(()) (t))P)
(re-1 ((a(()((((()m
N (rk-1(t) - (x)())
O((Amaxx 1.)
k=1 +((AAmaxx (t -E)
S m((Amma) ) - E)
O(a(mmax))
B.6 Connection between CBD & CBT
Proposition 3.3. Given Atmax m r(t)) and assume h. f.g are twice continu- ously differentiable with bounded second derivatives, the weighting function X(-) is bounded, and
E|IV x, log 911r ( [ < <c. Meanwhile, assume that L employs the one-step ODE solver in Eqn. [6 with ground truth pre-trained score model, i.e., Vt E [c,T 7 y ~ 9data( 8Φ (2 f, y) m log (x: = y). Then, we have: Aatmax Ammax of Afmax).
LCBT
The core technique for building the connection between consistency distillation and consistency training with Taylor Expansion also directly follows (58). The major difference lies in the form of the bridge ODE and the general noise schedule & the first-order ODE solver studied in our work.
Proof. First, for a twice continuously differentiable, multivariate, vector-valued function h(x, 6, y). denote 0₂ (x t, y) the Jacobian of h over the k-th variable. Consider the CBD objective with first-order ODE solver in Eqn. (16) (ignore terms taking expectation for notation simplicity):
L E (X(t)d (2 t, y). ho- (k. r)x, + le r) + kt
(45)
P.p.Re where (t,r) a=PeR= (2, or p
ka(t,r) arpy (0z
P.e.Pe are coeffi- P.
asp.P.
ps
Pt
cients It, Ve y in the first-order ODE solver in Eqn. (T). as pre-trained data predictor. By
applying first-order Taylor expansion Eqn. (45). we have:
Atmax
CBD
E (X(t)d (ho t, y), he (a, + (ki 1)x, kg(t, - - ka(t, (r t), v))] E. [A())d (he (0,, 1, y), h (æ,, 1, y) + 0 h (x,, L, ()((k1 1) + k2 r)ws + (l,rr)
+ (,), 1) o(l r())]
Here the enror term w.t. the first variable can obtained by applying Taylor expansion on k(t, r) = k(t,t) Ozk(t, (r t) + o(It r|) with tt, 1 = 0, fo (t, = ka(t,t) = 0. By applying
20

---

Taylor expansion on d, we have:
Mma
(h0 hg (2,, y)) + (() he (x,,
d he (oe, t, ())((( 1) + k2 r) + r)(t, + Ozho (or, (r -) + o(It r(]]
E{X(t)d(ho(zz h,) (m)
ho + (Ed 1, y), hg
(0e, [ ha (.,, (k) (t, 1)xx - k2(( + ka(l, [E( ho 2z, t, y), ho (z,, (er,,)))82l 82h (za, y)(r t)} + Blo(It - -r]}
Then we focus on the term related the first-order ODE solver:
(k1 1) + (2 () + ka(t, ryy.
By the transformation between data and score predictor T 21-012r and substitute sø (Ie, t, y) with we log (x) x = y), we have:
be
(t, 1)xx + (2(t,r) 2t a.ær of V., log qi (x y)
+ k)(t,r)y
by
Next, substituting the score Va, log (x)r((2/227)) = y) with the unbiased estimator:
E V æ, log aqjor (zt, k2, 21 |0,, 2T = y) = EE xt (a.TT + byzo))
[1, 2r
of
We then have:
(ho y), he (ar, t, (8, =[((()0)d(hh( y). he (æ,, t, y))
(æt, t, (k) (t,r) 1)xx + kelt, r)as + (t,rl)
t, y)l
m (ara axr + eZ
De, DT = y
(t, (2)),
+ ka(,r)
@ (E(((t)))d(ho( (ax,, y), hg (I1, t. ()) (x,, t, 8)l
(k1 - 1)a,, (1, 0,FT c7 2r (a.ey +b
(t(()))))
b =E(X(t)) 1, y), hag (æx, L, [0) he (æe, t, [k( (t, ((,r))+ + (t, r)w + ka(trr) - =x,,,
where (i) comes from the law of total expectation. Then we apply Taylor expansion in the reverse direction:
= (E())) (z y). ho-
+ EEAA())rz((ha L, y), ho (at, t, (0
ha(trr x;]}
+ (B((() (x,, h
(E((1)) d( (ho (zre 1, y) ho (x 4,y)
+ -ho (we, t, (a
+ Ord ho (wr, y) ho
1, y) he
r)y xx]]
t, (|
= (E((() dho (rr, 1, 1) ho- (m,, + 0 (1) h22 ka(1, - ac
+ -2hoo (x),, 1, 9) ())]) E(o r[)}
(>(( (dd))d((() 1o (x,, f, v) ho-
hz r) ha
+ Efo(It - rr)}
(68) (E((()) (d ((o (a+y + b,r + az, v), ho- (a,y + + 9)) o(I r|)
SS + o(It r(),
where (ii) follows the derivation in Eqn.
33
Eqn. 36 and N I).
21

---

C Additional Experimental Details
C.1 Details of Training and Sampling Configurations
We train CDBMs based on a series of pre-trained DDBMs. For two image-to-image translation tasks, we directly use the pre-trained checkpoints provided by DDBM's 72 official repository For image inpainting, we re-train a model with the same SB style noise schedule, network parameterization, and timestep scheme in Table. [T] as well as the overall network architecture. Unlike the training setup in SB, our network is conditioned on 27 y following DDBM and takes the class information of ImageNet as input, which we refer to as the base DDBM model for image inpainting on ImageNet. The model is initialized with the class-conditional version ImageNet 256 x 256 of guided diffusion [TO] We used a global batch size of 256 and a constant learning rate of 1e-5 with mixed precision (fp16) to train the model for 200k steps. We train the model with NVIDIA A800 80G GPUs for 9.5 days, achieving the FID reported in Table. with the first-order ODE solver in Equ. (16).
For training CDBMs, we use a global batch size of 128 and learning rate of 1e-5 with mixed precision (fp16) for all datasets using 8 NVIDIA A800 80G GPUs. For the constant training schedule r(t) = t At, we train the model for 50k steps, while for the sigmoid-sty train the model for 6s steps, e.g., 30k or 60k steps, due to numerical instability when - r(t) is small. For CBD, training a model for 50k steps on a dataset with 256 x 256 resolution takes 2.5 days, while CBT takes 1.5 days. In this work, we normalize all images within [-1, 1] and adopt the optimizer.
training schedule, we
RAdam 27 34
For sampling, we use a uniform timestep for all baselines with the ODE solver and CDBM on two image-to-image translation tasks with e = 0.0001, T" = 1.0. For CDBM on image inpainting on ImageNet, we manually assign the second timestep T 0.1 and make other timesteps uniformly distributed between [c. T 0.1), which we find yields better empirical performance on this task.
C.2 Details of Training Schedule for CDBM
We illustrate the effect of the hyperparamter in the sigmoid-like training schedule r(t) t(1 (tteraaa (1 ) Note that we further manually enforce r() to satisfy Atmax and A min-
D- (RM0)) H6
Di 5. (8618) m1
d S. poart #1
D- jaare) -3
- jiett m4
5 jietty
2.30
050
0--10 (tart)) wD
+000 Horb
--00 (tero)
1- 38 (Meoa) HS
I- 50 (100000
U- 20.0000011
6.0.
8 -63
022-
630-
026
036
650
875
436
660
100
050
85
8.0
6.0
Figure 6: Illustration of the effect of the parameter b on the sigmoid-style training schedule.
C.3 License
We list the used datasets, codes, and their licenses in Table 4
Table 4: The used datasets, codes and their licenses.
Name
URL
Citation
License
Edges Baages-HHandin Handhugs
https: http:///ttthuubbcc/uoaaaaee http://wwwnnnn/ptcctcotraaee Cww.ellll-aaaanaaa and- -px-pxx
23
BSDD
DIODE-Outdoor
/ttps://dood-
62
MIT
ImageNet
https: //wwv inage-net.....
9
Quiddd--ifffusion
StEpa://githb....
wtt.ttttbccooommmm com/ openas/ /ww.eeddiiffuiiii.
10
MIT
FSB
httpa: 3ttp://github... www/WV/aas///////ooooooccooooost 1288
33
CC-BY-NC-SA-40
DDBM
httpa://gsthb..c wttpc//////oooooo/////0P
72
22

---

D
Additional Samples
Condition
Ground Truth
DDBM, NFE=2
DDBM, NFE=100
CDBM. NFE=2
Condition
Ground Truth
DDBM, NFF=2
DDBM, NFE=100
CDBM, NFE=2
Condition
Ground Truth
DDBM, NFE=2
DDBM, N0E=100
CDBM. NFE=2
Figure 7: Additional Samples for Edges - Handbags.
23

---

Condition
Ground Truth
DDBM, NFE=2
DDBM, NFE=
CDBM NFE-2
Condition
Ground Truth
DDBM, NFE-2
DDBM, NFE=
CDBM, NFE-2
Figure 8: Additional Samples for DIODE-Outdoor.
24

---

Condition
Ground Truth
DDBM, NFE-2
DDBM, NFE-8
CDBM NFE=2
DDBM, NFE=10
CDBM NFE=10
Figure 9: Additional Samples for ImageNet 256 256.
25

---

Condition
Different samples starts from ( y)
Figure 10: Demonstration of sample diversity of the deterministic ODE sampler.
Condition
CDBM NFE-2
I'SB, NFE=2
SB NFE=4
CDBM NFE-4
I'SB. NFE-8
Figure 11: Qualitative comparison between CDBM and I'SB baseline ImageNet 256 x 256. Note that here the base model of CDBM is different from the officially released checkpoint of JSB we used for evaluation.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The discussion is located in the "Limitations and Broad Impact" section after the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: Assumptions are provided with the propositions and the detailed derivation and proof is in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental configurations are included in Section and Appendix C The information provided is sufficient to reproduce the results that support our claims.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: The release of the code needs official procedure related to the authors' affiliation, which is not approved yet.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc
Moolic//guides/Coeeeubbiis itw.ooeeeesuoiiiiooooooooioogg for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttps/cccppplcccuui.eeaeaecccuea
Stteee///ooSSSemsssS.oi.a www.ssssiicggcoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experiment configuration and details are included in Section 4 4and Ap- pendix C which is sufficient to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The metrics for evaluating generative models are typically stable and do not require error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided the information in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurrpp:: http:://wwurrpp.cpppppiibeee
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The discussion is located in the "Limitations and Broad Impact" section after the main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
30

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work is conducted with common academic image datasets with model capability restricted with specific tasks. There is little chance posing risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Licenses for existing assets are listed in Appendix
C
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
31

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
32

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "lpRd9I8TBm", "reviewer": "Reviewer_NqhJ", "summary": "This paper proposes to combine the consistency model (CM) with diffusion denoising bridge models (DDBMs) to build a consistency diffusion bridge model (CDBMs), which includes two paradigms of consistency bridge distillation and consistency bridge training. The experiments are conducted for image inpainting and image-to-image translation tasks.", "questions": "Please see the weaknesses for specific questions.", "limitations": "The author discusses the limitations of the proposed method in the last section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ This paper is well written and organized, with a clear structure for readers to follow up easily.\n\n+ The unified view of design space of several different DDBM models is an interesting summary for a clear comparison over existing works.\n\n+ The experiment's results especially with NFE=2 are promising to be comparable with other models with larger NFE steps as shown in Figure 4.", "weaknesses": "- The proposed method is reasonable but may not be very surprising. This work looks more like a direct combination of CM and DDBM, which models have been introduced a lot in previous works, including consistency distillation and consistency training. \n\n- Question about the motivation to combine CM and DDBM. CM is designed to reduce NFE steps requirement for diffusion sampling. DDBM models can also be able to achieve this goal by learning transformation between two data distributions, such as what is claimed in I2SB, while sacrificing the advantage of unsupervised learning. What is the key motivation and necessity to combine these two together?\n\n- In experiments results, some common metrics like PSNR and SSIM are not reported. In Table 3, the compared methods are most unsupervised learning approaches which are not trained with paired data. In this case, more NFE steps are expected. Therefore, for a fair comparison, the NFE for the compared baselines in Table 2 and 3 may also need to be noted. \n\n- For qualitative results, the comparison with previous DDBM method under the same NFE would be appreciated, since some DDBM method such as I2SB and CDDB [1] also claimed enabling image generation or restoration within a very few NFE steps from 1-4. It would be important comparison to compare with these baselines under the same experiment settings including NFE.\n\n- Due to the requirement for paired data training, the proposed model needs to be retrained to different tasks, which are limited compared with diffusion-based methods. What is the computational complexity and parameter generalization robustness to retrain the model on a new task? \n\n\n[1] Chung, Hyungjin, Jeongsol Kim, and Jong Chul Ye. \"Direct diffusion bridge using data consistency for inverse problems.\" Advances in Neural Information Processing Systems, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and feedback on our work. We provide our responses as follows:\n\n> W1: The proposed method may not be very surprising. This work looks more like a direct combination of CM and DDBM, which have been introduced in previous works.\n\n**A**: While the high-level idea of utilizing CM to accelerate the sampling of DDBM might appear direct, we would like to emphasize this work still makes some valid technical contributions to the community that can not be accomplished by simply combining the existing techniques from previous works.\n\nFirst, we demonstrate the effectiveness and advantages of the presented first-order ODE sampler for DDBM, which could achieve competitive, even superior performance to strong baselines with a deterministic sampling trajectory. To the best of our knowledge, there is no other better candidate in previous works. For example, DDBM [1] finds that using the ODE sampler in EDM [2] yielded poor performance and proposed a hybrid sampler that alternates between ODE and SDE steps. And I$^2$SB [3] uses a stochastic posterior sampling scheme. Neither of them can naturally fit the CM paradigm.\n\nSecond, our presented unified framework of empirical design space (i.e., noise schedule, network parameterization, and preconditioning) can encompass a wide range of diffusion bridges, which is completely decoupled from their different theoretical premises. As a result, people could efficiently reuse the successful empirical designs of previous diffusion bridges in a unified way according to our framework. For example, in our paper, we retrain an image inpainting model with DDBM's mathematical formulation and I$^2$SB's noise schedule, network parameterization, and preconditioning that has comparable performance to the original I$^2$SB. And we can further build a consistency model on top of it in a unified manner.\n\n\n> W2: The key motivation and necessity to combine DDBM and CM is unclear.\n\n**A**: We thank the reviewer for this question. For DDBMs, the benefit of utilizing the paired data during training is not only reducing the NFE requirement for sampling but also further improving the saturated performance given numerous NFE steps over unsupervised diffusion models. For example, the DDBM baseline in [1] needs **hundreds** NFE steps to achieve the saturated performance in some image translation tasks, surpassing the saturated performance of diffusion models. We show that introducing CM to DDBM can achieve these SOTA results $\\sim 50 \\times$ faster. In our opinion, this is analogous to why we need CM on diffusion models. On the other hand, even though some diffusion bridges (e.g., I$^2$SB) can achieve decent performance in a few NFE steps (e.g, $\\ge 10$ steps), we believe that further reducing $2.5 \\times$ NFE steps still makes a valid contribution.\n\n> W3: Some common metrics like PSNR and SSIM are not reported in experimental results.\n\n**A**: Thank you for the suggestion. We will add these two metrics in the appendix of our revised paper. We show the results in the comment for a short overview.\n\n> W4: The NFE for the compared baselines in Table 2 and 3 may also need to be noted.\n\n**A**: Thank you for the suggestion, we will add the NFE of the baselines in the revised paper. Most results are directly taken from the DDBM [1] and I$^2$SB [3] and some of them only have a rough number of NFE steps. We put them in the comment below for a simple overview.\n\n> W5: For qualitative results, the comparison with previous DDBM method under the same NFE would be appreciated.\n\n**A**: We thank the reviewer for the suggestion. Regarding the performance under a few NFE of the I$^2$SB baseline, we want to address that the DDBM (ODE-1) baseline in Table 3 is our replicated version of I$^2$SB on the center image inpainting task. As demonstrated in the answer of W1, we train an image inpainting model with DDBM's mathematical formulation and I$^2$SB's noise schedule, network parameterization, and preconditioning that has comparable quantitative performance to the original I$^2$SB. We put the results of both models under the same NFE in the comment for a simple overview.\n\nAs shown in the table, our replicated model can achieve the reported FID of 4.9 in I$^2$SB paper with 10 NFE steps. On the other hand, I$^2$SB only provides checkpoints with a fully deterministic forward process (i.e., the \"OT-ODE\" in their paper), which does not align with the DDBM formulation. Hence, we choose our version as the main baseline. We also add a qualitative comparison between I$^2$SB and CDBM in the one-page pdf.\n\nBesides, it seems like the mentioned CDDB [4] paper still needs $>50$, even thousands of NFE steps to achieve the saturated performance, which, again, justifies our motivation of introducing CM to achieve comparable performance with much fewer NFE steps.\n\n\n> W6: What is the computational complexity and parameter generalization robustness to retrain the model on a new task?\n\n**A**: The cost of the paired data training stage is mostly concentrated on the base DDBM model. For reference, training a DDBM from scratch on the DIODE (256 $\\times$ 256) image translation task would need $\\sim500$k iterations. In our experiment that replicates I$^2$SB, it takes us about 200k iterations to *fine-tune* a DDBM for center inpainting from an unsupervised diffusion model on ImageNet 256 $\\times$ 256, which is far easier than learning the diffusion model on this dataset. Once we have the base DDBM model, the consistency distillation/fine-tuning typically takes $<100$k iterations using the data for training the DDBM, which is more computationally efficient compared to training the base DDBM model itself.\n\nFor generalization robustness and efficiency on a new task, it would be an interesting research direction to figure out whether there is, or how to build a foundation model (such as a diffusion model) that can be efficiently transferred to a diffusion bridge on various new tasks by leveraging paired data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOqYvz1AxY", "reviewer": "Reviewer_2JER", "summary": "Motivated by faster sampling time, the authors investigate consistency model techniques applied to denoising diffusion bridge models, a particular formulation of conditioned diffusion model whereby the forward process is a mixture of diffusion bridge SDEs, with a conditioned terminal point.\n\nIn particular, the authors adapt typical consistency model training regimed known as \"consistency model training\" - training the model from scratch -  and \"consistency model distillation\" - distilling from a pretrained diffusion (bridge) model.", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "As far as I am aware consistency methods have not been applied to diffusion bridge models, though the significance of this is unclear vs performing consistency training on diffusion models or flow matching models.\n\nThe empirical performance showcased by the author's implementation appears competetitive to the baselines the authors consider.", "weaknesses": "Denoising diffusion bridge models are a particular formulation of conditioned diffusion model whereby the forward process is given by the SDE of a diffusion bridge, with a conditioned terminal point. This has the benefit of data to data translation, but in my opinion the authors exaggerate the significance of this and I would not consider it \"a new family of generative models\" [line 4]. Given the consistency methods applied to diffusion models transfer trivially to diffusion bridge models, I am of the opinion that the novelty of the methodological contribution is quite limited.\n\nFurthermore I would also question the usefulness of data to data generative models which are deterministic. For any ill-posed inverse problem, practitioners would want to be able to sample from many possible uncorrupted sample given a single corrupted sample, including some example provided here such as for inpainting.\n\nFor consistency models or indeed any flow type model, a Gaussian is typically chosen as marginal which adds stochasticity and permits the PF ODE to match the diffusion model in marginal distributions. This is not the case for diffusion bridge models with conditioned terminal point, there is no longer any stochasticity and hence the ODE derived cannot match the marginals of the stochastic generative backward process. Hence I wonder if PF-ODE of a diffusion bridge model even makes sense in the same way as a diffusion model? Given all the consistency approaches rely on the deterministic component, I believe this is quite fundamental and needs to be addressed.\n\nThere are a number of related works that could be discussed. In particular Augmented bridge matching (https://arxiv.org/abs/2311.06978) which draws an equivalence between denoising diffusion bridge models and conditioned bridge matching.\n\n\nOther:\n>  tractable class of Schrödinger Bridge and simulation-free, non-iterative training procedure [32, 6]\n\n[32,6] consider regular bridge matching without any connection to an optimal coupling and hence although tractable just a bridge matching model and not a schrodinger bridge. This is a common mistake.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and feedback on our work. We would like to provide our responses as follows:\n\n> W1: The significance of DDBM is exaggerated as a new family of generative models\n\n**A**: Thank you for pointing this out and we agree that DDBM itself can not be called as a family of generative models. We will revise this to \"a new formulation of generative modeling\".\n\n\n> W2: Given the consistency methods applied to diffusion models transfer trivially to diffusion bridge models, I am of the opinion that the novelty of the methodological contribution is quite limited\n\n**A**: While the high-level idea and core techniques of utilizing CM to accelerate the sampling of DDBM might appear direct, we would like to emphasize this work still makes some valid technical contributions to the community.\n\nFirst, we demonstrate the effectiveness and advantages of the presented first-order ODE sampler for DDBM, which could achieve competitive, or even superior performance to strong baselines with a deterministic sampling trajectory. To the best of our knowledge, there is no other better candidate in previous works. For example, DDBM [1] finds that using the ODE sampler in EDM [2] yielded poor performance and proposed a hybrid sampler that alternates between ODE and SDE steps. And I$^2$SB [3] uses a stochastic posterior sampling scheme. Neither of them can naturally fit the CM paradigm.\n\nSecond, our presented unified framework of empirical design space (i.e., noise schedule, network parameterization, and preconditioning) can encompass a wide range of diffusion bridges, which is completely decoupled from their different theoretical premises (e.g., either the method belongs to bridge matching or conditioned bridge matching). As a result, people could efficiently reuse the successful empirical designs of previous diffusion bridges in a unified way according to our framework. For example, in our paper, we retrain an image inpainting model with DDBM's mathematical formulation and I$^2$SB's noise schedule, network parameterization, and preconditioning that has comparable performance to the original I$^2$SB. And we can further build a consistency model on top of it in a unified manner.\n\n> W3: I would also question the usefulness of data to data generative models which are deterministic. For any ill-posed inverse problem, practitioners would want to be able to sample from many possible uncorrupted sample given a single corrupted sample, including some example provided here such as for inpainting.\n\n**A**: We would like to argue that the deterministic sampling process of data to data generative models, such as the presented first-order ODE sampler for DDBM, does not necessarily produce a deterministic sample given a fixed input. For the inverse problem, **it does produce diverse uncorrupted samples given a single corrupted sample.** This is because of the fact that the PF-ODE of DDBM is only well-defined for $0 \\le t < T$. Thus, the valid way to simulate the PF-ODE is to start from $q_{T-\\gamma|T}(x_{T-\\gamma}|x_T)$ for some $\\gamma > 0$, which introduces stochasticity and will result in sample diversity given a fixed $x_T$. In practice, once we have a plausible approximation of initial distribution, the ODE sampler will work similarly to its counterpart in noise-to-data generative models. We also provide a concrete illustration of the sample diversity with our first-order ODE sampler in the image inpainting task on the one-page pdf in the common response.\n\nOn the other hand, deterministic samplers may also have other advantages such as improved sampling efficiency (while ensuring sample diversity). For example, in our replicated image inpainting model with DDBM's formulation, the first-order ODE sampler is notably better than a first-order SDE sampler when NFE is small (e.g., $<100$). Another piece of evidence is that I$^2$SB's officially released checkpoint also adopts a deterministic sampler that removes the stochasticity of posterior sampling.\n\n\n\n> W4: For diffusion bridge models with conditioned terminal point, there is no longer any stochasticity and hence the ODE derived cannot match the marginals of the stochastic generative backward process\n\n\n**A**: As we explained in W3 and addressed in DDBM [1], the PF-ODE derived by DDBM is well-defined and models the evolution of $q_{t|T}(x_t | x_T)$ when $0 \\le t < T$. The singularity caused by the fixed terminal point can (and should) be removed by using a stochastic initial distribution, and doing so will enable valid ODE sampling and consistency modeling. We will add some notes about this point in our revised version. Furthermore, we find that the consistency losses in the current version allow us to sample $t=T$ from $\\mathcal U(\\epsilon, T)$ and we will revise the timestep distribution to $\\mathcal U(\\epsilon, T - \\gamma)$ with a pre-specified $\\gamma > 0$.\n\n> W5: There are a number of related works that could be discussed. In particular Augmented bridge matching, which draws an equivalence between denoising diffusion bridge models and conditioned bridge matching.\n\n**A**: Thank you for your suggestion. Augmented bridge matching is an excellent work that reveals and discusses how conditioning on the terminal point changes the theoretical properties of bridge matching, notably pointing out that conditioned bridge matching (or DDBM) preserves the empirical coupling. We have already mentioned this work in our initial submission and will add more discussion about it and the bridge matching technique in our revised version.\n\n> W6: [32,6] consider regular bridge matching without any connection to an optimal coupling and hence although tractable just a bridge matching model and not a schrodinger bridge. This is a common mistake.\n\n**A**: We thank the reviewer for pointing this out. We will add the discussion about bridge matching and properly classify these works as bridge matching & conditional bridge matching procedures in the related works section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qsLTlzcOY", "reviewer": "Reviewer_7nJA", "summary": "In this paper, the authors combine Denoising Diffusion Bridge Models (DDBMs), that build a transport map between two arbitrary target distributions (that can be coupled) through a stochastic process, with recent advances on consistency techniques [1], that were originally designed for denoising diffusion models. The motivation herein is to speed up the inference process of DDBMs, that turns out to be costly due to a high number of neural network evaluations. Following the approach from [1], the authors propose to learn the *consistency function* of the probability flow associated to an arbitrary DDBM, which outputs a prediction of the target distribution given any input of the DDBM process $x_t$, for any time $t$. To do so, they derive a consistency loss, which can be seen as the natural extension of the loss of [1] to the bridge setting. Then, they propose two paradigms for this approach: either *consistency bridge distillation*, which assumes to have access to a pretrained DDBM model, and *consistency bridge training*, where no model is available. Their formulation encompasses popular designs of diffusion bridges such as the Brownian brigde, the Image-to-Image Schrödinger bridge [2], the original bridges designed for DDBMs [3]... The authors provide practical guidelines on the neural network parameterization and preconditioning, the design of the ODE solver and the consistency schedule. Finally, they conduct numerical experiments on high-dimensional datasets that demonstrate that the proposed consistency bridge model is faster than the original model while having equal or better generative performance. They notably find out that fine-tuning a pretrained DDBM with consistency training provides better performance under lower computational budget than distillation.\n\n[1] Consistency models. Song et al. 2023.\n\n[2] I2SB: Image-to-Image Schrödinger Bridge. Liu et al. 2023.\n\n[3] Denoising diffusion bridge models. Zhou et al. 2023", "questions": "- Could you give further details and intuition on the design of the consistency schedules $r(t)$, which seem to be crucial so that consistency works (even if these reasons are heuristic) ? In particular, could you give examples of schedules that may be adapted in theory but do not work in practice ? It still unclear for me which design should be the best depending on the type of consistency (either distillation or training).\n- I have a question on the approximation made on the score in Equation (15) for consistency training, which can actually also be addressed to standard consistency models. Theoretically, up to the terminal conditioning in the bridge setting, the score evaluated in $x$ at time $t$ has to be understood as a conditional expectation over the posterior distribution of $X_0$ given $X_t=x$ by Tweedie's formula. This distribution has a very specific behaviour: (i) when $t\\approx0$ (close to the target distribution), it is concentrated like a Dirac mass; (ii), on the other hand when $t\\approx T$ (full noise), the posterior is approximately equal to the target distribution itself. This is well highlighted in the concept of duality of log-concavity in [1]. Therefore, the approximation of the score as presented in the current paper (that is evaluating the integrand in only one point $x_0$) is relevant when $t\\approx0$, but completely false when $t\\approx T$. Do you have an estimation of the error made in the latter case for tractable cases ? Does it have an impact on the training for this part of the process ? Have you come up with an idea to correct this intrinsic bias ?\n\n[1] Stochastic localization via iterative posterior sampling. Grenioux et al. 2024", "limitations": "The limitations are clearly indicated in Section 5. The authors notably acknowledge that bridge consistency models suffer from numerical instability (same as classic consistency models).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- Although this paper adapts lots of elements from [1] to the bridge setting, which may amortize the novelty of this contribution, I think that this contribution may be impactful for practical usage as it encompasses a large formulation of DDBMs and demonstrates great empirical performance, see Section 4.\n- The paper is well written and easy to follow. In particular, the mathematical statements are clear to understand.\n- The presentation of the background is well done and helps the readability of the paper.\n- The comparison with related work in the numerics section is well conducted. \n- All details on the experiments are available in the paper, which is a really good point for reproducibility.\n\n[1] Consistency models. Song et al. 2023.", "weaknesses": "- The current paper does not exhibit any particular novelty compared to the setting from [1], but this is not a major weakness to me.\n- I think that the paper does not provide enough intuition on the design of a good consistency schedule.\n\n[1] Consistency models. Song et al. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging our contribution and the valuable comments on our work. We would like to provide our responses as follows (in reverse order for better logical flow):\n\n> Q1: The approximation of the score as presented in the current paper (that is evaluating the integrand in only one point $x_0$) is relevant when $t \\approx 0$, but completely false when $t \\approx T$. Do you have an estimation of the error made in the latter case for tractable cases? Does it have an impact on the training for this part of the process? Have you come up with an idea to correct this intrinsic bias?\n\n**A**: Thank you for the question. We think that the original CM paper [1] has largely addressed the problem of justifying the use of the mentioned data score for consistency training. Specifically, in Theorem 6 of the CM paper, they reached a conclusion that, when the interval between two timesteps for training CM becomes infinitely small, then the gradient of the consistency training loss and the consistency distillation loss with the true score under Euler ODE solver will be identical. Therefore, the mismatch between the data score (posterior) and true score (marginal) can be addressed by shrinking the interval between the two timesteps in training CM. In practice, this becomes a principle for designing a consistency schedule that ensures $t - r(t)$ to be as small as possible, at least at the end of training. Nevertheless, we admit the inaccurate estimation will have an impact and we think one way to address it is by proposing more fine-grained consistency schedules, which is an interesting and important research direction.\n\n\n> Q2: Could you give further details and intuition on the design of the consistency schedules $r(t)$, which seem to be crucial so that consistency works (even if these reasons are heuristic)? In particular, could you give examples of schedules that may be adapted in theory but do not work in practice? It still unclear to me which design should be the best depending on the type of consistency (either distillation or training).\n\n**A**: Thank you for the question. First, the design space of training consistency model, such as consistency schedules, loss weighting, and distance metric in consistency loss, has become a specialized area of research, and there are several papers (e.g., [2, 3]) target on investigating better designs for these components. In this work, we mainly adopt the design guideline proposed in [3] and verified its effectiveness in a different setting, i.e., the training of a CM for the PF-ODE of DDBMs. In specific, the consistency schedule should gradually shrink $t - r(t)$ to be small to avoid optimization problems and error accumulation due to such small time intervals as much as possible, which is intuitively elaborated in Sec 3.2 in [3]. In our experiments, we found that the gradually shrinking schedule generally yields better performance than a fixed interval and could be applied to both consistency distillation and training. Hence, our conclusions on the consistency schedule are consistent with the intuition and insights of recent works studying this on diffusion models. However, how to rationalize the design of the process shrinking $t - r(t)$ (e.g., in conjunction with the accuracy of the score estimation discussed in Q1) is still an active area of research. \n\n[1] Song, Yang, et al. \"Consistency Models.\" International Conference on Machine Learning. ICML, 2023.\n\n[2] Song, Yang, and Prafulla Dhariwal. \"Improved techniques for training consistency models.\" ICLR 2024.\n\n[3] Geng, Zhengyang, et al. \"Consistency Models Made Easy.\" arXiv preprint 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aARCbJEmwK", "reviewer": "Reviewer_nr1H", "summary": "This paper works on efficient sampling of the denoising diffusion bridge models (DDBMs) or similar. In particular, the paper proposes to extend the consistency models (CMs) to DDBMs.\n\nCMs are generative models developed in the context of improving the sampling cost of diffusion-based generative models (DBGMs). Note that sampling of the probability flows of the DBGMs runs the integrators up to a few hundred steps or more. Similar to common DBGMs, assume we have a forward diffusion whose initial distribution—data—is denoted by $x_0$. In particular, when we rewrite the forward diffusion in the probability flow (PF) form, we get an ODE whose marginal is equivalent to the original forward diffusion. We further assume that we have the ODE-integrators that run $t$-decreasing direction starting from any given $t \\in (0, T]$ to $0$. Then, the CMs train to predict the output of the integrators, similar to denoising autoencoder, but the perturbation here is deterministic. Training such models uses the consistency condition. Once we train the CMs, we can use them instead of running a few hundred steps of DGBMs.\n\nDDBMs are input-conditional bridge models. More specifically, assume we have two distributions and their joint distribution, e.g., one is an image distribution, and the other is its edge images. The bridge models learn a diffusion to connect one distribution to the other. Unlike common bridge models, however, DDBMs conditions on its initial point, e.g., an edge image, so that the path generated by the learned diffusion preserves the information of the initial value. For several applications, such as image-to-image translations, these DDBMs show several benefits, unlike common Markovian models; in particular, the initial coupling of the joint distributions is preferred. Nevertheless, DDBMs are also required to run the SDE-integrators (or ODE-integrators for its PFs) a few hundred steps, similar to any other diffusion-based models.\n\nIn order to overcome sampling inefficiency in DDBMs, the paper proposes to extend CMs to DDBMs.\n\nFinally, the authors demonstrate the proposed method's effectiveness via various experiments, including image-to-image translations and image inpainting.\n\n---------------------------------------\nUpdated the rating from 4 to 7 after the authors' rebuttal", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "One of the paper's key contributions is the introduction of a novel solution to improve the sampling of the DDBM-like models. The proposed method addresses one of the significant challenges in diffusion-based models and offers a fresh perspective on how state-of-the-art techniques can be extended to the DDBM-like models. Furthermore, the proposed method exhibits comparable performance and scalability compared to popular diffusion-based generative models with a fewer number of evaluations, further underscoring the practical effectiveness of the proposed method.", "weaknesses": "The authors tackle a highly interesting problem and reach out for an excellent solution. However, I find that the paper requires a major revision due to a few theoretical loopholes in extending the consistency models (CMs) to the denoising diffusion bridge models (DDBMs).\n\nFor example, in expressing CM in the form of a stochastic differential equation (SDE), the paper uses a definition that is not mathematically well-defined. Particularly, in Equation 2, the notation of a reverse-time standard Wiener process is mentioned. However, a reverse-time standard Wiener process is not defined in general; more precisely, it is tricky to define such a concept. The reason is that the definition of the It\\'o integral is highly sensitive to the direction of time. Consequently, the change of variables used in calculus cannot be applied to SDE directly. As a result, we need to define an SDE-specific chain rule called Ito's lemma. Accordingly, one cannot rewrite an SDE by changing the sign of time. In order to apply the chain rule of calculus, the SDE literature uses the Stieltjes integral instead of the It\\'o integral, and a corresponding SDE notation needs to be defined. If this approach were taken for the submitted paper, however, the denoising diffusion and consistency model would need to be defined accordingly.\n\nIn conclusion, this paper's development deviates from the fundamental assumptions of the It\\'o integral-based SDEs. Note that the original CM circumvents this issue by using only ordinary differential equations (or partial differential equations). Therefore, unlike the It\\'o integral, it is free from concerns about the time directions. However, this does not apply to SDEs.\n\nIn my understanding, one would be able to reach the same conclusion as the submitted paper while sticking to the fundamentals of SDEs (related to It'o\\' diffusion). However, such modification would require a major revision from the current submission. \n\nIn addition, a few statements need to be addressed. In particular, to get an unbiased estimation of the expectation in Equation 15, one needs to sample $x_t$ and $x_T$ first and then sample $x_0$. However, the authors state that we can achieve this by sampling $x_0$ and $x_T$ first and then $x_t$. Such an estimator is biased except when we use a single sample.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive comments and valuable feedback. We would like to address the weaknesses as follows:\n\n> W1: This paper's development deviates from the fundamental assumptions of the It'o integral-based SDEs. \n\n**A**: We respectfully disagree with this statement and would like to address that **we are not expressing CM in the form of a stochastic differential equation (SDE).** The reverse SDE for denoising diffusion and DDBM in Eqn. (2) and Eqn. (7) are only mentioned once in Section 2 using the definition from established literature for completeness (which *can be removed without affecting the definition and development of CM* if it might cause a presentation issue). The consistency model for denoising diffusion and DDBM is defined with the PF-ODE in Eqn. (3) and Eqn. (8), respectively, which do not depend on the reverse SDE in Eqn. (2) and Eqn. (7).\n\n> W2: a few statements need to be addressed. In particular, to get an unbiased estimation of the expectation in Equation 15, one needs to sample $x_t$ and $x_T$ first and then sample $x_0$. However, the authors state that we can achieve this by sampling $x_0$ and $x_T$ first and then $x_t$. Such an estimator is biased except when we use a single sample.\n\n**A**: We appreciate the reviewer for rigorously pointing this out. We will revise this to: With a single sample $(x, y) \\sim q_{\\mathrm{data}}(x, y)$ and $x_t \\sim q_{t|0T}(x_t|x_0=x, x_T=y)$, the score $\\nabla_{x_t}\\log q_{t|T}(x_t|x_T=y)$ can be estimated with $\\nabla_{x_t}\\log q_{t|0T}(x_t|x_0=x, x_T=y)$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers:

We add some additional results as a part of our response in the one-page pdf, including:
* The demonstration of sample diversity of the deterministic ODE sampler for DDBM (Fig. 9)
* The qualitative comparison between I$^2$SB (their default setup with officially released checkpoint) and CDBM.


We kindly invite you to check the results if they are relevant to your mentioned weaknesses/questions.

Best,
Authors of Submission #17343

Meta-review: This manuscript incorporates ideas from previous work on "consistency models" to generalize recent work on denoising diffusion bridge models, to allow high-quality data generation with a coarser (and thus computationally cheaper) discretization of the underlying ODE.  Reviewers are positive about the approach, but had a number of concerns about technical details, which were ultimately addressed by the author rebuttal.  Please be sure that these clarifications and improvements are incorporated in the final manuscript, and work to make your results accessible to the broader NeurIPS community.
Decision: Accept (poster)