Consistency Diffusion Bridge Models

Authors: guande  he
kaiwen  zheng
jianfei  chen
fan  bao
jun  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 15:16:06

Abstract: Diffusion models (DMs) have become the dominant paradigm of generative modeling in a variety of domains by learning stochastic processes from noise to data. Recently, diffusion denoising bridge models (DDBMs), a new formulation of generative modeling that builds stochastic processes between fixed data endpoints based on a reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation. However, DDBM's sampling process typically requires hundreds of network evaluations to achieve decent performance, which may impede their practical deployment due to high computational demands. In this work, inspired by the recent advance of consistency models in DMs, we tackle this problem by learning the consistency function of the probability-flow ordinary differential equation (PF-ODE) of DDBMs, which directly predicts the solution at a starting step given any point on the ODE trajectory. Based on a dedicated general-form ODE solver, we propose two paradigms: consistency bridge distillation and consistency bridge training, which is flexible to apply on DDBMs with broad design choices. Experimental results show that our proposed method could sample $4\times$ to $50\times$ faster than the base DDBM and produce better visual quality given the same step in various tasks with pixel resolution ranging from $64 \times 64$ to $256 \times 256$, as well as supporting downstream tasks such as semantic interpolation in the data space.

Main paper: 
 introduction: 
 Diffusion models (DMs) [53, 21, 60] have reached unprecedented levels as a family of generative models in various areas, including image generation [10, 50, 48] , audio synthesis [5, 45] , video generation [20] , as well as image editing [41, 42] , solving inverse problems [25, 56] , and density estimation [59, 28, 37, 71] . In the era of AI-generated content, the stable training, scalability & state-of-the-art generation performance of DMs successfully make them serve as the fundamental component of large-scale, high-performance text-to-image [14] and text-to-video [18, 2] A critical characteristic of diffusion models is their iterative sampling procedure, which progressively drives random noise into the data space. Although this paradigm yields a sample quality that stands out from other generation models, such as VAEs [29, 46] , GANs [17] , and Normalizing Flows [11, 12, 30] , it also results in a notoriously lower sampling efficiency compared to other arts. In response to this, consistency models [58] have emerged as an attractive family of generative models by learning a consistency function that directly predicts the solution of a probability-flow ordinary differential equation (PF-ODE) at a certain starting timestep given any points in the ODE trajectory, designed to be a one-step generator that directly maps noise to data. Consistency models can be Figure 1 : Illustration of consistency models (CMs) on PF-ODEs of diffusion models and our proposed consistency diffusion bridge models (CDBMs) building on PF-ODEs of diffusion bridges. Different from diffusion models, the PF-ODE of diffusion bridge is only well defined in t < T due to the singularity induced by the fixed terminal endpoint. To this end, a valid input for CDBMs is some xt for t < T , which is typically obtained by one-step posterior sampling with a coarse estimation of x0 with an initial network evaluation.naturally integrated with diffusion models by adapting the score estimator of DMs to a consistency function of their PF-ODE via distillation [58, 26] or fine-tuning [15] , showing promising performance for few-step generation in various applications like latent space [40] and video [64] .Despite the remarkable achievements in generation quality and better sampling efficiency, a fundamental limitation of diffusion models is that their prior distribution is usually restricted to a non-informative Gaussian noise, due to the nature of their underlying data to noise stochastic process. This characteristic may not always be desirable when adopting diffusion models in some scenarios with an informative non-Gaussian prior, such as image-to-image translation. Alternatively, an emergent family of generative models focuses on leveraging diffusion bridges, a series of altered diffusion processes conditioned on given endpoints, to model transport between two arbitrary distributions [44, 36, 33, 54, 51, 72, 7] . Among them, denoising diffusion bridge models (DDBMs) [72] study the reverse-time diffusion bridge conditioned on the terminal endpoint, and employ simulation-free, non-iterative training techniques for it, showing superior performance in application with coupled data pairs such as distribution translation compared to diffusion models. However, DDBMs generally require hundreds of network evaluations to produce samples with decent quality, even using an advanced high-order hybrid sampler, potentially hindering their deployments in real-world applications. 
preliminaries 2.1 diffusion models: 
 Given the data distribution p data (x), x ∈ R m , diffusion models [53, 21, 60] specify a forward-time diffusion process from an initial data distribution p 0 = p data to a terminal distribution p T within a finite time horizon t ∈ [0, T ], defined by a stochastic differential equation (SDE):EQUATIONwhere w t is a standard Wiener process, f : R m × [0, T ] → R m and g : [0, T ] → R d are drift and diffusion coefficients, respectively. The terminal distribution p T is usually designed to approximate a tractable prior p prior (e.g., standard Gaussian) with the appropriate choice of f and g. The corresponding reverse SDE and the probability flow ordinary differential equation (PF-ODE) of the forward SDE in Eqn. ( 1) is given by [1, 60] :EQUATIONEQUATIONwhere wt is a reverse-time standard Wiener process and p t (x t ) is the marginal distribution of x t . Both the reverse SDE and PF-ODE can act as a generative model by sampling x T ∼ p prior and simulating the trajectory from x T to x 0 . The major difficulty here is that the score function ∇ log p t (x t ) remains unknown, which can be approximated by a neural network s θ (x t , t) with denoising score matching [63] :EQUATIONwhere U(0, T ) is uniform distribution, λ(t) > 0 is a weighting function, and p t|0 (x t |x 0 ) is the transition kernel from x 0 to x t . A common practice is to use a linear drift f (t)x t such that p t|0 (x t |x 0 ) is an analytic Gaussian distribution N (α t x 0 , σ 2 t I), where α t = e t 0 f (τ )dτ , σ 2 t = α 2 t t 0g 2 (τ ) α 2 τdτ is defined as the noise schedule [28] . The resulting score predictor s θ (x t , t) can replace the true score function in Eqn. (2) and (3) to obtain the empirical diffusion SDE and ODE, which can be simulated by various SDE or ODE solvers [55, 38, 39, 16, 70] .Given a trajectory {x t } T t=ϵ with a fixed starting timestep ϵ of a PF-ODE, consistency models [58] aim to learn the solution of the PF-ODE at t = ϵ, also known as the consistency function, defined as h : (x t , t) → x ϵ . The optimization process for consistency models contains the online network h θ and a reference target network h θ -, where θ -refers to θ with operation stopgrad, i.e., θ -= stopgrad(θ). The networks are hand-designed to satisfy the boundary condition h θ (x ϵ , ϵ) = x ϵ , which can be typically achieved with proper parameterization on the neural network. For PF-ODE taking the form in Eqn. (3) with a linear drift f (t)x t , the overall learning objective of consistency models can be described as:EQUATIONwhere r(t) is a function that specifies another timestep r (usually with t > r), d denotes some metric function with ∀x, y : d(x, y) ≥ 0 and d(x, y) = 0 iff. x = y. Here xr is a function that estimates x r = x t + r t dxτ dτ dτ , which can be done by simulating the empirical diffusion ODE with a pre-trained score predictor s ϕ (x t , t) or empirical score estimator -xt-αtx0 σ 2 t . The corresponding learning paradigms are named consistency distillation and consistency training, respectively.Given a data pair sampled from an arbitrary unknown joint distribution (x, y) ∼ q data (x, y), x, y ∈ R m and let x 0 = x, denoising diffusion bridge models (DDBMs) [72] specify a stochastic process that ensures x T = y almost surly via applying Doob's h-transform [13, 47] on a reference diffusion process in Eqn. (1) :EQUATIONwhere p T |t (x T = y|x t ) is the transition kernel of the reference diffusion process from t to T , evaluated at x T = y. Denoting the marginal distribution of Eqn. ( 6) as {q t } T t=0 , it can be shown that the forward bridge SDE in Eqn. ( 6) is characterized by the diffusion distribution conditioned on both endpoints, that is, q t|0T (x t |x 0 , x T ) = p t|0T (x t |x 0 , x T ), which is an analytic Gaussian distribution. A generative model can be obtained by modeling q t|T (x t |x T = y), whose reverse SDE and PF-ODE are given by:EQUATION)EQUATION)The only unknown term remains is the score function ∇ xt log q t|T (x t |x T = y), which can be estimated with a neural network s θ (x t , t, y) via denoising bridge score matching (DBSM):E t∈U (0,T ) E q data (x,y)q t|0T (xt|x0=x,x T =y) λ(t)∥s θ (x t , t, y) -∇ log q t|0T (x t |x 0 = x, x T = y)∥ 2 2 .(9) Replacing ∇ xt log q t|T (x t |x T = y) in Eqn. ( 7) and ( 8) with the learned score predictor s θ (x t , t, y) would yield the empirical bridge SDE and ODE that could be solved for generation purposes. 
consistency diffusion bridge models: 
 In this section, we introduce consistency diffusion bridge models, extending the techniques of consistency models to DDBMs to further boost their performance and sample efficiency. Define the consistency function of the bridge ODE in Eqn. (8) as h : (x t , t, y) → x ϵ with a given starting timestep ϵ, our goal is to learn the consistency function using a neural network h θ (•, •, y) with the following high-level objective similar to Eqn. ( 5):E t∈U (ϵ,T ),r=r(t) E q data (x,y)q t|0T (xt|x0=x,x T =y) [λ(t)d (h θ (x t , t, y), h θ -( xr , r, y))] .(10) To begin with, we first present a unified view of the design spaces such as noise schedule, network parameterization & precondition, as well as a general ODE solver for DDBMs. This allows us to: (1) decouple the successful practical designs of previous diffusion bridges from their different theoretical premises; (2) decouple the framework of consistency models from certain design choices of the corresponding PF-ODE, such as the reliance on VE schedule with Euler ODE solver of the original derivation of consistency models [58] . This would largely facilitate the development of consistency models that utilize the rich design spaces of existing diffusion bridges on DDBMs in a universal way. Then, we elaborate on two ways to train h θ based on different choices of xr , consistency bridge distillation, and consistency bridge training, with the proposed unified design spaces.Noise Schedule We consider the linear drift f (t)x t and define:EQUATIONwhich aligns with the common notation of noise schedules used in diffusion models by denoting σ t = α t ρ t . Then we could express the analytic conditional distributions of DDBMs as follows:EQUATION)The form of q t|0T is consistent with the original formulation of DDBM in [72] . Here, inspired by [6] , we opt to adopt a more neat set of notations for enhanced compatibility. As shown in Table 1 , with such notations, we could easily unify the design choices for diffusion bridges [33, 72, 6] that have shown effectiveness in various tasks and expeditiously employ consistency models on top of them. for a Gaussian N (a t x T + b t x 0 , c 2 t I) like q t|0T . Meanwhile, the inputs and outputs of the network F θ could be rescaled for a better-behaved optimization process, known as the network precondition. As shown in Table 1 , we could consistently use x 0 as the prediction target with different choices of network precondition to unify the previous practical designs for DBMs.default default † VP ‡ VE gmax VP Schedule T 1 1 1 T 1 1 f (t) 0 0 -1 2 β 0 0 0 -1 2 β 0 -1 2 β d t g 2 (t) σ 2 (η 1 -η 0 |2t -1|) 2 β 0 2t β 0 + β d t β 0 + β d t α t 1 1 e -1 2 β0t 1 1 e -1 2 β0t-1 4 βdt 2 σ 2 t σ 2 t t 0 g 2 (τ )dτ 1 -e -β0t t 2 β 0 t + 1 2 β d t 2 1 -e -β0t-1 2 βdt 2 ᾱt 1 1 e 1 2 β0-1 2 β0t 1 1 α t /α 1 ρ 2 t σ 2 t t 0 g 2 (τ )dτ e β0t -1 t 2 β 0 t + 1 2 β d t 2 e β0t+ 1 2 βdt 2 -1 ρ2 t σ 2 (1 -t) ρ 2 1 -ρ 2 t e β0 -e β0t T 2 -t 2 ρ 2 1 -ρ 2 t ρ 2 1 -ρ 2 t Parameters σ η 0 = √ β1- √ β0 2 η 1 = √ β1+ √ β0 2 β 0 = 0.1 β 1 = 0.3/1.0 β 0 T = 80 β 0 = 0.01 β d =The validity of a consistency model relies on an underlying PF-ODE that shares the same marginal distribution with the forward process. In the original DDBM paper [72] , the marginal preserving property of the proposed ODE is justified following an analogous logic from the derivation of the PF-ODE of diffusion models [60] with Kolmogorov forward equation. However, its validity suffers from doubts as there is a singularity at the deterministic starting point x T . Here, we provide a simple example to show that the ODE can indeed maintain the marginal distribution as long as we use a valid stochastic step to skip the singular point and start from T -γ for any γ > 0. Example 3.1. Assume T = 1 and consider a simple Brownian Bridge between two fixed points (x 0 , x 1 ):EQUATIONwith marginal distribution q t|01 (x t |x 0 , x 1 ) = N ((1 -t)x 0 + tx 1 , t(1 -t)). The ground-truth reverse SDE and PF-ODE are given by:EQUATION)EQUATION)Then first simulating the reverse SDE in Eqn. ( 14) from t = 1 to t = 1 -γ for some γ ∈ (0, 1) and then starting to simulate the PF-ODE in Eqn. (15) will preserve the marginal distribution.The detailed derivation can be found in Appendix. B.2. Therefore, the time horizon of the consistency model based on the bridge ODE needs to be set as t ∈ [ϵ, T -γ] for some pre-specified ϵ, γ > 0.Additionally, the marginal preservation of the bridge ODE for more general diffusion bridges can be strictly justified by considering non-Markovian variants, as done in DBIM [69] .Another crucial element for developing consistency models is the ODE solver, as a solver with a lower local error would yield lower error for consistency distillation, as well as the corresponding consistency training objectives [58, 57] . Inspired by the successful practice of advanced ODE solvers based on the Exponential Integrator (EI) [4, 22] in diffusion models, we present a first-order bridge ODE solver in a similar fashion: Proposition 3.1. Given an initial value x t at time t > 0, the first-order solver of the bridge ODE in Eqn. (8) from t to r ∈ [0, t] with the noise schedule defined in Eqn. (11) is:EQUATION)We provide detailed derivation in the Appendix B.1. Typically, an EI-based solver enjoys a lower discretization error and therefore has better empirical performance [16, 38, 39, 67, 70] . Another notable advantage of this general form solver, as we will show in Section 3.3, is that it could naturally establish the connection between consistency training and consistency distillation for any noise schedules that take the form in Eqn. (11) , eliminating the dependence of the VE schedule and the corresponding Euler ODE solver in the common derivation [58] .Analogous to consistency distillation with the empirical diffusion ODE, we could leverage a pretrained score predictor s ϕ (x t , t, y) ≈ ∇ xt log q t|T (x t |x T = y) to solve the empirical bridge ODE to obtain xr , i.e., xr = xϕ (x t , t, r, y), where xϕ is the update function of a one-step ODE solver with fixed s ϕ . We define the consistency bridge distillation (CBD) loss as:EQUATIONE t∈U (ϵ,T -γ),r=r(t) E q data (x,y)q t|0T (xt|x0=x,x T =y) [λ(t)d (h θ (x t , t, y), h θ -( xϕ (x t , t, r, y), r, y))] ,where t is sampled from the uniform distribution over [ϵ, T -γ], r(t) is a function specifies another timestep r such that ϵ ≤ r < t with ∆t max := max t {t -r(t)} and ∆t min := min t {t -r(t)}, λ(t) is a positive weighting function, d is some distance metric function with ∀x, y : d(x, y) ≥ 0 and d(x, y) = 0 iff. x = y, and θ -= stopgrad(θ). Similarly to the case of consistency distillation in empirical diffusion ODEs, we have the following asymptotic analysis of the CBD objective: Proposition 3.2. Given ∆t max = max t {t -r(t)} and let h ϕ (•, •, •) be the consistency function of the empirical bridge ODE taking the form in Eqn. (8) . Assume h θ is a Lipschitz function, i.e., there exists L > 0, such that for all t ∈ [ϵ, T -γ], x 1 , x 2 , y, we have∥h θ (x 1 , t, y) -h θ (x 2 , t, y)∥ 2 ≤ L∥x 1 -x 2 ∥ 2 . Meanwhile, assume that for all t, r ∈ [ϵ, T -γ], y ∼ q data (y) := E x [q data (x, y)],the ODE solver xϕ (•, t, r, y) has local error uniformly bounded by O((t -r) p+1 ) with p ≥ 1. Then, if L ∆tmax CBD = 0, we have:sup t,x,y ∥h θ (x, t, y) -h ϕ (x, t, y)∥ 2 = O((∆t max ) p ).The vast majority of the analysis can be done by directly following the proof in [58] with minor differences between the overlapped timestep intervals {t, r(t)} for t ∈ [ϵ, T -γ] used in Eqn. (17) and the fixed timestep intervals {t n } N n=1 used in [58] . We include it in Appendix B.5 for completeness. In this work, unless otherwise stated, we use the first-order ODE solver in Eqn. (16) as xϕ .In addition to distilling from pre-trained score predictor s ϕ , consistency models can be trained [58, 57] or fine-tuned [15] by maintaining only one set of parameters θ. To accomplish this, we could leverage the unbiased score estimator:EQUATIONthat is, with a single sample (x, y) ∼ q data and x t ∼ q t|0T (x t |x 0 = x, x T = y), the score ∇ xt log q t|T (x t |x T = y) can be estimated with ∇ xt log q t|0T (x t |x 0 , x T ). Substituting such an estimation of s ϕ into the one-step ODE solver xϕ in Eqn. (17) with the transformation between data and score predictorx ϕ = xt-atx T +c 2 t s ϕ bt, we can obtain an alternative xr that does not rely on the pre-trained s ϕ for any noise schedule taking the form in Eqn. (11) as follows (detail in Appendix B.3):EQUATIONwhere a r , b r , c r are defined as in Eqn. (11) , and z = xt-aty-btx ct ∼ N (0, I). Based on this instantiation of xr , we define the consistency bridge training (CBT) loss as:L ∆tmax CBT := (20) E t∈U (ϵ,T -γ),r=r(t) E q data (x,y) [λ(t)d (h θ (a t y + b t x + c t z, t, y), h θ -(a r y + b r x + c r z, r, y))] ,where t, r(•), λ(•), θ -1 are defined the same as in Eqn. (17) , and z ∼ N (0, I) is a shared Gaussian noise used in both h θ and h θ -1 . We have the following proposition demonstrating the connection between L ∆tmax CBT and L ∆tmax CBD with the first-order one-step ODE solver: Proposition 3.3. Given ∆t max = max t {t -r(t)} and assume d, h θ , f, g are twice continuously differentiable with bounded second derivatives, the weighting function λ(•) is bounded, andE[∥∇ xt log q t|T (x t |x T )∥ 2 2 ] < ∞.Meanwhile, assume that L ∆tmax CBD employs the one-step ODE solver in Eqn. ( 16) with ground truth pre-trained score model, i.e., ∀t ∈ [ϵ, T -γ], y ∼ q data (y) : s ϕ (x t , t, y) ≡ ∇ xt log q t|T (x t |x T = y). Then, we have:L ∆tmax CBD = L ∆tmax CBT + o(∆t max ).The core part of our analysis also follows [58] , except the connection between the CBD & CBT objective relies on the proposed first-order ODE solver and the estimated xr in Eqn. (19) with the general noise schedule for DDBM. We include the details in Appendix B.6.Network Precondition First, we focus on enforcing the boundary condition h θ (x ϵ , ϵ, y) = x ϵ of our consistency bridge model, which can be done by designing a proper network precondition. Usually, a variable substitution t = t -ϵ could work in most cases. For example, for the precondition for I 2 SB in Table 1 , we havex ϵ + σ εF θ = x ϵ + ϵ-ϵ 0 g 2 (τ )dτ = x ϵ .Also, the common "EDM" [24] style precondition used in DDBM also satisfies c skip (ε) = 1 and c out (ε) = 0. We also give a universal precondition to satisfy the boundary conditions based on the form of the ODE solver in Eqn. (16) in Appendix B.4 to cope with the case where the variable substitution is not applicable.Sampling As explained in Section 3.1, the PF-ODE is only well-defined within the time horizon 0 ≤ t ≤ T -γ for some γ ∈ (0, T ). Hence, the sampling of CDBMs should start with x T -γ ∼ q T -γ|T (x T -γ |x T = y), which can be obtained by simulating the reverse SDE in Eqn. (7) from T to T -γ. Here we opt to use one first-order stochastic step, which is equivalent to performing posterior sampling, i.e., x T -γ ∼ q T -γ|0T (x T -γ |x 0 = h θ (x T , T, y), x T = y). This sampling approach defaults to two NFEs (Number of Function Evaluations), which is aligned with the practical guideline that employing two-step sampling in CM allows for a better trade-off between quality and computation compared to other treatments such as scaling up models [15] . We could also alternate a forward noising step and a backward consistency step multiple times to further improve sample quality as consistency models do. 
experimental setup: 
 Task, Datasets, and Metrics In this work, we conduct experiments for CDBM on image-to-image translation and image inpainting tasks with various image resolutions and scales of the data set. For image-to-image translation, we use the Edges→Handbags [23] with 64 × 64 pixel resolution and DIODE-Outdoor [62] with 256 × 256 pixel resolution. For image inpainting, we choose ImageNet [9] 256 × 256 with a center mask of size 128 × 128. Regarding the evaluation metrics, we report the Fréchet inception distance (FID) [19] for all datasets. Furthermore, following previous works [33, 72] , we measure Inception Scores (IS) [3] , LPIPS [68] and Mean Square Error (MSE) for image-to-image translation and Classifier Accuracy (CA) of a pre-trained ResNet50 for image-inpainting. The metrics are computed using the complete training set for Edges→Handbags and DIODE-Outdoor, and a validation subset of 10,000 images for ImageNet.Training Configurations We train CDBM in two ways: distill pre-trained DDBM with CBD or finetuning DDBM with CBT. We keep the noise schedule and prediction target of the pre-trained DDBM unchanged and modify the network precondition to satisfy the boundary condition. Specifically, we adopt the design space of DDBM-VP and I 2 SB in Table 1 on image-to-image translation and image inpainting, respectively. We specify complete training details in Appendix C.We illustrate the specific design choices for CDBM. In this work, we use t ∈ [ϵ, 1 -γ] and set ϵ = 0.0001, γ = 0.001 and sample t uniformly during training. We employ two different sets of the timestep function r(t) and the loss weighting λ(t), also named the training schedule for CDBM. The first, following [58] , specifies a constant quantity for ∆t = t -r(t) with a simple loss weighting of λ(t) = 1. The constant gap ∆t is treated as a hyperparameter and we search it among {1/9, 1/18, 1/36, 1/60, 1/80, 1/120}. The other employs r(t) that gradually shrinks t -r(t) during the training process and a loss weighting of λ(t) = 1 t-r(t) , which enjoys a better trade-off between faster convergence and performance [58, 57, 15] . Following [15] , we use a sigmoid-style function r(t) = t(1 -1 q ⌊iters/s⌋ )(1 + k 1+e bt ), where iters is the number of training iterations, q, s, k, b are hyperparameters. We use q = 2, k = 8, and tune b ∈ {1, 2, 5, 10, 20, 50} and s ∈ {5000, 10000}. For image-to-image translation, as shown in Table . 2, we first observed that our proposed firstorder ODE solver has superior performance compared to the hybrid high-order sampler used in DDBM [72] . On top of that, CDBM's FID at NFE = 2 is close to or even better than DDBM's at NFE around 100 with the advanced ODE solver, achieving a sampling speed-up around 50×. This can be corroborated by the qualitative demonstration in Fig. 4 , where CDBMs drastically reduce the blurring effect on DDBMs under few-step generation settings while enjoying realistic and faithful translation performance.For image inpainting, as shown in Table . 3, the baseline ODE solver for DDBM achieves decent sample quality at NFE = 10. For CDBM, as shown in Fig. 2 , the acceleration ratio is relatively modest in such a large-scale and challenging dataset, achieving close to a 4× increase in sampling speed. Notably, CBT's FID at NFE = 4 matches DDBM at NFE = 10. Moreover, we find that CDBMs have better visual quality than DDBM given the same computation budget, as shown in Fig. 4 and Appendix D, which illustrates that CDBM yields a better quality-efficiency trade-off.Meanwhile, we observe that fine-tuning DDBMs with CBT generally produces better results than CBD in all three data sets, demonstrating fine-tuning a pre-trained score model to a consistency function is a more promising solution with less computational and memory cost compared to distillation, which is consistent with recent findings [15] . We also conducted an ablation study for CBD and CBT under different training schedules (i.e., the combination of the timestep function r(t) and the loss weighting λ(t)) on ImageNet 256 × 256. As shown in Fig. 3 , for a small timestep interval t -r(t), e.g., a small ∆t in Fig. 3a or a large b in Fig. 3b (detail in Appendix C.2), the performance is generally better but also suffers from training instability, indicated by the sharp increase in FID during training when ∆t = 1/120 and b = 50. While for a large timestep interval, the performance at convergence is usually worse. In practice, we found that adopting the training schedule that gradually shrinks r(t) -t with b = 20 or 50 with CBT could work across all tasks, whereas CBD generally needs a meticulous design for ∆t or b to ensure stable training and satisfactory performance.We show that CDBMs support performing downstream tasks, such as semantic interpolation, similar to diffusion models [55] . Recall that the sampling process for CDBM alternates between consistency function evaluation and forward sampling, we could track all noises and the corresponding timesteps to re-generate the same sample. By interpolating the noises of two sampling trajectories, we can obtain a series of samples lying between the semantics of two source samples, as shown in Fig. 5 , which demonstrates that CDBMs have a wide range of generative modeling capabilities, such as sample diversity and semantic interpolation. 
conclusion: 
 In this work, we introduce consistency diffusion bridge models (CDBMs) to address the sampling inefficiency of DDBMs and present two frameworks, consistency bridge distillation and consistency bridge training, to learn the consistency function of the DDBM's PF-ODE. Building on a unified view of design spaces and the corresponding general-form ODE solver, CDBM exhibits significant flexibility and adaptability, allowing for straightforward integration with previously established successful designs for diffusion bridges. Experimental evaluations across three datasets show that CDBM can effectively boost the sampling speed of DDBM by 4× to 50×. Furthermore, it achieves the saturated performance of DDBMs with less than five NFEs and possesses the broad capacity of generative models, such as sample diversity and semantic interpolation.Limitations and Broader Impact While significantly improving the sampling efficiency in the datasets we used, it remains to be explored how the proposed CDBM, along with the DDBM formulation, performs in datasets with larger-scale or more complex characteristics. Furthermore, the consistency model paradigm typically suffers from numerical instability and it would be a promising research direction to keep improving CDBM's performance from an optimization perspective. With enhanced sampling efficiency, CDBMs could contribute to more energy-efficient deployment of generative models, aligning with broader goals of sustainable AI development. However, it could also lower the cost associated with the potential misuse for creating deceptive content. We hope that our work will be enforced with certain ethical guidelines to prevent any form of harm. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The experiment configuration and details are included in Section 4 and Appendix C, which is sufficient to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The metrics for evaluating generative models are typically stable and do not require error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The discussion is located in the "Limitations and Broad Impact" section after the main paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work is conducted with common academic image datasets with model capability restricted with specific tasks. There is little chance posing risks for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Licenses for existing assets are listed in Appendix C.3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Diffusion Bridges Diffusion bridges [44, 36, 33, 54, 51, 72, 7, 6] are an emerging class of generative models with attractive flexibility in modeling the stochastic process between two arbitrary distributions. The flow matching [32] , and its stochastic counterpart, bridge matching [44] assume the access of a joint distribution and an interpolation, or a forward process, between the samples, then, another SDE/ODE is learned to estimate the dynamics of the pre-defined interpolation, which can be used for generative modeling from non-Gaussian priors [33, 6, 72, 69, 66] . In particular, the forward process can be constructed via Doob's h-transform [44, 36, 72] . Among them, DDBM [72] focuses on learning the reverse-time diffusion bridge conditioned on a particular terminal endpoint with denoising score matching, which has been shown to be equivalent to conducting a conditioned bridge matching that preserves the initial joint distribution [7] . Other works tackle solving the diffuion Schrödinger Bridge problem, such as using iterative algorithms [8, 51, 43] . In this work, we use a unified view of design spaces on existing diffusion bridges, in particular, bridge matching methods, to decouple empirical choices from their different theoretical premises and properties and focus on developing the techniques of learning the consistency function of DDBM's PF-ODE with various established design choices for diffusion bridges.Consistency Models Recent studies have continued to explore the effectiveness of consistency models [58] . For example, CTM [26] proposes to augment the prediction target from the starting point to the intermediate points along the PF-ODE trajectory from the input to this starting point. BCM [31] additionally expands the model to allow direct mapping at the PF-ODE trajectory points in both forward and reverse time. Beyond different formulations, several works aim to improve the performance of consistency training with theoretical and practical insights. iCT [57] systematically examines the design choices of consistency training and presents improved training schedule, loss weighting, distance metrics, etc. ECT [15] further leverages the insights to propose novel practical designs and show fine-tuning pre-trained diffusion models for learning consistency models yields decent performance with much lower computation compared to distillation. Unlike these works, we focus on constructing consistency models on top of the formulation of DDBMs with specialized design spaces and a sophisticated ODE solver for them.We first review the first-order ODE solver in Section 3.1: Proposition 3.1. Given an initial value x t at time t > 0, the first-order solver of the bridge ODE in Eqn. (8) from t to r ∈ [0, t] with the noise schedule defined in Eqn. ( 11) is:EQUATIONRecall the PF-ODE of DDBM in Eqn. ( 8) with a linear drift f (t)x t :EQUATIONAlso recall the noise schedule in Eqn. (11) and the analytic form of p t|0 and p T |t in diffusion models:EQUATIONWe thus have the corresponding score functions and the score-data transformation for s θ that predicts ∇ xt log q t|0T :EQUATIONEQUATIONEQUATIONWe use the data parameterization x θ (x t , t, y) in following discussions. For PF-ODE in Eqn. ( 21), substituting ∇ xt log q t|T (x t |x T = y) with Eqn. (25) and substituting p T |t (x T |x t ) in with Eqn. ( 23), we have the following after some simplification:EQUATIONwhich shares the same form as the ODE in Bridge-TTS [6] . In the next discussions, we present an overview of deriving the first-order ODE solver and refer the reader to Appendix A.2 in [6] for details.We begin by reviewing exponential integrators [4, 22] , a key technique for developing advanced diffusion ODE [16, 38, 39, 70] . Consider the following ODE:EQUATIONwhere F θ is a n-th differentiable parameterized function. By leveraging the "variation-of-constant" formula, we could obtain a specific form of the solution of the ODE in Eqn. ( 27) (assume r < t):EQUATIONThe integral in Eqn. ( 28) only involves the function F θ , which helps reduce discretization errors.With such a key methodology, we could derive the first-order solver for Eqn. (26) . First, collecting the coefficients for x t , y, x θ , we have:EQUATIONBy setting:a(t) = f (t) - g 2 (t) 2α 2 t ρ2 t + g 2 (t) 2α 2 t ρ 2 t , b 1 (t) = g 2 (t)ᾱ t 2α 2 t ρ2 t , b 2 (t) = g 2 (t) 2α t ρ 2 t .with correspondence to Eqn. (28) , the exponential terms could be analytically given by: e r t a(τ )dτ = α r σ r σr α t σ t σt , e r τ a(s)ds = α r σ r σr α τ σ τ στ .The exact solution for Eqn. ( 29) is thus given by:EQUATIONThe integrals in Eqn. (31) (without considering x θ ) can be calculated as:r t g 2 (τ ) α 2 τ ρ τ ρ3 τ dτ = 2 ρ 2 T ρ r ρr - ρ t ρt , r t g 2 (τ ) α 2 τ σ 3 τ στ dτ = 2 ρ 2 T ρt ρ t - ρr ρ rThen, with the first order approximation x θ (x τ , τ ) ≈ x θ (x s , s), we could obtain the first order solver in Eqn. (16) .Recall the provided example in Section 3.1: Example 3.1. Assume T = 1 and consider a simple Brownian Bridge between two fixed points (x 0 , x 1 ):EQUATIONwith marginal distribution q t|01 (x t |x 0 , x 1 ) = N ((1 -t)x 0 + tx 1 , t(1 -t)). The ground-truth reverse SDE and PF-ODE are given by:EQUATION)EQUATION)Then first simulating the reverse SDE in Eqn. ( 14) from t = 1 to t = 1 -γ for some γ ∈ (0, 1) and then starting to simulate the PF-ODE in Eqn. (15) will preserve the marginal distribution.Proof. We first demonstrate the effect of the initial SDE step, according to Table 1 and the expression of the relevant score terms in Eqn. (23) and Eqn. ( 25), the ground-truth reverse SDE can be derived as:dx t = x t -x 0 t dt + d wt .Then, the analytic solution of the reverse SDE in Eqn. (7) from time t to time s < t can be derived as:dx t - 1 t x t dt = - 1 t x 0 + d wt ⇐⇒ d 1 t x t = - 1 t 2 x 0 + 1 t d wt ⇐⇒ 1 s x s - 1 t x t = 1 s - 1 t x 0 + 1 s - 1 t ϵ, ϵ ∼ N (0, 1).Let t = 1, we have:x s = (1 -s)x 0 + sx 1 + s(1 -s)ϵ,i.e., x s has the same marginal as the forward process at time s. Similarly, the ground-truth PF-ODE can be derived as:dx t = 1 -2t 2t(1 -t) x t + 1 2(1 -t) x 1 - 1 2tx 0 dt, whose analytic solution from time t to time s < t can be derived as:dx t - 1 -2t 2t(1 -t) x t dt = 1 2(1 -t) x 1 dt - 1 2t x 0 dt ⇐⇒ d 1 t(1 -t) x t = t 2[t(1 -t)] 3/2 x 1 dt - 1 -t 2[t(1 -t)] 3/2 x 0 dt ⇐⇒ 1 s(1 -s) x s - 1 t(1 -t) x t = s s(1 -s) - t t(1 -t) x 1 + 1 -s s(1 -s) - 1 -t t(1 -t) x 0 ⇐⇒ x s = s(1 -s) t(1 -t) x t + s - s(1 -s) t(1 -t) t x 1 + 1 -s - s(1 -s) t(1 -t) (1 -t) x 0 . When x t ∼ N ((1 -t)x 0 + tx 1 , t(1 -t)), we have:x s = s(1 -s) t(1 -t) (1 -t)x 0 + tx 1 + t(1 -t)ϵ + s - s(1 -s) t(1 -t) t x 1 + 1 -s - s(1 -s) t(1 -t) (1 -t) x 0 = (1 -s)x 0 + sx 1 + s(1 -s)ϵ.Hence, once the singularity is skipped by a stochastic step, following the PF-ODE reversely will preserve the marginals in this case.Given (x, y) ∼ q data (x, y), x t ∼ q t|0T (x t |x 0 = x, x T = y) and an estimate of xr = xϕ (x t , t, y) based on the pre-trained score predictor s ϕ with the first-order ODE solver in Eqn. ( 16), our goal is to derive the alternative estimation of xr = x(x t , t, r, x, y) = a r y + b r x + c r z used in CBT, where z = xt-aty-bty ct ∼ N (0, I) and a r , b r , c r are defined in Eqn. (11) . We begin with the estimator with pre-trained score model and first-order ODE solver:EQUATIONwhere x ϕ is the equivalent data predictor of the score predictor s ϕ . By the transformation between data and score predictor x ϕ =xt-atx T +c 2 t s ϕ bt and substituting the score predictor s ϕ with the score estimator ∇ xt q t|0T (x t |x 0 = x, x T = y), we have:EQUATIONBy expressing x t = a t y + b t x + c t z, we could derive the corresponding coefficients for x, y, z on the right-hand side.For y:EQUATIONwhere (i) is due to the fact ᾱt = αt α T . For x:EQUATION)For z:EQUATION)Hence, we have the alternative model-free estimator xr = x(x t , t, r, x, y) = a r y + b r x + c r z, where z ∼ N (0, I) is the same Gaussian noise used in sampling x t = a t y + b t x + c t z. Substituting xϕ (x t , t, r, y) in the CBD objective in Eqn. (17) with x(x t , t, r, x, y) gives the CBT objective in Eqn. (20) .First, we show the detailed network parameterization for DDBM in Table . 1. Denote the neural network as F θ , the data predictor x θ (x, t, y) is given by:EQUATIONwhereEQUATION)andEQUATIONIt can be verified that, with the variable substitution t = t -ϵ, we have a ε = 0, b ε = 1, c ε = 0 and thus have c skip (ε) = 1 and c out (ε) = 0.Meanwhile, we could generally parameterize the data predictor x θ with the one-step first-order solver from t to ϵ, i.e.:EQUATIONwhich naturally satisfies f (x ϵ , ϵ, y) = x ϵ .B.5 Asymptotic Analysis of CBD Proposition 3.2. Given ∆t max = max t {t -r(t)} and let h ϕ (•, •, •) be the consistency function of the empirical bridge ODE taking the form in Eqn. (8) . Assume h θ is a Lipschitz function, i.e., there exists L > 0, such that for all t∈ [ϵ, T -γ], x 1 , x 2 , y, we have ∥h θ (x 1 , t, y) -h θ (x 2 , t, y)∥ 2 ≤ L∥x 1 -x 2 ∥ 2 . Meanwhile, assume that for all t, r ∈ [ϵ, T -γ], y ∼ q data (y) := E x [q data (x, y)],the ODE solver xϕ (•, t, r, y) has local error uniformly bounded by O((t -r) p+1 ) with p ≥ 1. Then, if L ∆tmax CBD = 0, we have:sup t,x,y ∥h θ (x, t, y) -h ϕ (x, t, y)∥ 2 = O((∆t max ) p ).Most of the proof directly follows the original consistency models analysis [58] , with minor differences in the discrete timestep intervals (i.e., non-overlapped in [58] and overlapped in ours) and the form of marginal distribution between p t (x t ) for the diffusion ODE and q t|T (x t |x T = y) for the bridge ODE.Proof. Given L ∆tmax CBD = 0, we have:E q data (x,y)q t|0T (xt|x0=x,x T =y) E t,r [λ(t)d (h θ (x t , t, y) -h θ -( xϕ (x t , t, r, y), r, y))] = 0 (41) Since λ(t) > 0, and for t ∈ [ϵ, T -γ], q t|0T (x t |x 0 = x, y 0 = y) takes the form of N (a t x T + b t x 0 , c t I) with c t > 0, which entails for any x t , t ∈ [ϵ, T -γ], q t|T (x t |x T = y) = E x [q t|0T (x t |x 0 = x, x T = y)] > 0.Hence, Eqn. (41) implies that for all t ∈ [ϵ, T -γ], (x, y) ∼ q data (x, y), x t ∼ q t|0T (x t |x 0 = x, x T = y), we have:EQUATIONBy the nature of the distance metric function d and the stopgrad operator, we then have:EQUATIONDefine the error term at timestep t ∈ [ϵ, T -γ] as:EQUATIONWe have:e t = h θ (x t , t, y) -h ϕ (x t , t, y) = h θ ( xϕ (x t , t, r(t), y), r(t), y) -h ϕ (x r(t) , r(t), y) = h θ ( xϕ (x t , t, r(t), y), r(t), y) -h θ (x r(t) , r(t), y) + h θ (x r(t) , r(t), y) -h ϕ (x r(t) , r(t), y) = h θ ( xϕ (x t , t, r(t), y), r(t), y) -h θ (x r(t) , r(t), y) + e r(t) .Since h θ is Lipschitz with constant L and the ODE solver xϕ (•, t, r, y) is bounded by O((t -r) p+1 ) with p ≥ 1, we have:∥e t ∥ 2 ≤ ∥e r(t) ∥ 2 + L∥ xϕ (x t , t, r(t), y) -x r(t) ∥ 2 = ∥e r(t) ∥ 2 + L • O((t -r(t)) p+1 ) = ∥e r(t) ∥ 2 + O((t -r(t)) p+1 ).From the boundary condition of the consistency function, we have:e ϵ = h θ (x ϵ , ϵ, y) -h ϕ (x ϵ , ϵ, y) = x ϵ -x ϵ = 0.Denote r m (t) as applying r on t for m times, since ∆t min = min t {t -r(t)} exists, there exists N such that r n (t) = ϵ for n ≥ N . We thus have:∥e t ∥ 2 ≤ ∥e ϵ ∥ 2 + N k=1 O((r k-1 (t) -r k (t)) p+1 ) = N k=1 O((r k-1 (t) -r k (t)) p+1 ) = N k=1 (r k-1 (t) -r k (t))O((r k-1 (t) -r k (t)) p ) ≤ N k=1 (r k-1 (t) -r k (t))O((∆t max ) p ) = O((∆t max ) p ) N k=1 (r k-1 (t) -r k (t)) = O((∆t max ) p )(t -ϵ) ≤ O((∆t max ) p )(T -ϵ) = O((∆t max ) p ).Proposition 3.3. Given ∆t max = max t {t -r(t)} and assume d, h θ , f, g are twice continuously differentiable with bounded second derivatives, the weighting function λ(•) is bounded, andE[∥∇ xt log q t|T (x t |x T )∥ 2 2 ] < ∞.Meanwhile, assume that L ∆tmax CBD employs the one-step ODE solver in Eqn. (16) with ground truth pre-trained score model, i.e., ∀t ∈ [ϵ, T -γ], y ∼ q data (y) : s ϕ (x t , t, y) ≡ ∇ xt log q t|T (x t |x T = y). Then, we have:L ∆tmax CBD = L ∆tmax CBT + o(∆t max ).The core technique for building the connection between consistency distillation and consistency training with Taylor Expansion also directly follows [58] . The major difference lies in the form of the bridge ODE and the general noise schedule & the first-order ODE solver studied in our work.Proof. First, for a twice continuously differentiable, multivariate, vector-valued function h(x, t, y), denote ∂ k h(x, t, y) as the Jacobian of h over the k-th variable. Consider the CBD objective with first-order ODE solver in Eqn. ( 16) (ignore terms taking expectation for notation simplicity):EQUATIONwherek 1 (t, r) = αrρr ρr αtρt ρt , k 2 (t, r) = αr ρ 2 T ρ2 r -ρtρr ρr ρt , k 3 (t, r) = αr α T ρ 2 T ρ 2 r -ρtρr ρr ρtare coefficients of x t , x ϕ , y in the first-order ODE solver in Eqn. ( 16), x ϕ is pre-trained data predictor. By applying first-order Taylor expansion on Eqn. (45), we have:L ∆tmax CBD =E [λ(t)d (h θ (x t , t, y), h θ -(x t + (k 1 (t, r) -1)x t + k 2 (t, r)x ϕ + k 3 (t, r)y, t + (r -t), y))] =E [λ(t)d (h θ (x t , t, y), h θ -(x t , t, y) + ∂ 1 h θ -(x t , t, y)[(k 1 (t, r) -1)x t + k 2 (t, r)x ϕ + k 3 (t, r)y] +∂ 2 h θ -(x t , t, y)(r -t) + o(|t -r|))] .Here the error term w.r.t. the first variable can be obtained by applying Taylor expansion onk(t, r) = k(t, t) + ∂ 2 k(t, t)(r -t) + o(|t -r|) with k 1 (t, t) -1 = 0, k 2 (t, t) = k 3 (t, t) = 0. By applyingTaylor expansion on d, we have:L ∆tmax CBD =E{λ(t)d(h θ (x t , t, y), h θ -(x t , t, y)) + λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[ ∂ 1 h θ -(x t , t, y)[(k 1 (t, r) -1)x t + k 2 (t, r)x ϕ + k 3 (t, r)y] + ∂ 2 h θ -(x t , t, y)(r -t) + o(|t -r|)]} =E{λ(t)d(h θ (x t , t, y), h θ -(x t , t, y))} + E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[∂ 1 h θ -(x t , t, y)[(k 1 (t, r) -1)x t + k 2 (t, r)x ϕ + k 3 (t, r)y]]} + E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))∂ 2 h θ -(x t , t, y)(r -t)} + E{o(|t -r|)}.Then we focus on the term related to the first-order ODE solver:(k 1 (t, r) -1)x t + k 2 (t, r)x ϕ + k 3 (t, r)y.By the transformation between data and score predictor x ϕ =xt-atx T +c 2 t s ϕ bt , and substitute s ϕ (x t , t, y) with ∇ xt log q t|T (x t |x T = y), we have:(k 1 (t, r) -1)x t + k 2 (t, r) x t -a t x T + c 2 t ∇ xt log q t|T (x t |x T = y) b t + k 3 (t, r)y.Next, substituting the score ∇ xt log q t|T (x t |x T = y) with the unbiased estimator:E[∇ xt log q t|0T (x t |x 0 , x T )|x t , x T = y] = E - x t -(a t x T + b t x 0 ) c 2 t |x t , x T = yWe then have:E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[∂ 1 h θ -(x t , t, y)[(k 1 (t, r) -1)x t + k 2 (t, r)x ϕ + k 3 (t, r)y]]} =E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[∂ 1 h θ -(x t , t, y)[ (k 1 (t, r) -1)x t + k 2 (t, r) x t -a t x T + c 2 t E -xt-(atx T +btx0) c 2 t |x t , x T = y b t + k 3 (t, r)y (i) =E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[∂ 1 h θ -(x t , t, y)[ (k 1 (t, r) -1)x t + k 2 (t, r) x t -a t x T -c 2 t xt-(atx T +btx0) c 2 t b t + k 3 (t, r)y] =E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[∂ 1 h θ -(x t , t, y)[k 1 (t, r)x t + k 2 (t, r)x + k 3 (t, r)y -x t ],where (i) comes from the law of total expectation. Then we apply Taylor expansion in the reverse direction:L ∆tmax CBD =E{λ(t)d(h θ (x t , t, y), h θ -(x t , t, y))} + E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[∂ 1 h θ -(x t , t, y)[k 1 (t, r)x t + k 2 (t, r)x + k 3 (t, r)y -x t ]]} + E{λ(t)∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))∂ 2 h θ -(x t , t, y)(r -t)} + E{o(|t -r|)}. =E{λ(t)[d(h θ (x t , t, y), h θ -(x t , t, y)) + ∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))[∂ 1 h θ -(x t , t, y)[k 1 (t, r)x t + k 2 (t, r)x + k 3 (t, r)y -x t ]] + ∂ 2 d(h θ (x t , t, y), h θ -(x t , t, y))∂ 2 h θ -(x t , t, y)(r -t)]} + E{o(|t -r|)} =E{λ(t)[d(h θ (x t , t, y), h θ -(x t , t, y) + ∂ 1 h θ -(x t , t, y)[k 1 (t, r)x t + k 2 (t, r)x + k 3 (t, r)y -x t ] + ∂ 2 h θ -(x t , t, y)(r -t))]} + E{o(|t -r|)} =E{λ(t)[d(h θ (x t , t, y), h θ -(k 1 (t, r)x t + k 2 (t, r)x + k 3 (t, r)y, r, y))]} + E{o(|t -r|)} (ii) = E{λ(t)[d(h θ (a t y + b t x + c t z, t, y), h θ -(a r y + b r x + c r z, r, y)]} + o(|t -r|) =L ∆tmax CBT + o(|t -r|), where (ii) follows the derivation in Eqn. (33) -Eqn. (36) , and z ∼ N (0, I). Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The release of the code needs an official procedure related to the authors' affiliation, which is not approved yet.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.https://github.com/alexzhou907/DDBM 


Decision: This manuscript incorporates ideas from previous work on "consistency models" to generalize recent work on denoising diffusion bridge models, to allow high-quality data generation with a coarser (and thus computationally cheaper) discretization of the underlying ODE.  Reviewers are positive about the approach, but had a number of concerns about technical details, which were ultimately addressed by the author rebuttal.  Please be sure that these clarifications and improvements are incorporated in the final manuscript, and work to make your results accessible to the broader NeurIPS community.

Reviews: review: paper_summary: paper_summary: This paper proposes to combine the consistency model (CM) with diffusion denoising bridge models (DDBMs) to build a consistency diffusion bridge model (CDBMs), which includes two paradigms of consistency bridge distillation and consistency bridge training. The experiments are conducted for image inpainting and image-to-image translation tasks.
strength_weakness: strength_weakness: - The proposed method is reasonable but may not be very surprising. This work looks more like a direct combination of CM and DDBM, which models have been introduced a lot in previous works, including consistency distillation and consistency training. 

- Question about the motivation to combine CM and DDBM. CM is designed to reduce NFE steps requirement for diffusion sampling. DDBM models can also be able to achieve this goal by learning transformation between two data distributions, such as what is claimed in I2SB, while sacrificing the advantage of unsupervised learning. What is the key motivation and necessity to combine these two together?

- In experiments results, some common metrics like PSNR and SSIM are not reported. In Table 3, the compared methods are most unsupervised learning approaches which are not trained with paired data. In this case, more NFE steps are expected. Therefore, for a fair comparison, the NFE for the compared baselines in Table 2 and 3 may also need to be noted. 

- For qualitative results, the comparison with previous DDBM method under the same NFE would be appreciated, since some DDBM method such as I2SB and CDDB [1] also claimed enabling image generation or restoration within a very few NFE steps from 1-4. It would be important comparison to compare with these baselines under the same experiment settings including NFE.

- Due to the requirement for paired data training, the proposed model needs to be retrained to different tasks, which are limited compared with diffusion-based methods. What is the computational complexity and parameter generalization robustness to retrain the model on a new task? 


[1] Chung, Hyungjin, Jeongsol Kim, and Jong Chul Ye. "Direct diffusion bridge using data consistency for inverse problems." Advances in Neural Information Processing Systems, 2023.
questions: questions: Please see the weaknesses for specific questions.
limitations: limitations: The author discusses the limitations of the proposed method in the last section.
paper_summary: paper_summary: Motivated by faster sampling time, the authors investigate consistency model techniques applied to denoising diffusion bridge models, a particular formulation of conditioned diffusion model whereby the forward process is a mixture of diffusion bridge SDEs, with a conditioned terminal point.

In particular, the authors adapt typical consistency model training regimed known as "consistency model training" - training the model from scratch -  and "consistency model distillation" - distilling from a pretrained diffusion (bridge) model.
strength_weakness: strength_weakness: Denoising diffusion bridge models are a particular formulation of conditioned diffusion model whereby the forward process is given by the SDE of a diffusion bridge, with a conditioned terminal point. This has the benefit of data to data translation, but in my opinion the authors exaggerate the significance of this and I would not consider it "a new family of generative models" [line 4]. Given the consistency methods applied to diffusion models transfer trivially to diffusion bridge models, I am of the opinion that the novelty of the methodological contribution is quite limited.

Furthermore I would also question the usefulness of data to data generative models which are deterministic. For any ill-posed inverse problem, practitioners would want to be able to sample from many possible uncorrupted sample given a single corrupted sample, including some example provided here such as for inpainting.

For consistency models or indeed any flow type model, a Gaussian is typically chosen as marginal which adds stochasticity and permits the PF ODE to match the diffusion model in marginal distributions. This is not the case for diffusion bridge models with conditioned terminal point, there is no longer any stochasticity and hence the ODE derived cannot match the marginals of the stochastic generative backward process. Hence I wonder if PF-ODE of a diffusion bridge model even makes sense in the same way as a diffusion model? Given all the consistency approaches rely on the deterministic component, I believe this is quite fundamental and needs to be addressed.

There are a number of related works that could be discussed. In particular Augmented bridge matching (https://arxiv.org/abs/2311.06978) which draws an equivalence between denoising diffusion bridge models and conditioned bridge matching.


Other:
>  tractable class of Schrödinger Bridge and simulation-free, non-iterative training procedure [32, 6]

[32,6] consider regular bridge matching without any connection to an optimal coupling and hence although tractable just a bridge matching model and not a schrodinger bridge. This is a common mistake.
questions: questions: See weaknesses.
limitations: limitations: See weaknesses.
paper_summary: paper_summary: In this paper, the authors combine Denoising Diffusion Bridge Models (DDBMs), that build a transport map between two arbitrary target distributions (that can be coupled) through a stochastic process, with recent advances on consistency techniques [1], that were originally designed for denoising diffusion models. The motivation herein is to speed up the inference process of DDBMs, that turns out to be costly due to a high number of neural network evaluations. Following the approach from [1], the authors propose to learn the *consistency function* of the probability flow associated to an arbitrary DDBM, which outputs a prediction of the target distribution given any input of the DDBM process $x_t$, for any time $t$. To do so, they derive a consistency loss, which can be seen as the natural extension of the loss of [1] to the bridge setting. Then, they propose two paradigms for this approach: either *consistency bridge distillation*, which assumes to have access to a pretrained DDBM model, and *consistency bridge training*, where no model is available. Their formulation encompasses popular designs of diffusion bridges such as the Brownian brigde, the Image-to-Image Schrödinger bridge [2], the original bridges designed for DDBMs [3]... The authors provide practical guidelines on the neural network parameterization and preconditioning, the design of the ODE solver and the consistency schedule. Finally, they conduct numerical experiments on high-dimensional datasets that demonstrate that the proposed consistency bridge model is faster than the original model while having equal or better generative performance. They notably find out that fine-tuning a pretrained DDBM with consistency training provides better performance under lower computational budget than distillation.

[1] Consistency models. Song et al. 2023.

[2] I2SB: Image-to-Image Schrödinger Bridge. Liu et al. 2023.

[3] Denoising diffusion bridge models. Zhou et al. 2023
strength_weakness: strength_weakness: - The current paper does not exhibit any particular novelty compared to the setting from [1], but this is not a major weakness to me.
- I think that the paper does not provide enough intuition on the design of a good consistency schedule.

[1] Consistency models. Song et al. 2023.
questions: questions: - Could you give further details and intuition on the design of the consistency schedules $r(t)$, which seem to be crucial so that consistency works (even if these reasons are heuristic) ? In particular, could you give examples of schedules that may be adapted in theory but do not work in practice ? It still unclear for me which design should be the best depending on the type of consistency (either distillation or training).
- I have a question on the approximation made on the score in Equation (15) for consistency training, which can actually also be addressed to standard consistency models. Theoretically, up to the terminal conditioning in the bridge setting, the score evaluated in $x$ at time $t$ has to be understood as a conditional expectation over the posterior distribution of $X_0$ given $X_t=x$ by Tweedie's formula. This distribution has a very specific behaviour: (i) when $t\approx0$ (close to the target distribution), it is concentrated like a Dirac mass; (ii), on the other hand when $t\approx T$ (full noise), the posterior is approximately equal to the target distribution itself. This is well highlighted in the concept of duality of log-concavity in [1]. Therefore, the approximation of the score as presented in the current paper (that is evaluating the integrand in only one point $x_0$) is relevant when $t\approx0$, but completely false when $t\approx T$. Do you have an estimation of the error made in the latter case for tractable cases ? Does it have an impact on the training for this part of the process ? Have you come up with an idea to correct this intrinsic bias ?

[1] Stochastic localization via iterative posterior sampling. Grenioux et al. 2024
limitations: limitations: The limitations are clearly indicated in Section 5. The authors notably acknowledge that bridge consistency models suffer from numerical instability (same as classic consistency models).
paper_summary: paper_summary: This paper works on efficient sampling of the denoising diffusion bridge models (DDBMs) or similar. In particular, the paper proposes to extend the consistency models (CMs) to DDBMs.

CMs are generative models developed in the context of improving the sampling cost of diffusion-based generative models (DBGMs). Note that sampling of the probability flows of the DBGMs runs the integrators up to a few hundred steps or more. Similar to common DBGMs, assume we have a forward diffusion whose initial distribution—data—is denoted by $x_0$. In particular, when we rewrite the forward diffusion in the probability flow (PF) form, we get an ODE whose marginal is equivalent to the original forward diffusion. We further assume that we have the ODE-integrators that run $t$-decreasing direction starting from any given $t \in (0, T]$ to $0$. Then, the CMs train to predict the output of the integrators, similar to denoising autoencoder, but the perturbation here is deterministic. Training such models uses the consistency condition. Once we train the CMs, we can use them instead of running a few hundred steps of DGBMs.

DDBMs are input-conditional bridge models. More specifically, assume we have two distributions and their joint distribution, e.g., one is an image distribution, and the other is its edge images. The bridge models learn a diffusion to connect one distribution to the other. Unlike common bridge models, however, DDBMs conditions on its initial point, e.g., an edge image, so that the path generated by the learned diffusion preserves the information of the initial value. For several applications, such as image-to-image translations, these DDBMs show several benefits, unlike common Markovian models; in particular, the initial coupling of the joint distributions is preferred. Nevertheless, DDBMs are also required to run the SDE-integrators (or ODE-integrators for its PFs) a few hundred steps, similar to any other diffusion-based models.

In order to overcome sampling inefficiency in DDBMs, the paper proposes to extend CMs to DDBMs.

Finally, the authors demonstrate the proposed method's effectiveness via various experiments, including image-to-image translations and image inpainting.

---------------------------------------
Updated the rating from 4 to 7 after the authors' rebuttal
strength_weakness: strength_weakness: The authors tackle a highly interesting problem and reach out for an excellent solution. However, I find that the paper requires a major revision due to a few theoretical loopholes in extending the consistency models (CMs) to the denoising diffusion bridge models (DDBMs).

For example, in expressing CM in the form of a stochastic differential equation (SDE), the paper uses a definition that is not mathematically well-defined. Particularly, in Equation 2, the notation of a reverse-time standard Wiener process is mentioned. However, a reverse-time standard Wiener process is not defined in general; more precisely, it is tricky to define such a concept. The reason is that the definition of the It\'o integral is highly sensitive to the direction of time. Consequently, the change of variables used in calculus cannot be applied to SDE directly. As a result, we need to define an SDE-specific chain rule called Ito's lemma. Accordingly, one cannot rewrite an SDE by changing the sign of time. In order to apply the chain rule of calculus, the SDE literature uses the Stieltjes integral instead of the It\'o integral, and a corresponding SDE notation needs to be defined. If this approach were taken for the submitted paper, however, the denoising diffusion and consistency model would need to be defined accordingly.

In conclusion, this paper's development deviates from the fundamental assumptions of the It\'o integral-based SDEs. Note that the original CM circumvents this issue by using only ordinary differential equations (or partial differential equations). Therefore, unlike the It\'o integral, it is free from concerns about the time directions. However, this does not apply to SDEs.

In my understanding, one would be able to reach the same conclusion as the submitted paper while sticking to the fundamentals of SDEs (related to It'o\' diffusion). However, such modification would require a major revision from the current submission. 

In addition, a few statements need to be addressed. In particular, to get an unbiased estimation of the expectation in Equation 15, one needs to sample $x_t$ and $x_T$ first and then sample $x_0$. However, the authors state that we can achieve this by sampling $x_0$ and $x_T$ first and then $x_t$. Such an estimator is biased except when we use a single sample.
questions: questions: N/A
limitations: limitations: N/A

Comments: title: Reply to Reviewer 2JER (Round 2)
Answer to the rebuttal
References
Reference and Additional Results
comment: Thank you for your feedback and we would like to provide the response to your follow-up question as follows:

> Q1: Note that the PSNR and SSIM results in Table for W3 of DDBM (ODE-1, NFE=2) are largely higher than the proposed method CBD / CBT (NFE=2). Could the author explain more about the reason? 

Thank you for the question. We would like to argue that, when evaluating generative models such as diffusion models that hold a certain level of sample diversity, the distortion metrics laid on the pixel space such as PSNR and SSIM may not be the proper major evaluator that aligns with sample quality. For example, DDBM (ODE-1, NFE=2) not only surpasses CBD / CBT (NFE=2) but also surpasses DDBM (ODE-1, NFE=100) in terms of PSNR & SSIM. One should agree that using hundreds of NFEs yields better sample quality than only two NFEs on the image translation task, though NFE=2 has better PSNR & SSIM than NFE=100, which can be justified by the blurry samples under NFE=2. Thus, following the community of diffusion models, we focus on FID as the major metric and claim the accelerated ratio according to this metric, where CBD / CBT (NFE=2) clearly surpasses DDBM (ODE-1, NFE=2). On the other hand, the results in Table for W3 actually further support our claim that CBD / CBT (NFE=2) has comparable sample quality to DDBM (ODE-1, NFE=100) measured by various metrics.

> Q2: It is kind of confusing about the argument with CDDB though. Although the CDDB paper also reports that more NFEs can help, the key is to compare in the same setting?

We apologize for the confusion caused by our argument with CDDB. In this paper, the target of the proposed consistency distillation/tuning is to achieve comparable performance in a few NFEs to their corresponding base model under numerous NFEs. Hence, we mainly compare the performance between the distilled/fine-tuned model (i.e., CBD/CBT) and the corresponding base model (DDBM ODE-1) under different NFEs. We agree that comparing different methods in the same setting is important but we think this should be conducted among different distillation/fine-tuning methods under the same base models targeted for accelerated sampling, which, to the best of our knowledge, is still a vacant area for the diffusion bridge community.

> Q3: The CDDB paper reports that "20 NFE CDDB outperforms 1000 NFE I2SB in PSNR by > 2 db". Can it suppose that CDDB would be a stronger baseline than I2SB to compare with?

We would like to address that CDDB is actually orthogonal to our method since it is a pure inference technique designed for the sampling fashion of diffusion bridges. The sampling process for CBT/CBD can fit into the inference process where CDDB is applicable and thus these two methods can be incorporated together. Moreover, the CDDB requires access to the Gaussian linear measurement for inverse problems, which might be a bit tricky to act as a baseline for fair comparison.
Thanks for the author to answer my questions. Some of my concerns are addressed. (1) Note that the PSNR and SSIM results in results Table for W3 of DDBM (ODE-1, NFE=2) are largely higher than the proposed method CBD / CBT (NFE=2). Could the author explain more about the reason? (2) Regarding the response to W5, the key point is if the experiments comparison is fair, under same NFE. Thanks again for adding new compared results between proposed method and I2SB with the same NFE in the pdf file. It is kind of confusing about the argument with CDDB though. Although the CDDB paper also reports that more NFEs can help, the key is to compare in the same setting? The CDDB paper reports that "20 NFE CDDB outperforms 1000 NFE I2SB in PSNR by > 2 db". Can it suppose that CDDB would be a stronger baseline than I2SB to compare with?
We are glad to see your concerns are successfully addressed and would like to thank you for the valuable review and discussion. We agree that DDBM did not address the validity of the probability flow ODE properly in Theorem 1 of their paper and we also agree that this is quite important. We will add a discussion about this and the full proof under the simple Brownian bridge case in the revised paper.
I was meaning the reverse process is not Markovian but I do not think this is a problem. 

 Thank you for the clarification, I think you are correct - apologies! I appreciate it and this is very interesting and a good contribution to the paper. I think this proof should be included in the paper, unless I've missed it somewhere. The DDBM paper does not have this initial stochastic step so the second part of Theorem 1 of DDBM is incorrect.
We are not totally clear about the Kolmogorov forward equation part you mentioned, but we believe the probability flow ODE part in DDBM is correct.

Firstly, we would like to argue that **diffusion bridges are Markovian processes**, as the condition $x_T$ is **fixed ahead of time**. As a simple example, the Brownian bridge is a Markovian process. This is exercise 5.11 in Oksendal's book "Stochastic Differential Equations". In the forward process, given $s<u<t$, $x_t$ depends on $x_u$, $x_T$ and is independent of $x_s$. In the reverse process, $x_s$ depends on $x_u,x_T$ and is independent of $x_t$. This is intuitive since the diffusion bridge can be represented as a simple SDE and simulated towards a single direction.

Secondly, we can **rigorously prove** that in the simple Brownian bridge case, simulating the SDE from $T$ to $T-\gamma$, and then simulating the ODE from $T-\gamma$ **maintains the marginals**. For simplification, assume $T=1$. In the time region $[1-\gamma,1]$, the SDE has no singularity so the marginal is the same as the forward process $p(x_t|x_0,x_1)=N(tx_1+(1-t)x_0,t(1-t))$ (you should agree with this). Then the ground-truth probability flow ODE can be derived as (we follow DDBM paper and omit the derivation process here, but we can provide the details if you doubt this)

$$
\frac{dx_t}{dt}=\frac{1-2t}{2t(1-t)}x_t+\frac{1}{2(1-t)}x_1-\frac{1}{2t}x_0
$$
The analytic solution of the ODE from time $t$ to time $s<t$ is (also we omit the derivation, but we can provide the details if you doubt this)

$$
\frac{x_s}{\sqrt{s(1-s)}}-\frac{x_t}{\sqrt{t(1-t)}}=\left(\frac{s}{\sqrt{s(1-s)}}-\frac{t}{\sqrt{t(1-t)}}\right)x_1+\left(\frac{1-s}{\sqrt{s(1-s)}}-\frac{1-t}{\sqrt{t(1-t)}}\right)x_0
$$
or
$$
x_s=\frac{\sqrt{s(1-s)}}{\sqrt{t(1-t)}}x_t+\left(s-\frac{\sqrt{s(1-s)}}{\sqrt{t(1-t)}}t\right)x_1+\left(1-s-\frac{\sqrt{s(1-s)}}{\sqrt{t(1-t)}}(1-t)\right)x_0
$$
When $x_t$ follows the marginal $N(tx_1+(1-t)x_0,t(1-t))$, it can be easily verified from the above relation that $x_s$ follows the marginal $N(sx_1+(1-s)x_0,s(1-s))$. Therefore, **in the simple case you mentioned, following the ODE reversely in time will maintain the marginals**.
It is not clear to me that the forward SDE and backward ODE proposed have the same marginals. Consider a brownian bridge x_t between 2 points x_0 and x_T. There is a non zero probability that x_t > L, say p(x_t>L|x_0, x_T)>l>0.  The heuristic proposed is supposedly valid for any gamma>0, then in the backward ODE process p(x_t>L|x_T) -> 0 as gamma -> 0 hence backward and forward do not coincide in distribution for all gamma > 0 ,so the argument proposed seems incorrect. 

The Kolmogorov forward equation typically requires the markov property which does not hold here. It is not clear to me if this is the issue or something else, I am aware of cases where Kolmogorov equations hold for non markov processes but this typically requires more justification than provided


I would be happy to raise my score if this is cleared up but it seems lack of a valid probability flow ODE is a major flaw in the paper.
We would like to make further clarification/discussion regarding the validity of the PF-ODE according to our understanding of DDBM.

First of all, the singularity at marginal points of the ODE is not an implementation/stability issue. Instead, this ODE is theoretically well-defined only on $ 0 \le t < T $, which means it can not be simulated from the given starting point theoretically. The valid way of utilizing the ODE would be first simulating the SDE from $T$ to $T - \gamma$ for some $\gamma > 0$ and then following the ODE, thus the whole sampling process will be stochastic and matches the marginal of the SDE.

Moreover, we think DDBM did give a sound derivation of the PF-ODE in Appendix A.3 of their paper, which utilizes the Kolmogorov forward/backward equations of the reference diffusion process rather than the diffusion bridge. According to this, we are inclined to agree with the reply of DDBM's authors regarding your mentioned two comments:

> We stress that (and also updated the text to include this) that our proposed deterministic process is more technically involved than simply defining a deterministic initial condition and dynamics. In particular, the introduction of randomness comes from the fact that our ODE is only well-defined on $ 0 \le t < T $, as Doob’s h-function causes a singularity at the boundary $ T $. When sampling using the SDE, we need to approximate $ x_{T-\epsilon} \approx y $ and follow the backwards SDE. For the ODE, the source of randomness comes from the initial distribution $p(x_{T-\epsilon})$, which is not the same as $y$. Instead, we sample $x_{T-\epsilon}$ (specifically approximating $x_{T - \epsilon^{\prime}} \approx y$) and then taking an Euler-Maruyama back to $T - \epsilon$), with which we can then sample with the valid probability flow ODE. Note that this clears the singularity and injects randomness while enabling ODE sampling.

In a summary, we think DDBM has already addressed the theoretical validity of the PF-ODE and we adopt their established conclusion as a basis of our work. However, we are open to further inspecting their conclusion if there are indeed critical/fundamental theoretical flaws to avoid propagating errors by follow-up works like ours.
Thank you for the response. 

My primary concern over the validity of the PF-ODE is a theoretical concern not an implementation/ stability issue over the singularity at marginal points. Your response does not address my concern. 

My understanding is that the initial stochastic sampling step is only at sampling / post training. Does this have any theoretical basis?

Let us ignore this heuristic and consider the actual reverse diffusion bridge between two points and the "ODE" the authors describe. The fact remains that this ODE is fully deterministic given the starting point as such it cannot match the marginals of the corresponding stochastic bridge it is determined from at all time intervals. Then it is not clear theoretically why the end marginal should match a data distribution?

DDBM [1] does not derive the PF ODE and provides a very hand wavy reference to Song 2021, however the regime is different so again it is not clear whether the PF ODE holds. The Kolmogorov forward/ backward equations are for Markov processes and the conditioned bridge SDE is clearly not Markov. 

The same issue has been raised with DDBM in review: https://openreview.net/forum?id=FKksTayvGo by both a reviewer and public commenter. I am inclined to agree with the comments there.


> The dynamics of () makes it clear why it is not possible to define a probability-flow ODE matching the marginal distributions of () on : in this hypothetical ODE, i.e. (7), both the initial condition and the dynamics would be deterministic, with no randomness left. The mentioned "averaged out" results correspond to the evolution of this deterministic ODE.

> ODE (7) is not a probability flow ODE matching the marginal distribution of SDE (6)

This is quite a fundamental issue and without being addressed, I am leaning towards lowering my score.
I would like to thank the authors for their precise answers. I will keep my score unchanged.
[1] Zhou, Linqi, et al. "Denoising Diffusion Bridge Models." ICLR 2024.

[2] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." NeurIPS 2022.

[3] Liu, Guan-Horng, et al. "I$^2$SB: Image-to-Image Schrödinger Bridge." ICML 2023.
## Reference
[1] Zhou, Linqi, et al. "Denoising Diffusion Bridge Models." ICLR 2024.

[2] Karras, Tero, et al. "Elucidating the design space of diffusion-based generative models." NeurIPS 2022.

[3] Liu, Guan-Horng, et al. "I$^2$SB: Image-to-Image Schrödinger Bridge." ICML 2023.

[4] Chung, Hyungjin, Jeongsol Kim, and Jong Chul Ye. "Direct diffusion bridge using data consistency for inverse problems." NeurIPS 2023.

---
## Additional Results
* ### Results for W3 (PSNR & SSIM for image translation tasks):

|                       | E2H             |                 | DIODE           |                 |
|-----------------------|-----------------|-----------------|-----------------|-----------------|
|                       | PSNR $\uparrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ |
| DDBM (heun, NFE=118)        |            8.36 |            0.29 |           22.64 |            0.63 |
| DDBM (ODE-1, NFE=2)   |           30.67 |            0.91 |           25.32 |            0.75 |
| DDBM (ODE-1, NFE=50)  |            29.1 |            0.89 |           23.52 |             0.70 |
| DDBM (ODE-1, NFE=100) |           28.91 |            0.89 |           23.41 |            0.69 |
| CBD (Ours, NFE=2)     |           25.25 |            0.84 |           22.68 |            0.65 |
| CBT (Ours, NFE=2)     |           27.97 |            0.88 |           23.44 |            0.68 |

* ### Results for W4 (NFE of baselines in Table 2 & 3):

For table 2:
| Method         | NFE      |
|----------------|----------|
| Pix2Pix        |        1 |
| DDIB           | $\ge 40$ |
| SDEdit         | $\ge 40$ |
| Rectified Flow | $\ge 40$ |
| I$^2$SB        | $\ge 40$ |

For table 3:
| Method   | NFE  |
|----------|------|
| DDRM     |   20 |
| $\Pi$GDM |  100 |
| DDNM     |  100 |
| Palette  | 1000 |
| I$^2$SB  | > 10 |


* ### Results for W5 (Quantative results of the I$^2$SB baseline):
| Model                        | NFE=2 | NFE=3 | NFE=4 | NFE=5 | NFE=8 | NFE=10 |
|------------------------------|-------|-------|-------|-------|-------|--------|
| I$^2$SB (posterior sampling) | 12.49 |  8.55 |   7.10 |  6.38 |  5.49 |   5.26 |
| DDBM (ODE-1)                 | 17.17 | 11.17 |  8.21 |  6.77 |  5.18 |   4.81 |