Title: Consistency Diffusion Bridge Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Bridges

Consistency Models

Image Translation
Abstract: Diffusion models (DMs) have become the dominant paradigm of generative modeling in a variety of domains by learning stochastic processes from noise to data. Recently, diffusion denoising bridge models (DDBMs), a new formulation of generative modeling that builds stochastic processes between fixed data endpoints based on a reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation. However, DDBM's sampling process typically requires hundreds of network evaluations to achieve decent performance, which may impede their practical deployment due to high computational demands. In this work, inspired by the recent advance of consistency models in DMs, we tackle this problem by learning the consistency function of the probability-flow ordinary differential equation (PF-ODE) of DDBMs, which directly predicts the solution at a starting step given any point on the ODE trajectory. Based on a dedicated general-form ODE solver, we propose two paradigms: consistency bridge distillation and consistency bridge training, which is flexible to apply on DDBMs with broad design choices. Experimental results show that our proposed method could sample $4\times$ to $50\times$ faster than the base DDBM and produce better visual quality given the same step in various tasks with pixel resolution ranging from $64 \times 64$ to $256 \times 256$, as well as supporting downstream tasks such as semantic interpolation in the data space.

Consistency Diffusion Bridge Models

Abstract

Diffusion models (DMs) have become the dominant paradigm of generative modeling in a variety of domains by learning stochastic processes from noise to data. Recently, diffusion denoising bridge models (DDBMs), a new formulation of generative modeling that builds stochastic processes between fixed data endpoints based on a reference diffusion process, have achieved empirical success across tasks with coupled data distribution, such as image-to-image translation. However, DDBM's sampling process typically requires hundreds of network evaluations to achieve decent performance, which may impede their practical deployment due to high computational demands. In this work, inspired by the recent advance of consistency models in DMs, we tackle this problem by learning the consistency function of the probability-flow ordinary differential equation (PF-ODE) of DDBMs, which directly predicts the solution at a starting step given any point on the ODE trajectory. Based on a dedicated general-form ODE solver, we propose two paradigms: consistency bridge distillation and consistency bridge training, which is flexible to apply on DDBMs with broad design choices. Experimental results show that our proposed method could sample 4 × to 50 × faster than the base DDBM and produce better visual quality given the same step in various tasks with pixel resolution ranging from 64 × 64 to 256 × 256 , as well as supporting downstream tasks such as semantic interpolation in the data space.

1 Introduction

Diffusion models (DMs) [53, 21, 60] have reached unprecedented levels as a family of generative models in various areas, including image generation [10, 50, 48], audio synthesis [5, 45], video generation [20], as well as image editing [41, 42], solving inverse problems [25, 56], and density estimation [59, 28, 37, 71]. In the era of AI-generated content, the stable training, scalability & state-of-the-art generation performance of DMs successfully make them serve as the fundamental component of large-scale, high-performance text-to-image [14] and text-to-video [18, 2] models.

A critical characteristic of diffusion models is their iterative sampling procedure, which progressively drives random noise into the data space. Although this paradigm yields a sample quality that stands out from other generation models, such as VAEs [29, 46], GANs [17], and Normalizing Flows [11, 12, 30], it also results in a notoriously lower sampling efficiency compared to other arts. In response to this, consistency models [58] have emerged as an attractive family of generative models by learning a consistency function that directly predicts the solution of a probability-flow ordinary differential equation (PF-ODE) at a certain starting timestep given any points in the ODE trajectory, designed to be a one-step generator that directly maps noise to data. Consistency models can be

naturally integrated with diffusion models by adapting the score estimator of DMs to a consistency function of their PF-ODE via distillation [58, 26] or fine-tuning [15], showing promising performance for few-step generation in various applications like latent space [40] and video [64].

Despite the remarkable achievements in generation quality and better sampling efficiency, a fundamental limitation of diffusion models is that their prior distribution is usually restricted to a non-informative Gaussian noise, due to the nature of their underlying data to noise stochastic process. This characteristic may not always be desirable when adopting diffusion models in some scenarios with an informative non-Gaussian prior, such as image-to-image translation. Alternatively, an emergent family of generative models focuses on leveraging diffusion bridges, a series of altered diffusion processes conditioned on given endpoints, to model transport between two arbitrary distributions [44, 36, 33, 54, 51, 72, 7]. Among them, denoising diffusion bridge models (DDBMs) [72] study the reverse-time diffusion bridge conditioned on the terminal endpoint, and employ simulation-free, non-iterative training techniques for it, showing superior performance in application with coupled data pairs such as distribution translation compared to diffusion models. However, DDBMs generally require hundreds of network evaluations to produce samples with decent quality, even using an advanced high-order hybrid sampler, potentially hindering their deployments in real-world applications.

In this work, inspired by recent advances in consistency models with diffusion ODEs [58, 57, 15], we introduce consistency diffusion bridge models (CDBMs) and develop systematical techniques to learn the consistency function of the PF-ODEs in DDBMs for improved sampling efficiency. Firstly, to facilitate flexible integration of consistency models in DDBMs, we present a unified perspective on their design spaces, including noise schedule, prediction target, and network parameterizations, termed the same as in diffusion models [28, 24]. Additionally, we derive a first-order ODE solver based on the general-form noise schedule. This universal framework largely decouples the formulation of DDBMs and the corresponding consistency models from highly practical design spaces, allowing us to reuse the successful empirical choices of various diffusion bridges for CDBMs regardless of their different theoretical premises. On top of this, we then propose two paradigms for training CDBMs: consistency bridge distillation and consistency bridge training. This approach is free of dependence on a restricted form of noise schedule and the corresponding Euler ODE solver as in previous work [58], thus enhancing the practical versatility and extensibility of the CDBM framework.

We verify the effectiveness of CDBMs in two applications: image translation and image inpainting by distilling or fine-tuning DDBMs with various design spaces. Experimental results demonstrate that our approach can improve the sampling speed of DDBMs from 4 × to 50 × , in terms of the Fréchet inception distance [19] (FID) evaluated with two-step generation. Meanwhile, given the same computational budget, CDBMs have better performance trade-offs compared to DDBMs, both quantitatively and qualitatively. CDBMs also retain the desirable properties of generative modeling, such as sample diversity and the ability to perform semantic interpolation in the data space.

2 Preliminaries

2.1 Diffusion Models

Given the data distribution p data ( x ) , x ∈ R m , diffusion models [53, 21, 60] specify a forward-time diffusion process from an initial data distribution p 0 = p data to a terminal distribution p T within a finite time horizon t ∈ [0 , T ] , defined by a stochastic differential equation (SDE):

where w t is a standard Wiener process, f : R m × [0 , T ] → R m and g : [0 , T ] → R d are drift and diffusion coefficients, respectively. The terminal distribution p T is usually designed to approximate a tractable prior p prior (e.g., standard Gaussian) with the appropriate choice of f and g . The corresponding reverse SDE and the probability flow ordinary differential equation (PF-ODE) of the forward SDE in Eqn. (1) is given by [1, 60]:

where ¯ w t is a reverse-time standard Wiener process and p t ( x t ) is the marginal distribution of x t . Both the reverse SDE and PF-ODE can act as a generative model by sampling x T ∼ p prior and simulating the trajectory from x T to x 0 . The major difficulty here is that the score function ∇ log p t ( x t ) remains unknown, which can be approximated by a neural network s θ ( x t , t ) with denoising score matching [63]:

where U (0 , T ) is uniform distribution, λ ( t ) > 0 is a weighting function, and p t | 0 ( x t | x 0 ) is the transition kernel from x 0 to x t . Acommonpractice is to use a linear drift f ( t ) x t such that p t | 0 ( x t | x 0 ) is an analytic Gaussian distribution N ( α t x 0 , σ 2 t I ) , where α t = e ∫ t 0 f ( τ )d τ , σ 2 t = α 2 t ∫ t 0 g 2 ( τ ) α 2 τ d τ is defined as the noise schedule [28]. The resulting score predictor s θ ( x t , t ) can replace the true score function in Eqn. (2) and (3) to obtain the empirical diffusion SDE and ODE, which can be simulated by various SDE or ODE solvers [55, 38, 39, 16, 70].

2.2 Consistency Models

Given a trajectory { x t } T t = ϵ with a fixed starting timestep ϵ of a PF-ODE, consistency models [58] aim to learn the solution of the PF-ODE at t = ϵ , also known as the consistency function , defined as h : ( x t , t ) ↦→ x ϵ . The optimization process for consistency models contains the online network h θ and a reference target network h θ -, where θ -refers to θ with operation stopgrad , i.e., θ -= stopgrad( θ ) . The networks are hand-designed to satisfy the boundary condition h θ ( x ϵ , ϵ ) = x ϵ , which can be typically achieved with proper parameterization on the neural network. For PF-ODE taking the form in Eqn. (3) with a linear drift f ( t ) x t , the overall learning objective of consistency models can be described as:

where r ( t ) is a function that specifies another timestep r (usually with t > r ), d denotes some metric function with ∀ x , y : d ( x , y ) ≥ 0 and d ( x , y ) = 0 iff. x = y . Here ˆ x r is a function that estimates x r = x t + ∫ r t d x τ d τ d τ , which can be done by simulating the empirical diffusion ODE with a pre-trained score predictor s ϕ ( x t , t ) or empirical score estimator -x t -α t x 0 σ 2 t . The corresponding learning paradigms are named consistency distillation and consistency training , respectively.

2.3 Denoising Diffusion Bridge Models

Given a data pair sampled from an arbitrary unknown joint distribution ( x , y ) ∼ q data ( x , y ) , x , y ∈ R m and let x 0 = x , denoising diffusion bridge models (DDBMs) [72] specify a stochastic process

that ensures x T = y almost surly via applying Doob's h -transform [13, 47] on a reference diffusion process in Eqn. (1):

where p T | t ( x T = y | x t ) is the transition kernel of the reference diffusion process from t to T , evaluated at x T = y . Denoting the marginal distribution of Eqn. (6) as { q t } T t =0 , it can be shown that the forward bridge SDE in Eqn. (6) is characterized by the diffusion distribution conditioned on both endpoints, that is, q t | 0 T ( x t | x 0 , x T ) = p t | 0 T ( x t | x 0 , x T ) , which is an analytic Gaussian distribution. A generative model can be obtained by modeling q t | T ( x t | x T = y ) , whose reverse SDE and PF-ODE are given by:

The only unknown term remains is the score function ∇ x t log q t | T ( x t | x T = y ) , which can be estimated with a neural network s θ ( x t , t, y ) via denoising bridge score matching (DBSM):

Replacing ∇ x t log q t | T ( x t | x T = y ) in Eqn. (7) and (8) with the learned score predictor s θ ( x t , t, y ) would yield the empirical bridge SDE and ODE that could be solved for generation purposes.

3 Consistency Diffusion Bridge Models

In this section, we introduce consistency diffusion bridge models, extending the techniques of consistency models to DDBMs to further boost their performance and sample efficiency. Define the consistency function of the bridge ODE in Eqn. (8) as h : ( x t , t, y ) ↦→ x ϵ with a given starting timestep ϵ , our goal is to learn the consistency function using a neural network h θ ( · , · , y ) with the following high-level objective similar to Eqn. (5):

To begin with, we first present a unified view of the design spaces such as noise schedule, network parameterization & precondition, as well as a general ODE solver for DDBMs. This allows us to: (1) decouple the successful practical designs of previous diffusion bridges from their different theoretical premises; (2) decouple the framework of consistency models from certain design choices of the corresponding PF-ODE, such as the reliance on VE schedule with Euler ODE solver of the original derivation of consistency models [58]. This would largely facilitate the development of consistency models that utilize the rich design spaces of existing diffusion bridges on DDBMs in a universal way. Then, we elaborate on two ways to train h θ based on different choices of ˆ x r , consistency bridge distillation, and consistency bridge training, with the proposed unified design spaces.

3.1 A Unified View on Design Spaces of DDBMs

Noise Schedule We consider the linear drift f ( t ) x t and define:

which aligns with the common notation of noise schedules used in diffusion models by denoting σ t = α t ρ t . Then we could express the analytic conditional distributions of DDBMs as follows:

The form of q t | 0 T is consistent with the original formulation of DDBM in [72]. Here, inspired by [6], we opt to adopt a more neat set of notations for enhanced compatibility. As shown in Table 1, with such notations, we could easily unify the design choices for diffusion bridges [33, 72, 6] that have shown effectiveness in various tasks and expeditiously employ consistency models on top of them.

Network Parameterization & Precondition In practice, the neural network F θ in DBMs does not always directly regress to the target score function; instead, it can predict other equivalent quantities, such as the data predictor x θ = x t -a t x T + c 2 t s θ b t for a Gaussian N ( a t x T + b t x 0 , c 2 t I ) like q t | 0 T . Meanwhile, the inputs and outputs of the network F θ could be rescaled for a better-behaved optimization process, known as the network precondition. As shown in Table 1, we could consistently use x 0 as the prediction target with different choices of network precondition to unify the previous practical designs for DBMs.

PF-ODE and ODE Solver The validity of a consistency model relies on an underlying PF-ODE that shares the same marginal distribution with the forward process. In the original DDBM paper [72], the marginal preserving property of the proposed ODE is justified following an analogous logic from the derivation of the PF-ODE of diffusion models [60] with Kolmogorov forward equation. However, its validity suffers from doubts as there is a singularity at the deterministic starting point x T . Here, we provide a simple example to show that the ODE can indeed maintain the marginal distribution as long as we use a valid stochastic step to skip the singular point and start from T -γ for any γ > 0 .

Example 3.1. Assume T = 1 and consider a simple Brownian Bridge between two fixed points ( x 0 , x 1 ) :

with marginal distribution q t | 01 ( x t | x 0 , x 1 ) = N ((1 -t ) x 0 + tx 1 , t (1 -t )) . The ground-truth reverse SDE and PF-ODE are given by:

Then first simulating the reverse SDE in Eqn. (14) from t = 1 to t = 1 -γ for some γ ∈ (0 , 1) and then starting to simulate the PF-ODE in Eqn. (15) will preserve the marginal distribution.

The detailed derivation can be found in Appendix. B.2. Therefore, the time horizon of the consistency model based on the bridge ODE needs to be set as t ∈ [ ϵ, T -γ ] for some pre-specified ϵ, γ > 0 . Additionally, the marginal preservation of the bridge ODE for more general diffusion bridges can be strictly justified by considering non-Markovian variants, as done in DBIM [69].

Another crucial element for developing consistency models is the ODE solver, as a solver with a lower local error would yield lower error for consistency distillation, as well as the corresponding

consistency training objectives [58, 57]. Inspired by the successful practice of advanced ODE solvers based on the Exponential Integrator (EI) [4, 22] in diffusion models, we present a first-order bridge ODE solver in a similar fashion:

Proposition 3.1. Given an initial value x t at time t > 0 , the first-order solver of the bridge ODE in Eqn. (8) from t to r ∈ [0 , t ] with the noise schedule defined in Eqn. (11) is:

We provide detailed derivation in the Appendix B.1. Typically, an EI-based solver enjoys a lower discretization error and therefore has better empirical performance [16, 38, 39, 67, 70]. Another notable advantage of this general form solver, as we will show in Section 3.3, is that it could naturally establish the connection between consistency training and consistency distillation for any noise schedules that take the form in Eqn. (11), eliminating the dependence of the VE schedule and the corresponding Euler ODE solver in the common derivation [58].

3.2 Consistency Bridge Distillation

Analogous to consistency distillation with the empirical diffusion ODE, we could leverage a pretrained score predictor s ϕ ( x t , t, y ) ≈ ∇ x t log q t | T ( x t | x T = y ) to solve the empirical bridge ODE to obtain ˆ x r , i.e., ˆ x r = ˆ x ϕ ( x t , t, r, y ) , where ˆ x ϕ is the update function of a one-step ODE solver with fixed s ϕ . We define the consistency bridge distillation (CBD) loss as:

where t is sampled from the uniform distribution over [ ϵ, T -γ ] , r ( t ) is a function specifies another timestep r such that ϵ ≤ r < t with ∆ t max := max t { t -r ( t ) } and ∆ t min := min t { t -r ( t ) } , λ ( t ) is a positive weighting function, d is some distance metric function with ∀ x , y : d ( x , y ) ≥ 0 and d ( x , y ) = 0 iff. x = y , and θ -= stopgrad( θ ) . Similarly to the case of consistency distillation in empirical diffusion ODEs, we have the following asymptotic analysis of the CBD objective:

Proposition 3.2. Given ∆ t max = max t { t -r ( t ) } and let h ϕ ( · , · , · ) be the consistency function of the empirical bridge ODE taking the form in Eqn. (8). Assume h θ is a Lipschitz function, i.e., there exists L > 0 , such that for all t ∈ [ ϵ, T -γ ] , x 1 , x 2 , y , we have ∥ h θ ( x 1 , t, y ) -h θ ( x 2 , t, y ) ∥ 2 ≤ L ∥ x 1 -x 2 ∥ 2 . Meanwhile, assume that for all t, r ∈ [ ϵ, T -γ ] , y ∼ q data ( y ) := E x [ q data ( x , y )] , the ODE solver ˆ x ϕ ( · , t, r, y ) has local error uniformly bounded by O (( t -r ) p +1 ) with p ≥ 1 . Then, if L ∆ t max CBD = 0 , we have: sup t, x , y ∥ h θ ( x , t, y ) -h ϕ ( x , t, y ) ∥ 2 = O ((∆ t max ) p ) .

The vast majority of the analysis can be done by directly following the proof in [58] with minor differences between the overlapped timestep intervals { t, r ( t ) } for t ∈ [ ϵ, T -γ ] used in Eqn. (17) and the fixed timestep intervals { t n } N n =1 used in [58]. We include it in Appendix B.5 for completeness. In this work, unless otherwise stated, we use the first-order ODE solver in Eqn. (16) as ˆ x ϕ .

3.3 Consistency Bridge Training

In addition to distilling from pre-trained score predictor s ϕ , consistency models can be trained [58, 57] or fine-tuned [15] by maintaining only one set of parameters θ . To accomplish this, we could leverage the unbiased score estimator:

that is, with a single sample ( x , y ) ∼ q data and x t ∼ q t | 0 T ( x t | x 0 = x , x T = y ) , the score ∇ x t log q t | T ( x t | x T = y ) can be estimated with ∇ x t log q t | 0 T ( x t | x 0 , x T ) . Substituting such an estimation of s ϕ into the one-step ODE solver ˆ x ϕ in Eqn. (17) with the transformation between data and score predictor x ϕ = x t -a t x T + c 2 t s ϕ b t , we can obtain an alternative ˆ x r that does not rely on the pre-trained s ϕ for any noise schedule taking the form in Eqn. (11) as follows (detail in Appendix B.3):

where a r , b r , c r are defined as in Eqn. (11), and z = x t -a t y -b t x c t ∼ N ( 0 , I ) . Based on this instantiation of ˆ x r , we define the consistency bridge training (CBT) loss as:

where t, r ( · ) , λ ( · ) , θ -1 are defined the same as in Eqn. (17), and z ∼ N ( 0 , I ) is a shared Gaussian noise used in both h θ and h θ -1 . We have the following proposition demonstrating the connection between L ∆ t max CBT and L ∆ t max CBD with the first-order one-step ODE solver:

Proposition 3.3. Given ∆ t max = max t { t -r ( t ) } and assume d, h θ , f, g are twice continuously differentiable with bounded second derivatives, the weighting function λ ( · ) is bounded, and E [ ∥∇ x t log q t | T ( x t | x T ) ∥ 2 2 ] < ∞ . Meanwhile, assume that L ∆ t max CBD employs the one-step ODE solver in Eqn. (16) with ground truth pre-trained score model, i.e., ∀ t ∈ [ ϵ, T -γ ] , y ∼ q data ( y ) : s ϕ ( x t , t, y ) ≡ ∇ x t log q t | T ( x t | x T = y ) . Then, we have: L ∆ t max CBD = L ∆ t max CBT + o (∆ t max ) .

The core part of our analysis also follows [58], except the connection between the CBD & CBT objective relies on the proposed first-order ODE solver and the estimated ˆ x r in Eqn. (19) with the general noise schedule for DDBM. We include the details in Appendix B.6.

3.4 Network Precondition and Sampling

Network Precondition First, we focus on enforcing the boundary condition h θ ( x ϵ , ϵ, y ) = x ϵ Usually, a variable substitution ˜ t = t -ϵ could work in most cases. For example, for the precondition of our consistency bridge model, which can be done by designing a proper network precondition. for I 2 SB in Table 1, we have x ϵ + σ ˜ ϵ F θ = x ϵ + √ ∫ ϵ -ϵ 0 g 2 ( τ )d τ = x ϵ . Also, the common 'EDM' [24] style precondition used in DDBM also satisfies c skip (˜ ϵ ) = 1 and c out (˜ ϵ ) = 0 . We also give a universal precondition to satisfy the boundary conditions based on the form of the ODE solver in Eqn. (16) in Appendix B.4 to cope with the case where the variable substitution is not applicable.

Sampling As explained in Section 3.1, the PF-ODE is only well-defined within the time horizon 0 ≤ t ≤ T -γ for some γ ∈ (0 , T ) . Hence, the sampling of CDBMs should start with x T -γ ∼ q T -γ | T ( x T -γ | x T = y ) , which can be obtained by simulating the reverse SDE in Eqn. (7) from T to T -γ . Here we opt to use one first-order stochastic step, which is equivalent to performing posterior sampling, i.e., x T -γ ∼ q T -γ | 0 T ( x T -γ | x 0 = h θ ( x T , T, y ) , x T = y ) . This sampling approach defaults to two NFEs (Number of Function Evaluations), which is aligned with the practical guideline that employing two-step sampling in CM allows for a better trade-off between quality and computation compared to other treatments such as scaling up models [15]. We could also alternate a forward noising step and a backward consistency step multiple times to further improve sample quality as consistency models do.

4 Experiments

4.1 Experimental Setup

Task, Datasets, and Metrics In this work, we conduct experiments for CDBM on image-to-image translation and image inpainting tasks with various image resolutions and scales of the data set. For image-to-image translation, we use the Edges → Handbags [23] with 64 × 64 pixel resolution and DIODE-Outdoor [62] with 256 × 256 pixel resolution. For image inpainting, we choose ImageNet [9] 256 × 256 with a center mask of size 128 × 128 . Regarding the evaluation metrics, we report the Fréchet inception distance (FID) [19] for all datasets. Furthermore, following previous works [33, 72], we measure Inception Scores (IS) [3], LPIPS [68] and Mean Square Error (MSE) for image-to-image translation and Classifier Accuracy (CA) of a pre-trained ResNet50 for image-inpainting. The metrics are computed using the complete training set for Edges → Handbags and DIODE-Outdoor, and a validation subset of 10,000 images for ImageNet.

Training Configurations Wetrain CDBM in two ways: distill pre-trained DDBM with CBD or finetuning DDBM with CBT. We keep the noise schedule and prediction target of the pre-trained DDBM

unchanged and modify the network precondition to satisfy the boundary condition. Specifically, we adopt the design space of DDBM-VP and I 2 SB in Table 1 on image-to-image translation and image inpainting, respectively. We specify complete training details in Appendix C.

Specification of Design Choices We illustrate the specific design choices for CDBM. In this work, we use t ∈ [ ϵ, 1 -γ ] and set ϵ = 0 . 0001 , γ = 0 . 001 and sample t uniformly during training. We employ two different sets of the timestep function r ( t ) and the loss weighting λ ( t ) , also named the training schedule for CDBM. The first, following [58], specifies a constant quantity for ∆ t = t -r ( t ) with a simple loss weighting of λ ( t ) = 1 . The constant gap ∆ t is treated as a hyperparameter and we search it among { 1 / 9 , 1 / 18 , 1 / 36 , 1 / 60 , 1 / 80 , 1 / 120 } . The other employs r ( t ) that gradually shrinks t -r ( t ) during the training process and a loss weighting of λ ( t ) = 1 t -r ( t ) , which enjoys a better trade-off between faster convergence and performance [58, 57, 15]. Following [15], we use a sigmoid-style function r ( t ) = t (1 -1 q ⌊ iters /s ⌋ )(1 + k 1+ e bt ) , where iters is the number of training iterations, q, s, k, b are hyperparameters. We use q = 2 , k = 8 , and tune b ∈ { 1 , 2 , 5 , 10 , 20 , 50 } and s ∈ { 5000 , 10000 } .

2

3

4

5

8

10

NFE

4

6

8

10

FID

DDBM (ODE-1)

CBD

CBT

0

5

10

15

20

25

30

Training Iterations (

×1000

)

9

10

11

12

13

14

FID

t

=1/9

t

=1/18

t

=1/36

t

=1/60

t

=1/80

t

=1/120

(a) CBD

0

5

10

15

20

25

Training Iterations (

×1000

)

10

11

12

13

14

FID

b=1

b=2

b=5

b=10

b=20

(b) CBT

4.2 Results for Few-step Generation

We present the quantitative results of CDBM on image-to-image translation and image inpainting tasks in Table 2 and Table 3. We adopt DDBM on the same noise schedule and network architecture, with the first-order ODE solver in Eqn. (16) as our main baseline (i.e., 'DDBM (ODE-1)'). We report the performance of the baseline DDBM under different Number of Function Evaluations (NFE) as a

Condition

DDBM, NFE=100

Condition

DDBM, NFE=100

Condition

DDBM, NFE=8

DDBM, NFE=10

DDBM, NFE=2

CDBM, NFE=2

DDBM, NFE=2

CDBM, NFE=2

DDBM, NFE=2

CDBM, NFE=2

CDBM, NFE=10

reference for the sampling acceleration ratio (Reduction factor of NFE to achieve the same FID) of CDBM. Following [72, 33], we report the result of other baselines with NFE ≥ 40 , which consists of diffusion-based methods, diffusion bridges with different formulations, or samplers. We mainly focus on the two-step generation scenario for CDBM, which is the minimal NFEs required for CDBM using the sampling procedure described in Section 3.4.

For image-to-image translation, as shown in Table. 2, we first observed that our proposed firstorder ODE solver has superior performance compared to the hybrid high-order sampler used in DDBM [72]. On top of that, CDBM's FID at NFE = 2 is close to or even better than DDBM's at NFE around 100 with the advanced ODE solver, achieving a sampling speed-up around 50 × . This can be corroborated by the qualitative demonstration in Fig. 4, where CDBMs drastically reduce the blurring effect on DDBMs under few-step generation settings while enjoying realistic and faithful translation performance.

For image inpainting, as shown in Table. 3, the baseline ODE solver for DDBM achieves decent sample quality at NFE = 10 . For CDBM, as shown in Fig. 2, the acceleration ratio is relatively modest in such a large-scale and challenging dataset, achieving close to a 4 × increase in sampling speed. Notably, CBT's FID at NFE = 4 matches DDBM at NFE = 10 . Moreover, we find that CDBMs have better visual quality than DDBM given the same computation budget, as shown in Fig. 4 and Appendix D, which illustrates that CDBM yields a better quality-efficiency trade-off.

Meanwhile, we observe that fine-tuning DDBMs with CBT generally produces better results than CBD in all three data sets, demonstrating fine-tuning a pre-trained score model to a consistency function is a more promising solution with less computational and memory cost compared to distillation, which is consistent with recent findings [15]. We also conducted an ablation study for CBD and CBT under different training schedules (i.e., the combination of the timestep function r ( t ) and the loss weighting λ ( t ) ) on ImageNet 256 × 256 . As shown in Fig. 3, for a small timestep interval t -r ( t ) , e.g., a small ∆ t in Fig. 3a or a large b in Fig. 3b (detail in Appendix C.2), the performance is generally better but also suffers from training instability, indicated by the sharp increase in FID during training when ∆ t = 1 / 120 and b = 50 . While for a large timestep interval, the performance at convergence is usually worse. In practice, we found that adopting the training schedule that gradually shrinks r ( t ) -t with b = 20 or 50 with CBT could work across all tasks, whereas CBD generally needs a meticulous design for ∆ t or b to ensure stable training and satisfactory performance.

4.3 Semantic Interpolation

We show that CDBMs support performing downstream tasks, such as semantic interpolation, similar to diffusion models [55]. Recall that the sampling process for CDBM alternates between consistency

function evaluation and forward sampling, we could track all noises and the corresponding timesteps to re-generate the same sample. By interpolating the noises of two sampling trajectories, we can obtain a series of samples lying between the semantics of two source samples, as shown in Fig. 5, which demonstrates that CDBMs have a wide range of generative modeling capabilities, such as sample diversity and semantic interpolation.

5 Conclusion

In this work, we introduce consistency diffusion bridge models (CDBMs) to address the sampling inefficiency of DDBMs and present two frameworks, consistency bridge distillation and consistency bridge training, to learn the consistency function of the DDBM's PF-ODE. Building on a unified view of design spaces and the corresponding general-form ODE solver, CDBM exhibits significant flexibility and adaptability, allowing for straightforward integration with previously established successful designs for diffusion bridges. Experimental evaluations across three datasets show that CDBM can effectively boost the sampling speed of DDBM by 4 × to 50 × . Furthermore, it achieves the saturated performance of DDBMs with less than five NFEs and possesses the broad capacity of generative models, such as sample diversity and semantic interpolation.

Limitations and Broader Impact While significantly improving the sampling efficiency in the datasets we used, it remains to be explored how the proposed CDBM, along with the DDBM formulation, performs in datasets with larger-scale or more complex characteristics. Furthermore, the consistency model paradigm typically suffers from numerical instability and it would be a promising research direction to keep improving CDBM's performance from an optimization perspective. With enhanced sampling efficiency, CDBMs could contribute to more energy-efficient deployment of generative models, aligning with broader goals of sustainable AI development. However, it could also lower the cost associated with the potential misuse for creating deceptive content. We hope that our work will be enforced with certain ethical guidelines to prevent any form of harm.

Acknowledgments and Disclosure of Funding

This work was supported by the National Science and Technology Major Project (2021ZD0110502), NSFC Projects (Nos. 62350080, 62106122, 92248303, 92370124, 62350080, 62276149, U2341228, 62076147), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J. Zhu was also supported by the XPlorer Prize.

References

Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, 1982.

Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233 , 2024.

Shane Barratt and Rishi Sharma. A note on the inception score. arXiv preprint arXiv:1801.01973 , 2018.

Mari Paz Calvo and César Palencia. A class of explicit multistep exponential integrators for semilinear problems. Numerische Mathematik , 102:367-381, 2006.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations , 2021.

Zehua Chen, Guande He, Kaiwen Zheng, Xu Tan, and Jun Zhu. Schrodinger bridges beat diffusion models on text-to-speech synthesis. arXiv preprint arXiv:2312.03491 , 2023.

Valentin De Bortoli, Guan-Horng Liu, Tianrong Chen, Evangelos A Theodorou, and Weilie Nie. Augmented bridge matching. arXiv preprint arXiv:2311.06978 , 2023.

Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems , 34:17695-17709, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255. IEEE, 2009.

Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems , volume 34, pages 87808794, 2021.

Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 , 2014.

Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations , 2016.

Joseph L Doob and JI Doob. Classical potential theory and its probabilistic counterpart , volume 262. Springer, 1984.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 , 2024.

Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. arXiv preprint arXiv:2406.14548 , 2024.

Martin Gonzalez, Nelson Fernandez, Thuy Tran, Elies Gherbi, Hatem Hajri, and Nader Masmoudi. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models. arXiv preprint arXiv:2305.14267 , 2023.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems , volume 27, pages 2672-2680, 2014.

Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662 , 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems , volume 30, pages 6626-6637, 2017.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , volume 33, pages 6840-6851, 2020.

Marlis Hochbruck, Alexander Ostermann, and Julia Schweitzer. Exponential rosenbrock-type methods. SIAM Journal on Numerical Analysis , 47(1):786-803, 2009.

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1125-1134, 2017.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems , 2022.

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Advances in Neural Information Processing Systems , 2022.

Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ODE trajectory of diffusion. In The Twelfth International Conference on Learning Representations , 2024.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems , 2021.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations , 2014.

Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems , 31, 2018.

Liangchen Li and Jiajun He. Bidirectional consistency models. arXiv preprint arXiv:2403.18035 , 2024.

Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations , 2022.

Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos Theodorou, Weili Nie, and Anima Anandkumar. I 2 sb: Image-to-image schrödinger bridge. In International Conference on Machine Learning , pages 22042-22062. PMLR, 2023.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In International Conference on Learning Representations , 2019.

Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations , 2022.

Xingchao Liu, Lemeng Wu, Mao Ye, and qiang liu. Learning diffusion bridges on constrained domains. In The Eleventh International Conference on Learning Representations , 2023.

Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning , pages 14429-14460. PMLR, 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems , 2022.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpmsolver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378 , 2023.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2022.

Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning , pages 16784-16804. PMLR, 2022.

Stefano Peluchetti. Diffusion bridge mixture transports, schrödinger bridge problems and generative modeling. Journal of Machine Learning Research , 24(374):1-51, 2023.

Stefano Peluchetti. Non-denoising forward-time diffusions. arXiv preprint arXiv:2312.14589 , 2023.

Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei. Diffusion-based voice conversion with fast maximum likelihood sampling scheme. In International Conference on Learning Representations , 2022.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

L Chris G Rogers and David Williams. Diffusions, Markov processes and martingales: Volume 2, Itô calculus , volume 2. Cambridge university press, 2000.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACMSIGGRAPH 2022 Conference Proceedings , pages 1-10, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo-Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems , 2022.

Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schrödinger bridge matching. Advances in Neural Information Processing Systems , 36, 2024.

Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Conditional simulation using diffusion schrödinger bridges. In Uncertainty in Artificial Intelligence , pages 1792-1802. PMLR, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Vignesh Ram Somnath, Matteo Pariset, Ya-Ping Hsieh, Maria Rodriguez Martinez, Andreas Krause, and Charlotte Bunne. Aligned diffusion schrödinger bridges. In Uncertainty in Artificial Intelligence , pages 1985-1995. PMLR, 2023.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations , 2023.

Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations , 2024.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning , pages 32211-32252. PMLR, 2023.

Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. In Advances in Neural Information Processing Systems , volume 34, pages 1415-1428, 2021.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. arXiv preprint arXiv:2203.08382 , 2022.

Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z Dai, Andrea F Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R Walter, et al. Diode: A dense indoor and outdoor depth dataset. arXiv preprint arXiv:1908.00463 , 2019.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolcm: Video latent consistency model. arXiv preprint arXiv:2312.09109 , 2023.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations , 2023.

Yuji Wang, Zehua Chen, Xiaoyu Chen, Jun Zhu, and Jianfei Chen. Framebridge: Improving image-to-video generation with bridge models. arXiv preprint arXiv:2410.15371 , 2024.

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In The Eleventh International Conference on Learning Representations , 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Kaiwen Zheng, Guande He, Jianfei Chen, Fan Bao, and Jun Zhu. Diffusion bridge implicit models. arXiv preprint arXiv:2405.15885 , 2024.

Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning , pages 42363-42389. PMLR, 2023.

Linqi Zhou, Aaron Lou, Samar Khanna, and Stefano Ermon. Denoising diffusion bridge models. arXiv preprint arXiv:2309.16948 , 2023.

A Related Works

Diffusion Bridges Diffusion bridges [44, 36, 33, 54, 51, 72, 7, 6] are an emerging class of generative models with attractive flexibility in modeling the stochastic process between two arbitrary distributions. The flow matching [32], and its stochastic counterpart, bridge matching [44] assume the access of a joint distribution and an interpolation, or a forward process, between the samples, then, another SDE/ODE is learned to estimate the dynamics of the pre-defined interpolation, which can be used for generative modeling from non-Gaussian priors [33, 6, 72, 69, 66]. In particular, the forward process can be constructed via Doob's h -transform [44, 36, 72]. Among them, DDBM [72] focuses on learning the reverse-time diffusion bridge conditioned on a particular terminal endpoint with denoising score matching, which has been shown to be equivalent to conducting a conditioned bridge matching that preserves the initial joint distribution [7]. Other works tackle solving the diffuion Schrödinger Bridge problem, such as using iterative algorithms [8, 51, 43]. In this work, we use a unified view of design spaces on existing diffusion bridges, in particular, bridge matching methods, to decouple empirical choices from their different theoretical premises and properties and focus on developing the techniques of learning the consistency function of DDBM's PF-ODE with various established design choices for diffusion bridges.

Consistency Models Recent studies have continued to explore the effectiveness of consistency models [58]. For example, CTM [26] proposes to augment the prediction target from the starting point to the intermediate points along the PF-ODE trajectory from the input to this starting point. BCM [31] additionally expands the model to allow direct mapping at the PF-ODE trajectory points in both forward and reverse time. Beyond different formulations, several works aim to improve the performance of consistency training with theoretical and practical insights. iCT [57] systematically examines the design choices of consistency training and presents improved training schedule, loss weighting, distance metrics, etc. ECT [15] further leverages the insights to propose novel practical designs and show fine-tuning pre-trained diffusion models for learning consistency models yields decent performance with much lower computation compared to distillation. Unlike these works, we focus on constructing consistency models on top of the formulation of DDBMs with specialized design spaces and a sophisticated ODE solver for them.

B Additional Details for CDBM Formulation, CBD, and CBT

B.1 Derivation of First-Order Bridge ODE Solver

We first review the first-order ODE solver in Section 3.1:

Proposition 3.1. Given an initial value x t at time t > 0 , the first-order solver of the bridge ODE in Eqn. (8) from t to r ∈ [0 , t ] with the noise schedule defined in Eqn. (11) is:

Recall the PF-ODE of DDBM in Eqn. (8) with a linear drift f ( t ) x t :

Also recall the noise schedule in Eqn. (11) and the analytic form of p t | 0 and p T | t in diffusion models:

We thus have the corresponding score functions and the score-data transformation for s θ that predicts ∇ x t log q t | 0 T :

We use the data parameterization x θ ( x t , t, y ) in following discussions. For PF-ODE in Eqn. (21), substituting ∇ x t log q t | T ( x t | x T = y ) with Eqn. (25) and substituting p T | t ( x T | x t ) in with Eqn. (23), we have the following after some simplification:

which shares the same form as the ODE in Bridge-TTS [6]. In the next discussions, we present an overview of deriving the first-order ODE solver and refer the reader to Appendix A.2 in [6] for details.

We begin by reviewing exponential integrators [4, 22], a key technique for developing advanced diffusion ODE solvers [16, 38, 39, 70]. Consider the following ODE:

where F θ is a n -th differentiable parameterized function. By leveraging the 'variation-of-constant' formula, we could obtain a specific form of the solution of the ODE in Eqn. (27) (assume r < t ):

The integral in Eqn. (28) only involves the function F θ , which helps reduce discretization errors.

With such a key methodology, we could derive the first-order solver for Eqn. (26). First, collecting the coefficients for x t , y , x θ , we have:

By setting:

with correspondence to Eqn. (28), the exponential terms could be analytically given by:

The exact solution for Eqn. (29) is thus given by:

The integrals in Eqn. (31) (without considering x θ ) can be calculated as:

Then, with the first order approximation x θ ( x τ , τ ) ≈ x θ ( x s , s ) , we could obtain the first order solver in Eqn. (16).

B.2 An Illustration Example of the Validity of the Bridge ODE

Recall the provided example in Section 3.1:

Example 3.1. Assume T = 1 and consider a simple Brownian Bridge between two fixed points ( x 0 , x 1 ) :

with marginal distribution q t | 01 ( x t | x 0 , x 1 ) = N ((1 -t ) x 0 + tx 1 , t (1 -t )) . The ground-truth reverse SDE and PF-ODE are given by:

Then first simulating the reverse SDE in Eqn. (14) from t = 1 to t = 1 -γ for some γ ∈ (0 , 1) and then starting to simulate the PF-ODE in Eqn. (15) will preserve the marginal distribution.

Proof. We first demonstrate the effect of the initial SDE step, according to Table 1 and the expression of the relevant score terms in Eqn. (23) and Eqn. (25), the ground-truth reverse SDE can be derived as:

Then, the analytic solution of the reverse SDE in Eqn. (7) from time t to time s < t can be derived as:

Let t = 1 , we have:

i.e., x s has the same marginal as the forward process at time s . Similarly, the ground-truth PF-ODE can be derived as:

whose analytic solution from time t to time s < t can be derived as:

When x t ∼ N ((1 -t ) x 0 + tx 1 , t (1 -t )) , we have:

Hence, once the singularity is skipped by a stochastic step, following the PF-ODE reversely will preserve the marginals in this case.

B.3 Derivation of the CBT Objective

Given ( x , y ) ∼ q data ( x , y ) , x t ∼ q t | 0 T ( x t | x 0 = x , x T = y ) and an estimate of ˆ x r = ˆ x ϕ ( x t , t, y ) based on the pre-trained score predictor s ϕ with the first-order ODE solver in Eqn. (16), our goal is to derive the alternative estimation of ˆ x r = ˆ x ( x t , t, r, x , y ) = a r y + b r x + c r z used in CBT, where z = x t -a t y -b t y c t ∼ N ( 0 , I ) and a r , b r , c r are defined in Eqn. (11). We begin with the estimator with pre-trained score model and first-order ODE solver:

where x ϕ is the equivalent data predictor of the score predictor s ϕ . By the transformation between data and score predictor x ϕ = x t -a t x T + c 2 t s ϕ b t and substituting the score predictor s ϕ with the score estimator ∇ x t q t | 0 T ( x t | x 0 = x , x T = y ) , we have:

By expressing x t = a t y + b t x + c t z , we could derive the corresponding coefficients for x , y , z on the right-hand side.

For y :

where ( i ) is due to the fact ¯ α t = α t α T .

For x :

For z :

Hence, we have the alternative model-free estimator ˆ x r = ˆ x ( x t , t, r, x , y ) = a r y + b r x + c r z , where z ∼ N ( 0 , I ) is the same Gaussian noise used in sampling x t = a t y + b t x + c t z . Substituting ˆ x ϕ ( x t , t, r, y ) in the CBD objective in Eqn. (17) with ˆ x ( x t , t, r, x , y ) gives the CBT objective in Eqn. (20).

B.4 Network Parameterization

First, we show the detailed network parameterization for DDBM in Table. 1. Denote the neural network as F θ , the data predictor x θ ( x , t, y ) is given by:

where

and

It can be verified that, with the variable substitution ˜ t = t -ϵ , we have a ˜ ϵ = 0 , b ˜ ϵ = 1 , c ˜ ϵ = 0 and thus have c skip (˜ ϵ ) = 1 and c out (˜ ϵ ) = 0 .

Meanwhile, we could generally parameterize the data predictor x θ with the one-step first-order solver from t to ϵ , i.e.:

which naturally satisfies f ( x ϵ , ϵ, y ) = x ϵ .

B.5 Asymptotic Analysis of CBD

Proposition 3.2. Given ∆ t max = max t { t -r ( t ) } and let h ϕ ( · , · , · ) be the consistency function of the empirical bridge ODE taking the form in Eqn. (8). Assume h θ is a Lipschitz function, i.e., there exists L > 0 , such that for all t ∈ [ ϵ, T -γ ] , x 1 , x 2 , y , we have ∥ h θ ( x 1 , t, y ) -h θ ( x 2 , t, y ) ∥ 2 ≤ L ∥ x 1 -x 2 ∥ 2 . Meanwhile, assume that for all t, r ∈ [ ϵ, T -γ ] , y ∼ q data ( y ) := E x [ q data ( x , y )] , the ODE solver ˆ x ϕ ( · , t, r, y ) has local error uniformly bounded by O (( t -r ) p +1 ) with p ≥ 1 . Then, if L ∆ t max CBD = 0 , we have: sup t, x , y ∥ h θ ( x , t, y ) -h ϕ ( x , t, y ) ∥ 2 = O ((∆ t max ) p ) .

Most of the proof directly follows the original consistency models analysis [58], with minor differences in the discrete timestep intervals (i.e., non-overlapped in [58] and overlapped in ours) and the form of marginal distribution between p t ( x t ) for the diffusion ODE and q t | T ( x t | x T = y ) for the bridge ODE.

Proof. Given L ∆ t max CBD = 0 , we have:

Since λ ( t ) > 0 , and for t ∈ [ ϵ, T -γ ] , q t | 0 T ( x t | x 0 = x , y 0 = y ) takes the form of N ( a t x T + b t x 0 , c t I ) with c t > 0 , which entails for any x t , t ∈ [ ϵ, T -γ ] , q t | T ( x t | x T = y ) = E x [ q t | 0 T ( x t | x 0 = x , x T = y )] > 0 . Hence, Eqn. (41) implies that for all t ∈ [ ϵ, T -γ ] , ( x , y ) ∼ q data ( x , y ) , x t ∼ q t | 0 T ( x t | x 0 = x , x T = y ) , we have:

By the nature of the distance metric function d and the stopgrad operator, we then have:

Define the error term at timestep t ∈ [ ϵ, T -γ ] as:

We have:

Since h θ is Lipschitz with constant L and the ODE solver ˆ x ϕ ( · , t, r, y ) is bounded by O (( t -r ) p +1 ) with p ≥ 1 , we have:

From the boundary condition of the consistency function, we have:

Denote r m ( t ) as applying r on t for m times, since ∆ t min = min t { t -r ( t ) } exists, there exists N such that r n ( t ) = ϵ for n ≥ N . We thus have:

B.6 Connection between CBD & CBT

Proposition 3.3. Given ∆ t max = max t { t -r ( t ) } and assume d, h θ , f, g are twice continuously differentiable with bounded second derivatives, the weighting function λ ( · ) is bounded, and E [ ∥∇ x t log q t | T ( x t | x T ) ∥ 2 2 ] < ∞ . Meanwhile, assume that L ∆ t max CBD employs the one-step ODE solver in Eqn. (16) with ground truth pre-trained score model, i.e., ∀ t ∈ [ ϵ, T -γ ] , y ∼ q data ( y ) : s ϕ ( x t , t, y ) ≡ ∇ x t log q t | T ( x t | x T = y ) . Then, we have: L ∆ t max CBD = L ∆ t max CBT + o (∆ t max ) .

The core technique for building the connection between consistency distillation and consistency training with Taylor Expansion also directly follows [58]. The major difference lies in the form of the bridge ODE and the general noise schedule & the first-order ODE solver studied in our work.

Proof. First, for a twice continuously differentiable, multivariate, vector-valued function h ( x , t, y ) , denote ∂ k h ( x , t, y ) as the Jacobian of h over the k -th variable. Consider the CBD objective with first-order ODE solver in Eqn. (16) (ignore terms taking expectation for notation simplicity):

where k 1 ( t, r ) = α r ρ r ¯ ρ r α t ρ t ¯ ρ t , k 2 ( t, r ) = α r ρ 2 T ( ¯ ρ 2 r -¯ ρ t ρ r ¯ ρ r ρ t ) , k 3 ( t, r ) = α r α T ρ 2 T ( ρ 2 r -ρ t ρ r ¯ ρ r ¯ ρ t ) are coefficients of x t , x ϕ , y in the first-order ODE solver in Eqn. (16), x ϕ is pre-trained data predictor. By applying first-order Taylor expansion on Eqn. (45), we have:

Here the error term w.r.t. the first variable can be obtained by applying Taylor expansion on k ( t, r ) = k ( t, t ) + ∂ 2 k ( t, t )( r -t ) + o ( | t -r | ) with k 1 ( t, t ) -1 = 0 , k 2 ( t, t ) = k 3 ( t, t ) = 0 . By applying

Taylor expansion on d , we have:

Then we focus on the term related to the first-order ODE solver:

By the transformation between data and score predictor x ϕ = x t -a t x T + c 2 t s ϕ b t , and substitute s ϕ ( x t , t, y ) with ∇ x t log q t | T ( x t | x T = y ) , we have:

Next, substituting the score ∇ x t log q t | T ( x t | x T = y ) with the unbiased estimator:

We then have:

where ( i ) comes from the law of total expectation. Then we apply Taylor expansion in the reverse direction:

C Additional Experimental Details

C.1 Details of Training and Sampling Configurations

We train CDBMs based on a series of pre-trained DDBMs. For two image-to-image translation tasks, we directly use the pre-trained checkpoints provided by DDBM's [72] official repository. 2 For image inpainting, we re-train a model with the same I 2 SB style noise schedule, network parameterization, and timestep scheme in Table. 1, as well as the overall network architecture. Unlike the training setup in I 2 SB, our network is conditioned on x T = y following DDBM and takes the class information of ImageNet as input, which we refer to as the base DDBM model for image inpainting on ImageNet. The model is initialized with the class-conditional version on ImageNet 256 × 256 of guided diffusion [10]. We used a global batch size of 256 and a constant learning rate of 1e-5 with mixed precision (fp16) to train the model for 200k steps. We train the model with 8 NVIDIA A800 80G GPUs for 9.5 days, achieving the FID reported in Table. 3 with the first-order ODE solver in Eqn. (16).

For training CDBMs, we use a global batch size of 128 and a learning rate of 1e-5 with mixed precision (fp16) for all datasets using 8 NVIDIA A800 80G GPUs. For the constant training schedule r ( t ) = t -∆ t , we train the model for 50k steps, while for the sigmoid-style training schedule, we train the model for 6 s steps, e.g., 30k or 60k steps, due to numerical instability when t -r ( t ) is small. For CBD, training a model for 50k steps on a dataset with 256 × 256 resolution takes ∼ 2.5 days, while CBT takes ∼ 1.5 days. In this work, we normalize all images within [ -1 , 1] and adopt the RAdam [27, 34] optimizer.

For sampling, we use a uniform timestep for all baselines with the ODE solver and CDBM on two image-to-image translation tasks with ϵ = 0 . 0001 , T = 1 . 0 . For CDBM on image inpainting on ImageNet, we manually assign the second timestep to T -0 . 1 and make other timesteps uniformly distributed between [ ϵ, T -0 . 1) , which we find yields better empirical performance on this task.

C.2 Details of Training Schedule for CDBM

We illustrate the effect of the hyperparamter b in the sigmoid-like training schedule r ( t ) = t (1 -1 q ⌊ iters /s ⌋ )(1 + k 1+ e bt ) . Note that we further manually enforce r ( t ) to satisfy ∆ t max and ∆ t min .

0.00

0.25

0.50

0.75

1.00

1.5

1.0

0.5

0.0

0.5

r

/

t

b

=5,

iter

/

s

=0

0.00

0.25

0.50

0.75

1.00

0.25

0.00

0.25

0.50

0.75

b

=5,

iter

/

s

=1

0.00

0.25

0.50

0.75

1.00

0.4

0.6

0.8

b

=5,

iter

/

s

=2

0.00

0.25

0.50

0.75

1.00

0.7

0.8

0.9

b

=5,

iter

/

s

=3

0.00

0.25

0.50

0.75

1.00

0.85

0.90

0.95

b

=5,

iter

/

s

=4

0.00

0.25

0.50

0.75

1.00

0.92

0.94

0.96

0.98

b

=5,

iter

/

s

=5

0.00

0.25

0.50

0.75

1.00

t

1.5

1.0

0.5

0.0

0.5

r

/

t

b

=50,

iter

/

s

=0

0.00

0.25

0.50

0.75

1.00

t

0.25

0.00

0.25

0.50

0.75

b

=50,

iter

/

s

=1

0.00

0.25

0.50

0.75

1.00

t

0.4

0.6

0.8

b

=50,

iter

/

s

=2

0.00

0.25

0.50

0.75

1.00

t

0.7

0.8

0.9

b

=50,

iter

/

s

=3

0.00

0.25

0.50

0.75

1.00

t

0.85

0.90

0.95

b

=50,

iter

/

s

=4

0.00

0.25

0.50

0.75

1.00

t

0.92

0.94

0.96

0.98

b

=50,

iter

/

s

=5

C.3 License

We list the used datasets, codes, and their licenses in Table 4.

D Additional Samples

Condition

Ground Truth

DDBM, NFE=2

DDBM, NFE=100

CDBM, NFE=2

Condition

Ground Truth

DDBM, NFE=2

DDBM, NFE=100

CDBM, NFE=2

Condition

Ground Truth

DDBM, NFE=2

DDBM, NFE=100

CDBM, NFE=2

AR

ERSUM 24

AR

AR

92

IAR

AR

Condition

Different samples starts from q T - γ | T ( x T - γ | x T = y )

Condition

CDBM, NFE=2

I

2

SB, NFE=4

CDBM, NFE=4

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The discussion is located in the 'Limitations and Broad Impact' section after the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are provided with the propositions and the detailed derivation and proof is in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental configurations are included in Section 4 and Appendix C. The information we provided is sufficient to reproduce the results that support our claims.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: The release of the code needs an official procedure related to the authors' affiliation, which is not approved yet.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experiment configuration and details are included in Section 4 and Appendix C, which is sufficient to understand the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The metrics for evaluating generative models are typically stable and do not require error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provided the information in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The discussion is located in the 'Limitations and Broad Impact' section after the main paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work is conducted with common academic image datasets with model capability restricted with specific tasks. There is little chance posing risks for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Licenses for existing assets are listed in Appendix C.3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on efficient sampling of the denoising diffusion bridge models (DDBMs) or similar. In particular, the paper proposes to extend the consistency models (CMs) to DDBMs.\n\nCMs are generative models developed in the context of improving the sampling cost of diffusion-based generative models (DBGMs). Note that sampling of the probability flows of the DBGMs runs the integrators up to a few hundred steps or more. Similar to common DBGMs, assume we have a forward diffusion whose initial distribution—data—is denoted by $x_0$. In particular, when we rewrite the forward diffusion in the probability flow (PF) form, we get an ODE whose marginal is equivalent to the original forward diffusion. We further assume that we have the ODE-integrators that run $t$-decreasing direction starting from any given $t \\in (0, T]$ to $0$. Then, the CMs train to predict the output of the integrators, similar to denoising autoencoder, but the perturbation here is deterministic. Training such models uses the consistency condition. Once we train the CMs, we can use them instead of running a few hundred steps of DGBMs.\n\nDDBMs are input-conditional bridge models. More specifically, assume we have two distributions and their joint distribution, e.g., one is an image distribution, and the other is its edge images. The bridge models learn a diffusion to connect one distribution to the other. Unlike common bridge models, however, DDBMs conditions on its initial point, e.g., an edge image, so that the path generated by the learned diffusion preserves the information of the initial value. For several applications, such as image-to-image translations, these DDBMs show several benefits, unlike common Markovian models; in particular, the initial coupling of the joint distributions is preferred. Nevertheless, DDBMs are also required to run the SDE-integrators (or ODE-integrators for its PFs) a few hundred steps, similar to any other diffusion-based models.\n\nIn order to overcome sampling inefficiency in DDBMs, the paper proposes to extend CMs to DDBMs.\n\nFinally, the authors demonstrate the proposed method's effectiveness via various experiments, including image-to-image translations and image inpainting.\n\n---------------------------------------\nUpdated the rating from 4 to 7 after the authors' rebuttal\n\nSTRENGTHS:\nOne of the paper's key contributions is the introduction of a novel solution to improve the sampling of the DDBM-like models. The proposed method addresses one of the significant challenges in diffusion-based models and offers a fresh perspective on how state-of-the-art techniques can be extended to the DDBM-like models. Furthermore, the proposed method exhibits comparable performance and scalability compared to popular diffusion-based generative models with a fewer number of evaluations, further underscoring the practical effectiveness of the proposed method.\n\nWEAKNESSES:\nThe authors tackle a highly interesting problem and reach out for an excellent solution. However, I find that the paper requires a major revision due to a few theoretical loopholes in extending the consistency models (CMs) to the denoising diffusion bridge models (DDBMs).\n\nFor example, in expressing CM in the form of a stochastic differential equation (SDE), the paper uses a definition that is not mathematically well-defined. Particularly, in Equation 2, the notation of a reverse-time standard Wiener process is mentioned. However, a reverse-time standard Wiener process is not defined in general; more precisely, it is tricky to define such a concept. The reason is that the definition of the It\\'o integral is highly sensitive to the direction of time. Consequently, the change of variables used in calculus cannot be applied to SDE directly. As a result, we need to define an SDE-specific chain rule called Ito's lemma. Accordingly, one cannot rewrite an SDE by changing the sign of time. In order to apply the chain rule of calculus, the SDE literature uses the Stieltjes integral instead of the It\\'o integral, and a corresponding SDE notation needs to be defined. If this approach were taken for the submitted paper, however, the denoising diffusion and consistency model would need to be defined accordingly.\n\nIn conclusion, this paper's development deviates from the fundamental assumptions of the It\\'o integral-based SDEs. Note that the original CM circumvents this issue by using only ordinary differential equations (or partial differential equations). Therefore, unlike the It\\'o integral, it is free from concerns about the time directions. However, this does not apply to SDEs.\n\nIn my understanding, one would be able to reach the same conclusion as the submitted paper while sticking to the fundamentals of SDEs (related to It'o\\' diffusion). However, such modification would require a major revision from the current submission. \n\nIn addition, a few statements need to be addressed. In particular, to get an unbiased estimation of the expectation in Equation 15, one needs to sample $x_t$ and $x_T$ first and then sample $x_0$. However, the authors state that we can achieve this by sampling $x_0$ and $x_T$ first and then $x_t$. Such an estimator is biased except when we use a single sample.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors combine Denoising Diffusion Bridge Models (DDBMs), that build a transport map between two arbitrary target distributions (that can be coupled) through a stochastic process, with recent advances on consistency techniques [1], that were originally designed for denoising diffusion models. The motivation herein is to speed up the inference process of DDBMs, that turns out to be costly due to a high number of neural network evaluations. Following the approach from [1], the authors propose to learn the *consistency function* of the probability flow associated to an arbitrary DDBM, which outputs a prediction of the target distribution given any input of the DDBM process $x_t$, for any time $t$. To do so, they derive a consistency loss, which can be seen as the natural extension of the loss of [1] to the bridge setting. Then, they propose two paradigms for this approach: either *consistency bridge distillation*, which assumes to have access to a pretrained DDBM model, and *consistency bridge training*, where no model is available. Their formulation encompasses popular designs of diffusion bridges such as the Brownian brigde, the Image-to-Image Schrödinger bridge [2], the original bridges designed for DDBMs [3]... The authors provide practical guidelines on the neural network parameterization and preconditioning, the design of the ODE solver and the consistency schedule. Finally, they conduct numerical experiments on high-dimensional datasets that demonstrate that the proposed consistency bridge model is faster than the original model while having equal or better generative performance. They notably find out that fine-tuning a pretrained DDBM with consistency training provides better performance under lower computational budget than distillation.\n\n[1] Consistency models. Song et al. 2023.\n\n[2] I2SB: Image-to-Image Schrödinger Bridge. Liu et al. 2023.\n\n[3] Denoising diffusion bridge models. Zhou et al. 2023\n\nSTRENGTHS:\n- Although this paper adapts lots of elements from [1] to the bridge setting, which may amortize the novelty of this contribution, I think that this contribution may be impactful for practical usage as it encompasses a large formulation of DDBMs and demonstrates great empirical performance, see Section 4.\n- The paper is well written and easy to follow. In particular, the mathematical statements are clear to understand.\n- The presentation of the background is well done and helps the readability of the paper.\n- The comparison with related work in the numerics section is well conducted. \n- All details on the experiments are available in the paper, which is a really good point for reproducibility.\n\n[1] Consistency models. Song et al. 2023.\n\nWEAKNESSES:\n- The current paper does not exhibit any particular novelty compared to the setting from [1], but this is not a major weakness to me.\n- I think that the paper does not provide enough intuition on the design of a good consistency schedule.\n\n[1] Consistency models. Song et al. 2023.\n\nQUESTIONS:\n- Could you give further details and intuition on the design of the consistency schedules $r(t)$, which seem to be crucial so that consistency works (even if these reasons are heuristic) ? In particular, could you give examples of schedules that may be adapted in theory but do not work in practice ? It still unclear for me which design should be the best depending on the type of consistency (either distillation or training).\n- I have a question on the approximation made on the score in Equation (15) for consistency training, which can actually also be addressed to standard consistency models. Theoretically, up to the terminal conditioning in the bridge setting, the score evaluated in $x$ at time $t$ has to be understood as a conditional expectation over the posterior distribution of $X_0$ given $X_t=x$ by Tweedie's formula. This distribution has a very specific behaviour: (i) when $t\\approx0$ (close to the target distribution), it is concentrated like a Dirac mass; (ii), on the other hand when $t\\approx T$ (full noise), the posterior is approximately equal to the target distribution itself. This is well highlighted in the concept of duality of log-concavity in [1]. Therefore, the approximation of the score as presented in the current paper (that is evaluating the integrand in only one point $x_0$) is relevant when $t\\approx0$, but completely false when $t\\approx T$. Do you have an estimation of the error made in the latter case for tractable cases ? Does it have an impact on the training for this part of the process ? Have you come up with an idea to correct this intrinsic bias ?\n\n[1] Stochastic localization via iterative posterior sampling. Grenioux et al. 2024\n\nLIMITATIONS:\nThe limitations are clearly indicated in Section 5. The authors notably acknowledge that bridge consistency models suffer from numerical instability (same as classic consistency models).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by faster sampling time, the authors investigate consistency model techniques applied to denoising diffusion bridge models, a particular formulation of conditioned diffusion model whereby the forward process is a mixture of diffusion bridge SDEs, with a conditioned terminal point.\n\nIn particular, the authors adapt typical consistency model training regimed known as \"consistency model training\" - training the model from scratch -  and \"consistency model distillation\" - distilling from a pretrained diffusion (bridge) model.\n\nSTRENGTHS:\nAs far as I am aware consistency methods have not been applied to diffusion bridge models, though the significance of this is unclear vs performing consistency training on diffusion models or flow matching models.\n\nThe empirical performance showcased by the author's implementation appears competetitive to the baselines the authors consider.\n\nWEAKNESSES:\nDenoising diffusion bridge models are a particular formulation of conditioned diffusion model whereby the forward process is given by the SDE of a diffusion bridge, with a conditioned terminal point. This has the benefit of data to data translation, but in my opinion the authors exaggerate the significance of this and I would not consider it \"a new family of generative models\" [line 4]. Given the consistency methods applied to diffusion models transfer trivially to diffusion bridge models, I am of the opinion that the novelty of the methodological contribution is quite limited.\n\nFurthermore I would also question the usefulness of data to data generative models which are deterministic. For any ill-posed inverse problem, practitioners would want to be able to sample from many possible uncorrupted sample given a single corrupted sample, including some example provided here such as for inpainting.\n\nFor consistency models or indeed any flow type model, a Gaussian is typically chosen as marginal which adds stochasticity and permits the PF ODE to match the diffusion model in marginal distributions. This is not the case for diffusion bridge models with conditioned terminal point, there is no longer any stochasticity and hence the ODE derived cannot match the marginals of the stochastic generative backward process. Hence I wonder if PF-ODE of a diffusion bridge model even makes sense in the same way as a diffusion model? Given all the consistency approaches rely on the deterministic component, I believe this is quite fundamental and needs to be addressed.\n\nThere are a number of related works that could be discussed. In particular Augmented bridge matching (https://arxiv.org/abs/2311.06978) which draws an equivalence between denoising diffusion bridge models and conditioned bridge matching.\n\n\nOther:\n>  tractable class of Schrödinger Bridge and simulation-free, non-iterative training procedure [32, 6]\n\n[32,6] consider regular bridge matching without any connection to an optimal coupling and hence although tractable just a bridge matching model and not a schrodinger bridge. This is a common mistake.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to combine the consistency model (CM) with diffusion denoising bridge models (DDBMs) to build a consistency diffusion bridge model (CDBMs), which includes two paradigms of consistency bridge distillation and consistency bridge training. The experiments are conducted for image inpainting and image-to-image translation tasks.\n\nSTRENGTHS:\n+ This paper is well written and organized, with a clear structure for readers to follow up easily.\n\n+ The unified view of design space of several different DDBM models is an interesting summary for a clear comparison over existing works.\n\n+ The experiment's results especially with NFE=2 are promising to be comparable with other models with larger NFE steps as shown in Figure 4.\n\nWEAKNESSES:\n- The proposed method is reasonable but may not be very surprising. This work looks more like a direct combination of CM and DDBM, which models have been introduced a lot in previous works, including consistency distillation and consistency training. \n\n- Question about the motivation to combine CM and DDBM. CM is designed to reduce NFE steps requirement for diffusion sampling. DDBM models can also be able to achieve this goal by learning transformation between two data distributions, such as what is claimed in I2SB, while sacrificing the advantage of unsupervised learning. What is the key motivation and necessity to combine these two together?\n\n- In experiments results, some common metrics like PSNR and SSIM are not reported. In Table 3, the compared methods are most unsupervised learning approaches which are not trained with paired data. In this case, more NFE steps are expected. Therefore, for a fair comparison, the NFE for the compared baselines in Table 2 and 3 may also need to be noted. \n\n- For qualitative results, the comparison with previous DDBM method under the same NFE would be appreciated, since some DDBM method such as I2SB and CDDB [1] also claimed enabling image generation or restoration within a very few NFE steps from 1-4. It would be important comparison to compare with these baselines under the same experiment settings including NFE.\n\n- Due to the requirement for paired data training, the proposed model needs to be retrained to different tasks, which are limited compared with diffusion-based methods. What is the computational complexity and parameter generalization robustness to retrain the model on a new task? \n\n\n[1] Chung, Hyungjin, Jeongsol Kim, and Jong Chul Ye. \"Direct diffusion bridge using data consistency for inverse problems.\" Advances in Neural Information Processing Systems, 2023.\n\nQUESTIONS:\nPlease see the weaknesses for specific questions.\n\nLIMITATIONS:\nThe author discusses the limitations of the proposed method in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-20"}