Direct Diffusion Bridge using Data Consistency for Inverse Problems
Authors: Hyungjin Chung
Jeongsol Kim
Jong Chul Ye
Venue: NeurIPS 2023 poster (2023)
Keywords: Diffusion models
Inverse problems
Diffusion bridge

Abstract: Diffusion model-based inverse problem solvers have shown impressive performance, but are limited in speed, mostly as they require reverse diffusion sampling starting from noise. Several recent works have tried to alleviate this problem by building a diffusion process, directly bridging the clean and the corrupted for specific inverse problems. In this paper, we first unify these existing works under the name Direct Diffusion Bridges (DDB), showing that while motivated by different theories, the resulting algorithms only differ in the choice of parameters. Then, we highlight a critical limitation of the current DDB framework, namely that it does not ensure data consistency. To address this problem, we propose a modified inference procedure that imposes data consistency without the need for fine-tuning. We term the resulting method data Consistent DDB (CDDB), which outperforms its inconsistent counterpart in terms of both perception and distortion metrics, thereby effectively pushing the Pareto-frontier toward the optimum. Our proposed method achieves state-of-the-art results on both evaluation criteria, showcasing its superiority over existing methods. Code is open-sourced [here](https://github.com/HJ-harry/CDDB).

Direct Diffusion Bridge using Data Consistency for Inverse Problems
Hyungjin Chung"
Jeongsol Kim
Jong Chul Ye²
1 *Dept. of Bio and Brain Engineering
Graduate School of AI
Korea Advanced Institute of Science and Technology (KAIST) chung, jeongsol, jong. ye @kaist.
Abstract
Diffusion model-based inverse problem solvers have shown impressive perfor- mance, but are limited in speed, mostly they require reverse diffusion sampling starting from noise. Several recent works have tried to alleviate this problem by building a diffusion process, directly bridging the clean and the corrupted for specific inverse problems. In this paper, we first unify these existing works under the name Direct Diffusion Bridges (DDB), showing that while motivated by dif- ferent theories, the resulting algorithms only differ in the choice of parameters. Then, we highlight a critical limitation of the current DDB framework, namely that it does not ensure data consistency. To address this problem, we propose modified inference procedure that imposes data consistency without the need for fine-tuning. We term the resulting method data Consistent DDB (CDDB), which outperforms
inconsistent counterpart in terms of both perception and distortion metrics, thereby effectively pushing the Pareto-frontier toward the opti- mum. Our proposed method achieves state-of-the-art results on both evaluation criteria, showcasing its superiority over existing methods. Code is open-sourced at https:://ittub. com HJ www.e//ooorrr///DD
1 Introduction
Diffusion models 15 38 have become the de facto standard of recent vision foundation models B2
31 B3). Among their capabilities the use of diffusion models as generative priors that can serve as plug-and-pla building blocks for solving inverse problems in imaging [[8 38 21 5]. Diffusion model-based inverse problem solvers (DIS) have shown remarkable performance and versatility, as one can leverage the powerful generative prior regardless of the given problem at hand, scaling to
linear 18 38 21,, non-linear
36.. and noisy problems (1
Although there are many advantages of DIS, one natural limitation is its slow inference. Namely, the overall process of inference-startingg from Gaussian noise and being repeatedly denoised to form a clean image-is kept the same, although there are marginal changes made to keep the sampling process consistent with respect to the given measurement. such cases, the distance between the reference Gaussian distribution and the data distribution remains large, requiring inevitably large number of sampling steps to achieve superior sample quality. On the other hand, the distribution of the measurements is much more closely related to the distribution of the clean images. Thus, intuitively, would cost us much less compute if we were allowed start the sampling process directly from the measurement, in the usual method of direct inversion in supervised learning schemes.
Interestingly, several recent works aimed to tackle this problem under several different theoretical motivations: Schrödinger bridge with paired data 26 2) new formulation of the diffusion process
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

DC0
DDS
CEOB CLEB (2005)
13033
C309
besthe
PSNB)
(a) NFE vs. PSNR (higher is better) (b) NFE vs. FID (lower better)
(c) Shift in pareto-frontier
Figure 1: Quantitative metric of SBSB 22 (denoted DDB) vs. proposed CDDB on sr4x-bicubic task.
via constant- -pped continual degradation [9, and 3) CeateinnUhlenbeckk stochastic differential equation (OU-SDE) 28]. While developed from distinct motivations, the resulting algorithms be understood in a unifying framework with minor variations: we define this new class of methods Direct Diffusion Bridges (DDB; Section 3.1). In essence, DDB defines the diffusion process from the clean image distribution / = 0 to the measurement distribution / = 1 as the convex combination between the paired data, such that the samples æ goes through continual degradation as t t= 0 1. In training, one trains a time-conditional neural network Gg that learns a mapping to zo for all timesteps, resulting in an iterative sampling procedure that reverts the measurement process.
Using such an iterative sampling process, one can flexibly choose the number of neural function evaluations (NFE) generate reconstructions that meet the desiderata: with low NFE, less distortion can be achieved as the reconstruction regresses towards the mean [9] with high NFE, one can opt for high perceptual quality at the expense of some distortion from the ground truth. This intriguing property of DDB creates Pareto Pareto-frontier reconstruction quality, where our desire would to maximally pull the plot towards high perception and low distortion (bottom right corner of Fig. Ic).
In this work, we assert that DDB is missing a crucial component of data consistency, and devise methods to make the models consistent with respect to the given measurement by only modifying the sampling algorithm, without any fine-tuning of the pre-trained model. We refer to this new class of models data Consistent Direct Diffusion Bridge (CDDB; Section 3.2) and show that CDDB is capable of pushing the frretterr further towards the optima (lower distortion: Fig. la higher perception: Fig. 1b overall trend: Fig. Ic) across variety of tasks. Theoretically, we show that CDDB is a generalization of DDS (Decomposed Diffusion Sampling) (6., a recently proposed method tailored for DIS with Gaussian diffusion, which guarantees stable and fast sampling. We then propose another variation, CDDB-deep, which can be derived as the DDB analogue of the DPS (5) by considering deeper gradients, which even further boosts the performance for certain tasks and enables the application to nonlinear problems where one cannot compute gradients the usual manner (e.g. JPEG restoration). In the experiments, we showcase the strengths of each algorithm and show how one can flexibly construct and leverage the algorithms depending on the circumstances.
Background
2.1 Diffusion models
Diffusion models
13
38
19
defines the forward data noising process p(2;(20)) as
T 0g 02z, ; NO I) for t [0,
(I)
where at, a, controls the signal component and the noise component, respectively, and are usually designed such that a² of = 15 (22) Starting from the data distribution Pdasa p(æa), the noising process in (T) gradually maps p(x;) towards isotropic Gaussian distribution as -> 1, i.e. p(æ1) ~ (). Training neural network to reverse the process amounts training residual denoiser
[ <1[2]
min Ex Camassp ) (m0
(2)

---

such that (x,)) 20 Furthermore, it can be shown that epsilon matching is equivalent to the denoising score matching (DSM) 16 37 objective up to constant with different parameterization
min Eg, 2o.e
[|i]" (xx) - V log 1og
(mx))/ox Moreover, for optimal f* and under regularity conditions, s8- (x v. log p Then, sampling from the distribution can be performed by solving the reverse-time generative SDE/ODE [38] [9] governed by the score function. It is also worth mentioning that the posterior mean, the so-called denoised estimate can be computed via Tweedie's formula 13
such that 1/0 (2:e) = 0..00
1
o 8 2x(00|aa))(201zz (2o
(x: + of Va, m, log p(a)) =
- (ax: + oi (xx)).
(4)
In practice, DDPM/DDIM solvers 15
35 work by iteratively refining these denoised estimates.
2.2 Diffusion model-based inverse problem solving with gradient guidance
Suppose now that we are given measurement y obtained through some Gaussian linear measurement process A, where our goal is to sample from the posterior distribution p(zlyy. Starting from the sampling process running the reverse SDE/ODE to sample from the prior distribution, one can modify the score function to adapt it for posterior sampling 38 []. By Bayes rule, Va, log p( (e y)
Va, log p(xx) + Va, log p(y xt). where wa, log p(x;) = sy- (xz). However, Va. log pogp( y (22) is intractable. Several methods have been proposed to approximate this deeedeepndennt likelihood,
two the most widely used being DPS [5] and IIGDM 33.. DPS proposes the following Jensen approximation
V a, log Jogplyla x4)
(DPS)
V s. log @ap))
Oioo 0||
A - bll Ozop
A A
A
(5)
dx,
Ox
of which the chain rule is based on the denominator layout notation 41]. Here, that the gradient term can be represented as the Jacobian (J) vector (V) product (JVP). In the original implementation of DPS, the two terms are not computed separately, but computed directly Væ, l A@oo ? where the whole term can be handled with backpropagation. By this choice, DPS can also handle non-linear operators when the gradients be computed, e.g. phase retrieval, forward model given as a neural network. On the other hand, IIGDM proposes
w; logr(y)a 2x)
(IGDM)
N(AAogje AAT +1) da.
A A@os)
(6)
where A = (AAAT)- is the Moore-Penrose pseudo-inverse. Using the JVP for implementation. is no longer required that the whole term is differentiable. For this reason, TTGDM can be applied to cases where have non-differentiable, non-linear measurements given that operation analogous to pseudo-inverse can be derived, c.g. JPEG restoration. Notably, the update step of DPS can be achieved by simply pre-conditioning IIGDM with A A. Implementing DIS with DPS (5) or IIGDM ( amounts augmenting the gradient descent steps in between the ancestral sampling iterations.
While these methods are effective and outperforms the prior projection-basee based approaches [8 they also have several drawbacks. Namely, the incorporation the U-Net Jacobian is slow, compute- heavy, and often unstable 12 34 For example, when applied to MRI reconstruction in medical imaging, DPS results in noisy reconstructions
6 possibly due to unstable incorporation of the Wirtinger derivatives [23). and IIGDM is hard use as it is non-trivial to compute At. 1. In order to proposed to use numerical optimization (i.e. conjugate gradients; CG) in the clean image domain, bypassing the need to compute Consequently, DDS achieves fast and stable reconstructions for inverse problems in medical imaging.
circumvent these issues, DDS 6

---

IS 26
InDI 9
Definition Motivation
Schrödinger bridge
Small-step MMSE
3, ( [min + 23at,
E [0. (0.0.5)
23 2Bat,
te [.05
Base process
Jo Sr dr, S. B, dr
Linear symmetric
Const
Diffussin process
dg 0i
/ 72) T8 77)
t e2
Sampling
22/12 (o -i-2
s/t
s (e) 27)
Table Comparison between different types of DDB. 3a == Smax in Appendix B
Buii Further details are given
Main Contributions
3.1 Direct Diffusion Bridge
We consider the case where we can sample æ := x ~ p(æ), and æ := p(y/ i.c. paired data for training. Adopting the formulation of SS (26) we define the posterior of æ, to be the product of Gaussians A(x:, 2o, 7²) and N Nar;; #1, 72). such that
p( [0, x1) N Øri
+
72 7
Note that the sampling of from can be done by the reparametrization trick
at (1 Oxp a + a + o,z, No I),
(8)
where := 5
This diffuusin bridge introduces a continual degradation process 21). starting from the clean image at t degradation at / 1, with additional stochasticity induced by the noise component a,, Our goal is to train a time-dependent neural network that maps any æ, to zo that recovers the clean image. The training objective for SS (26) analogous to denoising score matching (DSM) [16] reads
by taking convex combination
maximal
min E Byy-((ym)) m-pt)) [|, 2t 30
(9)
7
which is also equivalent training residual network Go with ming B[]]Go(x;) zu 11] For brevity, we simply denote the trained networks as Gg- even if it is parametrized otherwise. Once the network is trained,
reconstruct mo starting from æ by, for example, using DDPM ancestral sampling [15] where the posterior for & < reads
p( LD, vt.)
N/ (x.i (1 azi1) )ao + azi 1)
(10)
with a Cp e to yield w. ~ p(a, Coje 2.).
7/ 2 G- 2in At inference, mo is replaced with neural network-estimated
However, when the motivation is to introduce 1) tractable training objective that learns to recover the clean image along the degradation trajectory, and 2) devise a sampling method to gradually revert the degradation process, find that the choices made for the parameters in (8) can be arbitrary, as long as the marginal can be retrieved in the sampling process (10). In Table we summarize the
We ignore scaling constants that are related to the measurement noise for simplicity. For implementation,
this can be absorbed into the choice of step sizes.
2 For cases where there dimensionality mismatch, we use æ1 A y. keep this notation for simplicity. Hereafter, override the definition of signal, noise coefficients Cte, that was first defined in Eq. ①I.

---

Algorithm 1 CDDB
Algorithm 2 CDDB (deep)
Require: Co #1, Gi, 0.,0 1111 -/
Require: Go. 21, at C2, a.
I: for == N - to
1: for i N to do
2: Boj: t (x,)
2:
aaj Gg-(
3:
=~ 10
3:
2 ~ N(O.
4:
x/-1+ + (1 -1 Apjc
4:
x + (1-02-1))) Ø0|c
+ 11, 1 + 1
+ to 1 + 11
5:
g- A ly A
5:
9 s2:
Bāoli A ( A 1)
6: xc-1 t- x1 + P:-19
F;-1 f- a/ + Pi-19
7: end for
7: end for
8:
return 20
8: return Zo
choices made in [9,26] to emphasize that the difference stems mostly from the parameter choices and not something fundamental. Concretely, sampling x; from paired data can always be represented as (8): convex combination of øo and æi with some additional noise. Reverse diffusion at inference can be represented as (TO): a convex combination of zo and Ø; with some stochasticity. We define the methods that belong to this category as Direct Diffusion Bridge (DDB) henceforth. Below, formally state the equivalence between the algorithms, with proofs given in Appendix A
Theorem 1.
Let the parameters of InDI in Table 1 be t
c = ( + 72). Then,
InDI and SB are equivalent.
The equivalence relation will be useful when we derive our CDDB algorithm in Section. 3.2 As final note, IR-SDE (28) does not strictly fall into this category as the sampling process is derived from running the reverse SDE. However, the diffusion process can still be represented as (8) by setting
ag = - 01 of = A2 - 20.), and the only difference comes from the sampling procedure.
3.2 Data Consistent Direct Diffusion Bridge
Motivation Regardless of the choice in constructing DDB, there is a crucial component that is missing from the framework. While the sampling process (I0) starts directly from the measurement (or equivalent), as the predictions žoj: Go are imperfect and are never guaranteed to preserve the measurement condition y = Ax, the trajectory can easily deviate from the desired path, while the residual blows up. Consequently, this may result in inferior sample quality, especially in terms of distortion. In order to mitigate this downside, our strategy is to keep the DDB sampling strategy (TO) intact and augment the steps to constantly guide the trajectory to satisfy the data consistency. similar in spirit to gradient guidance DIS. Here, we focus on the fact that the clean image estimates Co & is produced at every iteration, which can be used to compute the residual with respect to the measurement y. Taking gradient step that minimizes this residual after every sampling step results in Algorithm| [ which we name data Consistent DDB (CDDB). In the following, we elaborate on how the proposed method generalizes DDS which was developed for DIS.
CDDB as generalization of DDS 6
Rewriting TTp with reparameterization trick t Z01:) +0 zooe Gg- (x:z)
Toj
(11)
Denoiso(@r))
Noise (022)
we see that the iteration decomposes into three terms: the denoised component, the deterministic
noise, and the stochastic noise. The key observation of DDIM (35) that if the score network is fully expressive, then the deterministic noise term 2t -ojz becomes Gaussian such that it satisfies the total variance condition
(2))00) + +22 =2
(12)
allowing (T] to restore the correct marginal N(xx;; wo o.). Under this condition, DDS showed that using a few step of numerical optimization ensure the updates from the denoised image To remain on the clean manifold. Furthermore, subsequent noising process using deterministic and stochastic noises can then be used to ensure the transition to the correct noisy manifold (6].
5

---

Under this view, our algorithm can be written concisely as
3x f- Dofe + pA y A@On)
+ ( - T01e) t0a
(13)
CDenoise(ar)
Noise(as)
where we make the update step only to the clean denoised component, and leave the other components as In order achieve proper sampling that obeys the marginals, important to show that the remaining components constitute the correct noise variance and the condition assuming Gaussianity should (12). In the following, we show that this is indeed satisfied for the two cases of direction diffusion bridge (DDB):
Theorem 2. The total variance condition
12 is satisfied for both PSB and InDI.
Proof. For InDI, considering the noise variance di tc, in Table 1
(a2),0)
+ + ole
22 + -
(14)
Due to the equivalence in Theorem I the condition is automatically satisfied in 12SB. We show that this is indeed the case in Appendix
In other words, given that the gradient descent update step in CDenoise x. does not leave the clean
data manifold, it is guaranteed that the intermediate samples generated by (13) will stay on the correct noisy manifold [6]. In this regard, CDDB can be thought of as the DDB-generalized version of DDS. Similar DDS, CDDB does not require the computation of heavy U-Net Jacobians and hence introduces negligible computation cost to the inference procedure, while being robust in the choice of step size.
CDDB-deep As shown in DPS and IIGDM, taking deeper gradients by considering U-Net Jacobians often beneficial for reconstruction performance. Moreover, even provides way impose data consistency for non-lincar inverse problems, where standard gradient methods are not feasible. In order devise analogous method, we take inspiration from DPS, and propose to augment the solver with gradient step that maximizes the time-dependent likelihood (w.r.t. the measurement) p(y x.). Specifically, we use the Jensen approximation from
f plut æen ((( ar)) dæc
plyla (x)) =
(15)
Ep(eo ax)) (p(y an)) p [20 (z..)) ply (
where the last equality is naturally satisfied from the training objective (9) Using the approximation used in 15 the correcting step under the Gaussian measurement model yields
Va, logp(y)æ;) Vz. ll -
(16)
Implementing (6)) in the place of the shallow gradient update step of Algorithm | we achieve CDDB-deep (see Algorithm 2 From our initial experiments, we find that preconditioning with At as in ITGDM improves performance by small margin, and hence use this setting as default.
4 Experiments
4.1 Setup
Model, Dataset For representative DDB, we choose I'SB [26] along with the pre-trained model weights for the following reasons: 1) it is open-sourced 2) it stands as the current state-of of of-the-art,
3) the model architecture is based on ADM [1I, which induces fair comparison against other DIS methods. All experiments are based on ImageNet 256x256 100, benchmark that is considered to be much more challenging for inverse problem solving based on generative models [5], compared to more focused datasets such as FFHQ [20]. We follow the standards of [26) and test our method on the following degradations: srix-ibi pool], deblur- -untfrrr gauss), and JPEG restoration with Ik validation images.
butps//ggihubb.om////ll/////

---

Groud Truth
PaP ADMM
FSB
CDDB (ours)
CDDB-deep
Figure 2: SR(x4)-bicubes reconstruction results. CDDB/CDDB-dey corrects details wrongly captured in SS: line texture in 1s row, color/texture errors in 2nd row, topology of wings in 3rd row (see arrows).
SR(x4)
Deblur
bicubic
pool
gRUSS
uniformm
Method
SSIM 1 LPIPS | FID
1
LIPPS
1 SSIM 1 LPPPS
PSNR
SSIM
LPIPS
FID
CDDB (ours)
26.41
0,860
0.19%
19.88
26.36
0.855
0.184
17.79
37.02
0.978
0.059
5.007
31.26
0.927
0.193
23.15
FSB 26
25.22
0.802
0.250
21.13
25.08
0,800
0.258
23.53
36.01
0,973
0.067
5,800
30.75
0.919
0.198
23.01
HGDM
26.20
0,850
0.252
26.07
0.256
26.97
DDRM X1
26.05
0.838
0,270
46.49
25.54
0.548
0.257
40.40
36.73
0.975
0.071
4.346
29.21
0.901
0.210
19.97
DDNM 10
26.41
0.901
0.250
38.63
26.04
0.218
33.15
PaP-ADMM
0.350
72.63
28
60.27
25.47
0.701
0.416
ADMM-TV
22.55
0.505
0.493
122.7
22.31
0.574
0.512
119.4
24.67
0.773
0.324
50.74
21.72
0.600
0.491
98.15
Table 2: Quantitative evaluation of SR, deblur task on ImageNet 256 256-1k; Bold: Best, under: second best. Colored: DDB methods.
Baselines, Evaluation Along with the most important comparison against I2SB, we also include comparisons with state-of-the-art DIS methods including DDRM (21]. DPS (5). IGGDM [36]. DDNM [40), and DDS [6]. For choosing the NFE and the hyper-parameters for each method, closely abide the original advised implementation: DDRM (20 NFE), DPS (1000 NFE), H1GDM (100 NFE), DDNM (100 NFE), DDS (100 NFE). We find that increasing the NFE for methods other than DPS does not induce performance gain. We also note that exclude IIGDM and DDNM for the baseline comparison in the deblurring problem, as directly leveraging the pseudo- inverse matrix may result in unfair boost in performance (29]. We additionally perform comparisons against PnP-ADMM [4] and ADMM-TV TV, which are methods tailored towards higher SNR. For I'SB along with the proposed method, most stable, and choose 1000 NFE for all other tasks. Further details on the experimental setup can be found in Appendix C
choose 100 NFE for JPEG restoration
found be the
4.2 Results
Comparison against baselines Throughout the experiments, thoroughly analyze both distortion and perception of the reconstructions obtained through our method against other DDB, DIS, and iterative optimization methods. Note that for the recent diffusion-baut methods, analysis has been
7

---

Degracie
PSB
CDDB (ours)
Grouné Truth
Degraded
BSB
CDDB (ours)
Ground Treth
Results impainting (Left) and deblurring (Right). For inpainting, boundaries are corrected (row 1) and artifacts are corrected/removed (row 2,3). For deblurring, halo artifacts are corrected, and grid patterns from the background are alleviated.
Figure 3:
mainly focused on perceptual metrics [5 36 26]. mainly because these methods excel on these metrics, but often compromising distortion metries. DDB methods often take this to the extreme, where one can achieve the best PSNR with very little NFE, and the PSNR consistently degrades as one increases the NFE [9] 26 (See Fig. Tc.. Despite this fact, the standard setting in DDB methods is to set high NEE as one can achieve much improved perceptual quality. On the other hand, conventional iterative methods are often highly optimized for less distortion, albeit with low perceptual quality. While this trade-off may seem imperative, we show that CDDB can improve both aspects, putting it in the place of the state-of-the-arr on most experiments (See Tab. 2 A similar trend can be observed in Fig. where we see that CDDB greatly improves the performance of DDB, while also outperforming DIS and iterative optimization methods.
CDDB pushes forward the Pareto-fronticn It is widely known that there exists an inevitable trade-off of distortion when aiming for higher perceptual quality [ (24) This phenomenon has been reconfirmed consistently in the re- cent DIS (5 36) and DDS ( 26 methods. For DDB, can flexibly control this trade-off by simply choosing dif- ferent NFE values, creating Pareto-frontier with higher NFE tailored towards perceptual quality. While this prop- erty is intriguing, the gain we achieve when increasing the NFE decreases exponentially, and eventually reaches a bound when NFE > 1000. In contrast, we show in Fig. thht CDDB pushes the bound further towards the optima. Specifically, 20 NFE CDDB outperforms 1000 NFE DDB in PSNR by > 2 db, while having lower FID (i.e. better perceptual quality). To this point, CDDB induces dramatic acceleration 50x) DDB.
Method
PSNR
SSIM 1 I.IPS |. FID
(ours)
19.48
FSB
26
26.
0.832
0.266
20.35
HGDMM 36 DDRM
26.09
0.842
0.282
30.27
21
26.33
0.829
0.330
47.02
Table 3: Quantitative evaluation of the JPEG restoration (QF 10) task.
CDDB CDDB-deep The two algorithms presented in this work share the same spirit but have different advantages. CDDB generally has higher speed and stability, possibly due to guar- anteed convergence. As result, it robustly in- creases the performance SR and deblurring. In contrast, considering the case of inpainting and JPEG restoration, CDDB cannot improve the performance of DDB. For inpainting, the default setting of SB ensures consistency by it- eratively applying replacement, implemented in [4]] As the measurement stays in the pixel space, the gradients cannot impose any con- straint on the missing pixel values. CDDB-deep useful such a situation, as the U-Net Jaco- bian has global effect on all the pixels, improv-
Degruded
FSB
CDDB (ours)
Ground Trath
Figure 4: Results on JPEG restoration (QF=10). CDDB recovers texture details (row 1,3), and color details (row 2).

---

ing the performance by inducing coherence. CDDB-deep also enables the extension to nonlinear inverse problems where one cannot take standard gradient steps. This is illustrated for the case of JPEG restoration in Tab. and Fig. 4 where we see overall improvement in performance compared PSB.
Noise robustness DIS methods are often de- signed such that they are robust to measurement noise 51. contrast, this is not the case for DDB as they are trained in supervised fashion: not explicitly trained with synthetic adding of noise, the method does not generalize well to noisy measurements, as can be seen in Fig. []] On the other hand, note that with CDDB, we are essentially incorporating a Gaussian likelihood model, which naturally enhances the robustness to noise. As a result, while SB tends to propa- gate noise (best seen in the background), we do not observe such artifacts when using CDDB.
Nokey (f - 0.015)
FS8l
CDB8 (nurs)
Garnd Trath
Figure 5: Results noisy SRx4 reconstruc- tion. SB propagates noise the reconstruction. CDDB effectively removes noise.
5 Discussion
Extension to other related works Going beyond the paired inverse problem setting and considering the Schrödinger Bridge (SB) problem [2 [) or more generally transport mapping problems [27] between the two unmatched distributions, it is often desirable to control the deviation from the start of sampling. A concrete example would the case of image-to-iman not want to alter the content of the image. As CDDB can be thought of as a regularization method that penalizes the deviation from the starting point, the application is general and can be extended to such SB problems at inference time by using the gradients that minimize the distance from the start point. We leave this direction for future work.
translation 443 where one does
Data consistency in supervised learning frameworks The first applications supervised deep learning to solve inverse problems in medical imaging (e.g. CT (17), MRI reconstruction (39) mostly involved directly inverting the measurement signal without considering the measurement constraints. The works that followed [I 14] naturally extended the algorithms by incorporating measurement consistency steps in between the forward passes through the neural network. Analogously, CDDB is natural extension of DDB but with high flexibility, we do not have to pre-determine the number of forward passes I or modify the training algorithm [4]
6 Conclusion
In this work, we unify the seemingly different algorithms under the class of direct diffusion bridges (DDB) and identify the crucial missing part of the current methods: data consistency. Our train -ree modified inference procedure named consistent DDB (CDDB) fixes this problem by incorporating consistency-impossie
gradient steps in between the reverse diffusion steps, analogous to the recent DIS methods. We show that CDDB can be seen as a generalization of representative DIS methods (DDS, DPS) in the DDB framework. We validate the superiority of our method with extensive experiments diverse inverse problems, achieving state-of-the-arr sample quality in both distortion and perception. Consequently, we show that CDDB can push the Pareto-frontier of the reconstruction toward the desired optimum.
Limitations and societal impact The proposed method assumes prior knowledge of the forward operator. While we limit our scope to non-blind inverse problems, the extension CDDB to blind inverse problems 7 30 will be possible direction of research. Moreover, for certain inverse problems (e.g. inpainting), even when do observe improvements qualitative results, the quantitative metrics tend to slightly decrease overall. Finally, inheriting from DDS/DIS methods, our method relies on strong priors that are learned from the training data distribution. This may potentially lead to reconstructions that intensify social bias and should be considered in practice.

---

Acknowledgments and Disclosure of Funding
This research was supported by the KAIST Key Research Institute (Interdisciplinary Research Group) Project, by the National Research Foundation of Korea under Grant SEFF-222RR/A2BBBHH
by the Korea Medical Device Development Fund grant funded by the Korea govermment (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 1711137899, KMDF_ SDFF__RR222000111110000000001091 by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea goverament(MSST) (No. 201900000000,,, Artificial Intelligence Graduate School Technology Development Project for Customs Admin-
Program (AISST)) and by the istration through National Research Foundation of KoRea(NRF) funded by the Ministry of Science & ICT and Korea Customs Service(NRF-202 IMM11
References
[1] Hemant K Aggarwal, Merry P Mani, and Mathews Jacob. MoDL: Model-based deep learning architecture for inverse problems. IEEE transactions on medical imaging, 38(2):394-405, 2018.
[2] Thilo Balke, Fernando Davis, Cristina Brendt Wohlberg. Scientific Computational Imaging COde (SCICO). Software library available from https: btttps:/ggitub... com/ lanl
Michael McCann, Luke Pfister, and
wwt.//ooomaannniiiiae /ccco 2022.
[3] Yochai Blau and Tomer Michaeli. The -istorion tradeoff. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6228-6237, 2018.
[4] Stanley H Chan, Xiran Wang, and Omar A Elgendy. Plug-and-play ADMM for image restora- tion: Fixed- exxnt-point&&e convergence and applications. IEEE Transactions on Computational Imaging. 3(1:884-98, 2016.
[5] Hyungjin Chung. Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. International Conference on Learning Representations, 2023.
[6] Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Fast diffusion sampler for inverse problems by geometric decomposition. arXiv preprint 0rXiv:2303.0575: 2023.
[7] Hyungjin Chung, Dohoon Ryu, Michael T Mccann, Marc L Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023,
[8] Valentin De Bortoli, James Thorton,, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34:1 34:17695-17709, 2021.
[9] Mauricio Delbracio and Peyman Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. arXiv preprint arXiv:2303.11433 2023.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, Iece, 2009.
[11] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, J.d Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[12] Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Composi- tional generation with energy-based diffusion models and meme. In International conference on machine learning. PMLR, 2023.
[13] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):1602-1111
2011. 2011.
10

---

[14] Harshit Gupta, Kyong Hwan Jin, Ha Q Nguyen, Michael T McCann, and Michael Unser. Cnn-based projected gradient descent for consistent ct image reconstruction. IEEE transactions on medical imaging, (7(6):1440-1433 2018.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851 2020.
[16] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4). 2005.
[17] Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. Deep convolu- tional neural network for inverse problems in imaging. IEEE Transactions on Image Processing. 26(9):450944522 2017.
[18] Zahra Kadkhodaie and Eero P Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit a denoiser. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[19] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurlPS, 2022.
[20] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[21] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[22] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint arXiv:2 2021.
[23] Ken Delgado The complex gradient operator and the cr-calculus. arXiv preprint arXiv:0906.48335, 2009.
[24] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4681-4690 2017.
[25] Christian Léonard. A survey of the schrödinger problem and some its connections with optimal transport. arXiv preprint arXiv:1308.000 021500000......1 2013.
[26] Guan-Horng Liu, Arash Vahdat, De-An Huang, Evangelos A Theodorou, Weili Nie, and Anima Anandkumar. SBS image-to-lmage Schrödinger Bridge. In International conference on machine learning. PMLR, 2023.
[27] Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
(28) Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Image restoration with mecantrev stochastic differential equations. In International conference on machine learning. PMLR, 2023.
[29] Sladjana Miljković, Marko Miladinović, Predrag Stanimirovié, and Igor Stojanović. Application of the pseudoinverse computation in reconstruction of blurred images. Filomat, 26(3):453-46 2012.
[30] Naoki Murata, Koichi Saito, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Gibbsddrm: A partially collapsed gibbs sampler for solving blind inverse problems with denoising diffusion restoration. In International conference on machine learning. PMLR, 2023.

---

[31] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[33] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494 2022.
[34] Tim Salimans and Jonathan Ho. Should ebms model the energy or the score? Energy Based Models Workshop-ICLR 2021,2000 2021.
[35] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations, ICLR, 2021.
[36] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pscudoinverse-guine ggended diffusion models for inverse problems. In International Conference Learning Representations, 2023.
[37] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, volume 32, 2019.
[38] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th International Conference on Learning Representations, ICLR, 2021.
[39] Shanshan Wang, Zhenghang Su, Leslie Ying, Xi Peng. Shun Zhu, Feng Liang, Dagan Feng, and Dong Liang. Accelerating magnetic resonance imaging via deep learning. In 2016 IEEE 13th international symposium biomedical imaging (ISBI), pages 514-517. IEEE, 2016.
[40] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023.
[41] Jong Chul Ye. Geometry of Deep Learning. Springer, 2022.
[42] Kai Zhang. Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE transactions on image processing, (6(7):3142-313 2017.
[43] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232. 2017.
12


Reviews:
[{"review_id": "ELObkAyXia", "reviewer": "Reviewer_7H1C", "summary": "The paper introduces a unified view of several existing diffusion-model-based methods for solving the inverse problem (i.e., DPS and $\\Pi$GDM), and proposes a new approach for this problem called Direct Diffusion Bridge (DDB) which is derived from the DDPM ancestral sampling, and to some extent, links to the Image-to-Image Schrodinger bridge (I$^2$SB). The main idea behind DDB is using a trained neural network $G_{\\theta^*}$ to compute a denoised image $\\hat{x}\\_{0|t}$, which is an approximation of $x_0$, from $x_t$. The paper also proposes an improved version of DDB called Consistent DDB (CDDB), which according to the authors, is more beneficial for reconstruction (of $x_0$).", "questions": "1) What are the architecture and configurations of the residual network $G\\_{\\theta^*}$? I couldn’t find any detail about $G\\_{\\theta^*}$ in the main paper or the supplementary material. Does it depend on time $t$ besides $x_t$\n\n2) What are the technical differences between the method in this paper and the use of Consistency Models for the inverse sampling problem?\n\n3) In Algorithm 1, is $x\\_1$ $y$ or Gaussian random noise? Since I$^2$SB can take the corrupted image $y$ as input, do the authors think their method can do so?\n\n4) In Eq 15, $\\mathbb{E}[x_t]$ should be $\\mathbb{E}[x_0|x_t]$.\n\n5) The main difference between Algorithm 1 and Algorithm 2 is the computation of $g$. In Alg. 1, $g$ is computed as $\\nabla_{\\hat{x}\\_{0|t}} \\log p(y|\\hat{x}\\_{0|t})$ while in Alg. 2, $g$ is computed as $\\nabla_{x_t} \\log p(y|\\hat{x}\\_{0|t})$. In my view, Alg. 2 is *more theoretically correct* than Alg. 1 rather than just being more beneficial for reconstruction as mentioned in the paper in lines (181 -> 183). Thus, I hope the authors could make this point clearer in their paper rather than just mentioning CDDB as an alternative for DDB when the inverse problem is non-linear. Even when the inverse problem is linear, we still need the term $\\frac{\\partial \\hat{x}\\_{0|t}}{\\partial x_t}$ which is the Jacobian in Eqs. 5, 6.\n\n6) At line 181, the authors should be clear about the “U-Net Jacobians”. Is the U-Net here the noise network $\\epsilon$ of the diffusion model or the network $G\\_{\\theta}$? I guess it should be $G\\_{\\theta}$ since we are considering $\\frac{\\partial \\hat{x}\\_{0|t}}{\\partial x_{t}}$. The authors give me the impression that they are trying to hide details about $G_{\\theta}$, which in fact plays a critical role in the paper.", "limitations": "Please refer to my questions above", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The unified view between DPS and $\\Pi$GDM is interesting though I think it’s not hard to figure out as both methods share the same motivation.\n- Experimental results seem to support the proposed method there are improvements in both perception and signal-to-noise ratio compared to previous works. However, the fairness in settings of different methods should be clarified in the paper.", "weaknesses": "1) The presentation in the paper makes the motivation for the proposed method “Direct Diffusion Bridge” (DDB) unclear. From my view, neither the unified view between DPS and $\\Pi$GDM (Section 2.2) nor the I$^2$SB model (Section 3.1) motivates the design of the denoised image $\\hat{x}_{0|t}$ in the paper. In fact, it seems like in Section 3.1 the authors try to link the posterior of $x_t$ in the I$^2$SB model (Eq. 7) with the posterior of $x_s$ (s < t) in the DDPM (Eq. 10), which, I guess, leads to the name “Direct Diffusion Bridge”.\n\n2) I think the main contribution *in terms of technique* in this paper is a new way to compute $\\hat{x}\\_0$ (line 2 in Algorithm). Other parts of Algorithm 1 (lines 3 -> 7) were already proposed in previous works (e.g., DPS, DDS). However, the trained network $G\\_{\\theta^*}$, which plays an important part in the algorithm, is not described in detail in the paper. It is only mentioned briefly at lines 120 -> 121. Besides, no ablation study was conducted to validate the use of $G\\_{\\theta^*}$ with other methods for computing $\\hat{x}\\_0$ from $x\\_t$ (e.g., Tweedie’s formula).\n\n3) The authors don’t mention how $G\\_{\\theta^*}$ can handle $x\\_t$ with different time steps $t$ in the paper. Does it take both $x_t$ and $t$ as input? In fact, from my view, $G\\_{\\theta^*}$ is very similar to the network in the paper Consistency Models [1]. However, I don’t see the authors discuss Consistency Models in their paper. Since Consistency Models can be directly used for DIS (a lot of experiments on inverse sampling can be found in [1]), I am keen to see the empirical comparison between this method and Consistency Models.\n\n4) Please correct me if I am wrong but I cannot find the NFE of the proposed method in the main paper or the supplementary material. Therefore, it is hard to fairly compare the performance of the proposed method with other baselines.\n\n[1] Consistency Models, Song et al., ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "After reading the comments, we would like to respectfully stress that there seems to be a **clear misunderstanding** of the paper. 1) Our work aims for the unified view of DDB, not DIS; 2) We do not introduce a new way to compute $\\hat{x}_0$, which is directly estimated from the neural network. Only the inference process changes; 3) Information about the network architecture, NFE, etc. are already clearly stated in the paper, which by no means do we attempt to hide. For detailed responses on each of the points, please see below.\n\n**W1. Motivation of the term “Direct Diffusion Bridge”?**\n\n**A.**  We emphasize that the aim of the paper was **not** to propose a unified view between DPS and PGDM, **nor** to link the posterior of I2SB with the posterior of DDPM. \nIn fact, the first aim of this paper was to unify the seemingly different approaches---I2SB and InDI---under a common framework. Note that different from DIS approaches (e.g. DPS, PGDM), these methods are trained in a paired fashion to directly invert the degraded image. This is most clearly seen in the case of t=1, where it would correspond to the simple supervised learning setting, hence the term **Direct**. Furthermore, these strategies train a neural network with multiple levels of degradation such that at test time, one can utilize an ancestral sampling procedure used in diffusion models for iterative refinement, hence the name **Diffusion**.\n\n**W2. How to compute $\\hat{x}_0$? Ablation studies?**\n\n**A.** \nWe are not introducing a new way to compute $\\hat{x}_0$. In fact, $\\hat{x}_0$ is a direct output of the neural network up to a pre-defined multiplicative and additive constant. \nThis can be interpreted as the minimum mean squared error (MMSE) estimate $\\mathbb{E}[x_0|x_t]$ up to parametrization/optimization error because the neural network was trained to minimize the l2 distance against the target $x_0$ given $x_t$. Note that $x_t$ here is different from the usual notion of $x_t$ in the diffusion model literature, where it is a Gaussian noised version of $x_0$. Rather, it is given by (8). Hence, we cannot use Tweedie’s formula directly as 1) retrieving $x_0$ from $x_t$ is not denoising, and 2) the trained neural network is not a score function. However, note that Tweedie’s formula is just a way of computing the posterior mean for the case of Gaussian noisy images. In this regard, the method of producing $x_0$ can be thought of as analogous to Tweedie’s formula for general degradations arising in direct diffusion bridges.\n\n**W3, Q1, Q2. Net. Arch., Comparison with CM[3]?**\n\n**A.**\nWe take the neural net $G_\\theta$ directly from I2SB without retraining (L194-195), and hence the model architecture follows that of ADM [4], which takes in both $x_t$ and $t$ as input. \n\nIn Consistency Models (CM)[3], $x_t$ is a Gaussian noisy version of $x_0$ as in diffusion models, and not as in DDBs. In this regard, solving inverse problems using Algorithm 4 of [3] falls into the category of DIS, and is especially similar to [5] and [6]. Note that our technical contribution is to take an existing trained DDB, and modify the inference procedure such that it boosts performance while being more faithful to the measurement. Thus, including a comparison against CMs would be adding yet another DIS to the current list of DIS methods used for comparison (DPS, PGDM, DDRM, DDNM), which, in our humble opinion, could be beneficial, not necessary.\n\nHaving said that, indeed, CMs share important aspects with DDBs, especially considering the inference procedure (i.e. Algorithm 1 of [3]). Since the network is distilled to produce a one-step estimation of the endpoint of PF-ODE trajectory, it can directly produce a clean image $\\hat{x}_0$ from $x_t$. It would be interesting to discuss the similarities and differences between the proposed method and CMs, which will be included in the modified discussion. In the current official repo of CMs, ImageNet 256x256 checkpoint is missing and we would have to train the model from scratch, which would be infeasible within a week considering the limited computational resources. However, we do plan to include this comparison after the rebuttal period and include them in the revised manuscript.\n\n**W4. NFE of CDDB**\n\n**A.** Please see L210-211. JPEG restoration: 100 NFE, others: 1000 NFE.\n\n**Q3.**\n\n**A.** Please see L111. $x_1=y$, the same way as in I2SB.\n\n**Q4.**\n\n**A.** Fixed.\n\n**Q5, Q6. Correctness of incorporating U-Net Jacobian**\n\n**A.** We agree that $\\nabla_{x_t}$ is more exact. We will make this clearer in the revised manuscript. However, we would like to note that in practice, due to the U-Net Jacobian being unstable, it is often beneficial to skip this, consistent with observations made in [7,8].\n\nWe respectfully disagree with your comment that we are hiding details about the network. U-Net Jacobian means the Jacobian of $G_\\theta$. It is very explicit that we take the models directly from I2SB, and give discussions on the model architecture (L194-197). Among the lines, we also mention that the model architecture stems from ADM, which is U-Net based, hence the name U-Net Jacobian, similar to how the term is used in diffusion model literature.\n\n**References**\n1. Guan-Horng et al. \"Image-to-image Schrödinger Bridge\" ICML (2023).\n2. Delbracio and Milanfar. \"Inversion by direct iteration: An alternative to denoising diffusion for image restoration.\" TMLR (2023).\n3. Yang et al. “Consistency models.\" ICML (2023).\n4. Prafulla and Nichol. “Diffusion models beat gans on image synthesis.” NeurIPS (2021)\n5. Yang et al. “Score-based generative modeling through stochastic differential equations.” ICLR (2021).\n6. Wang et al. “Zero-shot image restoration using denoising diffusion null-space model.” ICLR (2023).\n7. Poole et al. “Dreamfusion: Text-to-3d using 2d diffusion.” ICLR (2023).\n8. Chung et al. “Fast Diffusion Sampler for Inverse Problems by Geometric Decomposition.” arXiv (2023).\n"}, {"review_id": "y6OsnjKRsJ", "reviewer": "Reviewer_TLjQ", "summary": "This paper studies the existing works about Direct Diffusion Bridges (DDB) with a unified scheme and limitations, and proposes a modified inference procedure that imposes data consistency without the need for fine-tuning, called data Consistent DDB (CDDB), as a new diffusion model-based inverse problem solvers.\n", "questions": "Please see the weaknesses for specific questions. \n\nMinor comments:\n- In Table 2, the best results are not bolded correctly in the PSNR of “pool” images as mentioned in the caption? \n", "limitations": "The authors discuss the limitations and societal impact of the proposed method at the end of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written and organized. Especially, the writing of the background section clearly describes and sorts out several important relevant works in a unified perspective.\n\n+ This paper solves an important problem to study the data consistency problem with diffusion models, which can be used to solve inverse problems generally.\n\n+ The experiments performed on natural images are comprehensive, and results show satisfying reconstruction quality, with comparison methods in both DIS and DDB methods.\n", "weaknesses": "- The paper studies a series of DDB methods which match the clean data/image distribution with measurements distribution, with the key contribution to add the data consistency term into the inference sampling steps. Actually, this kind of data consistency term has been introduced in quite a few previous works about diffusion model-based inverse problem solving, so called DIS/DDS methods in this paper. Although this paper claims the proposed approach is a generalization of the DDS method, after reading the whole manuscript, one essential question is still not very clear why DDB methods should outperform DIS/DDS methods, if they share a similar scheme from some perspective and both consider data consistency constraints during the sampling steps.\n\n- As mentioned above, the idea of updating the clean part for data consistency from DDIM formulation has been introduced in some previous works, such as DDS paper [5] and DDNM paper [39]. Although the paper claims that the proposed approach is a generalization of method in [5], the comparison with these two similar papers is missing in the experiments.\n\n- Besides, for the proposed DDB methods, it seems to require the paired data for training the distribution matching, different from the series of DIS methods which generally are trained without paired data. Although the paper discuss the relevance with supervised learning frameworks, the proposed method may need to be also compared with other supervised methods or conditional diffusion methods, which are missing in the current experiments and results. Besides, in the Discussion section, the paper claims the CDDB is flexible and does not have to pre-determine the number of forward passes or modify the training algorithm. This flexibility is also carried with all the DIS methods generally which are used to solve inverse problems based on trained diffusion models, but not the unique characteristics from DDB methods, which may need to be clarified clearly in the paper.\n\n- In the proposed CDDB (deep) method, it relies on estimating a pseudo-inverse to preserve data consistency in each sampling step. But for solving non-linear inverse problems, it rarely uses a pseudo-inverse formulation, which may not be straightforward to obtain pseudo-inverse for most nonlinear problems.\n\n- As mentioned in the Discussion, in order to show the advantage of DDB methods to match two distributions with data consistency, a great example is image translation problem, which is kind of surprising not included in the scope of this paper. But meanwhile, for image translation problem, since the forward matrix A is not explicit, is it still possible to use the proposed CDDB method to preserve the data consistency in the image translation problem?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. Why would DDB outperform DIS?**\n\n**A.**\nThe main reason that DDB often outperforms DIS (DDS falls into this category) is that DDBs are trained *specifically* for a given task with paired datasets, rather than being a *general* solver. Moreover, it learns to directly start the diffusion process from the corrupted measurement, rather than from Gaussian noise. These two factors both compromise the ability of DDB to be a general solver but enhance the ability as a specific image restoration solver, especially given the same computational budget. We believe that this is an inevitable trade-off.\n\n**W2, 3. Further comparisons**\n\n**A.** Thank you for pointing this out. Please see below for the additional comparison against DDNM and DDS. Further, we did our best to include as many supervised learning and conditional diffusion baselines as additional comparison methods under the limited time span. We plan to add an exhaustive list of comparisons in the future modified version.\n\nWe will clarify that the flexibility holds not only for DDB methods but also for DIS methods. However, it is worth noting that for DIS, we do not generally see a smooth Pareto-frontier trade-off between perception and trade-off, as in the very low NFE regime of < 20, the methods usually fail catastrophically.\n\n|        | SR x4     |           |           |           |           |           |           |           |\n|--------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|\n|        | Bicubic   |           |           |           | Pool      |           |           |           |\n| Method | PSNR      | SSIM      | LPIPS     | FID       | PSNR      | SSIM      | LPIPS     | FID       |\n| CDDB   | **26.41** | **0.860** | **0.198** | **19.88** | **26.36** | **0.855** | **0.184** | **17.79** |\n| I2SB   | 25.22     | 0.802     | 0.260     | 24.13     | 25.08     | 0.800     | 0.258     | 23.53     |\n| DDNM   | **26.41** | 0.801     | 0.230     | 38.63     | 26.04     | 0.792     | 0.218     | 33.15     |\n| DDS    | **26.41** | 0.801     | 0.230     | 38.64     | 26.04     | 0.792     | 0.218     | 33.15     |\n| ESRGAN | 25.08     | 0.792     | 0.244     | 26.38     | 24.90     | 0.803     | 0.203     | 29.38     |\n| SR3    | 24.83     | 0.769     | 0.229     | 23.46     | -         | -         | -         | -         |\n\n**W4. Reliance on pseudo-inverse? Non-linear inverse problems?**\n\n**A.**\nGood point. While the experiments in the current manuscript focus on linear and semi-linear (JPEG restoration) inverse problems, it could be hard to obtain an estimate of the pseudo-inverse for more complex non-linear inverse problems. In such cases, however, we could easily use a DPS-like gradient step without preconditioning, with the reconstruction quality that does not differ too much from CDDB-deep (L191). We show the results with and without the preconditioning here.\n\n|                                | JPEG   |       |       |       |       |       |       |       |\n|--------------------------------|---------|-------|-------|-------|-------|-------|-------|-------|\n|                                | bicubic |       |       |       | pool  |       |       |       |\n| Method                         | PSNR    | SSIM  | LPIPS | FID   | PSNR  | SSIM  | LPIPS | FID   |\n| CDDB-deep (preconditioned)     | 26.81   | 0.876 | 0.177 | 18.25 | 26.70 | 0.865 | 0.188 | 17.81 |\n| CDDB-deep (not preconditioned) | 26.73   | 0.872 | 0.185 | 18.90 | 26.59 | 0.860 | 0.189 | 18.22 |\n\n**W5. Extension to image translation**\n\n**A.** Please see general comment 3. We agree that it would be very interesting to scale CDDB to image translation problems. However, DDB-type algorithms are not generally suitable for image translation where the exact match between the two domain images are difficult to obtain (eg. horse-to-zebra, Monet-to-Van Gogh). Even if this was the case as the reviewer mentions, we do not have an explicit forward matrix $A$ for image translation problems and hence would have to resort to an approximated pseudo-forward operator. For instance, if we could jointly train another network that is trained to recover back the starting signal as in cycleGANs, we would be able to use the neural network as our forward operator and use an algorithm similar to CDDB-deep to impose consistency on the starting image.\n\n**Minor Comment**\n\n**A.** Fixed."}, {"review_id": "llgI3Lwxyc", "reviewer": "Reviewer_UzBK", "summary": "In this paper, the authors propose Consistent Direct Diffusion Bridge (CDDB) a modification of the Direct Diffusion Bridge (DDB) procedure [1,2] that includes a data consistency term similar to [3,4]. The authors propose two ways to introduce the consistency: the first one forgets about the Jacobian term which is hard to compute in the guidance approximation. The second one (called CDDB-deep) compute the gradient through the observation term. It is more costly but can lead to improved results. The authors briefly justify their method in the spirit of [5] and then showcase the efficiency of their method on several inverse problems (inpainting/deblurring) on ImageNet 256x256.\n\n[1] Liu et al. (2023) -- SB: Image-to-Image Schrödinger Bridge\n\n[2] Delbracio, Milanfar (2023) -- Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration\n\n[3] Song et al. (2023) -- Pseudoinverse-Guided Diffusion Models for Inverse Problems\n\n[4] Chung et al. (2023) -- Improving Diffusion Models for Inverse Problems using Manifold Constraints", "questions": "* Another method that performs transfer tasks is Rectified Flow (albeit in a different context) [1] (or other transport based methods like [2]). Since these methods are not optimised to promote the fidelity to the term $Ax=y$ they should have a higher FID score but they might give better LPIPS, especially Rectified Flow which is deterministic. Can the authors comment on that?\n\n* I find the comment on the link with Schrodinger Bridge to be quite interesting but also quite confusing. In SB problems the coupling is not known beforehand. Hence, I don't really see how one could leverage a relation of the form $Ax=y$.\n\n* is it also possible to use some preconditioning with the pseudo-inverse in CDDB? (since in CDDB-deep \"we find that preconditioning with the pseudo inverse as in PiGDM improves performance\" it is a natural question.\n\n*Line 87: there is no footnote associated with the superscript 1.\n\n[1] Liu et al. (2023) -- Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow\n\n[2] Shi et al. (2023) -- Diffusion Schrödinger Bridge Matching", "limitations": "Adressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is very well-written and the explanations are very clear. The presentation of the method is sound and the authors did a good job at recalling the basics on diffusion models and I2SB.\n\n* I really appreciated the discussion on the difference (and ultimately the equivalence) between I2SB [1] and INDI [2]. This discussion is very well presented.\n\n* The experiments are strong. The authors have done a very thorough work investigating several metrics on challenging problems in ImageNet 256x256. \n\n* While one could complain that the novelty of the method is relatively low (basically combining I2SB with a guidance term), I think the experiments are extensive enough to warrant publication.", "weaknesses": "* I think this is overall a good paper and the next point is not the central problem tackled by the paper but I think it should be clarified (or at least the related remarks should be tamed). The authors spend a few lines deriving the variance preserving condition (Equation (12)) but not a lot is said afterwards. The authors basically refer to [1] to justify their method \"Furthermore, subsequent noising process using deterministic and stochastic noises can then be used to ensure the transition to the correct noisy manifold\". It is not clear to me that the results of [1] apply directly in the context of I2SB. Also, the current work is not self-contained as it relies heavily on the work of [1] and in particular the conjugate gradient method is never explained in the manuscript. I think that in the related work or in the theoretical section the authors should put more effort into giving details about what is [1] and how to adapt it to the current context (even though this is quite clear from reading the paper). In the experiments, the authors claim that that \"CDDB generally has higher speed and stability, possibly due to guaranteed convergence\". In the paper, no convergence results are provided. I don't see anything that would suggest that CDDB has guaranteed convergence. Overall, I think that the theoretical analysis is the weak point of the paper.\n\n* The authors \"exclude PiGDM for the baseline comparison in the deblurring problem, as directly leveraging the pseudo-inverse matrix may result in unfair boost in performances\". If this is the case then why not include PiGDM and use CDDB-deep which is also preconditioned with the pseudo-inverse? The fact that PiGDM performs (better?) than the proposed method because it makes a smart use of the pseudo-inverse does not seem like a good reason to remove it from the baseline.\n\n[1] Chung et al. (2023) -- Fast Diffusion Sampler for Inverse Problems by Geometric Decomposition", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1. Weak background for DDS, theory over-claimed**\n\n**A.**\nThank you for your encouraging comments. In the revised manuscript, we will spend more effort on the discussion of [1]. Moreover, we agree that we cannot guarantee the convergence of CDDB. The statement will be removed from the manuscript.\n\n**W2. Exclusion of methods that leverage pseudo-inverse for deblurring**\n\n**A.**\nWe would like to clarify that for all the experiments regarding deblurring, CDDB (Algorithm 1) was used. We refrained from the usage of CDDB-deep (Algorithm 2) to avoid the *inverse crime* as it is based on the perfectly known forward operator and the pseudo-inverse which may unfairly boost the restoration performance for the noiseless deblurring problem. Therefore, the metrics that were reported in Table 2 were obtained through CDDB (Algorithm 1), and hence no pseudo-inverse operation was used. This also holds for the SR experiments, and the only places where we do report the metrics on CDDB-deep are Figure 1 and Table 3, each corresponding to SR and JPEG restoration. In our humble opinion, this gives us the justification not to compare with $\\Pi$GDM.\n\n**Q1. Comparison with Rectified Flow, Diffusion Schrödinger Bridge Matching**\n\n**A.**\nThank you for pointing this out. Indeed, more general transport-based methods such as rectified flow and diffusion schrödinger bridge matching, which do not require paired data from the source and the target measure, but instead match the distributions directly, could induce better perceptual quality. That said, we are a bit confused with the reviewer’s comment, as we usually see lower FID and better LPIS at the same time regardless of the distribution matching or DDB with paired data. We therefore assume that the reviewer intended to mean worse distortion metric (PSNR/SSIM), but better perceptual metric (FID/LPIPS) - but please correct us if we are wrong). Although out of the scope of this work, if these methods were applied for solving inverse problems, CDDB-like consistency regularization would indeed boost the sample quality by correcting the improper deviations. However, we would like to note that while schrödinger bridge  methods are more versatile and can be applied to more general tasks, e.g. image translation, they are generally unstable especially when scaling to high-dimensional data. To the best of our knowledge, there has been not much progress in using general schrödinger bridge  methods for solving inverse problems in imaging.\n\n**Q2. Extension to Schrödinger Bridge Matching**\n\n**A.**\nPlease see general comment 3. While there exists no explicit notion of measurement consistency in SB problems, we could still induce a relaxed notion of consistency to constrain the transport path to be consistent with respect to the starting point. For instance, one could leverage a contrastive loss as in [1], or introduce a notion of cycle-consistency by jointly training a network that retrieves a sample from the starting measure.\n\n**Q3. Pre-conditioning for CDDB**\n\n**A.** This is indeed a natural question to ask. Unfortunately, so far we did not observe any performance gain through pre-conditioning for the case of CDDB. This issue may need further investigation in the future.\n\n**References**\n\n1. Kim, et al. \"Unpaired Image-to-Image Translation via Neural Schr\\\"odinger Bridge.\" arXiv preprint arXiv:2305.15086 (2023)."}, {"review_id": "fWZTOt3rZ0", "reviewer": "Reviewer_Qo4y", "summary": "This paper focuses on the diffusion model-based inversion problem. The paper first analyzes the current works and unifies them with the Direct Diffusion Bridges. Then, the authors point out that the data consistency is ignored by current works and propose CDDB. Experiments show that the proposed CDDB works well on various inverse tasks like inpainting, super-resolution, and deblurring.", "questions": "Please refer to the weakness part.", "limitations": "Some limitations are discusses in the Section 5.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper summarized the current work in a systematic way.\n\n2. The differences between this work and previous works are clear.\n\n3. The paper is supported well by theorems and experiments.", "weaknesses": "I am not so familiar with the related works and it is hard for me to judge the novelty and contribution of this paper, but the logic of this paper is reasonable, so I can only give borderline accept and I will increase my score if more experts support this paper. I am also wondering whether the work can be applied to the current popular textual inversion problem, since its current form only supports from one noisy image to a clean image.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your honesty and the best efforts to give a constructive feedback. While we are unsure if the method will be applicable to the case of personalization (e.g. textual inversion), please see general comment 3, where we discuss potential applications that are beyond the setting of inverse problems."}]

Author rebuttal:
We would like to thank the reviewers for their constructive and thorough reviews. We are encouraged that the reviewers think that our paper “provides an interesting unified view, with support of experimental results” (7H1C), is “well written, organized, with comprehensive experiments” (TLjQ), and is “very well presented, clear, with strong experiments” (UzBK).

We have summarized some of the major concerns that were raised by the reviewers below. Point-to-point responses were also included as a reply to each reviewer. We have also added a PDF file for additional experimental results.

**1. Clarification of the contribution, the reason for DDB being better than DIS**

We would like to clarify that the main contribution of the paper is two-fold: 1) Unification of seemingly different theories under the framework of direct diffusion bridges (DDB), and devising a new sampling algorithm that additionally imposes data consistency on DDB to yield better reconstruction results (CDDB). This direction is orthogonal and complementary to the recent endeavors to improve existing Diffusion Model-based Inverse problem Solver (DIS). A major difference, and also the reason why DDB often outperforms DIS is because DDBs are trained for specific inverse problems using paired datasets rather than being a universal solver for all tasks. As the inference distribution is pre-defined and tuned for each task, DDBs offer enhanced quality and stability.

**2. More extensive comparisons including DIS (PIGDM, DDNM, DDS), conditional diffusion-based methods (Palette), and supervised learning methods.** 

While we offered quite extensive set of comparison methods in the original submission, some of the reviewers pointed out several important works that are worth comparing against. Per the requests, we include the modified tables that now include additional methods.

**3. Discussion on how CDDB can be extended to more general cases, e.g. image translation**

For general translation tasks where there is no coupling between two specific data points (e.g. the general case of Schrödinger bridge), we cannot easily constrain the inference path to follow the consistency $y = Ax$. Even in such cases, we can generalize the notion of consistency, and impose the inference path so that one can regularize the resulting image to be similar to the starting image. One concrete example would be the use of contrastive loss [1], but one could use other strategies such as cycle consistency. 


**References**

[1] Kim, Beomsu, et al. "Unpaired Image-to-Image Translation via Neural Schrödinger Bridge." arXiv preprint arXiv:2305.15086 (2023).




Meta-review: This paper presents a modification to direct diffusion bridge to enforce data consistency (following recent works on consistency for acceleration). The paper introduces two variations (one faster and one better reconstruction results) and has a detailed comparison to multiple recent papers. This is a very competitive space and the authors did a great job in evaluating their method and showing very good results. The theoretical claims were over-stated but the authors were willing to revise their manuscript based on valid feedback on this limitation.
Decision: Accept (poster)