SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hyungjin Chung $^{1}$  Jeongsol Kim $^{1}$  Jong Chul Ye $^{2}$
Dept. of Bio and Brain Engineering
2Graduate School of AI
Korea Advanced Institute of Science and Technology (KAIST)
{hj.chung, jeongsol,jong.ye}@kaist.ac.kr
Diffusion model-based inverse problem solvers have shown impressive performance, but are limited in speed, mostly as they require reverse diffusion sampling starting from noise. Several recent works have tried to alleviate this problem by building a diffusion process, directly bridging the clean and the corrupted for specific inverse problems. In this paper, we first unify these existing works under the name Direct Diffusion Bridges (DDB), showing that while motivated by different theories, the resulting algorithms only differ in the choice of parameters. Then, we highlight a critical limitation of the current DDB framework, namely that it does not ensure data consistency. To address this problem, we propose a modified inference procedure that imposes data consistency without the need for fine-tuning. We term the resulting method data Consistent DDB (CDDB), which outperforms its inconsistent counterpart in terms of both perception and distortion metrics, thereby effectively pushing the Pareto-frontier toward the optimum. Our proposed method achieves state-of-the-art results on both evaluation criteria, showcasing its superiority over existing methods. Code is open-sourced at https://github.com/HJ-harry/CDDB
Diffusion models [15, 38] have become the de facto standard of recent vision foundation models [32, 31, 33]. Among their capabilities is the use of diffusion models as generative priors that can serve as plug-and-play building blocks for solving inverse problems in imaging [18, 38, 21, 5]. Diffusion model-based inverse problem solvers (DIS) have shown remarkable performance and versatility, as one can leverage the powerful generative prior regardless of the given problem at hand, scaling to linear [18, 38, 21], non-linear [5, 36], and noisy problems [21, 5].
Although there are many advantages of DIS, one natural limitation is its slow inference. Namely, the overall process of inference—starting from Gaussian noise and being repeatedly denoised to form a clean image—is kept the same, although there are marginal changes made to keep the sampling process consistent with respect to the given measurement. In such cases, the distance between the reference Gaussian distribution and the data distribution remains large, requiring inevitably a large number of sampling steps to achieve superior sample quality. On the other hand, the distribution of the measurements is much more closely related to the distribution of the clean images. Thus, intuitively, it would cost us much less compute if we were allowed to start the sampling process directly from the measurement, as in the usual method of direct inversion in supervised learning schemes.
Interestingly, several recent works aimed to tackle this problem under several different theoretical motivations: 1) Schrödinger bridge with paired data [26], 2) a new formulation of the diffusion process
(a) NFE vs. PSNR (higher is better)
Figure 1: Quantitative metric of  $\mathrm{I}^2\mathrm{SB}$  [26] (denoted DDB) vs. proposed CDDB on sr4x-bicubic task.
(b) NFE vs. FID (lower is better)
(c) Shift in pareto-frontier
via constant-speed continual degradation [9], and 3) Ornstein-Uhlenbeck stochastic differential equation (OU-SDE) [28]. While developed from distinct motivations, the resulting algorithms can be understood in a unifying framework with minor variations: we define this new class of methods as Direct Diffusion Bridges (DDB; Section 3.1). In essence, DDB defines the diffusion process from the clean image distribution in  $t = 0$  to the measurement distribution in  $t = 1$  as the convex combination between the paired data, such that the samples  $\pmb{x}_t$  goes through continual degradation as  $t = 0 \rightarrow 1$ . In training, one trains a time-conditional neural network  $G_{\theta}$  that learns a mapping to  $\pmb{x}_0$  for all timesteps, resulting in an iterative sampling procedure that reverts the measurement process.
Using such an iterative sampling process, one can flexibly choose the number of neural function evaluations (NFE) to generate reconstructions that meet the desiderata: with low NFE, less distortion can be achieved as the reconstruction regresses towards the mean [9]; with high NFE, one can opt for high perceptual quality at the expense of some distortion from the ground truth. This intriguing property of DDB creates a Pareto-frontier of reconstruction quality, where our desire would be to maximally pull the plot towards high perception and low distortion (bottom right corner of Fig. 1c).
In this work, we assert that DDB is missing a crucial component of data consistency, and devise methods to make the models consistent with respect to the given measurement by only modifying the sampling algorithm, without any fine-tuning of the pre-trained model. We refer to this new class of models as data Consistent Direct Diffusion Bridge (CDDB; Section 3.2), and show that CDDB is capable of pushing the Pareto-frontier further towards the optima (lower distortion: Fig. 1a, higher perception: Fig. 1b, overall trend: Fig. 1c) across a variety of tasks. Theoretically, we show that CDDB is a generalization of DDS (Decomposed Diffusion Sampling) [6], a recently proposed method tailored for DIS with Gaussian diffusion, which guarantees stable and fast sampling. We then propose another variation, CDDB-deep, which can be derived as the DDB analogue of the DPS [5] by considering deeper gradients, which even further boosts the performance for certain tasks and enables the application to nonlinear problems where one cannot compute gradients in the usual manner (e.g. JPEG restoration). In the experiments, we showcase the strengths of each algorithm and show how one can flexibly construct and leverage the algorithms depending on the circumstances.
Diffusion models [15, 38, 22, 19] defines the forward data noising process  $p(\pmb{x}_t|\pmb{x}_0)$  as
$$
\boldsymbol {x} _ {t} = \alpha_ {t} \boldsymbol {x} _ {0} + \sigma_ {t} \boldsymbol {z}, \boldsymbol {z} \sim \mathcal {N} (0, I) \text {f o r} t \in [ 0, 1 ], \tag {1}
$$
where  $\alpha_{t},\sigma_{t}$  controls the signal component and the noise component, respectively, and are usually designed such that  $\alpha_{t}^{2} + \sigma_{t}^{2} = 1$  [15, 22]. Starting from the data distribution  $p_{\mathrm{data}}\coloneqq p(\boldsymbol {x}_0)$ , the noising process in (1) gradually maps  $p(x_{t})$  towards isotropic Gaussian distribution as  $t\to 1$ , i.e.  $p(\pmb {x}_1)\simeq \mathcal{N}(0,\pmb {I})$ . Training a neural network to reverse the process amounts to training a residual denoiser
$$
\min  _ {\theta} \mathbb {E} _ {\boldsymbol {x} _ {t} \sim p (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {0}), \boldsymbol {x} _ {0} \sim p _ {\text {d a t a}} (\boldsymbol {x} _ {0}), \epsilon \sim \mathcal {N} (0, I)} \left[ \| \epsilon_ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) - \epsilon \| _ {2} ^ {2} \right], \tag {2}
$$
such that  $\epsilon_{\theta^{*}}^{(t)}(\pmb{x}_{t}) \simeq \frac{\pmb{x}_{t} - \alpha_{t}\pmb{x}_{0}}{\sigma_{t}}$ . Furthermore, it can be shown that epsilon matching is equivalent to the denoising score matching (DSM) [16, 37] objective up to a constant with different parameterization
$$
\min  _ {\theta} \mathbb {E} _ {\boldsymbol {x} _ {t}, \boldsymbol {x} _ {0}, \epsilon} \left[ \| \boldsymbol {s} _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) - \nabla_ {\boldsymbol {x} _ {t}} \log p (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {0}) \| _ {2} ^ {2} \right], \tag {3}
$$
such that  $s_{\theta^*}^{(t)}(\boldsymbol{x}_t) \simeq -\frac{\boldsymbol{x}_t - \alpha_t \boldsymbol{x}_0}{\sigma_t^2} = -\epsilon_{\theta^*}^{(t)}(\boldsymbol{x}_t) / \sigma_t$ . Moreover, for optimal  $\theta^*$  and under regularity conditions,  $s_{\theta^*}(\boldsymbol{x}_t) = \nabla_{\boldsymbol{x}_t} \log p(\boldsymbol{x}_t)$ . Then, sampling from the distribution can be performed by solving the reverse-time generative SDE/ODE [38, 19] governed by the score function. It is also worth mentioning that the posterior mean, or the so-called denoised estimate can be computed via Tweedie's formula [13]
$$
\hat {\boldsymbol {x}} _ {0 \mid t} := \mathbb {E} _ {p \left(\boldsymbol {x} _ {0} \mid \boldsymbol {x} _ {t}\right)} \left[ \boldsymbol {x} _ {0} \mid \boldsymbol {x} _ {t} \right] = \frac {1}{\alpha_ {t}} \left(\boldsymbol {x} _ {t} + \sigma_ {t} ^ {2} \nabla_ {\boldsymbol {x} _ {t}} \log p (\boldsymbol {x} _ {t})\right) \simeq \frac {1}{\alpha_ {t}} \left(\boldsymbol {x} _ {t} + \sigma_ {t} ^ {2} \boldsymbol {s} _ {\theta^ {*}} ^ {(t)} (\boldsymbol {x} _ {t})\right). \tag {4}
$$
In practice, DDPM/DDIM solvers [15, 35] work by iteratively refining these denoised estimates.
2.2 Diffusion model-based inverse problem solving with gradient guidance
Suppose now that we are given a measurement  $\pmb{y}$  obtained through some Gaussian linear measurement process  $\pmb{A}$ , where our goal is to sample from the posterior distribution  $p(\pmb{x}|\pmb{y})$ . Starting from the sampling process of running the reverse SDE/ODE to sample from the prior distribution, one can modify the score function to adapt it for posterior sampling [38, 5]. By Bayes rule,  $\nabla_{\pmb{x}_t}\log p(\pmb{x}_t|\pmb{y}) = \nabla_{\pmb{x}_t}\log p(\pmb{x}_t) + \nabla_{\pmb{x}_t}\log p(\pmb{y}|\pmb{x}_t)$ , where  $\nabla_{\pmb{x}_t}\log p(\pmb{x}_t) \simeq s_{\theta^*}(\pmb{x}_t)$ . However,  $\nabla_{\pmb{x}_t}\log p(\pmb{y}|\pmb{x}_t)$  is intractable. Several methods have been proposed to approximate this time-dependent likelihood, two of the most widely used being DPS [5] and IIGDM [36]. DPS proposes the following Jensen approximation<sup>1</sup>
$$
\nabla_ {\boldsymbol {x} _ {t}} \log p (\boldsymbol {y} | \boldsymbol {x} _ {t}) \stackrel {\text {(D P S)}} {\simeq} \nabla_ {\boldsymbol {x} _ {t}} \log p (\boldsymbol {y} | \hat {\boldsymbol {x}} _ {0 | t}) = \frac {\partial \hat {\boldsymbol {x}} _ {0 | t}}{\partial \boldsymbol {x} _ {t}} \frac {\partial \| \boldsymbol {A} \hat {\boldsymbol {x}} _ {0 | t} - \boldsymbol {y} \| _ {2} ^ {2}}{\partial \hat {\boldsymbol {x}} _ {0 | t}} = \underbrace {\frac {\partial \hat {\boldsymbol {x}} _ {0 | t}}{\partial \boldsymbol {x} _ {t}}} _ {\mathrm {J}} \underbrace {\boldsymbol {A} ^ {\top} (\boldsymbol {y} - \boldsymbol {A} \hat {\boldsymbol {x}} _ {0 | t})} _ {\mathrm {V}}, \tag {5}
$$
of which the chain rule is based on the denominator layout notation [41]. Here, we see that the gradient term can be represented as the Jacobian (J) vector (V) product (JVP). In the original implementation of DPS, the two terms are not computed separately, but computed directly as  $\nabla_{\boldsymbol{x}_t}\| \boldsymbol{y} - \boldsymbol{A}\hat{\boldsymbol{x}}_{0|t}\|_2^2$  where the whole term can be handled with backpropagation. By this choice, DPS can also handle non-linear operators when the gradients can be computed, e.g. phase retrieval, forward model given as a neural network. On the other hand, IIGDM proposes
$$
\nabla_ {\boldsymbol {x} _ {t}} \log p (\boldsymbol {y} | \boldsymbol {x} _ {t}) \stackrel {(\Pi \mathrm {G D M})} {\simeq} \mathcal {N} (\boldsymbol {A} \hat {\boldsymbol {x}} _ {0 | t}, \boldsymbol {A} \boldsymbol {A} ^ {\top} + \boldsymbol {I}) = \underbrace {\frac {\partial \hat {\boldsymbol {x}} _ {0 | t}}{\partial \boldsymbol {x} _ {t}}} _ {\mathrm {J}} \underbrace {\boldsymbol {A} ^ {\dagger} (\boldsymbol {y} - \boldsymbol {A} \hat {\boldsymbol {x}} _ {0 | t})} _ {\mathrm {V}}, \tag {6}
$$
where  $\mathbf{A}^{\dagger} \coloneqq \mathbf{A}^{\top}(\mathbf{A}\mathbf{A}^{\top})^{-1}$  is the Moore-Penrose pseudo-inverse. Using the JVP for implementation, it is no longer required that the whole term is differentiable. For this reason, IIGDM can be applied to cases where we have non-differentiable, non-linear measurements given that an operation analogous to pseudo-inverse can be derived, e.g. JPEG restoration. Notably, the update step of DPS can be achieved by simply pre-conditioning IIGDM with  $\mathbf{A}^{\top}\mathbf{A}$ . Implementing DIS with DPS (5) or IIGDM (6) amounts to augmenting the gradient descent steps in between the ancestral sampling iterations.
While these methods are effective and outperforms the prior projection-based approaches [38, 21], they also have several drawbacks. Namely, the incorporation of the U-Net Jacobian is slow, compute-heavy, and often unstable [12, 34]. For example, when applied to MRI reconstruction in medical imaging, DPS results in noisy reconstructions [6] possibly due to unstable incorporation of the Wirtinger derivatives [23], and IIGDM is hard to use as it is non-trivial to compute  $A^{\dagger}$ . In order to circumvent these issues, DDS [6] proposed to use numerical optimization (i.e. conjugate gradients; CG) in the clean image domain, bypassing the need to compute J. Consequently, DDS achieves fast and stable reconstructions for inverse problems in medical imaging.
I2SB [26]	InDI [9]
Definition
Motivation	Schrödinger bridge	Small-step MMSE
Base process	βt = {βmin + 2βdt, t ∈ [0,0.5)
2βmax - 2βdt, t ∈ [0.5,1.0]	-
γt = ∫0tβτdτ, γt = ∫t1βτdτ	-
Linear symmetric	Const
Diffusion process
αt	γt2/(γt2+ γt2)	t
σt2	γt2γt2/(γt2+ γt2)	t2εt2
Sampling
αs|t2	γs2/γt2	s/t
σs|t2	(γt2-γs2)γs2/γt2	s2(εs2-εt2)
Table 1: Comparison between different types of DDB.  $\beta_{\mathrm{d}} := \beta_{\mathrm{max}} - \beta_{\mathrm{min}}$ . Further details are given in Appendix B.
3 Main Contributions
3.1 Direct Diffusion Bridge
We consider the case where we can sample  $\pmb{x}_0 \coloneqq \pmb{x} \sim p(\pmb{x})$ , and  $\pmb{x}_1 \coloneqq \pmb{y} \sim p(\pmb{y}|\pmb{x})^2$ , i.e. paired data for training. Adopting the formulation of I²SB [26] we define the posterior of  $\pmb{x}_t$  to be the product of Gaussians  $\mathcal{N}(\pmb{x}_t; \pmb{x}_0, \gamma_t^2)$  and  $\mathcal{N}(\pmb{x}_t; \pmb{x}_1, \bar{\gamma}_t^2)$ , such that
$$
p \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {0}, \boldsymbol {x} _ {1}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t}; \frac {\bar {\gamma} _ {t} ^ {2}}{\gamma_ {t} ^ {2} + \bar {\gamma} _ {t} ^ {2}} \boldsymbol {x} _ {0} + \frac {\gamma_ {t} ^ {2}}{\gamma_ {t} ^ {2} + \bar {\gamma} _ {t} ^ {2}} \boldsymbol {x} _ {1}, \frac {\gamma_ {t} ^ {2} \bar {\gamma} _ {t} ^ {2}}{\gamma_ {t} ^ {2} + \bar {\gamma} _ {t} ^ {2}} \boldsymbol {I}\right). \tag {7}
$$
Note that the sampling of  $x_{t}$  from (7) can be done by the reparametrization trick
$$
\boldsymbol {x} _ {t} = \left(1 - \alpha_ {t}\right) \boldsymbol {x} _ {0} + \alpha_ {t} \boldsymbol {x} _ {1} + \sigma_ {t} \boldsymbol {z}, \boldsymbol {z} \sim \mathcal {N} (0, I), \tag {8}
$$
where  $\alpha_{t} := \frac{\gamma_{t}^{2}}{\gamma_{t}^{2} + \bar{\gamma}_{t}^{2}}$ ,  $\sigma_{t}^{2} := \frac{\gamma_{t}^{2}\bar{\gamma}_{t}^{2}}{\gamma_{t}^{2} + \bar{\gamma}_{t}^{2}}$ . This diffusion bridge introduces a continual degradation process by taking a convex combination of  $(x_0, x_1)$ , starting from the clean image at  $t = 0$  to maximal degradation at  $t = 1$ , with additional stochasticity induced by the noise component  $\sigma_{t}$ . Our goal is to train a time-dependent neural network that maps any  $x_{t}$  to  $x_0$  that recovers the clean image. The training objective for  $\mathrm{I}^2\mathrm{SB}$  [26] analogous to denoising score matching (DSM) [16] reads
$$
\min  _ {\theta} \mathbb {E} _ {\boldsymbol {y} \sim p (\boldsymbol {y} | \boldsymbol {x}), \boldsymbol {x} \sim p (\boldsymbol {x}), t \sim U (0, 1)} \left[ \| \boldsymbol {s} _ {\theta} (\boldsymbol {x} _ {t}) - \frac {\boldsymbol {x} _ {t} - \boldsymbol {x} _ {0}}{\gamma_ {t}} \| _ {2} ^ {2} \right], \tag {9}
$$
which is also equivalent to training a residual network  $G_{\theta}$  with  $\min_{\theta}\mathbb{E}[\| G_{\theta}(\pmb{x}_t) - \pmb{x}_0\| _2^2 ]$ . For brevity, we simply denote the trained networks as  $G_{\theta^{*}}$  even if it is parametrized otherwise. Once the network is trained, we can reconstruct  $x_0$  starting from  $x_{1}$  by, for example, using DDPM ancestral sampling [15], where the posterior for  $s < t$  reads
$$
p \left(\boldsymbol {x} _ {s} \mid \boldsymbol {x} _ {0}, \boldsymbol {x} _ {t}\right) = \mathcal {N} \left(\boldsymbol {x} _ {s}; \left(1 - \alpha_ {s \mid t} ^ {2}\right) \boldsymbol {x} _ {0} + \alpha_ {s \mid t} ^ {2} \boldsymbol {x} _ {t}, \sigma_ {s \mid t} ^ {2} \boldsymbol {I}\right), \tag {10}
$$
with  $\alpha_{s|t}^2 \coloneqq \frac{\gamma_s^2}{\gamma_t^2}$ ,  $\sigma_{s|t}^2 \coloneqq \frac{(\gamma_t^2 - \gamma_s^2)\gamma_s^2}{\gamma_t^2}$ . At inference,  $\pmb{x}_0$  is replaced with a neural network-estimated  $\hat{\pmb{x}}_{0|t}$  to yield  $\pmb{x}_s \sim p(\pmb{x}_s|\hat{\pmb{x}}_{0|t}, \pmb{x}_t)$ .
However, when the motivation is to introduce 1) a tractable training objective that learns to recover the clean image along the degradation trajectory, and 2) devise a sampling method to gradually revert the degradation process, we find that the choices made for the parameters in (8),(7) can be arbitrary, as long as the marginal can be retrieved in the sampling process (10). In Table 1, we summarize the
<sup>1</sup>We ignore scaling constants that are related to the measurement noise for simplicity. For implementation, this can be absorbed into the choice of step sizes.
For cases where there is a dimensionality mismatch, we use  $\pmb{x}_1 = \pmb{A}^\dagger \pmb{y}$ . We keep this notation for simplicity. Hereafter, we override the definition of signal, noise coefficients  $\alpha_{t},\sigma_{t}$  that was first defined in Eq. (1).
Algorithm 1 CDBD	Algorithm 2 CDBD (deep)
Require: Gθ*, x1, αi, σi, α2i-1|i, σ2i-1|i, ρi	Require: Gθ*, x1, αi, σi, α2i-1|i, σ2i-1|i, ρi
1: for i = N - 1 to 0 do	1: for i = N - 1 to 0 do
2: x0|i← Gθ*(xi)	2: x0|i← Gθ*(xi)
3: z ~ N(0, I)	3: z ~ N(0, I)
4: xi-1← (1 - α2i-1|i) x0|i + α2i-1|i xi + σi-1|i z	4: xi-1← (1 - α2i-1|i) x0|i + α2i-1|i xi + σi-1|i z
5: g ← A^T(y - A x0|i)	5: g ← ∂x0|i/∂xi A^†(y - A x0|i)
6: xi-1← x' i-1 + ρi-1 g	6: xi-1← x' i-1 + ρi-1 g
7: end for	7: end for
8: return x0	8: return x0
choices made in [9, 26] to emphasize that the difference stems mostly from the parameter choices and not something fundamental. Concretely, sampling  $\boldsymbol{x}_t$  from paired data can always be represented as (8): a convex combination of  $\boldsymbol{x}_0$  and  $\boldsymbol{x}_1$  with some additional noise. Reverse diffusion at inference can be represented as (10): a convex combination of  $\boldsymbol{x}_0$  and  $\boldsymbol{x}_t$  with some stochasticity. We define the methods that belong to this category as Direct Diffusion Bridge (DDB) henceforth. Below, we formally state the equivalence between the algorithms, with proofs given in Appendix A.
Theorem 1. Let the parameters of InDI [9] in Table 1 be  $t \coloneqq \frac{\gamma_t^2}{\gamma_t^2 + \bar{\gamma}_t^2}$ ,  $\epsilon_t^2 \coloneqq \frac{\bar{\gamma}_t^2}{\gamma_t^2} (\gamma_t^2 + \bar{\gamma}_t^2)$ . Then, InDI and  $I^2 SB$  are equivalent.
The equivalence relation will be useful when we derive our CDDB algorithm in Section 3.2. As a final note, IR-SDE [28] does not strictly fall into this category as the sampling process is derived from running the reverse SDE. However, the diffusion process can still be represented as (8) by setting  $\alpha_{t} = 1 - e^{-\bar{\theta}_{t}}$ ,  $\sigma_t^2 = \lambda^2 (1 - e^{-2\bar{\theta}_t})$ , and the only difference comes from the sampling procedure.
3.2 Data Consistent Direct Diffusion Bridge
Motivation Regardless of the choice in constructing DDB, there is a crucial component that is missing from the framework. While the sampling process (10) starts directly from the measurement (or equivalent), as the predictions  $\hat{\pmb{x}}_{0|t} = G_{\theta}(\pmb{x}_t)$  are imperfect and are never guaranteed to preserve the measurement condition  $\pmb{y} = \pmb{A}\pmb{x}$ , the trajectory can easily deviate from the desired path, while the residual blows up. Consequently, this may result in inferior sample quality, especially in terms of distortion. In order to mitigate this downside, our strategy is to keep the DDB sampling strategy (10) intact and augment the steps to constantly guide the trajectory to satisfy the data consistency, similar in spirit to gradient guidance in DIS. Here, we focus on the fact that the clean image estimates  $\hat{\pmb{x}}_{0|t}$  is produced at every iteration, which can be used to compute the residual with respect to the measurement  $\pmb{y}$ . Taking a gradient step that minimizes this residual after every sampling step results in Algorithm 1, which we name data Consistent DDB (CDDB). In the following, we elaborate on how the proposed method generalizes DDS which was developed for DIS.
CDDB as a generalization of DDS [6] Rewriting (10) with reparameterization trick
$$
\boldsymbol {x} _ {s} = \underbrace {\hat {\boldsymbol {x}} _ {0 | t}} _ {\text {D e n o i s e} (\boldsymbol {x} _ {t})} + \underbrace {\alpha_ {s | t} ^ {2} \left(\boldsymbol {x} _ {t} - \hat {\boldsymbol {x}} _ {0 | t}\right)} _ {\text {N o i s e} (\boldsymbol {x} _ {t})} + \sigma_ {s | t} \boldsymbol {z}, \quad \hat {\boldsymbol {x}} _ {0 | t} := G _ {\theta^ {*}} \left(\boldsymbol {x} _ {t}\right) \tag {11}
$$
we see that the iteration decomposes into three terms: the denoised component, the deterministic noise, and the stochastic noise. The key observation of DDIM [35] is that if the score network is fully expressive, then the deterministic noise term  $\pmb{x}_t - \hat{\pmb{x}}_{0|t}$  becomes Gaussian such that it satisfies the total variance condition
$$
\left(\alpha_ {s \mid t} ^ {2} \sigma_ {t}\right) ^ {2} + \sigma_ {s \mid t} ^ {2} = \sigma_ {s} ^ {2}, \tag {12}
$$
allowing (11) to restore the correct marginal  $\mathcal{N}(\pmb{x}_s;\pmb{x}_0,\sigma_s^2)$ . Under this condition, DDS showed that using a few step of numerical optimization ensure the updates from the denoised image  $\hat{\pmb{x}}_{0|t}$  remain on the clean manifold. Furthermore, subsequent noising process using deterministic and stochastic noises can then be used to ensure the transition to the correct noisy manifold [6].
Under this view, our algorithm can be written concisely as
$$
\boldsymbol {x} _ {s} \leftarrow \underbrace {\hat {\boldsymbol {x}} _ {0 | t} + \rho \boldsymbol {A} ^ {\top} (\boldsymbol {y} - \boldsymbol {A} \hat {\boldsymbol {x}} _ {0 | t})} _ {\operatorname {C D e n o i s e} \left(\boldsymbol {x} _ {t}\right)} + \underbrace {\alpha_ {s | t} ^ {2} \left(\boldsymbol {x} _ {t} - \hat {\boldsymbol {x}} _ {0 | t}\right)} _ {\operatorname {N o i s e} \left(\boldsymbol {x} _ {t}\right)} + \sigma_ {s | t} z, \tag {13}
$$
where we make the update step only to the clean denoised component, and leave the other components as is. In order to achieve proper sampling that obeys the marginals, it is important to show that the remaining components constitute the correct noise variance and the condition assuming Gaussianity should be (12). In the following, we show that this is indeed satisfied for the two cases of direction diffusion bridge (DDB):
Theorem 2. The total variance condition (12) is satisfied for both  $I^2 SB$  and InDI.
Proof. For InDI, considering the noise variance  $\sigma_t = t\epsilon_t$  in Table 1,
$$
\left(\alpha_ {s \mid t} ^ {2} \sigma_ {t}\right) ^ {2} + \sigma_ {s \mid t} ^ {2} = \frac {s ^ {2}}{t ^ {2}} t ^ {2} \epsilon_ {t} ^ {2} + s ^ {2} \left(\epsilon_ {s} ^ {2} - \epsilon_ {t} ^ {2}\right) = s ^ {2} \epsilon_ {s} ^ {2} = \sigma_ {s} ^ {2}. \tag {14}
$$
Due to the equivalence in Theorem 1, the condition is automatically satisfied in  $\mathrm{I}^2\mathrm{SB}$ . We show that this is indeed the case in Appendix A.