Step-wise Triple-Consistent Diffusion Sampling for Inverse Problems

## Abstract

Abstract Diffusion models (DMs) are a class of generative models that allow sampling from a distribution learned over a training set. When applied to solving inverse imaging problems (IPs), the reverse sampling steps of DMs are typically modified to approximately sample from a measurement-conditioned distribution in the image space. However, these modifications may be unsuitable for certain settings (such as in the presence of measurement noise) and non-linear tasks, as they often struggle to correct errors from earlier sampling steps and generally require a large number of optimization and/or sampling steps. To address these challenges, we state three conditions for achieving measurement-consistent diffusion trajectories. Building on these conditions, we propose a new optimization-based sampling method that not only enforces the standard data manifold measurement consistency and forward diffusion consistency, as seen in previous studies, but also incorporates backward diffusion consistency that maintains a diffusion trajectory by optimizing over the input of the pre-trained model at every sampling step. By enforcing these conditions, either implicitly or explicitly, our sampler requires significantly fewer reverse steps. Therefore, we refer to our accelerated method as S tep-w i se T riple- Co nsistent Sa m pling ( SITCOM ). Compared to existing state-of-the-art baseline methods, under different levels of measurement noise, our extensive experiments across five linear and three non-linear image restoration tasks demonstrate that SITCOM achieves competitive or superior results in terms of standard image similarity metrics while requiring a significantly reduced run-time across all considered tasks.

## 1 Introduction

Inverse problems (IPs) arise in a wide range of science and engineering applications, including computer vision [], signal processing [], medical imaging [], remote sensing [], and geophysics []. In these applications, the primary goal is to recover an unknown image or signal $\mathbf{x}\in\mathbb{R}^{n}$ from measurements or degraded image $\mathbf{y}\in\mathbb{R}^{m}$, which are often corrupted by noise. Mathematically, the unknown signal and the measurements are related as

$$ $\mathbf{y}=\mathcal{A}(\mathbf{x})+\mathbf{n}\>,$ (1) $$

where $\mathcal{A}(\cdot):\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ (with $m\leq n$) represents the linear or non-linear forward operator that models the measurement process, and $\mathbf{n}\in\mathbb{R}^{m}$ denotes the noise in the measurement domain, e.g., assumed sampled from a Gaussian distribution $\mathcal{N}(\mathbf{0},\sigma^{2}_{\mathbf{y}}\mathbf{I})$, where $\sigma_{\mathbf{y}}>0$ denotes the noise level. Exactly solving these inverse problems is challenging due to their ill-posedness in many settings, requiring advanced techniques to achieve accurate solutions.

Deep learning techniques have recently been utilized as a prior to aid in solving these problems [, ]. One framework that has shown significant potential is the use of generative models, particularly diffusion models (DMs) []. Given a training dataset, DMs are trained to learn the underlying distribution $p(\mathbf{x})$. During inference, DMs enable sampling from this learned distribution through an iterative procedure []. When employed to solving inverse problems, DM-based IP solvers often modify the reverse sampling steps to allow sampling from the measurements-conditioned distribution $p(\mathbf{x}|\mathbf{y})$ [, ]. These modifications typically rely on approximations that may not be suitable for all tasks and settings, and in addition to generally requiring many sampling iterations, often suffer from errors accumulated during early diffusion sampling steps []. In most DM-based IP solvers, these approximations are designed to enforce standard measurement consistency on the estimated image (or posterior mean) at every reverse sampling iteration, as in [], and may also include resampling using the forward diffusion process (which we refer to as forward diffusion consistency), such as in [, ].

A key bottleneck in DMs is their computational speed, as they are slower than other generative models due to the large number of sampling steps. Although various methods have been proposed to reduce sampling frequency (e.g., []), these improvements have yet to be fully realized for DMs applied to IPs. Most existing methods still require dense sampling, which continues to pose speed challenges.

#### Contributions:

In this paper, we: (i) identify key issues in accelerating DMs for IPs, (ii) propose three conditions that could fully leverage the information from the measurements and the pre-trained diffusion model to effectively address these issues, and (iii) present a new optimization-based method in the pixel space that satisfies these conditions. We refer to our accelerated sampling method as Step-wise Triple-Consistent Sampling (SITCOM). We evaluate our method on several image restoration tasks: Super Resolution, Box In-painting, Random In-painting, Motion Deblurring, Gaussian Deblurring, Non-linear Deblurring, High Dynamic Range, and Phase Retrieval. Compared to leading baselines, our approach consistently achieves either state-of-the-art or highly competitive quantitative results, while also reducing the number of sampling steps and, consequently, the computational time. See Figure 1 for examples.

Figure: Figure 1: Qualitative results on the FFHQ dataset on two linear tasks (top) and two non-linear tasks (bottom) under measurement noise of $\sigma_{\mathbf{y}}=0.05$. The PSNR and LPIPS values are given below each restored image. Zoomed-in regions show how SITCOM captures greater image details when compared to two general (non)linear DM-based methods (DPS [] and DAPS []).
Refer to caption: x1.png

## 2 Background: Diffusion Models & Their Usage in Solving IPs

Pre-trained Diffusion Models (DMs) generate images by applying a pre-defined iterative denoising process []. In the Variance-Preserving Stochastic Differentiable Equations (SDEs) setting [, ], DMs are formulated using the forward and reverse processes

$$ $d\mathbf{x}_{t}=-\frac{\beta_{t}}{2}\mathbf{x}_{t}dt+\sqrt{\beta_{t}}d\mathbf{ w}\>,~{}~{}~{}~{}~{}d\mathbf{x}_{t}=-\beta_{t}\Big{[}\frac{1}{2}\mathbf{x}_{t} +\nabla_{\mathbf{x}_{t}}\log p_{t}(\mathbf{x}_{t})\Big{]}dt+\sqrt{\beta_{t}}d \bar{\mathbf{w}}\>,$ (2) $$

where $\beta:\{0,\dots,T\}\rightarrow(0,1)$ is a pre-defined function that controls the amount of additive perturbations at time $t$, $\mathbf{w}$ (resp. $\bar{\mathbf{w}}$) is the forward (resp. reverse) Weiner process [], $p_{t}(\mathbf{x}_{t})$ is the distribution of $\mathbf{x}_{t}$ at $t$, and $\nabla_{\mathbf{x}_{t}}\log p_{t}(\mathbf{x}_{t})$ is the score function that is replaced by a neural network (typically a time-encoded U-Net []) $\bm{s}:\mathbb{R}^{n}\times\{0,\dots,T\}\rightarrow\mathbb{R}^{n}$, parameterized by $\theta$. In practice, given the score function $\bm{s}_{\theta}$, the SDEs in (2) can be discretized as in (3) where $\bm{\eta}_{t},\bm{\eta}_{t-1}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

$$ $\mathbf{x}_{t}=\sqrt{1-\beta_{t}}\mathbf{x}_{t-1}+\sqrt{\beta_{t}}\bm{\eta}_{t -1}\>,~{}~{}~{}~{}~{}\mathbf{x}_{t-1}=\frac{1}{\sqrt{1-\beta_{t}}}\Big{[} \mathbf{x}_{t}+\beta_{t}\bm{s}_{\theta}(\mathbf{x}_{t},t)\Big{]}+\sqrt{\beta_{ t}}\bm{\eta}_{t}\>.$ (3) $$

When employed to solve inverse problems, the score function in (2) is replaced by a conditional score function which, by Bayes’ rule, is $\nabla_{\mathbf{x}_{t}}\log p_{t}(\mathbf{x}_{t}|\mathbf{y})=\nabla_{\mathbf{x
}_{t}}\log p_{t}(\mathbf{x}_{t})+\nabla_{\mathbf{x}_{t}}\log p_{t}(\mathbf{y}|
\mathbf{x}_{t})$. Solving the SDE in (2) with the conditional score is referred to as posterior sampling []. As there doesn’t exist a closed-form expression for the term $\nabla_{\mathbf{x}_{t}}\log p_{t}(\mathbf{y}|\mathbf{x}_{t})$ (which is termed as the measurements matching term in []), previous works have explored different approaches, which we will briefly discuss below. We refer the reader to the recent survey in [] for an overview on DM-based methods for solving IPs.

A well-known method is Diffusion Posterior Sampling (DPS) [], which uses the approximation $p(\mathbf{y}|\mathbf{x}_{t})\approx p(\mathbf{y}|\hat{\mathbf{x}}_{0})$ where $\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})$ (or simply $\hat{\mathbf{x}}_{0}$) is the estimated image at time $t$ as a function of the pre-trained model and $\mathbf{x}_{t}$ (Tweedie’s formula []), given as

$$ $\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\Big{[} \mathbf{x}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta}(\mathbf{x}_{t}, t)\Big{]}=:f(\mathbf{x}_{t};t,\bm{\epsilon}_{\theta})\>,$ (4) $$

where $\bar{\alpha}_{t}=\prod^{t}_{j=1}\alpha_{j}$ and $\alpha_{t}=1-\beta_{t}$. We call the function $f$, defined in (4), as ‘Tweedie-network denoiser’ (also termed as ‘posterior mean predictor’ in []). Here, $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)=-\sqrt{1-\bar{\alpha}_{t}}\bm{s}_{
\theta}(\mathbf{x}_{t},t)$ [] outputs the noise in $\mathbf{x}_{t}$. Tweedie’s formula, like in our method, is also adopted in other DM-based IP solvers such as [, , ]. The drawback of these methods is that they require a large number of sampling steps.

The work in ReSample [], solves an optimization problem on the estimated posterior mean in the latent space for many steps to enforce measurement consistency, requiring many sampling and optimization steps.

The work in [] introduced RED-Diff, a variational Bayesian method that fits a Gaussian distribution to the posterior distribution of the clean image conditional on the measurements. This approach involves solving an optimization problem using stochastic gradient descent (SGD) to minimize a data-fitting term while maximizing the likelihood of the reconstructed image under the denoising diffusion prior (as a regularizer). However, the SGD process requires multiple iterations, each involving evaluations of the pre-trained DM on a different noisy image at some randomly selected time, making it quite computationally expensive.

Recently, Decoupling Consistency with Diffusion Purification (DCDP) [] proposed separating diffusion sampling steps from measurement consistency by using DMs as diffusion purifiers [, ], with the goal of reducing the run-time. However, DCDP requires tuning the number of forward diffusion steps for purification. Shortly after, Decoupled Annealing Posterior Sampling (DAPS) [] introduced another decoupled approach, incorporating gradient descent noise annealing via Langevin dynamics. DAPS, similar to DPS and RED-Diff, also requires a large number of sampling and optimization steps. Under measurement noise, DCDP achieves SOTA run-time across various linear restoration tasks, while DAPS sets the SOTA in restoration quality. Both will serve as primary baselines in our experiments.

## 3 SITCOM: Step-wise Triple-Consistent Sampling

### 3.1 Motivation: Addressing the Challenges in Applying DMs to IPs

Most inverse problems are ill-conditioned and undersampled. DMs, when trained on a dataset that closely resembles the target image, can provide critical information to alleviate ill-conditioning and improve recovery. Despite various previous efforts, a key challenge remains: How to efficiently integrate DMs into the framework of inverse problems? We will now elaborate on this challenge in detail.

The standard reverse sampling procedure in DMs consists of applying the backward discrete steps in (3) for $t\in\{T,T-1,\dots,1\}$, forming the standard diffusion trajectory for which $\mathbf{x}_{0}$ is the generated image. To incorporate the measurement $\mathbf{y}$ into these steps, a common approach adopted in previous works that demonstrate superior performance (e.g., [, , ]) is to the $\hat{\mathbf{x}}_{0}$ computed via (4) as follows:

$$ $\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})=\arg\min_{\mathbf{x}}\|\mathcal{ A}(\mathbf{x})-\mathbf{y}\|^{2}+\lambda\|\mathbf{x}-\hat{\mathbf{x}}_{0}( \mathbf{x}_{t})\|^{2}\>,$ (5) $$

where $\lambda\in\mathbb{R}_{+}$ is a regularization parameter. The $\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})$ obtained from (5) is close to $\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})$ while also remaining consistent with the measurements. When using $\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})$ to sample $\mathbf{x}_{t-1}$, the second formula in (3) can be rewritten as in (6), where the derivation is provided in Appendix A.

$$ $\mathbf{x}_{t-1}=\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha} _{t}}\mathbf{x}_{t}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{ t}}\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})+\sqrt{\beta_{t}}\bm{\eta}_{t}\>.$ (6) $$

By substituting $\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})$ into (6) with the measurement-consistent $\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})$, the modified sampling formula becomes:

$$ $\mathbf{x}_{t-1}=\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha} _{t}}\mathbf{x}_{t}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{ t}}\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})+\sqrt{\beta_{t}}\bm{\eta}_{t}\>.$ (7) $$

While this approach effectively ensures data consistency at each step, it inevitably causes $\hat{\mathbf{x}}^{\prime}_{0}$ to deviate from the diffusion trajectory(^1^11Diffusion trajectory refers to the path that leads to an in-distribution image, where the distribution is the one learned by the DM from the training set.), leading to two major issues:

- (I1)
The image $\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})$, initially constructed through Tweedie’s formula, usually appears quite natural (e.g., columns 3 to 5 of Figure 2 ); however, the modified version, $\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})$, is likely to exhibit severe artifacts (e.g., columns 6 to 8 of Figure 2).
- (I2)
Since the DM network, $\bm{\epsilon}_{\theta}$, is trained via minimizing the objective function $\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon}}\|\epsilon-\bm{\epsilon}_{\theta}(
\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},t
)\|^{2}$ (denoising score matching []) on a finite dataset, it performs best on noisy images lying in the high-density regions of the *training distribution* $\mathcal{N}(\mathbf{x}_{t};\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0},(1-\bar{
\alpha}_{t})\mathbf{I})$, $\mathbf{x}_{0}\sim p(\mathbf{x}_{0})$. We define an algorithm as forward-consistent if it likely applies $\bm{\epsilon}_{\theta}$ only to in-distribution inputs (i.e., those from the same distribution used for training). For example, if the forward diffusion used to train $\bm{\epsilon}_{\theta}$ adds Gaussian noise, the in-distribution input to $\bm{\epsilon}_{\theta}$ should ideally be sampled from a Gaussian with specific parameters. If Poisson noise is used in the forward process, inputs drawn from suitable Poisson distributions are more likely to fall within the well-trained region of the network. In summary, forward consistency requires that inputs to $\bm{\epsilon}_{\theta}$ during sampling align with the forward process. While the $\mathbf{x}_{t-1}$ generated from (6) is forward-consistent by design, the one generated from the modified formula (7) is not. Therefore, in the latter case, the DM network, $\bm{\epsilon}_{\theta}$, may be applied to many out-of-distribution inputs, leading to degraded performance.

We pause to verify our claimed Issue (I1) through a box-inpainting experiment. Columns 3 to 5 of Figure 2 show $\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})$ at various $t$. The results clearly demonstrate successful enforcement of data consistency, as the region outside the box aligns with the original image. However, this enforcement compromises the natural appearance of the image, introducing significant artifacts in the reconstructed area inside the box. Details about the setting of the results in Figure 2 are given in Section C.

Issue (I2) was previously observed in [], which proposed a remedy known as ‘resampling’. In this approach, the sampling formula in (7) is replaced by

$$ $\mathbf{x}_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{\mathbf{x}}_{0}+\sqrt{1-\bar{ \alpha}_{t-1}}\bm{\eta}_{t}\>.$ (8) $$

Provided $\hat{\mathbf{x}}_{0}$ is close to the ground truth $\mathbf{x}_{0}$, $\mathbf{x}_{t-1}$ generated this way will stay in-distribution with high probability. For a more detailed explanation of the rationale behind this remedy, we refer the reader to []. This method has since been adopted by subsequent works, such as [, ], and we will also employ it to address (I2).

Figure: Figure 2: Effects of enforcing backward-consistency in box-inpainting: Results of using Tweedie’s formula without measurement consistency (columns 3 to 5), enforcing measurement-consistency via (5) (columns 6 to 9), and enforcing both measurement-consistency and backward-consistency via (12) (columns 10 to 12) at different time steps $t^{\prime}$. Experimental details are given in Appendix C.
Refer to caption: x2.png

### 3.2 Network Regularization & Backward Diffusion Consistency

Previous studies, such as [, ], mitigate issue (I1) by using a large number of sampling steps, which inevitably increases the computational burden. In contrast, this paper proposes employing a *network regularization* to resolve issue (I1). This approach not only accelerates convergence but also enhances reconstruction quality. Let’s first clarify the underlying intuition.

It is widely observed that the U-Net architecture or trained transformers exhibit an effective image bias [, , , ]. From columns 3 to 5 of Figure 2, we observe that without enforcing data consistency, the reconstructed $\hat{\mathbf{x}}_{0}$, derived directly from Tweedie-network denoiser $f(\mathbf{x}_{t};t,\bm{\epsilon}_{\theta})$ for each time $t$, exhibits natural textures. This indicates that the reconstruction using the combination of Tweedie’s formula and the DM network has a natural regularizing effect on the image.

By definition, the output of $f(\mathbf{x}_{t};t,\bm{\epsilon}_{\theta})$ in (4) represents the *denoised* version of $\mathbf{x}_{t}$ at time $t$ using the Tweedie’s formula and the DM denoiser $\bm{\epsilon}_{\theta}$. Due to the implicit bias of $\bm{\epsilon}_{\theta}$, this denoised image tends to align with the clean image manifold, even if $\mathbf{x}_{t}$ does not correspond to a training image, as shown in columns 3 to 5 of Figure 2. We refer to this regularization effect of $f(\mathbf{x}_{t};t,\bm{\epsilon}_{\theta})$, which arises from network bias, as‘network regularization’.

By employing network regularization, we can address (I1) by ensuring that the data-consistent $\hat{\mathbf{x}}^{\prime}_{0}$ is also network-consistent. We refer the latter condition as Backward Consistency and define it formally as follows.

###### Definition 1 (Backward Consistency) .

We say an $\hat{\mathbf{x}}^{\prime}_{0}$ is backward-consistent with Tweedie’s formula and the DM neural network $\bm{\epsilon}_{\theta}$ at time $t$ if there exists some $\mathbf{v}_{t}$ such that $\hat{\mathbf{x}}^{\prime}_{0}=f(\mathbf{v}_{t};t,\bm{\epsilon}_{\theta}).$ In other words, backward consistency requires $\hat{\mathbf{x}}^{\prime}_{0}$ to be a ‘denoised version’ of some noisy image $\mathbf{v}_{t}$ via the Tweedie-network denoiser $f$ at time $t$.

The subset of images that are in the range of the function $f$ (i.e., backward-consistent) is denoted by $\mathcal{C}_{t}$ and defined as

$$ $\mathcal{C}_{t}:=\{f(\mathbf{v}_{t};t,\bm{\epsilon}_{\theta}):\mathbf{v}_{t} \in\mathbb{R}^{n}\}\>.$ (9) $$

Enforcing $\hat{\mathbf{x}}_{0}^{\prime}$ to be both
measurement- and backward-consistent involves solving the following optimization problem

$$ $\hat{\mathbf{x}}^{\prime}_{0},\hat{\mathbf{v}}_{t}:=\operatorname*{argmin}_{ \mathbf{v}^{\prime}_{t},\mathbf{x}^{\prime}_{0}}\Big{\{}\|\mathcal{A}\big{(}{ \mathbf{x}}^{\prime}_{0}\big{)}-\mathbf{y}\|^{2}_{2}~{}~{}~{}\text{subject to} ~{}~{}~{}{\mathbf{x}}^{\prime}_{0}=f(\mathbf{v}^{\prime}_{t};t,\bm{\epsilon}_{ \theta})\Big{\}}\>.$ (10) $$

However, (10) may violate forward consistency, as $\hat{\mathbf{v}}_{t}$ could possibly be far from $\mathbf{x}_{t}$. Therefore, we propose adding a regularization term, for which (10) becomes

$$ $\hat{\mathbf{x}}^{\prime}_{0},\hat{\mathbf{v}}_{t}:=\operatorname*{argmin}_{ \mathbf{v}^{\prime}_{t},\mathbf{x}^{\prime}_{0}}\Big{\{}\|\mathcal{A}\big{(}{ \mathbf{x}}^{\prime}_{0}\big{)}-\mathbf{y}\|^{2}_{2}+\lambda\|\mathbf{x}_{t}- \mathbf{v}^{\prime}_{t}\|_{2}^{2}~{}~{}~{}\text{subject to}~{}~{}~{}{\mathbf{x }}^{\prime}_{0}=f(\mathbf{v}^{\prime}_{t};t,\bm{\epsilon}_{\theta})\Big{\}}\>.$ (11) $$

During the reverse sampling process, at each time $t$, with the given $\mathbf{x}_{t}$, we seek a $\mathbf{v}_{t}^{\prime}$ in the nearby region (i.e., $\|\mathbf{x}_{t}-\mathbf{v}^{\prime}_{t}\|$ is small), such that $\mathbf{v}_{t}^{\prime}$ can be denoised by $f$ to produce a clean image $\mathbf{x}^{\prime}_{0}$ (i.e., ${\mathbf{x}}^{\prime}_{0}=f(\mathbf{v}^{\prime}_{t};t,\bm{\epsilon}_{\theta})$), which is also consistent with the measurements $\mathbf{y}$ (i.e., $\|\mathcal{A}\big{(}{\mathbf{x}}^{\prime}_{0}\big{)}-\mathbf{y}\|^{2}_{2}$ is small). We need to identify such a $\mathbf{v}_{t}^{\prime}$ because $\mathbf{x}_{t}$ itself cannot be directly denoised by $f$ to yield an image consistent with the measurements. By substituting the constraint into the objective function, the optimization problem in (11) is reduced to

$$ $\hat{\mathbf{v}}_{t}:=\operatorname*{argmin}_{\mathbf{v}^{\prime}_{t}}\Big{\{} \|\mathcal{A}\big{(}f(\mathbf{v}^{\prime}_{t};t,\bm{\epsilon}_{\theta})\big{)} -\mathbf{y}\|^{2}_{2}+\lambda\|\mathbf{x}_{t}-\mathbf{v}^{\prime}_{t}\|_{2}^{2 }~{}\Big{\}},\quad\hat{\mathbf{x}}^{\prime}_{0}=f(\hat{\mathbf{v}}_{t};t,\bm{ \epsilon}_{\theta}).$ (12) $$

The benefit of the considered backward consistency constraint is shown in columns 6 to 8 of Figure 2. After obtaining $\hat{\mathbf{x}}^{\prime}_{0}$, the resampling formula in (8) is used to obtain $\mathbf{x}_{t-1}$.

###### Definition 1 (Backward Consistency) .

### 3.3 Triple Consistency Conditions

We now summarize the three key conditions that apply at each sampling step.

Measurement Consistency: The reconstruction $\hat{\mathbf{x}}^{\prime}_{0}$ is consistent with the measurements This means that $\mathcal{A}(\hat{\mathbf{x}}^{\prime}_{0})\approx\mathbf{y}$.

Backward Consistency: The reconstruction $\hat{\mathbf{x}}^{\prime}_{0}$ is a denoised image produced by the Tweedie-network denoiser $f$. More generally, we define the backward consistency to include any form of DM network regularization (e.g., using the DM probability-flow (PF) ODE []) applied to $\hat{\mathbf{x}}^{\prime}_{0}$.

Forward Consistency: The pre-trained DM network $\bm{\epsilon}_{\theta}$ is provided with in-distribution inputs with high probability. To ensure this, we apply the resampling formula in (8) and enforce that $\hat{\mathbf{v}}_{t}$ remains close to $\mathbf{x}_{t}$.

We emphasize that C1-C3 aim to ensure that all intermediate reconstructions $\hat{\mathbf{x}}^{\prime}_{0}(\mathbf{x}_{t})$ (with $t>0$) are as accurate as possible, allowing us to effectively reduce the number of sampling steps. If reducing sampling steps is not necessary, these conditions become less critical, as the final reconstruction at $t=0$ can still be accurate with a large number of sampling steps, even if the intermediate reconstructions are less precise.

Previous works, such as [, ], enforce measurement consistency by applying $\mathcal{A}(\hat{\mathbf{x}}_{0})=\mathbf{y}$ exactly, whereas DPS [] does not ensure consistency along the diffusion trajectory.

### 3.4 The Proposed Sampler

Given $\mathbf{x}_{t}$, $\bm{\epsilon}_{\theta}$, and towards satisfying the above conditions, our method, at sampling time $t$, consists of the following three steps:

$$ $\displaystyle\begin{aligned} \leavevmode\hbox to64.22pt{\vbox to16.52pt{ \pgfpicture\makeatletter\hbox{\hskip 32.10919pt\lower-8.26118pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}\hbox{\hbox{{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{} {{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{30.10919pt}{8.26118pt}\pgfsys@lineto{-30.10919pt}{8.26118pt} \pgfsys@curveto{-31.21378pt}{8.26118pt}{-32.10919pt}{7.36577pt}{-32.10919pt}{6 .26118pt}\pgfsys@lineto{-32.10919pt}{-6.26118pt}\pgfsys@curveto{-32.10919pt}{- 7.36577pt}{-31.21378pt}{-8.26118pt}{-30.10919pt}{-8.26118pt}\pgfsys@lineto{30. 10919pt}{-8.26118pt}\pgfsys@curveto{31.21378pt}{-8.26118pt}{32.10919pt}{-7.365 77pt}{32.10919pt}{-6.26118pt}\pgfsys@lineto{32.10919pt}{6.26118pt} \pgfsys@curveto{32.10919pt}{7.36577pt}{31.21378pt}{8.26118pt}{30.10919pt}{8.26 118pt}\pgfsys@closepath\pgfsys@moveto{-32.10919pt}{-8.26118pt}\pgfsys@fill \pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-28.77618pt}{-2.29405 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$\hat{\mathbf{v}}_{t}:=\operatorname*{argmin}_{ \mathbf{v}^{\prime}_{t}}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}&\;\leavevmode\hbox to122.31pt{\vbox to27.75 pt{\pgfpicture\makeatletter\hbox{\hskip 61.15555pt\lower-13.87553pt\hbox to0.0 pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}\hbox{\hbox{{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 1.0,1.0,0.8}\pgfsys@color@rgb@fill{1.0}{1.0}{0.8}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 1.0,1.0,0.8}\pgfsys@color@rgb@fill{1.0}{1.0}{0.8}\pgfsys@invoke{ }{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{} }}{}{}\pgfsys@moveto{59.15555pt}{13.87553pt}\pgfsys@lineto{-59.15555pt}{13.875 53pt}\pgfsys@curveto{-60.26013pt}{13.87553pt}{-61.15555pt}{12.98012pt}{-61.155 55pt}{11.87553pt}\pgfsys@lineto{-61.15555pt}{-11.87553pt}\pgfsys@curveto{-61.1 5555pt}{-12.98012pt}{-60.26013pt}{-13.87553pt}{-59.15555pt}{-13.87553pt} \pgfsys@lineto{59.15555pt}{-13.87553pt}\pgfsys@curveto{60.26013pt}{-13.87553pt }{61.15555pt}{-12.98012pt}{61.15555pt}{-11.87553pt}\pgfsys@lineto{61.15555pt}{ 11.87553pt}\pgfsys@curveto{61.15555pt}{12.98012pt}{60.26013pt}{13.87553pt}{59. 15555pt}{13.87553pt}\pgfsys@closepath\pgfsys@moveto{-61.15555pt}{-13.87553pt} \pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-57.82254pt}{-7.54253 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$\|\mathcal{A}\big{(}\leavevmode\hbox to77.4pt{ \vbox to18.09pt{\pgfpicture\makeatletter\hbox{\hskip 38.70029pt\lower-9.04253 pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{} {{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor} {rgb}{0.85,1,0.85}\pgfsys@color@rgb@fill{0.85}{1}{0.85}\pgfsys@invoke{ }{{}{}{ { {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{ 0.85,1,0.85}\pgfsys@color@rgb@fill{0.85}{1}{0.85}\pgfsys@invoke{ }{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{} }}{}{}\pgfsys@moveto{36.70029pt}{9.04253pt}\pgfsys@lineto{-36.70029pt}{9.04253 pt}\pgfsys@curveto{-37.80487pt}{9.04253pt}{-38.70029pt}{8.14711pt}{-38.70029pt }{7.04253pt}\pgfsys@lineto{-38.70029pt}{-7.04253pt}\pgfsys@curveto{-38.70029pt }{-8.14711pt}{-37.80487pt}{-9.04253pt}{-36.70029pt}{-9.04253pt}\pgfsys@lineto{ 36.70029pt}{-9.04253pt}\pgfsys@curveto{37.80487pt}{-9.04253pt}{38.70029pt}{-8. 14711pt}{38.70029pt}{-7.04253pt}\pgfsys@lineto{38.70029pt}{7.04253pt} \pgfsys@curveto{38.70029pt}{8.14711pt}{37.80487pt}{9.04253pt}{36.70029pt}{9.04 253pt}\pgfsys@closepath\pgfsys@moveto{-38.70029pt}{-9.04253pt}\pgfsys@fill \pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-35.36728pt}{-2.25555 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left[\mathbf{v}^ {\prime}_{t}-\sqrt{1-\bar{\alpha}_{t}}\,\bm{\epsilon}_{\theta}(\mathbf{v}^{ \prime}_{t},t)\right]$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\big{)}-\mathbf{y}\|^{2}_{2}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\leavevmode\hbox to14.44pt{\vbox to13.33pt{ \pgfpicture\makeatletter\hbox{\hskip 7.22191pt\lower-6.66634pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}\hbox{\hbox{{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgffillcolor}{rgb}{ 1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{} {{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{5.22191pt}{6.66634pt}\pgfsys@lineto{-5.22191pt}{6.66634pt} \pgfsys@curveto{-6.32649pt}{6.66634pt}{-7.22191pt}{5.77092pt}{-7.22191pt}{4.66 634pt}\pgfsys@lineto{-7.22191pt}{-4.66634pt}\pgfsys@curveto{-7.22191pt}{-5.770 92pt}{-6.32649pt}{-6.66634pt}{-5.22191pt}{-6.66634pt}\pgfsys@lineto{5.22191pt} {-6.66634pt}\pgfsys@curveto{6.32649pt}{-6.66634pt}{7.22191pt}{-5.77092pt}{7.22 191pt}{-4.66634pt}\pgfsys@lineto{7.22191pt}{4.66634pt}\pgfsys@curveto{7.22191 pt}{5.77092pt}{6.32649pt}{6.66634pt}{5.22191pt}{6.66634pt}\pgfsys@closepath \pgfsys@moveto{-7.22191pt}{-6.66634pt}\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-3.8889pt}{-2.5pt} \pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$+$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\leavevmode\hbox to44.97pt{\vbox to18.47pt{ \pgfpicture\makeatletter\hbox{\hskip 22.48636pt\lower-9.23523pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}\hbox{\hbox{{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{0.7,1,1} \pgfsys@color@cmyk@fill{0.3}{0}{0}{0}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{0.7,1,1} \pgfsys@color@cmyk@fill{0.3}{0}{0}{0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{20.48636pt}{9.23523pt}\pgfsys@lineto{-20.48636pt}{9.23523pt} \pgfsys@curveto{-21.59094pt}{9.23523pt}{-22.48636pt}{8.33981pt}{-22.48636pt}{7 .23523pt}\pgfsys@lineto{-22.48636pt}{-7.23523pt}\pgfsys@curveto{-22.48636pt}{- 8.33981pt}{-21.59094pt}{-9.23523pt}{-20.48636pt}{-9.23523pt}\pgfsys@lineto{20. 48636pt}{-9.23523pt}\pgfsys@curveto{21.59094pt}{-9.23523pt}{22.48636pt}{-8.339 81pt}{22.48636pt}{-7.23523pt}\pgfsys@lineto{22.48636pt}{7.23523pt} \pgfsys@curveto{22.48636pt}{8.33981pt}{21.59094pt}{9.23523pt}{20.48636pt}{9.23 523pt}\pgfsys@closepath\pgfsys@moveto{-22.48636pt}{-9.23523pt}\pgfsys@fill \pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-19.15335pt}{-3.40222 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$\lambda\|\mathbf{x}_{t}-\mathbf{v}^{\prime}_{t}\|_ {2}^{2}$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\end{aligned}$ ( $\texttt{S}_{1}$ ) $$

$$ $\displaystyle\begin{aligned} \leavevmode\hbox to145.98pt{\vbox to18.9pt{ \pgfpicture\makeatletter\hbox{\hskip 72.98814pt\lower-9.44888pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}\hbox{\hbox{{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{0.85,1,0.85 }\pgfsys@color@rgb@fill{0.85}{1}{0.85}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{ 0.85,1,0.85}\pgfsys@color@rgb@fill{0.85}{1}{0.85}\pgfsys@invoke{ }{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{} }}{}{}\pgfsys@moveto{70.98814pt}{9.44888pt}\pgfsys@lineto{-70.98814pt}{9.44888 pt}\pgfsys@curveto{-72.09273pt}{9.44888pt}{-72.98814pt}{8.55347pt}{-72.98814pt }{7.44888pt}\pgfsys@lineto{-72.98814pt}{-7.44888pt}\pgfsys@curveto{-72.98814pt }{-8.55347pt}{-72.09273pt}{-9.44888pt}{-70.98814pt}{-9.44888pt}\pgfsys@lineto{ 70.98814pt}{-9.44888pt}\pgfsys@curveto{72.09273pt}{-9.44888pt}{72.98814pt}{-8. 55347pt}{72.98814pt}{-7.44888pt}\pgfsys@lineto{72.98814pt}{7.44888pt} \pgfsys@curveto{72.98814pt}{8.55347pt}{72.09273pt}{9.44888pt}{70.98814pt}{9.44 888pt}\pgfsys@closepath\pgfsys@moveto{-72.98814pt}{-9.44888pt}\pgfsys@fill \pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-69.65514pt}{-2.66191 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$\hat{\mathbf{x}}^{\prime}_{0}=f(\hat{\mathbf{v}}_{ t};t,\bm{\epsilon}_{\theta})\equiv\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left[\hat{ \mathbf{v}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\,\bm{\epsilon}_{\theta}(\hat{\mathbf {v}}_{t},t)\right]$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\end{aligned}$ ( $\texttt{S}_{2}$ ) $$

$$ $\displaystyle\begin{aligned} \leavevmode\hbox to141.96pt{\vbox to17.94pt{ \pgfpicture\makeatletter\hbox{\hskip 70.98158pt\lower-8.9719pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}\hbox{\hbox{{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{0.7,1,1} \pgfsys@color@cmyk@fill{0.3}{0}{0}{0}\pgfsys@invoke{ }{{}{}{{ {}{}}}{ {}{}} {{}{{}}}{{}{}}{}{{}{}}{}{}{}{}{} {\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgffillcolor}{rgb}{0.7,1,1} \pgfsys@color@cmyk@fill{0.3}{0}{0}{0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{} {{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{68.98158pt}{8.9719pt}\pgfsys@lineto{-68.98158pt}{8.9719pt} \pgfsys@curveto{-70.08617pt}{8.9719pt}{-70.98158pt}{8.07648pt}{-70.98158pt}{6. 9719pt}\pgfsys@lineto{-70.98158pt}{-6.9719pt}\pgfsys@curveto{-70.98158pt}{-8.0 7648pt}{-70.08617pt}{-8.9719pt}{-68.98158pt}{-8.9719pt}\pgfsys@lineto{68.98158 pt}{-8.9719pt}\pgfsys@curveto{70.08617pt}{-8.9719pt}{70.98158pt}{-8.07648pt}{7 0.98158pt}{-6.9719pt}\pgfsys@lineto{70.98158pt}{6.9719pt}\pgfsys@curveto{70.98 158pt}{8.07648pt}{70.08617pt}{8.9719pt}{68.98158pt}{8.9719pt}\pgfsys@closepath \pgfsys@moveto{-70.98158pt}{-8.9719pt}\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}{{{{}}\pgfsys@beginscope \pgfsys@invoke{ }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-67.64857pt}{-3.13889 pt}\pgfsys@invoke{ }\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\hbox{{$\mathbf{x}_{t-1}=\sqrt{{\bar{\alpha}}_{t-1}}\hat{ \mathbf{x}}^{\prime}_{0}+\sqrt{1-{\bar{\alpha}}_{t-1}}\bm{\eta}_{t}\>,~{}~{} \bm{\eta}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{I})\>.$}} }}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\end{aligned}$ ( $\texttt{S}_{3}$ ) $$

Figure: Figure 3: Illustrative diagram of the proposed procedure in SITCOM (left). Conceptual illustration of SITCOM, where $\mathcal{M}_{t}$ is the DM generative manifold at time $t$ and $\mathcal{C}_{t}$ is the subset of images that are backward-consistent, defined in (9) (right). Step (1) (solid arrow), Step (2) (dotted arrow), and Step (3) (dashed arrow) correspond to ($\texttt{S}_{1}$), ($\texttt{S}_{2}$), and ($\texttt{S}_{3}$), respectively.
Refer to caption: x3.png

The minimization in the first step optimizes over the input $\mathbf{v}_{t}^{\prime}$ of the pre-trained diffusion model at time $t$, where the first term of the objective enforces measurement consistency for the posterior mean estimated image, satisfying condition C1. The second term serves as a regularization term, implicitly promoting closeness between $\hat{\mathbf{v}}_{t}$ and $\mathbf{x}_{t}$ (i.e., condition C3), with $\lambda>0$ acting as the regularization parameter. The argument of the forward operator in ($\texttt{S}_{1}$) and the second step in ($\texttt{S}_{2}$) enforce that $\hat{\mathbf{v}}_{t}$ and $\hat{\mathbf{x}}^{\prime}_{0}$, respectively, maintain the diffusion trajectory through obeying Tweedie’s formula, thereby satisfying the backward consistency condition, C2. After obtaining the measurement-consistent estimate, $\hat{\mathbf{x}}^{\prime}_{0}$, as given in ($\texttt{S}_{2}$), it must be mapped back to time $t-1$ to generate $\mathbf{x}_{t-1}$. This is achieved through the forward diffusion step in ($\texttt{S}_{3}$) as outlined in the forward consistency condition, C3. A diagram of SITCOM procedure is provided in Figure 3 (left).

###### Remark 1 .

Obtaining the estimated image at time 0 0 given some $\mathbf{x}_{t}$ using the standard DM PF-ODE [] is more accurate compared to the one-step Tweedie’s formula. However, since PF-ODE is an iterative procedure, it requires more computational time. In SITCOM, PF-ODE could replace Tweedie’s formula in ($\texttt{S}_{2}$). Nevertheless, we chose not to use it, as this would increase the run time, and our empirical results are already highly competitive using Tweedie’s formula.

A conceptual illustration of SITCOM is shown in Figure 3 (right).
The DM generative manifold, $\mathcal{M}_{t}$, is defined as the set of all $\mathbf{x}_{t}$ sampled from $q(\mathbf{x}_{t}|\mathbf{x}_{0})=\mathcal{N}(\mathbf{x}_{t};\sqrt{\bar{\alpha}
_{t}}\mathbf{x}_{0},(1-\bar{\alpha}_{t})\mathbf{I})$, and $\mathbf{x}_{0}\sim p_{0}(\mathbf{x})$. This set coincides with the entire space $\mathbb{R}^{n}$ equipped with the probability measure induced by the distribution of $\mathbf{x}_{t}$, which we denote as $\mathcal{P}_{t}$. In Figure 3 (right), the variation of color around each $\mathcal{M}_{t}$ indicates the concentration of the measure $\mathcal{P}_{t}$, with darker colors representing higher concentration.
SITCOM’s Step (1) and Step (2) enforce measurement consistency and backward consistency, thus map $\mathbf{x}_{t}$ to $\hat{\mathbf{x}}^{\prime}_{0}=f(\hat{\mathbf{v}}_{t};t,\bm{\epsilon}_{\theta})$ which lies within the intersection of (i) measurement-consistent set $\{\hat{\mathbf{x}}^{\prime}_{0}:\mathcal{A}(\hat{\mathbf{x}}^{\prime}_{0})
\approx\mathbf{y}\}$ (the shaded black line) and (ii) the backward-consistent set $\mathcal{C}_{t}$ (the yellow ellipsoid) defined in (9). Subsequently, $\mathbf{x}_{t-1}$ is generated by inserting $\hat{\mathbf{x}}^{\prime}_{0}$ into the resampling formula, which enforces the forward consistency.

###### Remark 1 .

#### Handling Measurement Noise:

To avoid the case where the first term of the objective in ($\texttt{S}_{1}$) reaches small values yielding noise overfitting (i.e., when additive Gaussian noise in (1) is considered, $\sigma_{\mathbf{y}}>0$), we propose refraining from enforcing strict measurement fitting $\mathcal{A}(\mathbf{x})=\mathbf{y}$. Instead, we use the stopping criterion $\big{\|}\mathcal{A}\big{(}\frac{1}{\sqrt{\bar{\alpha}_{t}}}\big{[}\mathbf{v}^{
\prime}_{t}-\sqrt{1-\bar{\alpha}_{t}}\,\bm{\epsilon}_{\theta}(\mathbf{v}^{
\prime}_{t},t)\big{]}\big{)}-\mathbf{y}\big{\|}^{2}_{2}<\delta^{2}\>,$
where $\delta\in\mathbb{R}_{+}$ is a hyper-parameter that indicates the level of tolerance for noise and helps prevent overfitting. This is equivalent to enforcing an $\ell_{2}$ constraint, and is in spirit similar to []. Since the noise level cannot be accurately estimated, in our experiments, we use $\delta$ that is slightly larger than the actual level of noise in the measurements, i.e., $\delta>\sigma_{\mathbf{y}}\sqrt{m}$.

### 3.5 SITCOM with Arbitrary Stepsizes

In this subsection, we explain how to apply SITCOM with a large stepsize and present the final algorithm. The pre-trained DM is trained with $T$ diffusion steps. Given that our method is designed to satisfy measurement and diffusion consistency, SITCOM requires $N\ll T$ sampling iterations, using a step size of $\Delta t:=\lfloor\frac{T}{N}\rfloor$. Thus, we introduce the index $i$ instead of $t$ with a relation $t=i\Delta t$.

The procedure of SITCOM is outlined in Algorithm 1. As inputs, SITCOM takes $\mathbf{y}$, $\mathcal{A}(\cdot)$, $\bm{\epsilon}_{\theta}$, the number of sampling steps $N$, $\bar{\alpha}_{i}$ for all $i\in\{1,\dots,N\}$, the number of optimization steps $K$ per sampling step, stopping criteria $\delta$, and the learning rate $\gamma$.

Starting with initializing ${\mathbf{v}}^{(0)}_{i}$ as $\mathbf{x}_{i}$ (satisfying condition C3), lines 3 through 6 correspond to the first step of SITCOM, where ($\texttt{S}_{1}$) is solved via either gradient descent (as shown in the algorithm), or the ADAM optimizer []. In lines 5 and 6, the stopping criterion is applied to prevent strict data fidelity (avoiding noise overfitting). Following the gradient updates in the inner loop, $\hat{\mathbf{v}}_{i}$ is obtained in line 7, which is then used in line 8 to obtain $\hat{\mathbf{x}}^{\prime}_{0}$ as specified in ($\texttt{S}_{2}$), satisfying condition C2. Note that line 8 requires no additional computation, as the $\hat{\mathbf{x}}^{\prime}_{0}$ calculated here was already obtained while checking the stopping condition in line 6. After obtaining the double-consistent $\hat{\mathbf{x}}^{\prime}_{0}$, the resampling is applied to map the image back to time $t-1$ while ensuring $\mathbf{x}_{t-1}$ to be in-distribution, as indicated in line 9 of the algorithm. In the next iteration, the requirement that $\hat{\mathbf{v}}_{t-1}$ is close to $\mathbf{x}_{t-1}$ ensures that the input $\hat{\mathbf{v}}_{t-1}$ to the DM network, $\bm{\epsilon}_{\theta}$, is also in-distribution, thus satisfying the forward-consistency (condition C3).

Figure: Algorithm 1 Step-wise Triple-Consistent Sampling (SITCOM).

The computational requirements of SITCOM are determined by (i) the number of sampling steps $N$ and (ii) the number of gradient steps $K$ required for each sampling iteration. Given the proposed stopping criterion, this results in at most $NK$ Number of Function Evaluations (NFEs) of the pre-trained model (forward pass), $NK$ backward passes through the pre-trained model, and $NK$ applications each for the forward operator and its adjoint to solve the optimization problem in ($\texttt{S}_{1}$). With early stopping, the computational cost is lower. For example, for a linear operator $\mathcal{A}$ with dimensions $m\times n$, the cost of applying it (or its adjoint) to a vector is $\mathcal{O}(mn)$. For a network with width $M$ and depth $L$, the cost for making a forward pass is $\mathcal{O}(LM^{2})$. The gradients are computed w.r.t. the input of the DM network, requiring an additional backward pass. This backward pass has the same computational cost as the forward pass. Consequently, this procedure is significantly more efficient than network training, where the network weights are updated instead of the input.

### 3.6 Relation with Existing Approaches

While SITCOM and DPS [] both use Tweedie’s formula, there are two major differences. First, DPS does not enforce backward consistency. Specifically, it only considers one gradient descent step of the optimization in ($\texttt{S}_{1}$), whereas our method perform multiple steps, initializing with $\mathbf{x}_{t}$. Second, DPS does not enforce the forward diffusion consistency, namely, it does not use resampling ($\texttt{S}_{3}$). This means that DPS does not enforce a step-wise C1-C3.

Both SITCOM and the works in [, ] are optimization-based methods that modify the sampling steps to enforce measurement consistency, and both involve mapping back to time $t-1$ (as in step 3 of SITCOM). However, there is a major difference between them:
The optimization variable in these works is the estimated image at time $t$ (the output of the DM network), whereas in SITCOM, it is the noisy image at time $t$ (the input of the network). This means that these studies enforce C1 and C3, but not C2.

## 4 Experimental Results

#### Tasks:

Our experimental setup for IPs and noise levels used largely follows DPS []. For linear IPs, we evaluate five tasks: super resolution, Gaussian deblurring, motion deblurring, box inpainting, and random inpainting. For Gaussian deblurring and motion deblurring, we use 61$\times$61 kernels with standard deviations of 3 and 0.5, respectively. In the super-resolution task, a bicubic resizer downscales images by a factor of 4. For box inpainting, a random 128$\times$128 box is applied to mask image pixels, and for random inpainting, the mask is generated with each pixel masked with a probability of 0.7, as described in []. For nonlinear IP tasks, we consider three tasks: phase retrieval, high dynamic range (HDR) reconstruction, and nonlinear (non-uniform) deblurring. For phase retrieval, an oversampling rate of 2 is applied in frequency domain, and we report the best result out of four independent samples, consistent with [, ] (see Appendix D for more discussion on phase retrieval). In HDR reconstruction, the goal is to restore a higher dynamic range image from a lower dynamic range image (with a factor of 2). Nonlinear deblurring follows the setup in []. For measurement noise, we use $\sigma_{\mathbf{y}}\in\{0.01,0.05\}$ for all tasks.

#### Baselines & Datasets:

For baselines, in this section, we use DPS [], DDNM [], DCDP [], and DAPS []. The selection criteria is based on these baselines’ competitive performance on several linear and non-linear inverse problems under measurement noise. Additionally, we provide comparison results with three other baselines in Table 3 of Appendix E. We evaluate SITCOM and baselines using 100 test images from the validation set of FFHQ [] and 100 test images from the validation set of ImageNet [] for which the FFHQ-trained and ImageNet-trained DMs are given in [] and [], respectively, following the previous convention. For evaluation metrics, we use PSNR, SSIM [], and LPIPS [].

#### SITCOM Settings:

For Algorithm 1, we set $N=20$ and $K=30$ for most tasks. We show the impact of $N$ and $K$ in Appendix F.1. The parameter $\lambda$ is set to 0 for all tasks other than phase retrieval where we use $\lambda=1$, following the ablation study in Appendix F.2. The impact of the stopping criterion under the noisy setting is given in Appendix F.3. The learning rate for ($\texttt{S}_{1}$) is set to $\gamma=0.01$ across all measurements noise levels, datasets, and tasks. Table 6 in Appendix F.4 lists all the hyper-parameters used for every task. We note that the exact set of hyper-parameters is used for the FFHQ and ImageNet datasets. Our code is available online(^2^22[https://github.com/sjames40/SITCOM](https://github.com/sjames40/SITCOM)).

**Table 1: Average PSNR, SSIM, LPIPS, and run-time (minutes) of SITCOM and baselines using 100 test images from the FFHQ dataset (columns 3 to 7) and 100 test images from the ImageNet dataset with a measurement noise level of $\sigma_{\mathbf{y}}=0.05$. The results for the $\sigma_{\mathbf{y}}=0.01$ case are given in Table 2 of Appendix E. The first five tasks are linear, while the last three tasks are non-linear (underlined). For each task and dataset combination, the best results are bolded, and the second-best results are underlined. Values after $\pm$ represent the standard deviation. All results were obtained using a single RTX5000 GPU machine. For phase retrieval, the run-time is reported for the best result out of four independent runs. This is applied for SITCOM and baselines. More discussion about phase retrieval is given in Appendix D.**
| Task | Method | FFHQ | ImageNet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | Run-time ($\downarrow$) | PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | Run-time ($\downarrow$) |  |  |
| Super Resolution 4$\times$ | DPS | 24.44$\pm$0.56 | 0.801$\pm$0.032 | 0.26$\pm$0.022 | 1.26$\pm$0.52 | 23.86$\pm$0.34 | 0.76$\pm$0.041 | 0.357$\pm$0.069 | 2.38$\pm$1.02 |
| DAPS | 29.24$\pm$0.42 | 0.851$\pm$0.024 | 0.135$\pm$0.039 | 1.24 $\pm$0.22 | 25.67$\pm$0.73 | 0.802$\pm$0.045 | 0.256$\pm$0.067 | 2.16$\pm$0.45 |  |
| DDNM | 28.02$\pm$0.78 | 0.842$\pm$0.034 | 0.197$\pm$0.034 | 1.07$\pm$0.42 | 23.96$\pm$0.89 | 0.767$\pm$0.045 | 0.475$\pm$0.044 | 1.27$\pm$0.55 |  |
| DCDP | 27.88$\pm$1.34 | 0.825$\pm$0.07 | 0.211$\pm$0.05 | 0.52$\pm$0.34 | 24.12$\pm$1.24 | 0.772$\pm$0.000 | 0.351$\pm$0.00 | 1.45$\pm$0.00 |  |
| SITCOM (ours) | 30.68$\pm$1.02 | 0.867$\pm$0.045 | 0.142$\pm$0.056 | 0.45$\pm$0.58 | 26.35$\pm$1.21 | 0.812$\pm$0.021 | 0.232$\pm$0.038 | 1.12$\pm$0.52 |  |
| Box In-Painting | DPS | 23.20$\pm$0.89 | 0.754$\pm$0.023 | 0.196$\pm$0.032 | 1.57$\pm$0.55 | 19.78$\pm$0.78 | 0.691$\pm$0.052 | 0.312$\pm$0.025 | 2.28 $\pm$1.02 |
| DAPS | 24.17$\pm$1.02 | 0.787$\pm$0.032 | 0.135$\pm$0.032 | 1.35$\pm$0.45 | 21.43$\pm$0.40 | 0.736$\pm$0.020 | 0.218$\pm$0.021 | 2.54$\pm$1.02 |  |
| DDNM | 24.37$\pm$0.45 | 0.792$\pm$0.024 | 0.232$\pm$0.026 | 1.02$\pm$0.032 | 21.64$\pm$0.66 | 0.732$\pm$0.028 | 0.319$\pm$0.015 | 1.45$\pm$1.02 |  |
| DCDP | 23.66$\pm$1.67 | 0.762$\pm$0.07 | 0.144$\pm$0.05 | 0.56$\pm$0.25 | 20.45$\pm$1.22 | 0.712$\pm$0.07 | 0.298$\pm$0.04 | 1.127$\pm$0.25 |  |
| SITCOM (ours) | 24.68$\pm$0.78 | 0.801$\pm$0.042 | 0.121$\pm$0.08 | 0.35$\pm$0.25 | 21.88$\pm$0.92 | 0.742$\pm$0.032 | 0.214$\pm$0.021 | 1.12$\pm$0.35 |  |
| Random In-Painting | DPS | 28.39$\pm$0.82 | 0.844$\pm$0.042 | 0.194$\pm$0.021 | 1.52$\pm$0.30 | 24.26$\pm$0.42 | 0.772$\pm$0.02 | 0.326$\pm$0.034 | 2.27$\pm$0.25 |
| DAPS | 31.02$\pm$0.45 | 0.902$\pm$0.015 | 0.098$\pm$0.017 | 1.56$\pm$0.40 | 28.44$\pm$0.45 | 0.872$\pm$0.024 | 0.135$\pm$0.052 | 2.14$\pm$0.45 |  |
| DDNM | 29.93$\pm$0.67 | 0.889$\pm$0.032 | 0.122$\pm$0.056 | 1.45$\pm$0.35 | 29.22$\pm$0.55 | 0.912$\pm$0.034 | 0.191$\pm$0.048 | 1.54$\pm$0.52 |  |
| DCDP | 28.59$\pm$0.95 | 0.852$\pm$0.06 | 0.202$\pm$0.04 | 0.55$\pm$0.25 | 26.22$\pm$1.13 | 0.791$\pm$0.06 | 0.289$\pm$0.03 | 1.44$\pm$0.34 |  |
| SITCOM (ours) | 32.05$\pm$1.02 | 0.909$\pm$0.09 | 0.095$\pm$0.025 | 0.45$\pm$0.50 | 29.60$\pm$0.78 | 0.915$\pm$0.028 | 0.127$\pm$0.039 | 1.14$\pm$0.45 |  |
| Gaussian Deblurring | DPS | 25.52$\pm$0.78 | 0.826$\pm$0.052 | 0.211$\pm$0.017 | 1.50$\pm$0.50 | 21.86$\pm$0.45 | 0.772$\pm$0.08 | 0.362$\pm$0.034 | 2.55$\pm$0.45 |
| DAPS | 29.22$\pm$0.50 | 0.884$\pm$0.056 | 0.164$\pm$0.032 | 1.40$\pm$0.52 | 26.12$\pm$0.78 | 0.832$\pm$0.092 | 0.245$\pm$0.022 | 2.23$\pm$0.52 |  |
| DDNM | 28.22$\pm$0.52 | 0.867$\pm$0.056 | 0.216$\pm$0.042 | 1.56$\pm$0.45 | 28.06$\pm$0.52 | 0.879$\pm$0.072 | 0.278$\pm$0.089 | 1.75$\pm$0.63 |  |
| DCDP | 26.67$\pm$0.78 | 0.835$\pm$0.08 | 0.196$\pm$0.04 | 0.56$\pm$0.23 | 23.24$\pm$1.18 | 0.781$\pm$0.06 | 0.343$\pm$0.04 | 1.34$\pm$0.43 |  |
| SITCOM (ours) | 30.25$\pm$0.89 | 0.892$\pm$0.032 | 0.135$\pm$0.078 | 0.46$\pm$0.25 | 27.40$\pm$0.45 | 0.854$\pm$0.045 | 0.236$\pm$0.039 | 1.10$\pm$0.42 |  |
| Motion Deblurring | DPS | 23.40$\pm$1.42 | 0.737$\pm$0.024 | 0.270$\pm$0.025 | 2.40$\pm$0.55 | 21.86$\pm$2.05 | 0.724$\pm$0.022 | 0.357$\pm$0.032 | 2.56$\pm$0.40 |
| DAPS | 29.66$\pm$0.50 | 0.872$\pm$0.027 | 0.157$\pm$0.012 | 1.86$\pm$0.12 | 27.86$\pm$1.20 | 0.862$\pm$0.032 | 0.196$\pm$0.021 | 2.3$\pm$0.45 |  |
| SITCOM (ours) | 30.34$\pm$0.67 | 0.902$\pm$0.037 | 0.148$\pm$0.041 | 0.5$\pm$0.45 | 28.65$\pm$0.34 | 0.876$\pm$0.021 | 0.189$\pm$0.036 | 1.48$\pm$0.35 |  |
| Phase Retrieval | DPS | 17.34$\pm$2.67 | 0.67$\pm$0.045 | 0.41$\pm$0.08 | 1.50$\pm$0.34 | 16.82$\pm$1.22 | 0.64$\pm$0.08 | 0.447$\pm$0.032 | 2.17$\pm$0.24 |
| DAPS | 30.67$\pm$3.12 | 0.908$\pm$0.041 | 0.122$\pm$0.084 | 1.34$\pm$0.78 | 25.76$\pm$2.33 | 0.797$\pm$0.045 | 0.255$\pm$0.095 | 2.24$\pm$0.25 |  |
| DCDP | 28.52$\pm$2.50 | 0.892$\pm$0.19 | 0.167$\pm$0.92 | 3.30$\pm$0.45 | 24.25$\pm$2.25 | 0.778$\pm$0.14 | 0.287$\pm$0.089 | 3.49$\pm$0.52 |  |
| SITCOM (ours) | 30.97$\pm$3.10 | 0.915$\pm$0.064 | 0.112$\pm$0.102 | 0.52$\pm$0.34 | 25.45$\pm$2.78 | 0.808$\pm$0.065 | 0.246$\pm$0.088 | 1.40$\pm$0.40 |  |
| Non-Uniform Deblurring | DPS | 23.42$\pm$2.15 | 0.757$\pm$0.042 | 0.279$\pm$0.067 | 1.55$\pm$0.44 | 22.57$\pm$0.67 | 0.778$\pm$0.067 | 0.310$\pm$0.102 | 2.35$\pm$0.45 |
| DAPS | 28.23$\pm$1.55 | 0.833$\pm$0.052 | 0.155$\pm$0.041 | 1.42$\pm$0.41 | 27.65$\pm$1.2 | 0.822$\pm$0.056 | 0.169$\pm$0.044 | 2.14$\pm$0.45 |  |
| DCDP | 28.78$\pm$1.44 | 0.827$\pm$0.08 | 0.162$\pm$0.04 | 3.30$\pm$0.45 | 26.56$\pm$1.09 | 0.803$\pm$0.06 | 0.182$\pm$0.05 | 3.70$\pm$0.36 |  |
| SITCOM (ours) | 30.12$\pm$0.68 | 0.902$\pm$0.042 | 0.145$\pm$0.037 | 0.52$\pm$0.45 | 28.78$\pm$0.79 | 0.832$\pm$0.056 | 0.16$\pm$0.048 | 1.25$\pm$0.45 |  |
| High Dynamic Range | DPS | 22.88$\pm$1.25 | 0.722$\pm$0.056 | 0.264$\pm$0.089 | 1.45$\pm$0.34 | 19.33$\pm$1.45 | 0.688$\pm$0.067 | 0.503$\pm$0.132 | 2.42$\pm$0.46 |
| DAPS | 27.12$\pm$0.89 | 0.825$\pm$0.056 | 0.166$\pm$0.078 | 1.25$\pm$0.35 | 26.30$\pm$1.02 | 0.792$\pm$0.046 | 0.177$\pm$0.089 | 2.18$\pm$0.55 |  |
| SITCOM (ours) | 27.98$\pm$1.06 | 0.832$\pm$0.052 | 0.158$\pm$0.032 | 0.52$\pm$0.30 | 26.97$\pm$0.87 | 0.821$\pm$0.045 | 0.167$\pm$0.052 | 1.54$\pm$0.35 |  |

#### Main Results:

In Table 1, we present the quantitative results in terms of the average PSNR, SSIM, LPIPS, and run-time (minutes). Columns 3 to 6 correspond to the FFHQ dataset, while columns 7 to 10 reflect results for the ImageNet dataset. The table covers 8 tasks, 4 evaluation metrics, and 2 datasets, totaling 64 results. Among these, SITCOM reports the best performance in 58 out of 64 cases.

On average, SITCOM demonstrates strong reconstruction capabilities across most tasks. For the FFHQ dataset, SITCOM reports a PSNR improvement of over 1 dB in Super Resolution, random In-painting, and Gaussian Deblurring compared to the second-best method. On ImageNet, we observe more than a 1 dB improvement in random In-painting. Other than ImageNet Gaussian Deblurring and ImageNet Phase Retrieval, for which we under-perform by 0.66 dB and 0.31 dB, respectively, our PSNR improvement when compared to the second-best results are less than 1 dB. However, in terms of run-time, SITCOM consistently requires less computational time across all tasks. For FFHQ, SITCOM is over 3$\times$ faster in Box In-painting and motion Deblurring, and more than 2$\times$ faster in the remaining tasks, whereas on ImageNet, the run-time improvement ranges from 36 seconds (for HDR) to 62.4 seconds (for Super Resolution), when compared to DPS, DDNM, and DAPS.

For linear tasks, SITCOM requires slightly less run-time than DCDP on both datasets. However, across the two datasets, SITCOM achieves PSNR improvements of more than 1 dB, 2 dB, and 3 dB for the tasks of super resolution, box in-painting, and random in-painting (and Gaussian Deblurring), respectively, as compared to DCDP. For non-linear tasks, SITCOM not only provides PSNR improvements over DCDP but also significantly reduces run-time.

In summary, the results in Table 1 demonstrate that SITCOM either provides a notable improvement in restoration quality (e.g., cases where we report PSNR improvements of over 1 dB) or delivers comparable results to the baselines, all while significantly reducing computation time.

In Appendix E, we present the results with $\sigma_{\mathbf{y}}=0.01$ case (Table 2). Additionally, Table 3 includes quantitative results for three more baselines. In addition to the FFHQ restored images in Figure 1, we also provide additional samples from both datasets in the figures found in Appendix H.

## 5 Conclusion

In this paper, we proposed three conditions to achieve measurement- and diffusion-consistent trajectories for linear and non-linear inverse imaging problems using diffusion models (DMs) as priors. These conditions form the basis of our unique optimization-based sampling method, which optimizes the input of the diffusion model at each step. This approach allows for greater control over the diffusion process and enhances data consistency with the given measurements. Through extensive experiments across eight image restoration tasks, we evaluated the effectiveness of our method. The results showed that our sampler consistently delivers improved or comparable quantitative performance against state-of-the-art baselines, even with measurement noise. Notably, our method is efficient, requiring significantly less run-time than leading baselines, making it practical for real-world applications.

## Appendix A Derivation of ( 6 )

From [], we have

$$ $\displaystyle\mathbf{s}_{\theta}(\mathbf{x}_{t},t)$ $\displaystyle=-\frac{1}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}( \mathbf{x}_{t},t)\>.$ (13) $$

Rearranging the Tweedie’s formula in (4) to solve for $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ yields

$$ $\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)=\frac{\mathbf{x}_{t}-\sqrt{\bar{ \alpha}_{t}}\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})}{\sqrt{1-\bar{\alpha}_{t}}}\>.$ (14) $$

Now, we substitute into the recursive equation for $\mathbf{x}_{t-1}$:

$$ $\displaystyle\mathbf{x}_{t-1}$ $\displaystyle=\frac{1}{\sqrt{1-\beta_{t}}}\left[\mathbf{x}_{t}+\beta_{t} \mathbf{s}_{\theta}(\mathbf{x}_{t},t)\right]+\sqrt{\beta_{t}}\bm{\eta}_{t}$ (15) $\displaystyle=\frac{1}{\sqrt{1-\beta_{t}}}\left[\mathbf{x}_{t}+\beta_{t}\left( -\frac{1}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t) \right)\right]+\sqrt{\beta_{t}}\bm{\eta}_{t}$ (16) $\displaystyle=\frac{1}{\sqrt{1-\beta_{t}}}\left[\mathbf{x}_{t}-\frac{\beta_{t} }{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\right]+ \sqrt{\beta_{t}}\bm{\eta}_{t}$ (17) $\displaystyle=\frac{1}{\sqrt{1-\beta_{t}}}\left[\mathbf{x}_{t}-\frac{\beta_{t} }{\sqrt{1-\bar{\alpha}_{t}}}\left(\frac{\mathbf{x}_{t}-\sqrt{\bar{\alpha}_{t}} \hat{\mathbf{x}}_{0}(\mathbf{x}_{t})}{\sqrt{1-\bar{\alpha}_{t}}}\right)\right] +\sqrt{\beta_{t}}\bm{\eta}_{t}$ (18) $\displaystyle=\frac{1}{\sqrt{1-\beta_{t}}}\left[\mathbf{x}_{t}-\frac{\beta_{t} }{1-\bar{\alpha}_{t}}\left(\mathbf{x}_{t}-\sqrt{\bar{\alpha}_{t}}\hat{\mathbf{ x}}_{0}(\mathbf{x}_{t})\right)\right]+\sqrt{\beta_{t}}\bm{\eta}_{t}$ (19) $\displaystyle=\frac{1}{\sqrt{1-\beta_{t}}}\left[\left(1-\frac{\beta_{t}}{1- \bar{\alpha}_{t}}\right)\mathbf{x}_{t}+\frac{\sqrt{\bar{\alpha}_{t}}\beta_{t}} {1-\bar{\alpha}_{t}}\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})\right]+\sqrt{\beta_{t }}\bm{\eta}_{t}$ (20) $\displaystyle=\frac{\left(1-\bar{\alpha}_{t}-\beta_{t}\right)}{\sqrt{1-\beta_{ t}}\left(1-\bar{\alpha}_{t}\right)}\mathbf{x}_{t}+\frac{\sqrt{\bar{\alpha}_{t} }\beta_{t}}{\sqrt{1-\beta_{t}}\left(1-\bar{\alpha}_{t}\right)}\hat{\mathbf{x}} _{0}(\mathbf{x}_{t})+\sqrt{\beta_{t}}\bm{\eta}_{t}$ (21) $\displaystyle=\frac{\left(\alpha_{t}-\bar{\alpha}_{t}\right)}{\sqrt{\alpha_{t} }\left(1-\bar{\alpha}_{t}\right)}\mathbf{x}_{t}+\frac{\sqrt{\bar{\alpha}_{t}} \beta_{t}}{\sqrt{\alpha_{t}}\left(1-\bar{\alpha}_{t}\right)}\hat{\mathbf{x}}_{ 0}(\mathbf{x}_{t})+\sqrt{\beta_{t}}\bm{\eta}_{t}$ (22) $\displaystyle=\frac{\left(\sqrt{\alpha_{t}}-\sqrt{\alpha_{t}}\bar{\alpha}_{t-1 }\right)}{1-\bar{\alpha}_{t}}\mathbf{x}_{t}+\frac{\sqrt{\bar{\alpha}_{t-1}} \beta_{t}}{1-\bar{\alpha}_{t}}\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})+\sqrt{\beta _{t}}\bm{\eta}_{t}$ (23) $\displaystyle=\frac{\sqrt{\alpha_{t}}\left(1-\bar{\alpha}_{t-1}\right)}{1-\bar {\alpha}_{t}}\mathbf{x}_{t}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{ \alpha}_{t}}\hat{\mathbf{x}}_{0}(\mathbf{x}_{t})+\sqrt{\beta_{t}}\bm{\eta}_{t}\>,$ (24) $$

which is equivalent to the second formula in (3).

## Appendix B Limitations & Future Work

In SITCOM, the stopping criterion parameter is set slightly higher than the level of measurement noise, determined by $\sigma_{\mathbf{y}}$. As a result, our method requires access to (or estimation of) the measurement noise prior to the restoration process. Knowledge of noise level is also assumed in other works such as DAPS []. In practice, classical approaches, such as [, ], can be used to estimate the noise.

Additionally, the stated conditions and proposed sampler are limited to the non-blind setting, as SITCOM assumes full access to the forward model, unlike works such as [], which perform both image restoration and forward model estimation.

For future work, in addition to addressing the aforementioned limitations, we aim to extend SITCOM to the latent space and explore its applicability in medical image reconstruction.

## Appendix C Impact of the proposed Backward Consistency

Here, we demonstrate the impact of the proposed backward diffusion consistency in SITCOM using two experiments.

Figure: Figure 4: Results of applying optimization-based measurement consistency, for which the optimization variable is the DM output (resp. input), are shown in the first (resp. second) row for each task: Box Inpainting (top) and Gaussian Deblurring (bottom).
Refer to caption: x4.png

First, for the box-painting task, we compare optimizing over the input to the DM (as in SITCOM) with optimizing over the output of the DM network (as is done in DCDP [] and DAPS []) at time steps $t^{\prime}\in\{200,400,600\}$. For each case (selection of $t^{\prime}$), we start from $t=T$ and run SITCOM with a step size of $\lfloor\frac{T}{N}\rfloor$. At $t=t^{\prime}$, given $\mathbf{x}_{t^{\prime}}$, we perform two separate optimizations with intializing the optimization variable as $\mathbf{x}_{t^{\prime}}$: one iteratively over the DM network input (ours) and another iteratively over the DM network output (i.e., (5) but without the regularization), both running until convergence (i.e., when the loss stops decreasing).
For our approach, the result of the optimization from ($\texttt{S}_{1}$) is used as input to Tweedie’s formula in ($\texttt{S}_{2}$) to compute the posterior mean $\hat{\mathbf{x}}^{\prime}_{0}=\hat{\mathbf{x}}_{0}(\mathbf{v}_{t})$. For the case of optimizing over the DM output, we use (5) without regularization.
Figure 2 shows the results at different time steps. The consistency between the ground truth and the unmasked regions of the estimated images suggest the convergence of the measurement consistency. As observed, SITCOM produces significantly less artifacts in the masked region when compared to optimizing over the output. This is evident both at earlier time steps ($t^{\prime}=600$) and later steps ($t^{\prime}=400$ and $t^{\prime}=200$).

For the second experiment, the goal is to show that SITCOM requires much smaller number of optimization steps to remove the noise as compared to the case where the optimization variable is the output of the DM network. The results are given in Figure 4, where we repeat the above experiment with two tasks: Box-inpainting (top) and Gaussian Deblurring (bottom), this time using a fixed number of optimization steps for both SITCOM, and optimizing over the DM output. Specifically, we run SITCOM from $t=T$ to $t=t^{\prime}+1$. Then, we apply $K=20$ iterations (the setting in SITCOM) in ($\texttt{S}_{1}$), and $K=20$ when optimizing (5) (without regularization) where measurement noise is $\sigma_{\mathbf{y}}=0.05$. As shown, compared to optimizing over the DM output, SITCOM significantly reduces noise across all considered $t^{\prime}$, underscoring the effect of the proposed backward diffusion consistency when optimizing over the DM input.

## Appendix D Discussion on Phase Retrieval

As discussed in our experimental results section, for the phase retrieval task, we report the best results from 4 independent runs, following the convention in [, ]. For the phase retrieval results of Table 1 and Table 2 (given in Appendix E), we use this approach across all baselines where the run-time is reported for one run.

Figure: Figure 5: Results of Phase Retrieval on two images (top row) from the FFHQ dataset. Rows 2, 3, and 4 correspond to the results of DPS, DAPS, and SITCOM (ours), respectively.
Refer to caption: x5.png

The forward model for phase retrieval is adopted from DPS where the inverse problem is generally more challenging compared to other image restoration tasks. This increased difficulty arises from the presence of multiple modes that can yield the same measurements [].

In Figure 5, we present two examples comparing SITCOM, DPS, and DAPS. For each ground truth image, we show four results from which the best one was selected. In the first column, SITCOM avoids significant artifacts, while DAPS produces one image rotated by 180 degrees. In the second column, both SITCOM and DAPS exhibit one run with severe artifacts. However, the last image from SITCOM does exhibit more artifacts compared to the second worst-case result from DAPS. Additionally, the DPS results show severe perceptual differences in both cases, with artifacts being particularly noticeable in the second column.

## Appendix E Additional Comparison Results

In Table 2, we present the average PSNR, SSIM, LPIPS, and run-time (minutes) of DPS, DAPS, DDNM, and SITCOM using the FFHQ and ImageNet datasets for which the measurement noise level is set to $\sigma_{\mathbf{y}}=0.01$ (different from Table 1). The goal of these results is to evaluate our method and baselines under less noisy settings.

**Table 2: Average PSNR, SSIM, LPIPS, and run-time (minutes) of SITCOM and baselines using 100 test images from FFHQ and 100 test images from ImageNet with a measurement noise level of $\sigma_{\mathbf{y}}=0.01$. The first five tasks are linear, while the last three tasks are non-linear (underlined). For each task and dataset combination, the best results are bolded, and the second-best results are underlined. Values after $\pm$ represent the standard deviation. All results were obtained using a single RTX5000 GPU machine. For phase retrieval, the run-time is reported for the best result out of four independent runs. This is applied for SITCOM and baselines.**
| Task | Method | FFHQ | ImageNet |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | Run-time ($\downarrow$) | PSNR ($\uparrow$) | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | Run-time ($\downarrow$) |  |  |
| Super Resolution 4$\times$ | DPS | 25.20$\pm$1.22 | 0.806$\pm$0.044 | 0.242$\pm$0.102 | 1.31$\pm$0.44 | 24.45$\pm$0.89 | 0.792$\pm$0.052 | 0.331$\pm$0.089 | 2.33$\pm$0.40 |
| DAPS | 29.6$\pm$0.67 | 0.871$\pm$0.034 | 0.132$\pm$0.088 | 1.24$\pm$0.43 | 25.98$\pm$0.74 | 0.794$\pm$0.09 | 0.234$\pm$0.089 | 2.10$\pm$1.02 |  |
| DDNM | 28.82$\pm$0.67 | 0.851$\pm$0.043 | 0.188$\pm$0.13 | 1.07$\pm$0.35 | 24.67$\pm$0.78 | 0.771$\pm$0.06 | 0.432$\pm$0.34 | 1.38$\pm$0.55 |  |
| Ours | 30.95$\pm$0.89 | 0.872$\pm$0.045 | 0.137$\pm$0.046 | 0.50$\pm$0.34 | 26.89$\pm$0.86 | 0.802$\pm$0.057 | 0.224$\pm$0.056 | 1.34$\pm$0.45 |  |
| Box In-Painting | DPS | 23.56$\pm$0.78 | 0.762$\pm$0.034 | 0.191$\pm$0.087 | 1.52$\pm$0.43 | 20.22$\pm$0.67 | 0.69$\pm$0.034 | 0.297$\pm$0.077 | 1.55$\pm$0.44 |
| DAPS | 24.41$\pm$0.67 | 0.791$\pm$0.034 | 0.129$\pm$0.067 | 1.33$\pm$0.42 | 21.79$\pm$0.34 | 0.734$\pm$0.045 | 0.214$\pm$0.034 | 2.44$\pm$0.34 |  |
| DDNM | 24.67$\pm$0.067 | 0.788$\pm$0.024 | 0.229$\pm$0.055 | 1.02$\pm$0.42 | 21.99$\pm$0.54 | 0.737$\pm$0.034 | 0.315$\pm$0.022 | 1.42$\pm$0.45 |  |
| Ours | 24.97$\pm$0.55 | 0.804$\pm$0.045 | 0.118$\pm$0.022 | 0.37$\pm$0.34 | 22.23$\pm$0.44 | 0.745$\pm$0.034 | 0.208$\pm$0.023 | 1.23$\pm$0.44 |  |
| Random In-Painting | DPS | 28.77$\pm$0.56 | 0.847$\pm$0.034 | 0.191$\pm$0.023 | 1.55$\pm$0.34 | 24.57$\pm$0.45 | 0.775$\pm$0.023 | 0.318$\pm$0.26 | 2.12$\pm$0.30 |
| DAPS | 31.56$\pm$0.45 | 0.905$\pm$0.013 | 0.094$\pm$0.012 | 1.42$\pm$0.45 | 28.86$\pm$0.67 | 0.877$\pm$0.021 | 0.131$\pm$0.044 | 2.01$\pm$0.34 |  |
| DDNM | 30.56$\pm$0.56 | 0.902$\pm$0.013 | 0.116$\pm$0.023 | 1.25$\pm$0.42 | 30.12$\pm$0.45 | 0.917$\pm$0.012 | 0.124$\pm$0.032 | 1.89$\pm$0.23 |  |
| Ours | 33.02$\pm$0.44 | 0.919$\pm$0.012 | 0.0912$\pm$0.013 | 0.47$\pm$0.34 | 30.67$\pm$0.45 | 0.918$\pm$0.013 | 0.118$\pm$0.012 | 1.40$\pm$0.34 |  |
| Gaussian Deblurring | DPS | 25.78$\pm$0.68 | 0.831$\pm$0.034 | 0.202$\pm$0.014 | 1.33$\pm$0.44 | 22.45$\pm$0.42 | 0.778$\pm$0.067 | 0.344$\pm$0.041 | 2.12$\pm$0.44 |
| DAPS | 29.67$\pm$0.45 | 0.889$\pm$0.045 | 0.163$\pm$0.033 | 2.15$\pm$0.37 | 26.34$\pm$0.55 | 0.836$\pm$0.034 | 0.244$\pm$0.023 | 2.22$\pm$0.43 |  |
| DDNM | 28.56$\pm$0.45 | 0.872$\pm$0.024 | 0.211$\pm$0.034 | 1.24$\pm$0.34 | 28.44$\pm$0.021 | 0.882$\pm$0.021 | 0.267$\pm$0.00 | 1.76$\pm$0.33 |  |
| Ours | 32.12$\pm$0.34 | 0.913$\pm$0.024 | 0.139$\pm$0.045 | 0.45$\pm$0.25 | 28.22$\pm$0.45 | 0.891$\pm$0.014 | 0.216$\pm$0.021 | 1.34$\pm$0.25 |  |
| Motion Deblurring | DPS | 23.78$\pm$0.78 | 0.742$\pm$0.042 | 0.265$\pm$0.024 | 1.65$\pm$0.34 | 22.33$\pm$0.727 | 0.726$\pm$0.034 | 0.352$\pm$0.00 | 2.21$\pm$0.40 |
| DAPS | 30.78$\pm$0.56 | 0.892$\pm$0.034 | 0.146$\pm$0.023 | 1.44$\pm$0.34 | 28.24$\pm$0.62 | 0.867$\pm$0.023 | 0.191$\pm$0.017 | 2.12$\pm$0.44 |  |
| Ours | 32.34$\pm$0.44 | 0.908$\pm$0.028 | 0.135$\pm$0.028 | 0.52$\pm$0.34 | 29.12$\pm$0.38 | 0.882$\pm$0.025 | 0.182$\pm$0.025 | 1.45$\pm$0.31 |  |
| Phase Retrieval | DPS | 17.56$\pm$2.15 | 0.681$\pm$0.056 | 0.392$\pm$0.021 | 1.52$\pm$0.42 | 16.77$\pm$1.78 | 0.651$\pm$0.076 | 0.442$\pm$0.037 | 2.18$\pm$0.38 |
| DAPS | 31.45$\pm$2.78 | 0.909$\pm$0.035 | 0.109$\pm$0.044 | 1,85$\pm$0.32 | 26.12$\pm$2.12 | 0.802$\pm$0.023 | 0.247$\pm$0.034 | 2.32$\pm$0.35 |  |
| Ours | 31.88$\pm$2.89 | 0.921$\pm$0.067 | 0.102$\pm$0.078 | 0.54$\pm$0.45 | 25.76$\pm$1.78 | 0.813$\pm$0.032 | 0.238$\pm$0.067 | 1.31$\pm$0.45 |  |
| Non-Uniform Deblurring | DPS | 23.78$\pm$2.23 | 0.761$\pm$0.051 | 0.269$\pm$0.064 | 1.56$\pm$0.45 | 22.97$\pm$1.57 | 0.781$\pm$0.023 | 0.302$\pm$0.089 | 2.34$\pm$0.44 |
| DAPS | 28.89$\pm$1.67 | 0.845$\pm$0.057 | 0.150$\pm$0.056 | 1.41$\pm$0.37 | 28.02$\pm$1.15 | 0.831$\pm$0.082 | 0.162$\pm$0.034 | 2.23$\pm$0.56 |  |
| Ours | 31.09$\pm$0.89 | 0.911$\pm$0.056 | 0.132$\pm$0.45 | 0.56$\pm$0.37 | 29.56$\pm$0.78 | 0.844$\pm$0.045 | 0.147$\pm$0.042 | 1.34$\pm$0.44 |  |
| High Dynamic Range | DPS | 23.33$\pm$1.34 | 0.734$\pm$0.049 | 0.251$\pm$0.078 | 1.34$\pm$0.42 | 19.67$\pm$0.056 | 0.693$\pm$0.034 | 0.498$\pm$0.112 | 2.34$\pm$0.41 |
| DAPS | 27.58$\pm$0.829 | 0.828$\pm$0.00 | 0.161$\pm$0.067 | 1.26$\pm$0.44 | 26.71$\pm$0.088 | 0.802$\pm$0.032 | 0.172$\pm$0.066 | 2.12$\pm$0.32 |  |
| Ours | 28.52$\pm$0.89 | 0.844$\pm$0.045 | 0.148$\pm$0.035 | 0.51$\pm$0.42 | 27.56$\pm$0.78 | 0.825$\pm$0.037 | 0.162$\pm$0.046 | 1.45$\pm$0.41 |  |

Overall, we observe similar trends to those discussed in Section 4 for Table 1. On the FFHQ dataset, SITCOM achieves higher average PSNR values compared to the baselines across all tasks, with improvements exceeding 1 dB in 5 out of 8 tasks. For the ImageNet dataset, we observe more than 1 dB improvement on the non-linear deblurring task, while for the remaining tasks, the improvement is less than 1 dB, except for Gaussian deblurring (where SITCOM underperforms by 0.22 dB) and phase retrieval (underperforming by 0.36 dB).

In terms of run-time, generally, SITCOM significantly outperforms DDNM, DPS, and DAPS, with all methods evaluated on a single RTX5000 GPU. For the FFHQ dataset, SITCOM is at least twice as fast when compared to baselines. On ImageNet, SITCOM consistently requires much less run-time compared to DPS and DAPS. When compared to DDNM, SITCOM’s run-time is similar or slightly lower. For example, on the super-resolution task, both SITCOM and DDNM average 1.34 minutes, but SITCOM achieves over a 2 dB improvement.

In Table 3, we report the average PSNR and LPIPS results using three more baselines: Denoising Diffusion Restoration Models (DDRM) [], Plug-and-Play (PnP) ADMM [] (a non diffusion-based solver), and Regularization by Denoising with Diffusion (RED-Diff) []. The results of DDRM, PnP-ADMM, and RED-Diff are sourced from []. DDRM and PnP-ADMM present results for linear tasks whereas RED-Diff is used for the non-linear tasks. The results of SITCOM are as reported in Table 1.

When compared to DDRM and PnP-ADMM, SITCOM demonstrates notable improvements in both PSNR and LPIPS across all tasks and datasets. For instance, SITCOM achieves over a 5 dB improvement in random in-painting on both datasets. Compared to RED-Diff, SITCOM outperforms by 5 dB on FFHQ and more than 10 dB on ImageNet for phase retrieval. A similar trend is observed in the High Dynamic Range task. For non-linear non-uniform deblurring, although SITCOM performs better in terms of LPIPS, it reports approximately 1 dB (FFHQ) and 2 dB (ImageNet) less PSNR than RED-Diff, all without requiring external denoisers.

**Table 3: Average PSNR and LPIPS results of our method and other baselines over 100 FFHQ and 100 ImageNet test images. The measurement noise setting is $\sigma_{\mathbf{y}}=0.05$. The results of DDRM and PnP-ADMM (resp. RED-Diff) are sourced from Tables 1 and 3 (resp. 2 and 4) in []. The remaining results are as given in Table 1 of Section 4.**
| Task | Method | FFHQ | ImageNet |  |  |
| --- | --- | --- | --- | --- | --- |
| PSNR ($\uparrow$) | LPIPS ($\downarrow$) | PSNR ($\uparrow$) | LPIPS ($\downarrow$) |  |  |
| Super Resolution 4$\times$ | DDRM [] | 27.65 | 0.210 | 25.21 | 0.284 |
| PnP-ADMM [] | 23.48 | 0.725 | 22.18 | 0.724 |  |
| SITCOM (ours) | 30.68 | 0.142 | 26.35 | 0.232 |  |
| Box In-Painting | DDRM [] | 22.37 | 0.159 | 19.45 | 0.229 |
| PnP-ADMM [] | 13.39 | 0.775 | 12.61 | 0.702 |  |
| SITCOM (ours) | 24.68 | 0.121 | 21.88 | 0.214 |  |
| Random In-Painting | DDRM [] | 25.75 | 0.218 | 23.23 | 0.325 |
| PnP-ADMM [] | 20.94 | 0.724 | 20.03 | 0.680 |  |
| SITCOM (ours) | 32.05 | 0.095 | 29.60 | 0.127 |  |
| Gaussian Deblurring | DDRM [] | 23.36 | 0.236 | 23.86 | 0.341 |
| PnP-ADMM [] | 21.31 | 0.751 | 20.47 | 0.729 |  |
| SITCOM (ours) | 30.25 | 0.235 | 27.40 | 0.236 |  |
| Motion Deblurring | PnP-ADMM [] | 23.40 | 0.703 | 24.23 | 0.684 |
| SITCOM (ours) | 30.34 | 0.148 | 28.65 | 0.189 |  |
| Phase Retrieval | RED-Diff [] | 15.60 | 0.596 | 14.98 | 0.536 |
| SITCOM (ours) | 30.97 | 0.112 | 25.45 | 0.246 |  |
| Non-Uniform Deblurring | RED-Diff [] | 30.86 | 0.160 | 30.07 | 0.211 |
| SITCOM (ours) | 30.12 | 0.145 | 28.78 | 0.160 |  |
| High Dynamic Range | RED-Diff [] | 22.16 | 0.258 | 22.03 | 0.274 |
| SITCOM (ours) | 27.98 | 0.158 | 26.97 | 0.167 |  |

## Appendix F Ablation Studies

### F.1 Effect of the number of Optimization steps K 𝐾 K italic_K , & the number of Sampling steps N 𝑁 N italic_N

In this subsection, we perform an ablation study on the number of optimization steps, $K$, and the number of sampling steps, $N$. Specifically, for the tasks of Super Resolution, Motion Deblurring, and Random In-painting, we run SITCOM using combinations from $N\in\{10,20,30\}$ and $K\in\{20,30,40\}$. The average PSNR results over 20 test images from the FFHQ dataset are presented in Table 4. As shown, for the first three tasks, SITCOM consistently achieves strong PSNR scores across all $(N,K)$ pairs, demonstrating that its performance is not very sensitive to variations in $(N,K)$ within these ranges as the results vary by nearly 1 dB. For High Dynamic Range tasks, we observe that the best results are obtained with $(N,K)=(20,40)$. The selected $(N,K)$ values for our main results are listed in Table 6 of Appendix F.4.

**Table 4: Effect of the number of sampling steps ($N$) and optimization steps per sampling iteration ($K$) on the tasks listed in the first column for SITCOM. The reported PSNR values are averaged over 20 FFHQ test images.**
| $(N,K)$ | $(10,20)$ | $(10,30)$ | $(10,40)$ | $(20,20)$ | $(20,30)$ | $(20,40)$ | $(30,20)$ | $(30,30)$ | $(30,40)$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Super Resolution 4$\times$ | 29.654 | 29.771 | 29.815 | 29.913 | 29.952 | 29.961 | 30.009 | 30.027 | 30.033 |
| Motion Deblurring | 29.976 | 30.820 | 31.264 | 31.259 | 31.380 | 30.452 | 31.282 | 30.624 | 30.438 |
| Random Inpainting | 33.428 | 34.444 | 34.699 | 34.546 | 34.558 | 34.574 | 34.619 | 34.634 | 34.639 |
| High Dynamic Range | 25.902 | 26.290 | 27.873 | 26.957 | 27.104 | 27.874 | 27.171 | 27.127 | 26.806 |

### F.2 Effect of the Regularization Parameter λ 𝜆 \lambda italic_λ

In this subsection, we perform an ablation study to assess the impact of the regularization parameter, $\lambda$, in SITCOM. Table 5 shows the results across four tasks using various $\lambda$ values. Aside from phase retrieval, the effect of $\lambda$ is minimal. We hypothesize that initializing the optimization variable in ($\texttt{S}_{1}$) with $\mathbf{x}_{t}$ is sufficient to enforce forward diffusion consistency in C3. Therefore, we set $\lambda=1$ for phase retrieval and $\lambda=0$ for the other tasks.

Additionally, for all tasks other than phase retrieval, we observed that when $\lambda=0$, the restored images exhibit enhanced high-frequency details. For visual examples, see the results of $\lambda=0$ versus $\lambda=1$ in Figure 6.

**Table 5: Ablation Study on the impact of the regularization parameter $\lambda$.**
| $\lambda$ | 0 0 | $0.05$ | $0.5$ | $1$ | $1.5$ |
| --- | --- | --- | --- | --- | --- |
| Super Resolution 4$\times$ | 29.952 | 29.968 | 29.464 | 29.550 | 29.288 |
| Motion Deblurring | 31.380 | 31.393 | 31.429 | 31.382 | 31.150 |
| Random Inpainting | 34.559 | 34.537 | 34.523 | 34.500 | 34.301 |
| Phase Retrieval | 31.678 | 31.892 | 32.221 | 32.342 | 32.124 |

Figure: Figure 6: Results of running SITCOM using different regularization parameters in ($\texttt{S}_{1}$) for the task of Motion deblurring.
Refer to caption: x6.png

### F.3 Impact of the Stopping criterion For Noisy Measurements

In this subsection, we demonstrate the impact of applying the stopping criterion in SITCOM when handling measurement noise. For the tasks of super resolution and motion deblurring, we run SITCOM with and without the stopping criterion for the case of $\sigma_{\mathbf{y}}=0.05$. The results are presented in Figure 7. As shown, for both tasks, using the stopping criterion (i.e., $\delta>0$) not only improves PSNR values compared to the case of $\delta=0$, but also visually reduces additive noise in the restored images. This is because, without the stopping criterion , the measurement consistency enforced by the optimization in ($\texttt{S}_{1}$) tends to fit the noise in the measurements.

Figure: Figure 7: Impact of the stopping criterion in preventing noise overfitting. For the most right column, $\delta$ is set as in Table 6.
Refer to caption: x7.png

### F.4 Complete List of hyper-parameters in SITCOM

Table 6 summarizes the hyper-parameters used for each task in our experiments, as determined by the ablation studies in the previous subsections. Notably, the same set of hyper-parameters is applied to both the FFHQ and ImageNet datasets.

**Table 6: Hyper-parameters of SITCOM for every task considered in this paper. The same set of hyper-parameters is used for FFHQ and ImageNet. The learning rate in Algorithm 1 is set to $\gamma=0.01$ for all tasks, datasets, and measurement noise levels. For the stopping criterion column, $m_{\textrm{SR}}=64\times 64\times 3$, $m=256\times 256\times 3$, and $m_{\textrm{PR}}=384\times 384\times 3$.**
| Task | Sampling Steps $N$ | Optimization Steps $K$ | Regularization $\lambda$ | Stopping criterion $\delta$ for $\sigma_{\mathbf{y}}\in\{0.05,0.01\}$ |
| --- | --- | --- | --- | --- |
| Super Resolution 4$\times$ | $20$ | $20$ | 0 0 | {$0.051\sqrt{m_{\textrm{SR}}}$ ,$0.011\sqrt{m_{\textrm{SR}}}$} |
| Box In-Painting | $20$ | $20$ | 0 0 | {$0.051\sqrt{m}$ ,$0.011\sqrt{m}$} |
| Random In-Painting | $20$ | $30$ | 0 0 | {$0.051\sqrt{m}$ ,$0.011\sqrt{m}$} |
| Gaussian Deblurring | $20$ | $30$ | 0 0 | {$0.051\sqrt{m}$ ,$0.011\sqrt{m}$} |
| Motion Deblurring | $20$ | $30$ | 0 0 | {$0.051\sqrt{m}$ ,$0.011\sqrt{m}$} |
| Phase Retrieval | $20$ | $30$ | $1$ | {$0.051\sqrt{m_{\textrm{PR}}}$ ,$0.011\sqrt{m_{\textrm{PR}}}$} |
| Non-Uniform Deblurring | $20$ | $30$ | 0 0 | {$0.051\sqrt{m}$ ,$0.011\sqrt{m}$} |
| High Dynamic Range | $20$ | $40$ | 0 0 | {$0.051\sqrt{m}$ ,$0.011\sqrt{m}$} |

## Appendix G Detailed Implementation of tasks and Baselines

The forward models of all tasks are adopted from DPS. We refer the reader to Appendix B of [] for details. For baselines, we used the codes provided by the authors of each paper: DPS(^3^33[https://github.com/DPS2022/diffusion-posterior-sampling](https://github.com/DPS2022/diffusion-posterior-sampling)), DDNM(^4^44[https://github.com/wyhuai/DDNM](https://github.com/wyhuai/DDNM)), DAPS(^5^55[https://github.com/zhangbingliang2019/DAPS](https://github.com/zhangbingliang2019/DAPS)), and DCDP(^6^66[https://github.com/Morefre/Decoupled-Data-Consistency-with-Diffusion-Purification-for-Image-Restoration](https://github.com/Morefre/Decoupled-Data-Consistency-with-Diffusion-Purification-for-Image-Restoration)). Default configurations are used for each task.

## Appendix H Qualitative results

Figure 8 presents results with SITCOM, DPS, and DAPS using ImageNet. See also Figure 9, Figure 10, Figure 11, and Figure 12 for more images.

Figure: Figure 8: Qualitative results on the ImageNet dataset for five linear tasks and three non-linear tasks under measurement noise of $\sigma_{\mathbf{y}}=0.05$. The PSNR and LPIPS values are given below each restored image.
Refer to caption: x8.png

Figure: Figure 9: Super resolution (left) and box inpainting (right) results. First (resp. last) three rows are for the FFHQ (resp. ImageNet) dataset.
Refer to caption: x9.png

Figure: Figure 10: Motion deblurring (left) and Gaussian deblurring (right) results. First (resp. last) three rows are for the FFHQ (resp. ImageNet) dataset.
Refer to caption: x10.png

Figure: Figure 11: Random inpainting (left) and non-linear (non-uniform) deblurring (right) results. First (resp. last) three rows are for the FFHQ (resp. ImageNet) dataset.
Refer to caption: x11.png

Figure: Figure 12: Phase retrieval (left) and high dynamic range (right) results. First (resp. last) three rows are for the FFHQ (resp. ImageNet) dataset.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - How sensitive is the performance with respect to the parameter λ?

see the weakness part for more comments
soundness: 2
strengths: - The paper addresses a timely and important problem in inverse problem-solving using diffusion models.

- Experiments cover a variety of linear and nonlinear tasks, giving a broad perspective on the method's applicability.
summary: This paper addresses inverse problems using a plug-and-play approach with diffusion models, focusing on reducing the low sampling speeds that are characteristic of iterative diffusion models in these settings. To this end, the authors propose a "triple consistency" framework, adding a backward consistency component to the already established data and forward consistencies. The backward consistency aims to ensure that the solution obtained after data-consistency remains a valid diffusion model solution, essentially acting as a projection onto the intersection of data-consistent and diffusion model-consistent solutions.

The paper provides experimental results on both linear and nonlinear tasks and compares the proposed method with other existing techniques, showing performance improvements for certain tasks.
weaknesses: Modeling of Backward Consistency: The concept of backward consistency is interesting, but its implementation feels somewhat ad hoc. While it resembles a projection onto the intersection of data and prior-consistent solutions, this usually requires multiple iterations to reach a meaningful intersection. Here, however, only a single iteration is employed, which may be insufficient.

Alternative Approach: The backward consistency term could potentially be better represented as a fixed-point condition for the denoiser. This approach might be more systematic, aligning with methods like RED-diff, which regularize through denoising consistency.
Sufficiency of Eq. (4): Additionally, it's unclear if enforcing Eq. (4) (the Tweetie consistency) is sufficient to ensure backward consistency. This claim requires more justification.

Experimental Limitations:
- Comparison Scope: Tables 1 and 2 lack comparisons with key existing methods. Notably, the absence of comparisons with RED-diff and PGDM in Table 2 weakens the empirical analysis.

- Unfair Comparison: Simply reusing the hyperparameters from existing methods without adjusting them for the current dataset results in an unfair comparison. This could particularly impact performance for methods like DPS, RED-diff, and PGDM, which may not perform optimally without parameter tuning.

- Sampling Efficiency: A central claim of this work is the reduced number of iterations required by the proposed method. However, the paper lacks concrete experimental evidence to substantiate this, such as timing results that would demonstrate faster sampling speeds in seconds. Given the emphasis on efficiency, readers would expect clear evidence supporting this claim.

[PGDM] Song, J., Vahdat, A., Mardani, M., & Kautz, J. (2023, May). Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - How is the stopping criterion determined in practice? Does different $\delta$ influence the final output of SITCOM?
soundness: 2
strengths: - The paper is well-presented, with a clear and structured approach.
- The proposed algorithm introduces "forward consistency" and "backward consistency," which is a novel framework that may interest a broader research community.
- Experimental results on image restoration tasks are promising.
summary: This paper presents an optimization-based algorithm, SITCOM, to solve inverse problems with a pretrained diffusion model. The authors state three conditions to hold at each sampling step of the diffusion model. SITCOM optimizes measurement consistency on $x_t$ and uses a resampling procedure to enforce the consistency conditions. Experiments show that SITCOM performs better than existing methods in several image restoration tasks.
weaknesses: - The paper does not provide rigorous guarantees for the output of SITCOM, whereas several prior methods offer assurances on correct sampling from the desired posterior distribution [2,3,4,7].
- The technical novelty appears limited. The optimization of $x_t$ for measurement consistency using Tweedie’s approximation directly resembles DPS, while the resampling technique is derived from ReSample. The resulting algorithm appears to be a straightforward combination of existing methods.
- This paper lacks a comparison to many relevant works [1~8]. Some of them provide asymptotic exact posterior sampling [2,3,4,7], while [1] appears to achieve the proposed "backward consistency" as well. It seems strange that SITCOM is only compared with DPS, DDNM, and two closely related algorithms (DAPS, DCDP), which seemingly have not undergone peer review.
- The stopping criterion for preventing noise overfitting needs further justification. Given that the actual noise level $\sigma_y$ is typically unknown in practice, an ablation study on the algorithm's sensitivity to $\delta$ would be valuable.

### Reference
[1] Wang et al. "A Plug-in Method for Solving Inverse Problems with Diffusion Model." In NeurIPS 2024.

[2] Dou et al. "Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective." ICLR 2023.

[3] Gabriel et al. "Monte Carlo guided Denoising Diffusion models for Bayesian linear inverse problems." ICLR 2023.

[4] Wu et al. "Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors." In NeurIPS 2024.

[5] Rout et al. "Solving linear inverse problems provably via posterior sampling with latent diffusion models." In NeurIPS 2023.

[6] Song et al. "Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency." In ICLR 2024.

[7] Wu et al. "Practical and Asymptotically Exact Conditional Sampling in Diffusion Models." In NeurIPS 2023.

[8] Rout et al. "Beyond first-order Tweedie: Solving inverse problems using latent diffusion." In CVPR 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: The questions are integrated in the section above.
soundness: 2
strengths: The paper is well-written and easy to follow.
- It clearly motivates the issue of inconsistency in current models and provides an intuitive solution to address these inconsistencies.
- The proposed algorithm, which integrates these solutions, demonstrates improved performance over existing methods.

The authors showcase the applicability of their method across various image inverse problems.
summary: The authors identify limitations in current diffusion model-based inverse solvers, noting that the measurement-guidance term often pushes the trajectory toward inconsistency, leading to artifacts in intermediate samples and requiring a large number of correction steps.

To address these issues, they propose three conditions for achieving measurement-consistent diffusion trajectories: (1) standard data manifold measurement consistency, (2) forward diffusion consistency, and (3) backward diffusion consistency.

1. Measurement-consistent: Reconstruction is consistent with the measurements.
2. Forward-consistent: Intermediate samples during the diffusion trajectory should resemble in-distribution samples produced by forward diffusion.
3. Backward-consistent: The measurement-guided output, followed by Tweedie’s denoising, should also be the Tweedie-denoised output of some noisy data.

The authors introduce an algorithm that ensures all three consistencies, called Step-wise Triple-Consistent Sampling (SITCOM). The proposed method outperforms existing approaches in various image inverse problems.
weaknesses: The claims about consistency are not fully substantiated. It remains unclear if the proposed method genuinely addresses all three inconsistencies.
- Experimental validation is needed to quantify how effectively the method mitigates these inconsistencies. For example, could experiments be designed to measure the degree to which each inconsistency is reduced?

The impact of each individual component in the method is not verified.
- How would the results change if one component were removed?
- Additionally, could similar components be added to other algorithms, such as DPS or DDNM, to reduce inconsistencies in those methods as well? Would this approach similarly alleviate inconsistencies?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Please address my concerns in the Weaknesses part.
soundness: 1
strengths: 1. The intuitive explanation of the triple-consistency conditions is clear and reasonable. The paper identifies the potential factors that prevent existing methods from balancing quality and efficiency, summarizing as the triple-consistency conditions.

2. The paper consistently achieves comparable or superior empirical results across commonly used linear and nonlinear tasks as shown in Table 1.
summary: This paper identifies the key limitations in previous DMs for IPs and summaries, referred to as the triple-consistency conditions. The newly designed optimization-based method, SITCOM, achieves comparable or even superior restoration results with significantly reduced computational time. The paper demonstrates the superiority of the proposed method across commonly used benchmarks, including five linear tasks and three nonlinear tasks.
weaknesses: 1. **Incremental Modification and Unfair Comparisons**  
   The major issue with the paper is that it presents only an incremental modification over existing work and makes unfair comparisons.

   1. *Theoretical and Motivational Perspective*  
      Although the paper is the first to propose and formulate the triple-consistency conditions formally, each has already been individually adopted and utilized in other works. Specifically, measurement consistency is used in [4, 5, 6, 7], backward consistency in [1, 2, 3], and forward consistency in [4, 7]. The paper provides only intuitive explanations for each condition in Sections 3.1, 3.2, and 3.3, lacking clear evidence of how each contributes to the final restoration performance. A straightforward ablation study—removing each consistency condition from the proposed algorithm, SITCOM, and comparing performance with the full SITCOM while keeping other parameters—would give clearer hints of the significance of each condition. Unfortunately, the experiments fail to control variables, causing the conclusions of the paper potentially misleading.

   2. *Technical Perspective*  
      The proposed method, SITCOM (Algorithm 1 in the paper), differs from DCDP only in two aspects:
      * (1) Changing the optimization variable from $\hat{x}_0$ to $\hat{v}_i$, where these variables are connected by the Tweedie-network denoiser $\hat{x}_0' = f(\hat{v}_i; t, \theta)$.
      * (2) Adding an early stopping criterion (line 5 in Algorithm 1).

      My main concern is with (1). The paper claims this change is introduced to achieve backward consistency but comes at the cost of computing backward gradients through the Tweedie-network denoiser $f$. Consequently, when keeping other parameters the same (e.g., diffusion steps $N$, number of gradient updates $K$, stopping criterion $\delta$), SITCOM is significantly slower than DCDP due to the additional backward pass through $f$. However, the paper compares SITCOM with smaller $N$ and $K$ to DCDP with much larger $N$ and $N$ and concludes that SITCOM is more efficient in achieving comparable results. I strongly suggest that the authors conduct a detailed and fair comparison between SITCOM and DCDP by fixing all other parameters and only changing the optimization variables. Otherwise, it is unclear whether the observed efficiency gains are due to backward consistency or simply fewer diffusion and gradient steps.

2. **Other Minor Issues**
   1. The paper lacks comparison and discussion of several recently proposed methods [5, 6, 8].
   2. There is no formulation or experimentation on solving inverse problems with latent diffusion models, which are included in the baseline papers used [4, 7].
3. **Non-anonymous issue.** Files in the given code link [https://anonymous.4open.science/r/SITCOM-7539/README.md](https://anonymous.4open.science/r/SITCOM-7539/README.md) (`SITCOM.py`, `SITCOM_with_noise.py` and `SITCOM_with_noise_imagenet.py`, etc.)  contain non-anonymous information.

**References**

[1] Chung, Hyungjin, et al. "Diffusion Posterior Sampling for General Noisy Inverse Problems." *ICLR 2023*.

[2] Song, Jiaming, et al. "Pseudoinverse-Guided Diffusion Models for Inverse Problems." *ICLR 2023*.

[3] Song, Jiaming, et al. "Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation." *ICML 2023*.

[4] Li, Xiang, et al. "Decoupled Data Consistency with Diffusion Purification for Image Restoration." arXiv preprint arXiv:2403.06054 (2024).

[5] Wu, Zihui, et al. "Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors." arXiv preprint arXiv:2405.18782 (2024).

[6] Xu, Xingyu, et al. "Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstruction." arXiv preprint arXiv:2403.17042 (2024).

[7] Zhang, Bingliang, et al. "Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing." arXiv preprint arXiv:2407.01521 (2024).

[8] Dou, Zehao, et al. "Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective." *ICLR 2024*.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Could the authors provide additional intuition or examples for Definition 1 to make backward consistency more accessible?
2. Are there any performance curves that illustrate the effect of varying the number of optimization steps K and sampling steps N on both image quality and computational efficiency?
3. Can the authors elaborate on how SITCOM performs when using fewer sampling or optimization steps and if there are trade-offs between quality and computational cost?
4. Is there potential for SITCOM to be adapted into a deterministic sampler like DDIM? If so, what modifications would be needed?
5. How does the method handle noise characteristics that deviate significantly from those tested in the paper?
6. Are there theoretical justifications for the choice of optimization parameters, or is there an analysis showing the method's convergence properties?
Overall: This paper makes a meaningful contribution by introducing a theoretically rich and empirically validated approach for inverse problem-solving using diffusion models. The addition of backward consistency is an interesting theoretical insight, and the experimental results are compelling. However, the paper could benefit from clearer explanations, formal justifications for certain methodological choices, and more detailed computational metrics. Addressing these points would enhance the completeness and accessibility of the work.
soundness: 2
strengths: nnovative Theoretical Contributions: The concept of backward consistency (Definition 1) is an original addition that, along with measurement and forward consistency, forms a robust framework for inverse problem solvers using diffusion models. This mathematical insight is well-formulated and adds depth to the field (lines 216–269).
Optimization-Based Sampling: The detailed explanation of the optimization problem in Equation (S1) and the associated algorithm (lines 324–431) shows a strong command of integrating theoretical principles with practical implementation.
Comprehensive Experiments: The paper includes results on various tasks and datasets, demonstrating the robustness of the proposed approach (Table 1).
Efficiency Improvements: The reported reduction in run-time, while maintaining or improving performance, is a notable practical advantage (lines 486–539).
summary: The paper introduces SITCOM (Step-wise Triple-Consistent Sampling), a new framework for solving inverse problems using diffusion models. SITCOM enforces three consistency conditions—measurement, backward, and forward consistency—to guide the sampling process and improve the fidelity of reconstructions. The approach is validated through comprehensive experiments on tasks such as deblurring, super resolution, and in-painting, with results showing competitive performance and reduced run-time compared to baseline methods.
weaknesses: Clarity in Backward Consistency: Definition 1 (lines 216–269) could benefit from more intuitive explanations or examples to make the concept clearer to a broader audience.
Absence of Mathematical Justifications: While the theoretical contributions are solid, the paper lacks formal mathematical justifications for key aspects:
Choice of Optimization Parameters: There is no in-depth analysis on the selection of K (number of optimization steps) or λ (regularization parameter). Providing this would strengthen the confidence in the method's robustness (lines 324–431).
Necessity of Triple Consistency: The paper argues for the simultaneous use of measurement, backward, and forward consistency but does not include formal proof or theoretical analysis justifying why all three are essential for the claimed performance (lines 216–377).
Convergence Analysis: A formal convergence analysis is missing, which could help clarify the reliability and scalability of the SITCOM sampler. While the algorithm shows empirical success, having mathematical insights into its convergence properties and error bounds would offer more confidence in its reliability and scalability.
Mixed Results in Some Tasks: The improvements in tasks like ImageNet Gaussian Deblurring and Phase Retrieval are less significant compared to other baselines (Table 1), which may raise questions about the trade-offs in complexity versus marginal performance gains.
Lack of Computational Details: Detailed metrics like Number of Function Evaluations (NFE) are not presented in the main manuscript (it is located at appendix F). Including these would provide a clearer view of the computational cost. Please consider reorganize the contents.
Comparison to Deterministic Samplers: The paper does not explore whether the SITCOM method could be adapted into a deterministic version like DDIM or address how it compares to the optimized noise schedules like EDM.

Meta-review: Summary. This paper proposes to solve inverse problems using a plug-and-play approach with diffusion models. The proposed "triple consistency" framework adds a backward consistency component with already established data and forward consistencies. The backward consistency aims to ensure that the solution obtained after data-consistency remains a valid diffusion model solution. 

Strengths. The paper addresses a timely and important problem in inverse problem-solving using diffusion models. Experiments cover a variety of linear and nonlinear tasks, giving a broad perspective on the method's applicability. The paper is well-written. 

Weaknesses. Technical novelty of the paper appears limited. The proposed backward consistency approach seems ad hoc. The paper lacks comparison with some of the existing methods and additional experiments discussed during rebuttal show mixed results. Some reviewers argued that some of the comparisons seem unfair or unclear. The claims about consistency are not fully substantiated. It remains unclear if the proposed method genuinely addresses all three inconsistencies. 

Missing. 
The paper shares some similarities with other methods (mainly DCDP). A clear description of the differences and their importance is missing. 
The paper is mainly missing some convincing comparisons with existing methods that resemble the proposed method. 
The claims about consistency are not fully substantiated. It remains unclear if the proposed method genuinely addresses all three inconsistencies. 

Justification. 
Missing and mixed comparisons with related methods and discussion about similarities with DCDP were most important reasons for my decision.

Decision: Reject