Constrained Diffusion Implicit Models

## Abstract

Abstract This paper describes an efficient algorithm for solving noisy linear inverse problems using pretrained diffusion models. Extending the paradigm of denoising diffusion implicit models (DDIM), we propose constrained diffusion implicit models (CDIM) that modify the diffusion updates to enforce a constraint upon the final output.
For noiseless inverse problems, CDIM exactly satisfies the constraints; in the noisy case, we generalize CDIM to satisfy an exact constraint on the residual distribution of the noise. Experiments across a variety of tasks and metrics show strong performance of CDIM, with analogous inference acceleration to unconstrained DDIM: 10 10 10 10 to 50 50 50 50 times faster than previous conditional diffusion methods. We demonstrate the versatility of our approach on many problems including super-resolution, denoising, inpainting, deblurring, and 3D point cloud reconstruction.

## 1 Introduction

Denoising diffusion probabilistic models (DDPMs) have recently emerged as powerful generative models capable of capturing complex data distributions (Ho et al., ). Their success has spurred interest in applying them to solve inverse problems, which are fundamental in fields such as computer vision, medical imaging, and signal processing (Tropp & Wright, ; Hansen, ). Inverse problems require recovering unknown signals from (possibly noisy) observations. Linear inverse problems, where the observations consist of linear measurements of a signal, encompass tasks like super-resolution, inpainting, and deblurring.

Existing methods that apply diffusion models to linear inverse problems face several limitations. First, many previous works require task specific training or fine-tuning (Li et al., ; Xie et al., ). Second, methods that use pretrained diffusion models often introduce many additional network evaluations during inference (Dou & Song, ; Zhu et al., ). Finally, popular diffusion inverse methods such as diffusion posterior sampling (Chung et al., ) fail to exactly recover the input observations.

In this work, we propose constrained diffusion implicit models (CDIM), extending the inference acceleration of denoising diffusion implicit models (Song et al., ) to efficiently solve noisy linear inverse problems using a single pretrained diffusion model. Our method modifies the diffusion updates to enforce constraints on the final output, integrating measurement constraints directly into the diffusion process. In the noiseless case, this approach achieves exact recovery of the observations. For noisy observations, we generalize our method by optimizing the Kullback-Leibler (KL) divergence between the empirical residual distribution and a known noise distribution, effectively handling general noise models beyond the Gaussian assumption. Code can be found at our website(^1^11[https://grail.cs.washington.edu/projects/cdim/](https://grail.cs.washington.edu/projects/cdim/)).

Our contributions are as follows:

- •
Accelerated inference: we accelerate inference, reducing the number of model evaluations and wall-clock time by an order of magnitude—10 to 50 times faster than previous conditional diffusion methods—while maintaining comparable quality.
- •
Exact recovery of noiseless observations: we can find solutions that exactly match the noiseless observation.
- •
General noise models: we extend the CDIM framework to accommodate arbitrary observational noise distributions through distributional divergence minimization, demonstrating effectiveness given non-Gaussian noise, such as Poisson noise.

Figure: Figure 1: We show several applications of our method including image colorization, denoising, inpainting, and sparse recovery. We highlight the fact that we can handle general noise distributions, such as Poisson noise, and that our method runs in as little as 3 seconds.
Refer to caption: x1.png

## 2 Related Work

Figure: Figure 2: The inference speed and average LPIPS image quality score (inverted) averaged across multiple inverse tasks on the FFHQ dataset. The family of CDIM methods (top left corner) simultaneously achieves strong generation strong quality and fast inference, compared to other inverse solvers.
Refer to caption: extracted/5970634/images/performance_graph2.png

Diffusion methods have revolutionized generative modeling, building upon early work in nonequilibrium thermodynamics (Sohl-Dickstein et al., ) and implicit models (Mohamed & Lakshminarayanan, ). Diffusion models were first proposed in DDPM (Ho et al., ), which shared a framework analogous to score-based models using Langevin dynamics Song & Ermon (). Subsequent innovations focused on improving sampling efficiency, with denoising diffusion implicit models (DDIMs) (Song et al., ) introducing a method to speed up inference with implicit modeling. Further advancements in accelerating the sampling process emerged through the application of stochastic differential equations (Song et al., ) and the development of numerical ODE solvers, exemplified by approaches like PNDM (Liu et al., ), significantly enhancing the practical utility of diffusion models in various generative tasks.

Applying diffusion models to inverse problems has been an active research area. DPS uses alternating projection steps to guide the diffusion process (Chung et al., ). Methods like DDNM (Wang et al., ), DDRM (Kawar et al., ), SNIPS (Kawar et al., ), and PiGDM (Song et al., ) use linear algebraic approaches and singular value decompositions. Techniques such as DMPS (Meng & Kabashima, ), FPS (Dou & Song, ), LGD (Song et al., ), DPMC (Zhu et al., ), and MCG (Cardoso et al., ) focus on likelihood approximation for improved sampling. Guidance mechanisms were incorporated through classifier gradients (Dhariwal & Nichol, ), data consistency enforcement (Chung et al., ), and low-frequency feature matching Choi et al. (). Other approaches use projection (Boys et al., ; Chung et al., ) or optimization (Chan et al., ; Wahlberg et al., ).

## 3 background

We work in the context of DDPM (Ho et al., ), which models a data distribution $q({\mathbf{x}}_{0})$ by modeling a sequence $t=1,\dots,T$ of smoothed distributions defined by

$$ $q({\mathbf{x}}_{t}|{\mathbf{x}}_{0})=\mathcal{N}({\mathbf{x}}_{t};\sqrt{\alpha _{t}}{\mathbf{x}}_{0},(1-\alpha_{t}){\mathbf{I}}).$ (1) $$

The degree of smoothing is controlled by a monotone decreasing noise schedule $\alpha_{t}$ with $\alpha_{0}=1$ (no noise) and $\alpha_{T}=0$ (pure Gaussian noise).(^2^22We define $\alpha_{t}$ using the DDIM convention (Song et al., ); our $\alpha_{t}$ corresponds to $\bar{\alpha}_{t}$ in Ho et al. ().) The idea is to model a *reverse process* $p_{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})$ that that incrementally removes the noise in ${\mathbf{x}}_{t}$ such that $p_{\theta}({\mathbf{x}}_{T})=\mathcal{N}({\mathbf{x}}_{T};0,{\mathbf{I}})$ and $p({\mathbf{x}}_{0})$ approximates the data distribution, where $p({\mathbf{x}}_{0})$ is the marginal distribution of outputs from the reverse process:

$$ $p_{\theta}({\mathbf{x}}_{0})=\int p_{\theta}({\mathbf{x}}_{T})\prod_{t=1}^{T}p _{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})\,d{\mathbf{x}}_{1:T}.$ (2) $$

Given noisy samples ${\mathbf{x}}_{t}=\sqrt{\alpha_{t}}{\mathbf{x}}_{0}+\sqrt{1-\alpha_{t}}\bm{\epsilon}$, where ${\mathbf{x}}_{0}$ is a sample from the data distribution and $\bm{\epsilon}\sim\mathcal{N}(0,{\mathbf{I}})$, a diffusion model $\bm{\epsilon}_{\theta}({\mathbf{x}}_{t},t)$ is trained to predict $\bm{\epsilon}$:

$$ $\min_{\theta}\mathop{\mathbb{E}}_{{\mathbf{x}}_{t},\bm{\epsilon}}\left[\|\bm{ \epsilon}-\bm{\epsilon}_{\theta}\left({\mathbf{x}}_{t},t\right)\|^{2}\right].$ (3) $$

To parameterize the reverse process $p_{\theta}({\mathbf{x}}_{t-1}|{\mathbf{x}}_{t})$, DDIM (Song et al., ) exploits the Tweedie formula (Efron, ) for the posterior mean of a noisy observation:

$$ $\mathbb{E}\left[{\mathbf{x}}_{0}|{\mathbf{x}}_{t}\right]=\frac{1}{\sqrt{\alpha _{t}}}\left({\mathbf{x}}_{t}-\sqrt{1-\alpha_{t}}\nabla_{{\mathbf{x}}_{t}}\log q ({\mathbf{x}}_{t})\right).$ (4) $$

Using the denoising model $\bm{\epsilon}({\mathbf{x}}_{t},t)$ as a plug-in estimate of the score function $\nabla_{{\mathbf{x}}_{t}}\log q({\mathbf{x}}_{t})$ (Vincent, ) we define the Tweedie estimate of the posterior mean:

$$ $\hat{\mathbf{x}}_{0}\equiv\frac{1}{\sqrt{\alpha_{t}}}\left({\mathbf{x}}_{t}- \sqrt{1-\alpha_{t}}\bm{\epsilon}_{\theta}({\mathbf{x}}_{t},t)\right)\approx \mathbb{E}\left[{\mathbf{x}}_{0}|{\mathbf{x}}_{t}\right].$ (5) $$

And we use this estimator to define a DDIM forward process ${\mathbf{x}}_{t-1}=f_{\theta}({\mathbf{x}}_{t})$ defined by

$$ $f_{\theta}({\mathbf{x}}_{t})=\sqrt{\alpha_{t-1}}\hat{\mathbf{x}}_{0}+\sqrt{1- \alpha_{t-1}}\left(\frac{{\mathbf{x}}_{t}-\sqrt{\alpha_{t}}\hat{\mathbf{x}}_{0 }}{\sqrt{1-\alpha_{t}}}\right).$ (6) $$

Unlike DDPM, the forward process defined by Equation (6) is deterministic; the value $p_{\theta}({\mathbf{x}}_{0})$ is entirely determined by ${\mathbf{x}}_{T}\sim\mathcal{N}(0,{\mathbf{I}})$ thus making DDIM an implicit model.

With a slight modification of the DDIM update, we are able to take larger denoising steps and accelerate inference. Given $\delta\geq 1$, we define an accelerated denoising process

$$ $f^{\delta}_{\theta}({\mathbf{x}}_{t})=\sqrt{\alpha_{t-\delta}}\hat{\mathbf{x}} _{0}+\sqrt{1-\alpha_{t-\delta}}\left(\frac{{\mathbf{x}}_{t}-\sqrt{\alpha_{t}} \hat{\mathbf{x}}_{0}}{\sqrt{1-\alpha_{t}}}\right).$ (7) $$

Using this process, inference is completed in just $T^{\prime}\equiv T/\delta$ steps, albeit with degraded quality of the resulting sample ${\mathbf{x}}_{0}$ as $\delta$ becomes large.

## 4 methods

We are interested in solving linear inverse problems of the form ${\mathbf{y}}={\mathbf{A}}{\mathbf{x}}$, where ${\mathbf{y}}\in\mathbb{R}^{d}$ is a linear measurement of ${\mathbf{x}}\in\mathbb{R}^{n}$ and ${\mathbf{A}}\in\mathbb{R}^{d\times n}$ describes the form of our measurements. For example, if ${\mathbf{A}}\in\{0,1\}^{n\times n}$ is a binary mask (which is the case for, e.g., in-painting or sparse recovery problems) then ${\mathbf{y}}$ describes a partial observation of ${\mathbf{x}}$. We seek an estimate $\hat{\mathbf{x}}$ that is consistent with our observations: in the noiseless case, ${\mathbf{A}}\hat{\mathbf{x}}={\mathbf{y}}$. More generally, we seek to recover a robust estimate of $\hat{\mathbf{x}}$ when the observations ${\mathbf{y}}$ have been corrupted by noise. Given a noise distribution $r$, we seek to minimize $D_{\mathrm{KL}}(\hat{r}\;\|\;r)$, where $\hat{r}$ is the empirical distribution of $d$ residuals, e.g., ${\mathbf{y}}-{\mathbf{A}}\hat{\mathbf{x}}\in\mathbb{R}^{d}$, between noisy observations ${\mathbf{y}}$ and our estimates ${\mathbf{A}}\hat{\mathbf{x}}$.

We rely on a diffusion model $p_{\theta}({\mathbf{x}})$ to identify an estimate $\hat{\mathbf{x}}$ that is both consistent with the observed measurements ${\mathbf{y}}$ and likely according to the model. In Section 4.1, we propose a modification of the DDIM inference procedure to efficiently optimize the Tweedie estimates of $\hat{\mathbf{x}}_{0}$ to satisfy ${\mathbf{A}}\hat{\mathbf{x}}_{0}={\mathbf{y}}$ during the diffusion process, resulting in a consistent and likely final result ${\mathbf{x}}_{0}$. In Section 4.2 we extend this optimization-based inference procedure to account for noise in the observations ${\mathbf{y}}$. In Section 4.3 we describe an early-stopping heuristic to avoid overfitting to noisy observations, which further reduces the cost of inference. Finally, in Section 4.4 we show how to set the step sizes for these optimization-based methods.

### 4.1 Optimizing 𝐱 ^ 0 subscript ^ 𝐱 0 \hat{\mathbf{x}}_{0} over^ start_ARG bold_x end_ARG start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT to match the observations

For linear measurements ${\mathbf{A}}$, the Tweedie formula for $\hat{\mathbf{x}}_{0}$ (and the corresponding plugin-estimate Equation (5)) extends to a formula for the expected observations:

$$ $\mathbb{E}\left[{\mathbf{y}}|{\mathbf{x}}_{t}\right]={\mathbf{A}}\mathbb{E} \left[{\mathbf{x}}_{0}|{\mathbf{x}}_{t}\right]\approx{\mathbf{A}}\hat{\mathbf{ x}}_{0}.$ (8) $$

For noiseless observations ${\mathbf{y}}$, we propose a modification of the DDIM updates Equation (6) to find ${\mathbf{x}}_{t-1}$ that satisfies the constraint ${\mathbf{A}}\hat{\mathbf{x}}_{0}={\mathbf{y}}$. I.e., at each time step $t$, we force the Tweedie estimate of the posterior mean of $q({\mathbf{y}}|{\mathbf{x}}_{t})$ to match the observed measurements ${\mathbf{y}}$:

$$ $\displaystyle\begin{aligned} \operatorname*{arg\,min}_{{\mathbf{x}}_{t-1}} \quad&\|{\mathbf{x}}_{t-1}-f_{\theta}({\mathbf{x}}_{t})\|^{2}\\ \text{subject to}\quad&{\mathbf{A}}\hat{\mathbf{x}}_{0}={\mathbf{y}}.\end{aligned}$ (9) $$

We can interpret Equation (9) as a projection of the DDIM update $f_{\theta}({\mathbf{x}}_{t})$ onto the set of values ${\mathbf{x}}_{t-1}$ that satisfy the constraint ${\mathbf{A}}\hat{\mathbf{x}}_{0}={\mathbf{y}}$. The full inference procedure is analogous to projected gradient descent, whereby we alternately take a step $f_{\theta}({\mathbf{x}}_{t})$ determined by the diffusion model, and then project back onto the constraint ${\mathbf{A}}\hat{\mathbf{x}}_{0}={\mathbf{y}}$. We implement the projection step itself via gradient descent, initialized from ${\mathbf{x}}_{t-1}^{(0)}=f_{\theta}({\mathbf{x}}_{t})$ and computing

$$ ${\mathbf{x}}_{t-1}^{(k)}={\mathbf{x}}_{t-1}^{(k-1)}+\eta\nabla_{{\mathbf{x}}_{ t-1}}\|{\mathbf{y}}-{\mathbf{A}}\hat{\mathbf{x}}_{0}\|^{2}.$ (10) $$

As $t$ approaches 0 0, $\hat{\mathbf{x}}_{0}$ converges to ${\mathbf{x}}_{0}$ and $\|{\mathbf{y}}-{\mathbf{A}}\hat{\mathbf{x}}_{0}\|^{2}$ becomes a simple convex quadratic, which can be minimized to arbitrary accuracy by taking sufficiently many gradient steps. This allows us to guarantee exact recovery of the observations ${\mathbf{y}}={\mathbf{A}}{\mathbf{x}}_{0}$ in the recovered inverse ${\mathbf{x}}_{0}$.

For $t$ close to $T$, we face two conceptual challenges in optimizing Equation (9). First, for large $t$, no value ${\mathbf{x}}_{t}$ will satisfy ${\mathbf{A}}\hat{\mathbf{x}}_{0}={\mathbf{y}}$ and therefore the optimization is infeasible. Second, the estimate of the score function $\nabla_{{\mathbf{x}}_{t}}\log q({\mathbf{x}}_{t})$ using $\bm{\epsilon}_{\theta}({\mathbf{x}}_{t},t)$ may be inaccurate; we risk overfitting to a bad plug-in estimate $\hat{\mathbf{x}}_{0}$. We illustrate both these claims by considering the Tweedie estimator Equation (5) in the case $t=T$. In this case, ${\mathbf{x}}_{t}\sim\mathcal{N}(0,I)$ is independent of ${\mathbf{x}}_{0}$ and therefore $\mathbb{E}[{\mathbf{x}}_{0}|{\mathbf{x}}_{t}]=\mathbb{E}[{\mathbf{x}}_{0}]$, the mean of the data distribution $q({\mathbf{x}}_{0})$. Unless ${\mathbf{A}}\mathbb{E}[{\mathbf{x}}_{0}]={\mathbf{y}}$, the optimization is infeasible when $t=T$. Furthermore, we observe that when $t=T$, the plug-in estimator $\hat{\mathbf{x}}_{0}$ is not independent of ${\mathbf{x}}_{t}$ and $\hat{\mathbf{x}}_{0}\neq\mathbb{E}[{\mathbf{x}}_{0}]$. This is indicative of error in the plug-in estimator, especially at high noise levels.

In light of these observations, we replace Equation (9) with a Lagrangian

$$ $\operatorname*{arg\,min}_{{\mathbf{x}}_{t-1}}\|{\mathbf{x}}_{t-1}-f_{\theta}({ \mathbf{x}}_{t})\|^{2}+\lambda\|{\mathbf{y}}-{\mathbf{A}}\hat{\mathbf{x}}_{0} \|^{2}.$ (11) $$

We can interpret Equation (11) as a relaxation of Equation (9); the regularization by $\lambda\|{\mathbf{y}}-{\mathbf{A}}\hat{\mathbf{x}}_{0}\|^{2}$ is achieved implicitly by early stopping after $k=K$ steps of gradient descent. In contrast to projection, this Lagrangian objective is robust to both (1) the possible infeasibility of $\hat{\mathbf{y}}_{0}({\mathbf{x}}_{t})={\mathbf{y}}$ and (2) overfitting the measurements based on an inaccurate Tweedie plug-in estimator.

### 4.2 Optimizing the KL Divergence of Residuals

For noisy inverse problems, imposing a hard constraint ${\mathbf{A}}\hat{{\mathbf{x}}}_{0}={\mathbf{y}}$ will overfit to the noise $\bm{\sigma}$ in the observations, as illustrated by Figure 4.
Previous work accounts for noise using implicit regularization, by incompletely optimizing the objective ${\mathbf{A}}\hat{\mathbf{x}}_{0}={\mathbf{y}}$ (Chung et al., ). In contrast, we propose to exactly optimize the Kullback-Leibler (KL) divergence between the empirical distribution of residuals $R({\mathbf{A}}\hat{\mathbf{x}}_{0},{\mathbf{y}})$ and a known, i.i.d. noise distribution $r$:

$$ $\displaystyle\begin{aligned} \operatorname*{arg\,min}_{{\mathbf{x}}}\quad&\|{ \mathbf{x}}-{\mathbf{x}}_{t}\|^{2}\\ \text{subject to}\quad&D_{\mathrm{KL}}(R({\mathbf{A}}\hat{\mathbf{x}}_{0},{ \mathbf{y}})\;\|\;r)=0.\end{aligned}$ (12) $$

In Algorithm 1, we show how to optimize a constraint on categorical KL divergences to match arbitrary distributions of discretized residuals. We also provide a convenient objective for optimizing the empirical distribution of continuous residuals to match common noise patterns, including Gaussian and Poisson noise.

Figure: Algorithm 1 Constrained Diffusion Implicit Models with KL Constraints

Additive Noise. The general additive noise model is defined by ${\mathbf{y}}={\mathbf{A}}{\mathbf{x}}+\bm{\sigma}\in\mathbb{R}^{d}$, where $\bm{\sigma}\sim r^{\otimes d}$. By discretizing the distribution of residuals into $B$ buckets, we can compute a categorical KL divergence between observed residuals and the discrete approximation of $r_{B}$ of $r$:

$$ $D_{\mathrm{KL}}(R({\mathbf{A}}\hat{\mathbf{x}}_{0},{\mathbf{y}})\;\|\;r_{L})= \sum_{b=1}^{B}r_{B}(b)\log\left(\frac{r_{B}(b)}{\lfloor R({\mathbf{A}}\hat{ \mathbf{x}}_{0},{\mathbf{y}})\rfloor_{B}}\right).$ (13) $$

Figure: Input: Box inpainting with bimodal noise
Refer to caption: extracted/5970634/images/weird_noise_input.png

In Figure 3 we show results on the box inpainting task when the observation has been corrupted with bimodal noise: $p(\bm{\sigma}_{i}=-0.75)=p(\bm{\sigma}_{i}=0.75)=0.5$ for $i=1,\dots,n$, where image pixels are normalized values ${\mathbf{x}}_{i}\in[-1,1]$. We optimize the residuals using the discrete KL divergence and show that our result faithfully reconstructs the ground truth with high fidelity while filling in the missing section.

Gaussian Noise.
Additive Gaussian noise is defined by $\bm{\sigma}\sim\mathcal{N}(0,\sigma^{2}{\mathbf{I}})$, in which case the residuals $R({\mathbf{A}}{\mathbf{x}},{\mathbf{y}})\equiv{\mathbf{y}}-{\mathbf{A}}{
\mathbf{x}}\sim\mathcal{N}(0,\sigma^{2}{\mathbf{I}})$ are i.i.d. with distribution $r\sim\mathcal{N}(0,\sigma^{2}$). The empirical mean and variance of the residuals are

$$ $\hat{\mu}=\frac{1}{d}\sum_{i=1}^{d}R({\mathbf{A}}\hat{\mathbf{x}},{\mathbf{y}} )_{i},\quad\hat{\sigma}^{2}=\frac{1}{d}\sum_{i=1}^{k}d\left(R({\mathbf{A}}\hat {\mathbf{x}},{\mathbf{y}})_{i}-\hat{\mu}\right)^{2}.$ (14) $$

Using the analytical formula for KL divergence between two Gaussians (Kingma & Welling, ), we can match the empirical mean and variance of the residuals to $r$ by enforcing

$$ $D_{\mathrm{KL}}(R({\mathbf{A}}\hat{\mathbf{x}}_{0},{\mathbf{y}})\;\|\;r)=\log \left(\frac{\sigma^{2}}{\hat{\sigma}^{2}}\right)+\frac{\hat{\sigma}^{2}+\hat{ \mu}^{2}}{2\sigma^{2}}-\frac{1}{2}=0.$ (15) $$

Poisson Noise.
Possion noise is non-additive noise defined by $s{\mathbf{y}}\sim\textrm{Poisson}(s{\mathbf{A}}{\mathbf{x}})$, where ${\mathbf{y}}$ is interpreted as discrete integer pixel values. The scaling factor $s\leq 1$ controls the degree of Poisson noise. Poisson noise is not identically distributed across ${\mathbf{y}}$; the variance increases with the scale of each observation. To remedy this, we consider the Pearson residuals (Pregibon, ):

$$ $R({\mathbf{A}}\hat{\mathbf{x}}_{0},{\mathbf{y}})=\frac{\lambda({\mathbf{y}}-{ \mathbf{A}}\hat{\mathbf{x}}_{0})}{\sqrt{\lambda\hat{\mathbf{x}}_{0}}}.$ (16) $$

These residuals are identically distributed; moreover, they are approximately normal $r\sim\mathcal{N}(0,1)$ (Pierce & Schafer, ). We can therefore optimize the KL divergence between Pearson residuals and a standard normal using Equation (15) to solve inverse problems with Poisson noise. Although the Pearson residuals closely follow the standard normal distribution for positive values of $\hat{\mathbf{x}}_{0}$, this breaks down for values of $\hat{\mathbf{x}}_{0}$ close to zero, and extreme noise levels $s$. In practice we find the Gaussian assumption to be valid for natural images corrupted by as much noise as $s\approx 0.025$. In Figure 1 we show an example of denoising an image corrupted by Poisson noise with $s=0.05$.

### 4.3 Noise-Agnostic Constraints

In many practical situations, we will not know the precise distribution of noise $r$ in the observations. For these cases, we propose a noise-agnostic version of CDIM, assuming only that the noise is zero-mean with variance $\mathrm{Var}(r)$. The idea is to directly minimize the squared error of the residuals, with early stopping to avoid overfitting to the noise once $\mathrm{Var}(r)$ exceeds the empirical variance of the residuals. In experiments, we find that this noise-agnostic algorithm performs similarly to the noise-aware versions described in Section 4.2. Moreover, the noise-agnostic algorithm is more efficient: by stopping early with enforcement of the constraint, it avoids excess evaluations of the model during the final steps of the diffusion process. The complete process is shown in Algorithm 2.

Figure: (a)
Refer to caption: extracted/5970634/images/noisy_input.png

Figure: Algorithm 2 Constrained Diffusion Implicit Models with $L^{2}$ Constraints and Early Stopping

### 4.4 Choice of Step Size η 𝜂 \eta italic_η

An important hyperparameter of these algorithms is the step size $\eta$. DPS sets $\eta$ proportional to $1/\|{\mathbf{y}}-{\mathbf{A}}\hat{{\mathbf{x}}}_{0}\|$ Chung et al. (). We find that this fails to converge for KL optimization, and also produces unstable results for $L^{2}$ optimization when $T^{\prime}$ is small. One option is to set $\eta$ inversely proportional to the magnitude of the gradient $\|\nabla{\mathbf{x}}_{t-\delta}\|$ at every single optimization step. Although this is the easiest solution, it can result in unstable oscillations and slower convergence. Instead, we propose to set $\eta$ inversely proportional to $\mathbb{E}_{\mathbf{x}\sim\mathcal{X}_{\text{train}}}\left\|\nabla_{\mathbf{x}
_{t-\delta}}\right\|$, a common optimization heuristic (Amari, ; Pascanu & Bengio, ). In Appendix A we describe how to compute this expectation. In Figure 5 we show qualitatively what happens with different $\eta$ schedules.

Figure: Input, 50% inpainting
Refer to caption: extracted/5970634/images/eta_input.png

We find that for a specific optimization objective and task, the magnitude of the gradient $\|\nabla{\mathbf{x}}_{t-\delta}\|$ is highly similar across data points, datasets, and model architectures. While it is difficult to reason analytically about these magnitudes due to backpropagation through the network $\bm{\epsilon}({\mathbf{x}}_{t},t)$, we empirically demonstrate this observation in Appendix A. This suggests that a learned step size based on $\mathbb{E}_{\mathbf{x}\sim\mathcal{X}_{\text{train}}}\left\|\nabla_{\mathbf{x}
_{t-\delta}}\right\|$ generalizes as a good learning rate for unseen data. For all experiments, we estimate these magnitudes from FFHQ training data.

## 5 Results and Experiments

We conduct experiments to understand the efficiency and quality of CDIM across various tasks and datasets. In Section 5.1, we present quantitative comparisons to state-of-the-art approaches, followed by ablation studies in Section 5.2 examining inference speed and hyperparameters. In Section 5.3 we explore two novel applications of diffusion models for inverse problems.

### 5.1 Numerical Results on FFHQ and ImageNet

We evaluate CDIM on the FFHQ-1k (Karras et al., ) and ImageNet-1k (Russakovsky et al., ) validation sets, both widely used benchmarks for assessing diffusion methods for inverse problems. Each dataset contains 256 × 256 RGB images scaled to the range [0, 1]. The tasks include 4x super-resolution, box inpainting, Gaussian deblur, and random inpainting. Details of each task are included in the appendix. For all tasks, we apply zero-centered Gaussian observational noise with $\sigma=0.05$. To ensure fair comparisons, we use identical pre-trained diffusion models used in the baseline methods: for FFHQ we use the network from Chung et al. () and for ImageNet we use the network from Dhariwal & Nichol (). We use multiple metrics to measure the quality of the generated outputs: Frechet Inception Distance (FID) (Heusel et al., ) and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., ).
All experiments are carried out on a single Nvidia A100 GPU.

In Table 1 we compare CDIM with several other inverse solvers using the FID and LPIPS metrics on the FFHQ dataset. We present results using both our KL divergence optimization method (Algorithm 1) and our $L^{2}$ optimization method (Algorithm 2) with early stopping. For these experiments, we present results with $T^{\prime}=50$ and $K=3$ as well as $T^{\prime}=25$ and $K=1$ labeled as ”fast”. For ImageNet results please see Appendix B.3.

**Table 1: Quantitative results (FID, LPIPS) of our model and existing models on various linear inverse problems on FFHQ 256 × 256-1k validation dataset. (Lower is better). The best result is in bold and the second best is underlined.**
| FFHQ | Super | Inpainting | Gaussian | Inpainting | Runtime |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Res | (box) | Deblur | (random) | (seconds) |  |  |  |  |  |
| Methods | FID | LPIPS | FID | LPIPS | FID | LPIPS | FID | LPIPS |  |
| Ours - KL fast | 36.76 | 0.283 | 35.15 | 0.2239 | 37.44 | 0.308 | 35.73 | 0.259 | 2.57 |
| Ours - $L^{2}$ fast | 33.87 | 0.276 | 27.51 | 0.1872 | 34.18 | 0.276 | 29.67 | 0.243 | 2.4 |
| Ours - KL | 34.71 | 0.269 | 30.88 | 0.1934 | 35.93 | 0.296 | 31.09 | 0.249 | 10.2 |
| Ours - $L^{2}$ | 31.54 | 0.269 | 26.09 | 0.196 | 29.68 | 0.252 | 28.52 | 0.240 | 9.0 |
| FPS-SMC | 26.62 | 0.210 | 26.51 | 0.150 | 29.97 | 0.253 | 33.10 | 0.275 | 116.90 |
| DPS | 39.35 | 0.214 | 33.12 | 0.168 | 44.05 | 0.257 | 21.19 | 0.212 | 70.42 |
| DDRM | 62.15 | 0.294 | 42.93 | 0.204 | 74.92 | 0.332 | 69.71 | 0.587 | 2.0 |
| MCG | 87.64 | 0.520 | 40.11 | 0.309 | 101.2 | 0.340 | 29.26 | 0.286 | 73.2 |
| PnP-ADMM | 66.52 | 0.353 | 151.9 | 0.406 | 90.42 | 0.441 | 123.6 | 0.692 | 3.595 |
| Score-SDE | 96.72 | 0.563 | 60.06 | 0.331 | 109.0 | 0.403 | 76.54 | 0.612 | 32.39 |
| ADMM-TV | 110.6 | 0.428 | 68.94 | 0.322 | 186.7 | 0.507 | 181.5 | 0.463 | - |

### 5.2 Ablation Studies

Number of Inference Steps. CDIM offers the flexibility to trade off quality for faster inference time on demand. We investigate how generation quality changes as we vary the total computational budget during inference. Recall that the total number of network passes during inference is $T^{\prime}(K+1)$, where $T^{\prime}$ is the number of denoising steps and $K$ is the number of optimization steps per denoising step. We use the random inpainting task on the FFHQ dataset with the setup described in the previous section. For this experiment we use KL optimization (Algorithm 1). The total network forward passes are varied from $200$ to $20$, and we show qualitative results. Notably, CDIM yields high quality samples with as few as $50$ total inference steps, with quality degradations after that.

Figure: Input Random Inpainting
Refer to caption: extracted/5970634/images/ablation_2_input.png

$\mathbf{T^{\prime}}$ vs $\mathbf{K}$ trade-off. We consider the optimal balance between $T^{\prime}$ and $K$ when the total number of inference steps $T^{\prime}(K+1)$ is fixed. Using the random inpainting task on the FFHQ dataset with the previously described setup, we set $T^{\prime}(K+1)=200$ and analyze how PSNR, FID, and LPIPS change based on the chosen $T^{\prime}$ and $K$ values. Results are plotted in Figure 7. FID results consistently favor the maximum number of denoising steps $T^{\prime}$ with minimal optimization steps $K$. This is because FID evaluates overall distribution similarity rather than per-sample fidelity, and thus is not penalized by lower reconstruction-observation fidelity. In contrast, PSNR and LPIPS, which measure per-sample fidelity with respect to a reference image, achieve optimal results with a balanced mix of denoising and optimization steps.

Refer to caption: extracted/5970634/images/tk_fid.png

### 5.3 Additional Applications

Time-Travel Rephotography In Figure 1 we showcase an application of time-travel rephotography Luo et al. (). Antique cameras lack red light sensitivity, exaggerating wrinkles by filtering
out skin subsurface scatter which occurs mostly in the red channel. To address this, we input the observed image into the blue color channel and use the pretrained FFHQ model with Algorithm 2 to project the face into the space of modern images. We further emphasize the power of our approach; Luo et al. () trained a specialized model for this task while we are able to use a pretrained model without modification.

Sparse Point Cloud Reconstruction For this task, 20 different images from a scene in The Grand Budapest Hotel scene were entered into Colmap (Schönberger & Frahm, ) to generate a sparse 3D point cloud. Note that the sparse nature of the Colmap point cloud means that projections of the point cloud will have roughly $90\%$ of the pixels missing. Furthermore, the observations often contain significant amounts of non-Gaussian noise due to false correspondences. We can formulate this as a noisy inpainting problem and use our method to fill in the missing pixels for a desired viewpoint. To address the errors in the point cloud, we use Algorithm 2 along with a variance threshold that adequately captures the imprecise nature of the point cloud. We showcase the results in Figure 8.

Figure: (a)
Refer to caption: extracted/5970634/images/3d_recon_input.png

## 6 conclusion

In this paper we introduced CDIM, a new approach for solving noisy linear inverse problems with pretrained diffusion models. This is achieved by exploiting the structure of the DDIM inference procedure. By projecting the DDIM updates, such that Tweedie estimates of the denoised image $\hat{\mathbf{x}}_{0}$ match the linear constraints, we are able to enforce constraints without making out-of-distribution edits to the noised iterates ${\mathbf{x}}_{t}$. We note that our method cannot handle non-linear constraints, including latent diffusion, because for a non-linear function $h$, $\mathbb{E}\left[h({\mathbf{x}}_{0})\right]\neq h(\mathbb{E}\left[{\mathbf{x}}_
{0}\right]$. Therefore, unlike the linear case of Equation (8), we cannot extend Tweedie’s estimate of the posterior mean of ${\mathbf{x}}_{0}$ to an estimate of the posterior mean of non-linear observations $h({\mathbf{x}}_{0})$. However, for linear constraints, our method generates high quality images with faster inference than previous methods, creating a new point on the Pareto-frontier of quality vs. efficiency for linear inverse problems.

## 7 Acknowledgements

This work was supported by the UW Reality Lab, Lenovo, Meta,
Google, OPPO, and Amazon.

## Appendix A Calculationg 𝔼 ⁢ ‖ ∇ 𝐱 t − δ ‖ 𝔼 norm subscript ∇ subscript 𝐱 𝑡 𝛿 \mathbb{E}\left\|\nabla_{\mathbf{x}_{t-\delta}}\right\| blackboard_E ∥ ∇ start_POSTSUBSCRIPT bold_x start_POSTSUBSCRIPT italic_t - italic_δ end_POSTSUBSCRIPT end_POSTSUBSCRIPT ∥

To calculate our expected gradient magnitude, we first start with simple gradient normalization: $\eta\leftarrow 1/\left\|\nabla_{\mathbf{x}_{t-\delta}}\right\|$, which normalizes our step size by the gradient magnitude on the fly at every optimization step. We run the full CDIM algorithm on the target task with the desired number of steps $T$ and $K$ on images from the training set. We calculate and store each gradient magnitude $\left\|\nabla_{\mathbf{x}_{t-\delta}}\right\|$ during the optimization process at every step. Finally, we average the empirical gradient magnitudes at each step $t-\delta$ to find $\mathbb{E}\left\|\nabla_{\mathbf{x}_{t-\delta}}\right\|$ across data points and inner optimization steps $k$. In practice we find that very few images are required to calculate a stable value for the expected gradient magnitude. In all experiments the value was calculated by running an initial optimization on $10$ images from the training set.

Figure: Figure 9: A plot of $||\nabla_{\mathbf{x}_{t-\delta}}||$ for two models and datasets, ImageNet and FFHQ. In each task 100 images were used. First, note the variance in a single task/model, shown by the error bars, is small. Second, note that the variance across the two tasks/models is also small.
Refer to caption: extracted/5970634/images/grad_magnitude_comparison.png

## Appendix B Additional Experimental Details

### B.1 Task Details

We describe additional details for each inverse task used in our experiments.

Super Resolution Images are downsampled to $64\times 64$ using bicubic downsampling with a factor of $4$.
Box Inpainting A random box of size $128\times 128$ is chosen uniformly within the image. Those pixels are masked out affected all three of the RGB channels.
Gaussian Deblur A Gaussian Kernel of size $61x61$ and intensity $3$ is applied to the entire image.
Random Inpainting Each pixel is masked out with probability $92\%$ affecting all three of the RGB channels
$\mathbf{50\%}$ Inpainting In various figures, we showcase a a $50\%$ inpainting task where the top half of an image is masked out. This task is more challenging than box inpainting and can better illustrate differences between results.

### B.2 Measuring Runtime

To measure wall-clock runtime, we used a single A100 and ran all the inverse problems (super-resolution, box inpainting, gaussian deblur, random inpainting) on the FFHQ dataset. We only consider the runtime of the algorithm, without considering the python initialization time, model loading, or image io. For each task, we measured the runtime on 10 images and averaged the result to produce the final result. We note that the baseline runtimes are taken from Dou & Song (), where only the box inpainting task was considered. The runtime does not vary much between tasks when using CDIM, so we report our average runtime across tasks as a fair comparison metric.

### B.3 ImageNet Results

In table 2 we report FID and LPIPS for the ImageNet dataset.

**Table 2: Quantitative results (FID, LPIPS) of our model and existing models on various linear inverse problems on the Imagenet 256 × 256-1k validation dataset. (Lower is better)**
| Imagenet | Super | Inpainting | Gaussian | Inpainting |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Resolution | (box) | Deblur | (random) |  |  |  |  |  |
| Methods | FID | LPIPS | FID | LPIPS | FID | LPIPS | FID | LPIPS |
| CDIM - KL fast | 59.10 | 0.398 | 58.75 | 0.311 | 73.74 | 0.480 | 53.91 | 0.364 |
| CDIM - L2 fast | 53.70 | 0.378 | 52.00 | 0.267 | 56.10 | 0.393 | 51.96 | 0.370 |
| CDIM - KL | 47.77 | 0.347 | 48.26 | 0.2348 | 57.72 | 0.390 | 45.86 | 0.331 |
| CDIM - L2 | 47.45 | 0.339 | 50.31 | 0.251 | 38.69 | 0.347 | 46.20 | 0.332 |
| FPS-SMC | 47.30 | 0.316 | 33.24 | 0.212 | 54.21 | 0.403 | 42.77 | 0.328 |
| DPS | 50.66 | 0.337 | 38.82 | 0.262 | 62.72 | 0.444 | 35.87 | 0.303 |
| DDRM | 59.57 | 0.339 | 45.95 | 0.245 | 63.02 | 0.427 | 114.9 | 0.665 |
| MCG | 144.5 | 0.637 | 39.74 | 0.330 | 95.04 | 0.550 | 39.19 | 0.414 |
| PnP-ADMM | 97.27 | 0.433 | 78.24 | 0.367 | 100.6 | 0.519 | 114.7 | 0.677 |
| Score-SDE | 170.7 | 0.701 | 54.07 | 0.354 | 120.3 | 0.667 | 127.1 | 0.659 |
| ADMM-TV | 130.9 | 0.523 | 87.69 | 0.319 | 155.7 | 0.588 | 189.3 | 0.510 |

### B.4 Extended Results

Figure: Figure 10: FFHQ Super-resolution extended results
Refer to caption: extracted/5970634/images/extended_results_ffhq_superres.png

Figure: Figure 11: FFHQ Gaussian deblur extended results
Refer to caption: extracted/5970634/images/extended_results_ffhq_gaussian.png

Figure: Figure 12: FFHQ random inpainting extended results
Refer to caption: extracted/5970634/images/extended_results_ffhq_inpainting_random.png

Figure: Figure 13: ImageNet Gaussian deblur extended results
Refer to caption: extracted/5970634/images/extended_results_imagenet_gaussian.png

Figure: Figure 14: ImageNet random inpainting extended results
Refer to caption: extracted/5970634/images/extended_results_imagenet_inpainting_random.png

Figure: Figure 15: ImageNet box inpainting extended results
Refer to caption: extracted/5970634/images/extended_results_imagenet_inpainting_box.png

Figure: (a)
Refer to caption: extracted/5970634/images/church_top.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. **Naming Consistency**: The model is referred to as "conditional diffusion implicit models (CDIM)" within the text, yet the title uses "Constrained Diffusion Implicit Models." Additionally, there are existing conditional diffusion models, which may cause some confusion. Consistent naming throughout the paper could help to avoid this.
2. **Typos**:
    - Page 9, line 482: missing closing parentheses.
    - Page 13, line 672: "A. CALCULATIONG" (should be "CALCULATING").
    - Page 13, line 696: "A Gaussian Kernel of size '61x61' ~".
3. **Quantitative Results for Additional Applications**: For the Additional Applications section (Time-Travel Rephotography, Sparse Point Cloud Reconstruction), providing quantitative results would add further value and demonstrate the method's effectiveness.
4. **Highlighting Advantages of Using Only Pretrained Models**: The method reportedly improves certain aspects without additional modules, relying solely on pretrained models. Emphasizing this advantage more prominently could strengthen the appeal of the method.
5. **PSNR Measurements**: Table 1 currently lacks PSNR measurements. Including these would allow for more comprehensive performance assessment.
6. **Choice of Step Size in Section 4.4**: The paper notes that the DPS method fails in this context but doesn’t provide detailed reasons. A more thorough explanation here would be appreciated to clarify the underlying issues.
soundness: 2
strengths: The authors have presented a promising approach, CDIM, that demonstrates notable improvements over existing methods. Key strengths include:

- **Efficiency**: The CDIM method shows a faster wall-clock time than other DDPM-based approaches (e.g., DPS) and achieves better performance metrics compared to DDIM-based methods (e.g., DDRM).
- **Exact Recovery for Noiseless Observations**: By incorporating the inverse relationship directly into the diffusion process as a constraint, the method can achieve exact recovery in the case of noiseless observations.
- **General Noise Model Applicability**: CDIM also addresses scenarios involving general noise models, broadening its potential use cases.
summary: This paper suggests a new model, Conditional Diffusion Implicit Models(CDIM) for solving linear inverse problem with pretrained diffusion models.
CDIM can address a problem whether it is noisy or not in linear cases. By imposing constraint on the prior diffusion objective, it solves a linear inverse problem efficiently in both time and utility.
Also for more efficient convergence, this paper utilizes Early Stopping and adaptive learning rate.
In experiments, it shows that it is fast, powerful and easy to use pretrained diffusion models without additional modules.
weaknesses: While the paper has several strengths, there are some areas where further clarification and refinement would enhance its impact and precision:

- **Early Stopping Criterion**: The paper suggests that the method handles unknown noise by utilizing early stopping based on the variance of residuals. However, the rationale for selecting the variance of residuals as an early-stopping criterion could benefit from a more detailed explanation. Additionally, the logical connection between noiseless methods (which aim to minimize KL divergence) and the noise-agnostic method (which minimizes squared error via early stopping) feels less cohesive. Further clarification in this section would strengthen the reasoning.
- **Accelerated Inference**: The paper mentions that CDIM achieves inference times 10 to 50 times faster than previous conditional diffusion methods. Could you clarify whether this acceleration is solely due to the use of DDIM, or if it represents a unique contribution of your own? Clearer differentiation here would improve understanding.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. How much improvement does the optimization part make? Have the authors tried naive DDIM to solve inverse problems? What is the best result if we increase the number of optimization steps?
soundness: 2
strengths: 1. The core contribution lies in combining the DDIM sampling process with an optimization perspective to maintain alignment between the posterior mean and the observation.
2. The paper is well-written and easy to follow, comprehensive experiments have been conducted.
3. The authors conduct thorough research on the efficiency and accuracy of different diffusion-based posterior sampling algorithms.
summary: The paper proposes a new approach for solving noisy linear inverse problems with pretrained diffusion models from the perspective of optimization. By leveraging the DDIM sampling process, it is more efficient than other diffusion based posterior sampling algorithms. It is capable of dealing with arbitrary noise in the observations.
weaknesses: 1. The contribution is limited, the idea of using DDIM to accelerate the sampling process is not new.
1. More mathematical deductions in the appendix would be helpful for the readers to understand. For example, Eq.13, and Eq.14. Also, an introduction to the diffusion posterior sampling(DPS) algorithm in the related work section is also helpful.
2. DMPlug[1] proposes a similar idea. The difference will be that their method optimizes the noise space. I would suggest a comparison with their method.

[1] Wang H, Zhang X, Li T, et al. DMPlug: A Plug-in Method for Solving Inverse Problems with Diffusion Models[J]. arXiv preprint arXiv:2405.16749, 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Eq. (6), Eq. (7) have typo errors.
2. the Eq. (4) have deductive error, not $\sqrt{1-\alpha_{t}}\nabla_{x_t}\log q(x_t)$，should be ${(1-\alpha_{t})}\nabla_{x_t}\log q(x_t)$
soundness: 2
strengths: 1. propose a modification of the DDIM inference procedure to efficiently optimize the Tweedie estimates of $\hat{x_0}$ to satisfy $A\hat{x_0} = y$ during the diffusion process

2. propose to exactly optimize the Kullback-Leibler divergence between the empirical distribution of residuals $R(A\hat{x_0},y)$ and a known, i.i.d. noise distribution r to solve noisy inverse problems

3. give a new choice of $\eta$ to ensure the convergence for KL optimization and stable results of $L^2$ optimization
summary: The paper presents conditional diffusion implicit models(CDIM), which modify the diffusion updates to enforce a constraint upon the final
output to solve noisy linear inverse problems. CDIM satisfies the constraints absolutely for noiseless inverse problems. For noisy case, the author use KL divergence distance to generalize CDIM to constrain the residual distribution of the noise. Compared to other solvers, the family of CDIM method achieve good quality and fast inference time for inverse problems on FFHQ and Imagenet 1k dataset.
weaknesses: 1. The results of DSG[1] on FFHQ and ImageNet datasets are not given. The DSG shows better reconstruction quality and faster inference time on FFHQ and ImageNet datasets. 

2. The KL optimization method(Algorithm 1) is proposed to solve noisy linear inverse problems with known noise distribution. However, from Table.1 and Table.2, $L^2$ optimization has better performance than KL optimization in most tasks. The KL optmization seems meaningless.

3. The calculation of Var(r) are not shown clearly. The necessity of early stopping are not clarified. I doubt that early stopping cannot perform well in noise agnostic taks. 

\[1].Yang, Lingxiao, et al. "Guidance with spherical gaussian constraint for conditional diffusion." arXiv preprint arXiv:2402.03201 (2024).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See above.
soundness: 3
strengths: + Efficiency: 

The framework leverages DDIM, which bypasses the need for computing all 1000 denoising steps. As a result, it achieves impressive inference speeds (e.g., 2 seconds vs. 70 seconds for DPS).

+ Improved Performance: 

The method demonstrates better performance than DPS, as shown by FID scores, though it lacks evaluation through other metrics like PSNR.
summary: This paper proposes a linear non-blind inverse framework to solve inverse problems such as denoising, inpainting, and deblurring. The key contribution is the use of Denoising Diffusion Implicit Models (DDIM), which reformulates the diffusion process as a deterministic ODE, allowing it to bypass the full T sampling process. To ensure the denoised image aligns with the observed data, the method employs gradient projection to adjust the denoising trajectory. Additionally, a self-adaptive parameter control strategy is introduced to balance the data term and prior term dynamically. The approach significantly reduces inference time and demonstrates improved performance over Diffusion Posterior Sampling (DPS) across multiple applications.
weaknesses: I have several concerns regarding its baselines, claims, and equations. 

+ Baselines: 

DPS was a pioneering work that introduced diffusion priors for solving inverse problems. While this paper extends DPS by using DDIM model, the field has evolved rapidly. Recent advancements such as latentDPS (incorporating latent diffusion models), blindDPS (addressing blind inverse problems), and new methods like fastEM and other two arXiv works have shown improved performance using EM frameworks. The authors should include discussions about these recent developments and add comparisons with latentDPS or blindDPS, which have been available for over a year.

+ Claims and Contribution: 

The paper's efficiency seems to primarily come from switching from DDPM to DDIM, which is a known method for speeding up inference by reducing the number of denoising steps. This makes the paper’s core contribution somewhat limited, as it largely inherits benefits from DDIM. I also question where the performance improvement over DPS originates. Does the improvement come solely from DDIM? Typically, acceleration comes with a trade-off in performance, so the authors should clarify the source of the performance gains over DPS.

+ 3D Claims: 

The claim about "3D point cloud reconstruction" in the abstract is misleading. The paper focuses on 2D image completion, in the last figure, it is just projected points based 2D completion, which is far from true 3D reconstruction. The authors should rephrase this to more accurately reflect the work done. Additionally, the title could be clearer—something like "Solving Linear Inverse Problems with Constrained Diffusion Implicit Models" would better convey the focus of the paper.

+ Equations: 

Some equations lack clarity. In Equations (6) and (7), it would be helpful to explicitly include $x_{t-1}$, for instance: $x_{t-1}=f_{\theta}(x_t)=...$. Additionally, the explanation between lines 195-202, which suggests that one cannot get $x_0$ from $x_t$, is confusing. In DPS, the use of Tweedie’s formula to estimate  $\hat{x_0}$ instead of $x_0$ is mentioned and widely adopted, and the authors should rewrite this section to provide a clearer explanation.

Meta-review: The main claim of the paper is that the proposed approach can solve linear inverse problems 10-50 times faster than existing works on conditional diffusion models. The achieved speed-ups are encouraging and were appreciated by the reviewers. However, the contribution over DDIM is incremental and the overall method is ad/hoc. Theoretical guarantees of constraint satisfaction (which are important in many linear inverse problems) could make this a stronger submission, and I encourage the authors to take the reviewers suggestions into account for a resubmission. 

At this stage, I recommend to reject the work.

Decision: Reject