Linearly Constrained Diffusion Implicit Models
Authors: Vivek Jayaram
Ira Kemelmacher-Shlizerman
Steve Seitz
John Thickstun
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion
Inverse Problems
Optimization
Projection
Primary area: deep_learning
TL;DR: We solve noisy linear inverse problems 10-50x faster than existing diffusion methods and with exact recovery of measurement constraints

Abstract: We introduce Linearly Constrained Diffusion Implicit Models (CDIM), a fast and accurate approach to solving noisy linear inverse problems using diffusion models. Traditional diffusion-based inverse methods rely on numerous projection steps to enforce measurement consistency in addition to unconditional denoising steps. CDIM achieves a 10–50× reduction in projection steps by dynamically adjusting the number and size of projection steps to align a residual measurement energy with its theoretical distribution under the forward diffusion process. This adaptive alignment preserves measurement consistency while substantially accelerating constrained inference.
For noise-free linear inverse problems, CDIM exactly satisfies the measurement constraints with few projection steps, even when existing methods fail. We demonstrate CDIM’s effectiveness across a range of applications, including super-resolution, denoising, inpainting, deblurring, and 3D point cloud reprojection.

Linearly Constrained Diffusion Implicit Models
Vivek Jayaram'
Ira Kemelnacher-SSlizerman
Steven M. Seitz'
John Thickstun
University of Washington 2Cornell University
Abstract
We introduce Linearly Constrained Diffusion Implicit Models (CDIM), fast and accurate approach to solving noisy linear inverse problems using diffusion models. Traditional diffusion-basee based inverse methods rely on numerous projection steps enforce mcasurement consistency in addition to unconditional denoising steps. CDIM achieves 10-50x reduction in projection steps by dynamically adjusting the number and size of projection steps to align a residual measurement energy with its theoretical distribution under the forward diffusion process. This adap- tive alignment preserves measurement consistency while substantially accelerat- ing constrained inference. For noise-free linear inverse problems, CDIM exactly satisfies the measurement constraints with few projection steps, even when exist- ing methods fail. We demonstrate CDIM's effectiveness across a range applica- tions, including super-resolution, denoising, inpainting, deblurring, and 3D point cloud reprojection. Code and an interactive demo can be found on our project website.
Introduction
Recovering unknown signal from noisy linear measurements is fundamental challenge encom- passing tasks like super-resolution, inpainting, and denoising. These are examples of linear inverse problems, for which we observe partial measurements y E IRd that was generated through linear operator E dd applied to a signal x E g". the noisy case, measurements y are corrupted with noise sampled i.i.d. from gaussian distribution with variance of:
y Ax oo where ~(0,
We aim to use a pretrained diffusion model 1 as a prior to solve these underdetermined inverse problems. However, adapting diffusion models solve inverse problems has traditionally presented two key challenges. First, recovering the original signal from partial measurements is computation- ally expensive, requiring numerous additional network evaluations each step guide the diffusion towards the measurement constraint. Second, we would like to guarantee that the constraints are met arbitrary precision the noiseless case, difficult task when trying reduce overall steps.
A common approach to diffusion for inverse problems, used by methods like Diffusion Posterior Sampling (DPS) [ and others works 3 4. is to alternate unconditional diffusion steps with pro- jection steps on the measurement error: Vx, Axxx((x)) yl where (x(x)) (x)) is an estimate of the posterior mean E' [x [x]] However, the optimal number and size of these projection steps is diffi-
cult to determine as care must be taken to ensure that (1) the steps are sufficient to converge on the
measurement criteria, (2) unnecessary projection steps do not increase inference time, and (3) large
https://grail. htt.://aaal........a...g.ni.oooo wasshingcon..et//orooottttt//// edt//projects/cddmm
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Time-Travel Rephotography
Poisson Noise
Noisy Inpainting
Fast Inference: 3s
Figure We show several applications of our method including image colorization, denoising, inpainting. and sparse recovery. We highlight the fact that we can handle general noise distributions, such Poisson noise, and that our method runs in little seconds.
step sizes do not cause divergence or pull the iterate x, out of distribution. In light of these chal- lenges, methods like DPS opt for many small projection steps which each require a pass through the model. Therefore, despite advances accelerating unconditional sampling, reliance on numerous small projection steps still limits the speed of inference under constraints.
We propose linearly constrained diffusion implicit models (CDIM) address these challenges. CDIM is a new algorithm for projection-based DDIM sampling that can generate images satisfy- ing measurement constraints under highly accelerated sampling schedules. Our key insight to guide the number and size of the projection steps with the known distribution of A| - y un- der the forward noising process given y. This tractable chi-squared distribution, and we show that forcing Axx y to stay within a plausible region similarly enforces the optimization tar- get (x,) - yl2 to stay within plausible region under the forward noising process. Notably the former is computable without network evaluation, allowing us to apply strategies like binary search or line search on step size without costly overhead.
We emphasize that the speedup of CDIM does not simply come from adopting an accelerated un- conditional DDIM schedule, but rather from smarter projection strategy that greatly reduces the number of projection steps required for constrained sampling. This allows outputs to match partial observations under accelerated sampling without introducing large number of additional network evaluations. In contrast, works such as 4 use DDIM as a sampling strategy yet still require 200 additional network evaluations for constrained sampling, showing that DDIM alone is not enough to accelerate the entire proecess. To further demonstrate this, we present qualitative examples that simply using DPS with DDIM accelerated sampling yields blurry or divergent results, even when the projection step size is tuned for optimal performance.
for out-of-distribution inputs (impossible with DPS), while requiring 10-50x fewer projection steps. Under Gaussian measure- ment noise, our step size criterion naturally generalizes from the noiseless case without introducing additional network evaluations. We further extend CDIM to handle Poisson noise through a re- formulation based Pearson residuals. Empirically, CDIM achieves high-quality reconstructions in under 2.5 seconds, whereas prior methods such as DPS and MCG [ require over 70 seconds. Across tasks including super-resolution, box inpainting, deblurring, and random inpainting, CDIM delivers comparable or superior reconstruction quality while operating an order of magnitude faster.
Furthermore, CDIM exactly recovers noiseless measurements,
Related Work
Diffusion models 1 have emerged as powerful generative models, building upon early work in nonequilibrium thermodynamics (5) and implicit models (6. Denoising Diffusion Implicit Models (DDIM) [7] was a notable work that improved the efficiency of diffusion sampling through non-

---

markovian sampling. This was further advanced through stochastic differential equations 8 and numerical ODE solvers like PNDM 9
Applying diffusion models to inverse problems has been an active research area. DPS 2 was a notable method that uses alternating projection steps guide the diffusion process. DDNM (T), DDRM III, SNIPS [12), and PiGDM [3) use linear algebraic approaches and singular value de- compositions. Techniques such as DMPS [4], FPS [15], LGD [16], DPMC (4), and MCG (17),
and DAPs [8] focus on likelihood approximation for improved sampling. Guidance mechanisms
have been incorporated through classifier gradients 19, data consistency enforcement 20 and
low-frequency feature matching [21].
Other approaches projection 22
optimization
24 25 26 DMPlug 27 backprop-
agates through the entire diffusion process, leading to extremely slow inference. DSG 28 uses a
similar optimization update to for enforcing consistency with the partial observation; however, it does not guarantee matching a constraint exactly, instead using a soft constraint, like DPS, to handle observational noise (see Appendix F.I). Finally, works such as Blind DPS (29) and FastEM (30] solve inverse problems when the forward operator is unknown, a more difficult problem than the setting studied in this work.
Quality vs. Runtime
0.15
Average LPIPS (inverted)
0.20
CDIM
DFMC
FPS-SMC DPS
Oprs
0.25
Oury fast
0.30
DAP-50
0.35
DORM
MCG
0.40
0.45
PMRADDMM
Score SDE
0.50
10
20
50
60
Runtime (Seconds)
Figure 2: The family CDIM methods (top left corner) simultaneously achieves strong generation strong quality and extremely fast inference compared to other inverse solvers. We plot the inference speed and average LPIPS image quality score (inverted) averaged across multiple inverse tasks on the FFHQ dataset. "Ours" uses T 50 denoising steps while "Ours fast" uses T" 25 denoising steps
3 Background
which models data distribution g(xo) by modeling quence f = 1,..., Tof of smoothed distributions defined by
We work in the context of DDPM (1.
g(x:|xo) = -X,x Varxo, (1 - a.)...
The degree of smoothing is controlled by monotone decreasing noise schedule og with ag (no
noise) and âr = 0 (pure Gaussian noise). ??] The idea is to model reverse process pe (xx1|/x)) that
that incrementally removes the noise in x, such that po (xT) A ((x;;0, I) and p(xo) approximates the data distribution, where p(xo) is the marginal distribution of outputs from the reverse process:
po(xo) = -f po (xT) II 1=1
T
po (x dx1:r-
(3)
Given noisy samples x, = Vaxx0 + /1 OGE, where xo a sample from the data distribution and ~ (O, diffusion model eo (x,, t) is trained to predict
min E [[le eg (x,, t) [1 X..6
(4)
We define using the DDPM convention MI this corresponds in DDIM 7
3

---

To parameterize the reverse process po(x(-1(x)), DDIM 7 exploits the Tweedie formula [1 for the posterior mean of noisy observation:
E [xo
Vat (xt +
(1 (1 ât āt) Vx. log g(xx)).
(5)
Using the denoising model E e(x,, 1) as plug-in estimate of the score function via the relationship E [e(x,]] -
/1 de Xx log (xx)), we define the Tweedie estimate of the posterior mean:
xx(x,)
(x+ (x--vl (x,E 1)) & &E [xo/xe]-
(6)
Vat
Throughout this paper, we keep the formula for the Tweedie estimate in functional form X (x;) to make it clear that &y is function of our current iterate XT. We can then use this estimator to define DDIM forward process X = fxx) defined by
X2 Vā x0(x))) X fx() vār-i xx(x;) + VI VI- āc-1 V1-a
v1 dN
Unlike DDPM, the forward process defined by Equation (7) is deterministic; the value po is entirely determined by Xy ~ _N(0, I) thus making DDIM an implicit model.
With slight modification of the DDIM update, accelerate inference. Given 8 2 we define an accelerated denoising process
able to take larger denoising steps and
X v 04 xx(x.)
x (x,) v xx(x)) + vI V1-ā-
(8)
VI at
Using this process, inference is completed in just T T/6 steps, albeit with degraded quality of the resulting sample Xn as becomes large.
4 Methods
We are interested in solving linear inverse problems of the form Ax, where y E R is linear measurement of R" and E IRd describes our measurement operator. For example, if A [0.1) is binary mask (which is the case for, e.g., in-painting sparse recovery problems) then y describes a partial measurement of x. We seek an estimate that consistent with our measurements: in the noiseless case, Ax y. More generally, we seek to recover a robust estimate of x when the measurements have been corrupted by Gaussian noise: y Ax + or, where *N(0, oZ.).
In Section 4.1 we motivate the high-level approach of alternating unconditional DDIM steps with an adaptive number of gradient updates on the measurement residual energy Vx, (x, 2 We choose step size and number of gradient descent steps to ensure that each timestep t, the related quantity ||X,, y|l remains within standard deviation of its expected value Ee. [||Ax, 2 y] under the forward process; as shown in Section 4.2 this ensures that the residual we are optimizing, (x,) y 12, also remains in-distribution of the forward process with high probability. Finally, in Section 4.3 we discuss our algorithm for choosing step size 1) and stopping criteria based on a target residual energy. The full algorithm is described in Algorithm
4.1 Optimizing (x,) ) to match the measurements
For linear measurements A. the Tweedie formula for (xx)) (and the corresponding plugin- esstimate Equation (6)) extends to a formula for the expected measurements:
E. [y x.] AE
X xx] Ax,(
During the diffusion process, we would like the expected denoising trajectory to agree with the observation, i.e. (x, = y. Therefore, reasonable initial idea is to modify the DDIM updates Equation (7) to find the closest point x;-5 that satisfies the constraint A&o (x:-5) y. Lec. at each time step f, we force the Tweedie estimate of the posterior mean of q(y xx,) ) to match the observed measurements while making the smallest possible update from our unconditional denoising step:
arg min ||xe-5 f0(x
X
(10)
subject to
Axo =y.

---

(a)
(b)
Figure 3: Results 50% noisy inpainting task with og 0.2. (a) is the noisy partial observa- tion. (b) is generated by CDIM (Algorithm[] without considering the Gaussian measurement noise, showing that we can exactly match the constraint even when the observation is out of distribution. (c) is generated by CDIM and using the values from Appendix that consider the Gaussian mea- surement noise.
We face two conceptual challenges in optimizing Equation (TO). First, for t > 0, typically no value x, will satisfy (x, = and therefore the optimization is infeasible. Second, the estimate of the score function Vx, log g(x+) from the diffusion model, e (x,x () may inaccurate, particularly at large I; we risk overfitting to bad plug-in estimate xx(x)) which will pull our iterate far out of distribution during the denoising process.
In light of these observations, we replace Equation 10
with a soft optimization
arg min ||x - fo(xx))l + A - -
(11)
We can interpret Equation T as a relaxation of Equation (TOp and we implement this optimization via gradient descent, initialized from (0) = fx();) and computing it gradient steps
(k) (k-1) x 4-8 + V [| - y11
(12)
The regularization by A| A (x y|2 is achieved implicitly by stopping after k steps of gradient descent at given timestep f. In contrast to hard constraint at each timestep, this objective robust to both (1) the possible infeasibility of (xx(x:-8) = and (2) overfitting the measurements based on an inaccurate Tweedie plug-in estimator.
For notational purposes, we define L, as the energy of the measurement residual between the Tweedie posterior mean (x,) and our observation This is the quantity we actively reduce via gradient descent with each projection step:
L == (x(x y
(13)
We can therefore interpret Equation (1 as projection of the DDIM update fo (x:) onto the set of values X,-ò that sufficiently reduce the residual energy T.-8. The full inference procedure is analogous to projected gradient descent, whereby we alternately take a step (x;) determined by the diffusion model, and then project back onto the set of plausible values for Lt-s at timestep t t- ô. At high noise levels, the plausible domain is very large so our constraint is weak. As noise decreases, the plausible domain constricts to a radius of zero, achieving constraint satisfaction if we can ensure that L2 when =
As t approaches 0, (xx))) converges to and L, = [xxx - yl1 empirically behaves like simple convex quadratic ||Ax; 2 which can be minimized to arbitrary accuracy by taking sufficiently many gradient steps. This observation is why we can achieve exact recovery of the measurements y Ax in the final result Xo. We demonstrate in Figure [3c that we can match measurement even when it is out of distribution. We provide further calculations and empirical evidence to demonstrate the constraint convergence behavior in Appendix C
'We illustrate both these claims by considering the Tweedie estimator Equation in the case T. In this case, X₁ (V f) independent xo and therefore E|x, xx. E(xa]. the mean the data distribution (X))). Unless A Xu] y, the optimization is infeasible when T'. Furthermore, we observe that when t = T', the plug-in estimator & is not independent of x, and & o [[[o]]. This is indicative of error in the diffusion model, especially high noise levels.
5

---

Mcasurement Consistency Projection Step
Proxy Residual
&y(x,)
(
Ax,
Y
F,11AR,00) -I1
4 (XA v
R, = |1Ax, -
Figure 4: We show the conceptual overview of CDIM for 50% inpainting task without measure- ment noise. (Left) We compute the Tweedie posterior estimate X, (x,) then apply the linear operator
A. This value Ax)(X,) is compared with the observation y to obtain our loss L || (x,) yyl 2 We then update our iterate X, with steps on Vx, Lt. (Right) Our proxy residual RG |Ax; y has an anlytical x2 distribution under the forward noising process and used guide the step size and number of steps for the left side process.
4.2 Maintaining Ax, - y|1 Within its Plausible Region
When alternating unconditional diffusion steps with measurement consistency steps, key design choice is how aggressively to minimize L.- Specifically, at each timestep i, we must determine the domain of plausible values for the residual energy. and find the corresponding step size r) and number of gradient steps k to minimize Equation (11). Keeping L, within its plausible region means we must sufficiently minimize L, at each timestep while also not pulling our current iterate x, out of distribution of the unconditional diffusion process. Existing methods, such DPS (2). typically apply a single small projection step per diffusion timestep, which insufficiently minimizes the residual energy when using accelerated DDIM schedules, leading to under-convergence (see Figure 7. Alternatively, fixing & > 1 across all timesteps is both computationally inefficient and leave the plausible region, particularly at high noise levels where the residual is
may cause inherently large and uninformative.
To guide the optimization process more effectively, we propose to align the measurement residual with its expected behavior under the forward diffusion process. That is, at each reverse step - t- 5, we would like to ensure that the observed residual L,-8 is plausible under the distribution of residuals given by the forward process. This distribution can be defined formally via the forward noising process: Let ε (0,, I) be the forward noise and consider L. ||A Xo(x)) yl|² with x; = aa, Xo + V1- Or E, and y = Axp for some true data sample Xg. We then see that, under the forward diffusion process, p( Le ly) is well defined distribution that depends on the forward noise and ground truth data distribution Xg-
Our key idea is to compare the observed residual L, during the denoising process to its forward- process expectation Ex, [L, y] and adaptively adjust the optimization effort to maintain consis- tency. Intuitively, we want to keep our optimization target L, within a plausible region close to its expected value, which prevents under or over optimization of the projection steps. This idea is mo- tivated by prior work [32 3 that demonstrates improved generative performance when the reverse process mirrors the forward dynamics. Unfortunately, the forward expectation of (|Axx(x;)) - y is intractable to compute as it depends on the true data distribution. Furthermore, every function evaluation &n (and therefore L,) requires neural network evaluation, which would make it compu- tationally expensive to perform any kind of step size search even if we knew the plausible region..
Instead, we propose to use the distribution of proxy residual R, := ||Ax, - y1 guide our projection steps. This residual has several desirable properties compared to L,: it is analytically tractable under the forward diffusion process, inexpensive to compute, and strongly correlated with our target residual L,. Specifically, we calculate in the Appendix [ that under mild conditions, maintaining RG near its forward-process expectation, which we denote μ4 (y)), ensures that L, is similarly controlled with high probability. This provides a computationally efficient and principled mechanism for adaptively controlling projection effort at each timestep, without requiring additional model evaluations. This entire procedure is shown at a high level in Figure 4

---

(a) Random in- painting
Stopping
(c) Stopping (d) Stopping at (c) Ground Truth Ja + 0.10R
( 40
PR + OR
task input y
Figure 5: We show the results of stopping various points within the plausible region defined by Fn(y) c Rn(y) for different values of c. For < the results do not change dramatically, but numerically still improve (see Appendix D)
Proposition 1 Define the target residual energy Lt [x)) yl1² and the proxy residual energy R == ||x, - yll². Suppose that the proxy residual satisfies Rs - E[Roly]l I %. Then with high probability, L - E yyl a + O
(V-=a (VI-
4.2.1 The x² Distribution of R.
The distribution of the residual energy Ax - y11² y) under the forward noising process follows non-central generalized chi-squared distribution with tractable mean and variance:
ue prly) E,,, [R y] = (Vā, - 1) 2 + + tr)
(14)
of = Var., [& y)] = (r((())))))))))))))))))))))))()) + 4u Σxpe.
(15)
We derive this formally in Appendix B and also derive the corresponding mean and variance given noisy measurements = Ax + N0,
4.3 Choosing 7 and Stopping Criteria
Now that we have an analytical understanding of our proxy residual energy R, == |Ax - y/12, we can determine the optimal projection step size v and number of steps & for each timestep (.
To decide when to stop projection steps, compare the current residual energy Rg the forward expectation Ht (y) = E,, R y] derived above. We denote the plausible region boundary as (y) where we aim stay within some deviation of the mean. Lower values of tyyicaall produce better results at the tradeoff of more projection steps. We then halt the projection loop when the proxy residual lies within this plausible region boundary.
pe(y) = (y)
A y/1 py)
(16)
This rule avoids both under- and over- vee-prooectiion while ensuring the denoising trajectory aligns with the expected residual behavior. We show comparative results of choosing a different stopping criteria in Figure and in Section 5.3
To determine the optimal step size v), we precompute the gradient of our objective:
g -lx-,|lAx,,x,, 112
(17)
We then perform one-dimensional search (e.g., binary line search) select the r that yields a future residual energy || ||A (xxx-6 7g/) y closest to the plausible region boundary Pt-6- We find empirically that bringing the residual back to the boundary of the plausible region is better than trying to bring it directly to the center of the plausible region.
nj == arg min ( - ||A -gn -
(18)
gE
Since the objective is well-behaved quadratic function in v,, this search is efficient and stable in practice. We summarize the complete algorithm in Algorithm [ using generalized step size during the denoising process.

---

Algorithm Constrained Diffusion Implicit Models
(. for = T,T 8.., do
Xt-8 5- var-
Xe- /1 x-V1i-eeu((e.t1) + v a ax-oc0(Xe,))
{D Unconditional Generation)
vāx
pr(y) (4-6(()
+ O
{D Boundary of Plausible Region)
while |Ax,-8 y|1² > prly) do
& Var-o (X1-6 v1 60
9 V xI-1 (x -111
7f" = arg min PER (p(y)) - ||A (x 19) - y11
x f- X1-6 - '/9
end while end for
(a) Box inpaint- ing task input y
(b) T'=25
(c) T'=10
(d) T'=4
(c) Ground Truth
(31 Total Projec- tion Steps)
(15 Total Projec-
(10 Total Projec- tion Steps)
tion Steps)
Figure 6: We show how the results change as we decrease the number of denoising steps 7", Notably, we still produce reasonable results with only T denoising steps and corresponding 10 total projection steps, for total of 14 Neural Function Evaluations (NFEs) and < second inference.
4.4 Poisson Noise
Possion noise is non-additive noise defined by sy Poisson( Asx), where y is interpreted dis- crete integer pixel values. The scaling factor s I 1 controls the degree of Poisson noise. Poisson noise is not identically distributed across y; the variance increases with the scale of each measure- ment. To remedy this, we consider the Pearson residuals [33]:
R(Axo. y) A(y A&o)
(19)
VAxo
These residuals are identically distributed; morcover, they are approximately normal ~ NN 1) 34]. We can therefore treat the Pearson residuals standard normal noise and solve the inverse problems using the same method for Gaussian measurement noise. Although the Pearson residuals closely follow the standard normal distribution for positive values xg, this breaks down for values of &o close to zero, and extreme noise levels s. In practice we find the Gaussian assumption to be valid for natural images corrupted by as much noise as 0.025. In Figure I we show an example of denoising an image corrupted by Poisson noise with 0.05.
Results and Experiments
We conduct experiments to demonstrate the efficiency and quality of CDIM across various tasks
and datasets. Section [. we present quantitative comparisons to state-of-the-art approaches,
followed by a comparison against DPS using DDIM in Section 5.2 In Section 5.3 we describe
ablation studies examining inference speed and hyperparameters. Finally, in Section 5.4 we explore two novel applications of diffusion models for inverse problems.

---

Table 1: Quantitative results (FID, LPIPS) of model and existing models various linear inverse problems on FFHQ 256 x 256-1k 1k validation dataset. (Lower is better). The best result is in bold and the second best is underlined.
Super
Inpainting (box)
Gaussian Deblur
Inpainting (random)
Runtime (seconds)
FFHQ
Res
Methods
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
Ours T
25
33.87
0.276
27.51
0.1872
34.15 18
0.276
29.67
0.243
2.4
Ours T" 50
31.54
0.269
26.09
0.196
29.68
0.252
28.52
0.240
6.4
FPS-SMC
26.62
0.210
26.51
0.150
29.97
0.253
33.10
0.275
116.90
DPS
39.35
0.214
33.12
0.168
44.05
0.257
21.19
0.212
70.42
DDRM
62.15
0.294
42.93
0.204
74.92
0.332
69.71
0.587
2.0
MCG
87.64
0.520
40.11
0.309
101.2
0.340
29.26
0.286
73.2
PnP-ADMM
66.52
0.353
151.9
0.406
90.42
0.441
123.6
0.692
3.595
Score-SDE
96.72
0,563
60.06
0.331
109.0
0.403
76.54
0.612
32.39
ADMM-TV
110.6
0.428
68.94
0.322
186.7
0.507
181.5
0.463
5.1 Numerical Results on FFHQ and ImageNet
We evaluate CDIM on the FFHQ- Ik [35] and ImageNet-Ik Ik [36] validation sets. Each dataset con- tains 256 x 256 RGB images scaled to the range [0. 1]. The tasks include 4x super-resolution, box inpainting, Gaussian deblur, and random inpainting. Details of each task are included in the appendix. For all tasks, apply zero-centered Gaussian measurement noise with ensure fair comparisons, use identical pre-trained diffusion models used the baseline methods: for FFHQ we use the network from [2] and for ImageNet
0.05. To
the network from [19]. We report
Frechet Inception Distance (FID) [37 and Learned Perceptual Image Patch Similarity (LPIPS) [38]
with peak signal-to-noise ratio (PSNR) results in Appendix R.5
All experiments carried out
a single Nvidia A100 GPU.
In Table I compare CDIM with several other inverse solvers using the FID and LPIPS metrics on the FFHQ dataset. We present results using our method with both 0.1 for the number of standard deviations in the stopping criteria. For ImageNet results please see Appendix E.4
denoising steps and
T = 50 denoising steps. In all cases
5.2
Additional Comparisons
We show qualitative comparison against DPS [2] when we combine it DDIM and fewer steps (see Figure 7). We use the core DPS sampling algorithm, but with DDIM as the denoising algorithm instead of DDPM. The number of denoising steps is set to 15 (CDIM is limited to 15 projection steps) and the step size of DPS is increased in panel (c) to achieve the best constraint convergence
(a) Box inpainting with- out noise: input y.
(b) DPS DDIM MAE: 4%
DPS DDIM (large 7)) MAE: 0.3%
(d) CDIM MAE: 0.05%*
We show comparisons against DPS using DDIM sampling where all algorithms use 30 NFEs. For each output, we report the mean absolute eπror of the observed pixels, ||Ax - -yll, relative to the input. (b) If you simply run DPS with DDIM sampling, the constraint is not met. You can see this the hair, which is blurrier, along with the background which has changed. (c) If you try to increase the step size of DPS, the observed region matches better, but the results diverge. (c) CDIM achieves strong constraint satisfcaation and inpainting results. *The MAE of CDIM should be 0, but the diffusion schedule sets ag 0.999 for numeric stability.
Figure 7:

---

(b)
Figure 8: Noisy inpainting for sparse point cloud reprojection. (a) Shows sparse point cloud pro- jected to a desired camera angle. (b) Shows the result after our method is used for noisy inpainting.
possible. Note that when using DPS, no learning rate can lead to measurement consistency with the accelerated DDIM sampling scheduling without diverging.
We also compare against DAPs-50
18 DSG 28 and Diff-PIR
25 in Appendix
P
5.3
Ablation Studies
CDIM only contains two hyperparameters: the number of denoising steps T" and the plausible region stopping criteria constant c multiplied by (y). In Figure 6 we show how the results and total required projection steps change as we decrease the 7" to few 4 denoising steps. For the stopping criteria, Figure 5 shows qualitative results when changing e, and in Appendix D we show how quantitative results and number of projection steps change with c.
5.4 Additional Applications
Time-Travel Rephotography In Figure we showcase an application of time-travel rephotography
Antique cameras lack red light sensitivity, exaggerating wrinkles by filtering out skin subsur- face scatter which occurs mostly in the red channel. To address this, we input the observed image into the blue color channel and use the pretrained FFHQ model with Algorithm to project the face into the space of modern images. We further emphasize the power our approach: (39] trained a specialized model for this task while are able to a pretrained model without modification.
33).
Sparse Point Cloud Reprojection Twenty different images from a scene in The Grand Budapest Hotel scene were entered into Colmap 40 to generate a sparse 3D point cloud. Projections of this point cloud have roughly 90% of the pixels missing. Furthermore, the measurements often contain significant amounts of non-Gaussian noise due to false correspondences. We can formulate this as noisy inpainting problem and use Algorithm aloog with a variance threshold that adequately captures the imprecise nature of the point cloud. We showcase the results in Figure g
6 Conclusion
In this paper we introduced CDIM, new approach for accelerating noisy linear inverse recovery using pretrained diffusion models. By projecting DDIM steps into plausible region of of the forward process, we can enforce constraints without making out-of-distribution edits to the noised iterates X,. Note that our method cannot handle non-linear constraints because for non-linear function h. [l(x))] # 1<T [xo]). Therefore, we cannot extend Tweedie's estimate of the posterior mean X. to estimate of the posterior mean of non-lincar observations h(xx). However, for linear constraints, our method generates high quality images with faster inference than previous methods, creating point on the Pareto-frood fareto-frontier of quality vs. efficiency for linear inverse problems.
Acknowledgments
This work was supported by the UW Reality Lab, Lenovo, Meta, Google, OPPO, and Amazon. We also thank Tatsunori Hashimoto for fruitful discussions in the early development of this work.
10

---

References
[1] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, Ad- vances in Neural Information Processing Systems, 2020.
[2] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2022.
5.1 5.2
[3] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints, 2024.
2
4.2
[4] Yaxuan Zhu, Zehao Dou, Haoxin Zheng, Yasi Zhang, Ying Nian Wu, and Ruiqi Gao. Think twice before you act: Improving inverse problem solving with meme, 2024.
[5] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference machine learning, 2015. 2
[6] Shakir Mohamed and Balaji Lakshminarayanan. Learning implicit generative models. 2017.
[7] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. 2
[8] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[9] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2021. 2
[10] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffu- sion null-space model. In The Eleventh International Conference on Learning Representations, 2022.
[11] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restora- tion models, 2022.
[12] Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically, 2021.
[13] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guiee diffu- sion models for inverse problems. In International Conference on Learning Representations,
2023. 2
[14] Xiangming Meng and Yoshiyuki Kabashima. Diffusion model based posterior sampling for noisy linear inverse problems. arXiv preprint arXiv:2211.12344. 2022.
[15] Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving:
A filtering perspective. In The Twelfth International Conference on Learning Representations,
2023. E
[16] Jiaming Song, Qinsheng Zhang, Hongxu Yin, Morteza Mardani, Ming- Yu Liu, Jan Kautz, Yongxin Chen, and Arash Vahdat. Loss-guided diffusion models for plug-and-play controllable generation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research. pages 32483- 32498. PMLR, 23-29 Jul 2023. 2
[17) Gabriel Cardoso, Yazid Janati El Idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided diffusion for bayesian linear inverse problems, 2023. 2
11

---

[18] Bingliang Zhang, Wenda Chu, Julius Berner, Chenlin Meng, Anima Anandkumar, and Yang Song. Improving diffusion inverse problem solving with decoupled noise annealing. In 2025 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), page 20895-20905. IEEE, June 2025.
[19] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
5.1
[20] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diifuse-
Accelerating
conditional diffusion models for inverse problems through stochastic contraction, 2022. [
[21] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models, 2021. 2
[22] Benjamin Boys, Mark Girolami, Jakiw Pidstrigach, Sebastian Reich, Alan Mosca, and
O. Deniz Akyildiz. Tweedie moment projected diffusions for inverse problems, 2023,
[23] Stanley H. Chan, Xiran Wang, and Omar A. Elgendy. Plug-and-play admm for image restora- tion: Fixed point convergence and applications, 2016.
[24] Bo Wahlberg, Stephen Boyd, Mariette Annergren, and Yang Wang. An admm algorithm for class of total variation regularized estimation problems, 2012.
[25] Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration, 2023.
[26] William Huang, Yifeng Jiang, Tom Van Wouwe, and C. Karen Liu. Constrained diffusion with trust sampling, 2024.
[27] Hengkang Wang, Xu Zhang, Taihui Li, Yuxiang Wan, Tiancong Chen, and Ju Sun. Dmplug: plug-in method for solving inverse problems with diffusson models, 2024.7
[28] Lingxiao Yang, Shutong Ding, Yifan Cai, Jingyi Yu, Jingya Wang, and Ye Shi. Guidance with spherical gaussian constraint for conditional diffusion, 2024.
10
[29] Hyungjin Chung, Jeongsol Kim, Schui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems, 2022,
[30] Charles Laroche, Andrés Almansa, and Eva Coupete. Fast diffusion em: a diffusion model for blind inverse problems with application to deconvolution, 2023. 2
[31] Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):602-16114 2011.
[32] Giorgio Giannone, Akash Srivastava, Ole Winther, and Faez Ahmed. Aligning optimization trajectories with diffusion models for constrained design generation. In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023, 4.2
[33] Daryl Pregibon. Logistic regression diagnostics. Annals of Statistics, 9:705-724, 1981. 4.4
[34] Donald A. Pierce and Daniel W. Schafer. Residuals in generalized linear models. Journal of the American Statistical Association, 81 (399)))977-988 1986. 4.4
[35] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019. 5.1
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge, 2015. 5.1
[37] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium, 2018. 5.1
12

---

[38] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric, 2018. 5.1
[39] Xuan Luo, Xuaner (Cecilia) Zhang, Paul Yoo, Ricardo Martin-Brualla, Jason Lawrence, and Steven M. Seitz. Time-travel rephotography, ACM Transactions Graphics, 40(6):1-12, December 2021. 5.4
[40] Johannes L. Schönberger and Jan-Michael Frahm. Structure-frommmotion revisited. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4104-411 4113, 2016. 5.4
[41] D. L. Hanson and F. T. Wright. A Bound on Tail Probabilities for Quadratic Forms in Inde- pendent Random Variables. The Annals of Mathematical Statistics, 42(3):1079 1083, 1971.
[42] Mark Rudelson and Roman Vershynin. Hanson-wright inequality and sub- sub-ggassiian concentra- tion, 2013.
[43] Michael Hutchinson. A stochastic estimator the trace of the influence matrix for laplacian smoothing splines. Communications in -Simulaainnn and Computation, 18(3):1059- 1076, 1989. B.I
13

---

A Proposition: Controlling the Proxy Residual Controls the True Objective
Setup and notation. (0, 1). Let
Fix diffusini step E (I,...;7) with cumulative noise level at
One-step variance:
== 1 Ck.
a = Vā: 1,
E = AA E gaxd
Given the current state X2 vā Xo vbe, with E, NO,,1), define
Proxy residual:
T: ::: Ax y, R₂ == ||rell?,
Mean of proxy residual:
pc(y) = Be, Rs yl
X = X vbco(x(,t))
Tweedie estimate:
Vā
true objective: L == Axo yl1
model noise in measurement space:
G: = Aco(xi,t).
Throughout we assume the denoiser is conditionally unbiased: E [co( xi] ct.
Proposition Statement: Suppose that the proxy residual is close its expected value: Rs
[v() < Then with high probability, |L, E[L, yl| I a. + O / (1-a:) i.c the target
residual is close to its expectation.
Step 1: Write L, in terms of R²
Multiply & by A and subtract y:
Ax, vbG. Vony
Pt VbG. ay
A y
Voa
Vāa
Squaring the norm yields:
T 1 Rr + 111. a² 1|y|12 22b (r,, (e)
2a (rr,)) + 2av (G..)))
(20)
Step 2: Conditional expectation. Taking El. y] in Equation 20
using E((r,, (e) y] =
ElKG, y) yy] = (given the unbiased-score assumption and independence of e; from y), and using
E/r yl Bivā,Axo vbAs, - y] y gives:
1 BLL, y (prly) + tr + E 11/12 2a (1
04
(Mr(y)
+ tr -a22all1)))
a,
Step 3: Decompose the deviation. Now we can calculate how far our true objective deviates from its expected value. Write
A |c::1 - 2 (ra, Ag = (G..)) A : (re.v) a ll
14

---

Then
4 - LB
y]
(RR (+(1)) + ااا1 + 2vb/Aa/ + 2|a|v/|As/ + 2/al|/aa))
(21)
The first term in Equation 21 is the deviation of the proxy residual from its mean, which satisfies
R& - (y))
v ?.
We then use Hanson- Wright and
to bound the remainder of the terms.
Step 4: Concentration bounds for the A;
A1: Since Ç: A6 with 6t ~ (N( I), ||//1 ? a quadratic form a Gaussian vector. Hanson- Wright 41
gives
A & C (r
with probability at least - 2e and For measurement space dimension d: y E F and constant
C1 0.
Ag: Using Mtuchy-Schwara, x² tail for ||<all and the given bound ||rell (m((y) +
A < ||r ||</ll s vel( + C2vtr
Ag: We can use a standard Gaussian tail bound since (Ct, 8) ~ NO, y Ey). This yields
Asl S C2 |ylll Vr
Aq: This is a deterministic bound once ||r|l| bounded. Using |a| Vāa 1a1-11 1 S vb and ||rel s
V Mc(y) + +
A = (re, 9) |ly//1 s |rrll Ielll + |a| Ilvll (x(()) + 1l/l vb 1121?
Each of the three genuinely probabilistic bounds occurs with failure probability 2c for measure- ment space dimension y Rdd and absorbing constants into > 0. In typical proofs involving Hanson- Haason-Wright, c; 10 [42] and since d is typically much larger than 100, the failure probability remains exceedingly small.
Step 5: Assemble the pieces.
Insert the bounds for A1:4 into Equation 21 use Vir(y) +7 S Vu((y) + VT. and absorb numerical constants into universal C > 0:
Cvb
(V7 + (lvll) + Cb
L. - L. 3l
+ ā
|lvl2
a
Step 6: Conclusion.
Rg - 11(444(y)] 5 = L E vl 5
7
+ o
I
[
+
1-ae
l1vl3)
with probability at least 1 1- -ed Because -- = & for all practical timesteps, the additional terms are dominated by Vb/āz, atz leaving
L, BL vl
<
7 + O
VI
a
as claimed. Notice that As t 0 (ā: 1) the second term vanishes, so matching the proxy residual to its mean immediately controls the true objective with the same statistical precision.
15

---

B Distribution of | | -
yl1² with Noiseless and Noisy Observations
B.1 Noiseless Observations
Let us define our proxy residual vector T2 == Ax. y and the energy R₁ == 1r+11 ||Ax; - -yl
We start with the forward diffusion equation in Equation 2 and multiply both sides by A. We then subtract from both sides, yielding the residual
Ax: -y y (Vā 1)y VI Ac 4 I
(22)
This implies that p(r ~ (N),( Et), where
me(y) = (Vās- 1)y, 1yy,
Σ, a
The residual energy Rr := Axx - y11 = ||r]| is thus chi-squared distribution with the following mean Me (y) and variance of (y)):
(r(y) E,,, [R yy] = (Vā. - 1) trrAAA
(23)
of (y) = Var., R& y] = 2 2tr((2)))) + 4 Expe.
(24)
Note that often our linear operator is implemented functional form, where computing the actual matrix representation its transpose is inconvenient. In practice, can use trace estimators such as Hutchinson's 's method [43] to estimate all required values involving A, such as tr(AA) "),
tr((E))²) and μ. Eipt using only matrix-vector products.
B.2 Noisy Observations
We derive the mean and variance of residual energy R₁ ||Ax, y/1² during the forward diffusion process when we have measurement noise: y Ax, + By where By ~ of
Unbiased substitutions. Because quadratic forms with statistics that depend only on the noisy measurement and d == dim(y):
observe the latent projection Axg, replace its
||xx|| ||yl| do
(25)
(Axo) (Axo) = y "y By tr
(26)
with > AAT.
First two moments of the residual energy. Using the forward-diffusion decomposition va, Xg + vT - Or Et; E No, I), one finds the residual Ax, - y (Vo, -1) Ax + /1 11-di ā Ac; ag. After substituting the unbiased identities Equation (25) 255)-Equtionn 26 and
taking expectations over both noise sources e, and By. obtain closed-fort expressions that fully observable.
Expectation.
E Ru [yl
(1 ai) tr(E)
mo (Vā, 1)21 + (Vā. - 1))
(27)
diffusion noise
measurement noise
deterministic bius
Writing Σ, formula Var(Q.) = 2 (5 + 4 Enjt, gives
Variance.
āt) E of I and júr (vāg 1) , the non-central x2 moment
Varl R. [y] 2 (1 2z u{Σ + 2(11 a+) of tr mo²y
+ +4(vog - 1) (1 - āx) Ty of E) + -mm])] (28)
16

---

C Convergence of Constraint Satisfaction
We now analyze the convergence properties of the constraint satisfaction procedure in CDIM for the noiseless case. The algorithm alternates between unconditional diffusion updates and projection steps:
1. Unconditional update: fo(x)) = DDIM.step(x.))
2. Projection:
X arg m |
(x s.t. (x) =
where & (x,) denotes the Tweedie estimate [| x.] at timestep f. When the constraint is infeasible, we perform gradient descent on (x,) yl
We first show that the Tweedie estimate converges to the identity mapping, and characterize its rate of convergence. We also demonstrate this rate of convergence empirically Figure
Given that, we show that as - O, finding x; such that [|Ax,((x) - -yll is feasible and satisfiable via the proposed gradient descent algorithm.
Tweedie Convergence. In this section, we show that as + O, the Tweedie estimate converges to the identity mapping:
sup [>x(x,)))) - xrl12 s e(t)
(29)
where =(t) 0 as +
Consider the forward diffusion process:
X, = 1 1- B. +
Bie, e~01(
(30)
The Tweedie estimate (posterior mean) given by:
xx(x))
X
3
V
logp(x;)
(31)
V1-
3.
V1
Be
For small Be (as 0), we perform Taylor expansion:
3,
=1
+0()))
(32)
V1
3,
3
B3
1+
+ 0
(33)
3r
2
Substituting into xx(x)):
31
xo(x)) = Xt
1+ -
1+
Xx logp(x:) + 0(33)
(34)
2
B
Xt. +
X₁.
Vx, logp(x; (x,) ) + +(3g)
(35)
2
The deviation from is thus:
B
So(x:) - X; X₁ Vx. logp(xx) + 0(3))
(36)
Taking norms and applying the triangle inequality:
[xx(xx)) xel12 S 2 |xella + 3el Xx, +
(37)
To estimate the scaling of Vx, log (xx) 2/ recall that the marginal p(x.) is the Gaussian-smoothed density
pxxx) = (po (x))).)))xxx...
17

---

whose log -ogggrddten scales as the inverse of the noise standard deviation:
V x. logg(xx))le O ( (V
with high probability. Intuitively, as 3r 0. p(x;) becomes sharply peaked around Xo. and its log-density gradient increases proportionally to the inverse of the noise scale.
Substituting this scaling gives dominant term of v
[xo(X;)) xel12 O 3.)
(38)
with high probability as O.
This demonstrates that the Tweedie estimate of the posterior mean converges to the identity mapping at rate proportional to VB..
We show this empirically in Figure 9
Vb, vst
[{x)(x)) x,1l vs
1.2
300
170
0.4
20
250 800
1002
200
Figure 9: We show that [|x) - x.12 = O(VB)) demonstrating that the Tweedie estimate of the
posterior mean converges to the identity mapping as 0. (Left) plots v3, (Right) plots [x;) X; 2l2 for a Gaussian deblur task. At higher f, model inaccuracies in estimating Vx, log pl x,) cause deviations from the convergence pattern.
Convergence of Constraint Satisfaction. Based on the convergence of the Tweedie estimate X we show that for sufficiently small t:
1. The constraint set {xi Axx (x²) = y is non-empty.
2. Gradient descent on |x,) - y1 converges to point satisfying the constraint.
First, we show that the optimization landscape becomes increasingly well-behaved as - Con- sider the objective:
|Axx(x.) 2 = ||Ax + A(8o(x;) - x2)11?
(39)
= ||Ax, -yl
2(2xx: - Y, A(x xx))
+ ||A(xx(x)) x11
(40)
By Tweedie Convergence, the second and third terms bounded by (ss()). Therefore, 0. the objective converges to the convex quadratic ||Ax; - y which can be efficiently optimized via gradient descent.
For feasibility, note that as t - 0, finding X, such that A = y becomes equivalent to finding x, such that Ax, y up to emrоr of (({(t). The latter is feasible whenever y is in the range of A, which the standard assumption for linear inverse problems.
0, e(l) approaches zero, making the constraint feasible. Moreover, since the objective approaches convex quadratic, gradient descent will converge to the global minimum for sufficiently small t.
As /
18

---

D
Ablation Studies
D.1 Stopping Criteria
We consider variations of the stopping criteria where we use a different threshold besides I standard deviation on noiseless random inpainting task when T" = 25. We consider three different stopping criteria and report the LPIPS on the FFHQ test set as well as the total projection steps during the inference. Notice that having more generous stopping criteria does not always lead to fewer projection steps. We risk underfitting at early timesteps, and as the plausible region reduces to radius of 0, we must take extra steps at later time steps to ensure consistency.
Stopping criteria
Total Projection Steps
Random Inpainting LPIPS
PR + 3
24
0.296
+ OR
15
0.171
PR + 0.50R
25
0.169
Table 2: Comparing stopping criteria based on standard deviations to the expected residual energy.
E
Additional Experimental Details
E.1
Task Details
We describe additional details for each inverse task used in our experiments.
Super Resolution Images are downsampled to 64 x 64 using bicubic downsampling with a factor of 4.
Box Inpainting A random box of size 128 128 chosen uniformly within the image. Those pixels are masked out affected all three the RGB channels.
Gaussian Deblur A Gaussian Kernel of size 61 x 61 and intensity 3 is applied to the entire image.
Random Inpainting Each pixel is masked out with probability 92% affecting all three of the RGB channels
50% Inpainting In various figures, we showcase 50% inpainting task where the top half of an image is masked out. This task is more challenging than box inpainting and can better illustrate differences between results.
E.2 Number of Projection Steps
Across variety of tasks and dataset find that 7" 25 denoising steps leads average of 27.2 total projection steps and = 50 leads to an average of 46.5 total projection steps. For exper- iments where want strictly limit the NFEs for fair comparison, simply disallow additional projection steps after reaching the limit.
E.3 Measuring Runtime
To measure wall-clock runtime, we used single 1 and ran all the inverse problems (super- resolution, box inpainting, gaussian deblur, random inpainting) on the FFHQ dataset. We only consider the runtime of the algorithm, without considering the python initialization time, model loading, or image For each task, we measured the runtime on images and averaged the result to produce the final result. We note that the baseline runtimes are taken from [5), where only the box inpainting task was considered. The runtime does not vary much between tasks when using CDIM, so we report average runtime across tasks as a fair comparison metric.
19

---

E.4 ImageNet Results
In Table we report FID and LPIPS for ImageNet.
Table 3: Quantitative results (FID, LPIPS) of our model and existing models on various linear inverse problems on the Imagenet 256 x 256- 1kk validation dataset. (Lower is better)
Super
Inpainting
Gaussian Deblur
Inpainting
Imagenet
Resolution
(box)
(random)
Methods
FID
LPIPS
FID
LPIPS
FID
LPIPS
FID
LPIPS
Ours T 25
53.70
0.378
52.00
0.267
56.10
0.393
51.96
0.370
Ours T = 50
47.45
0.339
50.31
0.251
38.69
0.347
46.20
0.332
FPS-SMC
47.30
0.316
33.24
0.212
54.21
0.403
42.77
0.328
DPS
50.66
0.337
38.82
0.262
62.72
0.444
35.87
0.303
DDRM
59.57
0.339
45.95
0.245
63.02
0.427
114.9
0.665
MCG
144.5
0.637
39.74
0.330
95.04
0.550
39.19
0.414
PnP-ADMM
97.27
0.433
78.24
0.367
100.6
0.519
114.7
0.677
Score-SDE
170.7
0.701
54.07
0.354
120.3
0.667
127.1
0.659
ADMM-TV
130.9
0.523
87.69
0.319
155.7
0.588
189.3
0.510
E.5
PSNR Results
See Tables 4 and 5
Table 4: Quantitative results (PSNR) of our model and existing models on various linear inverse problems on the FFHQ 256- 1k validation dataset. (Higher is better)
Super
Inpainting
Gaussian Deblur
Inpainting (random)
Imagenet
Resolution
(box)
Methods
PSNR
PSNR
PSNR
PSNR
Ours T 25
27.08
23.20
26.77
26.49
Ours T" 50
27.30
23.47
27.03
27.10
FPS-SMC
28.10
24.70
26.54
27.33
DPS
25.67
22.47
24.25
25.23
DDRM
25.36
22.24
23.36
9.19
MCG
20.05
19.97
6.72
21.57
PnP-ADMM
26.55
11.65
24.93
8.41
Score-SDE
17.62
18.51
7.21
13.52
ADMM-TV
23.86
17.81
22.37
22.03
Table 5: Quantitative results (PSNR) of our model and existing models on various linear inverse problems on the Imagenet 256 x 256-16 1k validation dataset. (Higher is better)
Super
Inpainting
Gaussian Deblur
Inpainting (random)
Imagenet
Resolution
(box)
Methods
PSNR
PSNR
PSNR
PSNR
Ours T" 25
23.67
19.67
22.78
22.38
Ours T' 50
23.92
20.06
23.32
22.61
FPS-SMC
24.78
22.03
23.81
24.12
DPS
23.87
18.90
21.97
22.20
DDRM
24.96
18.66
22.73
14.29
MCG
13.39
17.36
16.32
19.03
PnP-ADMM
23.75
12.70
21.81
8.39
Score-SDE
12.25
16.48
15.97
18.62
ADMM-TV
22.17
17.96
19.99
20.96
20

---

Additional Comparisons
F.1 Comparison with DSG
We show a qualitative comparison against DSG [28] on 3 tasks in Figure I0 We used the official code from their github, and generated results with 25 DDIM diffusion steps for both DSG and CDIM (and K = for CDIM). As you can sec, the DSG results are blurrier and sometimes contain artifacts
DSG 25 Denoising Steps
CDIM 25 Denoising Steps
Input
GT
Random Inpainting
Gaussian
Deblur
Super Resolution
Figure 10: A comparison between CDIM and DSG (28 when both algorithms use 25 DDIM de- noising steps. Notice the artifacts the DSG random inpainting.
F.2 Comparison with Diff-PIR
Below we report quantitative comparisons against Diff-Pir 25 which uses 100 NFEs. These num- bers reported directly from their paper. Alhtough the LPIPs numbers comparable better with Diff-PIR on FFHQ, the FID numbers are noticeably worse and LPIPs is also worse on Ima- genet.
Table Comparison between CDIM and Diff-PIR (5
the FFHQ dataset.
Super Res
Gaussian Deblur
NFEs
FFHQ
Methods
FID
LPIPS
FID
LPIPS
Ours T"
25
33.87
0.276
34.18
0.276
~50
Ours T
31.54
0.269
29.68
0.252
~ 100
Diff-PIR
65.77
0.260
59.65
0.236
100
Table 7: Comparison between CDIM and Diff-PIR 25
the Imagenet dataset.
Super Res
Gaussian Deblur
NFEs
Imagenet
Methods
FID
LPIPS
FID
LPIPS
Ours
53.70
0.378
56.
0.393
50
Ours 7" = 50
47.45
0.339
38.69
0.347
100
Diff'
106.32
0.371
93.36
0.355
100
21

---

F.3 Comparison with DAPS
Below we report quantitative comparisons against DAPS-50 18 which uses 50 NFEs. We run the official code on 100 test images from FFHQ and report the LPIPS loss. These numbers differ from the numbers in the paper because (1) the paper uses By 0.05 for images in [-1. 1] while our paper and other baselines have images in [0, 1] which leads to twice as much noise and (2) the paper
reports L.PPs with model "alex" while ours and baselines use "vgg". Both of these cause the reported numbers in the paper to be significantly better, so we run experiments ourselves to ensure a fair comparison.
Input
GT
DAPS-50
CDIM T'=25
Gaussian Deblur
Random Inpainting
Super Resolution
Box Inpainting
Figure 11: Comparison between CDIM (T'=25) and DAPs-50 where both are capped at 50 NFEs. All of the DAPs-50 results contain noticeable artifacts compared to CDIM.
Table 8: Comparison between CDIM (T" 25) and DAPs-50 on the FFHQ dataset.
Methods
4x Super-Resolution
Box Inpainting
Gaussian Blur
Random Inpainting
DAPs-50
0.358
0.244
0.339
0.306
CDIM (T" = 25)
0.290
0.187
0.283
0.271
22

---

G
Extended Results
Ground Truth
Input
Ours T'=25
Ours T'=50
Figure 12: FFHQ Super-resolution extended results
23

---

Ground Truth
Input
Ours T'225
Ours T'=50
Figure 13: FFHQ Gaussian deblur extended results
24

---

Ground Truth
Input
Ours T'225
Ours T'=50
Figure 14: FFHQ random inpainting extended results
25

---

Ground Truth
Input
Ours T'=25
Ours T'=50
Figure 15: FFHQ box inpainting extended results
26

---

Ground Truth
Input
Ours T'=25
Ours T'=50
Figure 16: ImageNet Gaussian deblur extended results
27

---

Ground Truth
Input
Ours T'=25
Ours T'=50
Figure 17: ImageNet random inpainting extended results
28

---

Ground Truth
Input
Ours T'=25
Ours T'=50
Figure 18: ImageNet box inpainting extended results
a
(b)
Figure 19: Results on inpainting 50% of an image on LSUN Churches dataset.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we discuss the accelerated projection algorithm and discuss how speeds up constrained diffusion compared existing methods. We focus contri- bution of speedup
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss a few limitations such as not handling latent diffusion. However there could be more discussion of overall limitations
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [Yes]
Justification: Yes, our main claim (Proposition 1) is fully derived the appendix. Other claims, such the chi-squared distribution of residuals are derived in sufficient depth as they are straightforward.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relics upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe all experimental settings, hardware considerations, hyperparam- eters, and other requirements for running the experiments. Code will be released.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-souree models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
31

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code will be open sourced shortly after the review deadline.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
mublic/guides/ htaiii//giides//ddeeubbiissee powreddeuummssioooooiiooo
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. cttpsscccpubbll wwcpppplccccccccccccccccccc htw.blicgguides///eeeee http://guiddeeeeeuuuuuitiieeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we have discussed the training and test datasets, hyperparemeters, stop- ping criteria, etc.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We report many metrics like FID, LPIPS, PSNR, etc. These are standard in the community.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yss]
Justification:
Yes we describe the hardware used and execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : :/tn://nsnssp.pe hctpp//bbbccc.ccccccc.eeleelllll
Answer: [Yes]
Justification:
We have done our best to follow all ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are some negative socictal of diffusion posterior sampling but we do not believe it warrants a full broader impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not believe safeguards are required for this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions access the model or implementing safery filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
All IP is from standard community datasets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, wwtppppppppshooooo cow com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
We do not release no assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing was done
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects were used in experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not use LLMs outside of helping with basic writing/editing.
Guidelines:
The answer NA means that the method development in this rescarch does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neurihe ctttc//oeeeeeecce//2//////////// for what should or should be described.
36


Reviews:
[{"review_id": "FRhtJ6kKAK", "reviewer": "Reviewer_RQq2", "summary": "The paper introduces CDIM, a method for solving linear inverse problems through adaptive projection sampling (using a pretrained diffusion model). Authors show that under the forward noising process, the measurement residual follows a chi-square distribution, with tractable mean and variance. Using this, CDIM chooses projection step sizes and stopping criteria adaptively and projects $x_t$ just enough to keep residual within one standard deviation of its expected value at each step. \nIn the noiseless setting, this adaptive method guarantees exact constraint satisfaction and bounds the final reconstruction error with high probability. \n\nTheir empirical results show that CDIM achieves image quality on par with SoTA methods like DPS with 50x fewer model evaluations across multiple tasks on FFHQ and ImageNet, and some samples new applications like photo time travel and 3D point-cloud reconstruction.", "questions": "1. Would it be possible to extend this method for latent-based models?\n3. How sensitive are final results to the choice of \"±1 σ\" as the stopping criterion? What if you choose a dynamic tolerance (e.g. varying with t) improve performance?\n2. Section D of appendix looks incomplete. Subsection D.1 has typos and inconsistency in notation (like \"64 ∝ 64\" and \"showcase a a 50% inpainting\"). Subsection D.4 should be referring to table 3?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper proposes a novel method for using pretrained diffusion models to solve inverse problems that uses an adaptive projection mechanism grounded in the residual distribution.\n2. The method achieves the same level of measurement fidelity with an order of magnitude fewer denoiser calls.\n3. Empirical results are provided on FFHQ and ImageNet datasets which are standard sets for these tasks.\n\nWeakness:\n1. In a couple of cases, CDIM shows perofrmance that is close, but not exceeding the best slower baseline. For example, DPS outperformed  on random inpainting and FPS-SMC outperformed CDIM on super resolution FID.", "was_revised": true, "final_justification": "Authors have addressed my concerns and I continue to support acceptance of this work.", "rebuttal": "We would like to thank the reviewer for their positive feedback and thoughtful comments. We are glad that they appreciate the novelty and strength of the results. Below we address the specific concerns/questions raised. We plan to upload a revision with all these specific fixes.\n\n**“In a couple of cases, CDIM shows performance that is close, but not exceeding the best slower baseline.”**\n\nWe acknowledge that this is true, however the baselines mentioned use far more compute than us. When restricting those methods to a same amount of compute (such as the DPS + DDIM image in figure 5), our method performs significantly better.\n\n**“Would it be possible to extend this method for latent-based models”**\n\nWe note that, like DPS, our method can be extended to latent-based models with some amount of success. However, because Tweedie’s formula for the expected posterior mean does not pass through a non-linear function, $A(E[\\hat{x}_0]) \\neq E[A(\\hat{x}_0)]$, we cannot get the same theoretical guarantees on convergence of the constraint, or the same speedup. In addition, other heuristics may be required to get satisfactory convergence in latent models, which is why we have left this for a follow up work.\n\n**“How sensitive are final results to the choice of $\\pm \\sigma$ as the stopping criterion? What if you choose a dynamic tolerance (e.g. varying with t) improve performance?”**\n\nIn Figure 4 we have shown qualitative results when different stopping criteria than $\\pm 1 \\sigma$ are used.  Additionally, in Table 2 we have given quantitative results both on the resulting runtime and the generation quality for a chosen task when using these different stopping criteria. Although it would be possible to choose a dynamic tolerance, we note that the standard deviation $\\sigma$ is itself highly informative on the plausible region, decreasing dramatically as t->0, so a constant factor is sufficient.\n\n**“Section D of appendix looks incomplete.. typos and inconsistencies”**\n\nWe thank the reviewer for pointing this out. We will address the typos and inconsistency in the revised version. Indeed subsection D.4 should be referring to table 3 and 4, and we will revise the inconsistencies in experimental notation."}, {"review_id": "DDvzawsgEx", "reviewer": "Reviewer_iTEy", "summary": "The goal of this paper is to solve linear inverse problems using a pretrained generative model. While many existing methods in the literature (e.g., guidance-based or Plug-and-Play approaches) can tackle such problems, none of them explicitly enforce the linear constraint $Ax = y$ in the case of noiseless degradation. However, in certain applications—such as inpainting—it is meaningful to impose a hard constraint, since observed (non-masked) pixels should remain unchanged during reconstruction.\n\nThis paper aims to address this limitation by proposing an efficient method for solving inverse problems with hard constraints. The main contributions are:\n\n- An accelerated DDIM-based scheme for the prior denoising step.\n\n- A penalized formulation coupled with gradient descent to enforce constraint satisfaction.\n\n- Practical heuristics for step size selection during optimization.\n\n- Experimental results demonstrating competitive visual quality, faster computation compared to state-of-the-art methods, and better constraint satisfaction.", "questions": "First of all, I would like to thank the authors for this paper. I believe the two main questions addressed—hard constraints and acceleration—are relevant for the community. There are some interesting ideas in the paper, but I think the role of each component deserves further clarification.\n\n**1. On the empirical results and ablations**\n\nI appreciated the ablation study against DPS (Fig. 5). Do you have an explanation for why DPS performs poorly when combined with DDIM? Apart from the use of DDIM and the adaptive step size, is there any fundamental difference between your method and DPS?\n\nIn the same figure, the claim that “DPS doesn’t match the constraint” is not immediately obvious. You might consider reporting the numerical error on the constraint $Ax = y$ for each method.\n\nIt would also be interesting to highlight your method on box inpainting tasks, where other methods may suffer from hallucinations (i.e., inventing content in known regions). This would make the hard constraint enforcement more visible.\n\n**2. On the theoretical justification of constraint satisfaction**\n\nUnfortunately, I remain unconvinced by the theoretical arguments regarding constraint satisfaction:\n\na. Once the constrained optimization problem is replaced with a penalized one, there is little to no guarantee that the constraints will be exactly satisfied.\n\nb. The justification for the penalized formulation (footline on p.4) is questionable. If I understand correctly, the constraint $A \\hat{x_0}(x_{t-1}) = y$ should hold at $t = T$, not $A \\hat{x_0}(x_t) = y$. So the argument seems invalid as stated.\n\nc. Referring to Eq. (11) as a “Lagrangian” is incorrect. The formulation lacks dual variables and does not follow the standard Lagrangian structure.\n\nd. I went through the proof in Appendix C and found several steps questionable. For instance:\nHow do the assumptions in lines 336–337 imply that $x_t$ is bounded?\nI disagree with the claim made in line 348.\nThere seems to be a confusion in the status of $x_t$: sometimes it is defined by the diffusion model, other times it is treated as an optimization variable. In Eq. (37), for instance, $x_t$ is a variable being optimized, so the statement that $|A \\hat{x}_0 - x_t| \\to 0$ is not meaningful in this context.\n\ne. Have you considered using an Augmented Lagrangian approach? While this may come at a computational cost, it would provide a principled way to handle hard constraints.\n\n**3. Additional Baseline**\nIt would be useful to include a numerical comparison with Diff-PIR [1], which is a fast PnP method and may serve as a relevant baseline.\n\n**4. Additional suggestions and clarifications**\n\n- I could not fully understand Proposition 1. It seems the notations it relies on are only introduced later, making it hard to follow.\n\n- The Introduction and Related Work sections could benefit from further elaboration. In particular, it would help to provide more details about DPS, which is your main point of comparison.\n\n- To improve readability, I suggest consistently using the notation $\\hat{x}_0(x_t)$ to emphasize the dependence on $x_t$ in the reconstructed estimate.\n\n[1] Zhu, Y.; Zhang, K.; Liang, J.; Cao, J.; Wen, B.; Timofte, R. & Van Gool, L. Denoising diffusion models for plug-and-play image restoration, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2023, 1219-1229", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well written and relatively easy to follow overall.\n\n- The numerical results are convincing, particularly in terms of computational speed-up compared to methods like DPS.\n\n- The acceleration of the DDIM scheme is an interesting and relevant contribution.\n\n\n**Weaknesses**\n\n- *Theoretical gaps*: The authors claim that their method satisfies the hard constraints and provide informal arguments in support. However, these arguments appear to be mathematically incorrect or at least imprecise. While the numerical results are more convincing, the theoretical justification remains weak.\n\n- *Lack of focus and clarity on contributions*: The paper seems to pursue two distinct goals:\n(1) enforcing hard constraints in inverse problems, and\n(2) proposing a fast and visually effective diffusion-based method for inverse problem solving.\nThe first objective is less convincing, given the theoretical limitations mentioned above. The second is more promising, but the paper lacks a clear analysis of which components of the method (accelerated DDIM, step-size heuristics, gradient descent updates) are most responsible for the performance gains. A more modular ablation or discussion would improve clarity on this point.", "was_revised": true, "final_justification": "The authors were cooperative and addressed my questions. The additional numerical results convinced me that the method is valuable, especially given that relatively few papers tackle hard-constrained inverse problems. The authors also acknowledged that the paper is primarily empirical, and that the theoretical arguments are mostly heuristic.", "rebuttal": "We thank the reviewer for the detailed and helpful comments. We’re glad they found the results strong and the contribution relevant, and appreciate the careful review of the proofs. Below we address all comments and will fix the issues noted. We hope the author will consider the below for a score improvement.\n\nWe would like to start by addressing the main concern regarding the theoretical justification for constraint satisfaction. The reviewer asks about the proof in Appendix C. We note that this proof specifically describes the rate of convergence at which $\\lVert A\\\\hat{x}\\_0(x\\_{t-1}) - y\\rVert^2 \\\\rightarrow 0$ (which we also demonstrate empirically). However the fact that the constraint can be satisfied follows from the following, much simpler argument: As $t \\\\rightarrow 0$ the Tweedie's estimate becomes the identity function $\\\\hat{x}\\_0(x_t) \\\\rightarrow x_t$ and therefore the optimization problem $min\\_{x\\_{t-1}} ||A\\\\hat{x}\\_0(x\\_{t-1}) - y||^2$ becomes the **concave quadratic** $min\\_{x\\_{t-1}} ||Ax\\_{t-1} - y||^2$ . Specifically, at $t-1 = 0$, we can write this out as $x\\_0 \\gets min\\_{x\\_0} ||Ax\\_0 - y||^2$ and this quadratic can be optimized to arbitrary precision by following the gradients $\\\\nabla x\\_{t-1}$ of our current sample at $t-1 = 0$. We are performing exactly this minimization at $t-1 = 0$ because our stopping criteria is $||Ax\\_{t-1} - y||^2 \\\\leq \\\\mu_{t-1}(y) \\\\pm \\\\sigma\\_{t-1}$ where $\\\\mu\\_{t-1}(y) \\\\rightarrow 0$, $\\\\sigma\\_{t-1} \\\\rightarrow 0$ as $t-1 \\\\rightarrow 0$.  By using the optimal step size $\\\\eta$ via line search, the constraint is met to arbitrary precision (assuming that $y$ is in the range of $A$). This is in contrast to methods like DPS that make a single update to the constraint for each denoising update, and use heuristic step sizes that may not decrease the constraint error to zero. Later in this rebuttal we show quantitatively that the mean absolute pixel error for an observed region in inpainting goes to 0 for our method while it does not for DPS.\n\n\\\n**“The paper seems to pursue two distinct goals: (1) enforcing hard constraints and (2) fast diffusion-based inverse problem solving”**\n\nWe thank the reviewer for this comment and plan to strengthen our discussion of the connection between these two goals in the revision. These goals are closely related: accelerated unconditional sampling methods like DDIM struggle with constrained sampling because the constraint enforcement steps typically do not converge as fast as the unconditional steps. By proposing a new principled method that quickly decreases the constraints (enforcing the hard constraints even in accelerated sampling), we are able to produce high quality results for fast diffusion-based inverse problem solving. That fact that we guarantee hard constraints is a result of the proposed step size mechanism which is designed for fast posterior sampling by finding the optimal number of steps and step size at each iteration.\n\n\\\n**“Do you have an explanation for why DPS performs poorly when combined with DDIM? Apart from the use of DDIM and the adaptive step size, is there any fundamental difference between your method and DPS?”**\n\nDPS takes one constraint step per denoising step with a heuristic step size (footnote 5, p.6 of DPS) and doesn't take into account the additive noise distribution. In contrast, our method proposes a theoretically grounded method for taking optimal steps on the constraint, showing it guarantees constraint recovery and that the optimal step size can be analytically computed without a network pass, using only A’s functional form.\n\n\\\n**“DPS doesn’t match the constraint” is not immediately obvious. You might consider reporting the numerical error on the constraint”**\n\nWe thank the reviewer for this suggestion. Based on your suggestion we have run the following experiment to report in this rebuttal and for the paper revision. We performed box inpainting with no measurement noise, meaning we should exactly match the constraint, $Ax = y$ on the observed region. We ran both CDIM and DPS with DDIM, each with 50 total NFEs (25 denoising steps). Below we report the per-pixel mean absoluate error $\\\\frac{1}{m}\\\\Sigma|Ax\\_t - y|$ at the end of select timesteps: This is the measure of how far off the average pixel deviates from ground truth in the known region. (Pixel values between -1 and 1).\n\nT = 960\\\nCDIM: 0.72\\\nDPS: 0.66\n\nT = 480\\\nCDIM: 0.61\\\nDPS: 0.62\n\nT = 40\\\nCDIM: 0.0058\\\nDPS: 0.072\n\nT = 0\\\nCDIM: 0.00045 (In theory would be 0 but we have $\\\\overline{\\\\alpha}\\_0 = 0.9999$ not $1$ for numerical stability)\\\nDPS: 0.069\n\n\nCDIM matches the constraint to numerical precision, while DPS differs from observed pixels by an average of almost 7%. Any attempt to make DPS more closely match the constraint, for example increasing the gradient step size, results in divergence for accelerated sampling schedules.\n\n\n\\\n**“It would also be interesting to highlight your method on box inpainting task”**\n\nWe include several examples in the paper of a 50% inpainting task (harder than box inpainting) where we exactly match the constraint. As an example, consider Figure 3 where sub-panel b where we can force the observed region to match the noisy pixels exactly even when they are out of distribution.\n\n\\\n**“a. Once the constrained optimization problem is replaced with a penalized one, there is little to no guarantee that the constraints will be exactly satisfied.”**\n\nWe note that as $t \\\\rightarrow 0$, the penalized optimization turns into the hard constraint problem given by equation (10). This is because our stopping criteria $\\\\sigma$ goes to 0 as $t \\\\rightarrow 0$, so there is no early stopping. We continue running optimization steps on $||A\\\\hat{x}\\_0(x\\_{t-1}) - y||^2$ until we reach the desired level of convergence. Because this is concave quadratic, it can always be optimized to sufficiently meet the hard constraint. Therefore we have guarantee of exact constraint satisfaction at $t=0$ (and only at $t=0$ because higher noise levels do have the early stopping criteria).\n\n\\\n**“b. The justification for the penalized formulation (footline on p.4) is questionable. If I understand correctly, the constraint $A\\\\hat{x}\\_0(x\\_{t-1})$ should hold at t = T. So the argument seems invalid as stated.”**\\\nWe thank the reviewer for pointing out this subtle fact. It is true that the constraint in question is $\\\\hat{x}\\_0(x\\_{t-1})$, so even at the first optimization step, we are computing the Tweedie’s estimate $E[x\\_0 | x\\_{T-1}]$ and not $E[x\\_0 | x\\_{T}]$. However, the intuition holds that even at $t=T-1$, $x\\_t$ is largely uninformative, so finding $x\\_t$ where $A\\\\hat{x}\\_0(x\\_{t-1})=y$ is effectively impossible. The tweedie’s estimate is the expectation over all denoising trajectories, which would still be a blurry image at high noise values, even if say we set $x\\_{T-1}$ to our GT image.\n\n\\\n**“c. Referring to Eq. (11) as a Lagrangian is incorrect.”**\n\nWe thank the reviewer for pointing out this correction. We will change the description of the equation.\n\n\\\n**“d. Proof in Appendix C”**\\\nWe thank the reviewer for the careful reading and agree that Appendix C requires clearer exposition.  \n\n\n*Boundedness of $x_t$*\\\nSince the forward process is a convex combination of a bounded datum $x_0$ where $||x\\_0|| \\\\le R$ and Gaussian noise with coefficient $\\\\sqrt{\\\\beta_t}\\\\le 1$, the resulting sample is almost surely bounded: $||x\\_t|| \\\\le R+O(\\\\sqrt{\\\\beta_t})$. We will add this explicit bound. \n\n*Role of $x\\_t$ versus $\\\\hat x\\_0(x\\_t)$*\\\nThroughout Appendix C, $x\\_t$ is the optimization variable selected by the projection step, while $\\\\hat x\\_0(x\\_t)$ is its Tweedie estimate. Equation 37 is saying that the Tweedie’s estimate of our current optimization iterate converges to the identity function. We will make this distinction explicit to avoid confusion. We don't intend to say that $||A\\\\hat{x}\\_0 - x\\_t|| \\\\rightarrow 0$ but that $||\\\\hat{x}\\_0(x\\_t) - x\\_t|| \\\\rightarrow 0$, meaning that optimizing the tweedie estimate of the iterate is the same as optimizing the iterate itself at low noise values. We also show this empirically.\n\n*Convexity of the objective as $t\\\\to 0$*\\\nTweedie convergence gives $||\\\\hat x\\_0(x\\_t) - x\\_t|| = O(\\\\beta\\_t)$; expanding Eq.~(37) shows that $||A\\\\hat x\\_0(x\\_t) - y||^2 = ||A x\\_t - y||^2 + O(\\\\beta\\_t)$. Hence, for sufficiently small $t$, the landscape is an arbitrarily small perturbation of the convex quadratic $||A x_t - y||^2$, so standard gradient descent reaches any desired accuracy. \nThese clarifications will be incorporated, but they leave the main conclusion unchanged: as $t-1\\to 0$, gradient descent on $||A\\hat x_0(x_{t-1}) - y||^2$ can satisfy the constraint to arbitrary precision.\n\n**“It would be useful to include a numerical comparison with Diff-PIR [1]”**\\\nWe thank the reviewer for pointing out Diff-PIR. Below are some direct comparisons reporting LPIPS (lower is better), which are also added to the paper\n\n4x Super Resolution Imagenet:\\\nDiff-Pir (100 NFEs): 0.371 LPIPS\\\nOurs (100 NFEs): 0.339 LPIPS\n\nGaussian Deblur Imagenet:\\\nDiff-Pir (100 NFEs): 0.355 LPIPS\\\nOurs (100 NFEs): 0.347 LPIPS\n\nWith the same number of NFEs, our method performs better.\n\n\\\n**“I couldn't fully understand Proposition 1...relies on notation introduced later,”**\n\nThank you for pointing this out, indeed the notation was introduced after the proposition making it hard to follow. The key idea of Proposition 1 is that we can choose our optimal step size based on the value of $||Ax\\_t - y||$ (calculable via line search without a network pass) and that this also guarantees that our true target, $||A\\\\hat{x}\\_0 - y||$ decreases towards its desired value by a commensurate amount.\n\n\\\n**“To improve readability, I suggest consistently using the notation $\\\\hat{x}\\_0(x\\_t)$”**\\\nWe thank the reviewer for this suggestion. It has also been suggested by other reviewers and we will change the notation in the revised paper."}, {"review_id": "tn6oBOi37h", "reviewer": "Reviewer_kQS1", "summary": "The authors introduce CDIM, a method for solving noisy linear inverse problems using pretrained diffusion models. CDIM modifies DDIM sampling by projecting each step to enforce consistency with linear measurements. It handles Gaussian and Poisson noise, achieves exact recovery in the noiseless case, and performs well on tasks like inpainting and super-resolution. The authors conduct experiments across multiple linear inverse problems and 3D point cloud reprojection.", "questions": "Why are variational methods, such as RedDiff, and recent flow-based baselines omitted from the comparisons?\n\nHow does the method perform on more challenging settings like 8x SR?\n\nCan the approach be extended to latent diffusion models or nonlinear measurement settings, or is it inherently limited to pixel-space and linear operators?", "limitations": "The paper does not discuss any limitations, despite the fact that the method cannot be applied to nonlinear forward operators.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper is clearly written, well-motivated, and presents its core ideas in a structured and accessible manner. The proposed projection-based modification of DDIM sampling is easy to implement and allows for consistent reconstruction under linear constraints. \n\nWeaknesses:\nDespite these strengths, the contribution is somewhat limited in novelty. The primary reason for the speedup is the use of DDIM instead of DDPM, which is already well-established. Several strong baselines are missing, including variational approaches [1] and recent flow-based methods that currently define the state of the art. Some of the experiment, such as 4x super-resolution or sparse inpaintin are too simple and could be effectively addressed with classical or non-generative methods, limiting the practical relevance of the comparisons. Lastly, the method is restricted to pixel-space models due to its reliance on linear measurement operators, and cannot be applied in latent diffusion or nonlinear inverse problems.\n\n[1] A Variational Perspective on Solving Inverse Problems with Diffusion Models", "was_revised": true, "final_justification": "The authors addressed my main concerns in the rebuttal.", "rebuttal": "We thank the reviewer for their comments. We are slightly confused by the reviewer’s summary of the paper, as they mention how our method uses KL divergence minimization, something that is not mentioned once in the paper. KL divergence and the L2 residual minimization that the reviewer mentions are from a non-anonymized arxiv preprint of the paper from several months ago. The version submitted for review has a substantially different method that does not incorporate KL divergence minimization or discuss L2 residual minimization. We would like to make sure that the reviewer has engaged with the method submitted for review, given that the primary weakness stated is the claim that our method is not novel. \n\n\\\n**“The primary reason for the speedup is the use of DDIM instead of DDPM”**\n\nWe have shown experimental results that simply using DPS with DDIM does not yield good results (see Figure 5). Furthermore, methods like DPMC use DDIM and still require over 200 NFEs to get good results. The speedup comes from our conditional step (DDIM is only for the unconditional steps), which enables fast convergence to the constraint in fewer overall NFEs than previous methods.\n\n\\\n**“Why are variational methods, such as RedDiff, and recent flow-based baselines omitted from the comparisons?”**\nBased on feedback from reviewers we have added a comparison with DAPs, a recent state of the art work from CVPR 2025 that outperforms RedDiff.\n\n To facilitate a fair comparison, we used the official code and ran the configuration for DAPs-50 (50 NFEs) which has a comparable runtime as CDIM with T’ = 25 denoising steps. Below are the results on 100 images from the FFHQ test set reporting LPIPS (net=”vgg”). Lower is better.\n\nRandom inpainting:\\\nDAPs-50 : 0.306 LPIPS\\\nCDIM (T’=25): 0.271 LPIPS\n\nGaussian blur:\\\nDAPs-50: 0.339 LPIPS\\\nCDIM (T’=25): 0.283 LPIPS\n\n4x Super Resolution:\\\nDAPs-50: 0.358\\\nCDIM (T’=25): 0.290\n\n\nAs you can see, CDIM outperforms DAPs when both are restricted to the same fast sampling schedule (even though DAPs-1k exhibits stronger performance with 10x the runtime as CDIM).\n\nWe also have numerous qualitative comparisons that we will include in the revision and would like to share with the reviewer. The DAPs-50 images are slightly noisy due to the accelerated schedule, and the quality difference is visibly apparent.\n\nWe can therefore see that when restricted to a fast sampling environment (50 NFEs or 4 seconds), our method outperforms DAPS, a very recent and state-of-the-art method. \n\n\\\n**“How does the method perform on more challenging settings like 8x SR?”**\n\nBelow we include PSNR and LPIPS on 8x superresolution with $sigma_y = 0.05$ Gaussian measurement noise with T’=50 steps (approx 10 second inference) .Given that we cannot share images with the reviewer, we cannot send the qualitative 8x super resolution results, but will include some examples in the revised paper.  Regarding more challenging tasks, we also show several qualitative examples in the paper of 50% inpainting, a significantly harder task than the 25% box inpainting task used in the experiment section. \n\n8x superres results\\\nFFHQ\\\nLPIPS: 0.353\\\nPSNR: 23.63\n\nImageNet\\\nLPIPS: 0.457\\\nPSNR: 19.67\n\n\\\n**“Can the approach be extended to latent diffusion models or nonlinear measurement settings, or is it inherently limited to pixel-space and linear operators?”**\n\nJust like DPS, the method can work on latent and non-linear operators, but the theoretical guarantees fail since we rely on linearity of expectation of the Tweedie’s estimate: A(E(x)) = E(A(x)). Therefore the quality improvements in highly accelerated sampling schedules is not as evident for non-linear operators. We have run phase retrieval and non-linear deblurring and plan to include those results in the revision.\n\n\\\n**“The paper does not discuss any limitations despite the fact that the method cannot be applied to nonlinear forward operators”**\n\nNote that in the Conclusion we discuss how we do not handle non-linear constraints because of the linearity of expectation."}, {"review_id": "4pLcHQFwLf", "reviewer": "Reviewer_XQx7", "summary": "This paper proposes Conditional Diffusion Implicit Models (CDIM), a novel framework for solving linear inverse problems using pre-trained diffusion models. CDIM effectively addresses both noisy and noise-free linear inverse problems. By imposing constraints on the prior diffusion objective, it efficiently solves these problems in terms of both computational time and utility. The paper also incorporates techniques such as Early Stopping and an adaptive learning rate to facilitate more efficient convergence. Experimental results demonstrate that CDIM is fast, powerful, and straightforward to implement with pre-trained diffusion models, requiring no additional modules.", "questions": "Please address the weakness of the previous part", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. At the heart of this work is a novel integration of the DDIM sampling procedure within an optimization-based framework, ensuring close alignment between the posterior mean and the observed data. \n2. The manuscript is clearly written, logically organized, and supported by comprehensive experiments. \n3. It also provided a analysis how  CDIM projection strategy both enforces measurement consistency and dramatically cuts down the number of diffusion‐model evaluations.\n\n\nWeakness: \n1. While the proposed method shows potential, its performance still lags behind state-of-the-art approaches such as DAPs [6], and it has not been evaluated on nonlinear inverse problems (e.g., nonlinear deblurring or phase retrieval).\n\n2. The experimental section omits comparisons with several recent diffusion-based sampling algorithms for inverse problems [1–7]. At minimum, the authors should benchmark against leading methods like DAPs [6]\n\n[1] Song, Jiaming, et al. \"Pseudoinverse-Guided Diffusion Models for Inverse Problems.\" ICLR 2023.\n\n[2] Song, Jiaming, et al. \"Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation.\" ICML 2023.\n\n[3] Li, Xiang, et al. \"Decoupled Data Consistency with Diffusion Purification for Image Restoration.\" arXiv preprint arXiv:2403.06054 (2024).\n\n[4] Wu, Zihui, et al. \"Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors.\" arXiv preprint arXiv:2405.18782 (2024).\n\n[5] Xu, Xingyu, et al. \"Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstruction.\" arXiv preprint arXiv:2403.17042 (2024).\n\n[6] Zhang, Bingliang, et al. \"Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing.\" CVPR 2025.\n\n[7] Dou, Zehao, et al. \"Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective.\" ICLR 2024.", "was_revised": true, "final_justification": "I think the author solved my concern and I will raised my score to 4", "rebuttal": "We thank the reviewer for their positive comments and helpful suggestions. We are glad that the reviewer appreciated the novelty of the method and our comprehensive experiments, and also gave us a good score for all areas: quality, clarity, significance, and originality.\n\nWe appreciate the additional comparisons suggested. DAPs is a very recent state-of-the-art method that we did not initially compare against, so we report comparison numbers below. These will also be present in the paper revision both in the table and in figure 1.\n\nDAPs exhibits strong performance, and the results in the main table of DAPs are for DAPs-1k (1000 NFEs). To facilitate a fair comparison, we used the official code and ran the configuration for DAPs-50 (50 NFEs) which has a comparable runtime as CDIM with T’ = 25 denoising steps. Below are the results on 100 images from the FFHQ test set reporting LPIPS (net=”vgg”). Lower is better.\n\nRandom inpainting:\\\nDAPs-50 : 0.306 LPIPS\\\nCDIM (T’=25): 0.271 LPIPS\n\nGaussian blur:\\\nDAPs-50: 0.339 LPIPS\\\nCDIM (T’=25): 0.283 LPIPS\n\n4x Super Resolution:\\\nDAPs-50: 0.358\\\nCDIM (T’=25): 0.290\n\nAs you can see, CDIM outperforms DAPs when both are restricted to the same fast sampling schedule (even though DAPs-1k exhibits stronger performance with 10x the runtime as CDIM).\n\nWe also have numerous qualitative comparisons that we will include in the revision and would like to share with the reviewer when possible. The DAPs-50 images are slightly noisy due to the accelerated schedule, and the quality improvement is visibly apparent when using CDIM.\n\nWe can therefore see that when restricted to a fast sampling environment (50 NFEs or 4 seconds), our method outperforms DAPS, a very recent and state-of-the-art method. \n\nWe also thank the reviewer for the other references. We will make sure to include them all in the related work.\n\nRegarding non-linear problems, we can solve them by approximating the stopping criteria $\\mu_t(y)$ and $\\sigma_t(y)$ via a linearized approximation of the non-linear operator A. Although this method works to comparable quality as baselines when using a large number of denoising steps, we lose both the theoretical guarantees of constraint convergence and the high-quality results during accelerated schedules. This is why we didn't want to focus on nonlinear problems, as we wanted to focus on speed and constraint convergence. Nevertheless we will add this discussion into the paper to show how non-linear problems could be tackled with more denoising steps. (The need for more denoising steps when using nonlinear operators has been discussed in prior works, with DAPs using 4000 NFEs for their latent/non-linear results and only 1000 for linear results.)"}, {"review_id": "VN7LEUGk8O", "reviewer": "Reviewer_yBWp", "summary": "This work introduces new techniques for solving linear inverse problems with diffusion model priors. The work extends the sample-then-project approach of diffusion posterior sampling with a multi-step projection approach guided by the distribution of the residual under the forward process. The authors show how to define appropriate stopping criteria for the optimization at each step and employ an efficient line search for speed. In their experiments, the authors show that their approach is competitive with other recent approaches to solving linear inverse problems such as inpainting and deblurring as measured by FID and LPIPS, while being efficient in terms of runtime.", "questions": "I don't fully understand the paragraph from 145-150. Should the equation on line 147 be $\\hat{x}_0$? Could you shed some more light on why is this the best way to define the plausible region?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Stengths:\n\nThe runtime benefits of the proposed approach seem notable and are a significant improvement over some methods.The quality of the solutions are not necessarily state of the art, but appear to be quite good despite the fast sampling time. \n\nThe proposed heuristics for step-size and stopping criteria appear to be reasonably well-motivated, with some caveats. The experiments are using difficult real-world datasets and tasks with a wide variety of baselines\n\nThe background writing is clear and complete.\n\nWeaknesses:\n\nThe novelty is somewhat limited. The key update in equation 12 is the same as DPS and the idea of using multiple projection steps at each sampling time has been discussed in prior work, not cited here. In particular \"Constrained Diffusion with Trust Sampling\" for multiple step. \"Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models\" optimizes a version of equation 11, while combining with other techniques such as multi-step approximation and time-travel. Given the relationship between this method and those, they should be cited and how this method compares to and builds on them should be discussed.\n\nUnlike other extensions of DPS, such as PSLD (\"Solving Linear Inverse Problems Provably via\nPosterior Sampling with Latent Diffusion Models\") this approach doesn't seem to extend to latent diffusion models or general non-linear inverse problems, limiting its usefulness for many state of the art image generation models. The model used for imagenet experiments is somewhat outdated as model for natural images. \n\nMore qualitative examples of the results would be useful, particularly for more challenging tasks such as in-painting. I would like to be more confident the results are not cherry-picked.\n\nSome of the writing for the methods section is somewhat confusing. The notation for $\\hat{x}$ seems inconsistent, for instance in equations 10 & 11, its not clear if it's meant to approximate $E[x_0 | x_t]$ or $E[x_0 | x_{t-1}]$. It's slightly odd that equation 12 is not directly optimizing eq. 11, though this does end up being the same as DPS.", "was_revised": true, "final_justification": "The method is interesting, but the results could be significantly improved.", "rebuttal": "We thank the reviewer for their feedback and comments. We appreciate that the reviewer finds that our method produces good results and that the runtime improvements are notable and significant. We have addressed all the questions and hope the reviewer considers the additional evidence towards a score improvement.\n\nWe appreciate the two additional papers that are closely related. We have added these to the related works with a discussion. Specifically, the below is a comparison between our method and the two that are mentioned:\n\nConstrained Diffusion with Trust Sampling: This paper follows a similar motivation by noting that a single or fixed number of optimization steps at each denoising step is sub-optimal. They take multiple optimization steps based on a trust region that considers the variance of $x\\_t$ under the forward noising process. In contrast, our “trust region” is based on the analytical distribution of $||Ax\\_t - y||^2$ under the forward noising process. Furthermore they use a constant step size while we perform a line search for the optimal step size. As a result, their method requires 1000 NFEs (200 DDIM steps) while ours requires ~50 or ~100 NFEs for high quality results.\n\nTowards Coherent Image Inpainting Using Denoising Diffusion Implicit Models: COPAINT does Bayesian posterior shaping (jointly optimizing revealed and unrevealed regions with approximated likelihoods) whereas CDIM does projection-based constraint enforcement (optimize a simple quadratic residual after each diffusion step). COPAINT is heavier per step but builds coherence gradually; CDIM is simpler and faster, enforcing constraints deterministically at every iteration.\n\n\\\n**“This approach doesn't seem to extend to latent diffusion models or general non-linear inverse problems”**\nWe note that, like DPS, our method can be extended to non-linear problems but without the theoretical guarantees of exact constraint satisfaction and with using significantly more denoising steps. In this paper, we specifically wanted to focus on exact constraint matching and fast sampling, which are only possible for linear problems. We will add a deeper discussion of nonlinear problems of this in the paper and explain how they can be solved with more steps via a linearized approximation of a non-linear operator $A$, which converges on the constraint more slowly.\n\n\\\n**“I don't fully understand the paragraph from 145-150. Should the equation on line 147 be $\\hat{x}_0$”**\nYes! Thanks so much for pointing out this typo. The observed residual on line 147 should indeed by $||A\\\\hat{x}\\_0 - y||^2$. Hopefully this clears up any confusion.\n\n\\\n**“Why is this the best way to define the plausible region”**\nWe draw from several works (cited on line 149) showing that denoising sampling should mirror forward dynamics as closely as possible. Because the forward noising model adds zero mean Gaussian noise, the conditional mean $\\\\mathbb{E}\\_{\\varepsilon_t}\\\\left[ ||A \\\\hat{x}\\_0 - y||^{2} \\\\mid y \\\\right]$  is exactly the most probable (maximum-likelihood) value that this residual can take. Defining the ``plausible'' band as a narrow neighborhood around that mean therefore keeps our reconstructions statistically typical, anything much smaller or larger would be exponentially unlikely under the model and would signal over or under projection.\n\n\\\n**“The notation for $\\\\hat{x}\\_0$ seems inconsistent”**\nWe appreciate the suggestion on the notation of $E[x\\_0 | x\\_t]$, which has been mentioned by other reviewers. We have changed the notation $\\\\hat{x}\\_0$ to $\\\\hat{x}\\_0(x\\_t)$ throughout the paper to clarify this point"}]

Author rebuttal:

Meta-review: All reviewers have positive comments about this work, appreciating that the method offers substantial computational advantages over (strong) baseline methods for inference with pre-trained diffusion models, and is supported by some theoretical analysis of the diffusion process.  In the reviews and discussions, and number of points were raised relative to the clarity/details of the technical development, as well as the experimental results.  Please take care to thoroughly address these points in your final manuscript.
Decision: Accept (poster)