Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Authors: Yang Zheng
Wen Li
Zhaoqiang Liu
Venue: ICML 2025 poster (2025)
Keywords: diffusion model
inverse problems
Primary area: deep_learning->other_representation_learning

Abstract: Inverse problems (IPs) involve reconstructing signals from noisy observations. Recently, diffusion models (DMs) have emerged as a powerful framework for solving IPs, achieving remarkable reconstruction performance. However, existing DM-based methods frequently encounter issues such as heavy computational demands and suboptimal convergence. In this work, building upon the idea of the recent work DMPlug, we propose two novel methods, DMILO and DMILO-PGD, to address these challenges. Our first method, DMILO, employs intermediate layer optimization (ILO) to alleviate the memory burden inherent in DMPlug. Additionally, by introducing sparse deviations, we expand the range of DMs, enabling the exploration of underlying signals that may lie outside the range of the diffusion model. We further propose DMILO-PGD, which integrates ILO with projected gradient descent (PGD), thereby reducing the risk of suboptimal convergence. We provide an intuitive theoretical analysis of our approaches under appropriate conditions and validate their superiority through extensive experiments on diverse image datasets, encompassing both linear and nonlinear IPs. Our results demonstrate significant performance gains over state-of-the-art methods, highlighting the effectiveness of DMILO and DMILO-PGD in addressing common challenges in DM-based IP solvers.

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Yang Zheng'
Wen Li
Zhaoqiang Liu
Abstract
significant importance in wide range of real-world appli-
cations, such as medical imaging (Chung & Ye, 2022; Song et al., 2022), compressed sensing (Wu et al., 2019; Candès & Tao, 2005), and remote sensing (Twomey, 2019). Mathe- matically, the objective an IP is to recover an unknown signal x E R" from observed data y E R". typically modeled as (Foucart & Rauhut, 2013; Saharia al., 2022a):
Inverse problems (IPs) involve reconstructing sig- nals from noisy observations. Recently, diffu- sion models (DMs) have emerged powerful framework for solving IPs, achieving remarkable reconstruction performance. However, existing DM-based methods frequently encounter issues such as heavy computational demands and subop- timal convergence. In this work, building upon the idea of the recent work DMPlug (Wang et al., 2024), we propose two novel methods, DMILO and DMILO-PGD, to address these challenges. Our first method, DMILO, employs intermediate layer optimization (ILO) to alleviate the mem- ory burden inherent in DMPlug. Additionally, by introducing sparse deviations, we expand the range of DMs, enabling the exploration of under- lying signals that may lie outside the range of the diffusion model. We further propose DMILO- PGD, which integrates ILO with projected gra- dient descent (PGD), thereby reducing the risk of suboptimal convergence. We provide an intu- itive theoretical analysis of our approaches under appropriate conditions and validate their superi- ority through extensive experiments on diverse image datasets, encompassing both linear and nonlinear IPs. Our results demonstrate signifi- cant performance gains over state-of sate-of-the--a the-art art meth- ods, highlighting the effectiveness of DMILO and DMILO-PGID in addressing common challenges in DM-based IP solvers.
y = A(xx + +
(1)
where A(-) denotes the given forward operator (e.g., a linear transformation, convolution, or subsampling), and e E R" represents stochastic noise. A key challenge in solving IPs arises from their inherent ill ill-posednees: When m < n, even in the absence of noise, x* cannot be uniquely determined from y and A(-). This underdetermined nature necessitates the incorporation of prior knowledge regularization to ensure stable and reliable solutions.
Traditional approaches for solving IPs often rely on hand- crafted priors based on domain knowledge, such as sparsity in specific domains (c.g., Fourier or wavelet) (Tibshirani, 1996; Bickel et al., 2009). While effective in certain scenar- ios, these methods are limited in their ability to capture the rich and complex structures of natural signals and prone to suboptimal performance in highly ill ill-posed problems.
The advent of generative models has introduced new paradigms for addressing IPs by leveraging learned priors from large datasets. One class of generative model-based methods involves training models specifically for each IP using paired data (Aggarwal et al., 2018; Mousavi al., 2015; Yeh al., 2017). While these approaches can achieve high-quality reconstructions, their generalizability and flexi- bility are constrained, as they are tailored to specific tasks and often require extensive retraining for new problems. In contrast, another promising direction involves the utilization of pretrained generative models to solve IPs without addi- tional training (Bora et al., 2017: Shah & Hegde, 2018: Liu & Scarlett, 2020b). These methods assume that the underly- ing signals lie within the range of a variational autoencoder (VAE) (Kingma, 2014) 4) or generative adversarial network (GAN) (Goodfellow et al., 2014; Karras et al., 2018).
1. Introduction
Inverse problems (IPs) represent a broad class of challenges focused on reconstructing degraded signals, particularly images, from noisy observations. These problems are of
'University of Electronic Science and Technology of China. Correspondence to: Zhaoqiang Liu <zqliu12@ @eaaiemmmaaa..........
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Compared to these conventional generative models like VAEs and GANs, diffusion models (DMs) (Sohl-Dickstein

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
al., 2015; Ho et al., 2020; Song & Ermon, 2019; Dhariwal & Nichol, 2021) have recently emerged as a powerful frame- work for modeling high-dimensional data distributions with remarkable fidelity. These models have demonstrated signif- icant potential in addressing a variety of IPs, often achieving state-of-the-art (SOTA) performance, Motivated by the ef- fectiveness of DMs in solving IPs, this work explores DM based IP solvers, aiming to further advance the field by leveraging the properties and strengths of DMs (Wang et al., 2023; Whang et al., 2022; Saharia et al., 2022b;a; Alkan et al., 2023; Cardoso et al., 2023; Chung et al., 2023a; Feng & Bouman, 2023; Rout et al., 2023; Wu et al., 2023; Aali et al., 2025; Chung et al., 2024; Dou & Song, 2024; Song et al., 2024; Wu et al., 2024; Sun et al., 2024).
2022a;b; Zhao et al., 2024). These frameworks have been widely applied to IPs, achieving remarkable reconstruction performance.
DMs can be specifically trained for particular tasks, such as super-resolution (Gao et al., 2023; Shang et al., 2024), inpainting (Lugmayr et al., 2022), and deblurring (Sanghvi et al., 2025), but their generalization other tasks remains limited. In this paper, we focus on using pre-trained uncon- ditional DMs and propose general framework applicable to range of IPs. The key challenge is how to apply the knowledge of the prior distribution corresponding to the pre-trained DM during the sampling process.
Based on how algorithms reconstruct the underlying sig- nal x with prior distribution p(x:), works with pre-trained DMs can be further classified. Some methods apply Bayes' rule (Fabian et al., 2023; Fei et al., 2023; Zhang et al., 2023) to decompose the conditional score V. log p(zly)) into a tractable unconditional score V p(z) provided by the pretrained DMs and an intractable measurement- matching term, performing iterative corrections. For ex- ample, DDRM (Kawar et al., 2022) uses singular value decomposition (SVD) to approximate the measurement- matching term in spectral space. MCG (Chung et al., 2022) adapts Tweedie's ' formula in linear IPs to approximate the reconstructed signal, followed by a gradient- based poste- rior correction. It also performs projection with a manifold constraint to ensure that corrections remain on the data man- ifold. DPS (Chung et al., 2023b) discards the projection step in the reverse process of MCG to prevent the samples from falling off the manifold in noisy tasks and generalizes to both linear and nonlinear tasks. IIGDM (Song et al., 2023) employs pseudo-invers guidance to encourage data con- sistency between denoising results and the degraded image. Although these methods achieve high performance in many tasks, they are sometimes sensitive to guidance strength and may fail in simple scenarios, generating unnatural images or incorrect reconstructions.
1.1. Related Work
Inverse problems with conventional generative models: The central idea behind this line of work is to replace sparse priors with generative priors, specifically those from VAEs or GANs. The seminal work (Bora et al., 2017) proposes the CSGM method and demonstrates the use of pre-trained generative priors for solving compressed sensing tasks. The CSGM method aims to minimize |/y/ - A(xx)ll over the range of the generative model g()), and it has since been extended to various IP through numerous experiments (Oy- mak et al., 2017; Asim et al., 2020a;b; Liu et al., 2021; Jalal et al., 2021: Liu et al., 2022a;b; Chen et al., 2023b; Liu et al., 2024).
However, there are limitations to these methods. First, the underlying signal x" often lies outside the range of the gen- erative model, which can result in suboptimal reconstruction performance. Second, these approaches heavily depend on the choice of the initial vector, leading to potential conver- gence to local minima.
To address these issues, several methods employ projected gradient descent (PGD) (Shah & Hegde, 2018; 8; Hyder et al., 2019; Peng et al., 2020; Liu & Han, 2022), using iterative projections to mitigate the risk of local minima. Some ap- proaches allow for sparse deviations (Dhar et al., 2018) from the range of the generative model to capture signals outside the range. Additionally, the works (Daras al., 2021; 2022) further perform optimization in intermediate layers to better align the reconstruction with the measurements.
Another class of methods treats DMs as black-box gener- ative models and focuses on finding the initial latent vec- tor (Wang et al., 2024; Xu et al., 2024). In particular, DM- Plug (Wang et al., 2024) directly optimizes the initial latent vector and demonstrates robust performance across various tasks, with the advantage of not requiring specially designed guidance strengths. However, unlike conventional genera- tive methods that map from the latent to the image space with a single number of function evaluations (NFE), diffu- sion models require multiple NFEs for the reverse process. This creates a significant memory burden for DMPlug, as larger computational graphs are needed to store information throughout the entire process. Additionally, similar the CSGM method, if not initialized appropriately, DMPlug may converge to suboptimal points.
Inverse problems with diffusson models: DMs have un- locked new possibilities in variety of applications due to their ability to generate high-quality samples and model complex data distributions. Since the seminal works (Sohl- Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019), many studies have focused on improving the efficiency and quality of DMs, with efforts aimed at faster sampling speeds and higher output fidelity (Song et al., 2021a; Lu et al.,
2

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
1.2. Contributions
and S1 -S {8] 82 =8 S1, 82 S2). We define the E--ball (r) = (xr R ||a||| S sr..
In this paper, we focus on resolving the memory burden and suboptimal convergence problems prevalent in DM-based CSGM-type methods, such as DMPlug. Given that the mul- tiple sampling steps of DMs correspond to the composition of multiple functions, we initiate by applying Intermediate Layer Optimization (ILO) to relieve the memory burden. Moreover, inspired by the works in (Dhar et al., 2018; Daras al., 2021), we introduce sparse deviations to broaden the range of DMs. This expansion allows for the exploration of underlying signals beyond the original range of the DM. Additionally, we incorporate the Projected Gradient Descent (PGD) method, iteratively updating the predicted signals to circumvent suboptimal solutions that may arise due to the selection of the initial point. The main contributions of this study are as follows:
2. Preliminaries
A diffusion model (DM) comprises a forward process that progressively transforms real data into noise, and a reverse process that aims invert this transformation and recover the original data distribution. The forward process of DMs can be described by the following the stochastic differential equation (SDE):
dæ = f(t) at + (o(tddur 3o ~ po-
(2)
where [(() and g(() are the drift and diffusion coefficients, respectively, and we is a standard Wiener process. For each & E [o, Tj, the distribution of æt conditioned on 2o is Gaussian with 22 ~ (a) of I), where a and o are positive, differentiable functions with bounded deriva- tives, and their ratio ai /o (i.e., the signal-to-noise ratio) is strictly decreasing in t. The marginal distribution of æ, is denoted by Pt. and Por denotes the distribution of æ; con- ditioned on Zo. These functions and o are selected so that pr approximates a zero-mean Gaussian with covariance Iº I for some > 0.
We propose two novel methods, referred to as DMILO and DMILO-PGD. These methods integrate ILO with and without PGD respectively. They effectively (i) re- duce the memory burden and (ii) mitigate the suboptimal convergence issues associated with DM-based CSGM- type methods.
Under appropriate conditions, we offer an intuitive theo- retical analysis the proposed approach. This analysis demonstrates its efficacy solving IPs.
To guarantee that the SDE in (2) indeed induces the tran- sition distribution poz. the drift and diffusion coefficients must satisfy:
Through comprehensive numerical experiments con- ducted diverse image datasets, we verify the supe- rior performance of our method compared to SOTA tech- niques across various linear and nonlinear measurement scenarios.
f(t) = log an
do? ge(t) = dt - 2 dt
d log at of. (3)
dt
Reorusmetion
It is known from (Anderson et al., 1958) that the forward SDE in (2) has corresponding reverse-time diffusion pro- cess evolving from down to 0. This reverse SDE can be written as (Ho et al., 2020; Song et al., 2021b):
dx: = [f(t) x. g²(() Va log px:)) + g(t) dw,,
Intemnoiiti Layar Optiniztionn
(4)
where w. a Wiener process in reverse time, and Va (x;,) is the score function of p:.
Natre Gradient Updane
As shown by (Song al., 2021b), one may also construct deterministic process whose trajectories share the same marginal densities Pr( as the reverse SDE. This process is governed by the ordinary differential equation (ODE):
Figure 1: Illustration of our algorithm. Starting from zero vector, we first perform a naive gradient descent update step (blue arrow) and then projection step through intermediate layer optimization (red arrow) alternatively to reach the final estimate.
dx; = [f(t) x - gg) V lo:) (5)
Hence, sampling can be performed via numerical discretiza- tion or integration of this ODE.
1.3. Notation
We use upper-case boldface letters to denote matrices and lower-case boldface letters to denote vectors. For S₁ C R.
In practice the score function V (x(;)) is approxi- mated by a neural network. A common approach is to train noise prediction network en (a,,, t) to estimate the scaled
and S₂ C R.. S + S2 = {8) + 82 81 E S1, 82 E Sz)
3

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
score of V (xt)) The parameter set 0 is optimized by minimizing the following objective:
we adapt the PGD framework to our setting, thereby allevi- ating potential suboptimal solutions. In doing so, also highlight illustrative example that explains why using the forward model during projection yields more robust results compared to existing approaches.
L Pe- (on [Ileo(ar - oee,t) - <llidrr.
(6)
3.1. DMILO
By substituting V. log (x;)) = Eo(2,,,)) into (5) and exploiting its semi-linear structure, one obtains the nu- merical integration formula (Lu et al., 2022a):
ILO was originally introduced to improve the ability of conventional deep generative models to conform to given measurements (Daras et al., 2021). The key idea is to split the generative model into intermediate layers and iteratively optimize the latent representations at each layer. Specifi- cally, for conventional generative model such GAN, even if it only requires single sampling step map noise to image, ILO decomposes it into four intermediate lay- ers and optimizes the representation at each intermediate layer. However, this decomposition is often architecture- dependent and empirically determined, making it hard to generalize across diverse models.
f( eli ( co(zT, dT. 2
æt = cli (())dd xx +
(7)
An alternative strategy is to train a data prediction network (x( t) such that EØ t) (a+ a₁ (, f). Then, by substituting (3) into (7) and using the relation between 6o and æo, one obtains the following integration formula (Lu et al., 2022b):
Tt xx + af /'m (xx A) dA, (8) 0s
In contrast, when viewing the entire sampling process of a DM as generative function (-)), it naturally decomposed into composition of simpler functions:
Zt =
where A == log(ar/oz) is strictly decreasing in t and ad- mits an inverse function (-), and thus the data prediction network xa (A), (A)) can be rewritten as ie (æx. A).
g(-) = 9l 9 92
gN((),
(11)
where N denotes the total number of sampling steps, and (.).) represents the function corresponding to the i-th sam- pling step (see, e.g., (10)). Such a composition is indepen- dent of the architecture of the denoising network or sampling scheme, making it straightforward to be plugged into any DM. In this paper, we follow the DDIM setting and use Eq. (10) as the sampling function.
In the sampling process, DMs iteratively denoise an initial Gaussian noise vector to produce an image sample. Con- cretely, let [e, T' be partitioned by to t < .... < tN-1 tN T, where > is small to avoid numerical instabilities (Lu et al., 2022a). Then, by applying first- order approximation to (8), the transition from t; to 4 can be written as:
Based on this composition, our method operates iteratively, In each iteration, we first optimize the input é;, to the final function gi (-). Here, we use a technique inspired by (Dhar et al., 20,8), and introduce sparse vector ii, to search for the underlying signal outside the range of the generator:
Ou: 1 Wr. + Ohi1
ai4 1
Ott;
xo aT 4))
Obe
01.-1 oe:
(9)
This update rule matches the widely used DDIM sampling method (Song et al., 2021a). Note that the i-th sampling step corresponds to the following function:
it, in arg min [|A()t.)) A(gi (z) + v) + Al|Milaa 3,V
(12)
(xxx)
The + 0t,-1
Ott
12 ti),
where is the Lagrange multiplier, A(-) denotes the forward operator, and t is the estimated signal. Since the estimated signal is unavailable and often noisy, we replace A(#:,)
Tt
0t of,
(10)
and the entire sampling process corresponds to the composi- tion of 9; for i = N. N 1,... 1,...,1.
with the observed vector y in practice. As the minimization problem in Eq.(1) is highly non-convex and obtaining its globally optimal solutions is not feasible, we approximately solve this optimization problem using the Adam optimizer and apply fzregularizzatin to Øt, to avoid overfitting.
1.
3. Methods
In this section, we introduce proposed methods, DMILO and DMILO-PGD, two novel approaches for solving IPs using pretrained DMs. In Section 3.1, we present the con- cept of ILO and show how it can be naturally integrated into DMs to address the high memory usage found in recent DM- based CSGM methods such as DMPlug. Section 3.2,
We then proceed iteratively for the remaining layers (or sampling steps). Specifically, for each subsequent layer, we perform the optimization as follows:
L Pr. arg min |(+ (9,(x) + +X|p|l (13)
2.
4

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Unlike other DM-based CSGM-type approaches that must retain the entire gradient graph throughout the sampling process, our method requires only the gradient information of single sampling step at time. This offers a substantial reduction in memory usage and can be seamlessly applied to different sampling strategies. Additionally,, incorporating the sparse deviation term extends the range the generator and can yield better reconstruction quality in practice. For convenience, the iterative procedure for DMILO is presented in Algorithm 1.
Table I: Memory usage (in GB) for DMPlug, DMILO, and DMILO-PGD across different sampling steps. The model size is 2.75 GB for the super-resolution task on the LSUUN Bedroom dataset (Yu et al., 2015). "N/A" indicates that the method exceeds available memory.
Sampling Steps
DMPlug
DMILO DMILO-PGD
10.53
10.53
10.53
2
15.72
10.53
10.54
3
20.83
10.53
10.54
N/A
10.54
10.54
Algorithm DMILO
1: Input: Diffusion denoising model ((.) = 91 o ....
9N-1 o (.), total number of sampling steps N, noisy observation y. forward model A(-), iteration steps J,
without participant of forward operator A(.), we minimize [[A(G((r),)) - A(( ( where Ø de- notes the result from the gradient descent update (see Step in Algorithm 2). The efficacy of leveraging the forward operator guide the projection has been illustrated in Fig- ure 2, and the corresponding theoretical evidence has been provided in Theorem 4.4.
[x( - -t
Lagrange multiplier
2: Initialize (0) ~ _N(0, I) and (1) = 0
3: for to 2 do
4: [1)) (x( V
(0)
5: end for
6: for 1 do
7: Approximately solve (a! ( arg (in) lly (z) 1i { Alle/l1 using
Iepainting Openation
Projection Function
the Adam optimizer, initialized at (x, (x (1-1) 8: for i = 2 to N do Approximately arg min ||= - (a)(x) + v)ll + A||v/|a
.1 ")
iDD
solve (x) , (1
(1)
9:
using the Adam optimizer, initialized at (a/))
(-11 1
(1-1))
Orgin Image
Predicted Image = Az"
With Mosermmment (Right Projection)
10:
end for
11: for = N to do
12: 2t, (1)
+ V/ 91 [z), (1))
(
13: end for
14: end for
(J) 15: Return a = zta
Pordiceed Xi,
Predicte:
Messueenent (Wiong Projectio))
We also compare the memory usage of our methods with DMPlug using 2.75 GB diffusion model on an NVIDIA RTX 4090. As shown Table 1, our approaches signif- icantly reduce memory consumption and maintain a con- sistent memory footprint as the number of sampling steps increase, underscoring both their practical efficiency in real- world settings and their extensibility to different sampling methods and diffusion models.
Figure 2: Illustration of two projection strategies in DMILO-PGD. The top-right is the reconstructed image using method that leverages the forward operator to guide the projection (green arrow). In contrast, the bottom- right demonstrates how a purely "distance- projection (red arrow) can yield a problematic reconstruction.
4. Theoretical Analysis
3.2. DMILO-PGD
In this section, we present an intuitive theoretical analysis of the effectiveness of the proposed DMILO-PGD method. As observed from the experimental results, this method performs reasonably well for various instances of IPs. For the sake of simplicity, we build upon the results in (Daras et al., 2021) and consider only the case where the generator G R - R the composition of two functions with
We now describe how to adapt PGD to further address sub- optimal solutions. PGD alternates between a gradient de- scent step and a projection onto the range of the genera- tor (Shah& & Hegde, 2018). In our approach, we replace the projection step with DMILO, as outlined in Algorithm 2. Difference from conventional PGD which sorely minimize
5

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Algorithm 2 DMILL-PPGD
much smaller than n) such that
1: Input: Diffusion denoising model ((.) = 91 o .... 9N-1 o 9Nv (.), total number of sampling steps N, noisy observation y, forward model A(-), iteration steps E, learning rate 7), Lagrange multiplier X
log (22 S Cg. log
(6))
(15)
where Cg is a positive constant that depends on 92.
2: Initialize to
(0) = 0,
x0 ZiN ~N(0,, and ViN = 0
Next, we introduce the Set- Set-Restricted Eigenvalue Condition (S-REC), which has been extensively explored in CSGM- type methods (Bora et al., 2017).
3: for N to 2 do
4: ( (z( v V
(0)
5: end for
6: for 1 to E do
Definition 4.3. Let S C R". For some parameters 7 E (0, 1), 2 0, matrix A Rmxa is said satisfy the S-REC(S, ,,,C(,) if for all $1, 82 E E.S.
7: Tio
(e)
toa (e-1)
V y
A(x )11
(
8: Approximately
solve
(x
(e)
M/1 (e))
arg (..) ||A(
A(gi (2)) + A||n||
||A(s) 82)ll2 2 - 82l12 -
(16)
using the Adam optimizer,
initialized at
(0t, (e-1) ()-1))
M/
By proving that the Gaussian measurement matrix satisfies the S-REC over the set 91 (22 + (r)) for some r >> 0, we adapt the result from (Daras et al., 2021, Theorem 1) to derive the following theorem.
9: for = to N do
Approximately solve (wl.)) (())) (
10:
arg min using the Adam optimizer, initialized at 1/4
||
(x) + w)ll + A||l/laa
Theorem 4.4. Suppose that Assumptions 4.1 and 4.2 hold. Let A E R have i.i.d. NO 1/m) entries? For fixed 7 E (0, 1), O, and k (0, vn), let Li Consider the true optimum within the extended range:
(ar (x((-1)) (c-1))
11: end for
ká
12: for N do
gi (.) + V/
13: x,
14: end for
T == arg
min 21 Ex2 BP)
||" (21)),,,
(17)
15:
end for (E)
16:
Return a = Pio
and the measarement optimum within the extended range:
t1 = arg
min
||x=*
(21(21)
(18)
= 91 92, and we restrict our attention the linear task, where the measurement model in (I) simplifies to:
a Ex
Then, if the number of measurements m is sufficiently large with m = (kg log Lnn" + k² log(3n)), we have with prob- ability 1 -00m) that
y Ax +
(14)
Here, A E Rmxx represents the linear measurement matrix.
1-= 121 S (1
6 3) -++2) (2)1(()) -110)1100))))) x= x0112+
6. (19)
1l9n (.)) -
In theoretical studies of DMs, it is standard to assume that the denoising network zo (z, f) is Lipschitz continuous with respect to its first argument (Chen et al., 2022; 2023a; Li et al., 2024). Then, given that 91 corresponds to sam- pling step in DMs, it is reasonable to make the following assumption:
Similar Theorem I in (Daras et al., 2021), our Theorem 4.4 essentially states that if the number of measurements m is sufficiently large, the measurement optimum é is nearly as good as the true optimum T1. It is important to note
Assumption 4.1. The function 91 is L1 Lipschiit continu- ous for some L > 0.
FFor X' BR" and e-net of X' if, for all x E X, there exists some 8 E S such that ||s ls=aat ælIa S 6. The minimal cardinality it is finite) is denoted by N2') and is called the covering number of X' (with parameter e).
0, subset S C X' is said be an
e-net of X' (assuming
Furthermore, following the setting in (Li & Yan, 2024), and inspired by the common characteristic of natural image distributions that the underlying target distribution is con- centrated on or near low-dimensional manifolds within the high-din high-dimennsonal ambient space, we make the following dwweiieiniionlttty assumption regarding the range of g2-
"As noted in (Daras al., 2021), analogous results hold when A is partial circulant matrix, a case that is highly relevant to the blind image deblurring problem. In the context of partial circulant matrices, the number of measurements needed to establish the S- REC is of nearly the same order (neglecting additional logarithmic factors).
Assumption 4.2. Let X₂ be the range of 92. i.e., X₂ = 92 (. For any 8 > 0, we define the intrinsic dimension of X's as a positive quantity l2 (typically a positive integer
NNote that the optimization problem in (18) can be regarded as variant of (12), and the optimization problem in (17) can be regarded as variant of (13).
6

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
that reconstruction algorithm can only access the measure- ment error and can never compute ø1. Therefore, Theorem 4.4 also provides theoretical support for performing (12) instead of (13) in DMILO-PGD. The proof of Theorem 4.4 is deferred to Appendix A.
Mensroemen
Refeceace
EPS
5. Experiments
In this section, mainly adhere to the settings in DM- Plug (Wang et al., 2024) to assess our DMILO and DMILO- PGD methods. We compare our method with several recent baseline approaches on four linear inverse problems (IP) tasks, namely super-resolution, inpainting, and two linear image deblurring tasks involving Gaussian deblurring and motion deblurring. Furthermore, we evaluate on two non- linear IP tasks, including nonlinear deblurring and blind image deblurring (BID). To measure recovery quality, we follow (Blau& & Michaeli, 2018) and use three standard met- rics: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), which evaluate distortion, and Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) with the default backbone, which measures perceptual quality. Additionally, we compute the Fréchet Inception Distance (FID) on the linear image deblurring task to better illustrate the perceptual quality of the gener- ated images. We use bold to indicate the best performance, underline for the second-best, green to denote performance improvement, and red to signify performance decline.
IMPlag
DMILL (Oum)
ONILLOPGG (VERS)
Figure 3:
Visualization of sample results from our methods for inpainting with additive Gaussian noise (a = 0.01).
the outer learning rate ry is set to 0.5. For super-resolution, inner learning rate of 0.02 is also used, along with 400 inner iterations and 10 outer iterations. For DMILO-PGD in super-resolution. the outer learning rate 1) is set to 8. The quantitative results are presented in Table 2. Evidently, our DMILO-PGD approach surpasses all competing methods in most metrics. This clearly demonstrates its superior re- construction performance, highlighting its effectiveness and competitiveness in these two linear IP tasks.
5.2. Linear Image Deblurring
For linear image deblurring, we choose Gaussian deblur- ring and motion deblurring two linear image deblurring tasks. To validate the performance of different methods, we randomly sample 100 images from CelebA, FFHQ, and ImageNet (Deng et al., 2009). All images are resized to 256 x 256 x 3 and the noise level o is set to 0.01 sim- ilarly. We compare our methods with DPS, RED-diff, DPIR (Zhang et al., 2021) and DifiPIR, DMPlug. Note that we follow DMPlug to set the kernel size to 64 x 64, which differs from the kernel size initially employed in DPIR and DifPIR. Such a difference might impact the re- sults. The results presented in Tables 3 and 4 indicates that our methods generally perform well in terms of both real- ism and reconstruction metrics, demonstrating its ability to effectively balance perceptual quality and distortion. They achieve top performance across all metrics for motion de- blurring and achieve either the best competitive results on Gaussian deblurring.
5.1. Inpainting and Super-resolution
For inpainting and super-resolution two linear tasks, we create evaluation sets by sampling 100 images each from CelebA (Liu et al., 2015), FFHQ (Karras al., 2019), and LSUN-bedroom (Yu et al., 2015).4 4 All images are resized to 256 256 x 3 pixels. For super-resolution, we generate measurements by applying 4x bicubic downsampling, and for inpainting, we use random mask with 70% missing pixels. Consistent with the setting in DMPlug (Wang et al., 2024), all measurements are corrupted by additive zero mean Gaussian noise with a standard deviation of a = 0.01.
We compare our methods with DDRM (Kawar et al., 2022).
DPS (Chung et al., 2023b), IIDMM (Song al., 2023).
RED-diff (Song et al., 2023), DAPS (Zhang et al., 2025),
DiffPIR (Zhu et al., 2023) and DMPlug (Wang et al., 2024).
In the inpainting and super-resolution tasks, both DMILO and DMILO-PGD are configured with a Lagrange multiplier = 0.1. For inpainting, during the optimization process, both DMILO and DMILO-PGD employ the Adam optimizer with an inner learning rate of 0.02. They run for 200 inner it- erations and outer iterations. In the case of DMILO-PGD,
5.3. Nonlinear Deblurring
For nonlinear deblurring, use the learned blurring opera- tors from (Tran et al., 2021) with a known Gaussian-shaped kernel and additive zero-mean Gaussian noise with a stan- dard deviation of a = 0.01, following the setup of DMPlug. Similar to the process for linear tasks, our evaluation sets are created by sampling 100 images from CelebA, FFHQ.
"The experimental results for LSUN-hedroom are presented in Appendix C due to the page limit.
7

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Table 2: (Linear IPs) Super-resolution and inpainting with additive Gaussian noise (a = 0.01).
Super-resolution (4x)
Inpainting (Random 70%)
CelebA (256 x 256)
FFHQ (256 x 256)
CelebA (256 x 256)
FFHQ (256 x 256)
LPIPS!
PSNR1
SSIMT
LPIPS
PSNRT
SSIM1
LPIPS
PSNRT
SSIMT
LPIPS)
PSNR1
SSIM
DDRM
0.103
31.84
0.879
0.862
0.142
28.24
0.822
0.808
DPS
0.143
27.15
0.753
0.163
25.93
0.720
0.073
33.07
0.897
0.089
31,59
0.879
HGGMM
0.112
32.45
0.888
0.104
32.29
0.882
0.864
RED-diff
0.105
32.48
0.888
0.127
31,09
0.865
0.238
28.19
0.800
0.224
27.38
0.789
DAPS
0.111
29:98
0.814
0.149
28.67
0.803
0.051
32:40
0.886
30.95
0.879
DifIPIR
0.107
29.48
0.796
0.121
27.95
0.778
0.098
31,20
0.867
0.100
29.59
0.854
DMPlug
0.127
32.38
0.875
0.124
31.22
0.866
0.066
35.51
0.935
0.068
34.16
0.927
DMILO
0.133
30.81
0.785
0.129
30.26
0.789
0.025
36.07
0.951
0.029
34.23
0.937
DMILO-PGD
0.056
33.58
0,906
0.117
31.38
0.868
0,023
36.42
0,952
0.026
34.62
0,940
Ours vs. Best comp.
0.047
+1.10
+0.018
+0.038
+-006
-0.008
0..028
+0.91
+0.017
-0.020
+0046
-0.013
Table 3: (Linear IPs) Gaussian deblurring with additive Gaussian noise (a = 0.01).
CelebA
FFHQ
ImageNet
FID
LPIPS|
PSNRT
SSIMI
FID
LPIPS
PSNRT
SSIM
FID
LPIPS|
PSNR1
SSIM
DPS
0.309
27.65
0.752
91.45
0.150
25.56
0.717
147.99
23.30
RED-dilf
85.14
0.221
29.59
0.808
111.52
0.272
27.15
0.778
166.04
0.497
25.07
0.666
DPIR
92.05
0.256
31.30
0.861
134.07
29.06
26.67
0.741
DiPPIR
50.06
0.092
28.91
0.791
86.40
0.119
26.88
0.769
145.58
0.422
24.59
0,581
DMPlug
77.06
U.172
29.70
95.46
28.27
0.806
128.74
0.324
25.20
0.662
DMILO
54.42
0.092
30.89
0.816
72.34
0.110
29.60
0.852
142.13
0.316
24.81
0.702
AMILO--PG
80.69
0.157
30.74
0.811
111.17
0.176
28.65
0.799
164.42
0.411
26.03
0.706
Ours vs. Best comp.
+4.36
-0.000
-0.41
-0.045
-1406
-0.009
+0.54
0.008
-13.39
-0.008
-0.64
-0.035
and LSUN. Each image is resized to 256 x 256 x 3 pixels. As DDRM is only designed for linear IPs, we compare our DMILO and DMILO-PGD methods against DPS, IIGDM, RED-diff, and DMPlug.
deblurring.
5.4. Blind Image Deblurring
The BID problem aims to recover an underlying image x² and an unknown blur kernel k" from noisy observations y k² x x + e where "*" denotes convolution, k" is a spatially invariant kernel, and is the noise vector. In our experiments, we consider two types of blur kernels, namely motion kernels and Gaussian kernels, which are the same types used in the linear image deblurring task. To form the evaluation sets, we sample 100 images each from CelebA and FFHQ We compare our DMILO and DMILO-PGD methods against three recently proposed methods for blind IPs: ILVR (Choi et al., 2021), Blind-DPS (Chung et al., 2023a), and DMPlug.
Measnement
Reformex
CPS
D84P1RR
DMILO
DMILO. PCD
For both DMILO and DMILO-PGD, the Lagrange multi- plier X is set to 0.1. In BID tasks, DMILO employs the Adam optimizer with an inner learning rate of 0.01, 200 inner iterations, and 10 outer iterations. DMILO- -PGD uses Adam with an inner learning rate of 0.02, an outer learning rate = 2, 500 inner iterations, and 5 outer iterations. The experimental results are presented Tables and 7. These results clearly demonstrate that DMILO outperforms other methods across all tasks and metrics, underscoring its supe- rior performance in the BID task. However, DMILO-PGD
Figure 4: Visualization of sample results for nonlinear de- blurring with additive Gaussian noise (a = 0.01).
For the nonlinear deblurring task, both DMILO and DMILO- PGD use x= = 0.1, the Adam optimizer with inner leam- ing rate of 0.02, 200 inner iterations, and 5 outer iterations. DMILO-PGD further adopts an outer learning rate 0.3. As shown in Table 5, both methods outperform all com- petitors across all metrics on both datasets, demonstrating the superiority of DMILO and DMILO-PGD for nonlinear
8

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Table 4: (Linear IPs) Motion deblurring with additive Gaussian noise (a 0.01).
CelebA
FFHQ
ImageNet
FID
LPIPS)
PSNR
SSIM
FIDL
LPIPS
PSNR1
SSIM
FIDJ
LPIPS,
PSNR1
SSIMI
DPS
67.21
0.126
26.62
103.39
0.167
0.676
167.71
0.327
22.90
0.590
RED-diff
111.34
0.229
27:32
0.758
130.81
0.272
25.40
0.730
247:22
0.494
22.51
0,587
DPIR
120.62
0.192
31.09
0.826
116.72
0.181
29.67
0.820
134.73
0.289
26.98
0.758
DIIPIR
78.03
0.117
28.35
0.773
97.69
0.137
26.41
0.740
115.74
0.282
24.79
0.608
DMPlug
78:57
0.164
30.25
0.824
93.66
0.173
28.58
0.812
99.87
0.285
25.49
0.696
DMILO
31.08
0.044
34.15
0.908
41.48
0.044
33.21
0.909
53.77
0.098
29.67
0.841
DMILO-PGD
36.75
0.067
33.41
0.884
49.57
0.079
31,66
0.857
85.51
0.183
27.60
0.755
Ours vs. Best comp.
-36.13
-0.073
+3.06
+0.082
-52.18
00.933
+3.54
+0.089
-6.10
0.184
+2.69
+0.083
Table 5: (Nonlinear IP) Nonlinear deblurring with addi- tive Gaussian noise (a 0.01).
Table 6: (Nonlinear IP) BID (Gaussian Kernel) with addi- tive Gaussian noise (a 0.01).
CelebA (256
256)
FFHQ
256)
CelebA (256 x 256)
FFHQ (256 256)
LPIPS
PSNRT
SSIM
LPIPS)
PSNRT
SSIMT
LPIPSI
PSNR
SSIM
LPIPS
PSNR
SSIMI
DPS
0.221
24.28
0.668
0.225
24.00
0.664
0.287
19.03
0.503
0.258
20.18
0.541
HGGMM
0.099
28.51
0.859
0.125
27.27
0.843
Blind-DPS
0.146
24.86
0.715
0.176
28.60
0.804
RED-diff
0.224
29.89
0.796
0.235
28.64
0.773
DMPlug
0.146
29.58
0.790
0.161
28.93
0.825
DMPlug
0.095
30.32
0.851
31.37
0.866
DMILO
0,109
31.20
0.857
0.156
29.65
0.854
DMILO
0.076
32.74
0.897
0.066
33.05
0.915
DMILO-PGD
0.428
21.20
0,617
0.394
24.11
0.712
DMILO-PGD
0.058
32.87
0.897
0.047
34.02
0.919
Ours vs. Best comp.
-0.037
+1.62
+0.067
-0.005
+0.72
+0.029
Ours vs. Best comp.
0.037
+2255
4-0.038
0.055
4-2.65
+-0.553
Table 7: (Nonlinear IP) BID (Motion Kernel) with additive
Gaussian noise (a = 0.01).
CelebA (256 x 256)
FFHQ (256 x 256)
LPIPS
PSNRT
SSIM1
LPIPS
PSNRT
SSIMI
ILVR
0,328
17.47
0,442
0.324
17.99
0.424
Blind-DPS
0.119
26.47
0.772
0.146
24.86
0.715
DMPlug
0.123
28.92
0.801
0.148
27.63
0.778
DMILO
0.083
29.84
0.845
0.078
30.97
0.888
DMILO-PGD
0.391
19.71
0.546
0.414
19.50
0.544
Ours Best comp.
-0.036
+0.92
+0.044
-0.068
+3.34
+0.110
DMILO (ours)
DMILO PGD (sers)
over existing approaches.
Figure 5: Visualization of sample results from our methods for BID with a motion kernel with additive Gaussian noise (a 0.01).
Impact Statement
Our research on inverse problems in machine learning us- ing diffusion models makes significant contributions. The proposed DMILO and DMILO-PGD methods resolve the memory and convergence issues of DM-based solvers like DMPlug. This advancement enhances the performance of diffusion models in inverse problem applications, with broad practical use in medical imaging, compressed sensing, and remote sensing. Methodologically. it enriches the machine learning toolbox, offering theoretical insights and inspiring future algorithm development. Importantly, this work raises no ethical concerns. It aims to improve the efficiency and accuracy of data reconstruction without causing harm to individuals or society.
shows relatively less effectiveness. A possible reason for this is that the naive gradient update mechanism in DMILO- PGD might not be well-suited for updating the underlying blur kernel. The detailed algorithms for BID are provided in Appendix B for further reference.
6. Conclusion
In this paper, we presented two approaches, DMILO and DMILO-PGD, that address the memory burden and subopti- mal convergence issues in DMPlug. Our theoretical analysis and numerical results illustrate the effectiveness of these proposed methods, demonstrating significant improvements

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Acknowledgment
Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. R. Sampling is easy as learning the score: The- ory for diffusion models with minimal data assumptions.
This work was supported by National Natural Science Foun- dation of China 1, No. No.62176047) and Sichuan Natural Science Foundation (No.202ANSFTD0044),
http:://ardii.org/bb//2222 11219, 2022.
Choi, J., Kim, S., Jeong. Y., Gwon, Y., and Yoon. ILVR: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
References
Aali, A., Daras, G., Levac, B., Kumar, S., Dimakis, A. G., and Tamir, J. I. Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data. In ICLR, 2025.
Chung, H. and Ye, J. C. Score-based diffusion models for accelerated MRI. Medical Image Analysis, 2022.
Chung, H., Sim, B., Ryu, D., and Ye, J. C. Improving diffusion models for inverse problems using manifold constraints. In NeurIPS, 2022.
Aggarwal, H. K., Mani, M. P., and Jacob, M. Modl: Model- based deep learning architecture for inverse problems. IEEE Transactions Medical Imaging, 2018.
Chung, H., Kim, J., Kim, S., and Ye, J. C. Parallel diffusion models of operator and image for blind inverse problems. In CVPR, 2023a.
Alkan, C., Oscanoa, J., Abraham, D., Gao, M., Nurdi- nova, A., Setsompop, K., Pauly, M., Mardani, M., and Vasanawala, S. Variational diffusion models for blind MRI inverse problems. In NewrIPS, 2023.
Chung, H., Kim, J., Mecann, M. T. Klasky, M. L.., and Ye, J. C. Diffusson posterior sampling for general noisy inverse problems. In ICLR, 2023b.
Anderson, T. W., Anderson, T. W., Anderson, T. W., Ander- son, T. W., and Mathématicien, E.-U, An introduction to multivariate statistical analysis. Wiley New York, 1958.
Chung, H., Lee, S., and Ye, J. C. Decomposed diffusion sampler for accelerating large-scale inverse problems. In ICLR, 2024.
Asim, M., Daniels, M., Leong, O., Ahmed, A., and Hand, P. Invertible generative models for inverse problems: Mitigating representation error and dataset bias. In ICML., 2020a.
Daras, G., Dean, J., Jalal, A., and Dimakis, A. Intermedi- ate layer optimization for inverse problems using deep generative models. In ICML, 2021.
Asim, M., Shamshad, F., and Ahmed, Blind image decon- volution using deep generative priors. IEEE Transactions Computational Imaging, 2020b.
Daras, G., Dagan, Y., Dimakis, A., and Daskalakis, C. Score- guided intermediate level optimization: Fast Langevin mixing for inverse problems. In ICML, 2022.
Bickel, P. J., Ritov, Y., and Tsybakov, A. B. Simultaneous analysis of Lasso and Dantzig selector. The Annals of Statistics, 2009.
Deng, J.. Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Blau, Y. and Michaeli, T. The perception-distortion tradeoff. In CVPR, 2018
Dhar, M., Grover, A., and Ermon, S. Modeling sparse devi- ations for compressed sensing using generative models. In ICML, 2018.
Bora, A., Jalal, A., Price, E., and Dimakis, A. G. Com- pressed sensing using generative models. In /CML, 2017.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs image synthesis. In NeurlPS, 2021.
Candès, E. J. and Tao, T. Decoding by linear programming. IEEE Transactions Information Theory, 2005.
Dou, Z. and Song, Y. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In ICLR, 2024.
Cardoso, G., Le Corff, S., Moulines, E., et al. Monte Carlo guided denoising diffusion models for Bayesian linear inverse problems. In ICLR, 2023.
Fabian, Z., Tinaz, B., and Soltanolkotabi, M. DiracDif- fusion: Denoising and incremental reconstruction with assured conisssosstttncy..... In ICML, 2023.
Chen, H., Lee. H., and Lu, Improved analysis of score- based generative modeling: User-friendl bounds under minimal smoothness assumptions. In ICML, 2023a.
Fei, B., Lyu, Z., Pan, L., Zhang, J., Yang, W., Luo, T. Zhang, B., and Dai, B. Generative diffusion prior for unified image restoration and enhancement. CVPR, 2023.
Chen, J., Scarlett, J., Ng, M., and Liu, Z. A unified frame- work for uniform signal recovery in nonlinear generative compressed sensing. In NeurlPS, 2023b.
10

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Feng, B. T. and Bouman, K. L. Efficient Bayesian com- putational imaging with surrogate score-based prior. https://arxiv.org/ab//23999 01949, 2023.
Liu, Z., Luo, P., Wang, X., and Tang. X. Deep learning face attributes in the wild. In ICCV, 2015.
Liu, Z., Ghosh, S., and Scarlett, J. Robust 1-bit compres- sive sensing with partial Gaussian circulant matrices and generative priors. In ITW, 2021.
Foucart, S. and Rauhut, H. A Mathematical Introduction to Compressive Sensing. Springer New York, 2013.
Gao, S., Liu, X., Zeng, B., Xu, S., Li, Y., Luo, X., Liu, J., Zhen, X., and Zhang, Implicit diffusion models for continuous super-resolution. In CVPR, 2023.
Liu, Z., Liu, J., Ghosh, S., Han, J., and Scarlett, Genera- tive principal component analysis. In ICIR, 2022a.
Liu, Z., Wang, X., and Liu, J. Misspecified phase retrieval with generative priors. In NeurlPS, 2022b.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In NeurlPS, 2014.
Liu, Z., Li, W., and Chen, J. Generalized eigenvalue prob- lems with generative priors. In NeurIPS, 2024.
Ho, Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurIPS, 2020.
Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C.,, and Zhu, J. DPM- Solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurlPS, 2022a.
Hyder, R., Shah, V., Hegde, C., and Asif, M. S. Alterating phase projected gradient descent with generative priors for solving compressive phase retrieval. In ICASSP, 2019.
Lu, C., Zhou, Y., Bao, F, Chen, J., Li, C., and Zhu, J. DPM- Solver++: Fast solver for guided sampling of diffusion probabilistic models. https://arrii:oor/aaa//2 1..0199. 2022b.
Jalal, A., Arvinte, M., Daras, G., Price, E., Dimakis, A. G., and Tamir, J. Robust compressed sensing MRI with deep generative priors. NeurIPS, 2021.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Van Gool, L. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progressive growing of GANs for improved quality, stability, and variation. https://axii.orr//b///1.... 10196, 2018.
Mousavi, A., Patel, A. B., and Baraniuk, R. G. A deep learn- ing approach structured signal recovery. In Allerton, 2015.
Karras, T., Laine, S., and Aila, T. A style-based genera- tor architecture for generative adversarial networks. In CVPR, 2019.
Oymak, S., Thrampoulidis, C., and Hassibi, B. Near-optimal sample complexity bounds for circulant binary embed- ding. In ICASSP, 2017.
Kawar, B., Elad, M., Ermon, S., and Song, J. Denoising diffusion restoration models. In NeurIPS, 2022.
Kingma, D. P. Auto-encoding variational bayes. In ICLR, 2014.
Peng. P., Jalali, S., and Yuan, X. Solving inverse problems via auto-encoders. IEEE Journal on Selected Areas in Information Theory, 2020.
Li, and Yan, Adapting to unknown low-dimensional structures score-based diffusion models. In NeurlPS, 2024.
Rout, L., Raoof, N., Daras, G., Caramanis, C., Dimakis, A., and Shakkottai, S. Solving linear inverse problems prov- ably via posterior sampling with latent diffusion models. In NeurIPS, 2023.
Li, G., Wei, Y., Chen, Y., and Chi, Y. Towards faster non- asymptotic convergence for diffusion-baar based generative models. In ICLR, 2024.
Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., and Norouzi. M. Palette: Image-to-image diffusion models. In SIGGRAPH, 2022a.
Liu, Z. and Han, J. Projected gradient descent algorithms for solving nonlinear inverse problems with generative priors. In IJCAI, 2022.
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D. J., and Norouzi, M. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022b.
Liu, Z. and Scarlett, J. The generalized Lasso with nonlinear observations and generative priors. In NeurlPS, 2020a.
Liu, Z. and Scarlett, J. Information ceemmatioo--thorettio lower bounds for compressive sensing with generative models. IEEE Journal on Selected Areas in Information Theory, 2020b.
Sanghvi, Y., Chi, Y., and Chan, S. H. Kernel diffusion: An alternate approach to blind deconvolution. In ECCV, 2025.

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Shah, V. and Hegde, C. Solving linear inverse problems us- ing GAN priors: An algorithm with provable guarantees. In ICASSP, 2018.
Wang, Y., Yu, J., and Zhang. J. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2023.
Shang, S., Shan, Z., Liu, G., Wang, L.,, Wang. X., Zhang, Z., and Zhang, J. Resdiff: Combining enn and diffusion model for image super-resolution. In AAAI, 2024.
Whang. J., Delbracio, M., Talebi, H., Saharia, C., Dimakis, A. G., and Milanfar, P. Deblurring via stochastic refine- ment. In CVPR, 2022.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamies. In ICML, 2015.
Wu, L., Trippe, B., Naesseth, C., Blei, D., and Cunning- ham, J. P. Practical and asymptotically exact conditional sampling in diffusion models. In NeurlPS, 2023.
Sohoni, N. S., Aberger, C. R., Leszezynski, M., Zhang, J., and Ré, C. Low-memory neural network training: A
Wu, Y., Rosca, M., and Lillicrap, T. Deep compressed sensing. In ICML, 2019.
technical report. https://arzivvor//ab//////// 10631, 2019.
Wu, Z.., Sun, Y., Chen, Y., Zhang, B., Yue, Y., and Bouman, K. L. Principled probabilistic imaging using diffusion models as plug-and-play priors. In NeurIPS, 2024.
Song, B., Kwon, S. M., Zhang, Z., Hu, X., Qu, Q., and Shen, L. Solving inverse problems with latent diffusion models via hard data consistency. In ICLR, 2024.
Xu, T., Zhu, Z., Li, J., He, D., Wang, Y., Sun, M., Li, L., Qin. H., Wang. Y., Liu, J., and Zhang, Y.Q. Consistency model is an effective posterior sample approximation for diffusion inverse solvers. htpp:://artiv.or///bb/4433 /2063, 2024.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2021a.
Song. J., Vahdat, A., Mardani, M., and Kautz, Escudoinverse-guider diffusion models for inverse prob- lems. In ICLR, 2023.
Yeh, R. A., Chen, C., Yian Lim, T., Schwing, A. G., Hasegawa-Johnson, M., and Do, M. N. Semantic im- age inpainting with deep generative models. In CVPR, 2017.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Yu, F. Zhang, Y., Song, S., Seff, A., and Xiao, J. LSUN: Construction of a scrge-scale image dataset using deep learning with humans in the loop.
Song, Y., Shen, L., Xing, L., and Ermon, S. Solving inverse problems in medical imaging with score-based generative models. In ICLR, 2022.
http:://aniivoor/aa////500 00..:///aabb//..636666... 2015.
Zhang, B., Chu, W., Berner, J., Meng, C., Anandkumar, A., and Song, Y. Improving diffusion inverse problem solving with decoupled noise annealing. In CVPR, 2025.
Sun, Y., Wu, Z., Chen, Y., Feng, B. T., and Bouman, K. L. Provable probabilistic imaging using score-based genera- tive priors. IEEE Transactions on Computational Imag- ing, 2024.
Zhang, G., Ji, J., Zhang, Y., Yu, M., Jaakkola, T. S., and Chang, Towards coherent image impainting using de- noising diffusion implicit models. ICML, 2023.
Tibshirani, R. Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society Series B, 1996.
Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., and Timofte, R. pla--yy image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Tran, P., Tran, A. T., Phung, Q., and Hoai, M. Explore image deblurring via encoded blur kernel space. In CVPR, 2021.
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
Twomey, S. Introduction to the mathematics of inversion in remote sensing and indirect measurements. Courier Dover Publications, 2019.
Zhao, W., Bai, L., Rao, Y., Zhou, J., and Lu, J. UniPC: A unified predictor-corrector framework for fast sampling of diffusion models. In NeurlPS, 2024.
Vempala, S. S. The random projection method. American Mathematical Society, 2005.
Wang, H., Zhang, X., Li, T., Wan, Y., Chen, T., and Sun, J. DMPlug: A plug-in method for solving inverse problems with diffusion models. In NewrIPS, 2024.
Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R., and Van Gool, L. Denoising diffusson models for plug-and-play image restoration. In CVPR, 2023.
12

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
A. Proof of Theorem 4.4
First, we present the following basic concentration inequality for the Gaussian measurement matrix.
Lemma A.1. (Vempala, 2005, Lemma 1.3) Suppose that A R=x= has i...d. N 1/m) entries. For fixed æ E R", have for any e E 10 that
[11==) S || (1 2 2 1-11-2e--
22- 22
(20)
Based on Lemma A.1 and Assumptions 4.1 and 4.2, we present the proof of Theorem 4.4 as follows.
A.1. Proof of Theorem 4.4 From Assumption 4.2, we have
log N & (22) S Cg k2 log Lin 5
(21)
L
Additionally, using Maurey's 's Empirical Method (see (Daras al., 2021, Theorem 2)). we can derive:
L²
L2
log.N, N. (B)(r))
log 2m +) 1) S
log (3)..
(22)
82
82
Then, if setting & = r we have
log N a (B) S 82
log(2n +1) $ 2.2 log(3n).
(23)
Moreover, since 91 is Li-Lisschitz continuous, if M is a (6/In)-net of X₂ + (r),,, we have that gi (MM is 5-net of gi (X2 + (r)). Therefore, letting X' gi (N2 + BT (r)), we have
Lin k2 log(3n).
No(N < k2 log
(24)
Then, based on Lemma A.1 and the well-established chaining arguments in (Bora et al., 2017; Liu & Scarlett, 2020a), we have that when m = (kk log 4T + 122 log(3n)) with probability e-R(m), A satisfies S-REC(X' - X, 0). Then, we have
(191 x x1122 S (() 91 (a1))12 (x1) x 12
(25)
1
(IA((r( (1 12 6) 16 (xi) - x²112
(26)
< Al) (.)) - x3)122 | (a ll2 + d) + |l (x.) =x2
(27)
(21|A(x* - (Z())))l + 8) + (|ا x2112
(28)
1 2 - (x,))
(29)
where (26) follows from A satisfies S-REC(X 7, 8). (28) follows from (18), and (29) follows from Lemma A.1 with setting 1. This completes the proof.
B. Algorithms for BID
Following the kernel estimation method proposed DMPlug (Wang et al., 2024), we incorporate it into our framework and present the complete algorithms for our methods DMILO and DMILO-PGD for blind image deblurring tasks in Algorithms and 4, respectively.
13

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Algorithm 3 DMILO for BID
1: Input: Diffusion denoising model g(-) = 91 o ...... gV-1 gN (+), total number of sampling steps N, noisy observation y, iteration steps J, Lagrange multiplier
2: Initialize k(0) ~ ~O( I). Fiy ~N(0, and ix = 0
(0)
3: for i= N to 2 do
4: [0) = 9: (zt, t,-1 =
5: end for
6: for j
do
7: Approximately solve (at)) (1) 100)) = arg min l1 - k (g) + +2) + Alje/ using the Adam optimizer,
initailized at (ar (--1) (f-1) kb-1))
Ut
8: for 2 to N do
(0:(20) + +2)1 >||2/|| using the Adam optimizer, Approximately solve (at U/, = arg mi [1 ())
(i)
9:
initialized (x) w(--1) 10: end for
11: for i = N to do
(1) (z (1)) + (5)
12:
zt,
13: end for
14:
end for Return Vio
(J)
15:
C. More Experiment Results
C.1. Results on LSUN-bedroom Dataset
We conduct experiments on two linear inverse tasks and one nonlinear inverse task using 100 validation images randomly selected from the LSUN-Bedroom dataset. The settings of our algorithms follow the same configuration recommended in the main document. The results indicate that our algorithms DMILO and DMILO-PGD, perform well on the inpainting and nonlinear deblurring tasks, achieving the best performance across all metrics, while showing slightly lower performance on the super-resolution task.
Table Super-resolution, inpainting and nonlinear deblurring with additive Gaussian noise (a = 0.01) on the LSUN- bedroom dataset.
Super-resohtton (4x)
Inpainting (Random 70%)
Nonlinear Deblurring
LPIPS
PSNR1
SSIM?
LPIPS.,
PSNR
SSIMI
LPIPS
PSNR
SSIM
DDRM
0,200
27.62
0.820
0.223
25.53
0.790
N/A
NIA
N/A
DPS
0,275
24.03
0.677
0.124
29.92
0.863
0.633
22.40
0.312
HGDM
0.125
27.57
0.829
0.101
28.68
0,860
0.449
17.83
0.390
RED-diff
0.207
28.15
0,833
0.223
26.81
0.805
0.469
20.46
0.639
DMPlug
0.135
27.78
0.820
0.061
32.62
0.920
0.107
30.01
0.862
DMILO
0.167
27.63
0.790
0.034
32.62
0,941
0.061
31,89
0.919
DMILO-PGD
28.19
0.032
0.043
33.31
0.930
Ours Best comp.
+0.065
+0.04
.0.012
-0.029
+0.21
+0.021
.0.064
+3.30
+0.068
C.2. Total Number of Sampling Steps
There exists an intuitive idea that the final sampling step alone may be sufficient to as the entire generator, which can be formally expressed as:
g(-) = 91 (.).
(30)
To investigate this hypothesis, we conduct experiments where optimization is performed exclusively through the last timestep of the diffusion process. The results are summarized in Table 9, where "LTS" denotes methods that utilize only the last timestep for optimization.
14

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Algorithm 4 DMILO-PGD for BID
1: Input: Diffusion denoising model ((-) = 91 o..... -N-1 ON (.), total number of sampling steps N, noisy observation y, iteration steps E, learning rate lr. llk. Lagrange multiplier
2: Initialize Lo
(0) = 0, k()) ~ NO, I), Fix ~ (O,, I) and Vi = 0
(0)
3: for i N to 2 do
4: ziz-1 = 9: (z 1,-1
5: end for
6: for e to do
7: ( to
(e-1)
1/2 Vly kle-1), (2/6 11
8: Approximately solve (w/( ,/ (()) arg (w,) [k(le_1) ti kle-1) (g) = A||
using the Adam
optimizer, initialized at (
(e-1) 1/11
(c-1))
9: for = 2 to N do
Approximately solve (xt U/, (c)) = arg a || (gi(2) + +2]13 + A||2||1 using the Adam optimizer, (c-1) Vi (e-1))
10:
initialized at (z 11: end for
for N do
13: (.) Di (e)) Vi
14: end for
15: k(()) = (e- mh V 1 12 1 1le-1)
t (0)
16: end for (E)
17: Return = Zio
Our findings indicate that optimizing solely through the last timestep remains effective to some extent, though it leads to slight degradation in reconstruction performance compared to full multi-step optimization. We hypothesize that this performance drop is primarily due to suboptimal initialization. In principle, for the dast-timestep-only approach to perform well, the initialization vector should ideally lie within the range of the composition of all preceding sampling steps. However, identifying such an initialization is nontrivial and remains an open challenge.
Further exploration of appropriate initialization strategies for the last-timestep approach will be considered part of our future research directions.
Table 9: Experimental results for the inpainting task on 100 validation images from CelebA
LPIPS
PSNRT
SSIM
FID
DMPlug
0.066
35.51
0.935
49.98
DMILO
0.025
36.07
0.951
19.34
DMILO-LTS
0.041
34.22
0.934
25.54
DMILO-PGD
0.023
36.42
0.952
19.08
DMIT O.PGD.I.TS
0.031
34.42
0.937
19.46
C.3. Sparse Deviation
In DMILO, we introduce sparse deviations to expand the effective range of diffusion models, aiming to achieve better reconstruction performance. We hypothesize that incorporating sparse deviations can alleviate error accumulation caused by inaccurate intermediate optimization steps, thereby improving the overall quality of the reconstructed images.
To validate this hypothesis, conduct ablation study to examine the impact of sparse deviations the super-resolution task using the CelebA dataset. The results are summarized in Table 10, where "w/" denotes methods that utilize sparse deviations, and "w/o" represents methods without sparse deviations.
As shown in Table 10, the inclusion of sparse deviations consistently improves performance across multiple metrics, including LPIPS, PSNR, and SSIM. These results clearly demonstrate the effectiveness of the proposed sparse deviation mechanism in enhancing reconstruction quality.
15

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Table 10: Ablation study on the effect of sparse deviations for the super-resolution task on 100 validation images from the CelebA dataset.
LPIPSL
PSNRT
SSIMT
DMPlug
0.127
32.38
0.875
30.81
0.785
(w/o)
DMILO-PUD (w/)
0.056
33.58
0.906
DMILO-PGD (wlo)
0.173
32.07
0,870
C.4. Memory and Time Cost
To further validate the efficiency of our methods, we conduct a comparative experiment on the CelebA dataset for the image inpainting task to further demonstrate the time efficiency advantage of our proposed methods over DMPlug. The experiment performed using a single NVIDIA GeForce RTX 4090 GPU, with pretrained diffusion model having a parameter size of 357 MB. We compare the GPU memory consumption and the time required to reconstruct a single image under the optimal parameter settings for all compared methods. Additionally, we apply gradient checkpointing (Sohoni al., 2019) DMPlug optimize its memory consumption, where the variant with checkpointing is referred to HAMPlug-Ckpi in the table.
The experimental results are presented in Table 11. The gradient checkpointing strategy effectively reduces the memory burden DMPlug. However, this comes at the expense of increased computation time due to the overhead introduced by saving and loading intermediate gradients. In contrast, our proposed methods not only reduce memory consumption but also significantly lower the computation time compared to DMPlug, which because our methods employ a smaller gradient graph and then lessen the burden of gradient computation.
Table 11: Memory Cost (in GB) and Time Cost (in seconds)
DDRM
IIGDM
RED
DinPuR
DAPS
DMPlug
DMPlug Ckpt
DMILO
NFE.
20
1000
50
50
20
500
15000
15000
3000
3000
Time
43
1256
150
Memory
1.38
4.61
1.37
16

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
D. Visualization of Experimental Results
visualize our experimental results on two linear inverse tasks: inpainting and super-resolution and three nonlinear inverse tasks: nonlinear deblurring. blind Gaussian deblurring and blind motion deblurring tasks. All tasks are under Gaussian noise with the noise level 0.01.
DORM
RGON
SAATi
EMILO-IGD
CclebA
FFHQ
LSUN
Figure 6: Visualization of sample results for inpainting with additive Gaussian noise (a 0.01).
Hed TAl
CelebA
FFHQ
LSUN
Figure 7: Visualization of sample results for 4x super-resolution with additive Gaussian noise (a = 0.01).
17

---

Integrating Intermediate Layer Optimization and Projected Gradient Descent for Solving Inverse Problems with Diffusion Models
Jafrxarr
EPGIS
Xed Bdr
UNILLO
SNELLA
CelebA
FFHQ
LSUN
Figure 8: Visualization of sample results for nonlinear deblurring with additive Gaussian noise (a 0.01).
BMF
CelebA
FFHQ
Figure 9: Visualization of sample results for BID with Gaussian kernel with additive Gaussian noise (a = 0.01).
CelebA
FFHQ
Figure 10: Visualization of sample results for BID with a motion kernel with additive Gaussian noise (a = 0.01).
18


Reviews:
[{"review_id": "cHUUlWnlBi", "reviewer": "Reviewer_ve9x", "summary": "This paper proposes a novel approach for solving inverse problems using diffusion models through an iterative intermediate layer optimization strategy (DMILO). The optimization process is enhanced by introducing sparse deviations off the manifold of the diffusion trajectory, which allows the model to generalize beyond the range of the pre-trained diffusion model. Additionally, the method is further refined using projected gradient descent (PGD) to mitigate the risk of suboptimal convergence. The method achieves superior performance on linear tasks, nonlinear tasks, and blind image deblurring tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of this paper and the valuable feedback and suggestions. Our responses to the main concerns are given as follows.  \n\n(**My main concern is the computational cost of the proposed method. If the total number of function evaluations (NFEs) is fixed, does the proposed method still achieve better performance? Could the authors provide additional insights into the computational efficiency of their approach?**)   \nOur methods build on DMPlug and reduce its computational overhead. For instance, in the inpainting task, our methods outperform DMPlug while using significantly fewer NFEs. Specifically, our methods need only 3,000 NFEs in total, compared to the 15,000 NFEs required by DMPlug. Even when the number of NFEs is the same, our methods are computationally more efficient. This is because our methods employ a smaller gradient graph, which lessens the burden of gradient computation. In Table A1 below, we present the computational cost of reconstructing a validation image from the CelebA dataset for different methods for inpainting using an NVIDIA RTX 4090 GPU. The results demonstrate that our methods require less computational time than DMPlug.\n\n**Table A1: Computation cost for different approaches.**\n||DDRM|DPS|$\\Pi$GDM|RED-diff|DMPlug|**DMILO**|**DMILO-PGD**|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|NFE|20|1000|50|50|15000|3000|3000|\n|Time (s)|1|40|2|1|925|150|151|\n\n(**Did the author compare in terms of memory and computation cost by applying checkpointing to DMPlug?**)   \nFollowing your suggestion, we compare in terms of memory and computation cost by applying gradient checkpointing to DMPlug (see Table A2 below). The gradient checkpointing strategy effectively reduces the memory burden significantly. However, it simultaneously increases the computation cost because of the overhead associated with saving and loading gradients. In contrast, our methods manage to decrease both the memory and computation costs compared to DMPlug.\n\n**Table A2: Memory cost for different approaches.**\n||DMPlug|DMPlug-Ckpt|**DMILO**|**DMILO-PGD**|\n|:---:|:---:|:---:|:---:|:---:|\n|NFE|15000|15000|3000|3000|\n|Time (s)|925|1256|150|151|\n|Memory (GB)|6.94|3.01|3.33|3.34|\n\n(\"DMPlug-Ckpt\" denotes DMPlug with gradient checkpointing)\n\n(**There is no ablation study on the effect of sparse deviations., which is a major component in proposed method. Could the authors provide insights into how adding sparse deviations influences performance?**)  \nThank you for your insightful comment. We perform an ablation study on the impact of sparse deviations in the super-resolution task on the CelebA dataset (see Table A3 below). The results demonstrate the effectiveness of adding sparse deviations. In our understanding, adding sparse deviations not only broadens the range of diffusion models but also alleviates error accumulation resulting from inaccurate intermediate calculations, thus leading to improved reconstruction performance.\n\n**Table A3: Ablation study on the effect of sparse deviations for the super-resolution task on 100 validation images from CelebA.**\n||LPIPS|PSNR|SSIM|\n|:---:|:---:|:---:|:---:|\n|DMPlug|0.127|32.38|0.875|\n|DMILO (w/)|0.133|30.81|0.785|\n|DMILO (w/o)|0.202|29.23|0.699|\n|DMILO-PGD (w/)|0.056|33.58|0.906|\n|DMILO-PGD (w/o)|0.173|32.07|0.870|\n\n(\"w/\" denotes methods employing sparse deviations, and \"w/o\" denotes methods without adding sparse deviations.)\n\n(**Figure 1 appears inconsistent with its caption.**)   \nThank you for pointing this out. In the revised version, we will correct this to make Figure 1 consistent with its caption.", "claims_and_evidence": "The paper makes three primary claims:\n\n1. By replacing optimization over the entire deterministic diffusion sampling process, the proposed intermediate-layer optimization reduces memory burden.\n2. By leveraging sparse deviations, the approach gains additional flexibility to recover signals outside the range of the diffusion model.\n3. The proposed method achieves improved empirical performance compared to baseline approaches.\n\nThese claims are well-supported by extensive empirical experiments and intuitive theoretical analysis.", "methods_and_evaluation_criteria": "The method is evaluated on several linear inverse problems, nonlinear deblurring, and blind image deblurring tasks using the CelebA and FFHQ datasets. The evaluation metrics include LPIPS, PSNR, and SSIM.", "theoretical_claims": "A convergence bound is provided in Theorem 4.4.", "experimental_designs_or_analyses": "The experiment design is sound and valid.", "relation_to_broader_scientific_literature": "This work contributes to the broader field by presenting a memory-efficient and empirically stronger approach for solving inverse problems through iterative diffusion inversion over the deterministic diffusion sampling process.", "essential_references_not_discussed": "I did not identify any critical missing references.", "other_strengths_and_weaknesses": "Strength:\n\n1. The paper is well-motivated, addressing key challenges such as memory burden and suboptimal convergence in prior methods like DMPlug for solving inverse problems with diffusion models.\n2. The writing is clear and provides strong intuition behind each design choice.\n3. The proposed method consistently improves empirical performance across selected linear and nonlinear tasks.\n\nWeakness:\n\n1. My main concern is the computational cost of the proposed method. As described in Algorithms 1 and 2, DMILO and DMILO-PGD require solving $J\\cdot N$ and $E\\cdot N$ optimization problems via backward gradient passes through a **one-step** diffusion model. In contrast, the baseline DMPlug only requires solving a single optimization problem over a **three-step** diffusion model. If the total number of function evaluations (NFEs) is fixed, does the proposed method still achieve better performance? Could the authors provide additional insights into the computational efficiency of their approach?\n2. The paper states that the proposed method addresses the memory burden in DMPlug. However, an alternative strategy commonly used in practice is gradient checkpointing, which reduces memory consumption at a slight computational overhead. Did the author compare in terms of memory and computation cost by applying checkpointing to DMPlug?\n3. A minor issue is that DMILO-PGD performs worse on the blind image deblurring task, though a possible reason is provided.\n4. There is no ablation study on the effect of sparse deviations, which is a major component in proposed method. Could the authors provide insights into how adding sparse deviations influences performance?", "other_comments_or_suggestions": "Figure 1 appears inconsistent with its caption.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xTPiRrcx9K", "reviewer": "Reviewer_D1CM", "summary": "The paper proposes DMILO and DMILO-PGD, two novel methods for solving inverse problems using diffusion models. DMILO introduces Intermediate Layer Optimization (ILO) to reduce memory burden while improving reconstruction by allowing model variations. DMILO-PGD further integrates Projected Gradient Descent (PGD) to mitigate the lack of measurement fidelity in DMILO. The authors provide a theoretical analysis under certain conditions, demonstrating the effectiveness of their methods. Experiments across multiple linear and nonlinear inverse problems show significant improvements over state-of-the-art approaches in terms of memory efficiency and reconstruction quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive assessment of this paper and the valuable comments and suggestions. Our responses to the main concerns are given as follows.  \n\n(**Table 2 should be completed with linear deblurring (Gaussian or Motion), and baselines such as DPIR and/or DiffPIR. Would the authors be able to add comparisons to DPIR/DiffPIR?**)   \nWe conduct additional evaluations on linear Gaussian and motion deblurring tasks, comparing our methods with DPIR and DiffPIR on CelebA and FFHQ. The results are shown in Tables B1-B4, which reveal that while DiffPIR demonstrates outstanding performance in terms of the perceptual quality metric LPIPS and DPIR shows superiority in image distortion metrics such as PSNR and SSIM for linear Gaussian deblurring, our DMILO method generally attains the best performance across nearly all metrics, especially for linear motion deblurring. Note that we follow DMPlug to set the kernel size to $64 \\times 64$, which differs from the $61 \\times 61$ kernel size initially employed in DPIR and DiffPIR. Such a difference might impact the results.\n\nIn the revised version, we will further incorporate comparisons with DPIR and DiffPIR for other applicable tasks.\n\nWe sincerely hope that these additional comparisons have appropriately addressed your comments, and that you can kindly consider increasing your initial rating accordingly.\n\n**Table B1: Comparisons of different methods for linear Gaussian deblurring on 100 validation images from CelebA.**\n||LPIPS|PSNR|SSIM|\n|:---:|:---:|:---:|:---:|\n|DPS|0.109|27.65|0.752|\n|RED-diff|0.221|29.59|0.808|\n|DPIR|0.256|**31.30**|**0.861**|\n|DiffPIR|**0.092**|28.91|0.791|\n|DMPlug|0.172|29.70|0.776|\n|DMILO|**0.092**|30.89|0.816|\n|DMILO-PGD|0.157|30.74|0.811|\n\n**Table B2: Comparisons of different methods for the linear Gaussian deblurring task on 100 validation images from FFHQ.**\n||LPIPS|PSNR|SSIM|\n|:---:|:---:|:---:|:---:|\n|DPS|0.150|25.56|0.717|\n|RED-diff|0.272|27.15|0.778|\n|DPIR|0.271|29.06|0.844|\n|DiffPIR|0.119|26.88|0.769|\n|DMPlug|0.181|28.27|0.806|\n|DMILO|**0.110**|**29.60**|**0.852**|\n|DMILO-PGD|0.176|28.65|0.799|\n\n**Table B3: Comparisons of different methods for the linear motion deblurring task on 100 validation images from CelebA.**\n||LPIPS|PSNR|SSIM|\n|:---:|:---:|:---:|:---:|\n|DPS|0.126|26.62|0.730|\n|RED-diff|0.229|27.32|0.758|\n|DPIR|0.192|31.09|0.826|\n|DiffPIR|0.117|28.35|0.773|\n|DMPlug|0.164|30.25|0.824|\n|DMILO|**0.044**|**34.15**|**0.908**|\n|DMILO-PGD|0.067|33.41|0.884|\n\n**Table B4: Comparisons of different methods for the linear motion deblurring task on 100 validation images from FFHQ.**\n||LPIPS|PSNR|SSIM|\n|:---:|:---:|:---:|:---:|\n|DPS|0.167|24.34|0.676|\n|RED-diff|0.272|25.40|0.730|\n|DPIR|0.181|29.67|0.820|\n|DiffPIR|0.137|26.41|0.740|\n|DMPlug|0.173|28.58|0.812|\n|DMILO|**0.044**|**33.21**|**0.909**|\n|DMILO-PGD|0.079|31.66|0.857|\n\n\n(**I disagree with the authors in Paragraph 3.2 when they state: \"A key difference from conventional PGD is that…\"**)   \nThank you for the detailed comment. We will remove the phrase “a key difference” and reword the sentence to enhance its precision.\n\n(**Would the authors be able to precisely state the functional Algorithm 2 minimizes (which is not precisely (12))?**)  \nThank you for the question. In (12), we use $\\hat{\\mathbf{x}} _ {t _ 0}$ to represent the estimated signal. As Algorithm 1 does not provide an explicit form for $\\hat{\\mathbf{x}} _ {t _ 0}$, we substitute the inaccessible $\\mathcal{A}(\\hat{\\mathbf{x}} _ {t _ 0})$ with the observed vector $\\mathbf{y}$. In Algorithm 2, we first calculate $\\mathbf{x} _ {t _ 0}^{(e)}$ via simple gradient descent. This $\\mathbf{x} _ {t _ 0}^{(e)}$ serves as an explicit approximation of $\\hat{\\mathbf{x}} _ {t _ 0}$. We then fix $\\mathbf{x} _ {t _ 0}^{(e)}$ and use $\\mathcal{A}(\\mathbf{x} _ {t _ 0}^{(e)})$ in place of $\\mathcal{A}(\\hat{\\mathbf{x}} _ {t _ 0})$ in (12).", "claims_and_evidence": "The main claims and evidences are the following:\n\n**Claim1:** Theoretical analysis justifies the effectiveness of the proposed methods.\n**Evidence1:** The authors provide a low-dimensional manifold assumption, a Set-Restricted Eigenvalue Condition (S-REC), and a theorem (Theorem 4.4) proving that the learned measurement optimum is close to the true optimum under certain conditions (Section 4).\n\n**Claim2:** The proposed method improves over state-of-the-art methods for solving inverse problems.\n**Evidence2:** Large experiments are performed over a wide range of tasks and confirm the claim of the authors.", "methods_and_evaluation_criteria": "Methods and evaluation criteria (e.g. metrics) are appropriate and convincing.", "theoretical_claims": "I did check the proof of Theorem 4.4 and did not spot any mistake.", "experimental_designs_or_analyses": "I did check experimental designs and analyses, they are performed seriously. However, some comparisons are not extremely fair for linear IP. In particular, Table 2 should be completed with linear deblurring (Gaussian or Motion), and baselines such as DPIR https://arxiv.org/abs/2008.13751 and/or DiffPIR https://arxiv.org/abs/2305.08995, which are stronger baselines than e.g. RedDIff or DPS (in my experience).", "relation_to_broader_scientific_literature": "Relation to the scientific literature is well done for comparable methods, but related works to other approaches (non diffusion, e.g. PnP algorithms despite relevant, in particular in the context of implicitly learned priors) are missing, see landmark algorithms https://arxiv.org/abs/2008.13751, https://arxiv.org/abs/2305.08995.", "essential_references_not_discussed": "https://arxiv.org/abs/2008.13751 \nhttps://arxiv.org/abs/2305.08995.", "other_strengths_and_weaknesses": "Overall, this is a good paper with convincing results. Its main strength is its theoretical side. Its main limiation is its incremental aspect as well as missing baselines that I think would make the work even more convincing.\n\n**Strengths:**\n1. The approach, although incremental, is interesting\n2. The theoretical justification is rigorously justified\n3. Experimental results are convincing\n4. The paper is well written\n\n**Weaknesses:**\n1. This is an incremental work\n2. Some important linear inverse problems are missing, e.g. motion or gaussian deblurring\n3. Comparisons with other relevant methods could be included, in particular in the relatively easy case considered by the authors (i.e. deblurring with low noise level), where other baselines work particularly well (e.g. PnP, see the DPIR/DiffPIR mentionned above).", "other_comments_or_suggestions": "None", "questions_for_authors": "- Would the authors be able to add comparisons to DPIR/DiffPIR? This is the reason for my rating, which I'd be happy to reconsider. EDIT: this has been addressed by the authors.\n- I disagree with the authors in Paragraph 3.2 when they state: \"A key difference from conventional PGD is that we minimize $\\|\\|\\mathcal{A}(\\mathcal{G}(x))-\\mathcal{A}(x)\\|\\|^2$ (notations simplified) rather than $\\|\\|\\mathcal{G}(x) - x\\|\\|^2$ (again, notations simplified).\" I think that this sentence is not very accurate: PGD minimizes the sum of two terms (here the authors mention only one), and PGD can be used to minimize any type of function - with or without $\\mathcal{A}$. Furthermore, the difference between DMILO and DMILO-PGD lies (mainly) in step 4 of the PGD version, which is the usual gradient term from PGD... If I understand what the authors mean, it is probably wrt the proximal step, which would write in a different manner (e.g. $\\text{argmin} \\mathcal{G}(x) + \\frac{1}{2}\\|x-u\\|_2^2$). All in all, what the authors mean in this sentence is a bit vague and could be made more precise.\n- (optional) Would the authors be able to precisely state the functional Algorithm 2 minimizes (which is not precisely (12))? If yes this would be an interesting discussion in the appendix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sikab62HsJ", "reviewer": "Reviewer_ch18", "summary": "This paper introduces two novel methods, DMILO and DMILO-PGD, to address computational and convergence challenges in solving inverse problems (IPs) using diffusion models (DMs).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your useful comments and questions. Our responses to the main concerns are given as follows.  \n\n(**The analysis does not explicitly address whether the bound generalizes to N > 2, leaving open questions about scalability.**)  \nWe follow the work for ILO (Daras et al., 2021) to set $N = 2$. We believe that this is sufficient for an intuitive theoretical analysis of the effectiveness of our methods (e.g., Reviewer ve9x found that our claims are “well-supported by extensive empirical experiments and intuitive theoretical analysis”, Reviewer D1CM found that our theoretical analysis “justifies the effectiveness of the proposed methods”, and Reviewer oTx5 found “the theoretical analysis sufficient”).\n\n(**The theorem uses ℓ₁-regularization for ν, but the paper’s experiments employ Adam optimization without explicit guarantees of sparse recovery. This creates a gap between theory (exact ℓ₁ minimization) and practice (approximate optimization).**)  \nFirstly, it should be noted that even the globally optimal solutions to the $\\ell_1$ minimization problem might not exhibit sparsity. Secondly, given that the objective function of the $\\ell_1$ minimization problem is highly non-convex and obtaining its globally optimal solutions is not feasible, in practical applications, we employ the Adam optimizer to approximately solve the $\\ell_1$ minimization problem.\n\n(**It is recommended that the authors expand their scope to include inverse problems in other modalities, such as linear sparse view CT reconstruction and nonlinear metal artifact reduction in CT imaging. The selected inverse problems (super-resolution, inpainting, nonlinear deblurring, and blind image deblurring) are limited in scope and lack diversity in modalities (e.g., medical imaging, audio, or 3D data).**)   \nWe agree that tasks such as sparse-view CT reconstruction and metal artifact reduction in CT imaging are of practical importance. However, prior works closely related to our study, such as DDRM, DPS, $\\Pi$GDM, DiffPIR, and DMPlug, have concentrated their experiments on natural images for tasks including super-resolution, inpainting, nonlinear deblurring, and blind image deblurring. As is evident from the fact that these recent papers have been published in the topmost venues in ML or CV, and/or have been highly cited, this line of investigation has been widely accepted in the community and is a highly active area of research. We believe that extending the experiments to tasks like linear sparse-view CT reconstruction and multi-modal data such as medical imaging, audio, or 3D data is beyond the scope of the current work.\n\n(**The abstract is overly lengthy and contains redundant information. It should be more concise and focused on the key contributions and findings. Additionally, the inclusion of citations within the abstract is unconventional and should be removed.**)  \nThank you for the comment. In the revised version, we will shorten the abstract to enhance its conciseness, highlight key contributions, and remove the citation.\n\n(**The experimental setup is limited and does not comprehensively address the claimed issues of heavy computational demands and suboptimal convergence in DM-based methods.**)  \nOur methods build on DMPlug and reduce its computational overhead. Due to the character limit, please refer to Table D2 in our responses to Reviewer oTx5 for an illustration. As PGD is known for its capacity to alleviate the problem of suboptimal convergence (Shah & Hegde, 2018), the effectiveness of addressing suboptimal convergence in DM-based methods can be observed from the experimental results presented in Table 2 in the main document. Specifically, for super-resolution and inpainting tasks, DMILO-PGD yields the best reconstructions, thereby demonstrating this advantage.\n\n(**The proposed methods, DMILO and DMILO-PGD, exhibit noticeable performance fluctuations across different tasks. The authors should provide a detailed analysis of why these fluctuations occur and whether they are related to specific properties of the tasks.**)  \nWe found that for linear deblurring and BID tasks, DMILO achieves the best reconstruction performance, whereas DMILO-PGD gives comparatively inferior results. As mentioned in Section 5.3, performance fluctuations may arise from the naive gradient update, which may not be well-suited for all tasks. Nevertheless, across most tasks, both DMILO and DMILO-PGD show competitive performance. We leave the detailed analysis of why these fluctuations occur and whether they are related to specific properties of the tasks to future work.\n\n(**The paper does not adequately discuss the limitations of the proposed methods. For example: Can the methods handle highly ill-posed problems or tasks with significant noise?**)   \nWe primarily follow the experimental settings of DMPlug, which does not deal with highly ill-posed problems or tasks with significant noise. We leave the research on these aspects to future work.", "claims_and_evidence": "The core claims, such as memory efficiency, and improved convergence, are not supported by experiments. Thus, the claims are not convincing.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "### 1. Simplified Composition\n- The theorem assumes **G = g₁ ∘ g₂**, whereas DMs involve **multi-step compositions** (*g₁ ∘ g₂ ∘ ⋯ ∘ gₙ*).  \n  The analysis does not explicitly address whether the bound generalizes to *N > 2*, leaving open questions about scalability.\n\n### 2. Practical Relevance of Assumptions\n- The **Lipschitz continuity of g₁** may not hold strictly for real-world DMs due to non-linearities in neural networks.  \n  However, this is a common simplification in theoretical analyses of DMs (Chen et al., 2023; Li & Yan, 2024).\n\n### 3. Sparse Deviation Regularization\n- The theorem uses **ℓ₁-regularization** for *ν*, but the paper’s experiments employ Adam optimization without explicit guarantees of sparse recovery.  This creates a gap between theory (exact **ℓ₁** minimization) and practice (approximate optimization).", "experimental_designs_or_analyses": "The paper addresses the solution of inverse problems. While both linear and nonlinear inverse problems are considered, the selection of problems is rather limited and primarily confined to natural images. It is recommended that the authors expand their scope to include inverse problems in other modalities, such as linear sparse view CT reconstruction and nonlinear metal artifact reduction in CT imaging. Such an expansion would enhance the study's credibility and practical applicability.", "relation_to_broader_scientific_literature": "The authors proposed DMILO and DMILO-PGD, integrating ILO with and without PGD respectively. \n\n The authors offered an intuitive theoretical analysis of the proposed approach.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. The abstract is overly lengthy and contains redundant information. It should be more concise and focused on the key contributions and findings. Additionally, the inclusion of citations within the abstract is unconventional and should be removed. A well-structured abstract should briefly introduce the problem, describe the proposed methods, and summarize the main results without excessive detail or references.\n2. The experimental setup is limited and does not comprehensively address the claimed issues of heavy computational demands and suboptimal convergence in DM-based methods. While the paper demonstrates reduced memory usage, it lacks detailed analysis of computational efficiency  and convergence behavior. More rigorous experiments are needed to validate these claims.\n3. The selected inverse problems (super-resolution, inpainting, nonlinear deblurring, and blind image deblurring) are limited in scope and lack diversity in modalities (e.g., medical imaging, audio, or 3D data). Including a broader range of tasks and modalities would strengthen the generalizability of the proposed methods and better demonstrate their applicability to real-world scenarios.\n4. The proposed methods, DMILO and DMILO-PGD, exhibit noticeable performance fluctuations across different tasks. The authors should provide a detailed analysis of why these fluctuations occur and whether they are related to specific properties of the tasks.\n5. The paper does not adequately discuss the limitations of the proposed methods. For example: Can the methods handle highly ill-posed problems or tasks with significant noise?", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review."}}, {"review_id": "LN2NUEeJks", "reviewer": "Reviewer_oTx5", "summary": "This paper proposes a novel algorithm for zero-shot inverse problem solving using diffusion models. The method build off the recent DMPlug model, which optimizes the input to conform with partial methods. The authors highlight a key insight, the optimization through the diffusion sampling process can be done more efficiently when applied to each step separately. The paper validates the result of their method, and offers another variant based on PGD.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your recognition of this paper and the valuable comments and suggestions. Our responses to the main concerns are given as follows.  \n\n(**The metrics shown do not include image perceptual quality metrics, like FID or KID. & Evaluation datasets are limited in scope. Can the method operate on general images (such as ImageNet)?**)  \n\nThanks for the comments. We perform the experiments for linear motion deblurring on the ImageNet dataset, with reporting the FID metric. Due to the time constraint of the rebuttal period, we follow DiffPIR (https://arxiv.org/abs/2305.08995) to calculate the FID on 100 validation images. The following results show that our DMILO method performs well in terms of both realism and reconstruction metrics, demonstrating its ability to effectively balance perceptual quality and distortion. We will present relevant results for more tasks and more datasets in the revised version.\n\n**Table D1: Experimental results for the linear motion deblurring task on 100 validation images from ImageNet.**\n||LPIPS|PSNR|SSIM|FID|\n|:---:|:---:|:---:|:---:|:---:|\n|DiffPIR|0.282|24.79|0.608|115.74|\n|DMPlug|0.285|25.49|0.696|99.87|\n|DMILO|0.098|29.67|0.841|53.77|\n|DMILO-PGD|0.183|27.60|0.755|85.51|\n\n(**How does the method compare to alternative approaches in terms of NFEs? As a method that utilizes multiple derivatives of each diffusion step may be computationally prohibitive.**)  \nOur methods build on DMPlug and reduce its computational overhead. For instance, in the inpainting task, our methods outperform DMPlug while using significantly fewer NFEs. Specifically, our methods need only 3,000 NFEs in total, compared to the 15,000 NFEs required by DMPlug. Even when the number of NFEs is the same, our methods are computationally more efficient. This is because our methods employ a smaller gradient graph, which lessens the burden of gradient computation. In Table D2 below, we present the computational cost of reconstructing a validation image from the CelebA dataset for different methods for inpainting using an NVIDIA RTX 4090 GPU. The results demonstrate that our methods require less computational time than DMPlug.\n\n**Table D2: Computation cost for different approaches.**\n||DDRM|DPS|$\\Pi$GDM|RED-diff|DMPlug|**DMILO**|**DMILO-PGD**|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|NFE|20|1000|50|50|15000|3000|3000|\n|Time (s)|1|40|2|1|925|150|151|\n\n\n(**The authors do not mention the Perception-Distortion Tradeoff [1], which is key is analyzing inverse problem solutions using both distortion and perception metrics.**)   \nThank you for pointing this out. In the revised version, we will cite the mentioned paper [1] for the Perception-Distortion Tradeoff, and comprehensively explain the performance of our methods in terms of both distortion and perceptual quality.\n\n(**Could the authors explain why it is insufficient to only optimize through the last timestep of the diffusion process?**)   \nThank you for the insightful question. We conduct an ablation study on only optimizing through the last timestep of the diffusion process and find that it also works, although with slightly degraded reconstruction performance (see Table D3 below). We conjecture that the degraded reconstruction performance is primarily attributed to improper initialization. Specifically, when optimizing only through the last timestep of the diffusion process, in principle, the procedure should be initialized with a vector that lies within the range of the composition of functions corresponding to all timesteps except the last one. However, identifying such an initial vector seems to be a challenging task. We plan to delve deeper into this aspect in future research.\n\nWe believe this insufficiency arises because a single sampling step may introduce errors in detail, which are difficult to correct with only sparse deviations and accumulate during optimization. We will further explore this in the future study.\n\n**Table D3: Experimental results for the inpainting task on 100 validation images from CelebA.**\n||LPIPS|PSNR|SSIM|FID|\n|:---:|:---:|:---:|:---:|:---:|\n|DMPlug|0.066|35.51|0.935|49.98|\n|DMILO|0.025|36.07|0.951|19.34|\n|DMILO-LTS|0.041|34.22|0.934|25.54|\n|DMILO-PGD|0.023|36.42|0.952|19.08|\n|DMILO-PGD-LTS|0.031|34.32|0.937|19.46|\n\n(\"LTS\" denotes methods that optimize only through the last timestep of the diffusion process.)", "claims_and_evidence": "The claim in the paper are valid but I believe some supporting evidence is missing, see method and evaluation criteria.", "methods_and_evaluation_criteria": "Overall, the metrics do fit the problem, yet, I believe some issue remain:\n - The metrics shown do not include image perceptual quality metrics, like FID or KID. Without such metrics, it is hard to access the realism of the images, or whether they are closer to MMSE estimations given the observed data $\\mathbf{y}$. The example in Fig. 5 does in fact show blurred outputs, as typical of MMSE estimators.\n - Evaluation datasets are limited in scope. Can the method operate on general images (such as ImageNet)?\n - How does the method compare to alternative approaches in terms of NFEs? As a method that utilizes multiple derivatives of each diffusion step may be computationally prohibitive.", "theoretical_claims": "I found the theoretical analysis sufficient.", "experimental_designs_or_analyses": "The experimental design seems valid to me, withholding the concerns raised in ``Methods And Evaluation Criteria''.", "relation_to_broader_scientific_literature": "- The extension to DMPlug is relevant and interesting.\n- While many algorithms for solving inverse problems with diffusion models exist, the optimization based ones are underrepresented in the field.", "essential_references_not_discussed": "The authors do not mention the Perception-Distortion Tradeoff [1], which is key is analyzing inverse problem solutions using both distortion and perception metrics.\n\n[1] Blau, Yochai, and Tomer Michaeli. \"The perception-distortion tradeoff.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.", "other_strengths_and_weaknesses": "The paper is well written and easy to follow.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "Could the authors explain why it is insufficient to only optimize through the last timestep of the diffusion process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new method for inverse problem solutions by extending the ILO method to diffusions. The optimization search allows sparse deviations from the diffusion denoising, hence extending the range of the generative model. This is a natural and interesting idea. 

The authors show good results for linear inverse problems and also beyond those to blind and non-linear inverse problems. 
The experimental evaluation is quite thorough across a wide range of tasks, comparisons to numerous previous methods and different datasets.

The reviewers asked for additional experiments (e.g. for linear Gaussian deblurring on CelebA) and the Authors provided additional strong results.

One reviewer was critical and asked for further experimentation in additional tasks, but the paper contains already a sufficiently extensive evaluation, in my opinion. 

Overall this is a solid contribution on an important and competitive research space.
Decision: Accept (poster)