Pseudoinverse-Guided Diffusion Models for Inverse Problems
Authors: Jiaming Song
Arash Vahdat
Morteza Mardani
Jan Kautz
Venue: ICLR 2023 poster (2023)
Keywords: diffusion models
inverse problems

Abstract: Diffusion models have become competitive candidates for solving various inverse problems. Models trained for specific inverse problems work well but are limited to their particular use cases, whereas methods that use problem-agnostic models are general but often perform worse empirically. To address this dilemma, we introduce Pseudoinverse-guided Diffusion Models ($\Pi$GDM), an approach that uses problem-agnostic models to close the gap in performance. $\Pi$GDM directly estimates conditional scores from the measurement model of the inverse problem without additional training. It can address inverse problems with noisy, non-linear, or even non-differentiable measurements, in contrast to many existing approaches that are limited to noiseless linear ones. We illustrate the empirical effectiveness of $\Pi$GDM on several image restoration tasks, including super-resolution, inpainting and JPEG restoration. On ImageNet, $\Pi$GDM is competitive with state-of-the-art diffusion models trained on specific tasks, and is the first to achieve this with problem-agnostic diffusion models. $\Pi$GDM can also solve a wider set of inverse problems where the measurement processes are composed of several simpler ones.

Under review as a conference paper at ICLR 2023
PSEUDOINVERSE-GUDDE DIFFUSION MODELS FOR INVERSE PROBLEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Diffusion models have become competitive candidates for solving various inverse problems. Models trained for specific inverse problems work well but are limited to their particular use cases, whereas methods that use problem-agnostic mod- els are general but often perform worse empirically. To address this dilemma, we introduce beeudoinverse-guide Diffusion Models (IGGDM), an approach that uses problem-agnostic models close the gap in performance. IIGDM directly estimates conditional scores from the measurement model of the inverse problem without additional training. It can address inverse problems with noisy, non-linear, even non-differentiable measurements, in contrast to many existing approaches that are limited to noiseless linear ones. We illustrate the empirical effectiveness of IIGDM several image restoration tasks, including super- resolutos, inpainting and JPEG restoration. On ImageNet, IIGDM is competitive with state-of-the-art diffusion models trained specific tasks, and is the first to achieve this with diffusion models. IIGDM can also solve a wider set inverse problems where the measurement processes composed of several simpler ones.
INTRODUCTION
Vx, (x.,)
Random
Pseudainverse ITT
Derrsio
A (yr) (A(5.))
VUP
Xe satisfying measurement
y
Figure 1: High-level illustration of LIGDM. (Top) Problem-agnostic diffusion models perform an it- erative denoising operation to produce random samples. (Bottom) IIGDM utilizes gobllem-agnottcc diffusion models to solve inverse problems, a key component of which is pseudoinverse guidance (TIG). IIG converts the problem-agnostic score function into problem-specific one, using informa- tion about the measurements y and measurement model, denoted as h here (h is JPEG compression masking in this figure, best viewed zoomed in). The additional guidance term is vector-Jacobian product (VJP) that encourages consistency between the denoising result and the measurements, after a pseudoinverse transformation hi. 1IGDM applies the denoising process from IIG in an iterative fashion to generate valid solutions the inverse problem.

---

Under review as a conference paper at ICLR 2023
Reference
Gaussian noise + mask
JPEG mask
row JPEG + mask
Low dynamic range
RGDM
RIGDM
NGDM
RIGMM
Figure 2: ITGDM applies single problem ragoosi-agnostic diffusion model for various inverse problems, avoiding the cost of training multiple moblem--specife ones. Best viewed zoomed in.
Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021c) have been success- fully applied to various applications such as text-to-image generation (Rombach et al., 2022: Saharia et al., 2022b), natural language generation (Li et al., 2022), audio synthesis (Kong et al., 2020), and time series modeling (Tashiro et al., 2021). The ability to model complex, high-dimensional distri- butions also makes diffusion models strong candidates for solving inverse problems, where the goal to infer the underlying signal from measurements (Bora et al., 2017; Daras et al., 2021; Ongie al., 2020; Kadkhodaie & Simoncelli, 2021).
Most methods that solve inverse problems with diffusion models fall into one of the two paradigms. In the first paradigm, specific inverse problems, such as super-resolution (Saharia et al., 2021; Whang et al., 2021; Saharia al., 2022a). In the second paradigm, one uses problem-agnostic diffusion models that are trained for generative modeling but not train on any specific inverse problem; solutions are obtained via splug-and-play approach that combines the diffusion model and the measurement process, e.g., via Bayes' rule (Venkatakrishnan et al., 2013a; Bardsley, 2012; Laumont et al., 2022; Choi et al., 2021; Song et al., 2021 b; Jalal et al., 2021; Chung et al., 2021; Kawar al., 2021; 2022a; Chung et al., 2022b; Daras et al., 2022a). These methods can easily adapt to different tasks without re-training the diffusion model but tend to perform worse than problem-specifie diffussion models.
trains a problem-specifie conditional diffusion model that is limited to
To achieve the best of both worlds, we introduce psendoinverse guidance (IIG), which uses problem- agnostic diffusion models to reach the empirical performance problem-specific ones. Conditioned on the measurements and explicit measurement model, IIG estimates the problem-specifi. score function via Bayes' rule and uses these scores to draw samples. However, unlike classifien//lassiiiee free guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), TTG obtains the mroblem-specing score directly via the known measurement model, without training additional models. Intuitively, 1IG guides the diffusion process by matching the one-step denoising solution and the ground-truth measurements, after transforming both via a "pseudoinverse" of the measurement model (see Fig. I). This perspective allows IIG to be the first guidance-baseee handles measurements with Gaussian noise, as well as some non- non-linear, non- conddfferentiable measure- ment models, such as JPEG compression (Kawar al., 2022b).
approach for inverse problem solving that
We evaluate our method, termed Pseudoinverse-Guus Guided Diffusion Models (IIGDM), on various in- verse problems, such as super-resolution. inpainting, and JPEG restoration over ImageNet validation images, and show that it achieves similar performance when compared against state-of-ther tf-te-art task- specific diffusion models (Saharia et al., 2021; Dhariwal & Nichol, 2021; Saharia et al., 2022a). To the best of our knowledge, IIGDM is the first approach based on problem-agnostee models to achieve this quality on ImageNet. We further apply IIGDM to a wider range of inverse problems, where the measurement process is composed of different types of measurements. This allows to easily solve much wider set of problems, including ones have never been solved with diffusion models (see Fig. 2), such as low-resolution JPEG compression masking.
2

---

Under review as a conference paper at ICLR 2023
2 PRELIMINARIES: DIFFUSION MODELS
Let us denote the data distribution as (x() and define a family of distributions px(x,) by injecting
i..d Gaussian noise of standard deviation dr samples of po(x), i.e., Pr (x Xo) N of The standard deviation o; is monotonically increasing with respect to time E [0, T. with
0
and or being much larger than the standard deviation of the data'. Samples from (x(x) can be
simulated by the following family of stochastic differential equations (SDE), solving from t = T to
t = (Grenander & Miller, 1994; Karras al., 2022; Zhang al., 2022):
dx -0,0 Po Pa (x)dl _ log px))dx + /2
(1)
Probubilistic ODE
Langevin process
where Vx lolloogP Px()) is the score function, is the standard Wiener process, and Br is a function that describes the amount of stochastic noise injected in the process. B, = for all t, then Eq. becomes an ordinary differential equation (ODE) (Anderson, 1982). A common choice of 8 is not/o where 7) 1 corresponds to the eeriance-epploding SDE (VE-SDE, Song et al. (2021c)) and 17 = 0 corresponds to a version of denoising diffusion implicit models (DDIM, Song et al. (2021a)). Various forms of SDEs used by diffusion models in the literature can be described with Eq. with certain and 3, functions, up to time-dependent scaling factor over x.
Diffusion models, a.k.a. score-based generative models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2021c), solve Eq. I with two key approximations. The distribution with the highest noise level, px(), is approximated with (0, I), and the score function is approximated with a neu- Sx(x; dt). trained with denoising score matching objectives (Vincent, 2011). 1). Then, samples are drawn from diffusion models by solving the ODE or SDE in Eq. 1, such as with Euler's method, Euler-Maruyama, and higher order ODE solvers (Lu al., 2022; Karras al.,
ral network Vx Vx P:
2022; Zhang& & Chen, 2022).
3 METHODS
Problem statement Suppose we have measurements y E IR of some signal xo E IR", such that
= Hx2 + z,
(2)
where H E Ru is the known measurement matrix (model), and z 00,00 is an i.i.d. Gaussian noise vector with known dimension-wist standard deviation Ty. Our goal is to solve the inverse problem and recover xa E R from the measurements y. In later parts of the paper, may consider inverse problems whose measurements are not linear, which we denote as y = h(x)).
Diffusion models solve such inverse problems via Eq. 1, assuming that the problem-specine scores for all noise levels, i.e., Vx, log p. (x. are available. While it is possible to train a condi- tional diffusion model for specific H, it is computationally expensive do this for large family of problems, such as sparse reconstruction in medical imaging (Chung & Ye, 2022). Therefore, wish utilize more commonly available problem-agnostic score models So ox) that not trained specifically for the target inverse problem. If Vx, log (xxly) can be effectively approxi- mated with Se (xx: ot), then we can directly plug it in Eq. 1 to solve the inverse problem.
3.1 APPROXIMATING THE PROBLEM-SPECITEE SCORE FUNCTION
The problem-specific score be decomposed via Bayes' rule:
Vx, lo (x+ly) Vx, (x.) + Vx, log
where the first term can be approximated with the score network So(x;;0)) ( (Vincent, 2011), and the second term is guidance term which is the score of p. (y)x.).
Unfortunately, the score Vx, Po (y (x)) is intractable to compute, and we have to resort to ap- proximations efficiently estimate it. To see why this is true, we consider the underlying graphical model for Xo, Xt, and y. which is y +- Xp x,; is produced by adding independent Gaussian
"o save space, we use subscript index o represent the parentheses index (tt) for functions of
3

---

Under review as a conference paper at ICLR 2023
noise to Xo, so is independent of the measurement y when conditioned on Xo. Therefore, can write:
/ p(xo x )
Pe(y)x;)= =
(3)
which involves marginalization over The likelihood of p(y[xo) is tractable, yet samples from p(xo|x²) can only be approximated by the diffusion model with high precision (using the variational inference argument by Sohl-Dickstein et al. (2015); Ho et al. (2020)); when & - T, sampling from p(xolx)) essentially becomes sampling from the entire diffusion model. Even using Monte Carlo methods, it is computationally infeasible to estimate (y (x)), let alone its score (where the Monte Carlo estimate will also be biased).
Our solution to this issue is to reasonable approximations to the true pt (xoo/xx)) such that the resulting approximation to the score Vx, log (r(y)x,) is casy to compute. Intuitively, instead of representing pxx) with the entire diffusion model from time to O, we use one-step denoising process. Specifically, we first approximate pa (xoo)xx) with the following Gaussian:
(x = (Re, (&R.r1I) r21),
(4)
where the mean is obtained from Tweedie's formula:
& Exxx x,] = X, + of Vx, log Rs X + So;
(5)
Eq. represents the minimum mean squared error (MMSE) estimator of x given x, and the noise standard deviation 0N (Stein, 1981; Efron, 2011; Saremi & Hyvärinen, 2019), and Ta is time- dependent standard deviation value that should depend on the data (see discussion in App. A.3). Our choice for the mean (MMSE) can be justified using an argument related to variational inference (App. A.6).
Our next step is to approximate the score of pt (y).). Since the measurement model obtains y by performing linear transform on Xo and adding independent Gaussian noise (Eq. 2), and pa (Xe (xx
Gaussian under our approximation (Eq. 4), the distribution of conditioned on is also Gaussian under our approximation, as follows:
pr(()x)) R N(H HH + +z
(6)
Thus, have the following approximation the score?:
Vx, logp:(y(xx)
(y H (ii + +025)
-1
H
Ox:
Ox
vector
Jacobian
This is vector-Jacobian product and can be computed with backpropagation.
3.2 EXTENDING TO NON-LINEAR OPERATORS
In many cases, we have that By = 0, and thus, Eq. 7 can be simplified to:
O&: T
Vx, lygx: = x 2 ((H'
H H'H
(8)
Ox
where for a matrix with linearly independent rows, = H is the Moore- Penrose pseudoinverse of H. In this paper, guidance method, which uses Eq. 8 for noiseless measurements and Eq. 7 for noisy linear ones.
the pseudoinverse guidance (IIG) to denote
Notably, we only need to perform automatic differentiation explicitly through the score model, but not through the computational graph with H or H (see Listing in App. A. This allows us to extend IIG to measurements that are not necessarily linear or even differentiable. We note that the matrix pseudoinverse satisfies H H Hx = Hx for all x E X. Analogously, for some non-linear measurement function 1 E. we may find another function ht R. - R" such that hhh" (h())) = h(() for all x E R.,, similar to Kawar et al. (2022b). Two examples are as follows:
We use the numerator layout, gradient is the transpose of derivative.
4

---

Under review as a conference paper at ICLR 2023
Quantization Let h(x)
[xx be the element wise floor function of x R". Then define (x) := x for all x Z, and h(h² (h(x))) is still the floor function.
JPEG encoding Let ((x) be the JPEG encoding function, where quantization occurs after a dis- crete cosine transform operation. The corresponding JPEG decoding algorithm does not modify the values produced after quantization, so we can simply define h (x) as the JPEG decoding algorithm.
This idea can also be applied to other measurement models, such as the formation of a low dynamic range image (details in App. A.5). The corresponding TIG term would then become:
V logg) rr ((h ht ()(
(9)
Ox.
which generalizes the linear case (Eq. 8) when l((x) Hx and h² hx)) = H' for all x IR".
3.3 ADAPTIVE WEIGITTS IN GUIDED DIFFUSION MODELS
Similar to the guidance scalar in the classifier free) guidance literature (Dhariwal Nichol, 2021; Ho & Salimans, 2022), we introduce a scalar weight in front of the guidance term Vx, log (y (x(x), However, unlike most existing methods that apply a fixed weight for different diffusion times, we introduce heuristic that implicitly adapts the guidance weights according to the timestep. We use fxx; s, t, 77) to denote the one step update using the problem-agnostic score model from time to times (assuming
t), using the sampler introduced in the DDIM paper (Song et al., 2021a), with y E [0, 1] being a hyperparameter (details in App. A.4). Our one-step sampling update from time to time with pseudoinverse guidance is:
X s, + +r Vx log log
(10)
If we use the noiseless case in Eq. 9, this becomes:
x, f(x;st,,,) + (1xl(r) - ht (b(x)))
O&.
(11)
0x;
We describe the algorithm in Algorithm I (App. A. As the coefficients for the problem-agnostic score Vx, log Pe (x,) depend on the step s, this is equivalent to using the original DDIM sampler but adapting the weights to the pseudoinverse guidance term at different timesteps. To illustrate this, we can compare the ratio between the weights of our approach and the ones with w, = Ho al. (2022) (see Fig. 6). Intuitively, our approach increases the weights during the initial sampling phase and then decreases it to one towards the end. We also compare our weights with the ones used in Ho et al. (2022) image restoration problems, both of which use the pseudoinverse guidance with 100 diffusion steps and 17 = 0.2. On the super-resolution case (Fig. 7), our weights consistently produce sharp images. We further illustrate the advantages of weights on JPEG restoration in App. A.4, where large, fixed weights that worked better super-resolution could be unstable in another task.
3.4 DIFFERENCES FROM EXISTING GUIDANCE METHODS
Table 1: Comparison of different guidance methods.
Guidance
Expression
X differentiable
Train on (x,,)
Noisy
Classifier
V x. log ((y()))) V x1 lyy H&,1l Eqs. 7 to
Required
Yes
Reconstruction
Required
No
No
Pseudoinverse
Not required
No
Yes
Our approach is notably different from prior guidance-based methods in terms of how the condi- tional score Vx, logp(y/(xx is approximated (see Tab. 1). Compared with classifier classifier free guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), we do not require training on pairs of y) (noisy data and measurements). Compared with reconstruction guidance (Ho et al., 2022; Chung et al., 2022b; Ryu & Ye, 2022), ITG has three advantages:
5

---

Under review as a conference paper at ICLR 2023
Our approximation of p(xx(xx) is consistent, i.e., it does not depend on the measurement model H. The same cannot be said for reconstruction guidance, which makes isotropic Gaussian assumptions on y (see App. A.2).
In reconstruction guidance, the pseudoinverse H is replaced with matrix transpose H (see App. A.2), which is different for linear H whose singular values are not all 0 1.
IIG can be applied to noisy, non-linear, or non-differentiable measurement models, as dis- cussed in Sec. 3.2. In cases like JPEG, it is easier define a generalized notion of pseu- doinverse than a generalized notion of transpose (or adjoint).
4 RELATED WORK
Deep neural networks have been extensively used as priors for solving inverse problems (Venkatakr- ishnan et al., 2013b). Here, we focus on the setting where we can train models based on clean data but not on the problem, which is only known at inference time. This is reasonable in many real-world applications, such as medical imaging (Jalal et al., 2021; Chung & Ye, 2022) and JPEG restoration (Ehrlich et al., 2020). These inverse problem solvers may use different types of neural networks, such as randomly initialized networks (Ulyanov et al., 2018), denoisers (Romano et al., 2016), robust classifiers (Santurkar et al., 2019), and generative models (Bora al., 2017). Methods based generative adversarial networks (GANs, Goodfellow et al. (2014)) search for the latent variables and/or the generator parameters that would produce images aligning with the measure- ments (Bora et al., 2017; Pan et al., 2021; Menon et al., 2020); these methods often require hundreds not thousands of iterations, despite recent methods with improved efficiency (Daras et al., 2022a).
As another family of generative models, diffusion models are also used as inverse problem solvers, with two notable advantages over GANs: (i) it is trained with regression objectives over noisy data, so it can naturally deal with measurement noise without having perform inversion like in GANs; (ii) its close connections to SDE/ODE solvers allow the use of more efficient iterative updates. In particular, Denoising Diffusion Restoration Models (DDRM, Kawar et al. (2022a)) leverage both to derive efficient inverse problem solvers for both noisy and noiseless measurements.
Similar to DDRM, many works adopt a "replacement" approach, where consistency with the mea- surements are enforced by replacing parts of its intermediate predictions from the one-step denoiser with the measurements, sometimes in a transformed space (Song et al., 2021c; Choi et al., 2021; Song et al., b021b; Chung et al., 2021; Kawar et al., 2021). Despite being successful in many tasks, they have trouble dealing with sparse measurements, where the replacements have weaker impact on the sampling process. By computing additional gradients through the diffusion model, IIG allows the measurements to impact all the predicted values during the updates, regardless of sparsity. This is similar to reconstruction guidance, which also differentiates through the diffusion model during its updates (Ho et al., 2022: Ryu & Ye, 2022; Chung et al., 2022b). In fact, IIG is identical in the noiseless, linear case if the transpose of the measurement matrix is equal to its pseudoinverse (App. A.2). Nevertheless, TIG introduces principled ways of dealing with noisy, non-linear, or even non-differentiable measurements, discussed in Sec. 3.4.
5 EXPERIMENTS
Our approach, named "xeudoinverse-guided Diffusion Models (IGDM)), combines IIG (Eqs. 7 to 9) and the adaptive weight schedule (Eq. 10). While use sampler based on DDIM here, note that other samplers can be used as well. We evaluate quantitative results on the ImageNet dataset (Russakovsky et al., 2015) with publicly available diffusion models trained on images of size 256 x 256, as there are extensive prior results with problem-specific diffusion models trained on ImageNet (Dhariwal & Nichol, 2021: Saharia et al., 2021: 2022a).
First, we compare LIGDM against problem-specific models on 4x super-resolution, in- painting, and JPEG restoration. Despite the "unfair" advantage held by problem-specific models, HIGM is on par with them in terms of performance.
Next, we perform an ablation study over the two components introduced in this paper.
Finally, we apply IIGMM to inverse problems where the measurement process is composed of several steps, such as JPEG + super-resolution inpainting, denoising + inpainting, etc..

---

Under review as a conference paper at ICLR 2023
JPEG (QF - 5)
Palette
rIGMM
Reference
Figure 3: Results on JPEG restoration. From left to right: the compressed JPEG image, restoration results from Palette (task-specific) and IIGDM (task-agnostic,, and the reference image.
The compositional nature of these problems makes it infeasible to train diffusion models for each problem and highlights the strength of LIGDM.
5.1 QUANTITATIVE RESULTS
We consider two popular metrics, Frechet Inception Distance (FID, (Heusel et al., 2017)) and Clas- sifier Accuracy (CA) of a pre-trained ResNet50 model (He et al., 2015). Unless specified otherwise, use the noiseless version for pseudoinverse guidance (Eq. 8). We report super-resolution results on the full ImageNet validation set, and to follow the earlier practice established in Saharia et al. (2022a), we report inpainting and JPEG restoration results a subset that contains 10k images?, The ImageNet models that we use are trained with 1000 discrete timesteps, corresponding to 1000 discrete noise levels (Dhariwal & Nichol, 2021). For simplicity, we always use uniform spacing when we iterate the timesteps. Performance may further improve with better timestep scheduling, such the one that iterates more frequeetly lower noise levels (Karras et al., 2022). We use 100 iterations and 1.0 for IIGMM, and include additional task-specific details in App. B.
5.1.1 CIPEE-REESOLUTION
We apply average pooling (Pool) and bicubic interpolation (Bicubic, which applies convolution to the image) to produce two sets of 64 x 64 images, and then apply our 4x super-resolution algo- rithms to each. We consider both class-conditional (denoted as in Tab. 2) and class-uncondition models as the base generative model. Tab. 2, we report results from IIGDM and three other baselines: DDRM (Kawar et al. (2022a)), SR3 (Saharia et al. (2021)), and ADM-U (Dhariwal & Nichol (2021)). DDRM uses task-agnostic models, whereas SR3 and ADM-U use diffusion models specifically for the 64 + 256 super-resolution problem. On Pool, HIGDM significantly outper- forms DDRM, while only being slightly worse than ADM-U; on Bicubic, IIGDM outperforms all three baselines. Perhaps surprisingly, the ADM-U model performs much worse in Bicubic than Pool because it was trained on low-resolution images generated by average pooling" (i.e., the Pool prob- lem); when Bicubic images are used, the generated results become more blurry. This suggests that problem-specific diffusion models may fail to generalize beyond settings that they are trained on.
5.1.2 INPAINTING
We use the two types of inpainting masks used in (Saharia et al., 2022a): the center 128 x 128 pixels (Center), and freeform masks simulating brushstrokes that contain roughly 20% 30% of the pixels in each image (Freeform). In addition. problem with i. Gaussian noise of oy = 0.05 (the pixel intensity range is [0. 1)): the problem becomes harder the model has to perform denoising and inpainting the same time. For the noisy
report IIGDM results over the noisy inpainting
"accpe//biit.ly//vaappp22.
7

---

Under review as a conference paper at ICLR 2023
Guiddance
Reference
Mask 0.1 Gaussian
Reconstruction Guidance
NGDM
Figure 4: Results noisy inpainting problems. Reconstruction guidance (third column) does not handle measurement noise and will keep the noisy measurements, so only the masked regions are denoised.
Table 2: super-resolution results. Dark-colored rows indicate methods using cooblem-specifiee models.
Table 3: Inpainting results. Dark-colored rows indi- cate methods using problem-specifie models.
Filter
Method
FID 1
Mask
Method
FID-10k
CA 1
ADM (ec, Dhariwal & Nichol (2021))
3.1
73.4%
DeepFillv2 (Yu al., 2019)
18.0
64.3%
D1RM (Kawar 20222)
14.8
Palette (Saharia et al., 2022a)
6,6
69.3%
Pool
HGDM (Ours)
3.8
72.3%
Center
DDRM (Kawar et al., 2022a)
24.4
62.1%
D1RM Kawar (20226))
HIGM (Ouns)
HGGD (ec, Own)
3.6
72.2%
HGDM (noisy. Ours)
9.5
72.2%
SR3 (Saharia et al., 2021)
5.2
68.3%
DeepPillv22 (Yu et al., 2019)
ADM (ce, Dhariwal & Nichol (20213)
14.8
66.7%
Bicubic
IIGDM (Ours)
DDRM ( (Kawar et al., 2022a)
21.3
3.6
63.2%
72.1%
Freeform
DDRM (Kawar et al., 2022a)
Palette (Saharia al., 2022a)
8.6
5.2
72.3%
71.9%
DDRM (ec, Kawar (2022a))
19.6
65.3%
IIGDM (Ours)
5.3
75.3%
TIGDM (re, Ours)
3.2
75.1%
TIGDM (noisy, Ours)
7.3
74.5%
setting, we use Eq. 7 for IIG. In Tab. 3, report quantitative results on the two inpainting tasks, mainly comparing with Palette (Saharia al., 2022a), which trains diffusion model specifically on the inpainting task. While IIGDM achieves slightly worse FID compared with Palette, it has higher classifier accuracy in both cases. Morcover, 11GDM suffers merely a small performance drop when applied to the challenging denoising + inpainting task, demonstrating its robustness to noisy measurements. Methods based on reconstruction guidance, however, fail to perform denoising effectively, there are mechanisms to address measurement noise (see Fig, 4).
5.1.3 JPEG RESTORATION
We consider the three JPEG quality factors (QFs) used in Saharia et al. (2022a), which are 5, 10, and 20. In Tab. 4, we report quantitative results on JPEG, where we compare against regression- based baseline and Palette, both of which are trained specifically for JPEG images with QFs ranging from 5 to 30. Compared with Palette, ITGDM achieves slightly worse FID (less than 0.6) but higher classifier accuracy QFs 10 and 20. We note that the model used in ITGDM has any JPEG images compressed to these quality factors, demonstrating the strength of task-agnostic diffusion models.
5.2 ABLATION STUDIES
IIGDM introduces two key components: pseudoinverse guidance (IIG) for problem-speciffe score estimation and the adaptive guidance weight schedule for sampling (AW). To illustrate their ef- fectiveness, compare with alternative approaches. The alternative to IIG is the reconstruction guidance, whereas the alternative to AW is the standard weight schedule set with w, E (1,2,5) 5) in Ho al. (2022) (the w, with best performance is reported).
8

---

Under review as a conference paper at ICLR 2023
Table 4: JPEG restoration results. corl-colored indicate methods using wabblem--peciffe models.
Table 5: Ablation studies pseudoinverse guidance (LIG) and adaptive weight schedule (AW).
QF Method
Regression (Saharia al., 2022a)
29.0
52.8%
5 Palette (Saharia al., 2022a)
8.3
64.2%
Deblar
Bieubic
IIGMM (Ours)
8.6
64.1%
IIG
AW
PSNR
KID-Ik x x10*
FID-10K
CA
Regression (Saharia et ol., 2022a)
18.0
63.5%
x
x
21.98
41.03
18.6
40.6%
Palette (Saharia al., 2022a)
70.7%
IIGDM (Ours)
6.0
71.0%
x
31.95
0.98
15.4
67.6%
Regression (Sahuria al., 2022a)
11.5
69.7%
20
Palette (Sahoria 2022a)
4.3
73.5%
TIGDM (Ours)
4.7
74.4%
We consider the uniform kernel deblurring (Deblur) and bicubic 4x super-resolution (Bicubic) tasks discussed in (Kawar et al., 2022a). The measurement matrix for both tasks have varying singular values (see Fig. 5 in App. A.2), so IIG and reconstruction guidance updates are quite different since HT * H From Tab. we can see that methods that use IIG achieves significant improvement over reconstruction guidance, and switching to the pseudoinverse in the guidance term is critical. TIG itself achieves superior performance with AW, illustrating the importance of having a good sampling algorithm along with the guidance term. We provide additional experimental details and further ablation studies on the number of iterations per sample and 1) in App. B.
5.3 INVERSE PROBLEMS WITH COMPOSED MEASUREMENTS
Finally, we discuss cases where the measurement process consists of several simpler measurements, leading to some applications such as JPEG restoration with super-resolution + inpainting, etc., where the compositional nature of the measurements makes it too expensive train problem-specifte dif- fusion models individually. Specifically, let hx) = hı h22 hą (x) be measurement model composed of i smaller measurements. For certain measurements, such as low-resolution filtering, JPEG, and masking. we can approximate h h'(x) with A o o h h (x). and then use IIGDM with Eq. 9 directly. We illustrate some examples in Fig. 2 and Fig. 13 (App. B.3). To the best of our knowledge, many of these problems have not been solved with problem-agnosti diffusion models before (such super-resolution JPEG + inpainting).
DISCUSSIONS, LIMITATIONS, AND FUTURE WORK
In this paper, we introduced IIGDM, inverse problem solver using unconditional diffusion mod- els. On various tasks, IIGDM achieves competitive quality with conditional models while avoiding expensive groblem-speciin training. As a result, we can use problem aggnosiic diffusion models to solve certain problems that would be cost-ineffeetive to address individually with conditional diffu- sion models. leading much wider set of applications. The ability to handle measurement noise also gives ITGDM the potential to address certain real-world applications, such as MRT imaging with Gaussian noise (Sijbers& & Den Dekker, 2004).
Despite having better restoration results than DDRM (Kawar et al., 2022a), IIGM is slower, as each iteration costs more memory and compute due to the vector-Jacobian product over the score model. Therefore, it would be helpful to explore more efficient sampling techniques. It is also interesting to investigate if similar ideas as IIG can be used for diffusion models that do not directly operate the data space (Vahdat et al., 2021; Rombach et al.. 2022: Sinha al., 2021), on alternative forward diffusion models (Jing et al., 2022; Rissanen et al., 2022; Daras et al., 2022b; Hoogeboom & Salimans, 2022).
based
Reproducibility statement We have made the following efforts to facilitate reproducibility of our work. (i) Our experiments are conducted on publicly available datasets and model checkpoints (Sec. 5). (ii) We include a detailed description of our algorithm in Algorithm 1. (iil) We discuss all the key hyperparameters and evaluation metrics to reproduce our experiments in Sec. 3.3 and App. B. (iv) We provide more explanations to some statements in the main paper in App. A.2 to A.5.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Applications, 12(3):313-326, 1982.
Stochastic Processes and their
Johnathan M Bardsley. Meme-based image reconstruction with uncertainty quantification. SIAM Journal Scientific Computing, 340)) 2012.
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 1801 .0..01111111 2018.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using genera- tive models. arXiv preprint 100..0:0003302000 March 2017.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. ILVR: Con- ditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02933 August 2021.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical Image Analysis, pp. 102479, 2022.
Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Dloss-CCo---iffueeeeFee Faster: Accelerating conditional diffusion models for inverse problems through stochastic contraction. arXiv preprint arXiv:21 December 2021.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare I. Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint (xXiv22209.16687 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. arXiv preprint arXiv:2206.00944) 2022b.
Giannis Daras, Joseph Dean, Ajil Jalal, and Alexandros G Dimakis. Intermediate layer optimization for inverse problems using deep generative models. arXiv preprint arXiv:2102.07364, February 2021.
Giannis Daras, Yuval Dagan, Alexandros G Dimakis, and Constantinos Daskalakis. Score-guided intermediate layer optimization: Fast langevin mixing for inverse problem. arXiv preprint arXiv:2206.09 009.22006,91,,, 2022a.
Giannis Daras, Mauricio Delbracio, Hossein Talebi, Alexandros G Dimakis, and Peyman Milanfar. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.05443 2022b.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. arXiv preprint arXiv:2105.05233, May 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496):10221161, 2011.
Max Ehrlich, Larry Davis, Ser-Nam Lim, and Abhinav Shrivastava. Quantization guided JPEG artifact correction. arXiv preprint arXiv:2004.09320, April 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Ulf Grenander and Michael Miller. Representations of knowledge in complex systems. Journal of the Royal Statistical Society: Series B (Methodological), 56(4):549-581, 1994.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. aix preprint arXiv: 15w.::11....0 1512003385 December 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two Time-Scale update rule converge to a local nash equilibrium. arXiv preprint aivi 7767085000 June 2017.
10

---

Under review as a conference paper at ICLR 2023
Jonathan Ho and Tim Salimans. arXiv:2207.12595 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusson probabilistic models. arXiv preprint arXiv:2006.1 11399, June 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 2022.
Emiel Hoogeboom and Tim Salimans. Blurring diffusion models. arXiv preprint arXiv:2209.05557, 2022.
Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jonathan I Tamir. Robust compressed sensing MRI with deep generative priors. arXiv preprint arXiv:2108.01 August 2021.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion gener- ative models. arXiv preprint arXiv:2205.01499 2022.
Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in denoiser. Advances in Neural Information Processing Systems, 34:13242- 13254, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. arXiv preprint arXiv:2206.000:: 2022.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. SNIPS: Solving noisy inverse problems stochastically. arXiv preprint arXiv:2105. 14951, May 2021.
Bahjat Kawar, Michael Flad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201. 1793, 2022a.
Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. arXiv preprint arXiv:2209. 11888, 2022b.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint wxXv:20009.09761 2020.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal Imaging Sciences, 15(2):701-737. 2022.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B Hashimoto. Diffusion-Im improves controllable text generation. arXiv preprint arXiv:2205.144: 14217, 2022.
Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. Single-image hdr reconstruction by learning to reverse the camera pipeline. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
1651-1660, 2020.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00927, 2022.
Chenlin Meng, Yang Song, Jiaming Song. Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Im- age synthesis and editing with stochastic differential equations. arXiv preprint 1=XXi:21088.1107. 2021.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. PULSE: Self- Supervised photo upsampling via latent space exploration generative models. arXiv preprint arXiv:2003.03888 March 2020.
Gregory Ongie, Ajil Jalal, Christopher A Metzler, Richard G Baraniuk, Alexandros G Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39-56, 2020.

---

Under review as a conference paper at ICLR 2023
Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. Exploiting deep generative prior for versatile image restoration and manipulation. IEEE transactions on pattern analysis and machine intelligence, PP, September 2021. ISSN 0162-8828. doi: 10. 215423.
Severi Rissanen, Markus Heinonen, and Arno Solin. Generative modelling with inverse heat dissi- pation, arXiv preprint arXiv:2206. 13397, 2022.
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (RED). arXiv preprint arXiv: 1.02622, November 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International journal of computer vision, 115 (3):211-2522 December 2015. ISSN 0920-5691, 1573-1405
doi: 10. 10.1007/11263-0115 2660001-066--1
Dohoon Ryu and Jong Chul Ye. Pyramidal denoising diffusion probabilistic models. arXiv preprint arXiv:2208.01 0164, 2022.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement, arXiv preprint arXiv:2104.06636 2021.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans. David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 2022b.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with single (robust) classifier. arXiv preprint arXiv: pew.:0906009559 June 2019.
Saced Saremi and Aapo Hyvärinen. Neural empirical bayes. Journal of machine learning research: JMLR, 20(181):1-22 2019. ISSN 1532-4435, 1533-7928.
Jan Sijbers and AJ Den Dekker. Maximum likelihood estimation of signal amplitude and noise variance from mr data. Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 51(3):586-594 2004.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding mod- els for few-shot conditional generation. Advances Neural Information Processing Systems, 34: 12533-12548, 2021.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. arXiv preprint arXiv: 150.0355,,, March 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
Yang Song. Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging 2021b.
with score-based generative models. arXiv preprint
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based basee generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021c.
12

---

The annals of Under review as a conference paper at ICLR 2023
Charles M Stein. Estimation of the mean of multivariate normal distribution. Statistics, pp. 1135-1151, 1981.
Yusuke Tashrro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffu- sion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804-24816 2021.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446 9454,, 2018.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. arXiv preprint arXiv:2106.05931, June 2021.
Singanallur Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 945-948. IEEE, 2013a.
Singanallur Venkatakrishnan, Charles Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEER Global Conference on Signal and Information Processing, pp. 945-948. IEEE, 2013b.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-167, July 2011.
Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Pey- man Milanfar. Deblurring via stochastic refinement. arXiv preprint arXiv:21 December 2021.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. LSUN: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arv: 100.00335, June 2015.
Jiahui Yu, Zhe Lin, Jimei Yang. Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4471-4480, 2019.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902 April 2022.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.0557 255664 June 2022.
13

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL METHOD DETAILS
A.I ALGORITIM DETAILS
We illustrate PyTorch-like implementation for computing the pseudoinverse for the noiseless case in Listing For a different inverse problem, we only need to change the definitions for functions H and H. Hinv. In practice, many diffusion model architectures adopt the Variance-Presenn SDE, which scales the signal Xo down as the noise level increases, specifically:
(VP)
Xo + orE (VE)
x; 0 + vI- (1e
(VP).
(12)
To adjust for this in IIGDM, we need to scale the guidance term by vor (Dhariwal & Nichol, 2021). We list the full algorithm for IIGDM for VP-SDE in Algorithm 1.
# "H_pinv"; (b, m) -> (b, n), "H": (b, n) -> (b, m) are functions over batches # "y" has shape (b, m); "x_t" has shape (b, n)
"hatx_t" with shape (b, n) is the solution from the denoiser hatx_t denoise (I_t, sigma_t)
Compute the fixed coefficient; "mat" has shape (b, n) mat (ynvv(y) H_pinv (H(naxx_t)
# Compute the inner product between thatx_t" and "mat", and then sum over batch mat_x (mat. () hatx_ (xx_tt)_ (etx_t).suuu sum)(())
Compute the guidance term (without r_t).
guidance torch. autograd, grad x_t) [o]
Listing 1:
Pseudocode for computing the pseudoinverse guidance for the noiseless case.
Algorithm LIGDM for VP-SDE.
Inputs: y. hx() (noiseless) or H. By (noisy), x,, 17 E [0, 1], e-prediction diffusion model. Find a sequence timesteps (v: ] where th = 0 and uN T. Initialize x (O, I).
for i = N, .... do
1 14 9/-1
Get start and end times for this iteration >> Get a in VP-SDE
a; - Tto7
Eo <predictionxx: t)
>> Predict the (standardized) noise
x, + x x-vT-mice
b Predict the one-step denoised result
Vas
1 ax
> Get coefficients C1 in DDIM
C2 1 Oss
2..
noiseless then
gt ((hly)) -h (h(x))))
0K, 8x
else
g ((y H&.) (HHT
H
8& 8x.
end if
~ (0,1)
Sample i.i.d. Gaussian H1GDM update, first three terms are simply DDIM. Additional Vaz in front of comes from VP-SDE
+ Vagit cIE 0260 Varg
end for
A.2 STATEMENTS ABOUT RECONSTRUCTION GUIDANCE
We note that in reconstruction guidance (Ho et al., 2022), the following approximation is made:
Pt (ylx+) N (H&,,
(RC)
(13)
14

---

Under review as a conference paper at ICLR 2023
where we omit the og term in (Ho et al., 2022) use the Variance Exploding (VE) parametriza- tion throughout the paper". While Ho et al. (2022) only listed super-resolution and impainting as two examples, the general idea be extended to any linear H.
Transpose, not pseudoinverse. This approximation would lead to the following score function:
Vx, x, log Pt
(RG)
xx)
of (HY H H&,)
0&L
(14)
ox,
which essentially replaces the pseudoinverse term H in Eq. 8 with the transpose term H the differences in the variance approximation). Taking the singular value decomposition over H = UUV, we have that:
(ignoring
H H (UEV) H' H = - (UEEV) - V'I
VVVV (UEV)
(15)
(16)
where I(E²) the diagonal matrix which take if the corresponding entry in Ed is non-zero, and otherwise. Multiplying the former with a vector x (as in reconstruction guidance) will scale the sin- gular vectors by X² where as multiplying the latter (as in pseudoinverse guidance) keeps the scale for all singular vectors that correspond to nonzero singular values. When the measurement ma- trix has very different singular values (see Fig. 5), reconstruction guidance may improperly rescale the singular vectors, leading to reduced performance compared with pseudoinverse guidance (as in Table 5).
Bicubic Downsampling
9x9 Uniform Blur
0.25
0.8
gingeriake
0.20
0.6
0.15
0.4
0.10
0.05
0.2
3.00
0.0
2000
4000
10000
12000
20000
60000 B2000 100000 120000
160000
Singuler valuee Index
Singuler value index
Figure 5: Singular values of the bicubic downsampling and uniform blurring measurement matrix.
Consistent approximation of xx)) In reconstruction guidance, the isotropic Gaussian approximation over the distribution (x)))x means that the approximation of the distribution Pr(xo]Xx, (x)) will depend on the measurement model. For example, suppose we have two diagonal measurement matrices D, and D2 with all positive values along the diagonal, and D K D2. If
we use D2 as the measurement, then X = D and pt (RG) (II) (xolxe) = N of o but
D2 the measurement, then Xe D and (RG) (xo|xx) R. NX of 2 which is different from the earlier approximation. However, conditioned on X,, the distribution of Xu can be inferred from the diffusion model alone, so should not depend on the measurement model. Therefore, reconstruction guidance does not make a consistent approximation of the distribution (x (x)); pseudoinverse guidance, on the other hand, approximates p. (Xplx;) directly from the diffusion model and then approximates Pt (y xx,) by marginalization of Gaussians.
A.3 ABOUT THE VARIANCE OF THE APPROXIMATION
Our approximation for Pr (Xol xx) depends the variance term Tı, which should depend the variance of the data distribution. For example, if the data distribution (x(x)) = A/ I) is the standard normal distribution, then from Bayes' rule have the following closed-form solution for
Neverthelesss VE is equivalent to the variance preserving (VP) parametrization up to time-dependent scaling factor (Song al., 2021a), any sampling algorithm for VE can be adapted VP.
15

---

Under review as a conference paper at ICLR 2023
10
2.30
1000 222
525
750
1032
250
3200
Figure 6: Left: de as a function of t. Right: the ratio between our guidance weights and the Video Diffusion Models (VDM) weight w, = 1 (Ho et al., 2022) under different r) values, We take 100 uniformly spaced timesteps (out of possible of 1000 timesteps).
Low ros (64x64)
Our weights
VDM weights (w-11 VDM weights (w--21
VDM wrights (M)-S)
Reference
Low ros (64464)
Our weights
VDM wrights (w:111
VDM wrights (w VDM wiights (e)-S)
Reference
Figure 7: Two examples that compare adaptive weight schedule with the different weights w, in Video Diffusion Models (VDM, (Ho et al., 2022)) on 4x super-resolution (Bicubic). For fair comparison, ITG is used for all cases.
the posterior:
(x) px(xo)p,(xx(xx)) N
of +1 of of
Xx
(17)
so in this case, we have ro 0 and = 1. In Ho et al. (2022), ri is set as dz; this is reasonable in T (noise level is much higher than the
when t (noise level is small), but unrealistic when data variance). Nevertheless, in the noiseless case, we do not have to make the value of explicit, as we can simply rescale the gradient terms using the guidance weights.
In the noisy case, the choice of rt matters more it interacts with σy in Eq. 7. We simply use
Tt
+1
from Eq. 17, which provides good empirical results for our noisy inpainting experiments.
To see why this is the case, let take denoising as an example where H = I. When < By small, then r2 R 0g and r (r? + + v of becomes small, meaning that the noisy mea- surement provides little impact to the guidance term. Whereas when o; is large, then r; and rr(? + +o2) --2=~= ze 1, meaning that the guidance term will be impacted by the noisy measurements. The sampling procedure would first guide the unconditional samples towards the noisy measure- ments, and then perform denoising without overfitting them.
A.4 ABOUT ADAPTIVE GUIDANCE WEIGHTS
The sampling updates for the original DDIM paper is derived from the VP-SDE, so we rewrite the updates in the form of VE-SDE used in this paper:
f(x; 8, (, X ms + oi o0z 1/ (x: xx),
(18)
Xx + of Vx, (x.) + me-sst 0r - -12 1222000 Vx, loggs(x)) (19)
16

---

Under review as a conference paper at ICLR 2023
Low res (64x64)
Our weights
VDM weights (W,=1) VDM weights (wy-=2) VDM weights (W,=5)
Reference
Low (64x64)
Our weights
VDM weights (w,-1) VDM weights
VDM weights (w,-5)
Reference
Figure 8: Two examples that compare our adaptive weight schedule with the different weights wo in Video Diffusion Models (VDM, (Ho et al., 2022)) on JPEG (QF=10) restoration.
where NV, I) and
(o o
Ct
(20)
corresponds to the a1 coefficient in the original DDIM sampler (using or (VE) instead of ag (VP)
formuation). Note that & = x, of Vx. log (xx) is the denoised result.
In Ho et al. (2022), the guidance is applied X, for some constant Wyr such that?:
(RG) x + of Vx, (x. Wr Vx, ly H&
X
(21)
Using the DDIM sampler, the update for the next sample replaces R; in Eq. with x (RG), which further multiples a factor to the guidance term. In case, we directly add the guidance term to Eq. 19, which is more similar to the approach in Chung et al. (2022b). This would equivalent to the weights in Ho et al. (2022) if wy is different for different I, i.e., applying time-dependent guidance weights during sampling (hence being "adaptive"). While it is possible to tune w, to achieve decent results, we found that different tasks may require different wy For example, w, works well for super-resolution, but suffers from numerical overflow issues JPEG restoration (see Fig. 8).
A.5 ABOUT THE LOW DYNAMIC RANGE MEASUREMENT FUNCTION
Let h(x) be a function that reduces the dynamic range of an image. Typically, this consists of dynamic range clipping stage h1 non-linear mapping stage h2, and quantization stage h& (Liu et al., 2020). The non-linear mapping is also known as the camera response function, and it is fair to assume it is invertible (its inverse denoted as h)).
The dynamic range clipping function typically consists of the form hy(x) clip(æ, a, where and are lower and upper clipping ranges; we can define hf as follows:
y if E [a
k((x) = a if < a otherwise
(22)
Therefore, we can define h+((x) = (n) (x))) for pseudoinverse guidance. For ease of illustra- tion, assume h2 to be the identity function in qualitative results, and focus the clipping and quantization functions: in these cases, we clip images of range [-1, 1] to [-0.6, 0.6). and then quantize 8 bit representations into bits.
A Aaai,, the ae is missing because we VE instead of VP.
17

---

Under review as a conference paper at ICLR 2023
A.6 JUSTIFYING OUR APPROXIMATION
As discussed in Sec. 3.1. it is computationally infeasible to sample from more exact representations of p( (r)) (i.e, the diffusion model), so we need to approximate it. straightforward way is to approximate via variational inference: instead of the multi-step diffusion process, we use a simple Gaussian to approximate it. Let us denote the Gaussian as ((xo|xx); we can minimize the KI. divergence between q(xo) (x)) and p(xo (x), which gives us the following objective function:
min Fx(x))pxx, xx) (T. (xxxxxxx)))llxxxxx)x))) xx)]] min Ppxx x.) [log p (Xo, xx) log ((xo [x)]]
If we define (x,o|x;) as Gaussian with a fixed standard deviation and mean as function μ of X then the objective is equivalent to:
min Ex x (xx) - x 112).
(23)
which is exactly the denoising score matching / denoising autoencoder objective (Vincent, 201 I). Therefore, we can use the single step denoiser result as the mean of qf g(xo|xx xx))
Nevertheless, one might be interested in how "tight" is our approximation. While it intractable to compare with ground truth p(y|x, or even p(xo|xx), it is not hard to compare the score functions of p(xo|>x) xx) and our approximation g(xo xx) )8. In fact, denoting the denoiser as D, we have that:
Xx log xx) = Vx, log pl X xo) Vx, log ((x)
(24)
= (x0 x /o (D(x)) xi)/o? = - D(xi))/o)
(25)
and (overloading "gradient" notation for derivatives)
Xx, log xxx) x
D(x:)).
(26)
Therefore, the ground truth score is proportional (xo Dxx:)) whereas our score is proportional to [vx, Dxx))] [Xx D. The two terms are different by left matrix multiply of the gradient V., v,, (z:)). In the literature of plug-and-play methods, a reasonable assumption for the denoiser is
that it can be represented as a pseudolinear filter over the input (see Romano et al. (2016) for detailed explanations), so the gradient behaves roughly like a matrix. This suggests that our approximation is reasonably close, at least when the above score functions are concerned.
B EXPERIMENTAL DETAILS
B.1 ADDITIONAL EXPERIMENTAL SETUPS FOR QUANTITATIVE RESULTS
4x super-resolution Following CCDF (Chung et al., 2021) and SDEdit (Meng et al., 2021), we initialize our sampler with smaller noise level than the maximum to the linearly upsampled image (of size 256 x 256), which is chosen to be the one at the 500-th discrete timestep (where the model is trained with a total of 1000 discrete timesteps). Here, choose 100 iterations and = 1.0 for ITGDM. FID is evaluated over the restoration results on the entire ImageNet validation set, and compared against the statistics of the ImageNet training set.
by adding Gaussian noise
The baselines are run as follows. For DDRM (Kawar et al. (2022a)), we used the default hyper- parameter settings. For SR3 (Saharia et al. (2021)), we reported the official results from the paper. For ADM-U (Dhariwal & Nichol (2021)), we used their publicly available 64 + 256 ImageNet checkpoint and run 100 iterations for each image with the default command.
Inpainting For IGDM, we use a caass-oonditioall model, initialize our sampler from pure Gaus- sian noise at the maximum noise level ØT, apply 100 iterations to each image, and set 1/ 1.0. Following Saharia et al. (2022a), we evaluate FID over a 10k subset of the ImageNet validation set, and compare against the statisties of the ImageNet validation set.
'We note that tractable score functions does not imply tractable likelihood, as the latter requires the partition function, which itself requires marginalization step.
18

---

Under review as a conference paper at ICLR 2023
Table 6: 4x super-resolution results (Pool) from HIGDM using the class-unconditional model.
FID 1
CA f
Steps
20
50
100
20
50
100
1
0.0
6.5
4.4
4.3
70.0
70.3
69.2
0.5
7.2
4.5
3.9
70.0
71.4
70.6
1.0
10.9
6.1
3.8
68.2
71.3
72.3
Table 4x super-resolution results (Bicubic) from ITGDM using the class-conditional model.
FID 4
CA f
Steps
20
50
100
20
50
100
1
0.0
7.5
4.1
3.9
73.0
74.0
72.6
0.5
8.0
4.2
3.4
72.7
74.6
73.7
1.0
10.8
5.5
3.2
70.8
74.2
75.1
JPEG Restoration For each quality factor, we use the quantization matrix in Ehrlich et al. (2020) to compress the original 256 256 image, with x chroma subsampling. The quantization matrix is embedded in every JPEG file, so having this available to the algorithm is a natural and realistic setting. For IIGDM, we use a cnassuunconiiiinall model, initialize our sampler from pure Gaussian noise the maximum timestep, apply 100 iterations each image, and set 1.0. The FID is evaluated as in the inpainting case, following Saharia et al. (2022a).
B.2 ADDITIONAL ABLATION STUDIES AND DETAILS
Uniform deblurring We use the uniform 9 x 9 deblurring kernel used in Kawar et al. (2022a). The problem itself is relatively simple it has few non-zero singular values, and simply taking the pseudoinverse over the observations already gives good results. For all methods, we use a class- unconditional model, initialize our sampler from the 100-th discrete timestep using the CCDF / SDEdit approach. We use a total of 20 iterations for each image and 0.5 for the 1000 images used in the evaluation set of DGP DDRM (Pan et al., 2021; Kawar al., 2022a). We compare PSNR metrics with images scaled to [0, 1]. and Kernel Inception Distance (KID, (Binkowski et al., 2018)) metrics against the 1000 reference images, following the practice Kawar al. (2022a).
4x super-resolution The experiment setup is identical to the quantitative evaluation case, except that we evaluate the metrics over the 10k subset from Saharia et al. (2022a). We use a total of 100 iterations for each images and y) 1.0 for the 10000 images.
Ablation study over and number of iterations We report additional results over the hyper- parameters in the DDIM sampler, which are 77 (the amount of noise injected each step) and the number of iterations (steps) per image on super-resolution tasks. We consider the Pool and Bicubic 4x ruserttesolution task over the entire ImageNet validation set. From the results in Tabs. 6 and 7, we can draw similar conclusions as the ones from the DDIM paper (Song et al., 2021a): more iter- ations generally lead to improved performance, whereas the effect of 7) varies. When the number of iterations is small, smaller is better (as injects less noise in the process); when the opposite is true, larger 7) is better (due to the sampling process being more robust to errors in the score function).
B.3 ADDITIONAL FIGURES
We list additional qualitative results Figs. to 15, All the results with IIGM and reconstruction guidance are generated with 100 steps and 7) 1.0. We w, for reconstruction guidance.
19

---

Under review as a conference paper at ICLR 2023
Pool
Reconstruction guidance
ADM-U
ΠGDM
Reference
CEXO
Figure 9: Comparing methods for the Pool 4x super-resolution problem, including reconstruction
guidance (Ho et al., 2022), ADM-U (Dhariwal & Nichol, 2021), and IIGDM. Best viewed zoomed in.
20

---

Under review as a conference paper at ICLR 2023
Bicubic
Reconstruction guidance
ADM-U
ΠGDM
Reference
CEXO
Figure 10: Comparing methods for the Bicubic 4x super-resolution problem, including reconstruc- tion guidance (Ho et al., 2022), ADM-U (Dhariwal Nichol, 2021), and IIGM.. Best viewed zoomed in.
21

---

Under review as a conference paper at ICLR 2023
Masked
NGDM [Seed=1)
(Seed=2)
RIDM (Seed=3)
ΠIDMM (Seed=4)
IIGMM (Seed=5)
Figure 11: Inpainting results using IIGDM for the Freeform problem with multiple random samples.
22

---

Under review as a conference paper at ICLR 2023
JPEG (QF 5)
Palette
ΠGDM
Reference
Figure 12: Results for the JPEG restoration problem, including Palette (Saharia et al., 2022a) and HIGDM. Palette results are obtained from the official website.
23

---

Under review as a conference paper at ICLR 2023
Reference
Mask 0.1 Goussian
JPEG (QF-10) Mask
Low-res JPEG (QF-20)
LDR Quantization
Restored
Restored
Restored
Restored
JPEG (0h-10)
(0F-20)
Quantictionn
Reference
Mask 0.1 Causslan
JPEG (0F++0) Mask
Low-res +JEG (QF-220)
LDR Quantitation
Restored
Restored
Restored
Restored
Reference
Mack D.I. Gausslan
JPEG (QF-10) + Mask
Low ros JPEG (QF-20)
LDR Quantiatioon
Restored
Restored
Restared
Restored
Figure 13: Restoration results with IIGDM over various composed measurements. The same ran- dom seed is used for different problems.
24

---

Under review as a conference paper at ICLR 2023
Masked
NGDM (Seed=1)
NGDDM (Seed=2)
RGDM (Seed=3)
RIDM (Seed=4)
IGDM (Seed=5)
Figure 14: Inpainting results with IIGDM for the Center problem on the LSUN Bedroom dataset (Yu
al., 2015).
25

---

Under review as a conference paper at ICLR 2023
Reference
Low-res (4x)
RIDMM (4x)
Low-res (8x)
RGDM (8x)
Figure 15: Super-resolution results with IIGM for the Pool case on the LSUN Bedroom dataset (Yu
et al., 2015).
26

---

Under review as a conference paper at ICLR 2023
Table 8: NFEs of various algorithms. We list common baselines, such as Palette (Saharia et al., 2022a), ADM-U (Dhariwal & Nichol, 2021), SR3 (Saharia et al., 2021), DGP (Pan et al., 2021), SNIPS (Kawar et al., 2021), RED (Romano et al., 2016), DDRM (Kawar al., 2022a), and
DPS (Chung et al., 2022a).
ITGM (Ours)
Palette
Regression
ADM-U
SR3
DPS
DGP
SNIPS
RED
DDRM
NFEs
20 to 100
1000
100
1000
1000
1500
1000
500
20
Table 9: 4x super-resolution (Pool) and deblurring results on ImageNet 1K (256 x 256).
4x super-resolution (Pool)
Deblurring KID
Method
SSIM
KID
NFEs|
PSNR
NFEs
Regression
0.71
44.90
0
19.26
38.00
0
DGP
0.56
21.22
1500
22.70
27.60
1500
RED
0.73
53.55
100
26.16
21.21
500
SNIPS
0.22
35.17
1000
34.32
0.49
1000
DDRM
0.72
7.22
20
35.64
0.71
20
ITGDM (Ours)
0.73
1.24
100
39.36
0.00
20
B.4 RUNTIME DIFFERENT ALGORITHMS
Since the algorithm requires iterations with the diffusion model, the actual runtime of the algorithm would depend heavily on the number Neural Function Evaluations (NFEs). Kawar al. Kawar et al. (2022a) found that the runtime for diffusion models would dominate the total runtime, other computations (such as matrix operations on images) are negligible. Therefore, we use NFE as the unit for estimating the runtime of different algorithms. In Tab. 8, report the NFEs used by each algorithm.
We further note that IIGDM and DPS take additional backpropagation steps through the neural network, so each NFE roughly 3x as expensive as others. Thus, ITGDM is only beaten in terms of actual wall-clock time by regression and DDRM, and we have shown that it has superior restoration results than both in Tabs. to 4.
B.5 COMPARISON WITH VARIOUS BASELINES
In Tab. 9, we compare TIGDM against various baselines, including ones that are based on Plug- and-play (PnP) methods (Venkatakrishnan et al., 2013b), such as Deep Generative Prior (Pan et al. (2021)), Regularizing by Denoising (RED, Romano et al. (2016)), Solving Noisy Inverse Prob- lems Stochastically (SNIPS, Kawar et al. (2021)), and Denoising Diffusion Restoration Models (DDRM, Kawar et al. (2022a)). Similar to the setting for deblurring in Tab. 5, the comparison is done the 1000 validation examples listed in the DGP paper (Pan et al., 2021), and the KID computed with the 1000 ground truth images as the reference set.
We find that TGDDM achieves the best performance when compared with other PnP baselines; this is reasonable given that DDRM (Kawar et al., 2022a) shown to outperform the remaining competi- tors, and LIGDM outperforms DDRM from the results in
B.6 COMPARISON WITH DIFFUSION POSTERIOR SAMPLING
Diffusion Posterior Sampling (DPS, (Chung et al., 2022a)) is concurrent method that similarly uses a gradient-based guidance method. They approximate p(y|x,) with p(y[xo)) x which is similar to reconstruction guidance.
Findings. We investigate the performance of DPS over several tasks and over different learning rates and diffusion step hyperparameters. Similar what Chung et al. has found (Chung et al.,
27

---

Under review as a conference paper at ICLR 2023
Table 10: 4x super-resolution (Pool) comparisons with DPS, under different steps.
Steps
20
50
100
200
500
1000
DPS
0,559
0.511
0.414
0.335
0.215
0.162
LPIPS
IIGDM
0.164
0.140
0.140
DPS
0,504
0.580
0.634
0.696
0.756
0.778
SSIM
IIDM
0.779
0.772
0.777
Table 11: 4x super-resolution (Bicubic) comparisons with DPS, under different steps.
Steps
20
50
100
200
500
1000
DPS
0,558
0.510
0,410
0.329
0.221
0.171
LPIPS
IIDM
0.163
0.166
0.122
DPS
0,504
0.581
0.637
0.697
0.757
0.779
SSIM
IIGDM
0.789
0.730
0.773
2022a), we find that while DPS has strong performance with 1000 diffusion steps, its performance becomes much worse with less diffusion steps. Moreover, even with the full 1000 steps, one would still need to tune for the learning rate hyperparameter to get good performance. In contrast, our method is 10x faster than DPS in the slowest case, and achieves decent performance even with fewer diffusion steps (such 20 and 50 steps). In these settings, DPS does not produce reasonable results at all.
Image restoration quality. In our experiments, we evaluate performance of the models averaged over 5 validation images on ImageNet. For DPS, we consider different numbers of diffusion steps (from 20 to 1000), with the default learning rate being the one chosen in the DPS paper for 1000 steps. For IIGDM, we use the same settings as discussed in the paper; we report for diffusion steps up to 100 steps, except for deblurring (where the task is simple enough to get good results in 20 steps).
We report LPIPS and SSIM metrics for Pool, Bicubic, and 9 x 9 uniform deblurring in Tabs. 10 to 12. From the tables, it is obvious that DPS performance drops rapidly once number of diffusion steps decreases under 1000, whereas IIGMM performance remain competitive. We illustrate this trend visually in Fig. 16 for 4x super-resolution (Pool). In Tab. 13, we report the results for DPS under different learning rate hyperparameters:
found that the optimal hyperparameter can be problem-dependent: the optimal one for super-resolution is around 1 and 2, whereas using that for deblurring will produce NaNs; the optimal one for deblurring is 0.2, where super-resolution results tend to become less optimal.
Loss curves. We can treat the guidance terms in both DPS and IIGDM gradient step that optimizes the least squares loss function lyy Hx it is natural to visualize the loss each diffusion noise level. In Fig. 17, we visualize the loss curve of DPS and IIGDM on 4x super- resolution (Pool) example as function of the diffusion timestep level (so 1000 is highest noise
Table 12: 9 uniform deblurring comparisons with DPS, under different steps.
Steps
20
50
100
200
500
1000
DPS
0.539
0.472
0.412
0.340
0.260
0.245
LPIPS
IIGDM
0.004
DPS
0.516
0.575
0.604
0.662
0.698
0.719
SSIM
TGDDM
0.974
28

---

Under review as a conference paper at ICLR 2023
Table 13: Performance of DPS under different learning rates and 1000 diffusion steps.
Task
Super-resolution (Pool)
Deblurring
Learning rate
0.5
1.0
2.0
0.2
0.4
0.6
LPIPS
0.208
0.171
0.182
0.238
0.245
0.322
SSIM
0.755
0.779
0.775
0.700
0.719
0.608
DPS (20 steps)
DPS (50 steps)
DPS (100 steps)
DPS (200 steps)
DPS (500
DPS (1000 stepsl
RGDM (20 steps)
MGDM (50 steps)
RGDM (100 steps)
Reference
steps)
steps)
(100 steps)
steps)
DPS steps)
(1000 steps)
NGDM (20
M.DM steps)
NGDM (100 steps)
Reference
Figure 16: Super-resolution results with DPS (Chung al., 2022a) and IIGDM for the 4x super- resolution (Pool) under different number of diffusion steps.
29

---

Under review as a conference paper at ICLR 2023
1v Hxll
10*
DPS
IXGMM (100 steps)
(50 steps)
101
MGDM sleps)
10
10
1000
800
600
400
200
Diffusion limestep
Figure 17: Loss curves for lly Hx 12
using DPS (Chung et al., 2022a) and ITGDM for 4x
super-resolution (Pool) example.
level and 0 is lowest noise level). As expected, hoth methods start with the same loss. However, the loss function of DPS significantly increases around timesteps 1000 and 900, reaching 1000 of the initial loss; this means that the DPS learning rate schedule is too large at this initial stage. In contrast, HIGDM has a smooth loss curve over the entire process; in fact, the loss curves are quite consistent under 20, 50, 100 steps. Moreover, the final loss for IIGDM also smaller than that of DPS, further illustrating its superiority.
30


Reviews:
[{"review_id": "Z537TmgmGV", "reviewer": "Reviewer_JCDz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes Pseudoinverse-Guided Diffusion Models, which uses problem-agnostic diffusion models to reach the empirical performance of problem-specific ones. The proposed approach directly estimates conditional scores without additional training. In particular, it can address inverse problems with noisy and non-linear measurements, in contrast to existing approaches that are limited to noiseless linear ones. The authors illustrate the empirical performance of the proposed model on different image restoration tasks.  \n", "summary_of_the_review": "The work overall is interesting and well presented.", "strength_and_weaknesses": "Strength: (1) The work is overall well-written and joyful to read. The contributions of this work are clearly stated. (2) In general, the idea of this work is natural and interesting. The authors provide enough material to support their idea. \n(3) The experiments are solid and complete, including super-resolution, inpainting, and JPEG restoration. The experimental results are encouraging and well support the effectiveness of the proposed approach. \n\nWeakness: (1) It is not straightforward for the readers to know why the authors can approximate the score function via Eq. (3), Eq. (5), and Eq. (6). The authors should provide more details and insights.\n(2)  In Eq. (6), the authors derive the approximation of the score function. A natural question is how tight this approximation is. It seems that there are no guarantees. \n(3) It seems that this method is a heuristic approach. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Yp-VNMZX4gc", "reviewer": "Reviewer_v3Ma", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a pseudoinverse-guided diffusion model for solving (nonlinear) inverse problems using model-agnostic pre-trained score-based diffusion model (SDM) networks. The key component of the proposed in $\\Pi$GDM is the approximation of the conditional probability $p_t(y|x_t)$, which is based on Tweedie's formula and Gaussian assumption. With known $p_t(y|x_t)$, $\\Pi$GDM can the term as guidance to guide the generation of the image.\n\nIn summary, I think the contributions of the work are:\n1. Propose an approach to approximately calculate $p_t(y|x_t)$.\n2. Demonstrate the efficacy of the model on various (nonlinear) inverse problems, including JPEG restoration, image impainting, and image super-resolution.\n\n", "summary_of_the_review": "I have some concerns about the technical correctness of the proposed method. I am looking forward to the authors' clarification.", "strength_and_weaknesses": "**Strength:**\n1. Open code and data\n2. Thorough supplement\n3. Proposal of a new approach to compute $p_t(y|x_t)$\n\n**Weakness:**\n\n*Statement:*\n1. The statement \"To the best of our knowledge, ΠGDM is the first approach based on problem-agnostic models to achieve this quality on ImageNet.\" might be too ambitious. I found a concurrent paper (https://arxiv.org/abs/2209.14687) that also uses model-agnostic DPM for solving the general inverse problem and achieves very good results. A thorough comparison with existing methods needs to be presented to support this argument.\n2. I am not very clear what are the contributions of this work by reading the manuscript. I felt that using 'guidance' to condition DPM has already become a common strategy and many papers have investigated this problem. Perhaps it is due to my misunderstanding, but the text seems to imply that making model-agnostic pre-trained DPM work for specific inverse problems is one novelty/contribution tied to the paper.\n\n*Methodology:*\n\n3. The underlying assumption for ΠGDM to approximate $p_t(y|x_t)$ is that $p_t(x_0|x_t)$ is Gaussian, which is strong. For example, assume $p_t(x_t|x_0)$ is Gaussian and $p(x_0)$ is Gaussian as well. According to Baye's rule, \n$$p(x_0|x_t) = \\frac{p_t(x_t|x_0)p(x_0)}{p(x_t)}$$\nfor which a Gaussian approximation is too strong.\n\n4. Therefore, the practicality of the model might be severely limited.\n\n*Experiment:*\n\n5. In the experiments, the baseline methods are restricted to model-specific methods, failing to include other model-agnostic methods such as plug-and-play prior (PnP). It will be also helpful to see if ΠGDM improves over existing model-agnostic methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "C1ml8Sqtuj", "reviewer": "Reviewer_diJL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Pseudoinverse-guided Diffusion Models ($\\Pi$GDM), that use pretrained diffusion models (problem-agnostic) to solve inverse problems. The proposed $\\Pi$GDM works with noisy, non-linear, or even non-differentiable measurements, and demonstrates promising results.", "summary_of_the_review": "Good paper.", "strength_and_weaknesses": "## strengths\n- The paper is well-written and well-organized.\n- The problem of using pre-trained diffusion model to solve inverse problems is of great interest to the community. The paper demonstrates promising results and the discussion of related works is comprehensive.\n- The proposed pseudoinverse guidance has nice properties that can work with noisy, non-linear, or even non-differentiable measurements.\n## weakness and questions\n- Minor issues:\n\t- Table 1 is not referred to in the main text.\n- Questions:\n\t- Moore-Penrose: is there a specific reason to employ Moore-Penrose Pseudoinverse here? Is it possible to consider another more general pseudoinverse?\n\t- Adaptive weights: how does the adaptive weight compare to the weight used in [1], $1/||y-H \\hat{x}_t||$?\n\t- Non-linear extension: the paper discussed two examples of non-linear H, is it possible to extend it further to neural networks like a contrastive encoder, a classifier, or an autoencoder?\n\n[1] Chung, Hyungjin, Jeongsol Kim, Michael T. Mccann, Marc L. Klasky, and Jong Chul Ye. 2022. “Diffusion Posterior Sampling for General Noisy Inverse Problems.” arXiv [stat.ML]. arXiv. http://arxiv.org/abs/2209.14687.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9mCbMA66q3", "reviewer": "Reviewer_4ai2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduced a new approach called Pseudoinverse-guided Diffusion Models to deal with problem agnostic image inverse problems. Based on the recently popular diffusion models, it added the Pseudoinverse guidance by using a problem agnostic model S_theta(x;sigma_t) to approximate the score. Then the authors shows for 4 inverse problems their method performed on-par with the problem-specific method", "summary_of_the_review": "Not knowledgeable enough about the recent publications in the area, but the paper is novel and technically solid for me. Therefore, I recommend \"accept.\"", "strength_and_weaknesses": "Strength:\n\nThe paper is well written with a good story flow, making it much easier to follow for the audience.\nThe method is well motivated by the lack of problem-agnostic inverse methods and provided their solution by approximating the guidance term using the Pseudoinverse of the measurement matrix.\nThe ablation study of swapping the Psy-G with AW is solid evidence of Psy-G contributing to the performance robustness.\n\nWeakness:\n1. Although the problem motivation is clear, to me it has some problems: The solution is not really \"problem agnostic.\" The Pseudoinverse guidance involves informing the algorithm of the measurement function H or h(x). Is it difficult to come up with such a matrix for different inverse problems?\n\n2. How is the time performance (both training and testing) of such an algorithm compared to the baselines?\n\n3. How many times are the experiments repeated? How many test samples are combined to report the FID scores in Table 2&3. Is there a standard deviation to be reported?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster