A Restoration Network as an Implicit Prior
Authors: Yuyang Hu
Mauricio Delbracio
Peyman Milanfar
Ulugbek Kamilov
Venue: ICLR 2024 poster (2024)
Keywords: computational imaging
inverse problems
deep learning
plug-and-play priors
Primary area: optimization
TL;DR: A new method and theory for using deep restoration networks as implicit priors for solving inverse problems.

Abstract: Image denoisers have been shown to be powerful priors for solving inverse problems in imaging. In this work, we introduce a generalization of these methods that allows any image restoration network to be used as an implicit prior. The proposed method uses priors specified by deep neural networks pre-trained as general restoration operators. The method provides a principled approach for adapting state-of-the-art restoration models for other inverse problems. Our theoretical result analyzes its convergence to a stationary point of a global functional associated with the restoration operator. Numerical results show that the method using a super-resolution prior achieves state-of-the-art performance both quantitatively and qualitatively. Overall, this work offers a step forward for solving inverse problems by enabling the use of powerful pre-trained restoration models as priors.

Published as a conference paper at ICLR 2024
A RESTORATION NETWORK AS AN IMPLICIT PRIOR
Yuyang Hu', Mauricio Delbracio", Peyman Milanfar Ulugbek S. Kamiloy! Washington University in St. Louis, "Google Research
[h. yuyang, kamilow momilov]ed @wwsstl.... @ (mdelbras mi milanfar @google. com
edu,
ABSTRACT
Image denoisers have been shown to be powerful priors for solving inverse prob- lems in imaging. In this work, we introduce a generalization of these methods that allows any image restoration network to be used as an implicit prior. The pro- posed method uses priors specified by deep neural networks pre-trained as gen- eral restoration operators. The method provides a principled approach for adapt- ing state-of ofate-of-the-art are restoration models for other inverse problems. Our theoretical result analyzes its convergence to a stationary point of global functional associ- ated with the restoration operator. Numerical results show that the method using a super-resolution prior achieves state-of-the-art performance both quantitatively and qualitatively. Overall, this work offers step forward for solving inverse prob- lems by enabling the use of powerful pre-trained restoration models as priors.
INTRODUCTION
Many problems in computational imaging, biomedical imaging, and computer vision can be formu- lated as inverse problems, where the goal is to recover a high-quality images from its low-quality observations. Imaging inverse problems generally ill-posed, thus necessitating the use of prior models on the unknown images for accurate inference. While the literature on prior modeling of images vast, current methods are primarily based on deep learning (DL), where deep model is trained to map observations to images (Lucas et al., 2018; McCann et al., 2017; Ongie et al., 2020).
Image denoisers have become popular for specifying image priors for solving inverse prob- lems (Venkatakrishnan et al., 2013; Romano et al., 2017; Kadkhodaic & Simoncelli, 2021; Kamilov al., 2023). Pre-trained denoisers provide convenient proxy for image priors that does not require the description of the full density of natural images. The combination of state-of-the-an (SOTA) deep denoisers with measurement models has been shown to be effective in a number of inverse problems, including image super-resolution, deblurring, inpainting, microscopy, and medical imag- ing (Metzler et al., 2018: Zhang et al., 2017b; Meinhardt et al., 2017: Dong et al., 2019; Zhang et al., 2019; Wei et al., 2020; Zhang et al., 2022) (see also the recent reviews (Ahmad et al., 2020; Kamilov et al., 2023)). This success has led to active research on novel methods based on denoiser priors, their theoretical analyses, statistical interpretations, such as score matching and diffusion models (Chan et al., 2017: Romano et al., 2017: Buzzard et al., 2018; Reehorst & Schniter, 2019; Sun et al., 2019; Sun al., 2019; Ryu et al., 2019; Xu al., 2020; Liu et al., 2021; Cohen et al., 202 la; Hurault et al., 2022a;b; Laumont et al., 2022; Gan et al., 2023).
related approaches well as connections
Despite the rich literature on the topic, the prior work has narrowly focused on leveraging the sta- tistical properties of denoisers. There is little work on extending the formalism and theory to pri- ors specified using other types of image restoration operators, such as, for example, deep image super-resolution models. Such extensions would enable new algorithms that can leverage SOTA pre-trained restoration networks for solving other inverse problems. this paper, we address this gap by developing the Deep Restoration Priors (DRP) methodology that provides a principled ap- proach for using restoration operators as priors. We show that when the restoration operator is a minimum mean-squared error (MMSE) estimator, DRP can be interpreted as minimizing a compos- ite objective function that includes log of the density of the degraded image as the regularizer. Our interpretation extends the recent formalism based on using MMSE denoisers as priors (Bigdeli et al., 2017: Xu et al., 2020; Kadkhodaie & Simoncelli, 2021; 1; Laumont et al., 2022; Gan et al., 2023). We present theoretical convergence analysis of DRP stationary point of the objective function

---

Published as a conference paper at ICLR 2024
under set of clearly specified assumptions. We show the practical relevance of DRP by solving several inverse problems by using a super-resolution network as a prior. Our numerical results show the potential of DRP to adapt the super-resolution model to act an effective prior that can outper- form image denoisers. This work thus addresses a gap in the current literature by providing new principled framework for using pre-trained restoration models priors for inverse problems.
All proof' and some details that have been omitted for space appear in the supplementary material.
BACKGROUND
Inverse Problems. Many imaging problems can be formulated as inverse problems that seek to recover unknown image x E IR" from from its corrupted observation
Ax + e,
(1)
where A
| measurement operator and e E Tn is the noise. A common strategy for addressing inverse problems involves formulating them as an optimization problem
arg min (ax) with [(x) = g(x) + h(x));
#ER"
where g is the data-fidelity term that measures the fidelity to the observation y and h is the regular- izer that incorporates prior knowledge on æ. For example, common functionals in imaging inverse problems are the least-squares data-fide fidelity term g(x)= = 3 Ax: y and the total variation (TV) regularizer h(x:) = Da I1 where D is the image gradient, and > a regularization parameter.
Deep Learning. DL is extensively used for solving imaging inverse problems (McCann et al., 2017; Lucas et al., 2018; Ongie et al., 2020). Instead explicitly defining regularizer, DL methods often train convolutional neural networks (CNNs) to map the observations to the desired images (Wang et al., 2016; Jin et al., 2017: Kang et al., 2017; Chen et al., 2017; Delbracio et al., 2021: Delbracio & Milanfar, 2023). Model-based DL (MBDL) is widely-used sub-family of DL algorithmss that integrate physical measurement models with priors specified using CNNs (see reviews (Ongie et al., 2020; Monga al., 2021)). The literature of MBDL is vast, but some well-known examples include plug-and-play priors (PnP), regularization by denoising (RED). deep unfolding (DU), compressed sensing using generative models (CSGM), and deep equilibrium models (DEQ) (Bora et al., 2017; Romano et al., 7; Zhang & Ghanem, 2018; Hauptmann et al., 2018: Gilton et al., 2021; ;; Liu et al., 2022). These approaches come with different trade-offs in terms of imaging performance, compu- tational and memory complexity, flexibility, need for supervision, and theoretical understanding.
Denoisers as Priors. PnP (Venkatakrishnan al., 2013; Srechari et al., 2016) is one of the most popular MBDL approaches for inverse problems based on using deep denoisers as imaging priors (see recent reviews (Ahmad et al., 2020; Kamilov et al., 2023)). For example, the proximal-gradicn method variant of PnP can be written as (Hurault et al., 2022a)
x + ( with
k-1 yr (x - (=(a=11))
(3)
where D2 is denoiser with a parameter 0 for controlling its strength, >> 0 is regularization parameter, and > O is a step-size. The theoretical convergence of PnP methods has been estab- lished for convex functions g using monotone operator theory (Srechari et al., 2016; Sun al., 2019; Ryu et al., 2019), as well as for nonconvex functions based on interpreting the denoiser as MMSE estimator (Xu et al., 2020) ensuring that the term (I-D,) in (3) corresponds to gradient / of a function h parameterized by a deep neural network (Hurault et al., 2022a;b; Cohen et al., 2021a). Many variants of PnP have been developed over the past few years (Romano et al., 2017: Metzler et al., 2018; 8: Zhang et al., 201 7b; Meinhardt et al., 2017; Dong et al., 2019; Zhang et al., 2019; Wei et al., 2020), which has motivated an extensive research on its theoretical properties (Chan et al., 2017: Buzzard et al.. 2018; Ryu et al., 2019; Sun et al., 2019; Tirer & Giryes, 2019; Teodoro al., 2019; Xu et al., 2020; Sun et al., 2021; Cohen et al., 2021b; Hurault et al., 2022a; Laumont et al., 2022: Hurault al., 2022b; Gan et al., 2023).
This work is most related to two recent inspired methods using restoration operators instead of denoisers (Zhang al., 2019; Liu et al., 2020). Deep hlug-and-play super-resolution (DPSR) (Zhang et al., 2019) was proposed to perform image super-resolution under arbitrary blur kernels by using bicubic super-resolver prior. Regularization by artifact removal (RARE) (Liu al., 2020)
2

---

Published as a conference paper at ICLR 2024
was proposed to CNNs pre-trained directly on subsampled and noisy Fourier data as priors for magnetic resonance imaging (MRI). These prior methods did not leverage statistical interpretations of the restoration operators provide theoretical analysis for the corresponding PnP variants.
also worth highlighting the work of Gribonval and colleagues on theoretically exploring the relationship between MMSE restoration operators and proximal operators (Gribonval, 201 I; Gri- bonval & Machart, 2013; Gribonval & Nikolova, 2021). Some of the observations and intuition in that prior line of work is useful for the theoretical analysis of the proposed DRP methodology.
Our contribution. (1) Our first contribution the new method DRP for solving inverse problems using the prior implicit in pre-trained deep restoration network. Our method major extension of recent methods (Bigdeli et al., 2017: Xu et al., 2020; Kadkhodaic & Simoncelli, 2021; Gan et al., 2023) from denoisers to more general restoration operators. Our second contribution new theory that characterizes the solution and convergence of DRP under priors associated with the MMSE restoration operators. Our theory is general in the sense that it allows for nonsmooth data- fidelity terms and expansive restoration models. (3) Our third contribution is the implementation of DRP using the popular SwinIR (Liang et al., 2021) super-resolution model as prior for two distinct inverse problems, namely deblurring and super-resolution. We publicly share our implementation that shows the potential of using restoration models to achieve SOTA performance.
3 DEEP RESTORATION PRIOR
Image denoisers are currently extensively used as priors for solving inverse problems. We extend this approach by proposing the following method that uses more general restoration operator.
Algorithm Deep Restoration Priors (DRP)
1: input: Initial value x E IR" and parameters 7, >
2: for k 1. do
3: z² t
TTG(===1) where G(x) = x - -R(Hz)
4: x + sprox.. (z*)
5: end for
The prior in Algorithm is implemented in Line 3 using a deep model RP > IR" pre-trained to solve the following restoration problem
Hæ + n with ~~ n
Pæi
o²),
(4)
where H E RPX= is degradation operator, such as blur or downscaling, and n E Rº is the additive white Gaussian noise (AWGN) of variance a². The density Pa is the prior distribution of the desired class of images. Note that the restoration problem (4) is only used for training R and doesn't 't have to correspond the inverse problem (1) we are seeking to solve. When H the restoration operator R reduces an AWGN denoiser used in the traditional PnP methods (Romano al., 2017; Kadkhodaie & Simoncelli, 2021; Hurault et al., 2022a). The goal of DRP is to leverage a pre-trained restoration network R to gain access to the prior.
The measurement consistency is implemented in Line using the scaled proximal operator
{ |a = =|| + 7g(())
sprox..g (z) = prox H (2) = arg min
(5)
where ||и]| = D H Ho denotes the weighted Fuclidean seminorm of vector v. When H' is positive definite and g is convex, the functional being minimized in (5) strictly convex, which directly implics that the solution is unique. On the other hand, when g is not convex H'H is positive semidefinite, there might be multiple solutions and the scaled proximal operator simply returns one of the solutions. It is also worth noting that (5) has efficient solution when is the least-squares data- data-fidelity term (see for example the discussion in (Kamilov et al., 2023) on efficient implementations of proximal operators of squares)
The fixed points of Algorithm I can be characterized for subdifferentiable g (see Chapter 3 in (Beck, 2017) for discussion subdiiferentiabilltt)). When DRP converges, it converges vectors a:
3

---

Published as a conference paper at ICLR 2024
R" that satisfy (see formal analysis in Supplement A.1)
0 E 0g(=") 7 (HHH"
(6)
where Ög is the subdifferential of g and is defined in Line 3 of Algorithm As discussed in the next section, under additional assumptions, one can associate the fixed points of DRP with the stationary points of a composite objective function g + h for some regularizer h.
CONVERGENCE ANALYSIS OF DRP
In this section, we present theoretical analysis of DRP. We first provide more insightful interpre- tation of its solutions for restoration models that compute MMSE estimators of (4). We then discuss
the convergence the iterates generated by DRP. Our analysis will require several assumptions.
We will consider restoration models that perform MMSE estimation of æ E IR" for the problem (4)
>Pp]s(<;; 8) dæ /
Pale [ (8; x ())
R(s) = E [z]s]]
dæ.
(7)
Ps(s)
where we used the probability density of the observation E IRP
(s(s) = Pa| Pa|m (x) dx = Go Hx (x) dæ.
(8)
The function G2 in (8) denotes the Gaussian density function with the standard deviation o > Assumption 1. The prior density Pa is non-degenerate over E"
For example, probability density Pr. is degenerate over R", if it supported on space of lower di- mensions than n. Our goal is to establish an explicit link between the MMSE restoration operator (7) and the following regularizer
h(x)) = - log (Hæ)) 2 R",
where is the parameter in Algorithm 1, Pa is the density of the observation (8), and o² is the AWGN level used for training the restoration network. We adopt Assumption to have more intu- itive mathematical exposition, but one can in principle generalize the link between MMSE operators and regularization beyond non- deggeegeee priors (Gribonval & Machart, 2013). It is also worth ob- serving that the function h is infinitely continuously differentiable, since it is obtained by integrating Pa with a Gaussian density G2 (Gribonval, 2011; Gribonval & Machart, 2013).
Assumption 2.
The scaled proximal operator sprox 1g is well-defined in the sense that there exists solution to the problem for any z E R The function is subdifferentiable over R....
This mild assumption is needed for us to be able to run our method. There are multiple ways to ensure that the scaled proximal operator is well defined. For example, sprox.g is always well- defined for any that is proper, closed, and convex (Parikh & Boyd, 2014). DRP applicable with the popular least-squares data-fidelity term g(x) b y Azl13 One can relax the assumption of convexity by considering g that is proper, closed, and coercive, in which case sprox,g will have a solution (see for example Chapter 6 of (Beck, 2017)). Note that do require the solution to (5) to be unique: it is sufficient for sprox.. to return one of the solutions.
directly makes
We now ready theoretically characterize the solutions of DRP.
Theorem 1. Let R be the MMSE restoration operator (7) corresponding to the restoration prob- lem (4) under Assumptions 1-3, Then, any fixed-point x" E TR of DRP satisfies
O E 0g(x=) + Vh(x"),
where h is given (9).
The proof of the theorem is provided in the supplement and generalizes the well-known Tweedie 's formula (Robbins, 1956; Miyasawa, 1961; Gribonval, 2011) to restoration operators. The theorem implies that the solutions of DRP satisfy the first-order conditions for the objective function f f= g+ h. If is negative log- ligeliieldoood Pyja: then the fixed- a-posteriori probability (MAP) solutions corresponding to the prior density Ps- The density ps is
fixed-points points DRP can be interpreted as maximum-

---

Published as a conference paper at ICLR 2024
related to the true prior Pa through eq. (8), which implies that DRP has access to the prior Pa through the restoration operator R via density Pa- As + and o - 0, the density ps approaches the prior distribution Pæ.
The convergence analysis of DRP will require additional assumptions.
Assumption 3. The data-fidelity term g and the implicit regularizer h are bounded from below.
This assumption implies that there exists f > -oo such that f(x) 2 f for all E R"
Assumption 4. The function h has Lipschitz continutous gradient with constant L > 0. The degradation operator associated with the restoration network such that X > H 0.
This assumption is related to the implicit prior associated with restoration model and is needed to ensure the monotonic reduction of the objective f by the DRP iterates. As stated under eq. (9), the function h is infinitely continuously differentiable. We additionally adopt the standard optimization assumption that V Lipschitz continuous (Nesterov, 2004). It is also worth noting that the positive definiteness of H H in Assumption is relaxation of the traditional PnP assumption that the prior is a denoiser, which makes our theoretical analysis significant extension of the prior work (Bigdeli al., 2017; Xu al., 2020; Kadkhodaie & Simoncelli, 2021: Gan et al., 2023).
We are now ready to state the following results.
Theorem 2. Run DRP for for i 2 iterations under Assumptions 1-4 using a step-size 7 = p/ (aLb with >> 1. Then, for each iteration 1 S k t. there exists w w(x") ) Of such that
C(f(x") -
(==x"") (12
6
where C' > 0 is an iteration independent constant.
The exact expression for the constant C' given in the proof. Theorem shows that the iterates generated by DRP satisfy w(x*) 0 as / + cc. Theorems 1 and 2 do not explicitly require convexity or smoothness of 9. and non-expansiveness generalization of the existing theory from denoisers to more general restoration operators.
R. They can thus be viewed as a major
5 NUMERICAL RESULTS
We now numerically validate DRP several distinct inverse problems. Due to space limitations in the main paper, we have included several additional numerical results in the supplementary material.
We consider two inverse problems of form Ax ee (a) Image Deblurring and (b) Single Image Super Resolution (SISR). For both problems, we assume that e is the additive white Gaussian noise (AWGN). We adopt the traditional fo-norm loss the data-fidelity term in (2) for both problems. We use the Peak Signal-to-Noise Ratio (PSNR) for quantitative performance evaluation.
In the main manuscript, we compare DRP with several variants of denoiser-based methods, in- cluding SD-RED (Romano et al., 2017), PnP-ADMM (Chan et al., 2017), IRCNN (Zhang et al., 2017b), and DPIR (Zhang al., 2022). SD-RED and PnP-ADMM refer to the steepest-descent variant of RED and the ADMM variant of PnP, both of which incorporate AWGN denoisers based on DnCNN (Zhang et al., 2017a). IRCNN and DPIR based half-quadratic splitting (HQS) iterations that use the IRCNN and the DRUNet denoisers, respectively.
In the supplement, we present several additional comparisons, namely: (a) evaluation of the perfor- mance of DRP on the task of image denoising; (b) additional comparison of DRP with the recent provably convergent variant of PnP called gradient-step plug-and-play (GS-PnP) (Hurault et al., 2022a); (c) comparison of DRP with the diffusion posterior sampling (DPS) (Chung et al., 2023) method that uses a denoising diffusion model as a prior; (d) illustration of the improvement of DRP using SwinIR as a prior over the direct application of SwinIR SR using the Gaussian kernel; (e) presentation of quantitative results using SSIM and LPIPS metrics; (f) additional comparison with DPIR using with SwinlR trained as denoiser; (g) additional evaluation of DRP with non- MMSE restoration priors and other MMSE restoration priors; and (h) evaluation of robustness of numerical results to random seed.
5

---

Published as a conference paper at ICLR 2024
10"33
(b)
10
(6)
30
27.05
20.21
MNS 46
2,008 10
10
1,558 100
Figure I: Illustration of the convergence behaviour of DRP for image deblurring and single image super resolution on the Set3c dataset. (a)-(b): Deblurring with Gaussian blur kernels of standard deviations 1.6 and 2.0. (c)-(d): 2x and 3x super resolution with the Gaussian blur kernel standard deviation 2.0. Average distance | x 1 and PSNR relative to the groundtruth are plotted, with shaded areas indicating the standard deviation of these metrics across all test images.
Kernel
Datascts
SD-RED
PnP-ADMM
IRCNN+
DPIR
DRP
Set3c
27.14/0.121
29.11/0.099
28. 18/14/0.886
29.53/0.110
30.69/0.056
Sct5
29.78/0.166
32.31/0.137
29.46/0.130
32.38/0.152
32.79/0.109
CBSD68
25.78/0.344
28.90/0.287
26.86/0.307
28.86/0.307
29.10/0.246
McMaster
29.69/0.167
32.20/0.
29.15/0.153
32.42/0.151
32.79/0.107
Set3c
25.83/0.166 165
27.05/0.157
26.58/0.155
27.52/0.159
27.89/0.085
Sct5
18.13/0.193
30.77/0.181
28.75/0.182
30.94/0.188
31.04/0.143
CBSD68
24.43/0.455
27.45/0.360
25.97/0.416
27.52/0.376
27.46/0.296
McMaster
28.71/0.201
30.50/0.185
28.27/0.222
30.78/0.193
30.79/0.141
Table I: PSNRT/LPIPSJ, comparison of DRP and several SOTA methods for solving inverse prob- lems using denoisers on image deblurring with the Gaussian blur kernels of standard deviation 1.6 and 2.0 on Set3c, Set5, CBSD68 and McMaster datasets. The best and second best results are highlighted. Note how DRP can outperform SOTA PnP methods that use denoisers as priors.
5.1 SWIN TRANSFORMER BASED SUPER RESOLUTION PRIOR
Super Resolution Network Architecture, We pre-trained a gx super resolution model Rg using the SwinIR (Liang et al., 2021) architecture based on Swin Transformer. Our training dataset comprised both the DIV2K (Agustsson
Timofte, 2017) and Flick2K (Lim al., 2017) dataset, containing 3450 color images in total. During training, we applied qx bicubic downsampling to the input images with AWGN characterized by standard deviation a randomly chosen in [O, 10/255). We used three SwinIR SR models, each trained for different down-sampling factors: 2x, 3x and 4x.
Prior Refinement Strategy for the Super Resolution prior. Theorem 1 suggests that as - I, the prior in DRP converges to Pz- This process can be approximated for SwinIR by controlling the down-sampling factor of the SR restoration prior (.).) We observed through our numerical ex- periments that gradual reduction of g leads to less reconstruction artifacts and enhanced fine details. We will denote the approach of gradually reducing as prior refinement strategy. We initially set q larger down-sampling factor, which acts smaller value leading to preservation of finer details. This strategy is conceptually analogous to the gradual reduction of in the denoiser in the SOTA PnP methods such as DPIR (Zhang al., 2022).
aggressive prior; then reduce
5.2 IMAGE DEBLURRING
Image deblurring is based on the degradation operator of the form A = K, where K is convolution with the blur kernel k. We consider image deblurring using two 25 x 25 Gaussian kernels (with the standard deviations 1.6 and 2) used in (Zhang et al., 2019), and the AWGN vector corresponding to noise level of 2.55/255. For fair comparison, we use the official implementations provided by each baseline method and use the methods. The restoration model used as a prior in DRP is SwinIR introduced in Section 5.1, so that the operation H corresponds to the standard bicubic downsampling. The scaled proximal operator
random seed to ensure consistency of random noise for all

---

Published as a conference paper at ICLR 2024
Blurry Image
PnP-ADMM
IRCNN+
DPIR
DRP (Ours)
Ground Truth
28.11
26.07
28.54
28.72
PSNR
Blurry Image
PnP-ADMM
IRCNN+
DRP (Ours)
Ground Truth
28.43
27.91
28.60
29.05
PSNR
Figure 2: Visual comparison of DRP with several well-known methods on Gaussian deblurring of color images. The top row shows results for a blur kernel with a standard deviation (std) of 1.6, while the bottom left corner of blurry images show the blur kernels. Each image is labeled by its PSNR in dB with respect the original image. The visual differences highlighted in the bottom-right corner. Note how DRP using restoration prior improves over SOTA methods based on denoiser priors.
shows results for another blur kernel with std = 2. The squares the bottom-
PSNR
q 3 (28.12 dB)
(30.28 dB) 2 2and 3 (30.58 dB) Ground Truth
31
2and
21
Figure 3: Illustration of the impact of different SR factors in the prior used within DRP for image deblurring. We show three scenarios: (i) using only 3x prior, (ii) using only 2x prior, and (iii) the use of the prior refinement strategy, which combines both the 2x and 3x priors. Left: Convergence of PSNR against the iteration number for all three configurations. Right: Visual illustration of the final image for each setting. The visual difference is highlighted by the red arrow in the zoom-in box. Note how the reduction of can lead to about 0.3 dB improvement in the final performance.
sprox in (5) with data-fidelity term g(x)= 11y
K K can be written as
(2) = (K K + HH [K + H
(10)
We adopt standard approach of using few iterations of the conjugate gradient (CG) method (see for example (Aggarwal et al., 2019)) to implement the scaled proximal operator (10) by avoiding the direct inversion of (KT K + (HHHH H). As bicubic SR model is adopted as prior, the R(Hæ) Step 3 of Algorithm 1 performs a bicubic downsampling of the intermediate image x k-1 and inputs it into the bicubic SR SwinIR. In each DRP iteration, we run three steps of CG solver, starting from warm initialization from the previous DRP iteration. We fine-turned the hyper-pare parameter y, 7 and SR restoration prior rate q to achieve the highest PSNR value on the Set5 dataset and then apply the same configuration to the other three datasets.
Figure (a)-(b) illustrates the convergence behaviour of DRP on the Set3c dataset for two blur ker- nels. Table 1 presents the quantitative evaluation of the reconstruction performance two different blur kernels, showing that DRP outperforms the baseline methods across four widely-used datasets. Figure 2 visually illustrates the reconstructed results the same two blur kernels. Note how DRP can reconstruct the fine details of the tiger and starfish, as highlighted within the zoom-in boxes, while all the other baseline methods yield either oversmoothed reconstructions noticeable arti- facts. These results show that DRP can leverage SwinIR as an implicit prior, which not only ensures stable convergence, but also leads to competitive performance when compared denoisers priors.

---

Published as a conference paper at ICLR 2024
Kernel
Datasets
SD-RED
PnP-ADMM
IRCNN+
DPIR
DRP
Set3c
27.01/0.147
27.88/0.131
27.48/0.113
28.18/0.137
29.26/0.079
Set5
28.98/0.207
31.41/0.171
29.47/0.194
31.42/0.175
31.47/0.120
CBSD68
26.11/0.426
28. 28.00/0.338
26.66/0.413
27.97/0.356
28.12/0.288
McMaster
28.70/0.257
30.98/0.28 208
29.11/0.237
31.16/0.179
31.39/0.122
2x
Set3e
25.20/0.191
23.86/0.187
23.92/0.171
26.8010.171
27.41/0.107
Set5
28.57/0.246
30.06/0.205
28.91/0.241
30.36/0.241
30.42/0.163
CBSD68
25.77/0.529
26.88/0.393
26.06/0.510
26.98/0.401
26.98/0.327
McMaster
28.15/0.303
29.53/0.219
28.41/0020 29.
29.87/0.215
30.03/0.167
Sct3e
25.50/0.177
25.85/0.177
25.72/0. 158
26.64/0.16
27.77/0.090
SetS
28.75/0.220
30.09/0.15
29. 19/0.4/
30.39/0.202
30.83/0.149
CBSD68
25.69/0.475
26.78/0.378
26.01/0.460
26.80/0.394
27.18/0.299
McMaster
28.38/0.269
29.52/0 208
28.53/0.28
29.82/0.210
29.92/0.164
3x
Sct3c
24.55/0.220
24.87/0.216
24.87/0. 196
25.84/0.195
26.84/0.108
Set5
28. 28.19/0.280
29. 29/2.69
28.37/0.265
29.70/0.225
29.8%/0.162
CBSD68
25.40/0.556
26.28/0.420
25.56/0.530
26.39/0.422
26.60/0.324
McMaster
27.79/0.329
28. 28.72/0.246
27.85/0.320
29.11/0.236
29.47/0.167
Table 2: PSNR1/LPIPS., comparison of DRP and several baselines for SISR on Set3c, Set5, CBSD68 and McMaster datasets. The best and second best results are highlighted. Note the ex- cellent quantitative performance of DRP, which suggests the potential of using general restoration models as priors.
Figure illustrates the impact of the prior-refinement strategy described in Section 5.1. We compare three settings: (i) use of only 3x prior. (ii) use of only 2x prior, and (iii) use of the prior-refinement strategy to leverage both 3x and 2x priors. The subfigure on the left shows the convergence of DRP for each configuration, while the ones on the right show the final imaging quality. Note how the reduction of q leads to better performance, which is analogous to what was observed with the reduction of a in the SOTA PnP methods (Zhang al., 2022).
5.3 SINGLE IMAGE SUPER RESOLUTION
We apply DRP using the bicubic SwinIR prior to Single Image Super Resolution (SISR) task. The measurement operator in SISR can be written as A SK, where K is convolution with the blur kernel and performs standard d-fold nown-sampling with df = n/ The scaled proximal operator sprox>g in (5) with data-fidelity term g(x) = 1/ yy SK x 2 can be write as:
sprox (z) (K ST SK + H H S + +o + Hz).
(11)
where H is the bicubic downsampling operator. Similarly to deblurring in Section 5.2, we use CG to efficiently compute (11). We adjust the q for the best PSNR performance on Set5, and then use these parameters on the remaining datasets.
т. and the SR restoration prior factor
We evaluate super-resolution performance across two 25 x 25 Gaussian blur kernels, each with distinct standard deviations (1.6 and 2.0), and for two distinct downsampling factors (2x and 3x), incorporating an AWGN vector corresponding to noise level of 2.55/255. For fair comparison, we use the official implementations provided by each baseline method and use the same random seed to ensure consistency of random noise for all methods.
Figure (c)-(d) illustrates the convergence behaviour of DRP on the Set3e dataset for and 3x SISR. Figure 4 shows the visual reconstruction results for the same downsampling factors. Table summarizes the PSNR values achieved by DRP relative to other baseline methods when applied to different blur kernel and downsampling factors four commonly used datasets.
worth highlighting that the SwinIR model used in DRP was pre-trained for the bicubic super- resolution task. Consequently, the direct application of the pre-trained SwinIR to the setting consid- ered in this section leads to the suboptimal performance due to mismatch between the kernels used. See Supplement B.4 to how DRP improves the direct application of SwinIR.
8

---

Published as a conference paper at ICLR 2024
2x LR Image
PnP-ADMM
IRONN+
DPIR
DRP (Ours)
Ground Truth
27.30
96
27.10
96
27.47
28.03
PSNR
3x LR Image
PnP-ADMM
IRONN+
DPIR
DRP (Ours)
Ground Truth
25 25.69
25.75
26.46
PSNR
Figure 4: Visual comparison of DRP and several well-known methods on single image super resolu- tion. The top row displays performances for 2x SR, while the bottom row showcases results for 3x SR. The lower-left corner of each low-resolution image shows the blur kernels. Each image is la- beled by its PSNR in dB with respect the original image. The visual differences are highlighted by the boxes in the bottom-right corner. Note the excellent performance of the proposed DRP method using the SwinIR prior both visually and in terms of PSNR.
6 CONCLUSION
The work presented in this paper proposes a new DRP method for solving imaging inverse problems by using pre-trained restoration operators as priors, presents its theoretical analysis in terms of con- vergence, and applies the method to two well- inverse problems. The proposed method and its theoretical analysis extend the recent work using denoisers as priors by considering more general restoration operators. The numerical validation of DRP shows the improvements due to the use of learned SOTA super-resolution models. One conclusion of this work the potential effectiveness of going beyond priors specified by traditional denoisers.
LIMITATIONS AND FUTURE WORK
The work presented in this paper comes with several limitations. The proposed DRP method uses pre-trained restoration models as priors, which means that its performance is inherently limited by the quality of the pre-trained model. As shown in this paper, pre-trained restoration models provide convenient, principled, and flexible mechanism to specify priors; yet, they are inherently self- supervised and their empirical performance can thus be suboptimal compared to priors trained in supervised fashion for a specific inverse problem. Our theory based on the assumption that the restoration prior used for inference performs MMSE estimation. While this assumption is reasonable for deep networks trained using the MSE loss, is not directly applicable to denoisers trained using other common loss functions, such the f--norm or SSIM. Finally, is common with most theoretical work, our theoretical conclusions only hold when our assumptions are satisfied, which might limit their applicability in certain settings.
Our work opens several interesting directions for future research. First, the implicit prior in 9 can be seen an analysis prior with transform H (Elad al., 2007: Selesnick & Figueiredo, 2009), which suggests possibility of considering broader class of linear transforms for priors. Second, the excellent performance of the restoration priors beyond denoisers lead to interesting open question: what is the optimal linear transform H for a given measurement operator A. Third, while in this paper considered non-blind inverse problems, the extension of DRP blind inverse problems would be an interesting future direction of research (Gan et al., 2023).
REPRODUCIBILITY STATEMENT
We have provided the anonymous source code in the supplementary materials. The included README.md file contains detailed instructions how to run the code and reproduce the results reported in the paper. The psdudo-code of DRP is outlined in Algorithm The complete proofs and technical details for our theoretical analysis be found in the supplement.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
To the best of our knowledge this work does not give rise to any significant ethical concerns.
REFERENCES
H. K. Aggarwal, M. P. Mani, and M. Jacob. Modl: Model-based deep learning architecture for inverse problems. IEEE Trans. Med. Imag., 38(2):394 Feb. 2019. ISSN 1558-254X. doi: 10.1109/MM.2018222653
E. Agustsson and Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proc. IEEE Conf. Comput. Vis. and Pattern Recognit. (CVPR) workshops, July 2017.
R. Ahmad, C. A. Bouman, G. T. Buzzard, S. Chan, S. Liu, E. T. Reehorst, and P. Schniter. Plug- and-play methods for magnetic resonance imaging: Using denoisers for image recovery. IEEE Sig. Process. Mag., 37(1):105-116, 2020.
H. Attouch, J. Bolte, and B. F. Svaiter. Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backen backward splitting, and regularized Kauss-Seides methods. Math. Program. Ser. A, 137(1):91-129) 2013.
A. Beck. First-Order Methods in Optimization. MOS-SIAM Series on Optimization. SIAM, 2017.
A. Beck and M. Teboulle. A fast iterative shrinkage- thressoogggt algorithm for linear inverse prob- lems. SIAM J. Imaging Sciences, 2(1):183-20) 202, 2009.
A. Bigdeli, M. Zwicker, Favaro, and M. Jin. Deep mean-shift priors for image restoration. Proc. Adv. Neural Inf. Process. Syst., 30, 2017.
A. Bora, A. Jalal, E. Price, and Dimakis. Compressed sensing using generative priors. In Int. Conf. Mach. Learn., pp. 537-546, Sydney, Australia, August 2017.
G. Buzzard, S. H. Chan. S. Srechari, and C. A. Bouman. Plug-and- and-play unplugged: Optimization free reconstruction using consensus equilibrium. SIAM J. Imaging Sci., 11(3):2001-2200, Sep. 2018.
S. H. Chan, X. Wang, and O. A. Elgendy. Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Trans. Comp. Imag. 3(1):84-98, Mar. 2017.
H. Chen, Y. Zhang, M. K. Kalra, F. Lin, Y. Chen, Liao, Zhou, and G. Wang. Low-dose CT with a residual encoder-decoder convolutional neural network. IEEE Trans. Med. Imag., 36(12): 2535,, Dec. 2017.
H. Chung, J. Kim, M. T. Mccann, M. L. K., and J. C. Ye. Diffusson posterior sampling for general noisy inverse problems. In Int. Conf. on Learn. Represent., 2023. URL https: //openreiiew wwt./ee///orue forum? i 2--.nD92GAATOO
R. Cohen, Y. Blau, D. Freedman, and Rivlin. It has potential: Gradient-driven denoisers for convergent solutions to inverse problems. In Proc. Adv. Neural Inf. Process. Syst. 34, 202
R. Cohen, M. Elad, and P. Milanfar. Regularization by denoising via fixed-point projection (red-pro). SIAM Journal on Imaging Sciences, 14(3): 2021b.
M. Delbracio and Milanfar. Inversion by direct iteration: An alternative to denoising diffusion for image restoration. Trans. on Mach. Learn. Research, 2023. ISSN 2835-8856. URL https: oopnneeviiec ew net. forum? (inniiddvvyFFFF L3F. Featured Certification.
M. Delbracio, H. Talebei, and P. Milanfar. Projected distribution loss for image enhancement. In 2021 Int. Conf. on Comput. Photography (ICCP), pp. 1-12. IEEE, 2021.
W. Dong, P. Wang, W. Yin, G. Shi, E. Wu, and X. Lu. Denoising prior driven deep neural network for image restoration. IEFE Trans. Pattern Anal. Mach. Intell., 41(10):2305-333 1022005-3111 18, Oct 2019.
10

---

Published as a conference paper at ICLR 2024
M. Elad, P. Milanfar, and R. Rubinstein. Analysis versus synthesis in signal priors. Inverse Problems, 23(3):947-963 2007.
W. Gan, S. Shoushtari, Y. Hu, J. Liu, H. An, and U. S. Kamilov. Block coordinate plug-and-play methodsfor blind inverse problems. In Proc. Adv. Neural Inf. Process. Syst. 36, 2023.
D. Gilton, G. Ongie, and R. Willett. Deep equilibrium architectures for inverse problems in imaging. IEEE Trans. Comput. Imag. 1:112-1133, 2021.
R. Gribonval. Should penalized least squares regression be interpreted as maximum a posteriori estimation? IEEE Trans. Signal Process., 59(5):2405-2415 May 2011.
R. Gribonval and P. Machart. Reconciling "priors" & "priors" without prejudice? In Proc. Adv. Neural Inf. Process. Syst. 26, pp. 2193-2201, Lake Tahoe, NV, USA, December 5-10, 2013.
R. Gribonval and M. Nikolova. On Bayesian estimation and proximity operators. Appl. Comput. Harmon. Anal., 50:49-72, January 2021.
A. Hauptmann, F. Lucka, M. Betcke, N. Huynh, J. Adler, B. Cox, P. Beard, S. Ourselin, and S. Ar- ridge. Model-based learning for accelerated, limited-view 3-d photoacoustic tomography. IEEE Trans. Med. Imag. (7(6))
2018.
S. Hurault, A. Leclaire, and N. Papadakis. Gradient step denoiser for convergent plug-and-play. In Int. Conf. on Learn. Represent., Kigali, Rwanda, May 1-5, 2022a.
Hurault, Leclaire, and Papadakis. Proximal denoiser for convergent plug-and-play op- timization with nonconvex regularization. In Int. Conf. Mach. Learn., pp. 9483-9505, Kigali, Rwanda, 2022b. PMLR.
K. Jin, M. McCann, Froustey, and M. Unser. Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process., 26(9):4509-4522, Sep. 2017. doi: 10. 10.110991122001722133099
Kadkhodaie and E. Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. In Proc. Ady.Neural Inf. Process. Syst. 34, pp. 55242-13254 December 6-14, 2021.
U. S. Kamilov, C. A. Bouman, G. T. Buzzard, and B. Wohlberg. Plug-and-play methods for inte- grating physical and learned models in computational imaging. IEEE Signal Process. Mag., 40 (1):85-97, January 2023.
E. Kang, J. Min, and C. Ye. A deep convolutional neural network using directional wavelets for low-dose x-ray CT reconstruction. Med. Phys., (4(10):e360-e311 2017.
R. Laumont, V. De Bortoli, A. Almansa, J. Delon, A. Durmus, and M. Pereyra. Bayesian imaging using plug & play priors: When Langevin meets Tweedie. SIAM J. Imaging Sci., 15(2):701-737, 2022.
J. Liang, J. Cao, G. Sun, K. Zhang, L. Van G., and R. Timofte. Swinir: Image restoration using swin transformer. In Proc. IEEE Int. Conf. Comp. Vis. (ICCV), pp. 1833-1844, 2021.
B. Lim, Son, H. Kim, Nah, and K. Mu L. Enhanced deep residual networks for single image super-resolution. In Proc. IEEE Conf. Comput. Vis. and Pattern Recognit. (CVPR) workshops, pp. 136-144, 144, 2017.
Liu, Y. Sun, C. Eldeniz, W. Gan, H. An, and S. Kamilov. RARE; Image reconstruction using deep priors learned without ground truth. IEEE J. Sel. Topics Signal Process., 14(6): (086):088-1098 Oct. 2020.
Liu, S. Asif, B. Wohlberg, and U. S. Kamilov, Recovery analysis for alug-and-play priors using the restricted eigenvalue condition. In Proc. Adv. Neural Inf. Process. Syst. 34, pp. 5921-5933, December 6- 14, 2021.
Liu, X. Xu, W. Gan, S. Shoushtari, and U. S. Kamilov. Online deep equilibrium learning for regularization by denoising. In Proc, Adv. Neural Inf. Process. Syst., New Orleans, LA, 2022.
11

---

Published as a conference paper at ICLR 2024
A. Lucas, M. liadiis, R. Molina, and A. K. Katsaggelos. Using deep neural networks for inverse problems in imaging: Beyond analytical methods. IEEE Signal Process. Mag., 35(1):20-36 January 2018.
M. McCann, K. Jin, and M. Unser. Convolutional neural networks for inverse problems in imaging: A review. IEEE Signal Process. Mag. 34(6):85-95, 2017.
T. Meinhardt, M. Moeller, C. Hazirbas, and D. Cremers. Learning proximal operators: Using de- noising networks for regularizing inverse imaging problems. In Proc. IEEE Int. Conf. Comp. Vis., pp. 1799-1808, Venice, Italy, Oct. 22-29, 2017.
C. Metzler, P. Schniter, A. Veeraraghavan, and R. Baraniuk. prDeep: Robust phase retrieval with flexible deep network. In Proc. 36th Int. Conf. Mach. Learn., pp. 3501-3510, Stockholmsmässan, Stockholm Sweden, Jul. 10-15 2018.
K. Miyasawa. An empirical bayes estimator of the mean of a normal population. Bull. Inst. Internat. Statist, 38:1-2, 1961.
V. Monga, Y. Li, and Y. C. Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Process. Mag., 38(2):18 (22):18-4 44, March 2021.
Y. Nesterov. Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, 2004.
G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett. Deep learning techniques for inverse problems in imaging. IEEE J. Sel. Areas Inf. Theory, 1(1):39-56, May 2020.
N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trends in Optimization, 1(3):123- 231, 2014.
E. T. Rechorst and P. Schniter. Regularization by denoising: Clarifications and new interpretations. IEEE Trans. Comput. Imag., 5(1):52-61 67, March 2019.
H. Robbins. An empirical Bayes approach to statistics. Proc. Third Berkeley Symp. on Math. Statist. and Prob., Vol. (Univ. of Calif. Press, 1956), pp. 157-163, 1956.
Y. Romano, M. Elad, and P. Milanfar. The little engine that could: Regularization by denoising (RED). SIAM J. Imaging Sci., 10(44:180441844, 2017.
E. K. Ryu, J. Liu, S. Wang, X. Chen, Z. Wang, and W. Yin. Plug-and-play methods provably converge with properly trained denoisers. In Proc. 36th Int. Conf. Mach. Learn., volume 97, pp. 5546-5557, Long Beach, CA, USA, Jun. 09-15 2019.
I. W. Selesnick and M. A. T. Figueiredo. Signal restoration with overcomplete wavelet transforms: comparison of analysis and synthesis priors. In Proc. SPIE 7446, 74460D, 2009. doi: doi: 10.1117/12.8266633
S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman. Plug-and-play priors for bright field electron tomography and sparse interpolation. IEEE Trans. Comput. Imaging, 2(4):408 423, Dec. 2016.
Y. Sun, B. Wohlberg, and U. S. Kamilov. An online plug-and-play algorithm for regularized image reconstruction. IEEE Trans. Comput. Imag. 5(3):395-400) September 2019.
Y. Sun, S. Xu, Y. Li, Tian, B. Wohlberg, and S. Kamilov. Regularized Fourier ptychography using an online plug-and and--and-pp -nd- algorithm. In Proc. IEEE Int. Conf. Acoustics, Speech and Signal Process. (ICASSP), pp. 7665-7669, Brighton, UK, May 12- 17, 2019. doi: 10.1109/CASSPP2009, 8683057.
Y. Sun, Z. Wu, B. Wohlberg, and U. S. Kamilov. Scalable guarantees. IEEE Trans. Comput. Imag., 7:849-863 863,, July 2021.
ADMM with convergence
12

---

Published as a conference paper at ICLR 2024
A. M. Teodoro, J. M. Bioucas-Dias, and M. A. T. Figueiredo. A convergent image fusion algorithm using scene-adapted Gaussian maussian-mixtureeeee mixtmmm- based denoising. IEEE Trans. Image Process., 28(1):451- 463, January 2019.
T. Tirer and Giryes. Image restoration by iterative denoising and backward projections. IEEE Trans. Image Process., 28(3): 1220-1122,, Mar. 2019.
S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg. Plug-and-pla priors for model based reconstruction. In Proc. IEEE Global Conf. Signal Process. and Inf. Process., pp. 945-948, Austin, TX, USA, Dec. 3-5, 2013.
S. Wang, Z. Su, L. Ying, X. Peng. S. Zhu, F. Liang, D. Feng, and D. Liang. Accelerating magnetic resonance imaging via deep learning. In Proc. Int. Symp. Biomedical Imaging, pp. 514-517, April 2016. doi: 10. (09/ISBL.2011.799333.
Y. Wang, J. Yu, and Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.0044 2022.
K. Wei, A. Aviles-Rivero, J. Liang, Y. Fu, C.-B. Schönlieb, and H. Huang. Tuning-free plug-and- play proximal algorithm for inverse imaging problems. In Proc. 37th Conf. Mach. Learn., 2020.
X. Xu, Y. Sun, J. Liu, B. Wohlberg, and U. S. Kamilov. Provable convergence of plug-and-pla priors with mmse denoisers. IEEE Signal Process. Lett., 27:1280-1288 2020.
Z. Yue, Q. Zhao, J. Xie, L. Zhang, D. Meng, and K. K Wong. Blind image super-resolution with elaborate degradation modeling on noise and kernel. In Proc. IEEE Conf. Comput. Vis. and Pattern Recognit. (CVPR), pp. 2128-2138 2022.
J. Zhang and B. Ghanem. ISTA-Net: Interpretable optimization-inspree deep network for image compressive sensing. In Proc. IEEE Conf. Comput. Vision Puttern Recognit., pp. 1828-1837, 2018.
K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang. Beyond Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans. Image Process., 26(7):3142-3155, Jul. 2017a.
K. Zhang, W. Zuo. S. Gu, and L. Zhang, Learning deep CNN denoiser prior for image restoration. In Proc. IEEE Conf. Comput. Vis. and Pattern Recognit. pp- 1929-3933, Honolulu, USA, July 21 21-26, 2017b.
K. Zhang, W. Zuo, and L. Zhang. Deep plug-and-play super-resolution for arbitrary blur kernels. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1671-1681, Long Beach, CA, USA, June 16-20, 2019.
K. Zhang, L.. Van G., and R. Timofte. Deep unfolding network for image super-resolution. In Proc. IEEE Conf. Comput. Vis. and Pattern Recognit. (CVPR), pp. 3217-3226, 2020.
K. Zhang, J. Liang, L. Van G., and R. Timofte. Designing a practical degradation model for deep blind image super-resolution. In Pruc. IEEE Inf. Conf. Comp. Vis. (ICCV), pp. 4790- 2021.
K. Zhang, Y. Li, W. Zuo, L. Zhang, L. Van Gool, and R. Timofte. Plug-and-play image restoration with deep denoiser prior. IEEE Trans. Patt. Anal. and Machine Intell., 44(10))6360-633 6376, October 2022.
Y. Zhu, K. Zhang, J. Liang, J. Cao, B. Wen, R. Timofte, and L. Van G. Denoising diffusion models for plug-and-play image restoration. In Proc. IEEE Conf. Comput. Vis. and Pattern Recognit. (CVPR), pp. 1219-1229, 2023.
13

---

Published as a conference paper at ICLR 2024
This supplement to the main paper presents the details of our mathematical analysis and provides additional numerical results. The mathematical analysis builds on two distinct lines work: (a) relationship between the MMSE denoising and the probability density of the noisy image (Gribon- val, 2011; I; Gribonval & Machart, 2013; Bigdeli et al., 2017; Kadkhodaie & Simoncelli, 2021); (b) analysis of the proximal-gradient method (Beck & Teboulle, 2009; Attouch et al., 2013; Parikh & Boyd, 2014; Hurault et al., 2022a). Our results can be viewed as an extension of this line of research priors specified using more general restoration operators.
The structure of this supplementary document is follows. In Section A.I, we provide formal characterization of the fixed-points of DRP. In Section A.2, we prove the convergence of DRP under additional assumptions on 9 and R. Section B, we provide number of interesting numerical results that didn't make it to the main paper due to space. In particular, we evaluate the AWGN denoising performance of DRP, compare with another recent PnP method called GS-PnP, and show how DRP enables adaptation of SwinIR beyond problems it was trained on.
A THEORETICAL ANALYSIS OF DRP
A.1 PROOF OF THEOREM 1
Theorem. Let R be the MMSE restoration operator (7) corresponding to the restoration problem (4) under Assumptions 1-3. Then, any fixed-point x' E IR" of DRP satisfies
0 E 0g(x") + Vh(x"),
where h is given (9).
Proof. First note that any fixed point x" E _ of the DRP method can be expressed as
xx sprox (a -TGG(")) arg min | (a*
yr (rGG("") G))"")))ll))) l + 70(x))
(12)
=c
where we used the definition of the scaled proximal operator. From the optimality conditions of the scaled proximal operator, we then get
E 0g(x=) + HGH"")
(13)
On the other hand, the gradient of ps. defined in (8), can be expressed as
<sP ( Hrr s) Go (x)) dæ o² (HR(s) - -pp
(14)
where used the gradient of the Gaussian density with respect and the definition of the MMSE restoration operator in eq. (7). By rearranging the terms, we obtain the following relationship
HR(s) - s = V log (a()), E R
(15)
By using the definitions G(x) x R(Hæ) and h(x) To log Po (Hx), with æ R", in (15),
we obtain the following generalization of the well-known Tweedie's formula
HTHG(x:) H x R(Ha)) - 2 (H.)) Vh(x).
(16)
7
By combining(12) and (16), we directly obtain the desired result
0 E 0g(x") Vh()").
A.2 PROOF OF THEOREM 2
Theorem. Run DRP for t 2 1 iterations under Assumptions 1-4 using a step-size 7 = p/(a))
with a > 1. Then, for each iteration 1 < k < t, there exists w w(x E 0f(x*) such that
Σ w (=< 113 C(f(x") -f*
min 1skSt
[w )
t
where C 0 is iteration independent constant.
14

---

Published as a conference paper at ICLR 2024
Proof. Consider the iteration & 2 of DRP
x sprox (x y GT(G((")) with : R(Ha),
where R is the MMSE restoration operator specified (7). This implies that xt minimizes
((x))=
27 a x 2
H x²-1)²==000 HG ')]' (xc x x²-1)==--0) + g(())
a x² ( H(x x² + Vh(a*-1) - ===1)= + g(x);
where in the second inequality we used eq. (16) from the proof in Supplement A.1. By evaluating ş at a and x= we obtain the following useful inequality
g(x") S g(a
(x* 1a
-==1
Vh(a vh(a*~1))(("" x x²11). (17)
27
On the other hand, from the L-Lipschitz continuity of Vh, we have the following bound
h(**) S h( + Vh(a*~1) x² = + _ 11 1
(18)
By combining eqs. (17) and (18), we obtain
f(x*) S s(xx-11) - - (rt 7(a* ax-119 H II
(x. - x²h-1
$1( - (a (1) L 1| x
(19)
where we used y (alL) with a and μ defined Assumption 4. On the other hand, from the optimality conditions for 4,, we also have
0 E H H(# - - XTG(x* + 70(
H' x E 0g(x**) + Vh(x
where used eq. (16) from Supplement A.1. This directly implies that the following inclusion
w(x ) = HT x²11)~ Vh(x") - Vh(x*-1> 0f(x
The norm of the subgradient (x**) can be bounded as follows
)12 - HTH"" *x (a-1)1le + [/h((*)) Vh(a 1||
$ L (a(Apl) + 1) || x 2122
(20)
where we used the Lipschitz constant of Vh, 7 p/(aa), and 2 μ 0 defined in Assumption 4. By combining eqs. (19) and (20), we obtain the following inequality
w x >> < A A x² - < A2(S(x f(x*)) )), where A = (a((/)))) + 1)² > 0 and A₂ = t iterations, we can directly get the desired result
(21)
- 1)) > 0. Hence, by averaging over
1 ) 42(f(x²')(')
min Isket
w x MB <
(22)
This implies that w(x*) + 0 as + 20.
15

---

Published as a conference paper at ICLR 2024
B ADDITIONAL NUMERICAL RESULTS
B.1 IMAGE DENOISING
In this subsection, we show the performance of DRP on Gaussian image denoising. The corre- sponding measurement model is 2 + c, where e is AWGN with the standard deviation o and æ is the unknown clean image. We use the same SwinIR SR model, introduced in 5.1, as the prior for denoising. The degradation model in the SwinIR prior is the operation H correspond- ing to bicubic downsampling. The scaled proximal operator sprox in (5) with data-fidelity term
g(x) 2 ly 212 can be written
sprox,, (2) = (I -I+ + + Hz),
(23)
which can be efficiently implemented using CG, as in Section 5.2.
We compare DRP with one of SOTA denoising model DRUNet (Zhang et al., 2019) on noise level (a = 0.1). Figure and Figure illustrate the visual performance of DRP on the Set5 and CBSD68 datasets, respectively. Figure 7 further explores the impact of using different SR factors q priors, elucidating how these choices influence the visual quality of denoising.
0.1
DRP (Ours)
DRUNet
Ground Truth
20.85
32.39
33.47
PSNR
20.33
29.42
32.14
20.88
29.86
30.59
Figure 5: Illustration of denoising results of DRP on Set5 dataset with noise level = 0.1. Each image is labeled by its PSNR (dB) with respect to the original image.
16

---

Published as a conference paper at ICLR 2024
0.1
DRUNet
DRP (Ours)
Ground Truth
20.05
31.97
29.79
PSNR
20.84
31.53
29.22
30.05
Figure 6: Illustration of denoising results of DRP on CBSD68 dataset with noise level a = 0.1. Each image labeled by its PSNR (dB) with respect the original image.
a =
0.1
2x
3x
4x
Ground Truth
20.85
32.39
31.66
30.51
20.33
29.42
27.05
25.68
Figure 7: Illustration of denoising results of DRP on Set5 dataset with three SR level prior (2x, 3x and xx). Each image labeled by PSNR (dB) with respect to the original image.
17

---

Published as a conference paper at ICLR 2024
B.2 COMPARISON WITH GS-PNP
In this subsection, we will compare DRP with the recent gradient-step denoiser PnP method (GS- PnP) (Hurault al., 2022a). These comparisons were not included in the main paper due space, but are provided here for completeness. GS-PnP provides comparable performance on image de- blurring and single image super resolution as DPIR (Zhang al., 2019), but comes with theoretical convergence guarantees.
Table 3 shows that DRP outperforms both DPIR and GS-PnP image deblurring in most settings in terms of PSNR. Similarly, Table 4 shows that DRP can achieve better SISR performance in terms of PSNR compared to both methods. Figure 8 provides additional visual results on SISR showing that DRP can recover intricate details and sharpen features.
Kernel
Datasets
GS-PnP
DPIR
DRP
Set3c
29.53
29.78
30.69
CBSD68
28.86
28.70
29.10
Set3c
27.52
27.32
27.89
CBSD68
27.44
27.52
27.46
Table 3: PSNR performance of GS-PnP, DPIR, and DRP for image deblurring Set3c and CBSD68 datasets on two blur kernels. The best and second best results are highlighted.
SR
Kernels
Datasets
GS-PnP
DPIR
DRP
Set3e
28.23
28.18
29.26
CBSD68
28.03
27.97
28.12
2x
Set3e
26.19
26.80
27.41
CBSD68
26.79
26.98
26.98
Set3e
26.20
26.64
27.77
CBSD68
26.77
26.80
27.18
3x
Set3e
25.18
25.84
26.84
CBSD68
26.30
26.39
26.60
Table 4: PSNR performance of GS-PnP, DPIR, and DRP for 2x and 3x SISR on the Set3e and CBSD68 datasets, using two blur kernels. The best and second best results are highlighted.
2x LR Image
DPIR
GS-PnP
DRP (Ours)
Ground Truth
25.72
28.86
28.91
29.05
PSNR
3x LR Image
DPIR
GS-PnP
DRP (Ours)
Ground Truth
20.69
23.77
22.97
23.88
PSNR
Figure 8: Illustration of SISR results of DRP compared with two SOTA denoiser based PaP method DPIR (Zhang et al., 2019) and GS-PnP (Hurault et al., 2022a). The top row displays the performance for 2x SISR task, while the bottom row showcases results for 3x SISR task. Each image is labeled by its PSNR (dB) with respect to the original image, and the visual difference is highlighted by the boxes in the left bottom corner.
18

---

Published as a conference paper at ICLR 2024
B.3 COMPARISON WITH DIFFUSION POSTERIOR SAMPLING
There is a growing interest in using denoisers within diffusion models for solving inverse prob- lems (Zhu et al., 2023; Wang et al., 2022: Chung al., 2023). One of the most wiedely-adopted diffusion model in this context is the diffusion posterior sampling (DPS) method from (Chung et al., 2023), which integrates pre-trained denoisers and measurement models for posterior sampling. One may argue that DPS is related to PnP due to the use of image denoisers as priors. In this section, we present results comparing DRP with DPS for deblurring human faces. We used the public imple- mentation of DPS on the GitHub page that uses the prior specifically trained on human face image dataset (Chung et al., 2023). DRP uses the same SwinIR model trained on general image datasets (see Section 5.1). DPS and DRP are related but very different classes of methods. While DPS seeks to use denoisers to generate perceptually realistic solutions to inverse problems, DRP enables the adaptation of pre-trained restoration models as priors for solving other inverse problems.
Table presents PSNR results obtained by DPS and DRP for human face deblurring. While we omitted the visual results from the paper for the privacy reasons, will be happy to provide them requested by the reviewers. Overall, DPS achieves more perceptually realistic images, while DRP achieves higher PSNR and more closely matches the ground truth images. This is not surprising when considering the generative nature of DPS. A similar observation is available in the original DPS publication, which reported better PSNR and SSIM performance of PnP-ADMM relative to DPS on SISR and deblurring (see Supplement E in (Chung et al., 2023)).
Kernll
DPS
DRP
29.61
34.61
28.80
33.05
Table 5: PSNR performance of DPS and DRP for image deblurring on three sample images from FFHQ validation set (provided in the DPS GitHub project) with two blur kernels. The best results are highlighted.
19

---

Published as a conference paper at ICLR 2024
B.4 PERFORMANCE SWINIR TRAINED FOR BICUBIC SR ON A MISMATCHED SISR TASK
In this section, we make a noteworthy point: the SwinIR SR network we used as a prior in our DRP method is specifically trained for the bicubic SR task. Its direct application the SISR task (which is SR under Gaussian blur kernels and additive white Gaussian noise) leads to sub-optimal performance. This implies that our DRP method has the capacity to use mismatched restoration model as an implicit prior, effectively adapting it for other image restoration tasks,
Figure present qualitative and quantitative PSNR results on the Set3c dataset. Note how the direct use of SwinIR trained for bicubic SR does poorly on the SISR task, while using it within our DRP method as a prior leads the SOTA performance.
Image
SwinIR
DRP (Ours)
Ground Truth
21.37
21.25
29.01
PSNR
24.38
24.15
Figure 9: Illustration of 2x SISR using DRP compared with the directly use of SwinIR trained for bicubic SR. Each image is labeled by its PSNR (dB) with respect to the ground truth.
20

---

Published as a conference paper at ICLR 2024
B.5 ADDITIONAL EVALUATION USING SSIM AND LPIPS QUALITY METRICS
In this section, we present results using two popular quality metricc--SSM and LPIPS-for eval- uating image restoration. We compare the proposed DRP method using SwinIR trained for bicubic SR as a prior against PnP-ADMM and DPIR. The results below complement those using PSRN in Table and Table 2 in the main paper. As can be seen in Table and Table DRP always achieves better LPIPS and in most cases achieves better SSIM in all the debluring and SISR settings.
Kernel
Datascts
PnP-ADMM
DPIR
DRP
Set3e
29.11/0.925/0099
29.53/0. 930/0/ 110
30.69/0.440/0.o0
Sct5
32.31/0.9000 10000.00000113..
32.38/0.
22.38/0.901/0..0 901/0.152
32.79/0.902/0 109
CBSD68
08.90/0.832/0.28/
12.8660.836/0.330
29.10/0.837/0.2444
McMaster
32.20/0.895/0 895/0.136
32.42/0. 900/0.151
02.42/0.900/0.115
32.79/0.901//
07799/..9110.1111
Set3e
27.05/0.8888 888/0.157
27.52/0.
0022//090//.... 901/0.159
27.89/0. 900/0000000.000
Set5
60.77/0.873/0.11
30,94/0. 8004/0088000... 188
31.04/0.887 875/0.143
CBSD68
27.45/0. 783/0.360
27.52/0.787/0.376
27 27.46/0.773/M 773/0.296
McMaster
30,50/0.863/0.188
30.78/0.871/0.199
00.7990.86550.1441
Table 6: PSNR BENRR/SSIMEN /SSIM// of DRP and several SOTA PnP methods image deblurring with the Gaussian blur kernels of standard deviations 1.6 and 2.0 on Set3e, Set5, CBSD68, and McMaster datasets. The best and second best results highlighted. Note how DRP using image SR prior can outperform SOTA PnP methods that use denoisers as priors.
SR
Kernel
Datasets
PnP-ADMM
DPIR
DRP
Set3c
27.88/0.903/0. 131
18.11990090/0.0
29.26/0.915/00077
Set5
31.41/0.879/0.11
11.42/0.8830.117.
(1.47/0.876/0.111
CBSD68
25.00/0.796/0.330
07.97/0.801/0.355
28.12/0. 799/0.285
0..12/0..99/0...
McMaster
30.98/0.869/0. 208
31.16/0.87500.17
(1.39/0.873/0..
2x
Set3c
25.86/0.86170, 187
0..80/0.884/0.11
27.41/0.89 1/0.107
Set5
10.06/0.854/0.200
N0.36/0.862/0.222
30.42/0.8599/ 859/0.163
CBSD68
7..88/..749/0880
26.98/0. 401
26.98/0. 745/0.327
McMaster
29.53/0.836/0.22
29.87/0.84790.215
30.03/0.8 845/0.167
Set3e
25.85/0.8655 865/0.667117
16.64/0.883/0, 883/0,1000 169
27.7 27.77/0.898/ 898/0.090
Set5
0009/0.857/0.199
30. 30/39/0.50 863/0.202
30.83/0.868M/ 868/0.149
CBSD68
26.78/0.751/0.377
80/0
WwW.005550... 755/0.394
27. 17.1180.763// 763/0.2999
McMaster
29.52/0.839/0.1
19.8220.8488000 $48/0.210
29, 29.92/0. 24222/.041111
1/0.084441...... 1/6
3x
Set3e
24.87/0.834/0.21
25. 84/0. 863/0. 195
26.84/0.879) 879/0.108
Set5
29,26/0.836/0.22.
29/70/0.00
7770//00000...... 847/0.225
29.88/0.847// 847/0.166
CBSD68
26.28/0.722N 720000222/0442..
29/39/0. 730/0.42. 422
21.60/0.735/0.22
McMaster
28.72/0.814/0.266
29. 2..11/0.827/.222
29.47/0.832/0.16
Table 7: PSNRT/SSIMMY/LPISSL comparison of DRP and several baselines for SISR on Set3c, Set5, CBSD68, and McMaster datasets. The best and second best results are highlighted. Note the ex- cellent quantitative performance of DRP, which suggests the potential of using general restoration models as priors.
21

---

Published as a conference paper at ICLR 2024
B.6 COMPARISON ON SISR TASK WITH ADDITIONAL BASELINES
In this section, We will compare our proposed DRP method with some other baseline methods on 2x SISR task. Specificity, we will compare it with DIP prior-based methods BSRDM (Yue al., 2022) and pure learning-based method BSRNet (Zhang et al., 2021) on the same setting as we shown in Section 5.3. As observable in Table 8, DRP achieves better performance compared to both of them.
SISR
Kernel
Datasets
BSRDM
BSRGAN
DRP
Set3c
23.51/0.825/0.113
22.69/0.818/0. 102
29.26/0.915/00079
Sct5
27.80/0 827/0. 179
27.07700800//.111 124
31.47/0.876/0.122
CBSD68
25.66/0.718/0.33
25.51 7..1000.2330.225
28.12/0.799/0.29
McMaster
77.430.880/00.20
26,78/0. 1.900788/00132
31.39/0.87./0..
2x
Set3e
22.34/0. 768/0.228
22.34/0.768/0.233
22.62/0 801/0.123
27.41/0 .89 1/0.107
Set5
26.68/0.791/0 791300.99/00.46
06.88/0.789/0.11. 189
00.4220.85990..
CBSD68
24.86/0.670/0.
25.24/0.695/0.22 695/0.265
06.98/0.745/03277
McMaster
26.4770.768/0..
69900770/00157
30.03/0.84500.167
Table 8: PSNRT/SSIMt/LPIPS, comparison of DRP and several baselines for SISR on Setăc, Set5, CBSD68 and McMaster datasets. The best and second best results are highlighted. Note the ex- cellent quantitative performance of DRP, which suggests the potential of using general restoration models as priors.
22

---

Published as a conference paper at ICLR 2024
B.7 COMPARISON WITH DPIR (SWINIR): DPIR EQUIPPED WITH SWINIR DENOISER
In this section,we we will conduct an ablation study to shown the enhanced performance of DRP over denoiser-prior methods that do not originate from the SwinlR structure. To support this claim, we substitute the DRUNet denoiser utilized in DPIR with the SwinIR denoiser (Liang et al., 2021), called D aligning it with the same network structure employed in our SwinIR SR prior. As shown in table 9 and table 10, DRP can achieve better performance than both (PIR(SwinIR) and (PIR(DRUNet) in most of the debluring and SISR settings.
Kernel
Datascts
DPIR (SwinIR)
DPIR (DRUNct)
DRP
Set3e
30.66/0.932/0.099
29.53/0.93000
30.69/0.940/0.065
Sct5
32.35/0.888/0. 161
32.38/0.901 9010000010.155
$2.79/0.902/0.11
CBSD68
27.81/0. 01$100.669/0.66
0..86/0886/0..0
19.1000.837/0.244
McMaster
32,42/0. 900/0. 131
32.79/0.900 9119/09901/00.. 901/0.107
Set3e
27.77/0 0177/0.99/0..1.. 143
27.52/0. 901/0.159
27.89/0.
97.89/0.908/0..0 908/0.088
Set5
30.85/0. 01550.866/0.111 185
30. 30.94/0.A0.
144/0/88000.111
31.04/0.87 875/0.1143
CBSD68
27.29/0.7644M 764/0.378
27.52/0.787/0..
27. 27.46/0.7734 773/0.296
McMaster
30.69/0.859/0.188
30.75/0.871/0.193 30.79/0.865/0.144
Table 9: PSNR (dB) of DRP and several SOTA methods for solving inverse problems using denoisers on image deblurring with the Gaussian blur kernels of standard deviation 1.6 and 2.0 on Set3e, Set5, CBSD68 and McMaster datasets. The best and second best results are highlighted. Note how DRP can outperform SOTA PnP methods that use denoisers priors.
SISR
Kernel
Datasets
DPIR(SwinIR)
DRUN(D
DRP (SR)
Set3c
28.90/0.91
28.18/0.908.. 908
29.26/0.915/00079
Set5
31.38/0.872/0/ 122
11.42/0.88300111
31.47/0.876/0.
CBSD68
27.92/0.783/0..44
28.12/0. 799/0.285
McMaster
31.34/0.870/0 127
11.16/0.875/0.17
31.39/0.873/0.122
2x
Set3e
26.75/0.877/0 147
27.41/0.89 190
Set5
30.23/0,8524 8.4/0.24000
30.36/0.862/V 861
30,42/0. 0012//0.85//... 159/0.163
CBSD68
01.83/0.79/0.41
26.98/0.757/0.41
26.98/0.745/0.33 745/0.327
McMaster
1996/0884//0.22 207
29. 87/8 844/0.215
30.03/0. 845/0.1667
Set3e
0.81/0,883/0 136
26. 64/0 883/0. 169
27.77/0.898/0.00 898/0,090
Set5
30.35/0.8555 11100.855/0.1111 159
30.39 39/0 863/0.202
30.83/0.86800.0 868/0.149
CBSD68
26.79/0. 05.79/0.742/0.355 742/0.352
80/0 755/0,390 394
27.18/0.763/0.22
McMaster
29.94/0.8 19.94/0.84500.. 193
29. 99.8 848/0.20
29.92/0. 14100.841// 141/0.164
3x
Set3e
25.66/0.853/0. 853/0.153/.17
25.84/0 863/0.195
04.84/0.879/0..0
Set5
00.55/0.837/0.11
29.70/0 847/0.225
29.88/0.
29.88/0.847/0.166 847/0.162
CBSD68
26.29/0.7 1199/0.115/0455
26.39/0. 73000.30004440 422
26.60/0.73500.32
McMaster
29. 19.11/0.8217 827/0.233
29.47/0.832/0117
Table 10: PSNR comparison of DRP and several baselines for SISR Set3c, Set5, CBSD68 and McMaster datasets. The best and second best results are highlighted. Note the excellent quantitative performance of DRP, which suggests the potential of using general restoration models as priors.
23

---

Published as a conference paper at ICLR 2024
B.8 EVALUATION OF DRP USING IMAGE DEBLURRING NETWORK A PRIOR
In this section, we show that DRP can be used with restoration priors and network architectures (beyond the SwinIR-based bicubic SR prior). To that end, we evaluate DRP under the deep unfolding network for image deblurring as prior for solving Single Image Super Resolution (SISR) task.
B.8.1 RESTORATION PRIOR BASED ON DEBLURRING DEEP UNFOLDING NETWORK
Deep Unfolding Network Architecture. We pre-trained a deblurring model using the USR- Net (Zhang et al., 2020) architecture based deep unfolding network. Our training dataset consists of both the DIV2K (Agustsson & Timofte, 2017) and Flick2K (Lim et al., 2017) datasets, contain- ing 3,450 color images in total. During training, we applied synthesized blur kernels the input images, introducing Additive White Gaussian Noise (AWGN) characterized by randomly chosen in [0, 5/255]. This process uses the same synthesized blur kernels as detailed in (Zhang al., 2020).
The restoration network is trained only for the deblurring task, which limits its performance on the SISR task. The direct application the deblur prior on SISR is presented under the name "Deblur" in Table 11. Note how the direc use of the deblurring network on SISR without DRP does not work.
B.8.2 SINGLE IMAGE SUPER RESOLUTION USING DEBLURRING PRIOR
The measurement operator in SISR can be written as A = SK, where K is convolution with the blur kernel & and S performs standard d-fold down-sampling with d² n/mn The scaled proximal operator sprox>g in (5) with data-fidelity term g(x) 2 | SKx 2 can write as:
sprox (2) = (K* S SK H
HJ SK HH)
(24)
where H is the convolution operator with blur kernel k. Similarly to deblurring in Section 5.3, we use CG to efficiently compute (11). We adjust the hyper-parameter 7 T and then use these parameters on the remaining datasets.
We refer to DRP using the Deblurring Prior as DRP (Deblur) and DRP using the Prior as DRP (SR). Table 11 provides comprehensive quantitative evaluation of the reconstruction performance for the 2x Super-Resolution (SISR) task, using two distinct blur kernels. The results show that DRP (Deblur) outperforms DPIR and achieves comparable performance DRP (SR) across four datasets.
Kemel
Datasets
Deblur
DPIR
DRP (SR)
DRP (Deblur)
SeC3e
13.63/0.300/0.66
28.18/0.908/0.11
29.26/0.915/0.077
29.13/0.921/0.08
Set5
14.10/0.1120 172/0.81/
$1.42/0.883/0. 175
$1.44700876/0.11
31.81/0.8830..1
CBSDD68
13.69/0. 0..69/.115//.... 8599
77.9770.801/0.335
28.12/0.75 12200.99/0.288
28.27/0.704/0.283
McMaster
14.31/0. 14.31/0.158/0.66 158/0.844
11.11/0.875/0.1 179
31.39//.873/0. 122
31.52/0.881/0
2x
Set3c
7.85/0.090/0.955
26.80/0.8841 884/0. 171
27.41/0.89 I/0. 107
27.52/0.893/0.11
Set5
8. 8.77/0.052/1.00
336/0.862/0.222
30.42/0.8500
163
30.73/0.864/0,
175
CBSD68
8. 1.58/0.040/1.00
26.98/0. 1.V0.757/0.400
26,98/0. 745/0.327
27.19/0.555/0.333
McMaster
9,26/0.048/1.00
29.87/0.844/0.00
30.03/0.845/00 867
90.16/0.849/0.177
Table 11: PSNR//SSIM//LIPS. comparison of two DRP variants and several baselines for 2x
SISR on Set3e, Set5, CBSD68, and McMaster datasets. The best and second best results are high- lighted. Note how the deblurring model doesn't generalize well on SISR without DRP; while its integration as a prior into DRP yields SOTA performance.
24

---

Published as a conference paper at ICLR 2024
B.9 EVALUATION OF DIFFERENT RANDOM SEEDS
In this section, we present results illustrating the influence of different random seeds on the perfor- mance. We use three different random seeds for the image deblurring task the Set3c dataset. For each seed, DRP is compared with the best baseline method DPIR on the image deblurring task.
Seed ID
Kernel
DPIR
DRP
=
29.53/0.930/0.1 110
30.69/0.940/0056
ID
29.50/0.93000.10 30.65/0.940/00056
29.53/0.93000.11 30.67/0.941/0.058
=
27.52/0.901/0.10
27.89/0.906/0.088
ID 10
27.5 1/0.990000/
27. 27.85/0.907000055
ID 100
07.52/0.900/0.155
27.88/0.90900.066
Table 12: PSNRY/SSIMY/LPIPS/ performance of DPIR and DRP for image deblurring the Set3c dataset with two blur kernels and three different random seeds. The best results are highlighted.
B.10 NUMERICAL EVALUATION USING NON-MMSE PRIORS
In this section, we show that although our theory relies on the assumption that the restoration prior used for inference performs MMSE estimation, in practice, DRP work even for priors trained with other loss functions, such as the -norm or SSIM.
We retrained the SwinIR model for bicubic SR using the same setting as in Section 5.1, but with the -ormr loss instead of f2. Table 13 shows that there is not much difference between the two. This suggests that DRP can be stable in practice even without the MMSE assumption.
Kernel
DPIR
DRP (SRe,
DRP (SR(s)
29. 29.53/0.9300.110
30.67/0.941/0.052
30.69/0.940/0.056
27.52/0.901/0.159
27.85/0.909900086
27. 27.89/0.08800.08.
Table 13: PSNRT/SSIM//PIPSI, performance of DRP(SR,,) DRP(SR12) and DPIR for image deblurring on the Set3e dataset using two blur kernels. The best and second best results are high- lighted.
25


Reviews:
[{"review_id": "AewQixo1mV", "reviewer": "Reviewer_v5mr", "summary": "The authors propose to improve a family of solutions to inverse problems called plug and play reconstruction methods by replacing their implicit priors. Instead of utilizing only denoisers, the proposed method suggests other reconstruction methods can also be utilized as priors. They provide a theoretical study with key assumptions to illustrate the convergence. They present a numerical study on the convergence for two applications considered, deblurring and super-resolution. Lastly, the authors present results on multiple datasets compared against related previous works, showing improvements in most cases, according to their evaluation method.", "questions": "3) The authors should include other evaluation metrics, such as SSIM and LPIPS, for the results. That would help to make sure the performance is not biased towards a single evaluation metric.\n\n4) The authors mention one of the limitations for the proposed method is  “the assumption that the restoration prior used for inference performs MMSE estimation.” However, it is not clear if for the compared methods, they have included any of recent methods that are trained based on l_1 loss and SSIM to clarify how much this limitation affects the final performance in different applications.\n\n5) Using a number of different implicit priors rather than only a single architecture while comparing their performance numbers before and after applying the proposed method would clarify to what extend the method is sensitive to the implicit prior’s performance.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Originality: The main contribution of the work is extending the formulation of plug and play reconstruction methods to utilize implicit priors other than only denoisers.\n\nClarity: The theoretical analysis of their proposed idea is clearly presented.\n\nSignificance: While the performance numbers seem not necessarily significant in terms of improvements, it could lead to more improvements in subsequent works.", "weaknesses": "1) The method utilizes a pre-trained SwinIR [1]. However, according to the Table 2 from SwinIR, the performance of SwinIR for the task super-resolution is ~6db higher in terms of PSNR on set5, which is confusing. The authors should include the SwinIR for all of the test datasets as a baseline and explain the reason there is a performance drop after adapting their method.\n\n2) There is another main weakness in the experimental results. There seems to be no discussion on the sources of randomness for the proposed method and the compared works. I would assume for the proposed method, there could be randomness involved in all three main stages: the pre-training, the refinement process, and the Algorithm 1 itself. The same issue could also be true for the considered compared methods. The experimental study should consider multiple runs for each of the mentioned stages and provide individual or combined analysis on the distribution of results rather than just a single run. The reviewers need to be sure whether the provided performance numbers are the worst single runs of compared methods versus the best single run of the proposed method, or are average cases for all considered methods. Without providing such analysis using multiple runs, it is hard to assess the significance of the results for the proposed method.\n\n[1] Liang, J. Cao, G. Sun, K. Zhang, L. Van G., and R. Timofte. Swinir: Image restoration using swin transformer. ICCV, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aNtomym2n9", "reviewer": "Reviewer_U9fM", "summary": "In plug-and-play-type algorithms, a denoiser (such as a CNN trained to denoise images) is plugged in the place of the proximal operator which is part of the formally derived algorithm. This paper proposes a way of extending this approach in a way that allows the use not only denoisers but also other restoration networks (e.g., a trained super-resolution CNN). The extension is simple, although non-trivial, and the paper provides theoretical convergence guarantees and experimental validation.", "questions": "The form of Equation (9) suggests that using the regularizer trained to solve (4) is in some sense equivalent, or at least related, to analysis regularization/priors, where $p_s$ is a learned prior and ${\\bf H}$ the analysis operator. See https://doi.org/10.1117/12.826663 \nWhat do the authors think of this connection?\n\nHow much is lost by solving (10) and (11) by CG and how do the authors know that 3 iterations is enough? Would it be worth (at least for (10)) to give some careful thought to the form of the inversions needed; maybe it is possible to exploit the fact that bot ${\\bf H}$ and ${\\bf K}$ can are convolutions to obtain closed-form inversions using FFTs.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "As I mentioned above, the extension proposed in this paper is simple, although non-trivial, and the paper provides theoretical convergence guarantees and experimental validation. It is a solid paper and the authors show a clear and solid knowledge of the field and of the state of the art, which is comprehensively reviewed. In summary, although certainly not a breakthrough or a very exciting new method, it is a good quality piece of work.", "weaknesses": "Although this is a good quality paper, there are a few aspects that could be improved, some important, others less so. \n\nFirst, and most importantly, the paper lacks some discussion/analysis of why restoration networks for problems other than denoising can lead to better results than denoising regularizers. Do they learn better image priors? This is somewhat surprising, in that these other networks are expected to have learned (in addition to a \"prior\") also specific aspects of the particular problem in which they were trained. This begs the question: what problems yield the best restoration networks? Is this choice related to the main problem to be solved?\n\nThe previous comment is related to the following observation: although the authors present the method as working for general restoration networks, they end up only using restoration networks trained for super-resolution. Arguably, SR is a very particular type of inverse problem, in a sense, the one that is closer to pure denoising than any other inverse problem. The authors should comment on this. \n\nIn Section 4, there is some confusion regarding necessary and sufficient conditions. The authors write \"Our analysis will require several assumptions that act as sufficient conditions for our theoretical results.\" This sentence is self-contracting: if the assumptions are *required* they are *necessary* (\"necessary\" and \"required\" are synonyms). In fact, further down, the authors contradict that sentence, by writing \"This mild assumption is necessary ...\".\n\nThe first inequality in Theorem 2 is trivial and doesn't even need to be mentioned. \n\nMinor style issue: it is not good style to write \"...discussion in Kamilov et al. (2023) on ...\" or \"... in Chapter 3 in Beck (2017) ...\". Kamilov and Beck are not papers or books, but people. A nicer way to write these sentences is \"...discussion in the work of Kamilov et al. (2023) on ...\" or \"...discussion in the paper by Kamilov et al. (2023) on ...\" and \"... in Chapter 3 of the book by Beck (2017) ...\".\n\nA probability density function is degenerate, not only if it is supported on a subspace, but on any zero-measure set in the ambient space, for example, a d-manifold, with d < n. \n\nTypo: \"Our method is as a major extension ...\" should be \"Our method is a major extension ...\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pMkpuREiD5", "reviewer": "Reviewer_3U6y", "summary": "This work proposes the DRP method which utilizes a pre-trained restoration network as a prior to solve the inverse problem. Under several mild assumptions, it proves the convergence of DRP. In the experiments, two popular tasks (e.g., debluring and super-resolution) are considered to validate the effectiveness of the proposed method.", "questions": "1. According to my understanding, the motivation and idea of this work is very similar to IRCNN. The main difference to IRCNN is that it uses a more powerful restoration network SwinIR instead of DnCNN. The sprox operator in Eq. (5) corresponds to the sub-optimization problem Eq. (6a) of the paper of IRCNN. The core step 3 in Algorithm 1 corresponds to Eq. (6b) of IRCNN.\n2. Following the last question, step 3 introduce the restoration prior in Algorithm 1. I'm interested in that how to induce the updated procedure of step 3. In other word, a more intuitive explanation on step 3 should be provided.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written, and easy to follow.\n2. It provides sound theory proof on the convergence of DRP.", "weaknesses": "1. As for evaluation metric, more commonly-used metrics should be employed to have a comprehensive comparison, such as SSIM, LPIPS.\n2. More comparative methods should be considered, including the DIP prior-based methods DIPFKP (CVPR 2021) and BSRDM (CVPR 2022) and some directly trained based method, such as BSRNet (ICCV 2021), RealESRNet (ICCV 2021 workshop).\n3. This is a non-blind method. The experiments only verify its simple case with known degradation. I wonder that is it able to handle the real-world case?\n4.  As shown in the appendix, its performance is obviously inferior to DRUNet. Additionally, it relies on the SwinIR as a prior. It is necessary to conduct a comparison with SwinIR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uoPTQExDYj", "reviewer": "Reviewer_RzPk", "summary": "This paper aims at generalizing the idea of the implicit image prior parameterized by a pre-learned deep denoiser, for the plug-and-lay type image restoration. Specifically, the authors proposed the so-called deep restoration prior (DRP), which can be learned by more general image restoration tasks, such as super-resolution and deblurring. To demonstrate the effectiveness of the proposed method, the authors have trained a DRP based on super-resolution with a SwinIR architecture, and applied it to image deblurrring and super-resolution tasks.", "questions": "My concerns are provided in the \"weaknesses\" part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea is interesting and provides some new perspectives of the implicit image prior.\n\n2. Theoretical analysis has been provided.", "weaknesses": "1. The motivation of this work is not convincing enough. As I can understand, the aim of DRP is to better or more generally characterize the prior of images. However, it is doubtful whether this generalization is necessary since the deep denoising prior can be flexible enough by virtue of advanced neural network architectures.\n\n2. The application of the proposed DRP is more complicated. Specifically, the additional degradation operator H can make the image restoration optimization process more complex, as can be observed in Algorithm 1. Besides, the additional operator H raises another question, that is how to choose this operator. The authors experimented with super-resolution prior, which assumes H to be downscaling, but it is unknown what if other operator is adopted, and which operator is the best.\n\n3. The experimental comparison seems unfair. In specific, the authors used SwinIR for their DRP, while using DRUNet for DPIR, though DRUNet was adopted in the original DPIR work. It is necessary to re-implement DPIR with SwinIR, or re-implement DRP with DRUNet, such that the influences of network structure can be controlled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)