Gradient Step Denoiser for convergent Plug-and-Play
Authors: Samuel Hurault
Arthur Leclaire
Nicolas Papadakis
Venue: ICLR 2022 Poster (2022)
Keywords: Plug-and-Play
Inverse Problem
Image Restoration
Denoising

Abstract: Plug-and-Play methods constitute a class of iterative algorithms for imaging problems where regularization is performed by an off-the-shelf denoiser. Although Plug-and-Play methods can lead to tremendous visual performance for various image problems, the few existing convergence guarantees are based on unrealistic (or suboptimal) hypotheses on the denoiser, or limited to strongly convex data terms. In this work, we propose a new type of Plug-and-Play methods, based on half-quadratic splitting, for which the denoiser is realized as a gradient descent step on a functional parameterized by a deep neural network. Exploiting convergence results for proximal gradient descent algorithms in the non-convex setting, we show that the proposed Plug-and-Play algorithm is a convergent iterative scheme that targets stationary points of an explicit global functional. Besides, experiments show that it is possible to learn such a deep denoiser while not compromising the performance in comparison to other state-of-the-art deep denoisers used in Plug-and-Play schemes. We apply our proximal gradient algorithm to various ill-posed inverse problems, e.g. deblurring, super-resolution and inpainting. For all these applications, numerical results empirically confirm the convergence results. Experiments also show that this new algorithm reaches state-of-the-art performance, both quantitatively and qualitatively.

Published as a conference paper at ICLR 2022
GRADIENT STEP DENOISER FOR CONVERGENT PLUG- AND-
Samuel Hurault ; Arthur Leclaire & Nicolas Papadakis Univ. Bordeaux, Bordeaux INP, CNRS, IMB, UMR 5251,F-33400 Talence, France
ABSTRACT
Plug-and aong-and-Paay (PnP) methods constitute a class of iterative algorithms for imag- ing problems where regularization is performed by off-the-shelf denoiser. Al- though PnP methods can lead to tremendous visual performance for various im- age problems, the few existing convergence guarantees based unrealistic (or suboptimal) hypotheses on the denoiser, or limited to strongly convex data-fideligg terms. We propose new type of PnP method, based on half-quadratic splitting, for which the denoiser is realized as a gradient descent step on functional param- eterized by a deep neural network. Exploiting convergence results for proximal gradient descent algorithms in the nonconvex setting, we show that the proposed PnP algorithm a convergent iterative scheme that targets stationary points of an explicit global functional. Besides, experiments show that it is possible to learn such deep denoiser while not compromising the performance in comparison to other state-of-the-an deep denoisers used in PnP schemes. We apply our proximal gradient algorithm to various ill-posed inverse problems, e.g. deblurring, super- resolution and inpainting. For all these applications, numerical results empirically confirm the convergence results. Experiments also show that this new algorithm reaches state-of-the-ar performance, both quantitatively and qualitatively.
INTRODUCTION
Image restoration (IR) problems can be formulated as inverse problems of the form
x² arg min f() Ag(x)
(I)
where f' is term measuring the fidelity to a degraded observation y. and g is regularization term weighted by a parameter X 2 Generally, the degradation of clean image è can be modeled by linear operation y A& + E. where A is degradation matrix and & white Gaussian noise. In this context, maximum a posteriori (MAP) derivation relates the data-fidelity term to the likelihood fx) - p(y| 2 || y|l while the regularization term is related to the chosen prior.
Regularization is crucial since it tackles the ill-posedness of the IR task by bringing a priori knowl- edge on the solution. A lot of research has been dedicated to designing accurate priors g. Among the most classical priors, one can single out total variation (Rudin et al. 1992). wavelet sparsity (Malat 2009) patch-based Gaussian mixtures (Zoran & Weiss] 2011). Designing relevant prior g difficult task and recent approaches rather apply deep learning techniques to directly learn a prior from database of clean images (Lunz al. 2018 Prost al. 2021 González al. (021).
Generally, the problem T does not have a closed-form solution, and an optimization algorithm is re- quired. First-order proximal splitting algorithms (Combettes & Pesquet 2011 operate individually on ff and via the proximity operator
Prox y (x: arg min ; 111== 2112 f()).
Among them, half-quadratic splitting (HQS) (Geman & Yang 1995) alternately applies the proximal operators of ff and g. Proximal methods are particularly useful when either f or is nonsmooth.
Plug-and-Play (PnP) methods (Venkatakrishnan al. 2013) build on proximal splitting algorithms by replacing the proximity operator of with a generic denoiser, e.g. a pretrained deep network.
"Corresponding author: samuel.hurault@ math. wat..u-borcaaaue u-bordeaux.fr

---

Published as a conference paper at ICLR 2022
These methods achieve state-of-the-are results (Buzzard al. 2018 Ahmad al. 2020 Yuan al., 2020 Zhang et al., (022) in various IR problems. However, since a generic denoiser cannot gener- proximal mapping (Moreau 1965), convergence results, which stem from the properties the proximal operator, are difficult to obtain. Moreover, the regularizer g is only made implicit via the denoising operation. Therefore, PnP algorithms do not seek the minimization of an explicit objective functional which strongly limits their interpretation and numerical control.
ally be expressed
In order to keep tractability of a minimization problem, Romano et al. (2017) proposed, with reg- ularization by denoising (RED), an explicit prior that exploits a given generic denoiser D in the form (Ix) = (x, x D(=)). With strong assumptions on the denoiser (in particular a symmetric Jacobian assumption), they show that it verifies
V=g(x)
(xx).
Such a denoiser then plugged in gradient-based minimization schemes. Despite having shown very good results on various image restoration tasks, as later pointed out by Reehorst & Schniter (2018) or Saremi] (2019), existing deep denoisers lack Jacobian symmetry. Hence, RED does not minimize an explicit functional and is not guaranteed to converge.
Contributions. In this work, we develop PnP scheme with novel theoretical convergence guar- antees and state-of-the-art IR performance. Departing from the PnP-HQS framework, we plug denoiser that inherently satisfies equation (3) without sacrificing the denoising performance. The resulting fixed-point algorithm is guaranteed to converge to a stationary point of explicit func- tional. This convergence guarantee does not require strong convexity of the data data-fidelity term, thus encompassing ill-posed IR tasks like deblurring, super-resolution inpainting.
2 RELATED WORKS
PnP methods have been successfully applied in the literature with various splitting schemes: HQS (Zhang et al. 201 7b 2021). ADMM (Romano et al. 2017 Ryu al. 2019), Proximal Gra- dient Descent (PGD) Terms et al. 2020). First used with classical non deep denoisers such as BM3D (Chan al. 2016) and aseudo-lineat denoisers (Nair et al. 2021 Gavaskar et al.) (021), 1 more recent PnP approaches (Meinhardt et al.) 2017]] Ryu et al. 2019) rely on efficient the- shelf deep denoisers such DnCNN (Zhang al. 7a). State-of-the-ar IR results are currently obtained with denoisers that are specifically designed to be integrated in PnP schemes, like IR-
CNN (Zhang al. 2017b) or DRUNET (Zhang et al. 202I). Though providing excellent restora- tions, such schemes are not guaranteed to converge for all kinds of denoisers or IR tasks.
Designing convergence proof's for PnP algorithms is an active research topic. Sreehari et al. (0166) used the proximal theorem of Moreau (Moreau 1965) to give sufficient conditions for the denoiser be an explicit proximal map, which are applied a pseudo-linear denoiser. The convergence with pseudo-linear denoisers have been extensively studied (Gavaskar & Chaudhury 2020 Nair et al.) 2021 Chan 2019). However, state-of-the-an PnP results are obtained with deep denoisers. Various assumptions have been made to ensure the convergence of the related PnP schemes. With "bounded denoiser" assumption, Chan et al. (2016); Gavaskar Chaudhury (2019) showed convergence of PnP-ADMM with stepsizes decreasing to O. RED (Romano et al. 2017) and RED-PRO (Co- hen et al. 2021) respectively consider the classes of denoisers with symmetric Jacobian or demi- either too restrictive hard verify in practice. In Appendix A.3 more details are given on RED-based methods. Many works focus on Lipschitz pro- perties of PnP operators. Depending on the splitting algorithm in use, convergence can be obtained by assuming the denoiser averaged (Sun et al. 20195), firmly nonexpansive (Sun et al. 2021] Terrs ct al. 2020) simply nonexpansive (Reehorst Schniter 2018,Liv et al. (021). These settings are unrealistic as deep denoisers do not generally satisly such properties. Ryu et al. (201)); Terris et al. L.nstrineddLippshiiz Lipsccitz constants, order to fit the technical properties required for convergence. But imposing hard Lipschitz constraints on the network alters its denoising performance (Bohra et al. [202 Hertrich et al. (202I). Yet, Ret,Ryu et al. 2019) manages to get convergent PnP scheme without assuming the nonexpansiveness of D. This comes at the cost of imposing strong convexity on the data- fidte-fiyiitty term f. which excludes many IR inpainting. Hence, given the ill-posedness of IR problems,
contractive mappings, but these conditions
(2200) propose different ways to train deep denoisers with constrained
tasks like deblurring, super-resolution looking for unique solution via contractive operators is a restrictive assumption. In this work, we do not impose contractiveness, but still obtain convergence results with realistic hypotheses.
2

---

Published as a conference paper at ICLR 2022
One can relate the ideal deep denoiser to the "true" natural image prior p via Tweedie's Identity. In (Efron 2011), it is indeed shown that the Minimum Mean Square Error (MMSE) denoiser Da (at noise level σ) verifies (x) = I + o² log Pz(a)) where Pe is the convolution of p with the den- sity of (10, Id). In a recent line of research (Bigdeli et al. 201 Xu et al. 2020; Laumont et al., [202 Kadkhodaie & Simoncelli 2020), this relation is used plug a denoiser in gradient-based dynamics. In practice, the MMSE denoiser cannot be computed explicitly and Tweedie's 's Identity does not hold for deep approximations of the MMSE. In order to be as exhaustive as possible, we detailed the addressed limitations of existing PnP methods in Appendix A.
3 THE GRADIENT STEP PLUG-AND-PLAY
The proposed method is based on the PnP version of half-quadraaic-spliting (PnP-HQS) that amounts to replacing the proximity operator of the prior g with an off-the-shelf denoiser D.. In order to define a convergent PnP scheme, we first set up in Section 3.1 a Gradient Step (GS) de- noiser. We then introduce the Gradient Step PnP (GS-PnP) algorithm in Section 3.2
3.1 GRADIENT STEP DENOISER
We propose to plug a denoising operator D, that takes the form of a gradient descent step
D, ld - V
(4)
with go R" + R. Contrary to Romano et al. 2017). our denoiser exactly represents a conservative vector field. The choice of the parameterization of 9n is fundamental for the denoising performance. As already noticed in (Salimans & Ho (202T), experimentally found that directly modeling fa as neural network (e.g. a standard network used for classification) leads to poor denoising performance. In order to keep the strength of state-of-the-an unconstrained denoisers, we rather use
Jo (z) 2 ||0 No(x)113,
(5)
which leads to Do (x) = x 1 V. (x) = No JN. (x) (No()))))
(6)
where N₂ : IR= + Rn is parameterized by neural network and JN, (x) is the Jacobian of N₂ at point As discussed in Appendix A.2 the formulation (5) for go has been proposed in (Romano et al. 2017 Section 5.2) and (Bigdeli & Zwicker 2017) for a distinct but related purpose, and not exploited for convergence analysis. Thanks to our definition (6) for Dr., we can parameterize N, with any differentiable neural network architecture R" + R" that has proven efficient for image denoising. Although the representation power of the denoiser limited by the particular form (6), we show (see Section 5.1 that such parameterization still yields state-of-the-art denoising results. We train the denoiser D, for Gaussian noise by minimizing the MSE loss function
((Dx)
1I + (x) - x=]?],
(7)
or C(g.) E..pp. Vo E,) - 1.,
(8)
when written in terms of 9a using equation
Remark 1. By definition, the optimal solution 9 E arg min C related to the MMSE denoiser D*, that is, the best non-linear predictor of x given x + Eo. Therefore, it satisfies Tweedie's 's formula and Vg = log Pa |Efron 2011) i.e. 9-r = -02 Po + C. for some C E R. Hence approximating the MMSE denoiser with denoiser parameterized as 44 is related to approximating the logarithm of the smoothed image prior of Pa with -290. This relation was used for image generation with "Denoising Score Matching" by Saremi & Hyvarinen (019)) Bigdeli et al. 2020).
3.2 A PLUG-AND-PLAY METHOD FOR EXPLICIT MINIMIZATION
The standard PnP-HQS operator is Hos D orooxx i.e. (Id- V Vg. ) Prox, when using the GS denoiser as D,. For convergence analysis, we wish fit the proximal gradient descent (PGD) algorithm. We thus propose to switch the proximal and gradient steps and to relax the denoising step with a parameter X > 0. Our PnP algorithm with GS denoiser (GS-PnP) then writes
A Ga (11)) with A
Prox, oTTDD, + (1 7 Id),
GS-PuP
(9)
= Prox, (ld -.)gg).
3

---

Published as a conference paper at ICLR 2022
Under suitable conditions on f and 9o (see Lemma in Appendix C
fixed points of the PGD
operator TTA PaP correspond critical points of classical objective function in IR problems
F(z) f(z) + (z).
(10)
Therefore, using the GS denoiser from equation is equivalent to include an explicit regularization and thus leads to a tractable global optimization problem solved by the PnP algorithm. Our complete PnP scheme is presented in Algorithm [
It includes a backtracking procedure on the stepsize 7 that will be detailed in Section 4.2 Also, after convergence, we found it useful to apply an extra gradient step Id Xr Vr in order to discard the residual noise brought by the last proximal step Prox,
4 CONVERGENCE ANALYSIS
Algorithm 1: Plug and- image restoration
Param.: init. 20, 0, a 2 0, E > 0, To > O,
In this section, we introduce conditions on f and D, that will ensure the convergence of the PnP iterations (9) towards a solution of (IO). For that purpose, we make use of the litera- ture of convergence analysis (Attouch et al., 2013 Beek 2017: [Beck& & Teboulle| (2099) of the PGD algorithm in the nonconvex setting.
E 17 E (0, 1), (0,1/2).
Input : degraded image y-
Output: restored image a.
k = 0; x = Prox, (20): = To/1: A > E;
while ls < K and A do
= XD (3 + (1 Xr J
Ik+1 = Prox, (zk);
if F(xx) F (x++1)
< ? I 1k+1
4.1 CONVERGENCE RESULTS
then 17T;
else A = F(ax)) 1(a k = k +1;
A common setting for image restoration is the convex smooth 1,2 data-fidelity f = Ar + (1-
F(zo)
end
f())=1| Ar - yl? In order to cover the noiseless case or to deal with a broader range of common degradation models, like Laplace or Poisson noise, we only assume f to be proper, lower semicontinous and convex. Next, the regularizer Jo is assumed to be differentiable with Lipschitz gradient, but not necessarily convex. This assumption on 9. is reasonable from a practical perspective. Indeed, using a network No with differentiable activation functions, the function Io introduced in Section 3 is differentiable with Lipschitz gradient (details and proof are given in Appendix B). Without further assumptions on and fo. the following theorem establishes the convergence of both the objective function values and the residual for large variety of IR tasks.
Theorem (Proof in Appendix C.. Let - {+oo} and ga :R" be proper lower semicontinous functions with ʃ convex and 9o differentiable with L-Lipschitz grudient. Let X > O, F = f + Ago and assume that F is bounded from below. Then, for T < XL. the iterates 2y given by the iterative scheme g verify
(i) (F(rz) is non-increasing and converges.
(ii) The residual || - æx|l converges to O.
(iil) All cluster points of the sequence (Ix) are stationary points of (10).
Remark 2. In the nonconvex setting. the quantity 7 ming<i<k
T++1
x. is commonly used
to analyze the convergence rate of the algorithm (Beck & Teboulle 2009 Ochs et al. (2014). Fol-
lowing the proof of Theorem 1 in Appendix we can obtain <
F(aa)
F(ag) that is to say
(+)) convergence rate for the squared L² residual.
Remark 3. Even if most data-fidelity terms f encountered in image restoration are convex, Theo- rem can be extended to nonconvex f. The proof of (iii) requires technical adaptations that can be found in Li & Lin (2015) Theorem 1) (as the 1-Lipschitz property of Prox, does not hold anymore). Such nonconvex f appear for example in the context of phase retrieval (Metzler et al. 2018).
We can further obtain convergence of the iterates by assuming that the generated sequence (Ix)) is bounded and that f and ga verify the Kunlyka--ooasiewiizz (KL) property. The boundedness of (xx) is discussed in Appendix D The KL property (defined in Appendix E has been widely used to study the convergence of optimization algorithms in the nonconvex setting (Attouch et al. 2010; 2013 Ochs et al.) 2014). Very large classes of functions, in particular all the semi-algebracc functions, satisfy this property. In practice, in the extent of our analysis, the KI. property is always satisfied.
4

---

Published as a conference paper at ICLR 2022
Theorem 2 (Proof in Attouch et al. 2013)), Theorem 5.1). Let R" + RU and 9o : RR + R be proper lower semicontinous functions with f convex and 9a differentiable with L-Lipschitz gradient. Let X > 0, F = + Xga and assume that F is bounded from below. Assume that F verify the Kl. property. Suppose that T < XT, If the sequence (Ix) given by the iterative scheme ! is bounded, then it converges, with finite length,
critical point x² of F.
Remark 4. As explained in Attouch al. (2013) Remark 5.2), the continuity of is not quired since we use the "exact forward-backwatt splitting algorithm". We can thus deal with non-continuous data-fidelity terms, as it is the case with the inpainting application in Appendix J.3
A more detailed description of all the assumptions of Theorems 1 and is given in Appendix F
4.2 BACKTRACKING TO HANDLE THE LIPSCHITZ CONSTANT OF V
The convergence of Algorithm I actually requires to control the Lipschitz constant of Vg only on a small subset of images related to {. Therefore, estimating L for all images and setting the maximum stepsize т accordingly will lead to sub-optimal convergence speed. In order to avoid small stepsizes, we use the backtracking strategy of Beck (2017) Chapter 10) and Ochs et al. (2014).
The convergence study in the proof of Theorem is based on the sufficient decrease property of F established in equation (35). Without knowing the exact Lipschitz constant L, backtracking aims finding the maximal stepsize т yielding the sufficient decrease property. Given Y E (0, 1/2), 7) E [0, 1) and initial stepsize To > O, the following update rule on is applied at each iteration k:
while F(Xx)= F(TTs) (np((x)) Pap
ai 2x11?
nJ..
(11)
Proposition 1 (Proof in Appendix G Under the assumptions of Theorem 7 at each iter- ation of the algorithm, the backtracking procedure (17) is finite (i.e. a stepsize satifying ((Ix)-F(TTE) F(TGG. PuP (xx)) 2 211TGS ( =x is found in finite number if iterations), and with backtracking, the convergence results of Theorem I and Theorem? still hold.
Remark 5. In practice, as explained in Section 5 we choose to initialize the stepsize as XTo and backtracking hardly ever activates. In this particular case, our results prove convergence of the standard PnP-HQS scheme I++1 = Prox, oD oDx(xx) applied with our specific denoiser.
5 EXPERIMENTS
In this section, we first show the performance of the GS denoiser. Next empirically confirm that our GS-PnP method is convergent while providing state-of-th art results for different IR tasks.
5.1 GRADIENT-DESCENTTBAABD DENOISER
Denoising Network Architecture We choose to parameterize No with the architecture DRUNet (Zhang et al. 202I)) (represented in Appendix H)) a U-Net in which residual blocks are integrated. One first benefit of DRUNet is that it is built to take the noise level a as input, which is consistent with our formulation. Also, the U-Net models have previously offered good results in the context of prior approximation via Denoising Score Matching (Ho et al. 2020). Furthermore, Zhang al, (202T) showed that DRUNet yields state-of-the- results for denoising but also for PnP image restoration. In order to ensure differentiability w.r.t the input, we change RELU activations to ELU. We also limit the number of residual blocks to at each scale to lower the computational burden.
Training details We use the color image training dataset proposed in Zhang al. (021) i.e. a combination of the Berkeley segmentation dataset (CBSD) (Martin et al. [001]] ). Waterloo Explo- ration Database (Ma et al.) 2017). DIV2K dataset (Agustsson & Timofte 2017) and Flick2K data- set (Lim et al.) 2017). During training, the input images are corrupted with white Gaussian noise E with standard deviation a randomly chosen in [0. 55500/2 255). With our parameterization (6) of D., the network N, is trained to minimize the L loss (7). While the original DRUNet is trained with L' loss, we stick to the L loss to keep the interpretability of go as an approximation of the log prior (see Remark T). For each batch, the gradient Vg. is computed with PyTorc differentiation tools.
5

---

Published as a conference paper at ICLR 2022
We train the model on 128 128 patches randomly sampled from the training images. with batch size 16, during 1500 epochs. We use the ADAM optimizer with learning rate 10 divided by 2 every 300 epochs. It takes around one week to train the model on a single Tesla P100 GPU.
Denoising results We evaluate the PSNR performance of the proposed GS denoiser (GS-DDRNN on 256 x 256 color images center-cropped from the original CBSD68 dataset images (Martin al., 2001]. In [ we compare, for various noise levels o, our model with the simplified DRUNet (called "DRUNet light") that has the same architecture as our GS-DRUNet (2 residual blocks) and that is trained (with L loss) without the conservative field constraint. We also provide comparisons with the original DRUNet (Zhang et al. 2021) (with 4 residual blocks at each scale and trained with L' loss) and two ofte--of-the-art denoisers encountered in the PnP literature: FFDNet (Zhang et al., 2018] and DrCNN (Zhang et al., 2017a). For each network, we indicate in Table the average runtime while processing 256 x 256 color image on one Tesla P100 GPU.
Our GS-DRUNet denoiser, despite being con- strained to be an exact conservative field, reaches the performance of (and even slightly outperforms) its unconstrained counterpart DRUNet light. Sccond, departing from the lat- ter, we are able reduce the processing time by a large margin (#7) while keeping close PSNR to the original DRUNet (around -0.05dB) and maintaining a significant PSNR gap (around +O.5dB) with other deep denoisers like DnCNN and FFDNet. Note that the time difference be- tween GS-DRUNet and DRUNet light due to the computation of Vgr via backpropagation. These results indicate that GS-DRUNet is likely to yield competitive and fast PnP algorithm.
o (./255)
5
15
25
50
Time (ms)
FFDNet
39.95
33.53
30.84
27.54
1.9
DrCNN
39.80
33.55
30.87
27.52
2.3
DRUNet
40.31
33.97
31.32
28.08
69.8
DRUNet lighr 40.19
33.89
31.25
28.00
6.3
GS- DS-DRUNet 40.26
33.90
31.26
28.01
10.4
Table 1: Average PSNR denoising performance and runtime of our GS denoiser on 256 x 256 center-cropped images from the CBSD68 dataset, for various noise levels a. While keeping small runtime, GS-DRUNet slightly outperforms its un- constrained counterpart DRUNet light and outdis- tances the deep denoisers FFDNet and DrCNN
5.2 PLUG-AND-PLAY IMAGE RESTORATION
We show in this section that, in addition to being convergent, our PnP Alg. T Teaches sfate-of-the-ar art performance in deblurring and super-resolution (Sections 5.2.1 and 5.2.2) and realizes relevant in- painting (Appendix (.3). In all cases, we seek an estimate x of a clean image 2 E IR",, from an obser- vation obtained y A& + & E RR" with A X degradation matrix and & white Gaussian noise with zero mean and standard deviation v. The objective function minimized by Alg. iis
1 F(x) = 20 A - y/1² + 2 (1) - x111
(12)
In practice, for v > O, we multiply F in equation 12) by v² and consider F + with f(x) = - -1 2/1², 9. (x) = 2llNo(z) (z) - z||² and A Ava. With this formulation, the conver- I [and We also demonstrate in
gences of iterates and objective values are guaranteed by Theorems Appendix..3Jhhat our framework can be extended to other kinds of objective functions. For example, inpainting noise-free input images leads to a non differentiable data-fidelity term /.
Due to the large computational time of some compared methods, we use for evaluation and compar- ison subset of 10 color images taken from the CBSD68 dataset (CBSD10) together with the 3 fa- mous Set3C images (butterfly, leaves and starfish). Quantitative results run on the full CBSD68 data- set are given in Appendix D All images are center-cropped to the size 256 x 256. For each IR prob- lem, we provide default values for the parameters a and À that can be used to treat sucessfully a large class of images and degradations. The influence of both parameters is analyzed in Appendix D.5 Performance can be marginally improved by tuning À for each image, for example with the method of Wei et al. (220)) based on reinforcement learning. our experiments, backtracking is performed with 7 = 0.9 and 7- = 0.1. We observe (see Appendix B) that on majority of images, the Lipschitz constant L of Vg is slightly larger than 1. As convergence is ensured for XoT = >²>r= < +. we set the initial stepsize to To = (1²x)-1. At the first iteration, the gradient step in equation (9) is thus exactly D.. In the majority of our experiments, backtracking is never activated. The algorithm is initialized with a proximal step and terminates when the relative difference between consecutive values of the objective function is less than or the number of iterations exceeds K.

---

Published as a conference paper at ICLR 2022
5.2.1 DEBLURRING
For image deblurring, the degradation operator A boundary conditions. Therefore, H = FAAF, where F is the orthogonal matrix of the discrete diagonal matrix. The proximal operator of the data-fidelity term f(x) = H y|1 involves only element-win inversion and writes
is a convolution performed with circular
Fourier transform (and F its inverse), and
Prox+ = F -
2).
(13)
We demonstrate the effectiveness our method on large variety of blur kernels (represented in Table and noise levels. As in (Zhang et al., 2017b; Pesquet et al. [202]] Zhang et al. (202I), we use the 8 real-world camera shake kernels proposed Levin et al. (2009) as well as the
uniform
kernel and the 25x x 25 Gaussian kernel with standard deviation 1.6 (as in (Romano et al., 2017 We consider Gaussian noise with 3 noise levels v E (2.55. 7.65, 12.75) /25 ie. E f0.01. 0.03, 0.05). For all noise levels, we set
1.8v, X D²X X 0.1 for motion blur (kemnels (a) to (h)) and A 0.075 for static blur (kernels (i) and (i). Initialization is done with 20 y
we show in
Appendix (. the the robustness to the initialization. The stopping criteria are 10 and K 400.
We compare in Table our method (GS-PnP) against the patch-based method EPLL Zoran & Weiss. Hurault et a 20188) the deep PnP methods IRCNN (Zhang et al. 2017b). DPIR (Zhang al,
2011
2021 MMO (Pesquet et al. 2021). and the "RED-FP" algorithm (Romano et al. 2017) (with TNRD denoiser (Chen & Pock 2016)) referred to RED. Both IRCNN and DPIR PnP-HQS with fast decrease of r and a in a few iterations (8 iterations for DPIR) without guarantee of convergence.
DPIR the DRUNet denoiser from Table D MMO is the only compared method that guarantees convergence by plugging a DrCNN denoiser trained with Lipschitz constraints (but the only given network was trained for very low noise level). Finally, as RED only treats the channel in the YCbCr color space. we also indicate in Appendix J.1 for RED and the proposed method, the PSNR evaluated on the channel only.
Among all methods, GS-PnP closely follows DPIR in terms of PSNR for low noise level but per- forms equally or better for higher noise levels. Other comparisons are conducted in Appendix(I.ojon the Set3c and the full CBSD68 datasets TTables4aann and These results exhibit that GS-PnP reaches state-of-the-ar PnP deblurring for variety of kernels and noise levels. We underline that the con- vergence of GS-PnP is guaranteed, whereas DPIR can asymptotically diverge (see Appendix
(b)
(c)
(d)
(I)
(h)
Method
Avg
EPLI
28.32
28.24
28.36
25.80
29.61
27.15
26.90
26.69
25.84
26.49
27.34
RED
30.47
30.01
30.29
28.09
31.22
28.92
28.90
28.67
26.66
28.45
29.17
IRCNN
32.96
32.62
32.53
32.44
33.51
33.62
32.54
32.20
28.11
29.19
31.97
MMO
32.35
32.06
32.24
31.67
31.77
33.17
32.30
31.80
27.81
29.25 26
31.44
DPIR
33.76
33.30
33.04
33.09
34.10
34.34
33.06
32.77
28.34
29.16
32.50
GS-PnP
33.52
33.07
32.91
32.83
34.07
34.25
32.96
32.54
28.11
29.03
32.33
EPLL
25.31
25.12
25.82
23.75
26.99
25.23
25.00
24.59
24.34
25.43
25.16
RED
25.71
25.32
25.71
24.38
26.65
25.50
25.27
24.99
23.51
25.54
25.26
0.03
IRCNN
28.96
28.65
28.90
28.38
30.03
29.87
28.92
28.52
25.92
27.64
28.58
IRCNN
28.96
28.65
28.90
28.38
30.03
29.87
28.92
28.52
25.92
27.61
28.58
DPIR
29.38
29.06
29.21
28.77
30.22
30.23
29.34
28.90
26.19
27.81
28.91
GS-PnP
29.22
28.89
29.20
28.60
30.32
30.21
29.32
28.92
26.38
27.89
28.90
EPLL
24.08
23.91
24.78
22.57
25.68
23.98
23.70
23.19
23.75
24.78
24.04
RED
22.78
22.54
23.13
21.92
23.78
22.97
22.89
22.67
22.01
23.78
22.84
0.05
IRCNN
27.00
26.74
27.25
26.37
28.29
28.06
27.22
26.81
24.85
26.83
26.94
DPIR
27.52
27.35
27.73
27.02
28.63
28.46
27.79
27.30
25.25
27.11
27.42
GS-PaP
27.45
27.28
27.70
26.98
28.68
28.44
27.81
27.38
25.49
27.15
27.44
Table 2: PSNR(dB) comparison of image deblurring methods on CBSD10 with various blur ker- nels k and noise levels v. Best and second best results are displayed in bold and underlined.
For qualitative comparison, we show in Figure I c-f) the deblurring obtained with various methods on the image "starfish" (from set3C). Note that our algorithm, compared to competing methods, recover the sharpest edges. We also give convergence curves that empirically confirm the vergence of the values F(xx) ) (g) and of the residual minpsiss ||z/41 - x,1 22 1//1// (b). These observations supported by the additional experiment shown in Appendix J.1 Figure
7

---

Published as a conference paper at ICLR 2022
(a) Clean
(b) Observed (20.97dB)
(c) RED (25.92dB)
(d) IRCNN (28.66dB)
322
330
12"
(e) DPIR (29.76dB)
(1) GSPP (29.90dB)
(g) F (mr)
(h) (log scale)
Deblurring with various methods of "starfish" degraded with the indicated blur kernel and input noise level 0.03. Note that our algorithm better recovers the structures. In (g) and (h), show the evolution of F(xx) and 7 = mingsis* || - x<|1²/ / and in Appendix J.4 of the PSNR. We empirically verify convergence of functional values and residual. Note that the empirical convergence rate in (h) is faster than the O theoretical worst case rate established Remark
Figure 1:
5.2.2 SUPER-RESOLUTION
For single image super-resolution. the low-resolution image E R" is obtained from the high- resolution one x E En via y SHæ + & where E R2*x" the convolution with anti-aliasing kernel. The matrix S is the standard s-fold downsampling matrix of size m and n s² x m. In this context, we make use of the closed-form calculation of the proximal map for the data-fidelity term f(x) 1 SH yll², given by Zhao et al. (0016):
Prox, 2g F" (Im 5z AA") AF2
(14)
where 2, T + 2 and [A,,..., Aa E R with A = diag(A1.... Aae) a block- diagonal decomposition according to a x paving of the Fourier domain. Note that Im + _AAA a m x m diagonal matrix and its inverse is computed element-wise. As expected, with 8 1, equation (14) comes down equation (13).
As in Zhang et al. (2021), we evaluate super-resolution performance on 8 Gaussian blur kernels represented in Table 4 isotropic kernels with different standard deviations (0.7, 1.2, 1.6 and 2.0) and 4 anisotropic kernels. Results are averaged between isotropic and anisotropic. We consider downsampled images scale 2 and 3 and Gaussian noise with 3 different noise levels E {0.01, 0.03, 0.05}. Our method (GS-PnP) is compared against bicubic upsampling, RED, IRCNN ("IRCNN+" from (Zhang et al. (02T) and DPIR. We give again Appendix D.2 results obtained on the Set3C dataset (Table 7). All our results are obtained with X, = y²X 0.065 and
Initialization 2g done with bicubie interpolation of y (with shift correction (Zhang
al. 2021 and the stopping criteria are 1 and K = 400.
Besides being the only compared PnP method with convergence guarantee, GS-PnP outperforms in PSNR all other PnP algorithms over the considered range of blur kernels, noise levels and scale factors. We show in Figure 2 the super-resolution of the image "leaves" downsampled by 2, with an isotropic kernel and noise level 0.03. GS-PnP (f) recovers more accurately structures and color details than competing approaches (c-e), while converging in terms function values (g) and residual (h). Additional visual comparisons are presented in Appendix
8

---

Published as a conference paper at ICLR 2022
Kemels
Method
Avg
0.01
/ 0.03
v
0.05
V 0.01
0.03
0.05
Bicubic
24.85
23.96
22.79
23.14
22.52
21.62
23.15
RED
28.29
24.65
22.98
26.13
24.02
22.37
24.74
IRCNN
27.43
26.22
25.86
26.12
25.11
24.79
25.92
DPIR
28.62
27.30
26.47
26.88
25.96
25.22
26.74
GS-PnP
28.77
27.54
26.63
26.85
26.05
25.29
26.86
Bicubic
23.38
22.71
21.78
22.65
22.08
21.25
22.31
RED
26.33
23.91
22.45
25.38
23.40
21.91
23,90
IRCNN
25.83
24.89
24.59
25.36
24.36
23.95
24.83
DPIR 26.84
25.59
24.89
26.24
24.98
24.32
25.48
GS-PnP
26.80
25.73
25.03
26.18
25.08
24.31
25.52
PSN comparison of image super-resolution methods on CBSDI0 with various scales s, blur kernels l and noise levels v. PNSR results are averaged over kernels each row.
Table 3:
YIT
Clean
(b) Observed
RED (21.75dB)
(d) IRCNN (22.82dB)
32
32"
(e) DPIR (23.97dB)
(f) GS- PnP (24.81dB)
(g) F(x))
(h) 72 (log scale)
Super-resolution with various methods on "leaves" (set3C) downsampled by 2, with the indicated blur kernel and input noise level one that recovers sharpest leaves. In (g) and (h), we show the evolution of F(xx) and 7k minugisk || 22,13 /////////// and in Appendix the evolution of the PSNR.. The pirical convergence rate is faster than the ( theoretical worst case rate established in Remark
Figure 2:
0.03. Note that algorithm is the
CONCLUSION
In this work, introduce new PnP algorithm with convergence guarantees. A denoiser trained to realize an exact gradient step on regularization function that is formulated through a neural net- work. This denoiser is plugged in an iterative scheme closely related PnP-HQS. which proved to converge towards a stationary point of an explicit functional. One strength of this approach is to simultaneously allow for a non strongly convex (and non smooth) data-fidelity term with a denoiser that may not be nonexpansive. Experiments conducted on ill-posed imaging problems (deblurring, super-resolution, inpainting) confirm the convergence results and show that the proposed PnP algo- rithm reaches state-of-the-art image restoration performance. This work also opens several research perspectives. One could first examine which information is encoded in the proposed prior. For ex- question if relation prior and the gradient energy or sparsity. Also, it would be interesting to see the recovery analysis developed in (Liu et al. 2021) adapt the proposed framework.
ample, based on the sharp visual results,
be drawn between this

---

Published as a conference paper at ICLR 2022
7 REPRODUCIBILITY STATEMENT
Anonymous source code is given in supplementary material. It contains a README md file that explains step by step how run the algorithm and replicate the results of the paper. Moreover, the pseudocode of our algorithm is given Algorithm [ In Section S it is precisely detailed how all the hyper-parameters are chosen and, for each experiment, which dataset used. As for the theoretical results presented in Section complete proofs are given in the appendixes.
8
ETHICS STATEMENT
We believe that this work does not raise potential ethical concerns.
ACKNOWLEDGEMENTS
This work was funded by the French ministry of research through CDSN grant of ENS Paris- Saclay. This study has also been carried out with financial support from the French Research Agency through the PostProdLEAP and Mistic projects (ANR-19-CE23-0027-00 and ANR-I9-CE4000055,
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recog- nition workshops, pp. 126-135, 2017.
Rizwan Ahmad, Charles Bouman, Gregery Buzzard, Stanley Chan, Sizhuo Liu, Edward Reehorst, and Philip Schniter. Plug-and-play methods for magnetic resonance imaging: Using denoisers for image recovery. IEEE signal processing magazine, 37(1):005-116, 2020.
Hédy Attouch, Jérôme Bolte, Patrick Redont, and Antoine Soubeyran. Proximal alternating mini- mization and projection methods for nonconvex problems: An approach based the kurdyka- tojasiewicz inequality. Mathematics of operations research, (5(2):4384457 457, 2010.
Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter. Convergence of descent methods for semi- algebraic and tame problems: proximal algorithms, gewward-backward splitting, and regularized seidel methods. Mathematical Programming, 137(1-2):91-12) 2013.
H. H. Bauschke and P. L. Combettes. Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, 2011.
Amir Beck. First-order methods in optimization. SIAM, 2017.
Amir Beck and Mare Teboulle. Gradient-based algorithms with applications to signal recovery.
Convex optimization in signal processing and communications, pp. 42-88, 2009.
Siavash A Bigdeli, Geng Lin, Tiziano Portenier, L Andrea Dunbar, and Matthias Zwicker. Learning generative models using denoising density estimators. arXiv preprint arXiv:2001.02728. 2020.
Siavash Arjomand Bigdeli and Matthias Zwicker. Image restoration using autoencoding priors.
arXiv preprint arXiv:1703.000 7703.09944. 2017.
Siavash Arjomand Bigdeli, Meiguang Jin, Paolo Favaro, and Mathias Zwicker. Deep mean-shift priors for image restoration. arXiv preprint 179.07970,,,3749, 2017.
Pakshal Bohra, Alexis Goujon, Dimitris Perdios, Sébastien Emery, and Michael Unser. Learning lipschitz-controlled activation functions neural networks for plug-and-play image reconstruc- tion methods. In NeurlPS 202/ Workshop on Deep Learning and Inverse Problems, 2021.
Jérôme Bolte, Aris Daniliddis, Olivier Ley, and Laurent Mazet. Characterizations of tojasiewicz inequalities: subgradient flows, talweg, convexity. Transactions of the American Mathenatical Society, 362(6):311-3333, 2010.
10

---

Published as a conference paper at ICLR 2022
Gregery T Buzzard, Stanley H Chan, Suhas Sreehari, and Charles A Bouman. Plug-am and--adddddd play un- plugged: Optimization-Fi free reconstruction using consensus equilibrium. SIAM Journal on Imag- ing Sciences, 1133)2001-2020,, 2018.
Luca Calatroni and Antonin Chambolle. Backtracking strategies for accelerated descent methods with smooth composite objectíves. SIAM Journal on Optimization, 29(3):1772-1795 2019.
Stanley H Chan. Performance analysis of plug-and-play admm: A graph signal processing perspec-
tive. IEEE Transactions on Computational Imaging, 5(2):274-286, 2019.
Stanley H Chan, Xiran Wang, and Omar A Elgendy. Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Transactions on Computational Imaging, 3(1): 84-98, 2016.
Yunjin Chen and Thomas Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. IEEE transactions on pattern analysis and machine intelligence, 1966):256-1122,, 2016.
Regev Cohen, Michael Elad, and Peyman Milanfar. Regularization by denoising via exed-point projection (RED-PRO). SIAM Journal on Imuging Sciences, 14(3):137411406, 2021.
Patrick L. Combettes and Jean-Christoppe Pesquet. Proximal splitting methods in signal process- ing. In Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185-212. Springer, 2011.
Bradley Efron. Tweedic's 's formula and selection bias. Journal of the American Statistical Associa- tion, 106(496):1002-1611 2011.
Ruturaj G Gavaskar and Kunal N Chaudhury. Plug-and-plat ista converges with kernel denoisers. IEEE Signal Processing Letters, 27:610-614 614, 2020.
Ruturaj G Gavaskar, Chirayu D Athalye, and Kunal N Chaudhury. On plug-an woug-and-pol and-play regularization using linear denoisers. IEEE Transactions on Image Processing, 2021.
Ruturaj Girish Gavaskar and Kunal Narayan Chaudhury. On the proof of fixed-point convergence for pldg-and-play ADMM. IEEE Signal Processing Letters, 26(12):1817-181, 2019.
Donald Geman and Chengda Yang. Nonlinear image recovery with half-quadratic regularization. IEEE transactions on Image Processing, 4(7):932-946, 1995.
Mario González, Andrés Almansa, and Pauline Tan. Solving inverse problems by joint posterior maximization with autoencoding prior. arXiv preprint arX1v:2103.01648, 2021.
Johannes Hertrich, Sebastian Neumayer, and Gabriele Steidl. Convolutional proximal neural net-
works and
algorithms. Linear Algebra and its Applications, 631:203-222 234, 2021.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 1239, 2020.
Samuel Hurault, Thibaud Ehret, and Pablo Arias. Epll: an image denoising method using a gaussian mixture model learned on large set of patches. Image Processing On Line, 8:465-489, 2018.
Zahra Kadkhodaie and Eero Peter Simoncelli. Solving linear inverse problems using the prior im- plicit in a denoiser. NeurIPS 2020 Workshop on Deep Learning and Inverse Problems, 2020.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. arXiv preprint arXiv:2103.04715, 2021.
Anat Levin, Yair Weiss, Fredo Durand, and William Freeman. Understanding and evaluating blind deconvolution algorithms. In 2009 IEEE Conference Computer Vision and Pattern Recogni- tion, pp. 1964-1971. IEEE, 2009.
Huan Li and Zhouchen Lin. Accelcrated proximal gradient methods for nonconvex programming. Advances in neural information processing systems, 28:379-387, 2015.
11

---

Published as a conference paper at ICLR 2022
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution. In Proceedings of the IEEE conference on com- puter vision and pattern recognition workshops, pp. 136-144, 2017.
Jiaming Liu, Yu Sun, Cihat Eldeniz, Weijie Gan, Hongyu An, and Ulugbek S Kamilov. Rare: Image reconstruction using deep priors learned without groundtruth. IEEE Journal of Selected Topics in Signal Processing, 14(6):1088-1000 1088-1099, 2020.
Jiaming Liu, M Salman Asif, Brendt Wohlberg, and Ulugbek S Kamilov. Recovery analysis for plug-and-play priors using the restricted eigenvalue condition. arXiv preprint arXiv:2106.03666 03668, 2021.
Sebastian Lunz, Ozan Öktem, and Carola-Bibiane Schönlieb. Adversarial regularizers in inverse problems. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8516-8525, 2018.
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. Waterloo Exploration Database: New challenges for image quality assessment models. IEEE Transactions on Image Processing, 26(2):1004-1016 Feb. 2017.
Stéphane Mallat. A Wavelet Tour of Signal Processing, The Spurse Way. Academic Press, Elsevier, 3rd edition edition, 2009, ISBN 978-0-12-374370-1.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int't Conf. Computer Vision, volume 2, pp. 416-423, July 2001.
Tim Meinhardt, Michael Moller, Caner Hazirbas, and Daniel Cremers. Learning proximal operators: the IEEE
Using denoising networks for regularizing inverse imaging problems. In Proceedings the
International Conference Computer Vision, pp. 1781-1790, 2017.
Christopher Metzler, Phillip Schniter, Ashok Veeraraghavan, et al. prdeep: robust phase retrieval with a flexible deep network. In International Conference Machine Learning, pp. 3501-3510. PMLR, 2018.
Jean-Jacques Moreau. Proximité et dualité dans un espace hilbertien. Bulletin de la Société Mathématique France, 93:273-299, 1965. http://// http:/tppp//eelmmlooooooo/////// wogg/occ/// 87067
Pravin Nair, Ruturaj Girish Gavaskar, and Kunal Narayan Chaudhury. Fixed-point and objective convergence of plug-and-play algorithms. IEEE Transactions on Computational Imaging, 2021.
Peter Ochs, Yunjin Chen, Thomas Brox, and Thomas Pock. ipiano: Inertial proximal algorithm for nonconvex optimization. SIAM Journal on Imaging Sciences, 7(2):1388-143)) 2014.
Jean-Christophe Pesquet, Audrey Repetti, Matthieu Terris, and Yves Wiaux. Learning maximally monotone operators for image recovery. SIAM Journal on Imaging Sciences, 14(3):1206-122 1237,
2021.
Jean Prost, Antoine Houdard, Andrés Almansa, and Nicolas Papadakis. Learning local regulariza- tion for variational image restoration. arXiv preprint arXiv:2102.06115, 2021.
Edward T Rechorst and Philip Schniter. Regularization by denoising: Clarifications and new inter- pretations. IEEE transactions on computational imaging, 5(1):52-6) 67, 2018.
Yaniv Romano, Michael Flad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4): 180--18444 2017.
Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Phys. D, 60:259-268, 1992.
Emnst Ryu, Jialin Liu, Sicheng Wang, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. Plug-and- play methods provably converge with properly trained denoisers. In International Conference on Machine Learning pp. 5546-5557, PMLR, 2019.
12

---

Published as a conference paper at ICLR 2022
Tim Salimans and Jonathan Ho. Should EBMs model the energy the score? In Energy Based
Models Workshop-ICLL Macsshop-CLRRRA 2021, 2021.
Saeed Saremi. On approximating V with neural networks. arXiv preprint arXiv: 19t.:::991..1.744.,, 2019.
Saced Saremi and Aapo Hyvarinen. Neural empirical bayes. arXiv preprint arXiv:1903.0223: 2019.
Katya Scheinberg. Donald Goldfarb, and Xi Bai. Fast first-order methods for composite convex optimization with backtracking. Foundations of Computational Mathematics, 14(3):389-417, 2014.
Suhas Sreehari, Venkat Venkatakrishnan, Brendt Wohlberg. Gregery T Buzzard, Lawrence Drummy, Jeffrey P Simmons, and Charles A Bouman. Plug-and-pla priors for bright field elec- tron tomography and sparse interpolation. IEEE Transactions on Computational Imaging, 2(4): 408-423, 2016.
Yu Sun, Jiaming Liu, and Ulugbek S Kamilov. Block coordinate regularization by denoising. arXiv preprint 1rXiv:1905.05113. 2019a.
Yu Sun, Brendt Wohlberg, and Ulugbek S Kamilov. An online blug-and-play algorithm for regular- ized image reconstruction. IEEE Transactions on Computational Imaging, 5(3):395 408, 2019b.
Yu Sun, Jiaming Liu, Yiran Sun, Brendt Wohlberg, and Ulugbek S Kamilov. Async-red: A provably convergent asynchronous block parallel stochastic method using deep denoising priors. arXiv preprint XrXiv:2010.014 2020.
Yu Sun, Zihui Wu, Xiaojian Xu, Brendt Wohlberg, and Ulughek $ Kamilov. Scalable plug-and-play admm with convergence guarantees. IEEE Transactions Computational Imaging, 7:849-863, 2021.
Matthieu Temiss. Audrey Repetti, Jean-Chriss Christophe Pesquet, and Yves Wiaux. Building firmly non- expansive convolutional neural networks. In ICASSP 2020-2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
IEEE, 2020.
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 945-948. IEEE, 2013.
Kaixuan Wei, Angelica Aviles-Rivero, Jingwei Liang, Ying Fu, Carola-Bibiane Schönlieb, and Hua Huang. Tuning-free plug-and-play proximal algorithm for inverse imaging problems. In Interna- tional Conference on Machine Learning, pp. 10158-10169 PMLR, 2020.
Xiaojian Xu, Yu Sun, Jiaming Liu, Brendt Wohlberg, and Ulugbek S Kamilov. Provable convergence of plug-and-pll play priors with mmse denoisers. arXiv preprint arXiv:2005.07685, 2020.
Xin Yuan, Yang Liu, Jinli Suo, and Qionghai Dai. Plug-and-play algorithms for large-scale snapshot compressive imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26 (7):3142-3155, 2017a.
Kai Zhang. Wangmeng Zuo. Shuhang Gu, and Lei Zhang. Learning deep enn denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp.
2017b.
Kai Zhang, Wangmeng 7uo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for cnn- based image denoising. IEEE Transactions Image Processing, (009)4660846222 2018.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Lue Van Gool, and Radu Timofte. Plug-and-pla image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
13

---

Published as a conference paper at ICLR 2022
Ningning Zhao, Qi Wei, Adrian Basarab, Nicolas Dobigeon, Denis Kouamé, and Jean- Yves Tourneret. Fast single image super-resolutinn using a new analytical solution for l 23-44 le problems. IEEE Transactions Image Processing, 25(8):3683-366 -69777733377,78 2016.
Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In 2011 Interattinnal Conference on Computer Vision, pp. 479-486, IEEE, 2011.
A COMPARISON WITH THE PNP LITERATURE
A.1 LIMITATIONS OF PREVIOUS PNP METHODS
In the existing literature, PnP approaches have one of the following limitations:
They are not able to provide proof of convergence when non strongly convex data-fidelity terms are involved (Ryu et al. 2019). which is the case of some classical IR problems such as deblurring, super-resolution or inpainting.
They are restricted to (nearly) nonexpansive denoisers (Rechorst Schniter 2018 Ryu et al., 2019, Sun et al.
2019b Xu et al., (020) or denoisers with symmetric Jacobian
(Romano et al. 2017). But has already been shown that imposing symmetric Jacobian or Lipschitz property on a deep denoiser network alters its denoising performance (Bohra
et al. 2021 Hertrich et al. (021). We highlight that it was already empirically observed (Romano et al. 2017]] Zhang et al. 202T) that the performance of the denoiser directly impacts the performance of the corresponding PnP scheme for IR.
They show convergence of iterates thanks to decreasing time steps (Chan al. 2016), but there is no characterization of the obtained solution (it is not a minima or a critical point of any functional).
On the other hand, our method is proved to converge stationary point of an explicit functional including a non strongly convex data-fidelity term. It also relies on (possibly expansive) denoiser that, although being constrained to be conservative vector field, allows to produce stateofftheeas results for various ill-posed IR problems.
A.2 ON THE REGULARIZZTIIN 9a-
We first underline that the main point of method is to define the denoiser as D,- = Id Vd-V The choice for Jar is important for the denoising performance. With respect to the convergence properties of GS-PnP, this is nevertheless secondary issue, as our method would converge for other differentiable regularizers
The proposed regularization (x) (z)] was previously mentioned in the RED original paper (Romano et al., (017)) (but explicitly left aside) and used in the DAEP paper (Bigdeli & Zwicker (017)). The main difference between our regularizer and the one alternately proposed in
RED and DAEP is the following:
RED and DAEP both consider a generic given pretrained denoiser D. : IR" a R", which is then associated with the regularizer go (x) = +11 (x)2) and used as such in IR problems.
In our method, we set yo (xr = - No (x) (with No R + R. differentiable) and then we train the denoiser D, = ld Vg. with the loss function [I + - =I||3 for clean images z and additive white Gaussian noise (AWGN)
With this new formulation, field, without further assumptions on Ng. Thanks to this relation, the (slightly modified) PnP-HQS given in relation g becomes proximal gradient descent (PGD). We can then make use of conver- gence results of the PGD algorithm in the nonconvex setting to show the convergence of PnP-HQS.
ensured that D, = ld Vo is inherently conservative vector
In contrast to the original RED paper, we aimed finding setting of Plug-and-Pla image restoration that allows for a convergence proof with sufficiently general hypotheses. For this pur- pose, we had to consider this very particular form of regularization.
14

---

Published as a conference paper at ICLR 2022
A.3 RECENT LITERATURE ON REGULARIZATION BY DENOISING
We here provide a more detailed discussion on the follow-up literature on RED.
In parallel the RED method (Romano et al., 2017), the authors of Bigdeli & Zwicker (017)) propose to use the regularization, mentioned but not exploited in RED. g(a) [D,(r) =|
where Da is a pretrained denoising autoencoder. Next, Bigdeli et al. (2017) extended this work with a new prior, which is the Gaussian- -mootthed version of the natural image prior. Inspired by Tweedie's formula, they approximate the gradient of this log smoothed prior with the residual of a pretrained denoising autoencoder. With this new formulation, it is possible to optimize on the restored image but also on other parameters (e.g. the noise level and the used blur kernel).
Initially designed in the context of convex data-fidelity term, RED Romano et al. (017)) has been applied in the nonconvex setting for phase retrieval problems in PeDeee/Metzleen et al. (2018).
Regularization by Artifact-Removal (RARE) (Liu et al. 2020) extends the RED framework by re- placing the denoiser by a more general artifact-removal operator. The main advantage of this op- erator is that it can be trained without groundtruth data, but only by mapping pairs of artifact and noise contaminated images obtained directly from undersampled measurements. This is particularly useful for medical imaging applications where it is difficult to acquire fully-sampled training data.
The convergence of the original RED algorithm discussed in Reehorst & Schniter (2018). The authors provide a convergence proof for RED-PGD which requires the denoiser to be nonexpansive, which, as detailed in the previous sections, is a restrictive hypothesis.
The authors of Liu et al. (2021) provide recovery guarantee for the PnP framework, meaning con- vergence to a x= that satisfies = Axx" while being in the set Fix( D) of the fixed points of D. More precisely, they show the convergence of the PnP-PGD method towards such true solution x= under the assumptions that the denoiser residual R = Id D is bounded and Lipschitz, and that the mea- surement operator satisfies a "set-restricted eigenvalue condition" (S-REC, which can be understood as strong convexity on the image of the denoiser lm( D)). Under the additional assumptions that the denoiser is nonexpansive and that there exists they show that PnP and RED have the same solutions. As mentioned by the authors. it is neverthe- less difficult to verify the S-REC condition for a given measurement operator: since Im Im(D) DD) is not explicit, it is not clear how much S-REC relaxes the strong convexity. As explained in Sections 2 and 3 our results do not require strong convexity of the data-fidelity term.
Fi) that is also critical for the regularizer g.
Instead of including explicit regularization in the functional, RED-PRO (Cohen al. 2021
aims at minimizing the data-fidelity term on the set Fix(D) of fixed points of a generic denoiser D. The study is conducted under the hypothesis that the denoiser is demicontractive, which implies that Fix(D) is convex. thus leading to a convex optimization problem. However, this assumption seems difficult to verify in practice and the existence of fixed points for the RED-PRO operator does not appear straightforward. In contrast, the fixed points of the GS-PnP operator are directly related to the stationary points of the global functional F f + Àgo (Lemma) 1 in Appendix C whose existence is guaranteed as soon as F is coercive (see the discussion in Appendix D).
ASYNC-RED (Sun et al. 2020) enables faster computation of RED by decomposing the inference into a sequence of partial (block-coordinate) updates on æ which can be executed asynchronously in parallel over a multicore system. As in their previous work BC-RED (Sun et al. (019a)), the authors propose to further reduce the computational time by using only a random subset of measurements every iteration. Convergence of ASYNC-RED is shown, provided the denoiser is nonexpansive. A possible future extension of our work is the integration of the ASYNC framework to accelerate GS-PnP for large scale imaging inverse problems. As our GS-PnP converges without assuming nonexpansiveness of the denoising operation, it would be interesting to see if one can adapt the GS-PnP convergence properties to an ASYNC-GSPMP algorithm.
B LIPSCHITZ CONSTANT OF V Vgr
First, let us give a result which ensures that a large class of neural networks trained with differen- tiable activation functions have Lipschitz gradients with respect to the input image.
15

---

Published as a conference paper at ICLR 2022
Proposition 2. Let H = o... be composition of differentiable functions h4 IR - R Let us assume that for any the differential map RV is bounded and Lipschitz. Then H' is Lipschitz.
Proof. Let denote H, h4 o ... hy (and by convention, Ho Id). Let ||A]]lo be the best uniform bound on the operator norms (z)],, z E R 1 (which is also the best Lipschitz constant of h.). Let us also denote ||A/ ip the Lipschitz constant of ht. The chain rule gives that for any x, H' can be expressed composition of linear maps
(x)
(15)
Therefore, for any 2, y.
p-1
(1(x)) (v) 44010//00000000.-..1110440000110 (V)) (16)
-
(17)
We can thus bound the operator norms
(x) H''(() S (42
(18)
(H,))))))) ..M..))
(19)
and thus
Σ
II |13]] #+++1
([]l=o)]]|/(++1 11.1111111 1/ vll
(20)
which concludes because the chain-rule ensures that || S || ... -I
Proposition 2 applies for neural network obtained composition of fully-conmected layers with ELU activation functions, that is, by composing functions of the form
h(x)= = E(Az + b)
(21)
where a matrix, b a vector and the element-wise ELU defined by
E(x); = I; if x 2 0 E - if < 0.
(22)
is easy to see that E is differentiable and that E' is 1-Lipschitz with ||E' S 1. Therefore h1(x) = E' (Ar F'(Ar + b)
(23)
also bounded and Lipschitz.
Let us also mention that this proposition encompasses the case of U-nets which, in addition to composing fully-connected layers, also integrates skip-connections. For example, taking a skip- connection on a composition hag o h2 hı amounts to defining
H [x) his h2 (22(11()) hı h1(x)) ).
(24)
This can be simply rewritten = hy o h2 hi where
hx) (h2(h1 hz)),,
(25)
It is then clear that h22 has bounded Lipschitz differential soon as hy and h2 do.
The bound obtained the proof of Proposition 2 exponential in the depth of the neural net- work. We now provide some experiments showing that, in practice, the Lipschitz constant of Vgr does not explode. We show in Figure 3 for various noise levels a, the distribution of the spec- tral norms | (z)]lls on the training image set X, estimated with power iterations. The com- puted value varies a lot across images. Hence approximating the Lipschitz constant of Vgo with == max ( would lead unttrtest stepsizes and slow convergence most images. Backtracking solves this issue by finding at each iteration the optimal stepsize allowing sufficient decrease of the objective function.
16

---

Published as a conference paper at ICLR 2022
5=
8000
15
0=
6000
0=
2000
1|09ol00||s
Figure 3: Histogram of the values of the spectral norm || go (z)lls evaluated 128 x 128 im- ages from the training dataset, degraded with white Gaussian noise with various standard devia- tions (J255). Figure best seen in color.
C PROOF OF THEOREM
We first remind that function R. TR +oo is proper if its domain
dom(f) = fo E f(x) < +oc
(26)
non empty. Also, recall that f is lower semicontinuous at x if lim inf f(x) f(z").
Proof.
(i) For ease of notation, we consider A = 1. The generalisation for any À 0 is straightfor- ward by rescaling Oa (and L) accordingly. We denote the proximal gradient fixed point operator
T- = Prox,f o(Td - vV.g..). the objective function F = + and we introduce
1
Q+ v) = (v) (r y. (g) + || wll2 f(x).
(27)
2r
We have
argmin arg min (+(r,y) arg min 9- (v) + (x - y, Vg)( + ||x - w/12 + f(x)
2r
(28)
min f(x) + - | (y (1)1
2-
=Prox, oId V = (T)
By definition for the arg min, #i+1 (xxx) = (- xx.)) S Q+ 2x). Moreover, with go being T--mmothh we have by the descent lemma, for any I and any v, y E TR."
1
(x))) = (v) + (x y. Vg. (v)) + || - y11
(29)
2r
so that for every x, y E 1
(r(xx))= = F(x) and Qr(,,)) 2 F(x).
(30)
Therefore, at iteration k.
F(*+11) & (T xR) S (T,,, FT) = F(xx).
(31)
(F(zx) is thus non-increasing. Since F is lower-bounded, (F(Ix)) thus converges a limit F*
iii Note that Q, (Ix+1,,22)) xx) (2xR 24) implies
1 || xx
f(xx+1) S (Ix) - (#x+1 - Ik. (go(Ix))
(32)
2r
17

---

Published as a conference paper at ICLR 2022
Using also 29 with stepsize 2, get
F(xx+1) - [(xn+1) + 9o (xx+1)
s f(xx) (0N+1 - 2h, (go())))
| 2kk
2r
L
(33)
Do (TR) + (oR+11 The Vgo (xx))
|08+1 xx2l 11
-1-11 1.)
=((a))
|| Txl
2T 2
Summing over I: = 0, 1,.., m gives
|et+1 - sxll
<
(F(ro) - F(Em+1))
0=0
(34)
< 1 L (F(ro) F*).
Therefore, limg->oe ||x--1 - 2z =.
(iii) We begin by the two following lemmas characterizing the proximal gradient descent operator T- = Prox,f o(d (V.o))V.9)).
With the assumptions of Theorem 7 for x" E R., x² is a fixed point of the proximal gradient descent operator Tr Prox,f o(Id -T VV ie. T, (x²) x", if and only if x² is stationary point of problem [10). ie. - (x)) E Of(z*).
Lemma 1.
Proof. By definition of the proximal operator, we have
Tr Tr(x*) x² x² Prox, o(Id (V + x - -V - x' E rdf((*)
(35)
V= 0f(x*).
Lemma 2. With the assumptions of Theorem 1 Tr is LL Lipschitz.
Proof. Using the fact that for convex, Prax,/ is I-Lipschitz (Bauschke & Combettes
2011
Proposition 12.28), and by the Lipschitz property of Vago.
[T) (v)ll
||rrox, oId (z9
< [[d (() (Id (V)))))))
(36)
< (1 (1+ v/l-
Note that, by nonconvexity of Jo. the fixed point operator T, is not necessarily nonexpansive, but we can still show the convergence of the fixed-point algorithm towards critical point of the objective function. We can now turn to the proof of (iii). Let x² be cluster point of (ax)420- Then there exists subsequence (HK,) 322 converging to x² We have Vj 2
||x0 (x I| < ||0" Ik, | T, (Tx, )11 + (TT, T+ (x"))
(37)
< (2 T TL]|">" - Tky + || T, (21,)) by Lemma 2
Using (ii), the right-hand side of the inequality tends to 0 as j + cc. Thus ||2 - T. (x )|| = 0 and
x² (x")), which by Lemma means that xt is stationary point of problem 110
18

---

Published as a conference paper at ICLR 2022
D ON THE BOUNDEDNESS OF (Ix)
In order to obtain convergence of the iterates, in Theorem 2 the generated sequence (arx) is assumed to bounded. In the experiments (Section 5). we observed that under the rest assumptions of Theorem 2 boundedness was always verified. A sufficient condition for the boundedness of the iterates is the coercivity of the objective function, that is, lim F(x) = +oo (because the non-increasing property gives F(xx)= S F(zo)).
Similar to Laumont et al. 22221) we can constrain F to be coercive by choosing a convex compact set C' R" where the iterates should stay and by adding an extra term the regularization Jo:
= _ - No (x) +
(x( = (x) + | |l(())]l
|c(
(38)
with He the Euclidian projection on C. As yo is differentiable, the gradient step becomes
(Id [dd-rAva.)(a) = (Id -T-V =A(x IIc())).
(39)
In our experiments, we choose the compact set C as [-1,2]" In practice we observe that all the iterates always remain in C' and that the extra regularization term || [T) [ is never
activated. Therefore, we don't present this technical adaptation Algorithm but we let the reader aware that boundedness of (2x) is not limiting assumption.
E KL PROPERTY
Definition 1.
(KL) property (taken from Attouch et al. (2010)
(a) A function f Rn - R U too is said to have the
property at E dom(f) if there exists n (0, +00), a neighborhood U of x² and continuous concave function vb [0, - R4 such that (b( 0, sa is C¹ on (0, v/ > 0 on (0, n) and Vx EUN Uf[[f(")) < f(x") )= n), the Kurdyka- Lojask--wjiiiiiczz inequality holds:
(f((x) - fdiss 0f(z)) 21.
(40)
(b) Proper lower semicontinuous functions which satisfy the Kurdyka-Lojasievia inequality at each point of dom(Of) are called KL. function..
This condition can be interpreted as the fact that, up to a reparameterization, the function sharp i.e. we can bound its subgradients away from O. A big class of functions that have the KL-property given by real semi-algebraic functions. For more details and interpretations, we refer to Attouch et al. (2010) and Bolte et al. (2010).
F ON THE ASSUMPTIONS OF THEOREMS
I AND 2
In this section, we explicitly list and comment all the assumptions required by Theorems and 2 These assumptions are standard in nonconvex optimization. We now detail why each assumption verified for our plug-and-play image restoration algorithm.
Assumptions Theorem
Data-fidelity term f R R (+oo)
proper lower semicontinous and convex. This is general assumption that includes most of the data-fidelity terms classically used in IR problems. Note that we do not require differentiability of f. Degradations with Gaussian, Poisson Laplacian noise models fall into this hypothesis. As noticed in Remark our results can even be easily extended to a nonconvex data-fidelity term f. which encompasses applications like phase retrieval Metzler al. (011). In practice, it is helpful to have proximable, f.e. Prox, with closed-for foosed-form formula. Otherwise, Prox, needs to be calculated at each iteration with optimization algorithm.
19

---

Published as a conference paper at ICLR 2022
Regularization function funtiiioniiioiiiiinnnoooooooo 9o IR" R proper lower semicontinous and differentiable with L- L-Lipschitz gradient. We parametrize as 9o (z) = 2 211 No(=)]1 with N, a differen- tiable neural network. This assumption tive. Indeed, using a network N₂ with differentiable activation functions, our function is differentiable with Lipschitz gradient (details and proof are given in Appendix B
9a is thus reasonable from practical perspec-
Functional F
Ago bounded from below. This is straightforward all the terms
positive.
The stepsize
This is handled by backtracking (see Section 4.2
XL-
Assumptions of Theorem
Assumptions of T'hoorm
F verify the KL property. The KL property (defined in Appendix E has been widely used to study the convergence of optimization algorithms in the nonconvex setting (Attouch al.) 2010 2013 Ochs et al.) (014). Very large classes of functions, in particular all the semi- algebraic functions, satisty this technical property. It encompasses all the data-fidelity and regularization terms encountered in inverse problems.
The sequence (0x) given by the iterative scheme (9) is bounded. As discussed in Ap- pendix D the boundedness can be ensured with a potential additional projection each iteration. This is just a theoretical guarantee, as we observed that such a projection is never activated practice.
G BACKTRACKING AND PROOF OF PROPOSITION
Before giving the proof of Proposition I we first point out that our backtracking line search is classical Armijo-type backtracking strategy, already used for nonconvex optimization in (Beck)
2017 Chapter 10) or Ochs et al. (2014). Other procedures could investigated in future work. For instance, Li & Lin (2015) uses Barzilai-Borwer rule to initialize the backtracking line search.
Scheinberg et al. (011)) and Calatroni Chambollc (2019) have also proposed a backtracking strategy that allows for both decreasing and increasing of the stepsize.
We now give the proof of Proposition
Proof. For a given stepsize т, we showed in Appendix C equation 33 that
F(*K) ((T+(1))) N 2 T. ||T [ 11.
(41)
Taking J < 1-27 we get 1 L.) > 1 so that
F(xx) (xT) [ITT(((x) - xx|12
(42)
Hence, when <
1-2y the sufficient decrease condition equation 42 is satisfied and the back-
tracking procedure (T 1yr) must end.
In the proof of Theorem 1 we can replace the sufficient decrease (33 by 42 and finish the proof with the same arguments. In the way, in the proof of Theorem given (Attouch al., 2013 Theorem 5. !), our sufficient decrease (42) replaces (Attouch et al. 2013 Equation (52)).
20

---

Published as a conference paper at ICLR 2022
H DRUNET light ARCHITECTURE
The architecture of the DRUNet light denoiser of (Zhang et al. (2021) is given in Figure
Noise Level Map
Noisy Imtgee
Denoised Image
2 Residual Blocks
Skip Connection
2 Residual Blocks
Conv
56 ..nov
Toonv
Conv
Downscaling
Upscaling
Figure 4: Architecture of the DRUNet light denoiser Zhang et al. 2021 used to parameterize No.
I EXPANSIVENESS OF THE DENOISER
As Jo is not necessarily convex, GS-DRUNet denoiser D2 Id Vgo is not necessarily expansive and neither is the gradient step ld -r This is not an issue as, unlike previous the- oretical PnP studies (Terris al. 2020 Reehorst & Schniter 2018), our convergence results do require a nonexpansive denoising step. To advocate that our method converges without this assump- tion, show in Figure 5 the evolution of (D along the algorithm that
to obtain the super-resolution results of Figure 7 In this experiment, backtracking did get activated and stayed fixed at Xr = 1. The gradient step in the PGD algorithm was thus simply a denoising step D, Ar Vgr. Note that the Lipschitz constant of D, goes above but convergence still observed as shown by the two convergence curves in Figure
1-
0.8
0.6
0.4
40
Figure 5: Lipschitz constant D, along the iterates of the algorithm when performing the super- resolution experiments presented Figure 7 Note that the Lipschitz constant goes above i.e. D, is not nonexpansive, but we still empirically verified convergence (see convergence curves Figure
J ADDITIONAL EXPERIMENTS
J.1 DEBLURRING
We give here additional image deblurring experiments. We first present the PSNR performance comparison on the Set3e dataset in Table 4 We also provide evaluation of the best methods (GS-PnP, DPIR and IRCNN) on the full CBSD68 dataset in Table 5 For fair comparison with RED, also display in Table the PSNR calculated on the Y channel only. An additional visual
21

---

Published as a conference paper at ICLR 2022
comparison is finally shown in Figure captions.
Details and comments given in the corresponding
a
(b)
(c)
(d)
(e)
(f)
g
(h)
i
V Method
Avg
EPLL
23.83
24.14
24.83
19.85
26.08
21.77
21.53
21.57
22.45 43
21.36
22.74
RED
29.21
28.58
29.52
24.54
30.45
25.34
26.06
26.07
25.11
28.50
27.34
IRCNN
33.36
33.06
33.11
32.87
34.24
34.08
33.25
32.87
27.78
29.67
32.45
MMO
32.84
32.29
32.76
31.85
34.08
33.76
33.11
32.38
26.31
29.91
31.93
DPIR
34.94
34.46
34.25
34.34
35.57
35.53
34.49
34.21
28.14
29.63
33.56
GS-PnP
34.58
34.13
34.04
33.93
35.45
35.25
34.30
33.97
28.16
29.78
33.34
EPLL 21.21
21.10
22.65
18.78
24.12
20.77
20.42
19.89
20.61
20.60
21.02
RED 25.42
24.89
25.69
22.67
26.86
23.84
24.06
23.87
21.49
25.45
24.43
0.03
IRCNN 29.08
28.62
29.03
28.46
30.51
30.06
29.23
28.74
24.39
27.39
28.55
DPIR 30.33
29.74
29.87
29.67
31.27
31.08
30.21
29.72
25.02
27.84
29.48
GS-PnP 30.29
29.84
30.14
29.58
31.53
31.24
30.41
29.96
26.13
28.56
29.77
EPLL 19.84
19.60
21.40
17.71
22.77
19.68
19.02
18.24
19.81
20.12
19.82
RED 21.93
21.27
22.79
20.32
24.01
22.05
22.06
21.41
19.79
23.21
21.88
0.05
IRCNN 26.85
26.33
27.04
26.10
28.46
27.90
27.05
26.56
22.90
26.16 26.54
DPIR 27.96
27.37
28.07
27.44
29.42
29.04
28.32
27.56
23.57
26.93
27.57
GS-PnP 28.08
27.75
28.35
27.56
29.60
29.17
28.49
28.01
24.67
27.47
27.91
Table 4: PSNR(dB) comparison of image deblurring methods on set3C with various blur kernels and noise levels p. Best and second best results are displayed in bold and underlined. Similar to Table 2 for all kinds of kernels, the proposed method outperforms all competing methods at noise levels 0.03 and 0.05 and follows DPIR at lower noise level 0.01.
aa)
(b)
(c)
(d)
(e)
g
(h)
ii)
i
D Method
Avg
IRCNN 32.47
32.14
31.94
31.97
32.94
33.13
31.92
31.62
27.57
28.45
31.42
5
DPIR 33.2
83.26
32.82
32.48
32.65
33.57
33.85
32.49
32.22
27.65
28.26
31.93
GS-PnP 32.95
32.54
32.26
32.31
33.41
33.71
32.29
31.92
27.43
28.17
31.70
IRCNN 28.43
28.11
28.28
27.87
29.42
29.21
28.37
27.97
25.52
26.96
28.01
0.03
DPIR 28.88
28.53
28.55
28.30
29.58
29.62
28.69
28.28
25.60
26.96
28.30
GS-PnP 28.64
28.32
28.55
28.06
29.71
29.60
28.69
28.31
25.79
27.10
28.28
IRCNN 26.73
26.42
26.73
26.13
27.69
27.39
26.69
26.33
24.68
26.18
26.40
0.05
DPIR 27.04
26.
27.07
26.53
28.00
27.85
27.17
26.72
24.75
26.32
26.82
GS-PnP 26.93
26.72
27.07
26.45
28.09
27.87
27.21
26.82
25.
26.45
26.86
for image deblur- ring on the full CBSD68 dataset with various blur kernels and noise levels v, in the same conditions Table On CBSD10 (Table 2) CBSD68 (Table 5), observed very similar performance gaps between the compared methods, which confirms that CBSD10 is large enough to compare accurately the PnP methods.
Table 5: PSN performance of the fastest method (IRCNN/DPIR/GS
22

---

Published as a conference paper at ICLR 2022
(a)
(b)
(C)
(d)
(e)
(f)
(h)
Method
Avg
REDy 35.19
34.78
34.58
34.64
35.49
35.65
34.70
34.37
30.15
31.15
34.07
v2/255
GS-PnPy
36.20
35.76
35.21
35.55
36.33
36.87
35.16
34.79
29.21
29.48
34.45
REDy
33.52 33.20
33.20
33.05
34.20
34.21
33.22
32.90
28.77
30.44 32.67
0.01
GS-PnPy 33.88
33.39
33.15
33.15
34.35
34.58
33.22
32.81
28.23
29.12 32.59
REDy 29.26 28.83
29.28
28.53
30.64
30.48
29.50
29.06
26.18 18 28.78 29.05
0.03
GS-PnPy 29. 29.11
29.39
28.82
30.55 30.46
29.55
19.14
26.50 28.02 29.01
REDy 26.91 26.54 27.52 26.23
28.68 28.25 27.67
27.07 25.36 27 27.98 27.22
0.05
GS-PnPy 27.65 27.48 27.88 27.15 19 28.90 28.67 28.03 27.59 25.62 27.29 27.63
Table 6: PSNR(dB) performance, evaluated the luminance channel in YcbCr color space, of RED and GS-PnP for image deblurring on CBSD10. Remind that our method treats the RGB image as whole before being evaluated on the Y channel while RED treats the Y channel independently. in RED original paper [Romano al. 2011 ) Note that RED was optimized for kernels (i) and (j) and v / /2/255, and outperforms our method in this set of conditions. However, over the variety of kernels and noise levels, and in particular for motion blur, our method generally outperforms RED.
Compared to Table 2 add the case v /2/255
(a) Observed (18.31dB)
(b) RED (28.4344 43d))
(c) IRCNN (30.61dB)
(d) MMO (30.23dB)
12"
16-
12
12
4
(e) DPIR (30.85dB)
(1) GS-PnP (30.74dB)
(g) F(
(h) m2 (log scale)
Figure 6: Deblurring with various methods of an image from CSBD10 degraded with the indicated
blur kernel and input noise level v 0.01. In (g) and (h), we show the evolution of F(xx) and
Th minosisk ||I 2,/12/1|zo|1 along our algorithm. Note that GS-PuP and DPIR both cover fine textures while other methods tend to smooth details.
23

---

Published as a conference paper at ICLR 2022
J.2 RETERRREOOOTIOO
We also present additional super-resolution experiments. We realize a full PSNR performance comparison on the Set3e dataset Table We show additional visual comparisons between methods Figure 7 and Figure
Details and comments are given in the corresponding captions.
Kemels
Method
8
8
Avg
0.01
0.03
0.05
0.01
0.03
0.05
Bicubic
21.92
21.54
20.90
19.76
19.53
19.11
20.46
RED
28.22
25.62
23.61
24.91
23.38
21.82
24.59
IRCNN
28.35
26.40
25.27
25.61
24.45
23.37
25.58
DPIR
29.08
27.27
26.21
26.55
25.33
24.41
26.48
GS-PnP
29.24
28.03
26.65
25.90
25.56
24.60
27.00
Bicubic
19.82
19.58
19.16
18.95
18.76
18.40
19.11
RED
24.72
22.55
21.10
22.82
21.64
20.19
22.17
IRCNN 25.10
23.44
22.52
24.25
22.60
21.58
23.25
DPIR 26.22
24.52
23.56
25.34
23.57
22.50
24.29
GS-PnP 25.45
24.84
23.80
24.53
23.73
22.71
24.18
Table 7: PSNR(dB) comparison of image super-resolution methods on set3C with various scales s, blur kernels and noise levels v. Similar to Table 3 for isotropic and anisotropic kernels, the proposed method outperforms all competing methods at noise levels 0.03 and 0.05 and follows DPIR at lower noise level 0.01.
Kernels
Method
=
Avg
0.01 / 0.03 v = 0.05 v = 0.01 == 0.03 = 0.05
IRCNN
26.97
25.86
25.45
25.60
24.72
24.38
25.50
DPIR
27.79
26.58
25.83
26.05
25.27
24.66
26.03
CS-PuPP
27.88
26.81
26.01
25.97
25.35
24.74
26.13
IRCNN
25.41
24.52
24.18
24.94
24.01
23.61
24.45
DPIR
26.08
24.99
24.39
25.53
24.46
23.80
24.88
GS-PuP
25.98
25.07
24.53
25.47
24.56
23.92
24.92
Table 8: PSNR(dB) performance of the fastest method (IRCNN/DPIR/GSS-P)))
for image super-
resolution on the full CBSD68 dataset with various blur kernels it and noise levels v, the same conditions as Table 3 Once again, on CBSD10 (Table 2) or on CBSD68 (Table 5), we observed very similar performance gaps between the compared methods, which again confirms that CBSDI0 is large enough to compare accurately the PnP methods.
24

---

Published as a conference paper at ICLR 2022
(a) Clean
(b) Observed
(c) RED (21. (8d))
(d) IRCNN (23.15dB)
542
12"
3355
18
(e) DPIR (23.33dB)
(l) GS-PnP (23.47dB)
(g) F
(h) Ta (log scale)
Figure 7: Super-resolution with various methods CBSD10 image degraded with the indicated blur kernel, s = 2 and input noise level v = 0.05. In (g) and (h), we show the evolution of F(xx) and 7 mings || 22il2// / l along our algorithm. One notice that the proposed method GS-PnP manages to extract more structure the zoomed area than the competing methods.
(a) Clean
(b) Observed
(c) RED (25.10dB)
(d) IRCNN (25.12dB)
12"4
eg
22.4
15:
13"
460
(e) DPIR (25.4)dB)
(f) GS-PnP (25.46dB)
(g) F(XE)
(h) 74 (log scale)
Figure& 8: Super-resolution with various methods on a CBSD10 image degraded with the indicated blur kernel, s = 3 and input noise level 0.01. In (g) and (h), we show the evolution of F(x+)
and i = minosisa || - ///01/1/>/l1 l1 along our algorithm.
25

---

Published as a conference paper at ICLR 2022
J.3 INPAINTING (WITH NON-DIFFERENTIABLE DATA-FIDELITY TERM)
We now propose to apply our PnP scheme to image inpainting with the degradation model
y Axr
(43)
where A is diagonal matrix with values in {0, 1]. For inpainting, noise added to the degraded image. In this context, the data-fidelln term is the indicator function of A ((v)) { Ar
y): f(x) ( (which, by definition, equals on A ((y})
) and +00 elsewhere). Despite
being non differentiable, still verifies the assumptions of Theorems 1 and 2 and convergence is theoretically ensured. The proximal map becomes the orthogonal projection ((x))
Prox, / (z) = II -[A))(())(())) = Ay Az +
(44)
In our experiments, the diagonal of is filled with Bernoulli random variables with parameter = 0.5. We run our PnP algorithm with the backtracking strategy and keep fixed stepsize. Even we not exactly know the Lipschitz constant of Vga. observed in Figure 5 that, for small noise, it was almost always estimated as slightly larger than We thus choose Ar and empirically confirm convergence with this choice in follow-up experiments (see Figure 9 The algorithm is initialized with Zg y 0.5(Id A]y (masked pixels with value 0.5) and terminates when the number iterations exceeds K 100. We found it useful does not have noise, we found preferable not to run the last extra gradient pass from Algorithm
10/255. Given the form of f. we do not use
the first 10 iterations of the algorithm at larger noise level 50/255. As
We show inpainting results set3C images Figure 9 Our PnP restores the input images with high accuracy, including its small details. Furthermore, convergence of the residual at rate O is empirically confirmed.
GG-app
10"
10a
Clean
Observed
GS-PnP (31.65dB)
(a) 7/ (log scale)
10-1
10a
Clean
Observed
GS-PaP (33.65dB)
7/k: (log scale)
10"
10"1
10--
Clean
Observed
GS-PnP (33.71dB)
72: (log scale)
Figure 9: Inpainting results on set3C with pixels randomly masked with probability p = 0.5. In the last colomn, we show the evolution of 7* minooiss || x;/12/[[zol] along the iterations.
26

---

Published as a conference paper at ICLR 2022
J.4 PSNR CONVERGENCE CURVES
We first plot Figure T0 the evolution of the PSNR along the iterations of the PnP algorithm during the experiments of Figure and Figure 2 This illustrates that the minimization of F coincides with the maximization of the PSNR, which supports the interest of the optimized functional F = f -+>0a-
27
26
23
25
22
21
4
12
16
24
32
0
25
50
100
125
150
175
(a) Deblurring
(b) Super-rcsolution
Figure 10: Evolution the PSNR along the iterations the algorithm, during (a) the deblurring ex- periment of F (b) the super-resolution experiment of Figure Note that the convergence
PSNR follows the convergence function value (represented in Figures I and 2 gg))
J.5 INFLUENCE OF THE PARAMETERS
In this section we study more deeply the influence of the parameters involved the GS-PnP algo- rithm. Three parameters are involved: the stepsize т, the denoiser level and the regularization parameter
The stepsize 7 is automatically tuned with backtracking and is not tweaked heuristically, contrary to other competing methods based on PnP-HQS.
The first regularization parameter a is linked to the used denoiser.
The second regularization parameter is introduced so as to target the objective function f + Àgo, which is the main purpose of our method. It is classical formulation of inverse problems, and the trade-off parameter is usually tuned manually.
Thus, like PnP-HQS, we have two parameters that we are free tune manually. One additional mo- tivation for keeping both X and a as regularization parameters is to be able to use our PnP algorithm with noise-blind denoisers like DrCNN that are independent on a. In practice, our experiments, we first roughly estimated proportionally to the input noise level and tweaked À more precisely. Note that, for each inverse problem, our parameters À and a are fixed for a large variety of kernels, images and noise levels v. The parameters are not optimized for each image.
Figure I and Figure 12 respectively plot the average PSNR when deblurring the CBSDIO images with different values X and o/v, and fixed y 0.03. Both parameters control the strength of the regularization. We observe that A and a have a similar influence on the output: for small X or small a, the regularization involved by the denoising pass is not sufficient to counteract the noise amplification done by the proximal steps with large т (recall that when 7 - oo, Prox,f tends to the pseudo-inverse of A). On the contrary, expected, increasing À and tends over-smooth the output result.
On a single image, we also vary the main parameters of both GS-PnP and RED (Figure 13). For fair comparison, the PSNR is computed on the luminance channel only. This experiment confirms that, when manually optimizing the parameters for both methods, the PSNR results obtained with CS-PnP and RED remain close, as already observed in Table 6.
27

---

Published as a conference paper at ICLR 2022
25
20
15
mot on blur
0.0
0.4
Av
A = 0.05
= 0.1
A = 0.5
A = 10
Figure 11: Influence of the choice of the parameter X, for deblurring. Top: average PSNR when deblurring the images of CBSDI0, blurred with motion blurs static blurs, for different values of Nu The other parameters remain unchanged. Bottom: visual results when deblurring "starfish" with various A (in the same conditions Figure I
22.5
20.0
17.5
15.0
12.5
3.5
a/v=
0/V = 1.8
afv=4 44
a/v= 10
Figure 12: Influence of the choice of the parameter for deblurring. Top: average PSNR when deblurring the images of CBSD10, blurred with the 10 kernels, for different values of a/v, with 0.03. The other parameters remain unchanged. Bottom: visual results when deblurring "starfish" with various g/v, with 0.03 (in the same conditions as Figure
28

---

Published as a conference paper at ICLR 2022
24
24
0 0.5
22
0=1
22
PSWA
20
0
20
1.0
18
0
18
B ==1
0=
16
2.0
14
14
# =2555
109
A
(a) RED
(b) GSPnP
Figure 13: Influence of the parameters a and À for RED and GS-PnP when deblurring the single image "starfish" degraded with uniform kernel and 7.65 /25.. For fair comparison, like in Table the PSNR calculated on the channel only. Remember that the results Table were obtained with 3.25, À 0.02 for RED and 2v, À 0.075 for GS-PnP.
J.6 INFLUENCE OF THE INITIALIZATION
In Figure 14 we examine the robustness of the method to the initialization. As can be seen on this experiment, the output image does change much even for relatively large perturbation of the initialization. We thus observe a robustness to the initialization, both in terms of visual aspect and PSNR. We also observed that initializing with uniform image does change the output of the algorithm. We suggest that this robustness comes from the first proximal steps the data-fidelity term (with large 7), which prevent the algorithm to be stuck in a poor local minimum. Note that the use of large in the beginning of the algorithm is possible thanks to the backtracking procedure.
25
PSNR
20
15
(.255)
20/255
Tiii 40/255
Fiii 60/255
Figure 14: Influence of the initialitation 2n on the deblurring result. Instead of initializing with the blurred image
+ E with & an AWGN with standard deviation Oinit- By increasing the noise level Oinis we investigate the robustness of the result to changes in the initialization of the algorithm. Top: PSNR values, along with values of anit Bottom: corresponding visual results on "starfish" with various Jntit (in the same conditions Figure II. The algorithm is robust to noisy initializations up to a relatively large value of Jnit-
done in Section 5.2. we set
29

---

Published as a conference paper at ICLR 2022
J.7 CONVERGENCE OF DPIR (ZHANG ET AL. (2021)
In this section, we illustrate that, contrary to our method, the DPIR algorithm is not guaranteed to converge and even easily diverge. In Figure 15 we plot the convergence curves of both DPIR and our GS-PnP when deblurring the "starfish" image degraded with a motion kernel and v = 0.01. In the original DPIR paper Zhang et al. (202I). only 8 iterations are used with decreasing and a. More precisely. a decreases uniformly in log-scale from 49 to the input noise level v, and т is set proportional to z². In order to study the asymptotic behaviour of the method, we propose two strategies to run DPIR with 1000 iterations:
(i) Decreasing a from 49 to v over 1000 iterations instead of 8 (Figure 13 row 1).
(ii) Decreasing o from 49 to v in iterations, and then keep the last values a and 5 for the the remaining iterations (Figure 13 row 2).
As illustrated by the plot of Eisk || 2;11 (third column), DPIR fails to converge with both strategies, even if the residual | xx tends decrease with the second strategy. This divergence also involves loss of restoration performance terms of PSNR (first column). On the other hand, as theoretically shown in this paper, the residual | Ix I[2 with GS-PnP tends to (reaches ~ 10 13 before the activation of backtracking, versus ~ 10 for DPIR) and its series converges.
DPIR (1)
357
1871
080
124
DPIR (11)
12-
337
0.01
03017
18"
03012
1011
GS PPPP
020.8
03075
20
60
150 30
29 26
PSNR(EL)
[ - /xxl1//1| /|
(log scale)
Lick
|++1
Figure 15: Convergence of the DPIR algorithm versus convergence of GS-PnP when deblurring the "starfish" image. The two first rows display results obtained with DPIR with two different strategies used for decreasing a: in the first row, is decreased from 49 over 1000 iterations; in the second row, a is decreased in 8 iterations and then kept fixed for the remaining iterations.
30


Reviews:
[{"review_id": "oRvkGZD-M1r", "reviewer": "Reviewer_E8QG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Weakness**\n\n1. As stated, the proposed regularizer is similar to the alternate prior proposed in RED. What is the difference? Why the re-invention is new in the context of the paper.\n\n2. The convergence analysis seems to follow the standard proof in the non-convex optimization literature when the explicit regularizer is given. To me, the analysis is a straightforward extension of the classic results. Could the authors elaborate more on the difficulties in the proof?\n\n3. Could the authors explain more about how the network is trained? Since the output of the neural network is an image-size vector and the regularizer is $||x-N_\\sigma(x)||_2^2$, it is a little weird to me if one forces $\\nabla g$ to be the noise residual. Why not make $N_\\sigma$ a denoiser and explain $\\ell_2$-norm as accommodation of the Gaussian noise.\n\n4. Various paper has proposed backtracking steps to determine the step-size ($\\tau$ in GS-PnP). Perhaps a comparison and discussion are required.\n\n5. The final performance is about the same as the state-of-the-art plug-and-play algorithm (DPIR). If the authors can explain the novelty of the proposed regularizer properly, this will no longer be a downside. However, if the work is not novel in terms of theory, a marginal/zero improvement in the empirical performance will be a weakness.\n\n6. Given the linkage to RED, the current paper should provide a proper review of the progress of RED as well.", "summary_of_the_paper": "This paper makes an extension of the plug-and-play framework by formulating an explicit regularizer $g(x)=||x - N_\\sigma(x)||_2^2$ whose gradient $\\nabla g$ corresponds to the noise residual $x-D_\\sigma(x)$. By replacing the proximal of regularizer with this gradient step denoiser, the authors proposed the GS-PnP algorithm based on the half quadratic splitting algorithm. Since the explicit regularizer is known, a convergence analysis is hence established by assuming the Lipschitz continuity of $\\nabla g$.\n\nThis paper is closely related to the recent trend of learning a regularizer functional by using deep neural networks. The difference between the existing literature and this paper is that the former trains a deep neural network to directly output a scalar value, while the latter trains a neural network to output an image-size vector and then envelopes it with a $\\ell_2$-norm. In fact, the proposed regularizer shares the same formulation as the one stated in Sec 5.2 *\"An Alternative Prior\"* in Romano's RED paper. However, the difference between the two is not clearly stated in the paper.\n\n**Strength**\n1. An extension of the PnP with explicit regularizer formulated.\n2. (Non-convex) convergence analysis under the common assumptions of the Lipschitz continuity of $\\nabla g$ \n3. Extensive validation on image denoising, deblurring, and super-resolution.\n\n", "summary_of_the_review": "In general, I vote for *weak rejection* based on the current evaluation of the paper.\n\n—— After rebuttal ——\nI now vote for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aps8Uld7cGy", "reviewer": "Reviewer_xYLt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros.\nI enjoy reading this paper -- the writing is so great, the theory proved is so strong and general, the empirical simulation is also extensive and convincing, making this paper one of the best papers I have ever reviewed. \n\nThe proposed gradient step denoiser formulation in the PnP framework is particularly interesting. By such a modification, they can reach a very general theoretical convergence result without relying on assumptions of strongly convex data terms $f$ and non-expansiveness on $g$ --- the best result obtained from prior research. Moreover, they claim the established theorem can even be extended to non-convex $f$ --- this is also super exciting. \n\n## Cons.\nI have some questions that need clarification: \n- The main results are obtained based upon HQS algorithm. I'm wondering whether the theoretical results are still validated on other proximal algorithms, e.g., proximal gradient descent, ADMM, etc. \n- I think in Remark 1, the gradient of $g_\\sigma^*$ should be equivalent to the negative log gradient of the smoothed image prior $p_\\sigma$, according to Eq.(3) in [R1]. \n- The gradient step formulation of denoiser requires the calculation of Jacobian (Eq. (6)), which makes the training (in Eq. (8)) a bit cumbersome. Could the authors provide more details on how they compute the Jacobian? How to do that with the PyTorch automatic\ndifferentiation tools? \n- Recent advances on PnP methods, e.g., [R2] should be mentioned in this paper.\n\n[R1] Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser, arxiv 2021  \n[R2] Tuning-free Plug-and-Play Proximal Algorithm for Inverse Imaging Problems, ICML 2020\n", "summary_of_the_paper": "This paper presents a new form of the plug-and-play (PnP) half-quadratic splitting algorithm with provable convergence. \nIn contrast to directly formulating the denoiser as previous works, they propose to parameterize the denoiser with a learnable score-based function. This builds a bridge between the recently emerged score-based generative model and plug-and-play methods. And even more surprisingly, they show such a new formulation within the PnP framework leads to a very strong theoretical convergence guarantee under mild realistic assumptions. They also demonstrate good empirical results on three image restoration tasks i.e., deblurring, super-resolution and inpainting, verifying the empirical convergence rate is usually faster than $\\mathcal{O}(\\frac{1}{k})$ -- the worst-case rate established theoretically. ", "summary_of_the_review": "From my point of view, this is a truly innovative work with significance on both theoretical and practical sides. Since PnP is a very important tool in image reconstruction domains, this paper is thus worth being highlighted and would inspire further works in future. \n\nDisclaimer: I don't check the full details of mathematical derivations. The justification here is therefore contingent upon the prior assumption: all the Theorem/Remarks in this paper are rigorously established. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C_HJmFD7gV", "reviewer": "Reviewer_GZzY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the overall idea of this paper is great: designing a differentiable neural network, by construction, that equals the gradient descent step of an explicit regularizer function, the fact which provides new insights for solving an important open problem about the convergence of PnP and RED.  The idea sounds simple but promising and makes it possible to be directly applied for other variant PnP/RED algorithms. My main concerns / remark with this paper are as follows:\n\n1). It is unclear for me why RED led to poor results for both deblurring and super-resolution. This may be due to the improper parameter tunning such as regularization trade-off, step-size and denoising intensity of the denoisers. Since the fixed points of PnP/RED algorithms are directly affected by the step size used in those algorithms [Ahmad et al., 2020, p.108], to show the true performance of DPIR and RED, the step size must be fine-tuned against PSNR. This can be done directly on the testing images. Additionally, the neural network architecture of denoisers used within RED need to be clarified.\n\n2). To no loss of completeness, an additional interesting path to follow for this proposed method is to have a connection between convex regularization and CNNs, which would help to facilitate the optimization process and allows to derive recovery methods with global convergence. Hence, a brief discussion for providing insights of building convex $g_\\sigma(x)$ through deep neural networks $N_\\sigma$ is highly recommended.\n\n3). Since the setups for $g_\\sigma$ is not necessarily convex, resulting a non-convex problem. An initial point of the update sometimes is important. It would be interesting to see how different initialization influence the final reconstruction results.\n\n4). In order to have a better understanding about how the regularization parameter $\\lambda$ explicitly adjust the balance between the data-fit and the unconstrained deep neural network regularizer, it is useful to include a figure showing the evolution of images reconstructed by PnP-GS for different $\\lambda$.\n\n5). It is helpful to include the proposed GS-DRUNet and DRUNet light denoiser architecture in the supplementary materials to make this work easy to follow.\n\n6). Having a convergent plot against PSNR (dB) would help to see the speed of improvement in imaging quality. \n\nMinor comments:\n\n1). Table (5) in the supplement, “31.70” is displayed in bold while “31.93”is not. \n\n2). Some missing citations related to PnP:\n\n[1] Buzzard et al., 2018, Plug-and-Play Unplugged: Optimization Free Reconstruction using Consensus Equilibrium.\n\n[2] Ahmad et al., 2020, Plug-and-play methods for magnetic resonance imaging: Using denoisers for image recovery.\n", "summary_of_the_paper": "The paper considers a novel and interesting idea: designing a deep neural network denoiser that makes Plug-and-play priors (PnP) convergence analysis clean and simple, motivated by PnP-HQS [Zhang et al., 2017b] and regularization by denoising (RED) [Romano et al., 2017]. Existing works have proved the convergence of PnP and RED with contractive and nonexpansiven denoisers. It is an activate area for designing deep neural net denoisers that ensure PnP/RED convergence. Specifically, this paper focuses on designing denoisers that correspond to the gradient descent step in terms of the $L$-smooth regularizer function $g_\\sigma (x)$. By using such denoising step within PnP followed by the proximal of data fidelity $f$, one can guarantee convergence via traditional non-convex optimization given the objective function $F(x)$.  Since the handy $F(x)$ is available, a standard backtracking scheme is used in this work to ensure convergence without needs for tracking the exact Lipschitz constant of $\\nabla g_\\sigma$. Finally, the performance and stability of the proposed method is evaluated over three image inverse problems such as deblurring, super-resolution and inpainting, with satisfactory results compared to existing methods based on PnP and RED. ", "summary_of_the_review": "This paper is proposing an interesting alternative to training contractive/nonexpansiven denoiers for PnP and RED with convergence guarantee. Although I would expect a little bit more technical depth at least via numerical simulations, I believe this is a well-written paper with some new results that could spark further research in the important topic of designing explicit regularizers for PnP/RED. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1UFTpY3BcZT", "reviewer": "Reviewer_QQES", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis work is well-written and easy to follow for the most parts. \nIt discusses a new denoiser to be employed within a plug-and-play framework, which indeed is a hot topic and an interest of the community. However, the current version of the work is not ready for publication due to following reasons.\n\n\n\n\n**Primary issues:\n\n1 – the proposed regularizer in (5) is obscure and no interpretation around itself is provided. What does it impose on the recovery?  Besides, $N_{\\sigma}$ which seems to play the most important role in (5) is introduced only as “parametrized by a neural network”, and later it is restricted to be a differentiable neural net or “net with differentiable activation functions”; it is unclear what $N_{\\sigma}$ does, and how it is related to (8) and Remark 1. \n\n\n\n2 – The GS-PnP (9) is deviated from being a PnP-HQS while using (4) as the employed denoiser. The author(s) claimed it (“Departing from a slight modification of the PnP-HQS framework….”); however, the right-hand side of (9) seems like a heuristic approach which may lead to a better numerical result by adjusting parameters like $\\lambda$ and $\\tau$ which are not introduced in the denoiser (4).\n\nThe HQS for (1) is either $Prox_{\\tau f} \\circ Prox_{\\lambda \\tau g}$  or  $ Prox_{\\lambda \\tau g} \\circ Prox_{\\tau f} $, where PnP-HQS replaces  \\Prox_{\\lambda \\tau g} with a denoiser $D$. The strength of the filter $D$ is then either determined by $\\lambda$ and $\\tau$ (and the sparsifying transform used within $D$) consistent with the analysis of proximal operator, or one can use a heuristic approach to set the filtering strength (or $\\sigma$) through cross-validation. The results by the right-hand side of (9) might seem appealing (marginally) due to tweaking of the parameters $\\lambda$, $\\tau$ and $\\sigma$. \n\n\n3 – it is not clear whether the proposed GS denoiser (9) is limited to be employed within the HQS (half-quadratic splitting) method or can be used within other recovery frameworks such as ADMM and FBS. For instance, RED is independent of the recovery framework; is the GS (9) the same?\n\n\n\n\n\n4 – the Proof of Theorem 1 (i): First of all it is not clear how (16) is derived, by which the rest are built upon. The (16) can be valid only when “$f(x)$” satisfies restricted strong convexity. While the authors claimed “this new PnP algorithm allows for non strongly convex data terms”.\n“bounded and/or nonexpansive  denoiser”+ “decreasing step-size”+ “Lipschitz constraint”, “These settings are unrealistic as deep denoisers do not generally satisfy such properties.”\n“This comes at the cost of imposing strong convexity on the data term f, which excludes many IR tasks like deblurring, super-resolution and inpainting”\n“One strength of this approach is to simultaneously allow for a non strongly convex (and not smooth) data term …. .”\nNothing has been proved for the cases where “$f(x)$” is not strongly convex; however, the proof of theorem 1 is based on the strong convexity of “$f(x)$” and .\n\n\n\n5– “we plug a denoiser that inherently satisfies equation 3 without sacrificing the denoising performance” + “But imposing hard Lipschitz constraints on the network alters its denoising performance.”\nThe main objection on these claims is that how the performance of the denoising is inspected?\nIndeed, an stand alone denoising task is different from a general image restoration problem (e.g., deblurring, super-resolution, compressive sensing and inpainting) as the underlying degradation within each problem is different. Therefore, using the same denoiser for different image restoration problem is problematic, and expecting that a denoiser tuned for stand-alone AWGN denoising task works well for any IR task is unrealistic. \n\n\n6 – it is not clear how the tuning is performed for the other methods used for the comparison. For instance in Table 2, how are the results obtained by other comparative methods (e.g., IRCNN and DPIR)? Did you use the same set of parameters tuned by their authors for CBSD64 over CBSD10?\n“Due to large computational time of some compared methods ...” please ignore those slow methods (which I think is EPLL providing the least of results – or even RED in deblurring) and obtain the results for the whole CBSD64 for the sake of fair comparison and avoiding possible overfitting. This applies to other experiments too.\n\n**Secondary issues:\n1 – (2) is incorrect! It should be $Prox_{f}(z) = argmin_{x} \\{ \\frac{1}{2} ||x-z||_2^2 + f(x) \\}$ … \n\n\n2 – “Besides, this new PnP algorithm allows for non strongly convex data term, and thus can address ill-posed IR tasks ….” is indistinct and vague!! By “data term” do you mean “data-fidelity term”? If so, how and where non-convexity (or non-strongly convex!) data-fidelity is investigated? \n“Besides, this new PnP allows….” Does it mean other PnPs do not allow such capability but the proposed one does?\n\n“This comes at the cost of imposing strong convexity on the data term f, which excludes many IR tasks like ….” the same issue stated above!\n\n3 – “Though providing excellent restorations, such schemes are not guaranteed to converge for all kinds of denoiser or IR tasks.” is this claim correct? How is it related to your work? Does it mean the proposed work is able to work with any denoiser and for any IR tasks (e.g., compressive sensing)?\n\n\n\n4 – “contrary to Romano et al (2017), our denoiser exactly represents a conservative vector field.” any proof? Does it have a symmetric Jacobian?\n\n5 – “inpainting noise-free input images lead to a non differentiable data term $f$”! why?\n\n6 – what are the number of parameters in Table 1?\n\n\n**Minor issues/modifications/suggestions:\n\n1 – perhaps the regularization parameter in (1) should be positive, i.e. $\\lambda>0$\n\n2–  equation 4, equation (22), (6) … please use a consistent way of citing equations, e.g., use within parentheses.\n\n3– The forward model in page can be formulated in a separate equation due to its frequent use throughout the paper.\n\n4– HQS (half-quadratic splitting) is used but is not cited, please see https://doi.org/10.1109/83.392335\n\n5–  “external denoiser” is vague! Perhaps “generic denoiser” is a better alternative.\n\n6– “exact” proximal mapping is vague! Perhaps “explicit” proximal mapping.\n\n7– “subsequent integration in PnP schemes” + “subsequent PnP image restoration” are vague/incomprehensible!\n\n8– the first paragraph of page 3 is vague! It should be rewritten.\n\n “One can relate the deep denoiser” → “a deep”\n\n “The latter shows that the Minimum ….” what is the former?\n\n What are “$p$” and “$*$”? they are not introduced. Do they correspond respectively to density and convolution?\n\n “intractable” is vague!\n\n9– “equation 4 comes down to including”\n\n10– “Algorithm (9)” typo. Either (9) or Algorithm 1. \n\n11– “is non-increasing and converging” → converging toward?\n\n12– It would be better if only the average results are represented in Table 2, instead of so many numbers.\n\n13– “Both IRCNN and DPIR realize PnP-HQS”\n\n14– “i-e” typo. → “i.e.”", "summary_of_the_paper": "The paper proposed a new denoiser, based on the gradient of a proposed trained regularizer, to be employed within a plug-and-play (PnP) framework, which is claimed to have a convergence proof compared to other existing methods (which claimed to use \"unrealistic\" assumptions). Experiments are provided to support the efficacy of the proposed method.", "summary_of_the_review": "Post-rebuttal:\n\nI thank the authors for their careful responses to the reviewers’ concerns, comments and questions. Their response addresses most of my concerns. I highly recommend to include the responses to the major issues in the paper to rectify the ambiguities, and re-write Section 3 accordingly. I would like to change my evaluation based on the authors' response, to accept.\n\n\n-------------------------------------------------------------\n\nIn a nutshell, the submitted manuscript suffers from the following issues:\n1) lacking a clear motivation/description on the proposed denoiser (4) and (5),\n2) ambiguity of the proposed PnP method (9),\n3) unsupported claims and lacking a clear proof for Theorem 1 by which the contribution of paper is built upon, \n4) insufficient experiments,\nwhich overall could not convinced me to accept the paper, but to evaluate it as “reject”.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a plug-and-play method for solving imaging problems. Plug-and-play methods use a denoiser to solve linear inverse problems. The paper proposes a plug-and-play method and uses convex optimization tools from analyzing proximal gradient methods to provide convergence guarantees. The algorithm is applied to a variety of inverse problems showing that the method works well. 

After the discussion period, all four reviewers recommend acceptance. 
- Reviewer QQES provided a detailed review and raised a few concerns including a clear motivation for and description of the denoiser, and unsupported claims, in particular related to a proof in the paper. The authors revised the paper and responded in length to the claims, in particular they detailed steps and assumptions related to the theorem in their response. As a response, the reviewer changed their score to accept.
- Reviewer xYLt strongly supports acceptance based on the strong theoretical results and a very good exposition.
- Reviewer GZzY likes the overall idea of the paper and raised a few minor concerns and questions, which were addressed by the authors. 
- Finally, reviewer E8QG also appreciates the method, convergence analysis, and extensive validation. The reviewer also raised a few minor concerns and asked for clarification, and the response of the authors resolved those concerns. 

Based on my own reading and based on the reviews, I recommend acceptance. The paper provides a variant of a plug-and-play method, proves interesting convergence results for the method, and has a strong experimental evaluation of the method. I encourage the authors to take the feedback of the reviewers into account, which they have done for the most part already, and it would also be interesting to see the performance of the method for compressive sensing problems.
Decision: Accept (Poster)