It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems
Authors: Regev Cohen
Yochai Blau
Daniel Freedman
Ehud Rivlin
Venue: NeurIPS 2021 Poster (2021)
Keywords: Inverse solutions
denoising
regularization
deep learning

Abstract: In recent years there has been increasing interest in leveraging denoisers for solving general inverse problems. Two leading frameworks are regularization-by-denoising (RED) and plug-and-play priors (PnP) which incorporate explicit likelihood functions with priors induced by denoising algorithms.  RED and PnP have shown state-of-the-art performance in diverse imaging tasks when powerful denoisersare used, such as convolutional neural networks (CNNs). However, the study of their convergence remains an active line of research.  Recent works derive the convergence of RED and PnP methods by treating CNN denoisers as approximations for maximum a posteriori (MAP) or minimum mean square error (MMSE) estimators.  Yet, state-of-the-art denoisers cannot be interpreted as either MAPor MMSE estimators, since they typically do not exhibit symmetric Jacobians. Furthermore, obtaining stable inverse algorithms often requires controlling the Lipschitz constant of CNN denoisers during training.  Precisely enforcing this constraint is impractical, hence, convergence cannot be completely guaranteed. In this work, we introduce image denoisers derived as the gradients of smooth scalar-valued deep neural networks, acting as potentials. This ensures two things: (1) the proposed denoisers display symmetric Jacobians, allowing for MAP and MMSE estimators interpretation; (2) the denoisers may be integrated into RED and PnP schemes with backtracking step size, removing the need for enforcing their Lipschitz constant. To show the latter, we develop a simple inversion method that utilizes the proposed denoisers. We theoretically establish its convergence to stationary points of an underlying objective function consisting of the learned potentials. We numerically validate our method through various imaging experiments, showing improved results compared to standard RED and PnP methods, and with additional provable stability.

It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems
Regev Cohen Verily Research, Israel regevcohengooggle. com
Yochai Blau Google Research, Israel yochaib0google. com
Daniel Freedman Verily Research, Israel danieelfreedman@@oogllee com
Ehud Rivlin
Verily Research, Israel ehud@google. com
Abstract
In recent years there has been increasing interest in leveraging denoisers for solving general inverse problems. Two leading frameworks are regularization-y--deee (RED) and plug-and-play priors (PnP) which incorporate explicit likelihood func- tions with priors induced by denoising algorithms. RED and PnP have shown thie-offttheartt performance in diverse imaging tasks when powerful denoisers are used, such as convolutional neural networks (CNNs). However, the study of their convergence remains an active line of research. Recent works derive the convergence of RED and PnP methods by treating CNN denoisers as approxima- tions for maximum a posteriori (MAP) or minimum mean square error (MMSE) estimators. Yet, date-offthheart denoisers cannot be interpreted as either MAP or MMSE estimators, since they typically do not exhibit symmetric Jacobians. Furthermore, obtaining stable inverse algorithms often requires controlling the Lipschitz constant of CNN denoisers during training. Precisely enforcing this constraint is impractical, hence, convergence cannot be completely guaranteed. this work, we introduce image denoisers derived as the gradients of smooth ecalar-valued deep neural networks, acting potentials. This ensures two things: (1) the proposed denoisers display symmetric Jacobians, allowing for MAP and MMSE estimators interpretation; (2) the denoisers may be integrated into RED and PnP schemes with backtracking step size, removing the need for enforcing their Lipschitz constant. To show the latter, we develop a simple inversion method that utilizes the proposed denoisers. We theoretically establish its convergence to stationary points of underlying objective function consisting of the learned po- tentials. We numerically validate our method through various imaging experiments, showing improved results compared standard RED and PnP methods, and with additional provable stability.
Introduction
General Recovery Inverse problems aim to recover an unknown signal from its observed corrupted or compressed version. This fundamental task appears in a variety of fields such as machine learning, signal processing, computer vision and medical imaging. When model for the degradation process available, one may derive data fidelity signal through a forward operator. However, this term alone typically leads to ill-posed optimization problems with unstable solutions, requiring the incorporation of additional knowledge of the desired
which relates the measurements to the unknown
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

signal into the inversion process. This has given rise to diverse regularization methods which have been investigated extensively over the last few decades.
RED and PnP A very basic inverse problem is denoising, in which a clean signal is to be recovered from its measurements contaminated with additive noise. During the last period of time there has been an overwhelming advancement in the ability to perform denoising, creating techniques that approach optimal performance [1] 29 30 This striking progress in denoising has been the motivation for
To Tl methodology, which introduced the paradigm of using denoisers in order to solve general inverse problems. PnP incorporates implicit regularization into the inverse solutions by replacing proximal operators in common optimization algorithms with powerful denoisers. This simple modification has led remarkable performance in diverse inverse tasks 41 550 (8 [2 156, 3, [S]], as well as to the development of an arsenal of PnP- based techniques which includes proximal-type methods but also goes beyond them 33
the plug-and-play priors (PnP) 48
15
5
55 14 [7].
However, the lack of a clearly formulated regularization term corresponding to the denoiser poses a serious challenge in ensuring the stability of PnP-based methods, and typically requires careful
parameter-tuning [49
To fill this void, the regularization by denoising (RED) framework presents an explicit Laplacian- based regularization that relies on a denoiser. Under certain conditions on the denoiser [3)] 37], the RED functional is convex and its gradient is given by the denoising residual, allowing for easy incorporation in standard optimization procedures. RED has demonstrated impressive recovery performance while providing global convergence guarantees. However, various common denoisers do not satisfy the conditions required by RED 337. In such cases, there exists no objective function which justifies the RED method [37 and the corresponding convergence analysis does not hold. Thus RED, as well as PnP, shows state-of-the-ar empirical results but without provable stability. A comprehensive overview of PnP and RED is given in [3]
Related Work The study of the stability of both RED and PnP methods has become an active line of research in recent years. The convergence of two PnP techniques was proven in [TO] and 47
under the assumption of bounded denoisers. When minimum mean squared eтror (MMSE) denoisers are used, the PnP framework was shown to converge vis-a-vis an explicit objective function (53). Similarly, 7 18 proved the convergence of approximate message passing algorithms with MMSE denoisers for random froward operators. Multiple studies 41 43
45 derived the convergence of PnP by relying on nonexpansive denoisers. The latter has been the core assumption of [8) which explained the PnP framework, including its stability, through the notion of consensus equilibrium. A similar condition was used in (37], which adopted the score-matching framework to provide new interpretation of the RED algorithms and their convergence. [42 presented a block-coordinate RED denoisers. The work of (36
method and derived its fixed-point convergence for introduced a convergent PnP-type method using a maximally monotone denoiser, created by training nonexpansive neural network in supervised fashion. Alternatively, [40] provided convergence guarantees for certain PnP variants that use trained denoisers whose residuals are nonexpansive. This type of denoiser was used in 27 to develop Bayesian inference with PnP priors, including the
extension of the PnP framework to Monte Carlo sampling. The authors of 24 23 employ denoisers within Langevin dynamics to sample from the posterior distribution of any linear inverse problem. A different strategy was studied in (35. which proposed a special inner product derived from the given linear denoiser, and thus proved the convergence of PnP based on the theory of averaged nonexpansive operators. Recent work 11) introduced RED via fixed-projection framework (RED-PRO), which utilizes the fixed-points of the denoisers as regularization. RED-PRO unifies RED and PnP and derives their convergence for general demicontractive denoisers, including nonexpansive ones.
Recent studies, mentioned above, are motivated by or rely on the assumption that the denoiser is an MMSE or maximum a posteriori (MAP) estimator. However, such estimators are in the form of gradients, hence, they must exhibit symmetric Jacobians (0 19 21 amongst other properties.
Furthermore, Jacobian symmetry offers additional advantages as it often leads to stable algorithms and allows one to better understand the behavior of such non-linear estimators (4) Unfortunately, many state-of-the-art denoisers do not fulfill this condition (37. An additional drawback of current RED and PnP methods is that the Lipschitz constant of the denoiser often has be controlled to create nonexpansive operator, which in turn ensures convergence. This is typically done by learning deep network and adding regularization term to the training loss function, whose purpose is to bound the Lipschitz constant of the resultant denoiser. However, exactly enforcing this property of the denoiser is impractical and is only approximated in practice.

---

Contributions In this work, we develop provably convergent technique which exploits denoisers for solving general inverse problems. (I) We introduce image denoisers that are constructed as the gradients of smooth potential functions, represented by deep neural networks and trained in supervised fashion. (2) Equipped with these learned potential networks, we formulate a general inverse framework where we utilize our denoising potentials as an explicit regularization. (3) We use the existence of an underlying objective function to incorporate the proposed denoisers into standard optimization schemes with backtracking step size. Specifically, we present simple gradient descent algorithm where the step size is tuned automatically to provably guarantee convergence to stationary points of the objective function. (4) We numerically validate our method through imaging experiments, showing improved results compared to RED and PnP, with additional provable stability.
Background
Inverse Problems: Formulation We consider the recovery of an unknown signal x" E R from its corrupted observation y E RR The inversion task is traditionally formulated as an optimization problem of the form
X = arg min f(x;y) + ^x(x),
xE
where y(::y) is fidelity term related to the observations y. 9()) is a regularization functional that incorporates prior information or known statistics of the true signal x* to avoid ill- ill-posed problems, and À >> 0 balances between the terms. A common model for the measurements is Ax + w where A a linear degradation (forward) operator and w is an i.i.d white Gaussian noise, leading to the following oog-liielhood function
f f(x;y) - 1
- y11..
The above model applies to a wide variety of inverse problems such as compressed sensing, super resolution, deblurring, inpainting and much more.
In the following, we consider a general functional f(.) where we suppress its dependency on the observations y for brevity, Moreover, both g(-) and f(-) are typically assumed be convex where the former is generally non-differentiable, while the latter is a smooth function. By the definition of smooth functions, there exists Lipschitz constant L such that the gradient Vff of f(-) satisfies
[x) (x2))) S | x2ll. Vx,, X2 E Rdd..
The minimization problem ① has been investigated extensively over the decades, leading a series of diverse approaches. Many common optimization techniques for solving (1) rely on the proximal operator of g(.). defined as
Proximal Methods
Prg (x) A arg min zER
; ||x 2|| + Ag(z).
The proximal operator, which is a MAP estimator, provides an efficient way overcome the lack of differentiablity of g()), especially when it admits a closed-form expression, for example in the case of the E -oomm. A simple representative of this family of algorithms is the proximal gradient descent (PGD) method which updates the solution iteratively as follows
X
Pxx Xx
pa ((xx))
(5)
where μı > 0 is the step size. PGD has shown great performance in various applications along with strong convergence guarantees.
The PnP Framework
Notice that the proximal operator can be seen, by definition, as the solution to denoising problem. This observation serves the basis of the PnP framework which extends standard proximal methods by replacing the proximal operator with powerful denoisers. Following this rationale, (5) is modified to create the PnP-PGD method that performs the following update rule
X D pa (fxx))
(PnP-PGD)
where D ; 42 - d is an off-the-shellf denoiser.
3

---

The RED Framework An alternative strategy for leveraging advanced denoisers is the RED framework which utilizes the denoiser to construct an explicit regularization g(x) (x. D(x)). Under certain conditions on the denoiser [39 37 g(-) is convex and its gradient is readily given by
the denoising residual Vg(x)= = x (xx). This allows for the solution of [
by standard algorithms,
e.g., steepest (or gradient) descent
MC X X I+ A
+ ( (o(x-))())) D.
(RED-SD)
Surprisingly, RED-based methods have been found to be effective also when the required conditions on the denoiser are not satisfied and the expression for the regularization term is invalid. In such cases, RED uses the denoiser an implicit regularization, similar PnP.
Challenges RED and PnP Both RED and PnP provide extremely efficient means for exploiting powerful denoisers solve inverse problems. Yet their analysis poses two closely related challenges:
motivate or base their derivations on the assumption that the denoiser is an approximation of either MAP or MMSE estimator. It has been proven that when the Jacobian of such estimators is well-defined it must be symmetric 200 19 211. However, RED and PnP typically reach state-of-the- results by relying on denoisers which do not exhibit this property, as in the case of the popular DnCNN technique; result, they lose their interpretation as optimization solvers. In such cases, while there are alternative optimization-free explanations for RED and PnP (7). it is challenging to properly analyze their behavior and characterize the nature of their solutions.
1. Asymmetric Denoisers: Many studies RED and PnP 53
27 43
45
52
22
2. Convergence Issues: Continuing the above line of reasoning, often there exists regularization that explains the denoiser, namely no functional g(-) for which (x( D(x) or Vg(x) ((x)
[37. This lack of potential function creates difficulties in studying the theoretical convergence of the RED and PnP algorithms. However, a common sufficient condition to guarantee convergence is
for the denoiser D--) be (averaged) nonexpansive [3] implying that
[[D(x,) D(x2)ll - L| X2ll. Vx1, E
(6)
for some Lipschitz constant 0 L 1. However, most predefined denoisers do not satisfy this is to learn nonexpansive denoiser by adding regularization term the training loss which constrains its Lipschitz constant. Unfortunately. this remains an open challenge since in general there are no practical means to absolutely enforce the resultant denoiser to be nonexpansive.
assumption. An alternative approach 40
36
46
Our work addresses both of these challenges. We first introduce a new class of denoisers which stem from an explicit potential functional and thus demonstrate symmetric Jacobian by their construction. We then derive an inverse technique that utilizes our denoisers as well as their potentials to ensure convergence. Three important remarks are to be made with regard to work presented below. First, we emphasize that the proposed denoisers are not intended to compete with state-of-the-art denoisers, but rather are developed to exhibit favorable properties that benefit other inversion tasks. Second, our proposed recovery method is not derived to compete with end-to-end learned solutions to specific inverse problems [1 2 25 26). Rather, our learned denoisers, along with optimization framework, can be used for all inverse problems (of the form in Eq. [ without additional training. Lastly, while we present present specific inversion method, the potential-driven denoisers can benefit many RED and PnP algorithms, hence, our work complements these frameworks.
Recovery by Potential-Driven Denoisers
In this section, we present our denoisers driven by learned potential functions. Designed as gradient operators, the denoisers display symmetric Jacobians, allowing for a MAP or an MMSE estimator interpretation. More importantly we use the denoising potentials as an explicit regularization for general inverse problems, leading to provably convergent optimization schemes which utilize our proposed denoisers. Unlike previous works, we do not require bounding or approximation of the Lipschitz constants the denoisers.
3.1 Eikential-Driven Denoisers
MAP and MMSE Denoisers as Gradients Two fundamental forms of denoisers are MAP and MMSE estimators. The theoretical properties of MAP and MMSE denoisers are well-understood and

---

they serve as the basis of many works which study the convergence of RED and PnP methods. The following lemmas reveals that MAP and MMSE denosiers share a common structure:
Lemma 1 (Corollary 1211). Assume the denoiser is a proximal operator DD - Pe of some scalar- valued function @. Then, there exists a convex scalar-valued function (+)) such that Vab.
Lemma 2 (Tweedie's Formula [6]). Assume the denoiser is an MMSE estimator DD of images contaminated with white Gaussian noise of variance o². Then, 'D 2 I + o logp(-) where I is the identity operator and p(-) denotes a smooth probability density of images.
The above results imply that for any MAP or MMSE denoiser D there exists scalar-valued function o such that the denoiser, or equivalently its residual, can be expressed as D = Vp. This observation sets the main foundation of our gradient-driven driven denoisers, described next.
Construction of Gradient-Based Denoisers Now, we consider the task of training a deep network to perform denoising. While we focus on image denoising, the following derivations can be gen- eralized to any signal of interest. Motivated by the above observation, we design a denoising deep network which is the gradient of scalar-valie valued function, acting as potential function. Three possible approaches to construct gradient-driven denoisers are as follows:
1. Directly design deep network which is the gradient of certain potential function. This approach avoids building explicitly the potential network. In the Supplemental we describe certain conditions under which a given network is gradient.
Compose a potential function using an arbitrary deep network N₂ whose input an image. Simple examples of this approach are (x, Nx(x)), || (x) P. ||x (x)]l², etc. Note that in general the network No does not equal to the gradient the composite potential.
3. Model the potential function as scalar-valued deep network and use its gradient as the denoiser.
Here we focus on the last approach since it is the most general. We construct potential function g+(+) E IR defined as deep neural network with parameters 0. As illustrated in Fig. | the potential network follows a typical structure of ceed-forward network with skip-connections except from two main modifications:
The output of the network a single scalar.
The acitivations are continuously differentiable and smooth functions.
Mathematically, the potential network consists of M > layers described recursively as follows
7m 8m (Wim.m 7m + V. x + bm) m = 0,..., M 1
c. are the hidden variables where Zu = 0 and go (x) ZM the output of the network. The parameters of the network are 0 2 (bm, Win Vm, } )m=0 where bm is bias term and Wm, Vm are linear operators with Wo = 0. The activation functions are denoted by 8m. R
Here, x R the input noisy image, Zm
+ R and applied element-wise.
To allow the network to accept inputs of variable sizes, we choose the linear mappings to be standard convolutions. We ensure the output of the network a scalar by setting the number of output channels of the last convolution to be one, and using global average pooling as the final activation function. Another important design choice is that all activations are continuously differentiable and smooth functions. This creates a potential network that is smooth as the composition of smooth functions, which is crucial for guaranteeing convergence of our inversion method, presented in Section 3.2
Given the potential function (-). we define our denoiser Do(-) as
De (x) A gx).
Alternatively, we can write or R, Ve where RR is the denoising residual. This construction ensures that the resultant denoiser, termed gradient-driven denoising CNN (GraDnCNN), is indeed gradient of an explicit function, and thus it possesses a symmetric Jacobian. This form of gradient- driven denoisers represents more faithfully MAP and MMSE estimators, hopefully leading to better
5

---

Ch
128 Ch
Smooth Activattonn
Smoplh Aptivation
Global Pooling
X
Conv
Conv
Conv
- (x)
120 Ch
120 Ch
Ch
Conv
363 Convolution, Strde
Activation
Conv 36d Convolution, Strade
Global Pooting
Gioteal Ausreega Pozting
Figure 1: Network Architecture.
approximations. Finally, we perform supervised residual learning and train the denoiser (or equiv- alently the potential) by minimizing chosen loss function L(', with respect to the parameters
0" arg min Ex(x, x Ro(xx)),
where {x,, x are pairs of clean and noisy images respectively.
Special Potentials: Convex Functions and Difference Convex Functions An additional inter- esting path we follow is setting all (Wm)) (blue blocks in Fig. DD to be non-negative and all activation functions { (red blocks Fig. D to convex and non-decreasing mappings. This creates potential function ge (-) which is input convex neural network (ICNN) 44, leading to denoiser called denoising ICNN (DnICNN). As we explain later in Section 3.2] the use of convex potentials leads to convex inverse problems, thus it facilitates the optimization process and allows to derive recovery methods with global convergence guarantees.
The down side of convex potentials is that the non-negative weights (Wi)) } may limit the expressivity of the network. Hence, we propose an alternative where the potential network is defined as the difference of two ICNNS
(x) - go. (x) gog (x).
(10)
The resultant potential enjoys favorable properties as the difference of two convex functions 28 and can approximate non-convex functions (unlike the convex potentials). Thus, we present third denoiser, referred to as denoising by difference of ICNNs (DnDICNN), which acts as a compromise between GraDnCNN and DrICNN, In the following section, we study the properties of these three denoisers in the context of our recovery scheme, both in terms of stability and recovery performance.
3.2 Convergent Inversion Method
Inversion by Descent Given trained gradient-driven denoiser, we wish to exploit the denoising power within general recovery processes. The existence of an underlying learned potential go (-) allows us formulate the recovery as the following problem:
arg min (xx + Ago
(11)
xER4z
Here we assume f(-) is continuously differentiable smooth function, typically the log- likelihood
of the observations given by (2). The potential network acts as a regularization term or a smooth image prior which creates well-posed problem. Furthermore, by our construction the potential go (-) is smooth, thus, the complete objective, denoted by F(-). is smooth with some unknown Lipschitz constant 0. This facilities optimization process considerably and we can solve simple gradient descent algorithm
[T] via
X6+1 X₁ l4k V (Xx)),
(12)
where we start with some initial solution xp. Notice the above iteration can be written as
X
N₁
(v((x)) A(xx - Do (xx))
(13)

---

where Do(-) is our denoiser originating from go(-).
Backtracking Step Size Note that the step size μt must be set properly reach convergence. Typically, one only has access to the denoiser itself, hence the step size is tuned empirically which may lead either divergence or slow convergence due an overly small step size. By contrast, we have complete access to the objective function, thus, we can properly set the step size to ensure convergence. To that end we rely on the Descent Lemma 6 in the Supplemental, which implies that
Mkk. [ Vpu :0 :0 μi S T.
1
F( S FXx))
(14)
When (x()))) # O we obtain that 1 < F(x:): otherwise, we reach critical point. The challenge that remains setting the step size according the Lipschitz constant which is typically unknown. To address this, we set backtracking step size defined as 14k _ 1/f. where f. is an approximation of the true constant L. We start with some initial guess = Lu and at each iteration update L. and ja accordingly until inequality 14) is satisfied, ensuring we perform a descent step.
Algorithm 1: Regularization by Sitential-Drivens Denoising
Input: initial point Xo E RR Lo 0, B 1, À > 0, f()), go(-) and the related denoiser Do(-). Set F()) A f()) + >ge(.) and L = Lo.
for k=0, 1,2,... do
[ 1/LL.
X4+1 Xx. - M4 [V](xx)) + A(xx Dz (xx))) while F(xx+1) > F(xx)= 41 (((xx) 31 do
i. == B- 14k = 1/i:
N:- = X4 jau Vf((x)) + M(xe - Do(x)))).
The resulting inversion technique, outlined in Algorithm I can be viewed as RED-SD with backtrack- ing step size which is enabled by the existence of an available prior function ge(-). In addition, note that throughout the iterations it holds that i. S max BL) where L is the true unknown Lipschitz constant. Also note that the total number of updates of î. bounded by log, xi Ollog, L).
Convergence Guarantees We remark that other more sophisticated optimization techniques can be considered for solving (II). including acceleration schemes. However, rather than providing the most efficient recovery method, we focus here on simplicity to highlight the benefits of learned potential-driven denoisers. Moreover, there is vast literature which provides convergence guarantees for Algorithm[]] Junder jundr various conditions. In the following theorem we present a standard convergence analysis based on the smoothness of the objective function. The proof is given in the Supplemental.
Theorem 1. Assume that f(-) and go(-) ) are continuously differentiable and smooth functions. Define F(.) f(-) )go(.) (-) and assume it is bounded below by F*. Let (xx) be the sequence generated by Algorithm 7 Then.
min F 113 max Lo, (L) (F(xo) F²).
h:
The above theorem implies that FX - 0 as + do and that all limit points of the sequence {xg J220 stationary points of the objective function F(.). Notice this result applies all the three denoisers (or potentials) presented in Section[.]] This emphasizes the importance of the smoothness of the potential functions which ensures the convergence of the recovery, regardless to the highly non-convex nature of the objective function.
Commonly, the function f(() is convex function, e.g. when it is the log-likeli In such scenarios, the use of convex potential networks leads to a special case where the recovery problem is convex and we obtain global convergence guarantees:
Corollary 1. Assume that f(-) and go (.) are convex in addition to the conditions of Theorem Then, A 1 converges to global minima of the objective function F(.).
The proof is straightforward since according to Theorem the sequence generated by Algorithm converges to stationary points of the objective function, and any stationary point of convex function
7

---

is a global minimizer. Thus, in the context of inverse problems, our convex potential networks connect the well-known world of convex optimization and with the highly non-linear non-convex nature of deep neural networks.
Finally, for the completeness of our work, we study the case where the recovery problem includes additional constraints on the solution, leading to the following formulation
arg min f(x) + (g) + h(x),
(15)
xEX
where the function h(-) is convex yet possibly non-smooth. We provide the full analysis consisting of algorithmic and theoretical results in the Supplemental.
4 Numerical Experiments
Here we study the performance of A Juiig our three proposed potential-driven denoisers: GraDnCNN, DnICNN, DnDICNN. We compare ourselves to PnP-PGD and RED-SD, applied with the popular DnCNN denoiser (54) for the tasks of Gaussian deblurring and image super resolution.
Datasets For training the denoising networks for blind Gaussian denoising we use the public DIV2K dataset (2 that consists of total of 900 high resolution images, 800 for training and 100 for validation. We perform data augmentation by randomly cropping 100 patches of size 50 x 50 from each image of the DIV2K training set. We then contaminate each patch with an additive i.i.d. white Gaussian noise (WGN) with noise level an chosen uniformly at random from the range [1, 55), creating our 80,000 sample training set. For our validation set, we replicate each image of the DIV2K validation set 10 times, and we add WGN to each copy with random noise level in the specified range. As our test images for evaluating the inverse methods for deblurring and super resolution, we use two different datasets. The first set, provided by the authors of (39]. consists of 11 common color images (butterfly, starfish, etc), and we refer natural images from the public Berkeley segmentation dataset (BSD500) [3]. All pixel values of the images described above are in the range [0, 255].
as Setl For more through assessment, we utilize all 500
Training Denoisers We experiment with four different denoisers. The first is a variant of the DrCNN-B architecture (54) with 7 layers, 128 feature maps, and x 3 convolution kernels. For GraDnCNN we use the architecture shown in Fig. consisting of M = 6 layers, ELU activation functions, 128 feature maps, and 3 3 bias-free convolutions where the final two convolution layers have a single output channel, and the last activation is global average pooling to produce a sealar. A similar architecture is used for DnICNN, where we set certain weights to be non-negative. The latter architecture is then used to construct our DnDICNN as the difference of two DnICNNs.
Given the datasets detailed above, train each of the networks using Adam optimizer for 100 epochs with a constant learning rate of 10-3, 3, For the training loss, we use a modified version of mean squared error (MSE) cost function: Σn
d MSE X,, x,, Ro (x,)) where x², denotes clean patch, X, is the corresponding noisy patch contaminated with noise of level on, and Ro (xx) is the output of the network. Since samples with larger noise levels typically lead to larger training errors, we find that this modification of dividing the MSE by the noise level is required to prevent the denoisers from
over-smoothing, All experiments are performed in Tensorflow I where each model is trained on a single NVIDIA Tesla 32GB V100 GPU.
Inverse Problems We focus here on two common inverse problems: Gaussian deblurring and single image super resolution. For the task of deblurring perform the following process on the test datasets (Set1 BSD500) described above. Each image is convolved with a 2D Gaussian function 25 x 25 with a standard deviation of 1.6. Then, we add to the result an i.i.d. WGN of level = v2. For super resolution we follow the same procedure except we add an intermediate step, before noise contamination, in which we scale-down the image by a factor of 3 at each axis.
Recovery Given the degraded test images and the trained denoisers, we compare five different inverse methods: PnP-PGD and RED-SD, both with DrCNN, as well as our three potential-driven denoisers. For [ we use the constrained update rule, described in the Supplemental, where the proximal operator is the projection of pixel values of the solution into the range [0, 255]. We apply the same projection each intermediate solution of PnP-PGD
with
8

---

and RED-SD for improved stability. For properly setting the hyperparameters of the different tested methods we use Setll to perform an extensive hyperparameter search, provided in the Supplemental.
We first study the convergence by running all the techniques for fixed number of 400 iterations for both inverse tasks, where measure convergence by Xx++1 x sxll/Me. The results, averaged over each dataset separately, are given in F and 3 The foremost observation is that in all the scenarios PnP-PGD diverges at the certain point. We conjecture the reason of this instability of PnP-PGD lies in the fact that unlike the other methods PnP-PGD includes no hyperparameters for constraining the denoiser, thus, powerful denoisers may lead divergence by analogy with too large learning rate. This can be prevented by considering more sophisticated version of PnP or by performing early stopping. however, will introduce additional hyperparameter which need to be properly tuned for each data instance, making the technique less robust. Moreover, notice that convergence of Algorithm does not depend on the values of À and μ4, regardless of the potential-driven denoiser in use. This is desired property that most RED and PnP methods do not exhibit. We note that the convex potential leads to fastest convergence which is expected, since in this case the objective function is smooth and convex, facilitating the optimization process significantly.
Next, we quantitatively compare the inverse algorithms by computing the average peak signal noise ratio (PSNR) and structural similarity index (SSIM) of their solutions with respect to the ground truth images. As seen Table Algorithm with GraDnCNN produces the best performance in both tasks and for both test sets. Algorithm| with DrICNN yields the worst results among our potential-driven denoisers, yet comparable to PnP-PGD and RED-SD with DnCNN. The performance of Algorithm|] I with DrDICNN is consistent with the fact that DnDICNN presents compromise between GraDnCNN and DrICNN in terms the expressivity of the network. To visually support our claims, we provide selected recovered images in Fig. 4 while additional results are given in the Supplemental. An important remark is that in contrast to PnP and RED methods, presented here and many others, the solutions of Algorithm T and its performance are completely independed of the choice of the step-size μ. Hence, Algorithm is more robust.
PaP-PGD (DACNN)
PnP-PGD (DnCNN)
10¹
RED-SD (DnCNN)
RED-SD (DnCNN)
Algo. (GraDnCNN)
10a
Algo. (GraDnCNN)
Algo. (DnICNN)
Algo. 1 (DnICNN)
10a
Algo. (DADICNN)
Aloo (DnDICNN)
10a
10A
10"
150
200
400
300
# Iterations
Iterations
Figure Average Convergence for Setll (Left) Gaussian deblurring (Right) Super resolution
Table Average PSNR SSIM obtained for different methods and datasets.
PnP-PGD
RED-SD DrCNN
Algorithm 1 DrICNN
Datasets
DnCNN
GraDnCNN
DrDICNN
Gaussian Deblurring
Setll
28.81 08.81/0.80 0.86
28.69 0.82
29.42 0.86
28.91 0.85
29.07 0.84
BSD500
28.79 0.84
28.32/0.79
28.91 0.83
28.63/0.82
28.41 0.80
Super Resolution
Set1l
/ 0.77
26.14/ 0.76
26.69 0.79
25.95 0.76
26,20 / 0.76
BSD500
26.36 0.75
26.24 0.73
26.57 0.74
26.15/0.73
25.95/ 0.71

---

2785
451
2000
1000
RED-SD (DACNN)
200
RED-SD (DnCNN)
Algo.
(GraDnCNN)
Algo. 1 (DnICNN)
100
Algo. 1 (DnIENN)
200
Algo.
50
(DNDICNN)
50
10
20
10
2000
250
300
250
300
352
4.0
Iterations
Iterations
Figure 3: Average Convergence for BSD500 (Lefft) Gaussian deblurring (Right) Super resolution.
5 Conclusion
In this paper we developed provably convergent method with explicit denoising regularization for solving inverse problems. To that end, we first introduced the notion of potential-driven denoisers, ie. denoisers which are based on the gradient of learned scalar-valued networks. We then utilized our denoisers along with their potential functions to derive an inverse gradient descent method with backtracking step size, ensuring convergence. We demonstrate experimentally that the resulting methods provide improved performance compared standard RED and PnP methods, while offering provable stability.
The developed framework complements RED and PnP as potential-driven denoisers can be used within any RED or PnP techniques guarantee convergence, regardless of the data being used. Furthermore, while our work focus on learning denoisers, the concept of potential-driven networks may be generalized to benefit other image-to-image operations.
PnP-PGD (DnCNN) PSNR: 29.28. SSIM: 0.88
RED-SD (DnCNN) PSNR: 29.23, SSIM: 0.85
Algo. (GraDrCNN) PSNR: 30.05, SSIM: 0.89
Algo. (DriCNN)
Algo. 1 (DnDICNN) PSNR: 29.73, SSIM: 0.87
PSNR: 29.37, SSIM: 0.8B
PnP-PGD (DnCNN)
RED-SD (DnCNN)
Algo. (GCDDDCNN)
Algo (DniCNN)
Algo. (DnDICNN) 24.
Figure Recovery results the tasks of (top) Gaussian deblurring and (bottom) super resolution.
10

---

References
Abadi, M.. Barham, P.. Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving. G., Isard, M., et al.: Tensorflow: system for large-scale machine learning. In: Symposium on Operating Systems Design and Implementation. pp. 265-283 (2016)
[2] Agustsson, E., Timofte, R.: Ntire 2017 challenge on single image super-resolution: Dataset and study. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. pp. 126-135 (2017)
[3] Ahmad, R., Bouman, C.A., Buzzard, G.T., Chan, S., Liu, S., Reehorst, E.T., Schniter, P:: Plug-and-play methods for magnetic resonance imaging: Using denoisers for image recovery. IEEE Signal Processing Magazine 37(1), 105-116 (2020)
[4] Amos, B., Xu, I.,, Kolter, J.Z.: Input convex neural networks. In: International Conference on Machine Learning. pp. 146-155. PMLR (2017)
[5] Bayati, M., Montanari, A:: The dynamics of message passing dense graphs, with applications compressed sensing. IEEE Transactions on Information Theory 57(2), 764-78 (2011)
[6] Beck, A.. Teboulle, M.: Gradient-based algorithms with applications signal recovery. Convex optimiza- tion in signal processing and communications pp. 42-88 (2009)
Berthier, R., Montanari, A., Nguyen, P.M.: State evolution for approximate message passing with non- separable functions. Information and Inference: A Journal of the IMA 91), 33-79 (2020)
[8] Buzzard, G.T., Chan, S.H., Srechari, S., Bouman, C.A.: Plug-and-play unplugged: Optimization -rree reconstruction using consensus equilibrium. SIAM Journal on Imaging Sciences 11(3), 2001-2020 (2018)
[9] Chan, S.H.: Performance analysis of plug-and-play ADMM: A graph signal processing perspective. IEEE Transactions on Computational Imaging 5(2), 274-286 (2019)
[10] Chan, S.H., Wang, X., Elgendy, O.A.: Plug-and-play ADMM for image restoration: Fixed-poing conver- gence and applications. IEEE Transactions on Computational Imaging 3(1), 84-98 (2016)
[11] Chatterjee, P., Milanfar, P.: Is denoising dead? IEEE Transactions on Image Processing 19(4), 895-911 (2009)
[12] Cohen, R., Elad, M., Milanfar, P:: Regularization by denoising via fixed-point projection (RED-PRO). arXiv preprint (XXi:20008.00220 (2020)
[13] Combettes, P.L., Pesquet, J.C.: Fixed point stratogics in data science. IEEE Transactions on Signal Processing (2021)
[14] Danielyan, A., Katkovnik, V., Egiazarian, K.: Image deblurring by augmented Lagrangian with BM3D frame prior. In: Workshop Information Theoretic Methods in Science and Engineering (WITMSE). Tampere, Finland. pp. 16-18 (2010)
[15] Donoho, D.L., Maleki, A., Montanari, A.: Message- passiin algorithms for compressed sensing. Proceed- ings of the National Academy of Sciences 106(45), 18914-18918 18919 (2009)
[16] Efron, B.: Tweedie's formula and selection bias. Journal of the American Statistical Association 106(496), 1602-1614 (2011)
[17] Egiazarian, K., Katkovnik, V.: Single image super-resolution via BM3D sparse coding. In: IEEE European Signal Processing Conference (EUSIPCO), pp. 184992855, (2015)
[18] Fletcher, A.K., Pandit, P., Rangan, S., Sarkar, S., Schniter. P.: Plug-in estimation digh-diimnsional linear inverse problems: A rigorous analysis. In: Advances in Neural Information Processing Systems. pp. 7440-7449 (2018)
[19] Gribonval, R.: Should penalized least squares regression be interpreted as maximum a posteriori estimation? IEEE Transactions on Signal Processing 59(5). 2405-20220 2410 1)
[20] Gribonval, R., Nikolova, M.: On Bayesian estimation and proximity operators. Applied and Computational Harmonic Analysis (2019)
[21] Gribonval, R., Nikolova, M.: A characterization of proximity operators. Journa of Mathematical Imaging and Vision 62(6). 773-789 (2020)

---

[22] Kamilov, U.S.. Mansour, H. Wohlberg, B.: A plug-and-pla priors approach for solving nonlinear imaging inverse problems. IEEE Signal Processing Letters 24(12). 1872-1876 (2017)
[23] Kawar, B.. Vaksman, G., Elad, M.: SNIPS: Solving noisy inverse problems stochastically. arXiv preprint arXiv:2105.1493 (2021)
[24] Kawar, B., Vaksman, G., Elad, M.: Stochastic image denoising by sampling from the posterior distribution. arXiv preprint arXiv:2101 0..0000..1222210 (2021)
[25] Koh, J., Lee, J., Yoon, S.: -iagle deblurring with neural networks: A comparative survey. Computer Vision and Image Understanding 203, 103134 (2021)
[26] Lai, W.S., Huang, J.B., Hu, Z., Ahuja, N., Yang, M.H.: A comparative study single image blind deblurring. In: Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition. pp- 1701-1709 (2016)
[27] Laumont, R., De Bortoli, V., Almansa, A., Delon, J., Durmus, A., Pereyra, M.: Bayesian imaging using plug & play priors: when Langevin meets Tweedie. arXiv preprint arXiv:2103.04715 (2021)
[28] Le Thi, H.A., Dinh, T.P.: DC programming and DCA: thirty years of developments. Mathematical Programming 169(1) (),, 5-68 (2018)
[29] Levin, A., Nadler, B.: Natural image denoising: Optimality and inherent bounds, In: IEEE CVPR. pp. 2833-2840 (2011)
[30] Levin, A.. Nadler, B., Durand, F., Freeman, W.T.: Patch complexity, finite pixel correlations and optimal denoising. In: European Conference on Computer Vision. pp. 73-86. Springer (2012)
[31] Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 136-144 (2017)
[32] Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ccological statistics. In: Proc. 8th Int'l Conf. Computer Vision. vol. 2, pp. 416-423 (July 2001)
[33] Metzler, C.A., Maleki, A., Baraniuk, R.G.: From denoising to compressed sensing. IEEE Transactions on Information Theory 62(9). 5117-5144 (2016)
[34] Milanfar, P:: Symmetrizing smoothing filters. SIAM Journal on Imaging Sciences 6(1), 263 284 (2013)
[35] Nair, P., Gavaskar, R.G., Chaudhury, K.N.: Fixed-point and objective convergence of plug-and-play algorithms, IEEE Transactions on Computational Imaging 7, 337-348 (2021)
[36] Pesquet, J.C., Repetti, A., Terris, M.. Wiaux, Y.: Learning maximally monotone operators for image
recovery. arXiv preprint arXiv:2012.13247 (2020)
[37] Rechorst, E.T., Schniter, P.: Regularization by denoising: Clarifications and new interpretations. IEEE Transactions computational imaging 5(1), 52-67 67 (2018)
[38] Rick Chang, J., Li, C.L.,, Poczos, B., Vijaya Kumar. B., Sankaranarayanan, A.C.: One network to solve them all: Solving linear inverse problems using deep projection models. In: Proceedings of the IEBE International Conference on Computer Vision. pp. 5888-5897 (2017)
[39] Romano, Y., Elad. M., Milanfar, P.: The little engine that could: Regularization by denoising (RED). SIAM Journal on Imaging Sciences 10(4), 1804-1844 (2017)
[40] Ryu, E., Liu, J., Wang, S., Chen, X., Wang, Z., Yin, W.: Plug-and-play methods provably converge with properly trained denoisers. In: International Conference on Machine Learning. pp. 5546-5557. PMLR (2019)
[41] Srechari, S., Venkatakrishnan, S.V., Wohlberg, B., Buzzard, G.T., Drummy, L.F., Simmons, J.P., Bouman, C.A.: Plug-and-pel priors for bright field electron tomography and sparse interpolation. IEEE Transactions on Computational Imaging 2(4), 408 423 (2016)
[42] Sun, Y., Liu, J.. Kamilov, U.S.: Block coordinate regularization by denoising. In: Advances in Neural Information Processing Systems. pp. 380-390 (2019)
[43] Sun, Y., Wohlberg. B., Kamilov, U.S.: An online plug-and-play algorithm for regularized image recon- struction. IEEE Transactions Computational Imaging 5(3), 395 398-408888099 (2019)
12

---

[44] Teodoro, A.M., Bioucas-Dias. J.M., Figueiredo, M.A.: Scene-adapted plug-and-play algorithm with convergence guarantees. In: IEEE 27th International Workshop Machine Learning for Signal Processing (MLSP), pp. 1-6 (2017)
[45] Teodoro, A.M., Bioucas- Diass, J.M., Figueiredo. M.A.: A convergent image fusion algorithm using scene- adapted Gaussian mixture-based denoising. IEEE Transactions on Image Processing 28(1). 451-463 (2018)
[46] Terris, M., Repetti, A., Pesquet, J.C., Wiaux, Y: Building firmly nonexpansive convolutional neural networks. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8658 8662 (2020)
[47] Tirer, T., Giryes, R.: Image restoration by iterative denoising and backward projections. IEEE Transactions on Image Processing 28(3), 1220-1234 (2018)
[48] Venkatakrishnan, S.V., Bouman, C.A., Wohlberg, B.: Plug Plug-and-pul and-play priors for model based reconstruction. In: IEEE Global Conference on Signal and Information Processing. pp. 945 948 (2013)
[49] Wei, K., Aviles-Rivero, A., Liang, J., Fu, Y., Schönlieb, C.B., Huang, H.: Tuning-free plug-and-play proximal algorithm for inverse imaging problems. In: International Conference on Machine Learning, pp. 10158 10169. PMLR (2020)
[50] Wu, Z., Sun. Y., Liu, J., Kamilov, U.S.: Online regularization by denoising with applications to phase retrieval. In: Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0 (2019)
[51] Xing. J., Kamilov, U. Wu, W.. Wang, Y., Zhang, M.: Plug-and-play priors for reconstruction cconstruction-based based placental image registration. Smart Ultrasound Imaging and Perinatal, Preterm and Paediatric Image Analysis, pp. 133-142. Springer (2019)
[52] Xu, Liu, J., Sun, Wohlberg, Kamilov, U.S.: Boosting the performance of Plug-and- and-Play Priors via denoiser scaling. arXiv preprint arXiv:2002.1 1146 (2020)
[53] Xu, Sun, Liu, J., Wohlberg, B.. Kamilov, U.S.: Provable convergence of plug-and-pla priors with MMSE denoisers. IEBE Signal Processing Letters 27, 1280-1284 (2020)
[54] Zhang, K., Zuo, Chen, Y., Meng, D., Zhang, Beyond Gaussian denoiser: Residual leaminng of deep CNN for image denoising. IEEE Transactions on Image Processing 26(7). 3142-3155 3155 (2017)
[55] Zhang, K., Zuo. W.. Gu, S., Zhang, Learning deep CNN denoiser prior image restoration. In: Proceedings of the IEEE CVPR. pp. 3929 3938 (2017)
[56] Zhang, K. Zuo, W., Zhang, Deep plug-and and-play super-resolution for arbitrary blur kerels.. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1671-1681 (2019)
13


Reviews:
[{"review_id": "REzBeU8TQC8", "reviewer": "Reviewer_Lp9c", "summary": "\nThis paper considers the problem of designing an image denoiser that can be used as a prior within plug-and-play priors (PnP) or regularization by denosing (RED). Prior work has proved the convergence of PnP/RED for contractive, nonexpansive, and/or firmly nonexpansive denoisers. The design of denoisers that enable PnP/RED convergence is an active research area.\n\nThis paper focuses on denoisers that correspond to the gradients of some potential functions. By using such denoisers within RED, one can guarantee convergence via traditional optimization. The key idea in the paper is to use a deep neural network to parameterize a potential function (instead of the denoiser), then train that function for denoising by using its gradient. Additionally, when the weights of the network are nonlinear and activation functions are nondecreasing and convex, the denoiser corresponds to a convex function.\n\nThe key contribution of the paper is the formulation of a procedure for training a potential function that can be used to obtain a denoiser. The theoretical arguments presented in Lemma 2 and Theorem 1, are well known in optimization (see below for reference). Similarly, Algorithm 1 corresponds exactly to the RED-SD algorithm using a potential-driven denoiser. The empirical results compare three proposed variants of denoisers against PnP-PGD and RED-SD on problems of image deblurring and superresolution.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The central idea in the paper nicely complements the original work on RED, which attempted to obtain a PnP algorithm minimizing an explicit objective function. The recent literature on PnP/RED has focused on using deep neural nets to parameterize the denoiser. This paper argues that by instead parameterizing the potential function, one can obtain an explicit objective under milder assumptions than in the original RED paper. \n\nIssues: (a) The first major issue in this publication is the discussion in “Section 3.2 Convergent Inversion Method”. The theoretical results claimed in that section in Lemma 2 and Theorem 1 are standard results from optimization (see, for example, “Section 1.2.3 Gradient method” in Nesterov's book “An Introductory Lectures on Convex Optimization”). One can simply cut these results by including references and save 1 page for providing other details. (b) The second major issue in the paper is in inadequate simulations. The paper seems to want to show that the proposed denoiser leads to (i) convergence; (ii) better solutions in terms of PSNR/SSIM than PnP/RED with other denoisers. Regarding (i): (1) There are a number of variants of PnP/RED that achieve provable convergence using nonexpansive variants of DnCNN (see denoiser in [40]). Why would you pick the one that makes PnP-PGM diverge? (2) Having a convex optimization reference, such as FISTA-TV, would help to understand the benefit of having the potential function. (3) Having convergence plots in terms of PSNR (dB) would help to see the speed of improvement in imaging quality. (4) Having a basic deep learning reference such as UNet would help to evaluate that all methods are properly implemented and take advantage of the measurement model. (5) Having a convergence plot in terms of the objective function, would help to validate that the GraDnCNN is indeed minimizing an objective. Regarding (ii): Why would we expect the existence of the potential function to lead to better imaging quality? \n\nSummary: This paper is proposing an intriguing alternative to training nonexpansive denoisers for PnP/RED. While “it has potential”, the paper falls short in terms of fully convincing the benefit of the proposed method for solving inverse problems. Specifically, experimental validation is incomplete and needs to be significantly improved following my suggestions above. Additionally, the paper claims a number of classical results from optimization as contributions.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues", "review_text": "", "extra_scores": {"ethical_concerns": "No issues", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "KJjtCtgXQzo", "reviewer": "Reviewer_rP8a", "summary": "**Summary**:\nThis paper addresses an important issue in the framework of regularization by denoising (RED), which is that the explicit RED regularizer $\\frac{1}{2}x^T(x-D_\\sigma(x))$ exists only if a set of strict conditions on the denoiser are met. Due to this problem, RED fails to formulate explicit regularizers for advanced denoisers (e.g. CNN) since they generally do not satisfy the conditions. Although some work has established some theoretical analysis for RED algorithms using deep denoisers, a linkage between the denoiser and the regularizer is generally missing. \n\nIn this paper, the authors proposed to formulate the regularizer as a *smooth potential functional* $g(x)$, which is parameterized by a CNN with smooth activation functions. To link the potential functional to a denoiser, the authors trained the CNN such that the gradient $\\nabla g(x)$ outputs the noise residual for the AWGN. In this way, the proposed method can formulate the gradient-based RED update by directly taking the gradient of the objective function $F(x)$.\n\nSince the objective function is available, the authors also proposed to use the standard back-tracking trick in their framework to guarantee convergence. Additionally, a convergence analysis is also introduced by following the common nonconvex optimization analysis given the handy $F(x)$. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: Overall, this paper constitutes a solid contribution to the study of RED, proposing a new way to train a denoising prior and linking it back to a regularizer.\n\n**quality & clarity**: The paper is well-written and easy to read. The background review of the related work is very thorough and comprehensive\n\n**Significance**: I believe their contribution is of enough significance to the community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Weakness**\n\n1. In the description under eq. (7), it would be clearer if the authors can explicitly link $\\mathcal{W}$ and $\\mathcal{V}$ to the convolutional operations. \n\n2. Learning a denoiser as the gradient of a convex function is very interesting. But I do not get why having two convex potential $g_{\\theta_1}$ and $g_{\\theta_1}$ subtracting each other will increase the expressivity. \n\n3. It is nice to have a back-tracking algorithm implemented in RED since the Lipschitz constant of a CNN is not easy to estimate. However, it seems to me that *Liu et al* has also proposed a back-tracking algorithm for RED. What is the difference between yours and theirs?\n\n4. With all due respect and no intention to undermine the novelty, it is a little bit disappointing for me to see that there is only one theorem that only accounts for the nonconvex case. Given the fact that the authors also proposed a convex denoising functional, it may improve the completeness of the paper if the author can include another short theorem (even if it is well-known) or some remarks on the convergence of the proposed framework under convex $g(x)$.\n\n5. In the experiments, the batch normalization and biases are removed for DnCNN, which may significantly reduce the representation power of the network. Can the authors explain this? On the contrary, the proposed network architecture has biases, which makes the comparison unfair.\n\n6. Both PnP and RED achieve degraded performance compared with the results in other PnP/RED papers. I guess this is mainly due to the modified DnCNN architecture. The authors should compare their method with PnP/RED using standard DnCNN (with either batch normalization or biases). Moreover, please also compare against the Lipschitz-constrained (nonexpansive) DnCNN, which demonstrated good performance as well as convergence in many works.\n\n7. Some missing citations related to RED, back-tracking, and learning functionals:\n\n    [1] Sun et al Async-RED: A Provably Convergent Asynchronous Block Parallel Stochastic Method using Deep Denoising \n    Priors. ICLR 2021\n\n    [2] Liu et al RARE: Image Reconstruction Using Deep Priors Learned without Groundtruth. IEEE JSTSP 2020\n\n    [3] Lunz et al Adversarial Regularizers in Inverse Problems NeurIPS 2018", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour for writing response. 2 hours for reading and evaluation."}, "extra_text": {}}, {"review_id": "FljqkA0-3mU", "reviewer": "Reviewer_2weA", "summary": "Summary: \n- This submission deals with designing stable and smooth denoisers that could be directly used for regularization, or for plug-and-play (PnP) methods to solve inverse problems. \n- This is an improvement upon existing denoisers such as RED and PnP which are not necessarily stable. \n- The key idea is to craft neural networks that output scalar-valued potential function g whose gradient is the denoiser residual. To guarantee convergence and stability, g has to be smooth and h=1/2x^2 - g convex. \n- Advantages: the denoisers lead to symmetric Jacobians, allowing for a MAP estimation; denoisers can be integrated into optimization schemes with backtracking step size, and thus no need for enforcing Lipschitz continuity.\n- To construct such a network, GraDnCNN stacks conv layers and nonlinear activations, and proposes DnICNNs with non-negative weights and convex activations. Variants of DnICNN are discussed that achieve better denoising/recovery performance for Gaussian deblurring, and image superresolution.\n\nContributions:\n- construct stable denoisers based on neural networks as gradients of smooth potential functions g\n- integrate the denoisers as a regularizer for solving inverse problems and test it out for gaussian image deblurring and superresolution which outperforms RED and traditional PnP methods\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality and significance\n- a novel perspective to craft CNNs for designing stable denoiser that leverage the power of neural nets in a guaranteed manner\n- The design seems to be limiting, needing non-negative weights, and thus the scope and usefulness of this theory is not very clear. This is discussed in the comments next.\n\nClarity:\n- This paper is crowded, and it discusses different aspects of the potential based denosiers, so the main novelty of this work is lost. \n- It could benefit from a better organization. More details about the architecture of ICNN and requirements on the weights would help better understand the paper. This reviewer doesnt think the discussions about the convergence are of high priority as those come natural given the constraints of convexity and smoothness on the denoiser. \n\nComments:\n- This is an Interesting connection between convex regularization and CNNs. How far the network design can go? Let’s say we have off-the-shelf deep CNN based denoisers such as U-net and want to see their corresponding potential function, and regularization function.  It's important for the theory to be insightful and suggest good architectures. The Author's comment would be helpful.\n\n- About the GDCNN, what restications on the weights and layers are required to make sure the output is a legitimate gradient for a lipschitz and convex denoiser? It’s not clear that the non-convex training objective in (9) would naturally lead to network weights that would produce legitimate gradients?\n\n- The constraints on the design of DnICNN seem to be pretty stringent; non-negative weights and convex activations. It’s strange that no weight normalization is needed to ensure lipschitzness. Is that correct?\n\n- If I understand correctly, only ICNN is convex. DnDICNN and GraDnCNN are not legitimate for stability. right?\n\n- in order to see the usefulness of the connection between regularization and neural nets, it is useful to include off-the-shelf denoisers in the experiments Fig. 2,3. One such example is unrolled networks that are trained end-to-end, or the inverse problem with the generative model projection. Glancing through the literature, a few examples, very relevant to this work include [1], [2]. \n\n\n[1] Mardani, M., Sun, Q., Papyan, V., Monajemi, H., Vasanawala, S., Pauly, J., Donoho, D. (2018). Neural Proximal Gradient Descent for Compressive Imaging. Advances in Neural Information Processing Systems, 31, 9573-9583.\n\n[2] Van Veen D, Jalal A, Soltanolkotabi M, Price E, Vishwanath S, Dimakis AG. Compressed sensing with deep image prior and learned regularization. arXiv preprint arXiv:1806.06438. 2018 Jun 17.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "  ", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "  "}, "extra_text": {}}, {"review_id": "F3NuGZcEUD", "reviewer": "Reviewer_fHLE", "summary": "Motivated by the RED algorithm for solving inverse problems, the authors propose a neural-net denoiser that is the gradient of an explicit potential function.  This guarantees that the Jacobian is symmetric definite, which then guarantees that the RED algorithm minimizes an explicit cost function, which in turn makes convergence straightforward to prove, thereby solving an important open problem about the convergence of RED and plug-and-play (PnP)-type algorithms.  The fact that there is an explicit cost also enables the use of adaptive stepsize selection within RED, with one particular scheme proposed by the authors.  Numerical comparisons to RED and PnP algorithms show improved PSNR and SSIM performance, as well as enhanced stability.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the overall idea is great: design a neural-net denoiser that---by construction---equals the gradient of an explicit potential function.  To my knowledge, this is novel and it solves an important open problem about the convergence of PnP/RED methods.\n\nBut I fear that the numerical experiments are seriously flawed.  There are two reasons.\n1. It's important to understand that the fixed points of PnP and RED algorithms are directly affected by stepsizes used in those algorithms (please see [3, p.108 and p.112]).  This stands in sharp contrast to convex optimization, where the stepsize does not affect the fixed point.  Thus, to show the true potential of PnP/RED, one must tune the stepsize.  This can be easily accomplished using the validation data.\n2. There is a compounding problem, which has to do with differences among PnP/RED algorithms.  In the convex optimization setting, the PGD algorithm is unstable for large stepsizes, unlike the ADMM and primal-dual splitting (PDS) algorithms, where the stepsize can be chosen arbitrarily.  In the context of PnP, the PGD, ADMM, and PDS algorithms all have the same fixed points for a given stepsize (please see [3, p.111]), but PnP-PGD places severe limits on that stepsize, making many of those fixed points unstable.  In practice, the best stepsizes (e.g., in the sense of MSE or SSIM) are typically unstable for PnP-PGD, meaning that optimally tuned PnP-PGD is not competitive with optimally tuned PnP-ADMM and PnP-PDS.  Similar issues may arise with different RED algorithms.\nIn conclusion, if the goal is to compare to PnP algorithms, then PnP-PGD must be avoided and either PnP-ADMM or PnP-PDS must be properly tuned.  Likewise, the RED algorithm must be chosen wisely and tuned.  The experiments in the paper show PnP failing completely, but this is simply because the authors have not made a fair comparison to PnP.  Likewise, RED was not tuned and so it does not appear that the RED comparison is fair either.\n\nMinor comments:\n1. In the Related Work section, it may help to know that [3] is a comprehensive overview of PnP & RED methods and not just an application of PnP & RED to MRI.\n2. In the Related Work section, for completeness, it may help to note that the AMP [7] and VAMP [17] frameworks are unique in guaranteeing that an MMSE denoiser yields and MMSE solution to the inverse problem, at least under large random forward operators.  In the same context, they prove convergence for Lipschitz denoisers.\n3. Equation (13) is missing subscript k on x in two places.  Also, it should be noted immediately after (13) that (13) is RED-SD.\n4. I suggest removing \"It Has Potential\" from the title.  You could add this phrase to any paper; it has no value.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The reviewer does not see any potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "The reviewer does not see any ethical issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4oWdmqL3pQD", "reviewer": "Reviewer_UqjB", "summary": "The paper proposes image denoising based CNNs as regularizers that can be constructed as the gradient of smooth potential functions $g$. The main promise is mentioned to be making an optimization algorithm has explicit regularization, when equipped with those denoisers. Unlike original regularization by denoising (RED) that the explicit regularizer exists if the denoisers satisfy several strict conditions, the proposed denoisers only rely on smoothness. Particularly, the denoisers consist of smooth activation functions to ensure the differential continuity and are trained in a residual denoising fashion using MSE loss. As a result, the proposed algorithm can be formulated as steepest descent variant of RED optimized by directly adopting the gradient of the objective function $F$. Standard back-tracking is used to ensure convergence. A convergence analysis is provided following the common optimization theory. Finally, the performance of the proposed method is evaluated over two image inverse problems such as gaussian deblur and super-resolution, with satisfactory results compared to existing methods based on PnP and RED. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** This paper is well motivated and is written fairly well. The idea sounds simple but promising and makes it possible to be directly applied for other variant RED algorithms. Most importantly, this paper provides a new guide of training CNN denoisers as explicit regularizers, unlike current literature, where the link between denoisers and regularization is commonly missing.\n\n**Quality & Clarity:**  Overall the paper is  easy to read. The background review of the related work is comprehensive and clearly stated. However, I found it hard to follow at some specific sections due to insufficient details on the notations.  For one, it is a little bit confusing for me about the function $h$ mentioned in Lemma 1, line167 -- 172 and the one in Eq. 15.  \n\n**Significance:** I think this is a strong paper, in a line of research that is interesting and should be of interest to the NeurIPS community.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "**Weakness:**\n\n- Using the assumption that the denoiser $D$ is the proximal operator of some convex scaler function is to be fine to obtain some fundamental insights for the link between the CNN denoiser and regularization, but to me, it also relies on the convexity of the denoiser. Considering that Algorithm 1 equipped with non-convex denoiser achieves the best performance, it seems to be tricky to follow _Lemma 1_ to design the desnoisers.\n\n- For PnP-PGD, RED-SD and the proposed method with GraDnCNN using DnCNN are trained without lipschitz constrain for $L < 1$, resulting a non-convex problem. Hence an initial point of the update sometimes is important. It would be interesting to see how different initialization influence the final reconstruction results.\n\n- Similar to the above, both PnP and RED led to poor results compared to most of existing works. This may be due to the improper parameter tunning such as regularization trade-off $\\lambda$, step-size $\\gamma$ and denoising intensity. \nA recent work by (Xu’2021) introduce a simply way enable the tuning of the regularization parameter within PnP, it is not clear for me why this easy approach is not applied for PnP.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is proposing a new and interesting idea: a potential-based neural net denoiser that makes RED convergence analysis clean and simple. This is quite novel and suitable to NeurIPS audience. The reviewers raised serious concerns on the evaluation and how the comparisons to other RED methods were performed. I agree that the evaluation should be improved and the authors did a reasonable job on responding to reviewer input on this. I recommend that the authors revise their manuscript taking all this input into account. The analysis on convex functions is also a good suggestion that the authors should take into consideration.  


Decision: Accept (Poster)