It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems

1 Introduction . General Recovery Inverse problems aim to recover an unknown signal from its observed corrupted or compressed version . This fundamental task appears in a variety of fields such as machine learning , signal processing , computer vision and medical imaging . When a model for the degradation process is available , one may derive a data fidelity term which relates the measurements to the unknown signal through a forward operator . However , this term alone typically leads to ill-posed optimization problems with unstable solutions , requiring the incorporation of additional knowledge of the desired 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . signal into the inversion process . This has given rise to diverse regularization methods which have been investigated extensively over the last few decades . RED and PnP A very basic inverse problem is denoising , in which a clean signal is to be recovered from its measurements contaminated with additive noise . During the last period of time there has been an overwhelming advancement in the ability to perform denoising , creating techniques that approach optimal performance [ 11 , 29 , 30 ] . This striking progress in denoising has been the motivation for the plug-and-play priors ( PnP ) [ 48 , 10 , 41 ] methodology , which introduced the paradigm of using denoisers in order to solve general inverse problems . PnP incorporates implicit regularization into the inverse solutions by replacing proximal operators in common optimization algorithms with powerful denoisers . This simple modification has led to remarkable performance in diverse inverse tasks [ 41 , 50 , 38 , 22 , 56 , 3 , 51 ] , as well as to the development of an arsenal of PnP-based techniques which includes proximal-type methods but also goes beyond them [ 33 , 7 , 18 , 15 , 5 , 55 , 14 , 17 ] . However , the lack of a clearly formulated regularization term corresponding to the denoiser poses a serious challenge in ensuring the stability of PnP-based methods , and typically requires careful parameter-tuning [ 49 ] . To fill this void , the regularization by denoising ( RED ) [ 39 ] framework presents an explicit Laplacianbased regularization that relies on a denoiser . Under certain conditions on the denoiser [ 39 , 37 ] , the RED functional is convex and its gradient is given by the denoising residual , allowing for easy incorporation in standard optimization procedures . RED has demonstrated impressive recovery performance while providing global convergence guarantees . However , various common denoisers do not satisfy the conditions required by RED [ 37 ] . In such cases , there exists no objective function which justifies the RED method [ 37 ] and the corresponding convergence analysis does not hold . Thus RED , as well as PnP , shows state-of-the-art empirical results but without provable stability . A comprehensive overview of PnP and RED is given in [ 3 ] . Related Work The study of the stability of both RED and PnP methods has become an active line of research in recent years . The convergence of two PnP techniques was proven in [ 10 ] and [ 47 ] under the assumption of bounded denoisers . When minimum mean squared error ( MMSE ) denoisers are used , the PnP framework was shown to converge vis-a-vis an explicit objective function [ 53 ] . Similarly , [ 7 , 18 ] proved the convergence of approximate message passing algorithms with MMSE denoisers for random froward operators . Multiple studies [ 41 , 43 , 44 , 9 , 45 ] derived the convergence of PnP by relying on nonexpansive denoisers . The latter has been the core assumption of [ 8 ] , which explained the PnP framework , including its stability , through the notion of consensus equilibrium . A similar condition was used in [ 37 ] , which adopted the score-matching framework to provide a new interpretation of the RED algorithms and their convergence . [ 42 ] presented a block-coordinate RED method and derived its fixed-point convergence for block-nonexpansive denoisers . The work of [ 36 ] introduced a convergent PnP-type method using a maximally monotone denoiser , created by training a nonexpansive neural network in a supervised fashion . Alternatively , [ 40 ] provided convergence guarantees for certain PnP variants that use trained denoisers whose residuals are nonexpansive . This type of denoiser was used in [ 27 ] to develop Bayesian inference with PnP priors , including the extension of the PnP framework to Monte Carlo sampling . The authors of [ 24 , 23 ] employ denoisers within Langevin dynamics to sample from the posterior distribution of any linear inverse problem . A different strategy was studied in [ 35 ] , which proposed a special inner product derived from the given linear denoiser , and thus proved the convergence of PnP based on the theory of averaged nonexpansive operators . Recent work [ 12 ] introduced RED via fixed-projection framework ( RED-PRO ) , which utilizes the fixed-points of the denoisers as regularization . RED-PRO unifies RED and PnP and derives their convergence for general demicontractive denoisers , including nonexpansive ones . Recent studies , mentioned above , are motivated by or rely on the assumption that the denoiser is an MMSE or maximum a posteriori ( MAP ) estimator . However , such estimators are in the form of gradients , hence , they must exhibit symmetric Jacobians [ 20 , 19 , 21 ] amongst other properties . Furthermore , Jacobian symmetry offers additional advantages as it often leads to stable algorithms and allows one to better understand the behavior of such non-linear estimators [ 34 ] . Unfortunately , many state-of-the-art denoisers do not fulfill this condition [ 37 ] . An additional drawback of current RED and PnP methods is that the Lipschitz constant of the denoiser often has to be controlled to create a nonexpansive operator , which in turn ensures convergence . This is typically done by learning a deep network and adding a regularization term to the training loss function , whose purpose is to bound the Lipschitz constant of the resultant denoiser . However , exactly enforcing this property of the denoiser is impractical and it is only approximated in practice . Contributions In this work , we develop a provably convergent technique which exploits denoisers for solving general inverse problems . ( 1 ) We introduce image denoisers that are constructed as the gradients of smooth potential functions , represented by deep neural networks and trained in a supervised fashion . ( 2 ) Equipped with these learned potential networks , we formulate a general inverse framework where we utilize our denoising potentials as an explicit regularization . ( 3 ) We use the existence of an underlying objective function to incorporate the proposed denoisers into standard optimization schemes with backtracking step size . Specifically , we present a simple gradient descent algorithm where the step size is tuned automatically to provably guarantee convergence to stationary points of the objective function . ( 4 ) We numerically validate our method through imaging experiments , showing improved results compared to RED and PnP , with additional provable stability . 2 Background . Inverse Problems : Formulation We consider the recovery of an unknown signal x∗ ∈ Rdx from its corrupted observation y ∈ Rdy . The inversion task is traditionally formulated as an optimization problem of the form x̂ = argmin x∈Rdx f ( x ; y ) + λg ( x ) , ( 1 ) where f ( · ; y ) is a fidelity term related to the observations y , g ( · ) is a regularization functional that incorporates prior information or known statistics of the true signal x∗ to avoid ill-posed problems , and λ > 0 balances between the terms . A common model for the measurements is y = Ax +w where A is a linear degradation ( forward ) operator and w is an i.i.d . white Gaussian noise , leading to the following log-likelihood function f ( x ; y ) = 1 2 ||Ax− y||22 . ( 2 ) The above model applies to a wide variety of inverse problems such as compressed sensing , super resolution , deblurring , inpainting and much more . In the following , we consider a general functional f ( · ) where we suppress its dependency on the observations y for brevity . Moreover , both g ( · ) and f ( · ) are typically assumed to be convex where the former is generally non-differentiable , while the latter is a smooth function . By the definition of smooth functions , there exists a Lipschitz constant Lf > 0 such that the gradient∇f of f ( · ) satisfies ||∇f ( x1 ) −∇f ( x2 ) || ≤ Lf ||x1 − x2|| , ∀ x1 , x2 ∈ Rdx . ( 3 ) Proximal Methods The minimization problem ( 1 ) has been investigated extensively over the decades , leading to a series of diverse approaches . Many common optimization techniques for solving ( 1 ) rely on the proximal operator of g ( · ) , defined as Pλg ( x ) , argmin z∈Rdx 1 2 ||x− z||+ λg ( z ) . ( 4 ) The proximal operator , which is a MAP estimator , provides an efficient way to overcome the lack of differentiablity of g ( · ) , especially when it admits a closed-form expression , for example in the case of the ` 1-norm . A simple representative of this family of algorithms is the proximal gradient descent ( PGD ) method which updates the solution iteratively as follows xk+1 = Pλg ( xk − µk∇f ( xk ) ) ( 5 ) where µk > 0 is the step size . PGD has shown great performance in various applications along with strong convergence guarantees . The PnP Framework Notice that the proximal operator can be seen , by definition , as the solution to a denoising problem . This observation serves as the basis of the PnP framework which extends standard proximal methods by replacing the proximal operator with powerful denoisers . Following this rationale , ( 5 ) is modified to create the PnP-PGD method that performs the following update rule xk+1 = D ( xk − µk∇f ( xk ) ) , ( PnP-PGD ) where D : Rdx → Rdx is an off-the-shelf denoiser . The RED Framework An alternative strategy for leveraging advanced denoisers is the RED framework which utilizes the denoiser to construct an explicit regularization g ( x ) = 12 〈x , x−D ( x ) 〉 . Under certain conditions on the denoiser [ 39 , 37 ] , g ( · ) is convex and its gradient is readily given by the denoising residual∇g ( x ) = x−D ( x ) . This allows for the solution of ( 1 ) by standard algorithms , e.g. , steepest ( or gradient ) descent xk+1 = xk − µk 1 + λ ( ∇f ( xk ) + λ ( xk −D ( xk ) ) ) . ( RED-SD ) Surprisingly , RED-based methods have been found to be effective also when the required conditions on the denoiser are not satisfied and the expression for the regularization term is invalid . In such cases , RED uses the denoiser as an implicit regularization , similar to PnP . Challenges in RED and PnP Both RED and PnP provide extremely efficient means for exploiting powerful denoisers to solve inverse problems . Yet their analysis poses two closely related challenges : 1 . Asymmetric Denoisers : Many studies on RED and PnP [ 53 , 27 , 43 , 44 , 9 , 45 , 52 , 42 ] motivate or base their derivations on the assumption that the denoiser is an approximation of either MAP or MMSE estimator . It has been proven that when the Jacobian of such estimators is well-defined it must be symmetric [ 20 , 19 , 21 ] . However , RED and PnP typically reach state-of-the-art results by relying on denoisers which do not exhibit this property , as in the case of the popular DnCNN technique ; as a result , they lose their interpretation as optimization solvers . In such cases , while there are alternative optimization-free explanations for RED and PnP [ 8 , 37 ] , it is challenging to properly analyze their behavior and characterize the nature of their solutions . 2 . Convergence Issues : Continuing the above line of reasoning , often there exists no regularization that explains the denoiser , namely no functional g ( · ) for which∇g ( x ) = D ( x ) or∇g ( x ) = x−D ( x ) [ 37 ] . This lack of a potential function creates difficulties in studying the theoretical convergence of the RED and PnP algorithms . However , a common sufficient condition to guarantee convergence is for the denoiser D ( · ) to be ( averaged ) nonexpansive [ 13 ] , implying that ||D ( x1 ) −D ( x2 ) || ≤ L||x1 − x2|| , ∀ x1 , x2 ∈ Rdx , ( 6 ) for some Lipschitz constant 0 < L ≤ 1 . However , most predefined denoisers do not satisfy this assumption . An alternative approach [ 40 , 36 , 46 ] is to learn a nonexpansive denoiser by adding a regularization term to the training loss which constrains its Lipschitz constant . Unfortunately , this remains an open challenge since in general there are no practical means to absolutely enforce the resultant denoiser to be nonexpansive . Our work addresses both of these challenges . We first introduce a new class of denoisers which stem from an explicit potential functional and thus demonstrate symmetric Jacobian by their construction . We then derive an inverse technique that utilizes our denoisers as well as their potentials to ensure convergence . Three important remarks are to be made with regard to work presented below . First , we emphasize that the proposed denoisers are not intended to compete with state-of-the-art denoisers , but rather are developed to exhibit favorable properties that benefit other inversion tasks . Second , our proposed recovery method is not derived to compete with end-to-end learned solutions to specific inverse problems [ 31 , 2 , 25 , 26 ] . Rather , our learned denoisers , along with our optimization framework , can be used for all inverse problems ( of the form in Eq . ( 1 ) ) without additional training . Lastly , while we present a present a specific inversion method , the potential-driven denoisers can benefit many RED and PnP algorithms , hence , our work complements these frameworks .

Summary: Summary:  - This submission deals with designing stable and smooth denoisers that could be directly used for regularization, or for plug-and-play (PnP) methods to solve inverse problems.  - This is an improvement upon existing denoisers such as RED and PnP which are not necessarily stable.  - The key idea is to craft neural networks that output scalar-valued potential function g whose gradient is the denoiser residual. To guarantee convergence and stability, g has to be smooth and h=1/2x^2 - g convex.  - Advantages: the denoisers lead to symmetric Jacobians, allowing for a MAP estimation; denoisers can be integrated into optimization schemes with backtracking step size, and thus no need for enforcing Lipschitz continuity. - To construct such a network, GraDnCNN stacks conv layers and nonlinear activations, and proposes DnICNNs with non-negative weights and convex activations. Variants of DnICNN are discussed that achieve better denoising/recovery performance for Gaussian deblurring, and image superresolution.  Contributions: - construct stable denoisers based on neural networks as gradients of smooth potential functions g - integrate the denoisers as a regularizer for solving inverse problems and test it out for gaussian image deblurring and superresolution which outperforms RED and traditional PnP methods