It Has Potential: Gradient-Driven Denoisers for Convergent Solutions to Inverse Problems

1 Introduction . General Recovery Inverse problems aim to recover an unknown signal from its observed corrupted or compressed version . This fundamental task appears in a variety of fields such as machine learning , signal processing , computer vision and medical imaging . When a model for the degradation process is available , one may derive a data fidelity term which relates the measurements to the unknown signal through a forward operator . However , this term alone typically leads to ill-posed optimization problems with unstable solutions , requiring the incorporation of additional knowledge of the desired 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . signal into the inversion process . This has given rise to diverse regularization methods which have been investigated extensively over the last few decades . RED and PnP A very basic inverse problem is denoising , in which a clean signal is to be recovered from its measurements contaminated with additive noise . During the last period of time there has been an overwhelming advancement in the ability to perform denoising , creating techniques that approach optimal performance [ 11 , 29 , 30 ] . This striking progress in denoising has been the motivation for the plug-and-play priors ( PnP ) [ 48 , 10 , 41 ] methodology , which introduced the paradigm of using denoisers in order to solve general inverse problems . PnP incorporates implicit regularization into the inverse solutions by replacing proximal operators in common optimization algorithms with powerful denoisers . This simple modification has led to remarkable performance in diverse inverse tasks [ 41 , 50 , 38 , 22 , 56 , 3 , 51 ] , as well as to the development of an arsenal of PnP-based techniques which includes proximal-type methods but also goes beyond them [ 33 , 7 , 18 , 15 , 5 , 55 , 14 , 17 ] . However , the lack of a clearly formulated regularization term corresponding to the denoiser poses a serious challenge in ensuring the stability of PnP-based methods , and typically requires careful parameter-tuning [ 49 ] . To fill this void , the regularization by denoising ( RED ) [ 39 ] framework presents an explicit Laplacianbased regularization that relies on a denoiser . Under certain conditions on the denoiser [ 39 , 37 ] , the RED functional is convex and its gradient is given by the denoising residual , allowing for easy incorporation in standard optimization procedures . RED has demonstrated impressive recovery performance while providing global convergence guarantees . However , various common denoisers do not satisfy the conditions required by RED [ 37 ] . In such cases , there exists no objective function which justifies the RED method [ 37 ] and the corresponding convergence analysis does not hold . Thus RED , as well as PnP , shows state-of-the-art empirical results but without provable stability . A comprehensive overview of PnP and RED is given in [ 3 ] . Related Work The study of the stability of both RED and PnP methods has become an active line of research in recent years . The convergence of two PnP techniques was proven in [ 10 ] and [ 47 ] under the assumption of bounded denoisers . When minimum mean squared error ( MMSE ) denoisers are used , the PnP framework was shown to converge vis-a-vis an explicit objective function [ 53 ] . Similarly , [ 7 , 18 ] proved the convergence of approximate message passing algorithms with MMSE denoisers for random froward operators . Multiple studies [ 41 , 43 , 44 , 9 , 45 ] derived the convergence of PnP by relying on nonexpansive denoisers . The latter has been the core assumption of [ 8 ] , which explained the PnP framework , including its stability , through the notion of consensus equilibrium . A similar condition was used in [ 37 ] , which adopted the score-matching framework to provide a new interpretation of the RED algorithms and their convergence . [ 42 ] presented a block-coordinate RED method and derived its fixed-point convergence for block-nonexpansive denoisers . The work of [ 36 ] introduced a convergent PnP-type method using a maximally monotone denoiser , created by training a nonexpansive neural network in a supervised fashion . Alternatively , [ 40 ] provided convergence guarantees for certain PnP variants that use trained denoisers whose residuals are nonexpansive . This type of denoiser was used in [ 27 ] to develop Bayesian inference with PnP priors , including the extension of the PnP framework to Monte Carlo sampling . The authors of [ 24 , 23 ] employ denoisers within Langevin dynamics to sample from the posterior distribution of any linear inverse problem . A different strategy was studied in [ 35 ] , which proposed a special inner product derived from the given linear denoiser , and thus proved the convergence of PnP based on the theory of averaged nonexpansive operators . Recent work [ 12 ] introduced RED via fixed-projection framework ( RED-PRO ) , which utilizes the fixed-points of the denoisers as regularization . RED-PRO unifies RED and PnP and derives their convergence for general demicontractive denoisers , including nonexpansive ones . Recent studies , mentioned above , are motivated by or rely on the assumption that the denoiser is an MMSE or maximum a posteriori ( MAP ) estimator . However , such estimators are in the form of gradients , hence , they must exhibit symmetric Jacobians [ 20 , 19 , 21 ] amongst other properties . Furthermore , Jacobian symmetry offers additional advantages as it often leads to stable algorithms and allows one to better understand the behavior of such non-linear estimators [ 34 ] . Unfortunately , many state-of-the-art denoisers do not fulfill this condition [ 37 ] . An additional drawback of current RED and PnP methods is that the Lipschitz constant of the denoiser often has to be controlled to create a nonexpansive operator , which in turn ensures convergence . This is typically done by learning a deep network and adding a regularization term to the training loss function , whose purpose is to bound the Lipschitz constant of the resultant denoiser . However , exactly enforcing this property of the denoiser is impractical and it is only approximated in practice . Contributions In this work , we develop a provably convergent technique which exploits denoisers for solving general inverse problems . ( 1 ) We introduce image denoisers that are constructed as the gradients of smooth potential functions , represented by deep neural networks and trained in a supervised fashion . ( 2 ) Equipped with these learned potential networks , we formulate a general inverse framework where we utilize our denoising potentials as an explicit regularization . ( 3 ) We use the existence of an underlying objective function to incorporate the proposed denoisers into standard optimization schemes with backtracking step size . Specifically , we present a simple gradient descent algorithm where the step size is tuned automatically to provably guarantee convergence to stationary points of the objective function . ( 4 ) We numerically validate our method through imaging experiments , showing improved results compared to RED and PnP , with additional provable stability . 2 Background . Inverse Problems : Formulation We consider the recovery of an unknown signal x∗ ∈ Rdx from its corrupted observation y ∈ Rdy . The inversion task is traditionally formulated as an optimization problem of the form x̂ = argmin x∈Rdx f ( x ; y ) + λg ( x ) , ( 1 ) where f ( · ; y ) is a fidelity term related to the observations y , g ( · ) is a regularization functional that incorporates prior information or known statistics of the true signal x∗ to avoid ill-posed problems , and λ > 0 balances between the terms . A common model for the measurements is y = Ax +w where A is a linear degradation ( forward ) operator and w is an i.i.d . white Gaussian noise , leading to the following log-likelihood function f ( x ; y ) = 1 2 ||Ax− y||22 . ( 2 ) The above model applies to a wide variety of inverse problems such as compressed sensing , super resolution , deblurring , inpainting and much more . In the following , we consider a general functional f ( · ) where we suppress its dependency on the observations y for brevity . Moreover , both g ( · ) and f ( · ) are typically assumed to be convex where the former is generally non-differentiable , while the latter is a smooth function . By the definition of smooth functions , there exists a Lipschitz constant Lf > 0 such that the gradient∇f of f ( · ) satisfies ||∇f ( x1 ) −∇f ( x2 ) || ≤ Lf ||x1 − x2|| , ∀ x1 , x2 ∈ Rdx . ( 3 ) Proximal Methods The minimization problem ( 1 ) has been investigated extensively over the decades , leading to a series of diverse approaches . Many common optimization techniques for solving ( 1 ) rely on the proximal operator of g ( · ) , defined as Pλg ( x ) , argmin z∈Rdx 1 2 ||x− z||+ λg ( z ) . ( 4 ) The proximal operator , which is a MAP estimator , provides an efficient way to overcome the lack of differentiablity of g ( · ) , especially when it admits a closed-form expression , for example in the case of the ` 1-norm . A simple representative of this family of algorithms is the proximal gradient descent ( PGD ) method which updates the solution iteratively as follows xk+1 = Pλg ( xk − µk∇f ( xk ) ) ( 5 ) where µk > 0 is the step size . PGD has shown great performance in various applications along with strong convergence guarantees . The PnP Framework Notice that the proximal operator can be seen , by definition , as the solution to a denoising problem . This observation serves as the basis of the PnP framework which extends standard proximal methods by replacing the proximal operator with powerful denoisers . Following this rationale , ( 5 ) is modified to create the PnP-PGD method that performs the following update rule xk+1 = D ( xk − µk∇f ( xk ) ) , ( PnP-PGD ) where D : Rdx → Rdx is an off-the-shelf denoiser . The RED Framework An alternative strategy for leveraging advanced denoisers is the RED framework which utilizes the denoiser to construct an explicit regularization g ( x ) = 12 〈x , x−D ( x ) 〉 . Under certain conditions on the denoiser [ 39 , 37 ] , g ( · ) is convex and its gradient is readily given by the denoising residual∇g ( x ) = x−D ( x ) . This allows for the solution of ( 1 ) by standard algorithms , e.g. , steepest ( or gradient ) descent xk+1 = xk − µk 1 + λ ( ∇f ( xk ) + λ ( xk −D ( xk ) ) ) . ( RED-SD ) Surprisingly , RED-based methods have been found to be effective also when the required conditions on the denoiser are not satisfied and the expression for the regularization term is invalid . In such cases , RED uses the denoiser as an implicit regularization , similar to PnP . Challenges in RED and PnP Both RED and PnP provide extremely efficient means for exploiting powerful denoisers to solve inverse problems . Yet their analysis poses two closely related challenges : 1 . Asymmetric Denoisers : Many studies on RED and PnP [ 53 , 27 , 43 , 44 , 9 , 45 , 52 , 42 ] motivate or base their derivations on the assumption that the denoiser is an approximation of either MAP or MMSE estimator . It has been proven that when the Jacobian of such estimators is well-defined it must be symmetric [ 20 , 19 , 21 ] . However , RED and PnP typically reach state-of-the-art results by relying on denoisers which do not exhibit this property , as in the case of the popular DnCNN technique ; as a result , they lose their interpretation as optimization solvers . In such cases , while there are alternative optimization-free explanations for RED and PnP [ 8 , 37 ] , it is challenging to properly analyze their behavior and characterize the nature of their solutions . 2 . Convergence Issues : Continuing the above line of reasoning , often there exists no regularization that explains the denoiser , namely no functional g ( · ) for which∇g ( x ) = D ( x ) or∇g ( x ) = x−D ( x ) [ 37 ] . This lack of a potential function creates difficulties in studying the theoretical convergence of the RED and PnP algorithms . However , a common sufficient condition to guarantee convergence is for the denoiser D ( · ) to be ( averaged ) nonexpansive [ 13 ] , implying that ||D ( x1 ) −D ( x2 ) || ≤ L||x1 − x2|| , ∀ x1 , x2 ∈ Rdx , ( 6 ) for some Lipschitz constant 0 < L ≤ 1 . However , most predefined denoisers do not satisfy this assumption . An alternative approach [ 40 , 36 , 46 ] is to learn a nonexpansive denoiser by adding a regularization term to the training loss which constrains its Lipschitz constant . Unfortunately , this remains an open challenge since in general there are no practical means to absolutely enforce the resultant denoiser to be nonexpansive . Our work addresses both of these challenges . We first introduce a new class of denoisers which stem from an explicit potential functional and thus demonstrate symmetric Jacobian by their construction . We then derive an inverse technique that utilizes our denoisers as well as their potentials to ensure convergence . Three important remarks are to be made with regard to work presented below . First , we emphasize that the proposed denoisers are not intended to compete with state-of-the-art denoisers , but rather are developed to exhibit favorable properties that benefit other inversion tasks . Second , our proposed recovery method is not derived to compete with end-to-end learned solutions to specific inverse problems [ 31 , 2 , 25 , 26 ] . Rather , our learned denoisers , along with our optimization framework , can be used for all inverse problems ( of the form in Eq . ( 1 ) ) without additional training . Lastly , while we present a present a specific inversion method , the potential-driven denoisers can benefit many RED and PnP algorithms , hence , our work complements these frameworks .

Summary:  This paper considers the problem of designing an image denoiser that can be used as a prior within plug-and-play priors (PnP) or regularization by denosing (RED). Prior work has proved the convergence of PnP/RED for contractive, nonexpansive, and/or firmly nonexpansive denoisers. The design of denoisers that enable PnP/RED convergence is an active research area.  This paper focuses on denoisers that correspond to the gradients of some potential functions. By using such denoisers within RED, one can guarantee convergence via traditional optimization. The key idea in the paper is to use a deep neural network to parameterize a potential function (instead of the denoiser), then train that function for denoising by using its gradient. Additionally, when the weights of the network are nonlinear and activation functions are nondecreasing and convex, the denoiser corresponds to a convex function.  The key contribution of the paper is the formulation of a procedure for training a potential function that can be used to obtain a denoiser. The theoretical arguments presented in Lemma 2 and Theorem 1, are well known in optimization (see below for reference). Similarly, Algorithm 1 corresponds exactly to the RED-SD algorithm using a potential-driven denoiser. The empirical results compare three proposed variants of denoisers against PnP-PGD and RED-SD on problems of image deblurring and superresolution.