Recovery Analysis for Plug-and-Play Priors using the Restricted Eigenvalue Condition

1 Introduction . Many imaging problems—such as denoising , inpainting , and super-resolution—can be formulated as an inverse problem involving the recovery of an image x∗ ∈ Rn from noisy measurements y = Ax∗ + e , ( 1 ) where A ∈ Rm×n is the measurement operator and e ∈ Rm is the noise . Compressed sensing ( CS ) [ 1 , 2 ] is a related class of inverse problems that seek to recover a sparse vector x∗ from m < n measurements . The sparse recovery is possible under certain assumptions on the measurement matrix , such as the restricted isometry property ( RIP ) [ 1 ] or the restricted eigenvalue condition ( REC ) [ 3 , 4 ] . While traditional CS recovery relies on sparsity-promoting priors , recent work on compressed sensing using generative models ( CSGM ) [ 5 ] has broadened this perspective to priors specified through pre-trained generative models . CSGM has prompted a large amount of follow-up work on the design and theoretical analysis of algorithms that can leverage generative models as priors for image recovery [ 6–9 ] . Plug-and-play priors ( PnP ) [ 10 , 11 ] and regularization by denoising ( RED ) [ 12 ] are two methods related to CSGM that can also leverage pre-trained deep models as priors for inverse problems . However , unlike CSGM , the regularization in PnP/RED is not based on restricting the solution to the range of a generative model , but rather on denoising the iterates with an existing additive white Gaussian noise ( AWGN ) removal method . The effectiveness of PnP/RED has been shown in a number of inverse problems [ 13–18 ] , which has prompted researchers to investigate the theoretical properties and interpretations of PnP/RED algorithms [ 19–30 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Despite the rich literature on both PnP/RED and CSGM , the conceptual relationship between these two classes of methods has never been formally investigated . In particular , while PnP/RED algorithms enjoy computational advantages over CSGM by not requiring nonconvex projections onto the range of a generative model , they lack theoretical recovery guarantees available for CSGM . In this paper , we address this gap by presenting the first recovery analysis of PnP/RED under the assumptions of CSGM . We show that if a measurement matrix satisfies a variant of REC from [ 5 ] over the range of a denoiser , then the distance of the PnP solutions to the true x∗ can be explicitly characterized . We also present conditions under which the solutions of both PnP and RED coincide , providing sufficient conditions for the exact recovery of x∗ using both methodologies . Our results highlight that the regularization in PnP/RED is achieved by giving preference to images near the fixed points of pre-trained deep neural networks . Besides new theory , this paper also presents numerical results directly comparing the recovery performance of PnP/RED against the recent algorithms in compressed sensing from random projections and subsampled Fourier measurements . These numerical results lead to new insights highlighting the excellent recovery performance of both PnP and RED , as well as the benefit of using priors specified as pre-trained artifact removal ( AR ) operators rather than AWGN denoisers . All proofs and some technical details that have been omitted for space appear in the Supplement , which also provides more background and simulations . The code for our numerical evaluation is available at : https : //github.com/wustl-cig/pnp-recovery . 2 Background . Inverse problems . A common approach to estimating x∗ in ( 1 ) is to solve an optimization problem : min x∈Rn g ( x ) + h ( x ) with g ( x ) = 1 2 ‖y −Ax‖22 , ( 2 ) where g is a data-fidelity term that quantifies consistency with the observed data y and h is a regularizer that encodes prior knowledge on x . For example , a widely-used regularizer in inverse problems is the nonsmooth total variation ( TV ) function h ( x ) = τ‖Dx‖1 , where D is the gradient operator and τ > 0 is the regularization parameter [ 31–33 ] . Compressed sensing using generative models . Generative priors have recently become popular for solving inverse problems [ 5 ] , which typically require solving the optimization problem : min z∈Rk 1 2 ‖y −AW ( z ) ‖22 , ( 3 ) where W : Rk → Im ( W ) ⊆ Rn is a pre-trained generative model , such as StyleGAN-2 [ 34 , 35 ] . The set Im ( W ) is the image set ( or the range set ) of the generator W. In the past few years , several algorithms have been proposed for solving this optimization problem [ 6–9 ] , including the recent algorithms PULSE [ 36 ] and intermediate layer optimization ( ILO ) [ 37 ] that can recover highlyrealistic images . The recovery analysis of CSGM was performed under the assumption that A satisfies the set-restricted eigenvalue condition ( S-REC ) [ 5 ] over the range of the generative model : ‖Ax−Az‖22 ≥ µ‖x− z‖22 − η ∀x , z ∈ Im ( W ) , ( 4 ) where µ > 0 and η ≥ 0 . S-REC implies that the pairwise distances between vectors in the range of the generative model must be well preserved in the measurement space . It thus broadens the traditional notions of REC and the restricted isometry property ( RIP ) in CS beyond sparse vectors [ 38 ] . PnP and RED . PnP [ 10 , 11 ] refers to a family of iterative algorithms that are based on replacing the proximal operator proxγh of the regularizer h within a proximal algorithm [ 39 ] by a more general denoiser D : Rn → Im ( D ) ⊆ Rn , such as BM3D [ 40 ] or DnCNN [ 41 ] . For example , the widely used proximal gradient method ( PGM ) [ 42–45 ] can be implemented as a PnP algorithm as [ 46 ] xk = T ( xk−1 ) with T : = D ( I− γ∇g ) , ( 5 ) where g is the data-fidelity term in ( 2 ) , I denotes the identity mapping , and γ > 0 is the step size . Remarkably , this heuristic of using denoisers not associated with any h within a proximal algorithm exhibited great empirical success [ 13–18 ] and spurred a great deal of theoretical work on PnP [ 19–28 ] . In particular , it has been recently shown in [ 23 ] that , when the residual of D is Lipschitz continuous , PnP-PGM converges to a point in the fixed-point set of the operator T that we denote Fix ( T ) . RED [ 12 ] is a related method , inspired by PnP , for integrating denoisers as priors for inverse problems . For example , the steepest descent variant of RED ( SD-RED ) [ 12 ] can be summarized as xk = xk−1 − γG ( xk−1 ) with G : = ∇g + τ ( I− D ) , ( 6 ) where γ > 0 is the step size and τ > 0 is the regularization parameter . For a locally homogeneous D that has a strongly passive and symmetric Jacobian , the solution of RED solves ( 2 ) with h ( x ) = ( τ/2 ) xT ( x− D ( x ) ) [ 12 , 22 ] . Subsequent work has resulted in a number of extensions of RED [ 29 , 30,47–49 ] . For example , it has been shown in [ 29 ] that , when D is a nonexpansive operator , SD-RED converges to a point in the zero set of operator G that we denote as Zer ( G ) . Other related work . While not directly related to our main theoretical contributions , it is worth briefly mentioning other important related families of algorithms that also use deep neural nets for regularizing ill-posed imaging inverse problems ( see recent reviews of the area [ 50–53 ] ) . This work is most related to methods that rely on pre-trained priors that are integrated within iterative algorithms , such as a class of algorithms in compressive sensing known as approximate message passing ( AMP ) [ 54–57 ] . Another related family of algorithms are those based on the idea of deep unrolling ( for an overview see Section IV-A in [ 53 ] ) . Inspired by LISTA [ 58 ] , the unrolling algorithms interpret iterations of a regularized inversion as layers of a CNN and train it end-to-end in a supervised fashion [ 59–64 ] . Deep image prior [ 65 ] and deep decoder [ 66 ] also use neural networks as prior for images ; instead of using a pre-trained generative network , they learn the parameters of the network while solving the inverse problem using the available measurements . 3 Recovery Analysis for PnP and RED . We present two sets of theoretical results for PnP-PGM ( 5 ) using the measurement model ( 1 ) and the least-squares data-fidelity term ( 2 ) . We first establish recovery bounds for PnP under a set of sufficient conditions , and then address the relationship between the solutions of PnP and RED . The proofs of all the theorems will be provided in the Supplement . We start by discussing two assumptions that serve as sufficient conditions for our analysis of PnP . Assumption 1 . The residual R : = I−D of the operator D is bounded by δ and Lipschitz continuous with constant α > 0 , which can be written as ‖R ( x ) ‖2 ≤ δ and ‖R ( x ) − R ( z ) ‖2 ≤ α‖x− z‖2 , ∀x , z ∈ Rn . ( 7 ) The rationale for stating Assumption 1 in terms of the residual R is based on our interest in residual deep neural nets that take a noisy or an artifact-corrupted image at the input and produce the corresponding noise or artifacts at the output . The success of residual learning in the context of image restoration is well known [ 41 ] . Prior work has also shown that Lipschitz constrained residual networks yield excellent performance without sacrificing stable convergence [ 23 , 29 ] . Related assumptions have been used in earlier convergence results for PnP [ 19 , 23 ] . For example , one of the most-widely known PnP convergence results relies on the boundedness of D [ 19 ] . The Lipschitz continuity of the residual R has been used in the recent analysis of several PnP algorithms in [ 23 ] . Both of these assumptions are relatively easy to implement for deep priors . For example , the boundedness of R can be enforced by simply bounding each output pixel to be within [ 0 , ν ] for images in [ 0 , ν ] n ⊂ Rn for some ν > 0 . The α-Lipschitz continuity of R can be enforced by using any of the recent techniques for training Lipschitz constrained deep neural nets [ 23 , 67–69 ] . Fig . 1 presents an empirical evaluation of the Lipschitz continuity of R used in our simulations . Assumption 2 . The measurement operator A ∈ Rm×n satisfies the set-restricted eigenvalue condition ( S-REC ) over Im ( D ) ⊆ Rn with µ > 0 , which can be written as ‖Ax−Az‖22 ≥ µ‖x− z‖22 , ∀x , z ∈ Im ( D ) . ( 8 ) S-REC in Assumption 2 was adopted from the corresponding assumption for CSGM stated in ( 4 ) , which establishes a natural conceptual link between those two classes of methods . The main limitation of Assumption 2 , which is also present in the traditional RIP/REC assumptions for compressive sensing , lies in the difficulty of verifying it for a given measurement operator A . There has been significant activity in investigating the validity of related conditions for randomized matrices for different classes of signals [ 4,70–72 ] , including for those synthesized by generative models [ 5,6,9,37 ] . Despite this limitation , Assumption 2 is still conceptually useful as it allows us to relax the strong convexity assumption used in the convergence analysis in [ 23 ] by stating that it is sufficient for the strong convexity to hold only over the image set Im ( D ) rather than over the whole Rn . This suggests a new research direction for PnP on designing deep priors with range spaces restricted to satisfy S-REC for some A . In the Supplement , we present an empirical evaluation of µ for the measurement operators used in our experiments by sampling from Im ( D ) . Consider the set Fix ( D ) : = { x ∈ Rn : x = D ( x ) } of the fixed points of D. Note that Fix ( D ) is equivalent to the set Zer ( R ) : = { x ∈ Rn : R ( x ) = 0 } of the zeros of the residual R = I − D. Intuitively , Zer ( R ) consists of all images that produce no residuals , and therefore can be interpreted as the set of all noise-free images according to the network . Similarly , when R is trained to predict artifacts in an image , Zer ( R ) is the set of images that are artifact-free according to R. In the subsequent analysis , we use the notation Zer ( R ) , but these results can be equivalently stated using Fix ( D ) . We first state the PnP recovery in the setting where there is no noise and x∗ ∈ Zer ( R ) . Theorem 1 . Run PnP-PGM for t ≥ 1 iterations under Assumptions 1-2 for the problem ( 1 ) with no noise and x∗ ∈ Zer ( R ) . Then , the sequence xt generated by PnP-PGM satisfies ‖xt − x∗‖2 ≤ c‖xt−1 − x∗‖2 ≤ ct‖x0 − x∗‖2 , ( 9 ) where x0 ∈ Im ( D ) and c : = ( 1 + α ) max { |1− γµ| , |1− γλ| } with λ : = λmax ( ATA ) . The proof of the theorem is available in the Supplement . Theorem 1 extends the theoretical analysis of PnP in [ 23 ] by showing convergence to the true solution x∗ of ( 1 ) instead of the fixed points Fix ( T ) of T in ( 5 ) . The condition x0 ∈ Im ( D ) can be easily enforced by simply passing any initial image through the operator D. One does not necessarily need α < 1 , for the convergence result in Theorem 1 . As shown in [ 23 ] , the coefficient c in Theorem 1 is less than one if 1 µ ( 1 + 1/α ) < γ < 2 λ − 1 λ ( 1 + 1/α ) , ( 10 ) which is possible if α < 2µ/ ( λ− µ ) . Since all PnP algorithms have the same fixed points [ 20 , 24 ] , our result implies that PnP can exactly recover the true solution x∗ to the inverse problem , which extends the existing theory in the literature that only shows convergence to Fix ( T ) . We now present a more general result that relaxes the assumptions in Theorem 1 . Theorem 2 . Run PnP-PGM for t ≥ 1 iterations under Assumptions 1-2 for the problem ( 1 ) with x∗ ∈ Rn and e ∈ Rm . Then , the sequence xt generated by PnP-PGM satisfies ‖xt − x∗‖2 ≤ c‖xt−1 − x∗‖2 + ε ≤ ct‖x0 − x∗‖2 + ε ( 1− ct ) ( 1− c ) , ( 11 ) where x0 ∈ Im ( D ) and ε : = ( 1 + c ) [ ( 1 + 2 √ λ/µ ) ‖x∗ − projZer ( R ) ( x∗ ) ‖2 + 2/ √ µ‖e‖2 + δ ( 1 + 1/α ) ] ( 12 ) and c : = ( 1 + α ) max { |1− γµ| , |1− γλ| } with λ : = λmax ( ATA ) . Theorem 2 extends Theorem 1 by allowing x∗ to be outside of Zer ( R ) and extends the analysis in [ 5 ] by considering operators D that do not necessarily project onto the range of a generative model . In the error bound ε , the first two terms are the distance of x∗ to Zer ( R ) and the magnitude of the error e , and have direct analogs in standard compressed sensing . The third term is the consequence of the possibility for the solution of PnP not being in the zero-set of R and one can show that when Zer ( R ) ∩ Zer ( ∇g ) 6= ∅ , then the third term disappears . As reported in the Supplement , we empirically verified that the distance of the PnP solution to Zer ( R ) is small for both the denoiser and the artifact-removal operators used in our experiments . Our final result explicitly relates the solutions of PnP and RED . In order to obtain the result , we need an additional assumption that the denoiser D = I− R is nonexpansive . Assumption 3 . The denoiser D is nonexpansive ‖D ( x ) − D ( z ) ‖2 ≤ ‖x− z‖2 , ∀x , z ∈ Rn . This is related but different from Assumption 1 that assumes the residual R is α-Lipschitz continuous . The convergence of SD-RED in ( 6 ) to Zer ( G ) can be established for a nonexpansive operator D [ 29 ] . In principle , the nonexpansiveness of D can be enforced during the training of the prior in the same manner as that of the more general Lipschitz continuity . However , the prior in our numerical evaluations is trained to have a contractive residual R without any explicit constraints on D. As a reminder , the nonexpansiveness of R is only a necessary ( but not sufficient ) condition for the nonexpansiveness of D [ 73 ] . Despite this fact , our empirical evaluation of the Lipschitz constant of D in Fig . 1 indicates that D used in our experiments is nonexpansive . Theorem 3 . Suppose that Assumptions 1-3 are satisfied and that Zer ( ∇g ) ∩ Zer ( R ) 6= ∅ , then PnP and RED have the same set of solutions : Fix ( T ) = Zer ( G ) . As a reminder , the solutions of PnP correspond to the fixed-points of the operator T defined in ( 5 ) , while those of RED to the zeroes of the operator G defined in ( 6 ) . The assumption that Zer ( ∇g ) ∩ Zer ( R ) 6= ∅ implies that there exist vectors that are noise/artifact free according to R and consistent with the measurements y . While this assumption is not universally applicable to all the inverse problems and priors , it still provides a highly-intuitive sufficient condition for the PnP/RED equivalence . Although the relationship between PnP and RED has been explored in the prior work [ 22 , 30 ] , to the best of our knowledge , Theorem 3 is the first to prove explicit equivalence . If one additionally considers PnP-PGM with a step size that satisfies the condition in ( 10 ) , then T is a contraction over Im ( D ) , which implies that PnP-PGM converges linearly to its unique fixed point in Im ( D ) . The direct corollary of our analysis is that , in the noiseless scenario y = Ax∗ with x∗ ∈ Zer ( R ) , the image x∗ is the unique fixed point of both PnP and RED over Im ( D ) . It is worth mentioning that several of our assumptions have been stated in a way that simplifies mathematical exposition , but can alternatively be presented in a significantly weaker form . For example , the boundedness assumption in ( 1 ) —which is used only in the proof of Theorem 2—does not have to hold everywhere , but only at the fixed points of PnP . Indeed , as can be seen in eq . ( 6 ) of the Supplement , the constant δ is only used to bound the norm of the residual at the fixed-point of PnP-PGM . Similarly , we do not need the nonexpansiveness of D in Assumption 3 to be true everywhere , but only at the fixed points of RED-SD ( see page 4 of the Supplement ) . In summary , our theoretical analysis reveals that the fixed-point convergence of PnP/RED algorithms can be strengthened to provide recovery guarantees when S-REC from CSGM is satisfied . Since PnP/RED algorithms do not require nonconvex projections onto the range of a generative model , they enjoy computational benefits over methods that use generative models as priors . However , the literature on generative models is rich with theoretical bounds and recovery guarantees compared to that of PnP/RED . We believe that our work suggests an exciting new direction of research for PnP/RED by showing that a similar analysis can be carried out for PnP/RED .

Summary: In this work, the authors analyze Plug and Play (PnP) methods for solving inverse problems, and utilize the framework from compressive sensing with generative models to derive theoretical guarantees. In particular, the authors show that in a linear inverse problem, if 1) the residual operator of a denoiser is Lipschitz and 2) the measurement matrix satisfies a restricted eigenvalue condition over the range of the denoiser, then linear convergence of the PnP algorithm can be established for suitable parameter choices. Moreover, under a similar set of assumptions, an equivalence between PnP and Regularization by Denoising (RED) is derived, in that both algorithms have the same set of solutions. Empirical evaluations also show 1) supporting evidence for aspects of the theory holding for trained neural network-based denoisers and 2) that PnP algorithms compete with, and can outperform, state-of-the-art compressive sensing approaches using deep generative priors.