Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: 
This paper considers the problem of designing an image denoiser that can be used as a prior within plug-and-play priors (PnP) or regularization by denosing (RED). Prior work has proved the convergence of PnP/RED for contractive, nonexpansive, and/or firmly nonexpansive denoisers. The design of denoisers that enable PnP/RED convergence is an active research area.

This paper focuses on denoisers that correspond to the gradients of some potential functions. By using such denoisers within RED, one can guarantee convergence via traditional optimization. The key idea in the paper is to use a deep neural network to parameterize a potential function (instead of the denoiser), then train that function for denoising by using its gradient. Additionally, when the weights of the network are nonlinear and activation functions are nondecreasing and convex, the denoiser corresponds to a convex function.

The key contribution of the paper is the formulation of a procedure for training a potential function that can be used to obtain a denoiser. The theoretical arguments presented in Lemma 2 and Theorem 1, are well known in optimization (see below for reference). Similarly, Algorithm 1 corresponds exactly to the RED-SD algorithm using a potential-driven denoiser. The empirical results compare three proposed variants of denoisers against PnP-PGD and RED-SD on problems of image deblurring and superresolution.
main_review: Strengths: The central idea in the paper nicely complements the original work on RED, which attempted to obtain a PnP algorithm minimizing an explicit objective function. The recent literature on PnP/RED has focused on using deep neural nets to parameterize the denoiser. This paper argues that by instead parameterizing the potential function, one can obtain an explicit objective under milder assumptions than in the original RED paper. 

Issues: (a) The first major issue in this publication is the discussion in “Section 3.2 Convergent Inversion Method”. The theoretical results claimed in that section in Lemma 2 and Theorem 1 are standard results from optimization (see, for example, “Section 1.2.3 Gradient method” in Nesterov's book “An Introductory Lectures on Convex Optimization”). One can simply cut these results by including references and save 1 page for providing other details. (b) The second major issue in the paper is in inadequate simulations. The paper seems to want to show that the proposed denoiser leads to (i) convergence; (ii) better solutions in terms of PSNR/SSIM than PnP/RED with other denoisers. Regarding (i): (1) There are a number of variants of PnP/RED that achieve provable convergence using nonexpansive variants of DnCNN (see denoiser in [40]). Why would you pick the one that makes PnP-PGM diverge? (2) Having a convex optimization reference, such as FISTA-TV, would help to understand the benefit of having the potential function. (3) Having convergence plots in terms of PSNR (dB) would help to see the speed of improvement in imaging quality. (4) Having a basic deep learning reference such as UNet would help to evaluate that all methods are properly implemented and take advantage of the measurement model. (5) Having a convergence plot in terms of the objective function, would help to validate that the GraDnCNN is indeed minimizing an objective. Regarding (ii): Why would we expect the existence of the potential function to lead to better imaging quality? 

Summary: This paper is proposing an intriguing alternative to training nonexpansive denoisers for PnP/RED. While “it has potential”, the paper falls short in terms of fully convincing the benefit of the proposed method for solving inverse problems. Specifically, experimental validation is incomplete and needs to be significantly improved following my suggestions above. Additionally, the paper claims a number of classical results from optimization as contributions.
limitations_and_societal_impact: No issues
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary: 
- This submission deals with designing stable and smooth denoisers that could be directly used for regularization, or for plug-and-play (PnP) methods to solve inverse problems. 
- This is an improvement upon existing denoisers such as RED and PnP which are not necessarily stable. 
- The key idea is to craft neural networks that output scalar-valued potential function g whose gradient is the denoiser residual. To guarantee convergence and stability, g has to be smooth and h=1/2x^2 - g convex. 
- Advantages: the denoisers lead to symmetric Jacobians, allowing for a MAP estimation; denoisers can be integrated into optimization schemes with backtracking step size, and thus no need for enforcing Lipschitz continuity.
- To construct such a network, GraDnCNN stacks conv layers and nonlinear activations, and proposes DnICNNs with non-negative weights and convex activations. Variants of DnICNN are discussed that achieve better denoising/recovery performance for Gaussian deblurring, and image superresolution.

Contributions:
- construct stable denoisers based on neural networks as gradients of smooth potential functions g
- integrate the denoisers as a regularizer for solving inverse problems and test it out for gaussian image deblurring and superresolution which outperforms RED and traditional PnP methods

main_review: Originality and significance
- a novel perspective to craft CNNs for designing stable denoiser that leverage the power of neural nets in a guaranteed manner
- The design seems to be limiting, needing non-negative weights, and thus the scope and usefulness of this theory is not very clear. This is discussed in the comments next.

Clarity:
- This paper is crowded, and it discusses different aspects of the potential based denosiers, so the main novelty of this work is lost. 
- It could benefit from a better organization. More details about the architecture of ICNN and requirements on the weights would help better understand the paper. This reviewer doesnt think the discussions about the convergence are of high priority as those come natural given the constraints of convexity and smoothness on the denoiser. 

Comments:
- This is an Interesting connection between convex regularization and CNNs. How far the network design can go? Let’s say we have off-the-shelf deep CNN based denoisers such as U-net and want to see their corresponding potential function, and regularization function.  It's important for the theory to be insightful and suggest good architectures. The Author's comment would be helpful.

- About the GDCNN, what restications on the weights and layers are required to make sure the output is a legitimate gradient for a lipschitz and convex denoiser? It’s not clear that the non-convex training objective in (9) would naturally lead to network weights that would produce legitimate gradients?

- The constraints on the design of DnICNN seem to be pretty stringent; non-negative weights and convex activations. It’s strange that no weight normalization is needed to ensure lipschitzness. Is that correct?

- If I understand correctly, only ICNN is convex. DnDICNN and GraDnCNN are not legitimate for stability. right?

- in order to see the usefulness of the connection between regularization and neural nets, it is useful to include off-the-shelf denoisers in the experiments Fig. 2,3. One such example is unrolled networks that are trained end-to-end, or the inverse problem with the generative model projection. Glancing through the literature, a few examples, very relevant to this work include [1], [2]. 


[1] Mardani, M., Sun, Q., Papyan, V., Monajemi, H., Vasanawala, S., Pauly, J., Donoho, D. (2018). Neural Proximal Gradient Descent for Compressive Imaging. Advances in Neural Information Processing Systems, 31, 9573-9583.

[2] Van Veen D, Jalal A, Soltanolkotabi M, Price E, Vishwanath S, Dimakis AG. Compressed sensing with deep image prior and learned regularization. arXiv preprint arXiv:1806.06438. 2018 Jun 17.


limitations_and_societal_impact:
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes image denoising based CNNs as regularizers that can be constructed as the gradient of smooth potential functions $g$. The main promise is mentioned to be making an optimization algorithm has explicit regularization, when equipped with those denoisers. Unlike original regularization by denoising (RED) that the explicit regularizer exists if the denoisers satisfy several strict conditions, the proposed denoisers only rely on smoothness. Particularly, the denoisers consist of smooth activation functions to ensure the differential continuity and are trained in a residual denoising fashion using MSE loss. As a result, the proposed algorithm can be formulated as steepest descent variant of RED optimized by directly adopting the gradient of the objective function $F$. Standard back-tracking is used to ensure convergence. A convergence analysis is provided following the common optimization theory. Finally, the performance of the proposed method is evaluated over two image inverse problems such as gaussian deblur and super-resolution, with satisfactory results compared to existing methods based on PnP and RED. 
main_review: **Originality:** This paper is well motivated and is written fairly well. The idea sounds simple but promising and makes it possible to be directly applied for other variant RED algorithms. Most importantly, this paper provides a new guide of training CNN denoisers as explicit regularizers, unlike current literature, where the link between denoisers and regularization is commonly missing.

**Quality & Clarity:**  Overall the paper is  easy to read. The background review of the related work is comprehensive and clearly stated. However, I found it hard to follow at some specific sections due to insufficient details on the notations.  For one, it is a little bit confusing for me about the function $h$ mentioned in Lemma 1, line167 -- 172 and the one in Eq. 15.  

**Significance:** I think this is a strong paper, in a line of research that is interesting and should be of interest to the NeurIPS community.

limitations_and_societal_impact: **Weakness:**

- Using the assumption that the denoiser $D$ is the proximal operator of some convex scaler function is to be fine to obtain some fundamental insights for the link between the CNN denoiser and regularization, but to me, it also relies on the convexity of the denoiser. Considering that Algorithm 1 equipped with non-convex denoiser achieves the best performance, it seems to be tricky to follow _Lemma 1_ to design the desnoisers.

- For PnP-PGD, RED-SD and the proposed method with GraDnCNN using DnCNN are trained without lipschitz constrain for $L < 1$, resulting a non-convex problem. Hence an initial point of the update sometimes is important. It would be interesting to see how different initialization influence the final reconstruction results.

- Similar to the above, both PnP and RED led to poor results compared to most of existing works. This may be due to the improper parameter tunning such as regularization trade-off $\lambda$, step-size $\gamma$ and denoising intensity. 
A recent work by (Xu’2021) introduce a simply way enable the tuning of the regularization parameter within PnP, it is not clear for me why this easy approach is not applied for PnP.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Motivated by the RED algorithm for solving inverse problems, the authors propose a neural-net denoiser that is the gradient of an explicit potential function.  This guarantees that the Jacobian is symmetric definite, which then guarantees that the RED algorithm minimizes an explicit cost function, which in turn makes convergence straightforward to prove, thereby solving an important open problem about the convergence of RED and plug-and-play (PnP)-type algorithms.  The fact that there is an explicit cost also enables the use of adaptive stepsize selection within RED, with one particular scheme proposed by the authors.  Numerical comparisons to RED and PnP algorithms show improved PSNR and SSIM performance, as well as enhanced stability.
main_review: I think the overall idea is great: design a neural-net denoiser that---by construction---equals the gradient of an explicit potential function.  To my knowledge, this is novel and it solves an important open problem about the convergence of PnP/RED methods.

But I fear that the numerical experiments are seriously flawed.  There are two reasons.
1. It's important to understand that the fixed points of PnP and RED algorithms are directly affected by stepsizes used in those algorithms (please see [3, p.108 and p.112]).  This stands in sharp contrast to convex optimization, where the stepsize does not affect the fixed point.  Thus, to show the true potential of PnP/RED, one must tune the stepsize.  This can be easily accomplished using the validation data.
2. There is a compounding problem, which has to do with differences among PnP/RED algorithms.  In the convex optimization setting, the PGD algorithm is unstable for large stepsizes, unlike the ADMM and primal-dual splitting (PDS) algorithms, where the stepsize can be chosen arbitrarily.  In the context of PnP, the PGD, ADMM, and PDS algorithms all have the same fixed points for a given stepsize (please see [3, p.111]), but PnP-PGD places severe limits on that stepsize, making many of those fixed points unstable.  In practice, the best stepsizes (e.g., in the sense of MSE or SSIM) are typically unstable for PnP-PGD, meaning that optimally tuned PnP-PGD is not competitive with optimally tuned PnP-ADMM and PnP-PDS.  Similar issues may arise with different RED algorithms.
In conclusion, if the goal is to compare to PnP algorithms, then PnP-PGD must be avoided and either PnP-ADMM or PnP-PDS must be properly tuned.  Likewise, the RED algorithm must be chosen wisely and tuned.  The experiments in the paper show PnP failing completely, but this is simply because the authors have not made a fair comparison to PnP.  Likewise, RED was not tuned and so it does not appear that the RED comparison is fair either.

Minor comments:
1. In the Related Work section, it may help to know that [3] is a comprehensive overview of PnP & RED methods and not just an application of PnP & RED to MRI.
2. In the Related Work section, for completeness, it may help to note that the AMP [7] and VAMP [17] frameworks are unique in guaranteeing that an MMSE denoiser yields and MMSE solution to the inverse problem, at least under large random forward operators.  In the same context, they prove convergence for Lipschitz denoisers.
3. Equation (13) is missing subscript k on x in two places.  Also, it should be noted immediately after (13) that (13) is RED-SD.
4. I suggest removing "It Has Potential" from the title.  You could add this phrase to any paper; it has no value.

limitations_and_societal_impact: The reviewer does not see any potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: **Summary**:
This paper addresses an important issue in the framework of regularization by denoising (RED), which is that the explicit RED regularizer $\frac{1}{2}x^T(x-D_\sigma(x))$ exists only if a set of strict conditions on the denoiser are met. Due to this problem, RED fails to formulate explicit regularizers for advanced denoisers (e.g. CNN) since they generally do not satisfy the conditions. Although some work has established some theoretical analysis for RED algorithms using deep denoisers, a linkage between the denoiser and the regularizer is generally missing. 

In this paper, the authors proposed to formulate the regularizer as a *smooth potential functional* $g(x)$, which is parameterized by a CNN with smooth activation functions. To link the potential functional to a denoiser, the authors trained the CNN such that the gradient $\nabla g(x)$ outputs the noise residual for the AWGN. In this way, the proposed method can formulate the gradient-based RED update by directly taking the gradient of the objective function $F(x)$.

Since the objective function is available, the authors also proposed to use the standard back-tracking trick in their framework to guarantee convergence. Additionally, a convergence analysis is also introduced by following the common nonconvex optimization analysis given the handy $F(x)$. 
main_review: **Originality**: Overall, this paper constitutes a solid contribution to the study of RED, proposing a new way to train a denoising prior and linking it back to a regularizer.

**quality & clarity**: The paper is well-written and easy to read. The background review of the related work is very thorough and comprehensive

**Significance**: I believe their contribution is of enough significance to the community.

limitations_and_societal_impact: **Weakness**

1. In the description under eq. (7), it would be clearer if the authors can explicitly link $\mathcal{W}$ and $\mathcal{V}$ to the convolutional operations. 

2. Learning a denoiser as the gradient of a convex function is very interesting. But I do not get why having two convex potential $g_{\theta_1}$ and $g_{\theta_1}$ subtracting each other will increase the expressivity. 

3. It is nice to have a back-tracking algorithm implemented in RED since the Lipschitz constant of a CNN is not easy to estimate. However, it seems to me that *Liu et al* has also proposed a back-tracking algorithm for RED. What is the difference between yours and theirs?

4. With all due respect and no intention to undermine the novelty, it is a little bit disappointing for me to see that there is only one theorem that only accounts for the nonconvex case. Given the fact that the authors also proposed a convex denoising functional, it may improve the completeness of the paper if the author can include another short theorem (even if it is well-known) or some remarks on the convergence of the proposed framework under convex $g(x)$.

5. In the experiments, the batch normalization and biases are removed for DnCNN, which may significantly reduce the representation power of the network. Can the authors explain this? On the contrary, the proposed network architecture has biases, which makes the comparison unfair.

6. Both PnP and RED achieve degraded performance compared with the results in other PnP/RED papers. I guess this is mainly due to the modified DnCNN architecture. The authors should compare their method with PnP/RED using standard DnCNN (with either batch normalization or biases). Moreover, please also compare against the Lipschitz-constrained (nonexpansive) DnCNN, which demonstrated good performance as well as convergence in many works.

7. Some missing citations related to RED, back-tracking, and learning functionals:

    [1] Sun et al Async-RED: A Provably Convergent Asynchronous Block Parallel Stochastic Method using Deep Denoising 
    Priors. ICLR 2021

    [2] Liu et al RARE: Image Reconstruction Using Deep Priors Learned without Groundtruth. IEEE JSTSP 2020

    [3] Lunz et al Adversarial Regularizers in Inverse Problems NeurIPS 2018
review_title: null

Meta-review: This paper is proposing a new and interesting idea: a potential-based neural net denoiser that makes RED convergence analysis clean and simple. This is quite novel and suitable to NeurIPS audience. The reviewers raised serious concerns on the evaluation and how the comparisons to other RED methods were performed. I agree that the evaluation should be improved and the authors did a reasonable job on responding to reviewer input on this. I recommend that the authors revise their manuscript taking all this input into account. The analysis on convex functions is also a good suggestion that the authors should take into consideration.  


Decision: Accept (Poster)