Stochastic Deep Restoration Priors for Imaging Inverse Problems

Authors: Yuyang Hu, Albert Peng, Weijie Gan, Peyman Milanfar, Mauricio Delbracio, Ulugbek S. Kamilov
Primary area: optimization
Keywords: computational imaging, inverse problems, deep learning

Abstract:
Deep neural networks trained as image denoisers are widely used as priors for solving imaging inverse problems. While Gaussian denoising is thought sufficient for learning image priors, we show that priors from deep models pre-trained as more general restoration operators can perform better. We introduce Stochastic deep Restoration Priors (ShaRP), a novel method that leverages an ensemble of such restoration models to regularize inverse problems. ShaRP improves upon methods using Gaussian denoiser priors by better handling structured artifacts and enabling self-supervised training even without fully sampled data. We prove ShaRP minimizes an objective function involving a regularizer derived from the score functions of minimum mean square error (MMSE) restoration operators, and theoretically analyze its convergence. Empirically, ShaRP achieves state-of-the-art performance on tasks such as magnetic resonance imaging reconstruction and single-image super-resolution,  surpassing both denoiser- and diffusion-model-based methods without requiring retraining.

Decision: reject
Reviewer scores: 3, 5, 6, 1, 6

Reviews and discussion:
<thread>
creation date:
1729776967266

thread_type:
official_review

<thread_object>
creator: reviewer_bq8e

<content>
summary:
This paper introduces Stochastic deep Restoration Priors (ShaRP), a method that leverages an ensemble of deep restoration models to regularize imaging inverse problems. ShaRP improves upon Gaussian denoiser-based methods by handling structured artifacts more effectively, enabling self-supervised training without fully sampled data.

strengths:
- The paper is well-written and easy to follow.
- The theoretical analysis of the convergence is thorough and well-explained.

weaknesses:
1. **Comparison to Diffusion-Based Methods:**
   The paper lacks a comprehensive comparison to diffusion-based methods, such as DiffIR [1] and DDRM [2]. Including these comparisons would strengthen the results and provide clarity on how ShaRP performs relative to other leading methods in the field.

2. **Supervised vs. Self-Supervised ShaRP:**
   In line 154, the authors mention a key contribution: "We implement ShaRP with both supervised and self-supervised restoration models as priors and test it on two inverse problems: CS-MRI and SISR." However, the experimental section does not provide a direct comparison between the self-supervised and supervised versions of ShaRP for the same task. This comparison is necessary to assess the benefits of the self-supervised approach.

3. **Self-Supervised Nature:**
   For a restoration network to be trained on a set of tasks, such as a set of blur kernels \(H_i\), access to ground truth data is still required. This approach, which involves sampling multiple times from the ground truth, raises questions about whether the method can truly be considered self-supervised. Clarification is needed regarding the self-supervised claim.

4. **Use of Multiple Degradation Operators:**
   The rationale for using a set of degradation operators \(H_1, H_2, \ldots, H_k\) in cases where the target problem involves only a single fixed operator (e.g., \(H_1\)) is unclear. It would be helpful if the authors could explain why introducing multiple degradation operators is necessary or beneficial when solving a fixed-task problem.

**References:**

- [1] DiffIR: Efficient Diffusion Model for Image Restoration
- [2] Denoising Diffusion Restoration Models

questions:
What is the practical inference time of the proposed method in comparison to state-of-the-art (SOTA) methods? Additionally, the visual comparisons presented do not clearly demonstrate significant improvements. It would be beneficial to include more compelling visual examples to better illustrate the advantages of the proposed approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730075705272

thread_type:
official_review

<thread_object>
creator: reviewer_e3gi

<content>
summary:
This paper extends the "deep restoration priors" approach from Hu et al 2024c from one degradation operator to multiple ones.

strengths:
The paper is clearly written, for the most part.

weaknesses:
This paper is a trivial extension of Hu et al 2024c.  In this paper, the degradation operator is randomly chosen from a set, whereas in Hu et al 2024c there was a single operator.  But this change led to no new challenges or questions.  For example, the "ShaRP" regularizer in (6) is a trivial extension of (9) in Hu et al 2024c that now takes an expectation over H.  Figure 2 in this paper is a direct copy of Figure 1 from Hu et al 2024c.  The experiments are conducted on different linear inverse problems, but there is no intellectual contribution there.

questions:
I don't have any questions for the authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730111194984

thread_type:
official_review

<thread_object>
creator: reviewer_kamm

<content>
summary:
This paper introduces a novel concept termed ShaRP for stochastic priors for the regularization of inverse problems. At its core, a set of $b$ restoration problems is considered, for which the MMSE is minimized. In this setting, ShaRP is the expectation of the probability of the degraded version of the images given the probability density of the observations. Under certain assumptions, a closed-form solution of the gradient of the regularizer is derived and the convergence of $\nabla f$ is shown. The results are numerically tested for CS-MRI and SISR.

strengths:
The paper introduces a novel concept for stochastic priors, which is complemented by two important theoretical assertions (see summary). The assumptions can be regarded as mild. Both results are highly relevant for numerical experiments. 
The paper remarkably advances SOTA for CS-MRI and is on par with competing methods in the case of SISR.

weaknesses:
The presentation of the paper could be improved in some places. In particular, I am missing a more concise introduction to the motivation behind the actual concept of ShaRP in Section 4.
The parameter $b$ representing the number of restoration problems is essential for the numerical performance. However, I am lacking a substantial discussion of the role of this parameter. In particular, no details about the role of $\alpha$ (see Section 5.1) and its impact on the results were provided.

questions:
1. Can you please provide more details about the impact of the choice of $b$ on the numerical results? An ablation study might help here. Likewise, you could numerically evaluate the impact of $\alpha$.
2. What happens if you modify the number of restoration priors in B.1.1?
3. Please rewrite the motivation in Section 4. In particular, I am missing a good motivation and reasoning for the actual definition in equation (6), which might help the reader to better understand ShaRP. In addition, the object $G_\sigma(s-Hx)$ could be better motivated from a mathematical point of view.
4. The results in Table 4 do not show a clear tendency for DiffPIR, DRP, or ShaRP. Is there a reason for this available? Are there specific conditions under which each method performs best?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730356391108

thread_type:
official_review

<thread_object>
creator: reviewer_xtix

<content>
summary:
The paper proposes a new approach for learning a regularizer for linear imaging inverse problems. While Gaussian denoisers have been successfully used as effective image priors, the authors extend the idea by using more general restoration operators as image priors. In the numerical experiments, the authors show the reconstruction performance with their new “ShaRP” regularizers for two important inverse problems (compressive MRI and single-image super-resolution). The numerical experiments are performed with ShaRP regularizers trained in supervised and self-supervised manners.

strengths:
1. Using general restoration operators instead of simple Gaussian denoisers is interesting and novel. Theorem 1 also puts this idea on a solid theoretical foundation. 

2. The experimental results are compelling. Comparison of ShaRP with different baseline methods, especially PnP algorithms that utilize Gaussian denoisers, provides evidence for the claim made by the authors in the abstract.

weaknesses:
1. I am not particularly convinced about the motivation for using a general restoration prior, that arises out of training a deep reconstruction operator for linear inverse problems (with several degradation operators), for solving another linear inverse problem. While learning Gaussian denoisers is a problem that is fundamentally easier than solving a general inverse problem (with an ill-posed operator), this is not the case here. 

2. The novelty in the convergence analysis (namely, the proof of Theorem 2) is unclear to me. To my understanding, the assumptions and the result are along the same lines as in the paper entitled “A Guide Through the Zoo of Biased SGD” by Demidovich et al. (which the authors have cited). 

3. Although the authors show the performance of shaRP trained in an unsupervised manner, I don’t think the training loss used in the unsupervised case (e.g., the loss in Algorithm 3) corresponds to a restoration operator that approximates the conditional expectation $E[x|s, H]$. Consequently, the theoretical analysis does not explain the experimental results with a restoration operator trained in an unsupervised manner.

questions:
1. Page 1: “...Tweedie’s formula (Robbins, 1956; Efron, 2011) seemingly implies that Gaussian denoising alone might be sufficient for learning priors,...”: That is indeed true and this work does not disprove this fact. 

2. Page 2: “...ShaRP provides a richer and more flexible representation of image priors…”: At this stage, it is rather vague as to what “richer and more flexible” means. 
3. Page 2: “Unlike Gaussian denoisers, the restoration models in ShaRP can often be directly
trained in a self-supervised manner”: Even Gaussian denoisers can be trained in an unsupervised manner using only noisy images (e.g., using a SURE loss). 

4. It might be good to put some of the math background in the intro, effectively shortening the material in the first three pages. 

5. Page 3: “We introduce a novel regularization concept for inverse problems that encourages solutions that produce degraded versions closely resembling real degraded images.”: Firstly, this would be the case if the degradation is the same for which the restoration network is trained, not the forward operator “A” for which you want to solve the inverse problem. Secondly, can one not promote this property using a simple Gaussian denoiser together with a data-consistency loss? 

6. Theorem 1: “Assume that the prior density $p_x$ is non-degenerate over $\mathbb{R}^n$”: This is almost always not true. 

7. Assumption 3: Define what “$b(x)$” is. 

Requested changes:

1. Provide a comprehensive review of the convergence analysis of biased SGD in terms of the assumptions and results to put your theoretical contributions in perspective. 

2. Clarify whether one can approximate the conditional expectation $E[x|s, H]$ using a restoration operator trained on a self-supervised loss. If not, the interpretation of ShaRP as biased SGD does not hold in this case.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724745132

thread_type:
official_review

<thread_object>
creator: reviewer_n69z

<content>
summary:
This paper extends the work of [Hu et al., 2024c] for the regularization of inverse problems using a deep restoration prior trained on multiple restoration tasks. The authors theoretically analyze the induced objective function and the convergence of the proposed algorithm. Empirical results demonstrate that the framework outperforms Gaussian denoiser priors in image reconstruction tasks, including MRI and super-resolution.

strengths:
1. The experiments show that training a deep restoration network on multiple tasks can improve its regularization capability.

2. Theoretical analysis is also provided.

weaknesses:
1. The paper seems limited in novelty and closely resembles [Hu et al., 2024c]. The main contribution is the training of a restoration prior across the same task with varying levels of ill-posedness. The authors should clearly explain how the proposed methodology differs from [Hu et al., 2024c].

2. The authors claim that the proposed prior trained on a general restoration task outperforms Gaussian denoisers; however, there is a lack of sufficient experiments to support this claim. Gaussian denoisers can be used for solving general inverse problems, but the experiments presented here are confined to the same inverse problem and forward operator on which the prior was trained, although with different levels of ill-posedness. For example, in Section 5.1, the restoration prior is trained on MRI with an x8 acceleration and applied to the same problem with x4 and x6 acceleration factors. Although Algorithm 1 suggests that the proposed prior can be used to solve any other inverse problems, the experiments are limited to the same problem type. To validate the claimed superiority over Gaussian denoisers, the authors are requested to include experiments where the trained prior is applied to a different inverse problem from the one it was originally trained on. For example, training the prior on super-resolution and using it for solving MRI reconstruction. You can also train your prior on multiple distinct tasks as suggested by Algorithm I, for example training the prior on super-resolution and image in-painting tasks, and use it for solving a different task like MRI. This comparison is crucial as Gaussian denoisers can solve any other different inverse problems once trained.

Hu, Yuyang, et al. "A Restoration Network as an Implicit Prior." ICLR 2024.

questions:
Please take a look at the comments under weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1732227953263

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all:

comment:
Thank you all for the feedback. We provide detailed answers to all the comments below. To better address some of them, we ran additional experiments. These results were included in the dedicated sections of the revised supplementary material. We have highlighted the revised parts in the paper and the supplementary material with **orange**.

Below, we address two general questions that highlight the motivation for our work:

**Why go beyond Gaussian denoisers to general restoration priors?**

An important point often overlooked in the literature is that Gaussian-denoiser networks are suboptimal when applied to images degraded by other, non-Gaussian artifacts. In this paper, we derive a closed-form prior that leverages multiple restoration networks that can better deal with non-Gaussian artifacts due to the inverse problem. As shown in the paper, this leads to performance that significantly surpasses methods using Gaussian-denoising networks as priors.

**Why propose a new prior based on restoring from multiple types of degradations?**
The strength of our framework lies in its versatility, enabling seamless integration of a wide-range of restoration models within a unified formulation. Note how our approach achieves substantial performance improvements over methods using Gaussian-denoising priors, as well as the DRP method from Hu et al. (2024c), which relies on a single restoration model (see Tables 1 and 3). Additionally, in principle, our framework remains compatible with Gaussian-denoising networks, as they can be seamlessly incorporated into our prior using $H = I$.
Furthermore, our prior can be trained without fully-sampled measurements: Unlike Gaussian denoisers, ShaRP models can be trained without requiring fully sampled/clean measurement data, making it more versatile (see Table 2).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses deep learning reconstruction for computational imaging. It builds on the work on deep restoration priors by Hu et al. from 2024 by introducing multiple degradation operator in place of one, and using a different optimization algorithm which is an adaptation of SNORE (Renaud et al 2024b). This new combination results in improved numerical results, in particular on subsampled MRI.

Reviewers made positive remarks about the clarity and the numerical results, but there has been a broad agreement  that the novelty in this paper relative to Hu et al. 24 and other prior work is limited and not sufficient for acceptance at ICLR. While the rebuttal phase has been intense, no fundamentally new information emerged from the discussions, while the reviewers made precise arguments to support their position.

The manuscript may benefit from a "full-disclosure" rewrite that proactively addresses the criticism of N69Z, XTiX, and E3Gi. A focus on domain-specific numerics (MRI, medical imaging in general) and practical impact may further strengthen the paper. The reviewers provided precise, detailed remarks that should help guide these revisions effectively.

While the leading-order criticisms were about novelty,  part of the dialogue between the authors and the reviewers revolved around the meaning of "prior" and its relation to denoisers via Tweedie's formula. A prior distribution doesn't depend on a specific observation process. Priors learned by Gaussian denoisers (via Tweedie) and "priors" that result from restoration networks which, as the authors explain, are tuned to a particular problem, are different objects. The authors are justified in stating that "In practice, the effectiveness of a denoiser is tied to how well the noise model it's trained on matches the actual noise in the image". But this is a statement about structured denoising more than about priors. A bona fide prior and a known forward model allow to compute any estimator, at least in principle. They can be used to generate a dataset and do any kind of ML. The question is how to build effective algorithms that exploit this (with finite samples) and at this point one may obtain improved results by schemes such as the one proposed by the authors. This distinction might benefit from further clarification in the revised manuscript.

For these reasons I am recommending a rejection. Nonetheless, I hope that the authors will find the reviewers' feedback and the discussions valuable as they work on refining their paper.