RestoreGrad: Signal Restoration Using Conditional Denoising Diffusion Models with Jointly Learned Prior

Authors: Ching-Hua Lee, Chouchang Yang, Jaejin Cho, Yashas Malur Saidutta, Rakshith Sharma Srinivasa, Yilin Shen, Hongxia Jin
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Denoising diffusion probabilistic model, prior distribution, posterior, speech enhancement, image restoration

Abstract:
Denoising diffusion probabilistic models (DDPMs) estimate the data distribution by sequentially denoising samples drawn from a prior distribution, which is typically assumed to be the standard Gaussian for simplicity. Owing to their capabilities of generating high-fidelity samples, DDPMs can be utilized for signal restoration tasks in recovering a clean signal from its degraded observation(s), by conditioning the model on the degraded signal. The degraded signals are themselves contaminated versions of the clean signals; due to this correlation, they may encompass certain useful information about the target clean data distribution. However, naively adopting the standard Gaussian as the prior distribution in turn discards such information. In this paper, we propose to improve conditional DDPMs for signal restoration applications by leveraging a more informative prior that is jointly learned with the diffusion model. The proposed framework, called RestoreGrad, exploits the correlation between the degraded and clean signals to construct a better prior for restoration tasks. In contrast to existing DDPMs that just settle on using pre-defined or handcrafted priors, RestoreGrad learns the prior jointly with the diffusion model. To this end, we first derive a new objective function from a modified evidence lower bound (ELBO) of the data log-likelihood, to incorporate the prior learning process into conditional DDPMs. Then, we suggest a corresponding joint learning paradigm for optimizing the new ELBO. Notably, RestoreGrad requires minimum modifications to the diffusion model itself; thus, it can be flexibly implemented on top of various conditional DDPM-based signal restoration models. On speech and image restoration tasks, we show that RestoreGrad demonstrates faster convergence (5-10 times fewer training steps) to achieve on par or better perceptual quality of restored signals over existing DDPM baselines, along with improved robustness to using fewer sampling steps in inference time (2-2.5 times fewer steps), advocating the advantages of leveraging jointly learned prior for efficiency improvements in the diffusion process.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729442962286

thread_type:
official_review

<thread_object>
creator: reviewer_vrxe

<content>
summary:
The paper proposes RestoreGrad, a method that aims to enhance conditional restoration DDPM models. The main idea is to add to the training 2 auxiliary networks that will train jointly with the original DDPM model and allow the DDPM to use at inference non-gaussian prior that is dependent on the degraded input. RestoreGrad is tested on image restoration and speech enhancement and shows faster convergence at training as well as robustness to inference with fewer steps. The quality of the results is either on-par or better for the reported metrics.

strengths:
-	RestoreGrad converges faster with the same performances relative to the baselines, and the auxiliary nets do not add many parameters (only 2%).
-	The experiments contain comparisons against many SotA algorithms.
-	The paper is clear and well written. 
-	The robustness to sampling using fewer timesteps is important and works well.

weaknesses:
-	**Novelty:** A core claim of novelty is the idea of using a learned, non-gaussian prior. However, this idea is not new. For example, PriorGrad already discussed in the past this exact same idea (however not achieving better performances). The authors mention this and previous non-learened non-gaussian priors as different conceptually or technically (L88). However, Grad-TTS also used a learned prior, and successfully. 
This limits the novelty contribution to only adaptation to the image domain.
-	**Evaluations:** 
  -	The paper lacks subjective evaluations via user studies.
  -	In L1057 the authors claim higher PSNR yields better perceptual image quality, which is known to not be the case as it is a distortion metric only (See e.g., Blau & Michaeli 2018 or Freirich et al.2021).
  -	The important LPIPS and FID metrics are only present in the appendix for a single task (RainDropDiff). The authors should move them to the main part of the manuscript and add these metrics for the rest of the domains and models (and specifically in the tables and not only as graphs).
  -	There are no links to qualitative samples for the speech enhancement task.
-	**Minor**: 
  -	In the abstract please clarify that you improve conditional DDPM for signal restoration (and not just general conditional models).
  -	L463 missing the device & train time (the latter is mentioned in later paragraph, but it's a bit unorganized).

questions:
-	Will code be published on acceptance?
-	I would suggest maybe a more perceptual speech enhancement metric such as automatic speech recognition before and after the degradations, e.g. using whisper (see Benita et al. 2024).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730446015850

thread_type:
official_review

<thread_object>
creator: reviewer_rjt2

<content>
summary:
This paper proposes a method to improve the approach of using diffusion models conditioned on degraded data to solve restoration tasks in the audio and image domains. This method is inspired by PriorGrad, which uses a data-dependent adaptive prior, and attempts to extend this approach. Specifically, it simultaneously trains a Prior Net that estimates the parameters of the data-dependent prior. During training, it also simultaneously trains a Posterior Net that estimates the parameters of the data-dependent prior using the true data x_0 as input, along with the main score function. While PriorGrad handcrafts the prior, this method learns it simultaneously. The model is trained with a modified ELBO as the objective function, and experiments show that the proposed method converges 5 to 10 times faster in training compared to the baseline and achieves good performance with 2 to 2.5 times fewer steps.

strengths:
- The experiments in Section 5 demonstrate that RestoreGrad shows notable improvements over baseline models in both speech and image restoration tasks. This method contributes significantly to the field as it has a certain level of generalizability, meaning it can be applied to many techniques with similar frameworks.
- The number of parameters in the Prior Net and Posterior Net is only about 2% of the main diffusion model's total parameters. This means that the additional computational/memory cost is kept to a minimum.
- This method includes hyperparameters \eta and \lambda that balance the weights in the loss function. However, the performance of the model doesn't seem to be very sensitive to the values of these parameters. This is a significant advantage for practitioners who will use this method in real-world applications.

weaknesses:
- While one of the strength of this method lies in its generalizability, the speech-related experiments only use CDiffSE (proposed in 2022) as a baseline. This alone might not be sufficient to fully demonstrate the validity of the method. It seems necessary to address its applicability to other techniques as well.
- In the field of SE, there are clear improvements compared to the baseline. However, looking at Table 3, it appears that the improvements in the image processing domain are not as significant.

questions:
1. In addition to PriorGrad, I understand that SpecGrad is also related to this paper and shares similar objectives, albeit for different tasks. Could you explain the relationship with SpecGrad and discuss the advantages of this method?
2. While this method seems applicable to various techniques, what are its limitations? For instance, does it require the amplitude information of the signal to be significant? If so, is it difficult to apply to latent diffusion model-based methods? This is briefly mentioned in the conclusion, but could you elaborate on this a bit more?
3. Can you provide more insights into the values output by the Prior Net (or the Posterior Net output)? How does the characteristics of the output signal sequence differ from PriorGrad or SpecGrad? I felt it was outputting the amplitude information of the signal to be estimated in the restoration task. If so, is it possible to replace this prior net module with the output of a fixed SE module?
4. Comparing the SE results with the IR results, the improvements in SE seem more pronounced. Can you discuss this difference? Is it possible to discuss this in light of the characteristics of each type of data? (For example, is it important that speech has a zero mean?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676819023

thread_type:
official_review

<thread_object>
creator: reviewer_z1ga

<content>
summary:
This paper focuses on signal restoration tasks and explores diffusion models with learned prior distributions capturing information about corrupted signal $y$. The pipeline the authors propose resembles VAE with the decoder represented by a conditional diffusion model and two encoders. Thus, at inference information about corrupted signal is used not only as conditioning to the network estimating diffusion noise, but also parameterizes the diffusion prior. Three networks are trained jointly, and the posterior encoder (skipped at inference) takes as input both clean signal $x_0$ and its corrputed verison $y$ and helps the prior encoder learn data-dependent task-specific prior. The novel loss function providing upper bound on negative log-likelihood for such combined VAE-diffusion model is proposed. Experiments on speech enhancement and image restoration tasks show the benefits of the proposed approach compared to conditional diffusion models with data-independent priors or the ones based on heuristics when applicable.

strengths:
1. The authors introduced a novel model design combining ideas from VAE and diffusion modeling: the proposed model consists of two encoders parameterising prior distribution and a conditional diffusion based decoder.
2. A novel ELBO-like loss was proposed allowing for stable training of the mentioned model. The learned priors look quite reasonable. Thus, this approach can become a good alternative to hand-crafted diffusion priors in signal restoration tasks.
3. The experiments clearly demonstrate that diffusion models can benefit from the priors learned in the way the authors propose. The number of training steps necessary to achieve the same performance level decreases compared to data-independent or hand-crafted priors. Moreover, RestoreGrad achieves better performance at convergence.
4. The presentation quality is good (except for one point, see below); the text is clear and the paper is easy to follow.

Overall, I think that this is a good paper from the point of view of originality and clarity.

weaknesses:
I can point out three main weaknesses.

1. As I far as I understand, the role of the posterior net is to better capture the correlation between $x_0$ and $y$ and to pass this knowledge to the prior net whose only input is the corrupted signal $y$ which is achieved by prior matching terms in Formula (9). But what happens if we get rid of this posterior encoder? It is possible to think of the following design: both at training and inference we have the same scheme, the prior net $\psi$ is conditioned only on $y$ and parameterizes diffusion prior, and the conditional diffusion $\theta$ restores a signal from this prior. This model wouldn't differ from the one the authors propose at inference. As for training loss, we obviously would not have prior matching terms anymore, and  $\Sigma_{post}$ would be replaced with $\Sigma_{prior}$ in the latent regularization and denoising matching terms. Such a design looks more simple (only one encoder instead of two) and also allows for learnable prior. I think that it could be a good baseline in the experimental section and it could help to support the idea that the posterior encoder is helpful since it is trained on both $y$ and $x_0$.

2. Non-diffusion baselines in the experimental section seem to be a bit outdated. E.g., in speech enhancement best methods have started to achieve > 3.0 PESQ on a common SE benchmark (Valentini dataset) since the year 2022 [1] outperforming CDiffuSE and RestoreGrad, and you just mention quite old methods with PESQ < 2.5. Although I don't have experience with image deblurring, I also suspect that the baselines you choose for this task are far from SOTA because the papers you refer to were published no later than in 2020.

3. What is $\sigma_t$ in Formula (9)? On the one hand, in the text of the paper $\sigma_t$ is mentioned in the context of common diffusion models (e.g. lines 153 or 189) and it suggests that the reverse process $p_{\theta}$ has fixed (unlearnable) covariance matrix $\Sigma_{\theta}=\sigma^2_t I$. In your setting it must mean something different, since the Algorithm 2 in the Appendix suggests that at each reverse diffusion step the noise you add has covariance matrix $\sigma^2_t\Sigma_{prior}$. Anyway, you should define the reverse process $p_{\theta}$ in your setting more clear, like in lines 188-190, especially since you use $\sigma_t$ that seems to have something to do with it in Proposition 1.

Also, I feel there's something wrong with the values of $\gamma_t$ in Proposition 1. In practice you set them all to $1$, so it is not a big issue, but still I'll mention my concerns. On the one hand, in lines 250-251 and 824-825 it is defined as $\beta_t^2/2\sigma_t^2\alpha_t(1-\bar{\alpha_t})$ for $t>1$. On the other hand, in the proof of Proposition 1 we have lines 858-859 implying a different expression $\gamma_{t}=\beta_t/2\alpha_t(1-\bar{\alpha}_{t-1})$. 

[1] Dual-branch attention-in-attention transformer for single-channel speech enhancement, Yu et al.

questions:
1. I have a question regarding the proof in lines 858-860: we see that the expectation of KL divergence between two Gaussians $q(x_{t-1}|x_t,x_0)$ and $p_\theta(x_{t-1}|x_t,y)$ contains only MSE term $\Vert\epsilon-\epsilon_{\theta}\Vert_{\Sigma^{-1}}^2$ and does not contain the log-determinant terms implying that it is KL divergence between two Gaussians with the same covariance matrices $\Sigma_A$ and $\Sigma_B$ (and $\log\frac{\Sigma_A}{\Sigma_B}$ cancels out).  In the assumptions in Proposition 1 you explicitly define how the forward diffusion paths look like. It's easy to see that for such forward diffusions the covariance of $q(x_{t-1}|x_t,x_0)$ is $\tilde{\beta}_{t}\Sigma$. So, is it true that reverse process has this covariance matrix?

2. One of latent regularization terms is multiplied by $\alpha_T$ which should be very little since the mean of the prior should be very close to zero. So, this term should have very little impact in theory. Why do you still use it at training? Does it have any impact on the training in practice?

3. Is the training process sensitive to the choice of loss weights $\lambda$ and $\eta$?

Also, I'd like to make a couple of additional remarks here. These are just the remarks to consider, not questions:

4. Typo in line 043: "DDPMs typically consistS..."

5. It's interesting to see what happens with your model if we consider continuous-time diffusion formalism as in [2]. Continuous-time diffusions also allow for likelihood estimation (but in a different way, without ELBO-like derivation). In my understanding, in this case prior matching terms in the total loss (9) would remain unchanged, denoising matching terms would look similar, but I'm not sure about Latent Regularization terms. Would they still be there in the loss function? If they would, what whould the exact expression look like, and how would these changes affect model training/final performance?

[2] Score-Based Generative Modeling through Stochastic Differential Equations, Song et al.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733483556

thread_type:
official_review

<thread_object>
creator: reviewer_wuzf

<content>
summary:
This paper proposes RestoreGrad, a framework that enhances signal restoration tasks using conditional denoising diffusion probabilistic models (DDPMs) with a jointly learned prior. The authors introduce a data-dependent prior distribution to improve the efficiency of DDPMs for recovering clean signals from degraded versions. RestoreGrad’s framework incorporates two additional networks, Prior Net and Posterior Net, to capture more relevant information for the reverse diffusion process.

strengths:
1. Adaptive Prior Distribution: RestoreGrad learns a data-dependent prior rather than relying on a fixed Gaussian, potentially improving efficiency by tailoring the prior to the degraded signal. 

2. Joint Training with Minimal Modifications: The proposed approach requires only the addition of Prior and Posterior networks and a modified loss function, making it compatible with various DDPM architectures without significant changes. 

3. Efficiency in Training and Inference: RestoreGrad exhibits faster convergence and robustness to fewer sampling steps during inference, making it computationally efficient compared to baseline DDPMs.

weaknesses:
1.  Lack of Novelty in Methodology: The method heavily builds on existing techniques, including the use of conditional DDPMs, adaptive priors (as seen in PriorGrad), and standard VAE and DDPM loss formulations. The Prior and Posterior networks adapt concepts from encoder-decoder architectures, similar to those in VAEs, rather than presenting new model designs. 

2. Learned Prior Concept is not novel: The idea of using an adaptive, learned prior is inspired by existing models, like PriorGrad [1]. PriorGrad introduced the concept of data-dependent priors by using the degraded signal to inform the prior distribution, so RestoreGrad's approach of a learned prior based on the degraded input builds directly on this idea rather than introducing a fundamentally new approach  
3. Limited Novelty in Loss Function Design: The modified ELBO loss primarily integrates known VAE and DDPM loss terms without introducing novel loss structures or constraints tailored to signal restoration. 

4. No New Restoration-Specific Techniques: The Prior and Posterior networks are based on standard architectures (e.g., ResNet), and there is no exploration of restoration-specific methods, such as task-specific losses or architectures, that might further enhance the model's performance in different restoration tasks. 

5. Missing Computational Time Comparison: The paper lacks a clear and detailed computational time comparison with baseline models, which is critical for understanding the added cost of the Prior and Posterior Nets. Including this comparison in training and inference steps, along with performance metrics, would provide a clearer assessment of RestoreGrad's computational demands. 

6. Missing FID details in Tables 3, 4, and 7 - While the paper reports PSNR and SSIM metrics for assessing image quality, it omits the FID (Fréchet Inception Distance) metric, a widely used measure for evaluating the realism and distributional similarity of generated images. Including FID would give a more comprehensive understanding of RestoreGrad's performance in terms of perceptual quality. 

7. No Discussion on Trade-offs: The method introduces additional computational layers and complexity, yet there's little discussion on the trade-offs between quality improvement and increased model size or computation. 

[1]. Lee, S. G., Kim, H., Shin, C., Tan, X., Liu, C., Meng, Q., ... & Liu, T. Y. (2021). Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. arXiv preprint arXiv:2106.06406.

questions:
How the authors address the above weakness will be important for re-evaluating the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732269085912

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We thank all the reviewers for the insightful comments that have helped us improve our submission substantially. We were able to conduct additional experiments to address reviewers’ feedback and concerns, further improve the manuscript clarity, and justify the contributions and innovation of RestoreGrad. Here we provide an overview of our responses. Detailed responses can be found in the reviewers' individual comments. We have also updated the manuscript accordingly, highlighting the changes in blue for ease of review.

---

## Paper summary

- We study improvements to conditional denoising diffusion probabilistic models (DDPMs) for signal restoration tasks, where clean signals are recovered from their degraded observations by utilizing the generative modeling capabilities of diffusion models.

- To address inefficiencies due to the discrepancy between the actual data and assumed prior distributions used in existing DDPMs, we propose a new framework called RestoreGrad that jointly learns the prior distribution with the diffusion model through a *prior encoder* for improved model efficiency, based on exploiting the correlation between the targe clean signal and input degraded signal encoded by an auxiliary *posterior encoder*. The framework is formally derived from a novel integration of the VAE and DDPM.

- We show that the new, two-encoder DDPM learning framework leveraging jointly learned priors leads to faster convergence, better restored signal quality, and robustness to using fewer sampling steps during inference over existing DDPM baselines on speech and image restoration tasks.

---

## Strengths as per the reviewers

- Reviewers WuZF, RjT2 appreciated the generalizability and effectiveness of the proposed work. Specifically, they highlighted the significance of the proposed idea being applicable to many techniques of similar frameworks, without requiring complicated modifications to the diffusion models and loss functions.

- Reviewers VRxE, RjT2 acknowledged the parameter efficiency of our framework, as RestoreGrad achieves notable improvements while introducing only minimum additions of parameters over baseline conditional DDPMs in signal restoration tasks.

- Reviewer Z1ga appreciated the idea of combining techniques from VAEs and DDPMs via a novel ELBO-based loss function to arrive at the two-encoder design for jointly learning the prior. Reviewer VRxE acknowledged the robustness to sampling using fewer timesteps as an important feature of RestoreGrad. Reviewer RjT2 highlighted the performance of RestoreGrad not being sensitive to the values of introduced hyper-parameters as a significant advantage for real-world applications.

---

## Major questions from the reviewers

While we respond to the individual queries in reviewer specific rebuttals, here we list some of the important ones and our responses.

> **Distinction with existing approaches**

Reviewers WuzF and VRxE had some questions on the novelty of the learned prior concept and distinction with existing works on adaptive and learned priors (e.g., PriorGrad and Grad-TTS). We summarize our main novel points here. 

- Our framework adopts a **two-encoder** design, i.e., prior and posterior encoders, for **jointly** learning the diffusion prior distribution by incorporating posterior information based on the correlation between the target clean signal and degraded observation(s). Such a network design of using prior and posterior encoders for DDPMs has not been seen in the literature to our knowledge.

- Our learning framework is established based on **a novel loss function** formally derived from integrating the ELBOs of VAEs and DDPMs to enjoy the advantages of both methodologies. The derivation and resulting loss function are both new.

- Our proposed method is general and can be adopted by DDPM-based signal restoration models for several data modalities including images and audio. The **generality** of our approach further distinguishes it from prior works that focused on just speech.

> **Completeness of experiments**

The reviewers have several questions on missing detailed information of the experiments and also suggestions on additional results. We have revised our manuscript accordingly to provide more complete evidence. Below we summarize main additional experiments conducted **in the revised manuscript**.
- Reviewer Z1ga suggested an interesting ablation study to observe the benefits brought by including the posterior information coming from the target signal $\mathbf{x}\_0$ and conditioning signal $\mathbf{y}$ through the Posterior Net. We presented the new results in Section 5.1.2, **Table 3**.  

- Reviewer VRxE suggested a more perceptual SE metric using ASR. We added the results using open-source ASR models in Appendix C.1, **Table 10**.

- Reviewer WuzF asked about the trade-offs between quality and complexity. We added related discussion by showing the performance versus encoder model size in Appendix C.1, **Table 11**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732872750722

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors before End of Discussion Period (1/2)

comment:
We thank the AC for their efforts so far in facilitating reviews and their efforts in the future during the rest of the conference. We would also like to thank all the reviewers again for their time and effort on providing valuable feedback during the discussion period by far. We are glad that by addressing the reviewers' concerns and questions, our paper has apparently improved. It was especially encouraging to see that **Reviewers VRxE** and **RjT2** have increased their scores to reflect more positive recognition of our work, and **Reviewer Z1ga** stays with their initial positive view. On the other hand, *it was unfortunate that we have not heard back from **Reviewer WuzF** so far* after we responded to their initial review comments, who had concerns about the novelty of RestoreGrad. We therefore would like to take this chance to further highlight our novel aspects and distinctions from the most similar prior works, from both the Algorithm and Application perspectives. We hope this helps you better appreciate the originality and significance of our work.

## Originality on the Algorithm Side

### **1. New Evidence Lower Bound (ELBO) on Conditional Data Log-likelihood for Generative Modeling**

Let us compare the ELBOs used in different generative modeling frameworks:

- Conditional VAEs (e.g., Refs [R1, R2]):
$$
\\log p(\\mathbf{x}\_0|\\mathbf{y})\\geq\\mathbb{E}\_{q\_\\phi(\\boldsymbol{\\epsilon}|\\mathbf{x}\_0,\\mathbf{y})}\\left[\\log p\_\\theta(\\mathbf{x}\_0|\\mathbf{y},\\boldsymbol{\\epsilon})\\right] - D\_{\\text{KL}}\\left(q\_\\phi(\\boldsymbol{\\epsilon}|\\mathbf{x}\_0,\\mathbf{y}) || p\_\\theta(\\boldsymbol{\\epsilon}|\\mathbf{y})\\right).
$$

- Conditional DDPMs (e.g., Refs [R3, R4, R5]):
$$
\\log p(\\mathbf{x}\_0|\\mathbf{y})\\geq\\mathbb{E}\_{q(\\mathbf{x}\_{1:T}|\\mathbf{x}\_0)}\\left[\\log\\frac{p\_\\theta({\\mathbf{x}\_{0:T}}|\\mathbf{y})}{q(\\mathbf{x}\_{1:T}|\\mathbf{x}\_0)}\\right].
$$

- RestoreGrad (ours):
$$
\\log p(\\mathbf{x}\_0|\\mathbf{y})\\boldsymbol{\\geq}\\mathbb{E}\_{q_\\phi(\\boldsymbol{\\epsilon}|\\mathbf{x}\_0,\\mathbf{y})}\\Biggr[\\mathbb{E}\_{q(\\mathbf{x}\_{1:T}|\\mathbf{x}\_0)}\\left[\\log\\frac{p\_{\\theta}(\\mathbf{x}\_{0:T}|\\mathbf{y},\\boldsymbol{\\epsilon})}{q(\\mathbf{x}\_{1:T}|\\mathbf{x}\_0)}\\right]\\Biggr] - D\_{\\text{KL}}\\bigr(q\_{\\phi}(\\boldsymbol{\\epsilon}|\\mathbf{x}\_0,\\mathbf{y}) || p\_{\\psi}(\\boldsymbol{\\epsilon}|\\mathbf{y})\\bigr).
$$

We can see that our method presents *a new form of ELBO that seamlessly integrates the DDPM into the VAE framework.* The motivation behind the combination is to embrace the best of both worlds, i.e., promising generative ability (DDPM) and modeling efficiency (VAE). Consequently, RestoreGrad achieves improved output signal quality and training/sampling efficiency *simultaneously*.

### **2. New Loss Function for Conditional DDPM Training**

The closest work to ours is PriorGrad (Ref [R6]) who also improved conditional DDPMs with data-dependent priors. Motivated by their adaptive prior concept, our framework further generalizes to *a jointly learnable prior framework by employing a prior encoder and a posterior encoder, $\psi$ and $\phi$.* To be more specific, we can compare the loss functions of ours with PriorGrad:
- PriorGrad (Ref [R6]):
$$
\\min\_{\\theta} \\,\\,\\, ||\\boldsymbol{\\epsilon}-\\boldsymbol{\\epsilon}\_\\theta(\\mathbf{x}\_t,\\mathbf{y},t)||^2\_{\\boldsymbol{\\Sigma}^{-1}\_{y}}, \\,\\,\\, \\text{where} \\,\\,\\, \\boldsymbol{\\Sigma}\_{y}=f(\\mathbf{y}) \\, \\text{and} \\, f(\cdot) \\, \\text{is a pre-defined function that maps} \\, \\mathbf{y} \\, \\text{into the prior distribution.}
$$

-  RestoreGrad (ours):
$$
\\min_{\\theta,\\phi,\\psi} \\,\\,\\, \\eta\\bigr(\\bar{\\alpha}\_T||\\mathbf{x}\_0||^2\_{\\boldsymbol{\\Sigma}^{-1}\_{\\text{post}}}+\\log|\\boldsymbol{\\Sigma}\_{\\text{post}}|\\bigr)+||\\boldsymbol{\\epsilon}-\\boldsymbol{\\epsilon}\_\\theta(\\mathbf{x}\_t,\\mathbf{y},t)||^2\_{\\boldsymbol{\\Sigma}^{-1}\_{\\text{post}}}+\\lambda \\bigr(\\log\\frac{|\\boldsymbol{\\Sigma}\_{\\text{prior}}|}{|\\boldsymbol{\\Sigma}\_{\\text{post}}|}+\\text{tr}(\\boldsymbol{\\Sigma}\_{\\text{prior}}^{-1}\\boldsymbol{\\Sigma}\_{\\text{post}})\\bigr), \\,\\,\\, \\text{where} \\,\\,\\, \\boldsymbol{\\Sigma}\_{\\text{prior}}\\coloneqq\\boldsymbol{\\Sigma}\_{\\text{prior}}(\\mathbf{y};\\psi) \\,\\,\\, \\text{and} \\,\\,\\,
\\boldsymbol{\\Sigma}\_{\\text{post}}\\coloneqq\\boldsymbol{\\Sigma}\_{\\text{post}}(\\mathbf{x}\_0,\\mathbf{y};\\phi).
$$

We can see that our framework is more *generalized* as it takes care of the prior estimation by utilizing two encoders $\psi$ and $\phi$ to jointly learn the distribution with the DDPM $\theta$, therefore by-passing the requirement to search for a suitable mapping function $f(\cdot)$ usually having to be based on the domain knowledge of a given task (e.g., spectral analysis in speech tasks). Our framework is thus more general.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a framework called RestoreGrad to deal with the signal restoration tasks using conditional diffusion models with a jointly learned prior. The key idea of RestoreGrad is to integrate VAE and DDPM to better capture the correlation between clean and degraded signals.

Strengths:

1. The integration of VAE and DDPM into a single  framework is interesting.  

2. Compared to baseline models,  RestoreGrad demonstrates faster convergence and improvement with fewer sampling steps.  


Weaknesses:
1. The proposed framework is similar to PriorGrad which makes the technical contribution marginal.

2. The comparisons is insufficient, which include some old baselines for image restoration tasks.

3.  More discussion is needed on computational complexity versus performance trade-offs. 

4.  The writing is not very clear, e.g., the role of the posterior encoder and the derivation of the ELBO.

While this is an interesting paper, based on the above weaknesses raised by reviewers,  I have to recommend rejection in its current form but I sincerely encourage  the authors to revise it for future submission.