Blind Inversion using Latent Diffusion Priors

Authors: WeiminBai, Siyi Chen, Wenzheng Chen, He Sun
Primary area: generative models
Keywords: Computational Imaging, Diffusion Model, Inverse Problem

Abstract:
Diffusion models have emerged as powerful tools for solving inverse problems due to their exceptional ability to model complex prior distributions. However, existing methods predominantly assume known forward operators (i.e., non-blind), limiting their applicability in practical settings where acquiring such operators is costly. Additionally, many current approaches rely on pixel-space diffusion models, leaving the potential of more powerful latent diffusion models (LDMs) underexplored. In this paper, we introduce LatentDEM, an innovative technique that addresses more challenging blind inverse problems using latent diffusion priors. At the core of our method is solving blind inverse problems within an iterative Expectation-Maximization (EM) framework: (1) the E-step recovers clean images from corrupted observations using LDM priors and a known forward model, and (2) the M-step estimates the forward operator based on the recovered images. Additionally, we propose two novel optimization techniques tailored for LDM priors and EM frameworks, yielding more accurate and efficient blind inversion results. As a general framework, LatentDEM supports both linear and non-linear inverse problems. Beyond common 2D image restoration tasks, it enables new capabilities in non-linear 3D inverse rendering problems. We validate LatentDEM's performance on representative 2D blind deblurring and 3D sparse-view reconstruction tasks, demonstrating its superior efficacy over prior arts.

Decision: reject
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729507365311

thread_type:
official_review

<thread_object>
creator: reviewer_rjxb

<content>
summary:
The paper presents a new diffusion-based model for blind inverse problems, which alternates between a diffusion step with a fixed parameter (eg a fixed blur kernel), and a step estimating the kernel given an estimate of the image. The paper motivates this algorithm as an instance of the expectation-maximization (EM) framework. In particular, the approach leverages powerful latent diffusion models such as Stable Diffusion v1.5, and performs the bulk of the diffusion in the latent space. 

The method is demonstrated in blind image deblurring and sparse view 3D reconstruction, obtaining a better performance in terms of both distortion and perceptual quality than the competing baselines.

strengths:
- introduces an approach that leverages pretrained encoder/decoders which reduces the computational burden of working in the pixel space.
- good experimental results in blind deblurring and sparse view 3D reconstruction.

weaknesses:
- I believe the link to the EM framework is not sufficiently well founded: the MAP kernel estimation step uses \hat{x_0} instead of a sample from the posterior p(x|y, \phi). It is not clear to me what is the role of q(x|y) in equation 5, and more generally the equation does not add much to the explanation in my opinion.
- The experimental setup is not sufficiently well described in the main text, i) raises doubts about testing on a training set (see related questions below) and ii) the papers seems to use circular padding which is not a very realistic setting.
- The presentation of the paper could be improved:
    - the citation style is incorrect in many parts of the paper (use citep{} for references that are not part of a sentence)
    - the mathematical notation is not consistent: equation 11 uses Z uppercase for denoting a vector, whereas the rest uses lowercase, the letter 'z' is used for both latent space vectors and auxiliary variables in 11, the letter D is used for a denoiser and a decoder. What is the  'F^2' operator in equation 11? Acronyms are defined multiple times throughout the text (eg. EM). The method Zero123 is not followed by a reference when introduced on page 7.

questions:
- What is the denoiser used for estimating the blur kernels? how was that denoiser trained? 
- Is there an overlap between training kernels and test kernels? How were the test kernels generated/chosen? 
- Was the Stable Diffusion model was trained on the FFHQ and ImageNet images used for testing the models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730236988474

thread_type:
official_review

<thread_object>
creator: reviewer_yuwp

<content>
summary:
This paper introduces LatentDEM, a novel method for addressing blind inverse problems through latent diffusion models. The proposed method leverages an Expectation-Maximization (EM) framework: in the expectation step, a clean image is recovered using a latent diffusion prior, and in the maximization step, the forward model is estimated based on the recovered clean image. The authors demonstrate the method’s application to blind deblurring and 3D pose-free sparse-view reconstruction.

strengths:
The paper is well-organized and clearly communicates its results. 

The authors provide extensive experimental results, comparing their method to prominent existing methods for blind inverse problems, such as BlindDPS and FastEM. 

They provide an ablation study examining the influence of gradient estimation skipping and annealing consistency on performance.

weaknesses:
1. A primary concern is the paper’s original contribution. The methodology appears to combine elements previously explored in the literature. For instance, latent diffusion models for inverse problems have been investigated in [1], while Expectation-Maximization approaches for blind inverse problems have been studied in [2]. Simply substituting a latent diffusion model as the prior does not, in itself, seem to constitute a novel contribution. Explicitly listing the contributions could improve clarity.

2. Another concern is the runtime of the proposed method. As seen in Table 2, the method appears to trade time for performance, with comparable results to BlindDPS but with similar computational demands. In other words, we would not gain any performance advantage compared to BlindDPS, if are limited by the runtime of the method. 

3. The reported performance metrics for FastEM and BlindDPS appear inconsistent with previous results. For instance, in the FastEM paper, performance on the FFHQ dataset is as follows: MPRNet: 19.52, BlindDPS: 24.05, and FastEM: 25.75, demonstrating comparability to DPS given the GT kernel. However, in this paper, the results are MPRNet: 21.60, BlindDPS: 22.58, and FastEM: 17.46. Could the authors provide an explanation for this performance discrepancy? Is the comparison of FastEM accurate? Can they reproduce FastEM’s results by strictly following its settings for a more direct comparison?

[1] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.

[2] Charles Laroche, Andrés Almansa, and Eva Coupete. Fast diffusion em: a diffusion model for blind inverse problems with application to deconvolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5271–5281, 2024.

questions:
Have the authors observed any instances of non-converging patterns in their algorithm? Specifically, does a particular choice of annealing or tunable parameters (Equations 11 or 12) lead to divergence?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730325194343

thread_type:
official_review

<thread_object>
creator: reviewer_e53d

<content>
summary:
The paper proposed a method for solving blind restoration problems (examined mostly on blind deblurring) using a pretrained latent diffusion model (LDM). It is based on adding to the iterations of unconditional LDM sampling steps (which are sometimes skipped) where the the observation/forward model parameters (e.g., blur kernel) are updated and being used for guiding the diffusion (I do not see a clear relation to expectation maximization). The proposed method is quite heuristic and shares similar ideas with existing works that use guided LDM for non-blind restoration and works that use guided pixel-space DMs for blind restoration. In the reported experiments it outperforms alternative methods.

strengths:
- As far as I know, this is the first work that uses a pretrained LDM for blind restoration.

- The proposed method appears to have empirical advantages over competitors.

weaknesses:
- The contribution seems incremental in terms of the techniques being used and the lack of theoretical motivation and analysis of the proposed method. 

- The presentation is not good in terms of explaining why the authors refer to the proposed method as an expectation maximization (EM) technique.

questions:
- The presentation of EM in Section 3 differs from the plain EM formulation (as in Dempster et al., 1977 and textbooks).
For example, what are the unobserved/missing data? Where in the E-step do you compute (conditional) expectation of the log likelihood/posterior with respect to the missing data?
The authors may refer to some variational Bayes modification of EM but not to the actual EM. Make the discussion more clear and point to the reference that justifies your formulation.

- Consequently, the presentation of the method in Section 4 needs to be improved. The derivation of the method does not match the EM paradigm. It looks like some alternating optimization scheme that lacks formal connection to EM.

- Lack of theoretical analysis for the proposed modifications to the baseline alternating optimization approach.  
For "Technique 1" there is some motivation in the appendix, but no formal mathematical argument that actually considers the effect of error in phi. The other modifications are quite heuristic.  

- Regarding the experiments, you need to provide more details on the kernels, noise levels, etc., that you use in training and in testing.  
Also, state in some table the hyperparameters that you use. Do you use the same hyperparameters for all the settings and for both 2D and 3D?

-  The reported results of the competitors for blind deblurring on FFHQ are lower than those reported in previous works  (Chung et al., 2022a), (Laroche et al., 2024) (e.g., worse kernel estimation for BlindDPS and huge deterioration for FastEM).  
How can you explain it? Do you use the same setup as these works? If not, then explain why.

- Please compare also with GibbsDDRM:  
Murata, Naoki, et al. "Gibbsddrm: A partially collapsed gibbs sampler for solving blind inverse problems with denoising diffusion restoration." ICML, 2023.

- Present also the results for the non-blind methods on which BlindDPS and FastEM, and your approach, are based. They can be interpreted as the upper bound on the results, as done in GibbsDDRM and FastEM.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730731802601

thread_type:
official_review

<thread_object>
creator: reviewer_6zdj

<content>
summary:
This paper aims to tackle the challenging problem of blind inverse tasks by employing latent diffusion priors within an expectation-maximization (EM) framework, which iteratively estimates both the unknown measurement operator (within M-step) and the underlying image (within E-step). The proposed approach has been validated for tasks of 2D blind deburring and 3D pose-free sparse-view reconstruction. The parameters of the forward model are estimated in the M-step through using a pre-trained denoiser of Gaussian noise within a PnP framework. In the E-step, a so called "annealing consistency" approach is proposed through introducing a tunable hyper-parameter in the score of likelihood term to stabilize the training. In addition to the score of likelihood term,  a "gluing" regularization (Rout et al, 2024) is employed within diffusion posterior sampling for 2D deblurring task.

strengths:
This problem holds broad interest across the machine learning and signal processing communities.
The paper is generally well-structured and somewhat clear.

weaknesses:
Several technical ambiguities require clarification. These include the choice of likelihood formulation, the role of “gluing” regularization versus annealing consistency, and the performance discrepancies across datasets. More experimental validation, particularly isolating the effects of different regularization techniques, would strengthen the claims. Additionally, a detailed comparison of hyperparameters with competing methods would improve transparency and reproducibility. For further information of the detected ambiguities please refer to the section "Questions".

questions:
the following issues and ambiguities require further clarification or validation:

1. While (Chung et al. 2022b) formulates the score of the data likelihood as $\nabla_{z_t} \log p_t(y|z_t)$ in Eq.(14), this paper opts for Eq.(9), citing “annealing consistency” for enhanced stability and performance. The authors should provide empirical evidence or theoretical justification for this choice. How does “annealing consistency” compare to the “vanilla” latent DPS term? Also, clarify the behavior of Eq.(9) and Eq.(18) when measurements are noiseless, i.e.  $\sigma = 0$.

2. For the 2D blind deblurring task, Eq.(16) includes a “gluing” regularization term (in addition to “vanilla” latent DPS-based term) as per (Rout et al. 2024). It is unclear whether the reported improvements are due to this regularization or the proposed “annealing consistency.” An experiment isolating these factors in 2D deblurring would be helpful.

3. The paper should explain why BlindDPS performs much better on FFHQ than ImageNet, and detail the training setup. Additionally, comparing FID metrics between this work, FastEM (Rombach et al. 2022), and BlindDPS (Chung et al. 2022a) would improve clarity [as used in the aforementioned papers as well]. Why does FastEM show negligible improvement over initial observations (e.g., see the estimated blurring kernel which is close to Dirac delta)?

4. Clarify the parameter $\phi$ for widely studied image inverse tasks (e.g., compressed sensing, tomography, super-resolution, inpainting, spectral imaging) to enhance understanding of its role across applications.

5. The paper aims to tackle [general] ‘blind’ inverse problems through estimating the parameters of forward measurement operator in the M-step of the EM framework as eq.(10).  To this aim, the parameter $\phi$ (e.g., blurring kernel for the task of 2D deblurring) is estimated through a PnP denoising step as eq.(19). Please elucidate on whether a denoiser pre-trained on Gaussian noise can generalize to other inverse tasks like compressed sensing, and explain what signal regularities it might exploit in estimating $\phi$.

6. A comprehensive list of trainable and tunable hyperparameters for the proposed method, along with those used in Fast Diffuse EM (Laroche et al. 2024) and BlindDPS (Chung et al. 2022a), would facilitate fair comparisons.

7. Please specify the network used in the E-step and provide experimental setup details for Table 1.

Minor Comments:

- In the context “these priors cannot capture the complex natural image distributions, limiting the solvers’ ability to produce high-quality reconstructions”, please consider replacing the (Candes & Romberg, 2007) citation with the following, which better aligns with the stated limitations of image priors:
https://doi.org/10.1109/TIP.2005.863057
https://doi.org/10.1109/ICIP.2007.4379013
https://doi.org/10.1109/GlobalSIP.2013.6737048
https://doi.org/10.1137/16M1102884

- Please clarify initialization of $\mathcal{A}_{\phi^{(t)}}$ in Algorithm 1 and specify corresponding M-step equations (perhaps Appendix C.2 ?)

- Differences between Eq.(11) of this paper and Eq.(24)/Eq.(B.9) in (Laroche et al. 2024) require clarification.
    
- Please correct discrepancies in Section 4.3 regarding the noiseless assumption in Eq.(1).

- Minor typographical and grammatical errors include:
  * Typo corrections: “a intermediate” and “an one-to-many.”
  * Redundant citations: “Liu et al. (2023b)” and “Liu et al. (2023c)” refer to the same work.
  * Misuse of “While” and “However” following Eq.(4).
  * Replace “introduce” with “utilize” or “employ” when referring to “gluing” regularization as it is not part of the contribution of this work.
  * The paper overlooks the following relevant work:
        Murata et al., “Gibbsddrm: A partially collapsed Gibbs sampler for solving blind inverse problems with denoising diffusion restoration.” ICML, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732213142135

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for their constructive and detailed feedback. Below, we summarize the major concerns and our responses. Point-by-point replies have been provided to each reviewer, and the manuscript has been updated accordingly.

---

1. **Contribution and Novelty**

    1.  Pre-trained LDMs for Blind Inverse Problems: 

        This work is the first to use pre-trained latent diffusion models (LDMs) for challenging blind inverse problems. The integration of LDMs and EM framework is non-trivial because of two major issues: 

        (a) Instability: The encoder-decoder structure of LDMs, combined with unknown model errors, makes diffusion posterior sampling fragile.

        (b) limited efficiency: iterative gradient backpropagation through the decoder leads to substantial computational costs. 

        To address these issues, we introduced **annealing consistency** and **skip-gradient techniques** that effectively stabilize and accelerate the inference process, enabling our method to outperform existing approaches, such as FastEM and BlindDPS.

    2. New Applications:

        We demonstrate a novel application of our method in **pose-free sparse-view 3D reconstruction**, a high-dimensional, non-linear inverse problem more challenging than blind deblurring. This success further highlights the robustness and versatility of our algorithm, providing a new benchmark for testing.

---

2. **Mathematical Formulation of the EM Framework**

Our approach is firmly grounded in the EM algorithm, not an alternating optimization scheme. In our method:

1) The **E-step** involves posterior sampling from the diffusion prior.

2) The **M-step** maximizes the expected log data fidelity to estimate model parameters.

The only approximation lies in using Tweedie’s formula in the E-step to accelerate computation, which does not alter the EM framework's fundamental nature. While the EM derivation in the original paper is derived from a variational Bayes perspective, we have provided additional mathematical derivations following the classic algorithm, and added more explanations and discussions in our detailed responses to each reviewer.

---

3. **Fair Comparison with Baselines**

The differences in reconstruction scores between our work and prior methods primarily arise from variations in kernel sizes and datasets: The original FastEM paper tested only on small kernels, and the original BlindDPS paper evaluated only on FFHQ and AFFHQ datasets. In contrast, our experiments cover a broader range of kernel sizes, complexities, and datasets (e.g., ImageNet). Additional experiments, included in the updated manuscript and reviewer responses, confirm that our comparisons are **fair, accurate, and more comprehensive** than previous work.

---

We hope these clarifications and additional results address the reviewers' concerns. We are happy to discuss any remaining points during the discussion phase. Thank you again for your insightful feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733156719999

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
### **Dear Reviewers, ACs, and PCs,**

We sincerely thank all the reviewers for their valuable and constructive feedback, as well as for dedicating their time to reviewing our paper. Based on the insightful suggestions provided during the rebuttal phase, we have conducted a thorough revision addressing the reviewers' key concerns. Below, we summarize these concerns and detail the revisions and updates included in the final submission. We hope this provides the reviewers, ACs, and PCs with a clearer understanding of the progress and outcomes of the rebuttal discussion.

---

### **Concerns**

+ **[6ZdJ, E53D, rjXb]** Expectation-Maximization algorithm and the derivation of our method.
  + **[Authors]**  We explain in the response. *[Response to 6ZdJ, E53D, rjXb] [General Response: Mathematical Formulation]*
  + **[Authors]** Additional clarifications and step-by-step derivation . *[L205-L223] [Appendix G]*
+ **[6ZdJ, E53D, yUwp]** Comparison with baselines.
  + **[Authors]**  We explain in the response. *[Response to 6ZdJ, E53D, yUwp] [General Response: Fair Comparison with Baselines]*
  + **[Authors]** Additional experiments. *[L1409-1426, Figure 15]*
+ **[6ZdJ, E53D, rjXb]** Detailed experimental settings.
  + **[Authors]** We explain in the response. *[Response to 6ZdJ, E53D, rjXb]*
+ **[E53D, yUwpn]** Contribution and novelty.
  + **[Authors]** We explain in the response. *[Response to E53D, yUwpQ] [General Response: Contribution and Novelty]*
+ **[6ZdJ, E53D]** Missing reference of GibbsDDRM.
  + **[Authors]** We explain in the response. *[Response to 6ZdJ, E53D]*
  + **[Authors]** We add discussion and experiments. *[L1438-L1452, Figure 16]*
+ **[6ZdJ]** Role of annealing consistency technique.
  + **[Authors]**  We explain in the response. *[Response to 6ZdJ]*
  + **[Authors]** We conduct additional isolating experiments to demonstrate the proposed technique. *[L1350-L1377, Figure 14]*

---

### **Revisions**

+ **[Preliminary]** We restate the derivation of the Expectation-Maximum algorithm to make it precise, to address concerns raised by Reviewer 6ZdJ, E53D and rjXb. *[L 205-233, Preliminary]*
+ **[Method]** We rewrite the Half-Quadratic Splitting (HQS) optimization according to Reviewer rjXb's concern. *[L332-L337, Section 4.3]*
+ **[Appendix F]** We add all additional experimental results here for clarification. *[L1382-1493, Appendix F]*
+ **[Appendix G]** We derive the Expectation-Maximum algorithm step-by-step, according to concerns raised by Reviewer 6ZdJ, E53D and rjXb. *[L1514-1565, Appendix G]*

---

### **Other Points**

+ **[6ZdJ]** Replace “introduce” with “employ”. *[L 853, Appendix C.1]*
+ **[6ZdJ]** Typos. *[Corrected]*
+ **[rjXb, 6ZdJ]** Citations miused. *[Corrected]*

---

### **Summary**

We sincerely thank the reviewers for their valuable suggestions, which have helped us strengthen our revised submission. We believe the revised submission could be much more robust and contribute to broadening the field of diffusion-based inverse imaging, offering valuable insights to the community. Notably, our approach not only excels at 2D blind deblurring, but also achieves superior performance on the challenging pose-free multi-view consistent 3D reconstruction. We deeply appreciate the efforts of the reviewers, ACs, and PCs during the rebuttal period.

*Best regards,*

*Authors*
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes LatentDEM, a novel method for addressing blind inverse problems through latent diffusion models within an Expectation-Maximization (EM) framework. The authors claim that their method outperforms existing methods for 2D blind deblurring and enables new capabilities in non-linear 3D inverse rendering problems, such as pose-free sparse-view 3D reconstruction.  

Strengths:
- The paper is generally well-written and easy to follow.  
- It proposes a novel method for addressing blind inverse problems using latent diffusion models.  
- The method is shown to be effective for both 2D and 3D inverse problems.  

Weaknesses:
- Some reviewers have raised concerns about the clarity and accuracy of the EM formulation.  
- There are ambiguities regarding the likelihood formulation, the role of "gluing" regularization versus annealing consistency, and the handling of performance discrepancies across datasets.  
- The novelty of the "annealing consistency" technique is questioned.  
- More experimental validation is needed, particularly isolating the effects of different regularization techniques and providing a detailed hyperparameter comparison for reproducibility.  

The decision to reject this paper is based on the following reasons:
- The concerns about the EM formulation are not fully addressed. This hinders the understanding of the proposed formulation.
- The technical ambiguities and the lack of novelty of the "annealing consistency" technique raise concerns about the overall soundness of the method.
- The experimental validation is not sufficient to support the claims.

This paper presents interesting ideas but their potential impact remains unclear due to the current presentation.  A significant revision is necessary to address the reviewers' concerns and fully articulate the paper's contributions. In particular, the following aspects could help make the work clearer and better:

- Clearly respond to concerns regarding the E and M steps.
- Discuss the drawbacks of the proposed methods transparently.
- Clarify that annealing consistency is an existing technique, not a novelty of the paper.
- Provide a comparison between annealing and gluing performances.
- Review, cite, and compare overlooked related works.
- Detail the experimental setup and configurations for all methods.
- Elaborate on the poor performance of comparative methods.