Blind Inversion using Latent Diffusion Priors

## Abstract

Abstract Diffusion models have emerged as powerful tools for solving inverse problems due to their exceptional ability to model complex prior distributions. However, existing methods predominantly assume known forward operators ( i.e . , non-blind), limiting their applicability in practical settings where acquiring such operators is costly. Additionally, many current approaches rely on pixel-space diffusion models, leaving the potential of more powerful latent diffusion models (LDMs) underexplored.
In this paper, we introduce LatentDEM, an innovative technique that addresses more challenging blind inverse problems using latent diffusion priors. At the core of our method is solving blind inverse problems within an iterative Expectation-Maximization (EM) framework: (1) the E-step recovers clean images from corrupted observations using LDM priors and a known forward model, and (2) the M-step estimates the forward operator based on the recovered images. Additionally, we propose two novel optimization techniques tailored for LDM priors and EM frameworks, yielding more accurate and efficient blind inversion results.
As a general framework, LatentDEM supports both linear and non-linear inverse problems. Beyond common 2D image restoration tasks, it enables new capabilities in non-linear 3D inverse rendering problems. We validate LatentDEM’s performance on representative 2D blind deblurring and 3D sparse-view reconstruction tasks, demonstrating its superior efficacy over prior arts.

## 1 Introduction

Inverse problems aim to recover underlying signals $\boldsymbol{x}$ from partial or corrupted observations $\boldsymbol{y}$ generated by a forward operator $\mathcal{A}_{\phi}(\cdot)$. Such problems are prevalent in computer vision and graphics, encompassing a variety of tasks ranging from 2D image restoration(denoising, deblurring, and inpainting ; ) to 3D reconstruction(CT, NLOS, inverse rendering ; ; ), *etc*.
Typically, inverse problem solvers assume the forward model $\mathcal{A}$ and its physical parameters ${\phi}$ are known (*i.e*., non-blind) . However, acquiring accurate forward models is often challenging or impractical in real-world settings. This necessitates solving blind inverse problems, where both the hidden signals $\boldsymbol{x}$ and the forward model parameters ${\phi}$ must be jointly estimated.

Being heavily ill-posed, inverse problems largely rely on data priors in their computation.
Traditional supervised learning approaches train an end-to-end neural network to map observations directly to hidden images ($\boldsymbol{y}\to\boldsymbol{x}$) ; ; .
Recently, diffusion models (DMs) ; ; have emerged as powerful inverse problem solvers due to their exceptional ability to model the complex data distribution $p(\boldsymbol{x})$ of underlying signals ${\boldsymbol{x}}$.
DMs approximate $p(\boldsymbol{x})$ by learning the distribution’s score function $\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{x}_{t}})$ , allowing data-driven priors to be integrated into Bayesian inverse problem solvers (*e.g*., diffusion posterior sampling ). Later, latent diffusion models (LDMs) have evolved as a new foundational model standard  by projecting signals into a lower-dimensional latent space $\boldsymbol{z}$ and performing diffusion there. This strategy mitigates the curse of dimensionality typical in pixel-space DMs and demonstrates superior capability, flexibility, and efficiency in modeling complex, high-dimensional distributions, such as those of videos, audio, and 3D objects ; ; ; .

Although both DM-based and LDM-based solvers have demonstrated impressive posterior sampling performance in diverse computational imaging inverse problems, existing methods predominantly focus on non-blind settings (*i.e*., optimizing images $\boldsymbol{x}$ with known forward model parameters ${\phi}$) ; ; .
Blind inversion poses more challenges since jointly solving $\boldsymbol{x}$ and ${\phi}$ involves non-convex optimization, often leading to instabilities.
While recent advances have explored the feasibility of solving blind inverse problems using pixel-based DMs ; , these methods suffer from computational inefficiencies and limited capability in modeling complex image priors, rendering them unsuitable for more challenging, high-dimensional blind inversion tasks like 3D inverse rendering.

In this paper, we introduce LatentDEM, a novel approach that solves blind inverse problems using powerful LDM priors.
The core concept of LatentDEM involves a variational EM framework that alternates between reconstructing underlying images $\boldsymbol{x}$ through latent diffusion posterior sampling (E-step) and estimating forward model parameters ${\phi}$ using the reconstructed images (M-step). We further design an annealing optimization strategy to enhance the stability of the vulnerable latent space optimization,
as well as a skip-gradient method to accelerate the training process. Consequently, LatentDEM allows us to leverage the capabilities of pre-trained foundational diffusion models to effectively solve a wide range of blind 2D and 3D inverse problems.

To the best of our knowledge, LatentDEM is the first method that incorporates powerful LDM priors  in the blind inverse problems. We first validate our method with Stable Diffusion  priors and perform the representative 2D blind motion deblurring task, where we showcase superior imaging quality and efficiency over prior arts.
LatentDEM further demonstrates new capabilities in more challenging non-linear 3D inverse rendering problems. Given a set of unposed sparse-view input images, we apply Zero123 priors  to synthesize the corresponding novel view images, supporting pose-free, sparse-view 3D reconstruction. Our results exhibit more 3D view consistency and achieve new state-of-the-art novel view synthesis performance.

Figure: Figure 1: We apply our method on two representative blind inverse problems: 2D Blind Deblurring and Pose-free Spare-view 3D Reconsturction. Notably, in 2D task, our method achieves more accurate image recovery and kernel estimation over BlindDPS , while in 3D task we successfully reconstruct consistent novel view images from unposed input views.

## 2 Related Work

Inverse Problems.    The goal of general inverse problems is to recover signals $\boldsymbol{x}\in\mathbb{R}^{D}$ from partial observations $\boldsymbol{y}\in\mathbb{R}^{M}$:

$$ $\boldsymbol{y}=\mathcal{A}_{\phi}\left(\boldsymbol{x}\right)+\boldsymbol{n},$ (1) $$

where $\mathcal{A}$, ${\phi}$ and $\boldsymbol{n}\sim\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})$ represent the forward operator, its parameters, and the observation noise, respectively.
The signal $\boldsymbol{x}$ can be either solved by supervised learning approaches ; ; , or recovered within the Bayesian framework to maximize the posterior: $p(\boldsymbol{x}|\boldsymbol{y})\propto p(\boldsymbol{x})p(\boldsymbol{y}|
\boldsymbol{x})$, where data priors $p(\boldsymbol{x})$ are of vital importance.
Traditional methods use handcrafted priors such as sparsity or total variation (TV) ; .
However, these priors cannot capture the complex natural image distributions, limiting the solvers’ ability to produce high-quality reconstructions ; ; .
Recent advances in diffusion models (DMs), particularly latent diffusion models (LDMs), have made them attractive for inverse problems due to their powerful data prior modeling capabilities ; ; .
In this paper, we focus on solving *blind* inverse problems using latent diffusion models ; .

Diffusion Models for 2D Inverse Problems.    DMs have been applied to a wide range of 2D inverse problems, including natural image deblurring, denoising, and super-resolution tasks ; ; ; ; , as well as medical and astronomy image enhancement ; ; ; . Diffusion Posterior Sampling (DPS) pioneered the use of DMs as strong data priors to solve non-blind 2D inverse problems in a maximum-a-posteriori (MAP) manner ; ; . Later works ; evolved DPS with Latent Diffusion Model (LDM) priors, demonstrating improved performance due to better priors. While these methods ; ; all address non-blind problems, BlindDPS  extends DPS to the blind setting by modeling diffusion priors of both data and forward model parameters. Similar to our approach, FastEM  proposes to address blind inversion within an Expectation-Maximization (EM) framework. However,  ; remain limited to pixel-based DMs, as the instability of LDMs makes the optimization even harder. In this paper, we investigate how to integrate more powerful LDM priors with EM frameworks in blind inversion tasks and demonstrate new state-of-the-art results.

Diffusion Models for 3D Inverse Problems.    3D reconstruction from 2D images, also known as inverse graphics, has long been a significant goal in the fields of vision and graphics ; ; .
Recently, diffusion models are also largely involved in tackling this problem ; ; ; ; .
In this context, the underlying signals $\boldsymbol{x}$ and the observation $\boldsymbol{y}$ represent 3D data and 2D images, while $\mathcal{A}$ denotes the forward rendering process and ${\phi}$ are the camera parameters.
Although the most straightforward way
is to directly model 3D distributions ; , this way is not feasible due to the scarcity of 3D data ; . Alternatively, recent works focus on utilizing 2D diffusion priors to recover 3D scenes with SDS loss  but suffer from view inconsistency issues ; ; ; .

To mitigate this problem, a branch of work fine-tunes Latent Diffusion Models (LDMs) with multi-view images, transforming LDMs into conditional renderers ; ; . Given an image and its camera parameter, they predict the corresponding novel views of the same 3D object. In other words, these models can also be utilized to provide 3D data priors.
However, existing methods typically operate in a feed-forward fashion, still leading accumulated inconsistency during novel view synthesis and requiring further correction designs ; ; . In contrast, LatentDEM treats the sparse-view 3D reconstruction  task as a blind inverse problem. Given sparse-view input images without knowing their poses, we apply Zero123  priors to jointly optimize their relative camera parameters and synthesize new views. Our method utilizes information of all input views  and produces significantly better view-consistent objects compared to feed-forward baselines ; .

## 3 Preliminary

Diffusion Models and Latent Diffusion Models.    DMs ; ; model data distribution by learning the time-dependent score function $\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{x}_{t}})$ with a parameterized neural networks $\boldsymbol{s}_{\theta}$.
In the forward step, it progressively injects noise into data through a forward-time SDE; while in the inverse step, it generates data from noise through a reverse-time SDE :

$$ $\begin{split}\text{Forward-time SDE:}\quad&\mathrm{d}{\boldsymbol{x}}=-\frac{ \beta_{t}}{2}\boldsymbol{x}\mathrm{d}{t}+\sqrt{\beta_{t}}\mathrm{d}{ \boldsymbol{w}},\\ \text{Reverse-time SDE:}\quad&\mathrm{d}{\boldsymbol{x}}=\left[-\frac{\beta_{t }}{2}\boldsymbol{x}-\beta_{t}\nabla_{\boldsymbol{x}_{t}}\log p_{t}({ \boldsymbol{x}_{t}})\right]\mathrm{d}{t}+\sqrt{\beta_{t}}\mathrm{d}{\overline{ \boldsymbol{w}}},\end{split}$ (2) $$

where $\beta_{t}\in(0,1)$ is the noise schedule, $t\in[0,T]$, $\boldsymbol{w}$ and $\overline{\boldsymbol{w}}$ are the standard Wiener process running forward and backward in time, respectively.
This equation is also called variance-preserving SDE (VP-SDE) that equals DDPM .
Through this paper, we define $\alpha_{t}:=1-\beta_{t},\bar{\alpha}_{t}:=\prod_{i=1}^{t}\alpha_{i}$ following , as adopted in Algorithm 1, 2.

A significant drawback of pixel-based DMs is that they require substantial computational resources and a large volume of training data.
To reduce the computation overhead, a generalized family of Latent Diffusion Models (LDMs) are proposed ; .
LDMs embed data into a compressed latent space through $\boldsymbol{z}=\mathcal{E}(\boldsymbol{x})$ for efficiency and flexibility, and decode the latent code $\boldsymbol{z}$ back to the pixel space through $\boldsymbol{x}=\mathcal{D}(\boldsymbol{z})$, where $\mathcal{E}:\mathbb{R}^{D}\to\mathbb{R}^{N}$ and $\mathcal{D}:\mathbb{R}^{N}\to\mathbb{R}^{D}$ are the encoder and decoder, respectively. LDMs fuel state-of-the-art foundation models such as Stable Diffusion , which can serve as a powerful cross-domain prior.
The versatility of LDMs makes them promising solvers for inverse problems.
However, such an efficient paradigm is a double-edged sword, as LDMs are notorious for their instability due to the vulnerability of latent space ; .

Figure: Figure 2: Overview of LatentDEM. Top: One EM iteration. Given currently estimated data and kernel, in the E-step, we draw new samples with LDM priors with the proposed *annealing technique*. In the M-step we apply maximum-a-posterior (MAP) to update forward parameters. Middle: Evolution of the optimized signals and forward parameters.
Refer to caption: x1.png

Diffusion Models for Inverse Problems.    A common approach to apply DM priors in non-blind inverse problems is to replace unconditional score function $\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{x}_{t}})$ with conditional score function $\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{x}_{t}|\boldsymbol{y}})$ and apply posterior sampling ; . With the Bayesian rules, we have

$$ $\displaystyle\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{x}_{t}| \boldsymbol{y}})$ $\displaystyle=\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{x}_{t}})+ \nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{y}|\boldsymbol{x}_{t}})$ (3) $\displaystyle\approx\boldsymbol{s}_{\theta}(\boldsymbol{x}_{t},t)+\log p( \boldsymbol{y}|\hat{\boldsymbol{x}}_{0}(\boldsymbol{x}_{t}))$ (4) $$

While $\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{x}_{t}})$(Eq. 3 middle) can be approximated by diffusion models $\boldsymbol{s}_{\theta}(\boldsymbol{x}_{t},t)$(Eq. 4 left). However, $\nabla_{\boldsymbol{x}_{t}}\log p_{t}({\boldsymbol{y}|\boldsymbol{x}_{t}})$(Eq. 3 right) is not tractable as the likelihood $p_{t}(\boldsymbol{y}|\boldsymbol{x}_{t})$ is not known when $t\neq 0$.
Following DPS , we also assume $p_{t}(\boldsymbol{y}|\boldsymbol{x}_{t})=\int_{\boldsymbol{x}_{0}}p(
\boldsymbol{y}|\boldsymbol{x}_{0})p(\boldsymbol{x}_{0}|\boldsymbol{x}_{t})
\mathrm{d}{\boldsymbol{x}_{0}}\approx p(\boldsymbol{y}|\hat{\boldsymbol{x}}_{0
}(\boldsymbol{x}_{t}))$, where $\hat{\boldsymbol{x}}_{0}(\boldsymbol{x}_{t})=\mathbb{E}\left[\boldsymbol{x}_{0
}|\boldsymbol{x}_{t}\right]$, which is computational efficient and yields reasonable results. Moreover, we apply the same trick for the latent space sampling as well .

Expectation-Maximum Algorithm.    The Expectation-Maximization (EM) algorithm ; is an iterative optimization method used to estimate the parameters $\phi$ of the statistical models that involve underlying variables $\boldsymbol{x}$ given the observations $\boldsymbol{y}$. When the true values of the underlying variables are unknown, directly applying maximum likelihood estimation (MLE) to recover $\phi$ becomes infeasible.
Instead, the EM algorithm minimizes the Kullback-Leibler (KL) divergence with respect to the model parameters $\phi$ and a variational distribution $q(\boldsymbol{x}|\boldsymbol{y})$,

$$ $\displaystyle D_{KL}(q(\boldsymbol{x}|\boldsymbol{y})\parallel p_{\phi}( \boldsymbol{x}|\boldsymbol{y}))$ $\displaystyle=\int q(\boldsymbol{x}|\boldsymbol{y})\log\frac{q(\boldsymbol{x}| \boldsymbol{y})p(\boldsymbol{y})}{p_{\phi}(\boldsymbol{y}|\boldsymbol{x})p( \boldsymbol{x})}d\boldsymbol{x}$ (5) $\displaystyle=\mathbb{E}_{q(\boldsymbol{x}|\boldsymbol{y})}\left[\log q( \boldsymbol{x}|\boldsymbol{y})+\log p(\boldsymbol{y})-\log p_{\phi}( \boldsymbol{y}|\boldsymbol{x})-\log p(\boldsymbol{x})\right],$ $$

where $p_{\phi}(\boldsymbol{x}|\boldsymbol{y})$ denotes the true posterior distribution. To solve this optimization problem, the EM algorithm iterates between optimizing $q(\boldsymbol{x}|\boldsymbol{y})$ and $\phi$, known as the Expectation step (E-step) and the Maximization step (M-step), respectively.
In the E-step, $q(\boldsymbol{x}|\boldsymbol{y})$ is optimized by sampling underlying variables from $p_{\phi}(\boldsymbol{x}|\boldsymbol{y})$ assuming known model parameters $\phi$. In the M-step, $\phi$ is optimized using the posterior samples obtained from the E-step. Through this iterative approach, the EM algorithm converges towards a local maximum of the observed data log-likelihood, making it a versatile tool for estimation problems with underlying variables.

## 4 Method

We now describe how to solve *blind* inverse problems using latent diffusion priors. Our method is formulated within a variational Expectation-Maximization (EM) framework, where the signals and forward operators are iteratively optimized through EM iterations.
In the E-step (Sec.4.1), we leverage latent diffusion priors to draw signal posterior samples, where we introduce an annealing technique to stabilize the optimization process.
In the M-step (Sec.4.2), we estimate forward operators in a maximum-a-posteriori (MAP) manner, and adopt a skip-gradient method to improve the efficiency.
In Sec.4.3, we show how our framework can be applied to solve representative problems such as 2D blind deblurring and 3D pose-free sparse-view reconstruction.

### 4.1 E-step: Posterior Sampling via Latent Diffusion

The goal of LatentDEM’s E-step is to solve for the posterior distribution $p_{\phi}(\boldsymbol{x}|\boldsymbol{y})$ by leveraging the reverse time SDE in Eq. 2.
To utilize the latent priors, inspired by PSLD , we conduct posterior sampling in the latent space by defining a conditional latent diffusion process:

$$ $\displaystyle\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}| \boldsymbol{y}})$ $\displaystyle=\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}})+ \nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{y}|\boldsymbol{z}_{t}})$ (6) $\displaystyle\approx\boldsymbol{s}_{\theta}^{*}{(\boldsymbol{z}_{t},t)}+\nabla _{\boldsymbol{z}_{t}}\log p_{\phi}({\boldsymbol{y}|\mathcal{D}^{*}\left( \mathbb{E}[{\boldsymbol{z}}_{0}|\boldsymbol{z}_{t}]\right)})$ (7) $\displaystyle=\boldsymbol{s}_{\theta}^{*}{(\boldsymbol{z}_{t},t)}-\frac{1}{2 \sigma^{2}}\nabla_{\boldsymbol{z}_{t}}\left\|\boldsymbol{y}-\mathcal{A}_{\phi} \left(\mathcal{D}^{*}\left(\mathbb{E}[{\boldsymbol{z}}_{0}|\boldsymbol{z}_{t}] \right)\right)\right\|_{2}^{2}$ (8) $$

where $\boldsymbol{s}_{\theta}^{*}{(\boldsymbol{z}_{t},t)}$ is the pre-trained LDM that approximates the latent score function $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}})$, $\mathcal{A}_{\phi}$ is the parameterized forward model, $\sigma$ is the standard deviation of the additive observation noise, $\mathcal{D}^{*}$ is the pre-trained latent decoder, and $\mathbb{E}[{\boldsymbol{z}}_{0}|\boldsymbol{z}_{t}]$ can be estimated through a reverse time SDE from $\boldsymbol{z}_{t}$ .
However, Eq. (8) works only when $\mathcal{A}_{\phi}$ is known, *i.e*., non-blind settings . In the context of blind inversion, $\mathcal{A}_{\phi}$ is randomly initialized so that significant modeling errors perturb the optimization of $\boldsymbol{z}_{t}$ in the latent space. Consequently, there are significant artifacts when directly applying Eq. 8. We have to introduce an annealing technique to stabilize the training process:

###### Technique 1 (Annealing consistency)

Suppose that the estimated forward operator $\mathcal{A}_{\phi}$ is optimized from coarse to fine in the iterations, we have that:

$$ $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y}}) \approx\boldsymbol{s}_{\theta}^{*}{(\boldsymbol{z}_{t},t)}-\frac{1}{2\zeta_{t} \sigma^{2}}\nabla_{\boldsymbol{z}_{t}}\left\|\boldsymbol{y}-\mathcal{A}_{\phi} \left(\mathcal{D}^{*}\left(\mathbb{E}[{\boldsymbol{z}}_{0}|\boldsymbol{z}_{t}] \right)\right)\right\|_{2}^{2},$ (9) $$

where $\zeta_{t}$ is a time-dependent factor that decreases over time, *e.g*., it anneals linearly from 10 at $t=1000$ to 1 at $t=600$ and then holds.

We refer to this scaling technique as Annealing consistency. Intuitively, $\mathcal{A}_{\phi}$ is randomly initialized at the beginning, which cannot provide correct gradient directions. Therefore, we reduce its influence on the evolution of $\boldsymbol{z}_{t}$ with a large factor ($\zeta_{t}=10$). As sampling progresses, $\mathcal{A}_{\phi}$ gradually aligns with the underlying true forward operator. We then anneal the factor ($\zeta_{t}=1$) to enforce data consistency. We find that this annealing technique is critical for blind inversion with latent priors; without it, the optimized signal $\boldsymbol{x}$ consistently exhibits severe artifacts, as shown in Figure 5. Further theoretical explanations can be found in Appendix A.

###### Technique 1 (Annealing consistency)

### 4.2 M-step: Forward Operator Estimation

The goal of LatentDEM’s M-step is to update the forward operator parameters ${\phi}$ with the estimated samples $\hat{\boldsymbol{x}}_{0}$ from the E-step. This can be achieved by solving a maximum-a-posterior (MAP) estimation problem:

$$ ${\phi}^{*}=\arg\max_{\phi}\mathbb{E}_{\hat{\boldsymbol{x}}_{0}}\left[\log p_{ \phi}(\boldsymbol{y}|\hat{\boldsymbol{x}}_{0})+\log p({\phi})\right]=\arg\min_ {\phi}\mathbb{E}_{\hat{\boldsymbol{x}}_{0}}\left[||\boldsymbol{y}-\mathcal{A}_ {\phi}(\hat{\boldsymbol{x}}_{0})||^{2}_{2}+\mathcal{R}(\phi)\right],$ (10) $$

where $p({\phi})$ is the prior distribution of $\phi$, $\hat{\boldsymbol{x}}_{0}=\mathcal{D}^{*}(\hat{\boldsymbol{z}}_{0})=\mathcal{D}
^{*}(\mathbb{E}\left[\boldsymbol{z}_{0}|\boldsymbol{z}_{t}\right])$, and $\mathcal{R}({\phi})$ is an equivalent regularizer. $\mathcal{R}({\phi})$ can include sparsity, patch-based priors, plug-and-play denoisers ; , etc. This MAP estimation problem can be solved using either gradient-based optimization  or neural networks . Compared to BlindDPS , which jointly optimizes
$\boldsymbol{x}$ and ${\phi}$ using two diffusion processes, our method leverages the properties of EM , resulting in faster convergence and better performance.

Different from pixel-space diffusion models, latent diffusion models require encoding and decoding operations that map between latent space and pixel space (*i.e*., $\boldsymbol{x}=\mathcal{D}^{*}(\boldsymbol{z})$ and $\boldsymbol{z}=\mathcal{E}^{*}(\boldsymbol{x})$), which takes primary time consumption.
Therefore, we further design an acceleration method that “skips” these operations to improve the efficiency of LatentDEM.
Specifically, in the whole EM iteration, the E-step comprises two sub-steps: prior-based diffusion ($\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}})$)
and data likelihood-based diffusion ($\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{y}|\boldsymbol{z}_{t}})$).
(See Eq. 6 for the two terms), where the former happens in latent space($\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}})$) while the latter happens in pixel space that requires encoder-decoder operations. Moreover, the M-step also involves the encoder-decoder operations. We propose to skip these operations to accelerate the training process:

###### Technique 2 ( Skip gradient )

In early stages ($t>S_{T}$), performing $K$ times E-step in latent space with $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}})$ only, then perform the whole EM-step in both latent space and image space.

We refer to this new technique as Skip gradient. We find it largely accelerates the training process without hurting the performance for two reasons. First, similar to the annealing case, in the early stages of the diffusion posterior sampling process, the sampled data $\hat{\boldsymbol{x}}_{0}$ and forward parameters ${\phi}$ are far from the true value, making frequent LDM encoding and decoding unnecessary, as they won’t provide useful gradient signals.
Second, though in the skip-gradient steps we rely on latent diffusion ($\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}})$) only which is unconditional, to some extent, the optimization will follow the previous conditional sampling trajectory, still leading meaningful convergence, as is also observed in  .

We typically set $S_{T}=500$ and $K=8$, which means the total skipped number $M=(T-S_{T})(1-1/K)=(1000-500)(1-1/8)\approx 437$ full gradient computation steps.
We show it significantly reduces computation overhead while keeping PSNR values approximate to the non-skip version, as demonstrated in Table 2. Our full algorithm is described in Algorithm 1.

###### Technique 2 ( Skip gradient )

### 4.3 Blinding Inversion Tasks

Figure: Algorithm 1 LatentDEM for Blind Deblurring

Our framework incorporates powerful LDM priors within the EM framework, which enables solving both linear and non-linear inverse problems. We showcase two representative tasks: the 2D blind debluring task, and the high-dimension, non-linear 3D blind inverse rendering problem.

2D Blind Deblurring.    In the blind deblurring task, we aim to jointly estimate the clean image $\boldsymbol{x}$ and the blurring kernel $\phi$ given a blurry observation $\boldsymbol{y}=\mathcal{A}_{\phi}(\boldsymbol{x})=\phi*\boldsymbol{x}$. The LatentDEM approach proceeds as follows:

- •
E-step: Assuming a known blurring kernel $\phi$, sample the latent code $\boldsymbol{z}_{t}$ and the corresponding image $\hat{\boldsymbol{x}}_{0}^{(t)}=\mathcal{D}(\hat{\boldsymbol{z}}_{0}(
\boldsymbol{z}_{t}))$ based on Eq. 8. To enhance training stability, we adopt the "gluing" regularization to address the non-injective nature of the latent-to-pixel space mapping. More discussions about this regularization are shown in Appendix B.
- •
M-step: Estimate blur kernels using Half-Quadratic Splitting (HQS) optimization ; :
$\mathbf{Z}^{(t-1)}=\mathcal{F}^{-1}(\frac{\mathcal{F}(\boldsymbol{y})\mathcal{
F}(\hat{\boldsymbol{x}}_{0}^{(t)})+\delta\sigma^{2}\mathcal{F}({\phi}^{(t)})}{
\mathcal{F}^{2}(\hat{\boldsymbol{x}}_{0}^{(t)})+\delta\sigma^{2}}),\ \ \ \ \ {
\phi}^{(t-1)}=\mathbf{D}_{\sqrt{\lambda/\delta}}(\mathbf{Z}^{(t-1)}),$
(11)
where $\mathbf{Z}$ is a intermediate variable, $\mathbf{D}$ is a Plug-and-Play (PnP) neural denoiser ; , $\mathcal{F}$ and $\mathcal{F}^{-1}$ are forward and inverse Fourier transforms, $\sigma$ defines the noise level of measurements, and $\lambda$, $\delta$ are tunable hyperparameters . The superscripts ^(t-1) and ^(t) index diffusion steps. More details on implementation are provided in Appendix B.

Pose-free Sparse-view 3D Reconstruction.    We also demonstrate for the first time that LDM-based blind inversion can be applied to sparse-view, unposed 3D reconstruction, a challenging task that jointly reconstructs the 3D object and camera parameters. Zero123, a conditional LDM, is utilized to approximate the 3D diffusion prior in our task. Given an input image $\boldsymbol{y}$ and camera parameters $\phi=(R,T)$ at a target view, Zero123 generates a novel-view image $\hat{\boldsymbol{x}}_{0}^{(t)}=\mathcal{D}(\hat{\boldsymbol{z}}_{0}),\hat{
\boldsymbol{z}}_{0}=\mathbb{E}[{\boldsymbol{z}}_{0}|\boldsymbol{z}_{t}]$ through a conditional latent diffusion process $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y},\phi})$. However, current Zero123 is limited to view synthesis and 3D generation from a single image.

By integrating Zero123 into LatentDEM, we can reconstruct a view-consistent 3D object from multiple unposed images. Without loss of generality, we illustrate this with two images $\boldsymbol{y}_{1}$ and $\boldsymbol{y}_{2}$ without knowing their relative pose. The LatentDEM approach becomes:

- •
E-step: Assuming known camera parameters $\phi_{1}$ and $\phi_{2}$, aggregate information through a joint latent diffusion process $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y}_{1},
\phi_{1},\boldsymbol{y}_{2},\phi_{2}})$ to create view-consistent latent codes $\boldsymbol{z}_{t}$ and synthesized image $\hat{\boldsymbol{x}}_{0}^{(t)}$.
- •
M-step: Estimate camera parameters based on $\hat{\boldsymbol{x}}_{0}^{(t)}$ by aligning unposed images to synthetic and reference views via gradient-based optimization:
$\phi_{2}^{(t-1)}=\phi_{2}^{(t)}-\lambda\nabla_{\phi_{2}^{(t)}}\|\boldsymbol{z}
_{t}(\boldsymbol{y}_{2},\phi_{2}^{(t)})-\boldsymbol{z}_{t}(\hat{\boldsymbol{x}
}_{0}^{(t)},\mathbf{0})\|_{2}^{2}-\delta\nabla_{\phi_{2}^{(t)}}\|\boldsymbol{z
}_{t}(\boldsymbol{y}_{2},\phi_{2}^{(t)})-\boldsymbol{z}_{t}(\boldsymbol{y}_{1}
,\phi_{1})\|_{2}^{2},$
(12)
where $\boldsymbol{z}_{t}(\cdot,\cdot)$ represents the time-dependent latent features of an image after specified camera transformation, $\mathbf{0}$ indicates no transformation, and $\lambda,\delta$ are tunable hyperparameters. Note only $\phi_{2}$ is optimized, as $\phi_{1}$ defines the target view.

Through the synthesis of multiple random novel views from input images and subsequent volumetric rendering, we finally generate a comprehensive 3D representation of the object. This approach extends to arbitrary n unposed images, where n-1 camera poses should be estimated. More views yield better 3D generation/reconstruction performance. It outperforms state-of-the-art pose-free sparse-view 3D baselines  and generates well-aligned images for detailed 3D modeling . Further details, including the derivation of view-consistent diffusion process from traditional Zero123 and 3D reconstruction results from various numbers of images, are provided in Appendix C.

## 5 Experiments

In this section, we first apply our method on the 2D blind deblurring task in Sec. 5.1. We then demonstrate our method on pose-free, sparse-view 3D reconstruction in Sec. 5.2. Lastly, we perform extensive ablation studies to demonstrate the proposed techniques.

Figure: Figure 3: Blind motion deblurring results. Row (1-2): ImageNet. Row (3-4): FFHQ. Our method successfully recovers clean images and accurate blur kernels, consistently outperforming all the baselines, even under challenging cases where the observations are severely degraded.

### 5.1 2D Blind Motion Deblurring

#### Dataset.

We evaluate our method on the images from the widely used ImageNet  and FFHQ . We randomly choose 64 validation images from each dataset, where the resolutions are both $256\times 256$. We chose to use the state-of-the-art Stable Diffusion v-1.5 model  as our cross-domain prior.
For quantitative comparison, we evaluate the image quality with three metrics: peak signal-to-noise-ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS).
Besides, we assess the estimated kernel via mean-squared error (MSE), and maximum of normalized convolution (MNC) ,
All experiments are done on NVIDIA A100 80GB GPUs.

**Table 1: Quantitative evaluation (PSNR, SSIM, LPIPS) of blind deblurring task and (MSE, MNC) of kernel estimation on ImageNet and FFHQ. Bold: Best, under: second best.**
| Method | ImageNet ($\mathbf{256}\times\mathbf{256}$) | FFHQ ($\mathbf{256}\times\mathbf{256}$) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Image | Kernel | Image | Kernel |  |  |  |  |  |  |  |
| PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | MSE $\downarrow$ | MNC $\uparrow$ | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | MSE $\downarrow$ | MNC $\uparrow$ |  |
| MPRNet | 19.85 | 0.433 | 0.470 | - | - | 21.60 | 0.517 | 0.399 | - | - |
| Self-Deblur | 16.74 | 0.232 | 0.493 | 0.016 | 0.036 | 18.84 | 0.328 | 0.493 | 0.017 | 0.045 |
| BlindDPS | 17.31 | 0.472 | 0.309 | 0.036 | 0.274 | 22.58 | 0.583 | 0.245 | 0.048 | 0.270 |
| FastEM | 17.36 | 0.422 | 0.377 | 0.440 | 0.266 | 17.46 | 0.554 | 0.169 | 0.035 | 0.399 |
| Ours | 19.35 | 0.496 | 0.256 | 0.010 | 0.441 | 22.65 | 0.653 | 0.167 | 0.009 | 0.459 |

#### Results.

We provide motion deblurring results in Figure 3 and Table 1.
Our method is compared with two state-of-the-art methods that directly apply pixel-space diffusion models for blind deblurring: BlindDPS  and FastEM , and three widely applied methods: MPRNet , DeblurGAN V2 , and Self-Deblur .
Several interesting observations can be found here.
First, LatentDEM outperforms all the baselines qualitatively. As shown in Fig. 3, in challenging cases with severe motion blur and aggressive image degradation, previous methods are unable to accurately estimate the kernel, while the proposed method enables accurate kernel estimation and high-quality image restoration.
We attribute this to the fact that the powerful LDM priors provide better guidance than pixel-space DM priors in the posterior sampling, together with the deliberately designed EM optimization policies.
Moreover, as shown in Table 1, LatentDEM also achieves the best scores in most metric evaluations, especially for kernel estimation, which demonstrates the efficacy of our EM framework. Interestingly, MPRNet shows higher PSNR in ImageNet dataset but visually it produces smooth and blurry results, which indicates the quality of deblurring cannot be well-reflected by PSNR. Nevertheless, we still largely outperform it in SSIM and LPIPS metrics.

### 5.2 Experiments on Pose-free Sparse-view 3D Reconstruction

#### Dataset.

We evaluate the pose-free sparse-view 3D reconstruction performance on Objaverse dataset , which contains millions of high-quality 3D objects. We pick up 20 models and for each model, we randomly render two views without knowing their poses.
Our goal is to synthesize novel views and reconstruct the underlying 3D model from the unposed sparse-view inputs, which is a very challenging task  and cannot be easily addressed by NeRF  or Gaussian Splatting  that require image poses.

Figure: Figure 4: Pose-free sparse-view 3D reconstruction results. Our method successfully synthesizes consistent novel view images given two sparse input views. In contrast, Zero123 produces images missing the engine handle that are not consistent with the input views, while LEAP fails to generate photo-realistic images.

Figure: Figure 5: Effectiveness of our annealed consistency technique. Left: blur kernel accuracy curve (green) on 10 examples (std are represented by shadow). It indicates that the kernel is very wrong at the beginning but becomes meaningful when $t<600$, which corresponds to the annealing factor curve (red). Right: we further show that simply applying LDM priors in blind inversion produces images with severe artifacts due to the fragile latent space, while the annealing technique stabilizes the optimization and generates much better results.
Refer to caption: x2.png

#### Results.

We provide novel view synthesis results in Fig 4. Our model is built on top of Zero123  priors. The performance of Zero123 has demonstrated expressive in synthesizing high-quality novel-view images, but it sometimes fails in creating view-consistent results across different views, as shown in the first row of Fig. 4 .
A major reason is that it synthesizes new views from a single input only and cannot capture the whole information of the 3D object, resulting in hallucinated synthesis.
Instead, our method could easily crack this nut by aggregating the information of all input views and embedding them together through hard data consistency . We show we acquire much more consistent novel view images from only two input views.
Moreover, our method doesn’t need to know the poses of images as they can be jointly estimated, which supports the more challenging pose-free, sparse view 3D reconstruction. We compare our method with the current state-of-the-art pose-free 3D reconstruction baseline, LEAP . As shown in the second row of Fig. 4, while LEAP fails to generate photo-realistic new views, our method could leverage the powerful Zero123 prior to overcome texture degradation and geometric misalignment, maintaining fine details like the geometry and the texture of the jacket. We provide more results and analysis in Appendix C.4, C.5, D.

### 5.3 Ablation Studies

#### Annealed Consistency.

A major problem when using LDMs instead of pixel-based DMs is the vulnerable latent space optimization. In the context of blind inverse problems, the inaccurate forward operators at the beginning could make the problem even worse, where the optimal solutions significantly deviate from the true value and contain strong image artifacts, as demonstrated in Fig. 5 right.
To ensure stable optimization, we should set our empirical annealing coefficients($\zeta_{t}$ anneals linearly from 10 at $t=1000$ to 1 at $t=600$ and then holds) based on the forward modeling errors, as shown in Fig. 5.
This technique show stabilizes the optimization process and produces a more accurate estimation(Fig. 5 right). We provide more annealing analysis in Appendix A.

#### Skip Gradient for Acceleration.

We also investigate the influence of the skip-gradient technique, where we compute the full EM step every $K$ steps while in the middle steps we only run the latent space diffusion.
We validate 4 different groups of hyperparameters and compare their running times and imaging quality on 2D blind deblurring task (Table 2).
We find the running time for LatentDEM linearly decreases as the number of accelerated steps $M$ increases. In the accelerated steps, the encoder-decoder inference and gradients are ignored, therefore largely reducing the total optimization time.
Moreover, though we have skipped a lot of computation burden, due to the fact that the gradient descent tends to follow the previous optimization trajectory, it still results in meaningful convergence. In an extreme setting (case f) where we skip 900 gradients, our method still outperforms baselines, as well as achieving the fastest optimization speed.

**Table 2: Effectiveness of our skip gradient technique. We evaluate different skip gradient schemes on the blind deblurring task. Compared to the setting without skipping, skipping half steps or even more steps performs similarly better than baselines while reducing the running time.**
|  | Method | Running | Image | Kernel |
| --- | --- | --- | --- | --- |
|  | Time | PNSR | MSE |  |
| a | FastEM | 1min30sec | 17.46 | 0.048 |
| b | BlindDPS | 1min34sec | 22.58 | 0.035 |
| c | No skipping. ($S_{T}=1000,K=1,M=0$) | 6min33sec | 23.45 | 0.011 |
| d | Skip-grad. ($S_{T}=500,K=8,M=437$) | 4min17sec | 23.44 | 0.011 |
| e | Skip-grad. ($S_{T}=0,K=8,M=875$) | 2min54sec | 23.00 | 0.010 |
| f | Skip-grad. ($S_{T}=0,K=16,M=937$) | 1min20sec | 22.56 | 0.010 |

## 6 Conclusion

In this work, we proposed LatentDEM, a novel method that incorporates powerful latent diffusion priors for solving blind inverse problems.
Our method jointly estimates underlying signals and forward operator parameters inside an EM framework with two deliberately designed optimization techniques, which showcases more accurate and efficient 2D blind debluring performance than prior arts, as well as demonstrates new capabilities in 3D sparse-view reconstruction.
Its limitation includes that in 3D tasks it still relies on LDMs fine-tuned with multi-view images. It is interesting to think about how to combine LatentDEM with SDS loss to directly run 3D inference from purely 2D diffusion models.

## Appendix A Theoretical Explanations for Annealing Consistency

We consider the blind inversion as a time-dependent modeling process and the annealing consistency technique helps address time-dependent modeling errors. Specifically, we express the image formation model as:

$$ $\boldsymbol{y}=\mathcal{A}_{{\phi}^{(t)}}(\hat{\boldsymbol{x}}_{0})+w_{t}+ \boldsymbol{n},\quad w_{t}\sim\mathcal{N}(\mathbf{0},\nu_{t}^{2}\mathbf{I}), \quad\boldsymbol{n}\sim\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I}),$ (13) $$

where ${\phi}^{(t)}$ represents the estimated forward model parameters at step $t$, $\hat{\boldsymbol{x}}_{0}=\mathcal{D}^{*}(\mathbb{E}(\boldsymbol{z}_{0}|
\boldsymbol{z}_{t}))$,
$w_{t}$ is the time-dependent modeling noise which is assumed to follow a Gaussian distribution with a time-dependent standard deviation of $\nu_{t}$, and $\boldsymbol{n}$ is the observation noise with a constant standard deviation of $\sigma$.
Therefore, following  [] we derive below data likelihood term to account for both modeling errors and observation noise during the diffusion posterior sampling:

$$ $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{y}|\boldsymbol{z}_{t}}) \approx-\frac{1}{2(\nu_{t}^{2}+\sigma^{2})}\nabla_{\boldsymbol{x}_{t}}\left\| \boldsymbol{y}-\mathcal{A}_{\phi}\left(\mathcal{D}^{*}\left(\mathbb{E}[{ \boldsymbol{z}}_{0}|\boldsymbol{z}_{t}]\right)\right)\right\|_{2}^{2},$ (14) $$

where $\nu_{t}^{2}$ should gradually decrease from a large value to zero as the estimated model parameters converge to the ground truth.
This is consistent with the proposed technique, where in Eq. 9 $\zeta_{t}$ linearly anneals from a large number to a constant.
As a result, annealing consistency aligns better with the blind inversion problem and brings more stable optimization, as well as superior performance.

## Appendix B Implementation Details of 2D Blind Deblurring

### B.1 E-step

#### Diffusion Posterior Sampling with Gluing Regularization.

In the 2D blind deblurring task, the E-step performs latent diffusion posterior sampling (DPS) to reconstruct the underlying image, assuming a known blur kernel.
The basic latent DPS takes the form as follows:

$$ $\begin{split}\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}| \boldsymbol{y}})&\approx\boldsymbol{s}_{\theta}^{*}{(\boldsymbol{z}_{t},t)}+ \nabla_{\boldsymbol{z}_{t}}p\left(\boldsymbol{y}|\mathcal{D}^{*}\left(\mathbb{ E}\left[\boldsymbol{z}_{0}|\boldsymbol{z}_{t}\right]\right)\right),\\ &=\boldsymbol{s}_{\theta}^{*}{(\boldsymbol{z}_{t},t)}-\frac{1}{2\zeta_{t} \sigma^{2}}\nabla_{\boldsymbol{z}_{t}}\left\|\boldsymbol{y}-\mathcal{A}_{\phi} \left(\mathcal{D}^{*}\left(\mathbb{E}[{\boldsymbol{z}}_{0}|\boldsymbol{z}_{t}] \right)\right)\right\|_{2}^{2},\end{split}$ (15) $$

which simply transform the equation from pixel-based DPS [] to the latent space. However, this basic form always produces severe artifacts or results in reconstructions inconsistent with the measurements [].
A fundamental reason is that the decoder is an one-to-many mapping, where numerous latent codes $\boldsymbol{z}_{0}$ that represent underlying images can match the measurements.
Computing the gradient of the density specified by Eq. 15 could potentially drive $\boldsymbol{z}_{t}$ towards multiple different directions.
To address this, we introduce an additional constraint called “gluing” [] to properly guide the optimization in the latent space:

$$ $\displaystyle\nabla_{\boldsymbol{z}_{t}}\log p\left(\boldsymbol{y}|\boldsymbol {z}_{t}\right)$ $\displaystyle=\underbrace{\nabla_{\boldsymbol{z}_{t}}p\left(\boldsymbol{y}| \mathcal{D}^{*}\left(\mathbb{E}\left[\boldsymbol{z}_{0}|\boldsymbol{z}_{t} \right]\right)\right)}_{\text{DPS vanilla extension }}$ (16) $\displaystyle+\gamma\underbrace{\nabla_{\boldsymbol{z}_{t}}||\mathbb{E}\left[ \boldsymbol{z}_{0}|\boldsymbol{z}_{t}\right]-\mathcal{E}^{*}\left(\mathcal{A}_ {\phi^{(t-1)}}^{T}\boldsymbol{y}+\left(\boldsymbol{I}-\mathcal{A}_{\phi^{(t-1) }}^{T}\mathcal{A}_{\phi^{(t-1)}}\right)\mathcal{D}^{*}\left(\mathbb{E}\left[ \boldsymbol{z}_{0}|\boldsymbol{z}_{t}\right]\right)\right)\|^{2}_{2}}_{\text{` `gluing'' regularization}},$ $$

where $\gamma$ is a tunable hyperparameter.
The gluing objective [] is critical for LatentDEM as it constrains the latent code update following each M-step, ensuring that the denoising update, data fidelity update, and the gluing update point to the same optima. Note that gluing is also involved in the skip-gradient technique, *i.e*., we will also ignore it during the skipped steps.

### B.2 M-step

The M-step solves the MAP estimation of the blur kernel using the posterior samples $\hat{\boldsymbol{x}}_{0}$ from the E-step. This process is expressed by:

$$ ${\phi}^{*}=\arg\min_{\phi}\mathbb{E}_{\hat{\boldsymbol{x}}_{0}}\left[\frac{1}{ 2\sigma^{2}}||\boldsymbol{y}-\mathcal{A}_{\phi}(\hat{\boldsymbol{x}}_{0})||^{2 }_{2}+\mathcal{R}(\phi)\right],$ (17) $$

where $\sigma^{2}$ denotes the noise level of the measurements and $\mathcal{R}$ is the regularizer.
Common choices of the regulation term can be $l_{2}$ or $l_{1}$ regularizations on top of
the physical constraints on the blur kernel (non-negative values that add up to one).
Despite being quite efficient when the blurry image does not have noise, they generally fail to provide high-quality results when the noise level increases [].
Therefore, we decide to leverage a Plug-and-Play (PnP) denoiser, $\mathbf{D}_{\sigma_{d}}$, as the regularizer.
We find that training the denoiser on a dataset of blur kernels with various noise levels ($\sigma_{d}$) can lead to efficient and robust kernel estimation.
Now with this PnP denoiser as the regularizer, we can solve Eq. 17 with the Half-Quadratic Splitting (HQS) optimization scheme:

$$ $\mathbf{Z}_{i+1}=\arg\min_{\mathbf{Z}}\left[\frac{1}{2\sigma^{2}}\|\mathcal{A} _{\mathbf{Z}}(\hat{\boldsymbol{x}}_{0})-\boldsymbol{y}\|_{2}^{2}+\frac{\delta} {2}\|\mathbf{Z}-\phi_{i}\|_{2}^{2}\right],$ (18) $$

$$ $\phi_{i+1}=\mathbf{D}_{\sqrt{\lambda/\delta}}(\mathbf{Z}_{i+1})=\arg\min_{\phi }\left[\lambda\mathcal{R}(\phi)+\frac{\delta}{2}\|\phi-\mathbf{Z}_{i+1}\|_{2}^ {2}\right],$ (19) $$

where $\mathbf{Z}$ is a intermediate variable, $\mathbf{D}_{\sigma_{d}}$ is a PnP neural denoiser [, ], $\sigma$ defines the noise level of measurements, and $\lambda$, $\beta$ are tunable hyperparameters []. The subscripts i and i+1 index iterations of Eq. 18 and Eq. 19 in one M-step.
For the deconvolution problem, Eq 18 can easily be solved in the Fourier domain and Eq 19 corresponds to the regularization step. It corresponds to the MAP estimator of a Gaussian denoising problem on the variable
$\mathbf{Z}_{i+1}$.
The main idea behind the PnP regularization is to replace this regularization step with a pre-trained denoiser.
This substitution can be done becaue of the close relationship between the MAP and the MMSE estimator of a Gaussian denoising problem.
In the end, the M-step can be expressed by Eq 11.

#### Plug-and-Play Denoiser.

We train a Plug-and-Play (PnP) denoiser to serve as the kernel regularizer in the M-step.
For the architecture of the denoiser, we use a simple DnCNN [] with 5 blocks and 32 channels.
In addition to the noisy kernel, we also take the noise level map as an extra channel and feed it to the network to control the denoising intensity.
The settings are similar to one of the baseline methods, FastEM [].
In the data preparation process, we generate 60k motion deblur kernels with random intensity and add random Gaussian noise to them.
The noisy level map is a 2D matrix filled with the variance and is concatenated to the kernel as an additional channel as input to the network.
We train the network for 5,000 epochs by denoising the corrupted kernel and use the MSE loss. All the training is performed on a NVIDIA A100 which lasts for seven hours. We also try different network architectures like FFDNet but find the DnCNN is sufficient to tackle our task and it’s very easy to train.

#### Hyperparameters.

For the motion deblur task, we leverage the codebase of PSLD [], which is based on Stable Diffusion-V1.5.
Besides the hyperparameters of the annealing and skip-gradient technique, we find it critical to choose the proper parameters for the gluing and M-step.
Improper parameters result in strong artifacts.
In our experiments, we find the default hyperparameters in  [] won’t work, potentially due to the more fragile latent space. The hyperparameters in our M-step are $\lambda=1$ and $\delta=5e6$, and We iterate Eq. 11 20 times to balance solution convergence and computational efficiency.

## Appendix C Implementation Details of Pose-free Sparse-view 3D Reconstruction

### C.1 Basics

#### Problem Formulation.

The pose-free, sparse-view 3D reconstruction problem aims to reconstruct a 3D object from multiple unposed images. This task can be formulated as a blind inversion problem with the following forward model:

$$ $\boldsymbol{y}=\mathcal{A}_{\phi}\left(\boldsymbol{x}\right),\boldsymbol{y}=\{ \boldsymbol{y}_{1},\cdots,\boldsymbol{y}_{n}\},\quad{\phi}=\{{\phi}_{1},\cdots ,{\phi}_{n}\},$ (20) $$

where $\boldsymbol{x}$ represents the underlying 3D object, $\boldsymbol{y}$ is the set of observations containing images from multiple views, and $\phi$ denotes the camera parameters corresponding to different views. The task requires us to jointly solve for both the 3D model, $\boldsymbol{x}$, and the image poses, $\phi$, to reconstruct a view-consistent 3D object. 3D models can be represented in various forms, including meshes, point clouds, and other formats. In our paper, we implicitly represent the 3D model as a collection of random view observations of the 3D object. These views can then be converted into 3D geometry using volumetric reconstruction methods such as Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (3DGS).

#### Camera Model Representation.

Figure: Figure 6: Spherical coordinate system [].
Refer to caption: x3.png

We employ a spherical coordinate system to represent camera poses and their relative transformations. As shown in Fig. 6, we place the origin of the coordinate system at the center of the object. In this system, $\theta$, $\phi$, and $r$ represent the polar angle, azimuth angle, and radius (distance from the center to the camera position), respectively.
The relative camera pose between two views is derived by directly subtracting their respective camera poses. For instance, if two images have camera poses $(\theta_{1},\phi_{1},r_{1})$ and $(\theta_{2},\phi_{2},r_{2})$, their relative pose is calculated as $(\theta_{2}-\theta_{1},\phi_{2}-\phi_{1},r_{2}-r_{1})$.

#### Zero123.

Zero123 is a conditional latent diffusion model, $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y},\phi})$, fine-tuned from Stable Diffusion for novel view synthesis. It generates a novel-view image at a target viewpoint, $\boldsymbol{x}_{0}=\mathcal{D}(\boldsymbol{z}_{0})$, given an input image, $\boldsymbol{y}$, and the relative camera transformation, $\phi$, where $\mathcal{D}$ maps the latent code to pixel space. This model enables 3D object generation from a single image: by applying various camera transformations, Zero123 can synthesize multiple novel views, which can then be used to reconstruct a 3D model. As a result, Zero123 defines a powerful 3D diffusion prior.

Figure: Figure 7: Graphic model of the E-step.
Refer to caption: x4.png

Figure: Algorithm 2 LatentDEM for Pose-free Sparse-view 3D Reconstruction

### C.2 E-Step

In the E-step, we perform a view-consistent diffusion process to generate target novel views from multiple input images, where we assume the camera pose of each image is known.

#### Derivation of View-consistent Diffusion Model from Zero123.

The view-consistent generation is governed by a latent diffusion process conditioned on multiple images, defined as $\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y}_{1},
\phi_{1},\boldsymbol{y}_{2},\phi_{2}})$. In the graphical model of the 3D reconstruction problem, the input images $\boldsymbol{y}_{1}$ and $\boldsymbol{y}_{2}$ represent different views of the same object.
These views should be independent of each other given the geometry of the 3D object, which is described by the latent code $\boldsymbol{z}_{t}$. Consequently, the view-consistent diffusion can be derived from Zero123 as follows:

$$ $\begin{split}p(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{1},\phi _{1},\boldsymbol{y}_{2},\phi_{2})&\propto p(\boldsymbol{y}_{1},\boldsymbol{y}_ {2}|\boldsymbol{z}_{t-1},\boldsymbol{z}_{t},\phi_{1},\phi_{2})\\ &=p(\boldsymbol{y}_{1}|\boldsymbol{z}_{t-1},\boldsymbol{z}_{t},\phi_{1})p( \boldsymbol{y}_{2}|\boldsymbol{z}_{t-1},\boldsymbol{z}_{t},\phi_{2})\\ &\propto p(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{1},\phi_{1} )p(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{2},\phi_{2}),\end{split}$ (21) $$

where the conditional diffusions from single images, $p(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{1},\phi_{1})$ and $p(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{2},\phi_{2})$, are defined by Zero123, and they both follow Gaussian distributions according to the Langevin dynamics defined by the reverse-time SDE (Eq. 2),

$$ $\begin{split}p_{t}(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{1}, \phi_{1})=\mathcal{N}\left(\frac{1}{\sqrt{1-\beta_{t}}}\left[\boldsymbol{z}_{t }+\beta_{t}\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}| \boldsymbol{y}_{1},\phi_{1}})\right],\beta_{t}I\right)\\ p_{t}(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{2},\phi_{2})= \mathcal{N}\left(\frac{1}{\sqrt{1-\beta_{t}}}\left[\boldsymbol{z}_{t}+\beta_{t }\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y}_{2}, \phi_{2}})\right],\beta_{t}I\right)\end{split}$ (22) $$

In our pose-free 3D reconstruction task, we account for potential inaccuracies in $\phi_{2}$ during early diffusion stages, so the diffusion process is modified as:

$$ $p_{t}(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{2},\phi_{2})= \mathcal{N}\left(\frac{1}{\sqrt{1-\beta_{t}}}\left[\boldsymbol{z}_{t}+\beta_{t }\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y}_{2}, \phi_{2}})\right],(\beta_{t}+\nu_{t}^{2})I\right),$ (23) $$

where $\nu_{t}$ represents the standard deviation of time-dependent model errors. As a result, $p(\boldsymbol{z}_{t-1}|\boldsymbol{z}_{t},\boldsymbol{y}_{1},\phi_{1},
\boldsymbol{y}_{2},\phi_{2})$ becomes

$$ $\mathcal{N}\left(\frac{1}{\sqrt{1-\beta_{t}}}\left[\boldsymbol{z}_{t}+\beta_{t }\frac{(\beta_{t}+\nu_{t}^{2})\nabla_{\boldsymbol{z}_{t}}\log p_{t}({ \boldsymbol{z}_{t}|\boldsymbol{y}_{1},\phi_{1}})+\beta_{t}\nabla_{\boldsymbol{ z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y}_{2},\phi_{2}})}{2\beta_{t }+\nu_{t}^{2}}\right],\frac{\beta_{t}(\beta_{t}+\nu_{t}^{2})}{2\beta_{t}+\nu_{ t}^{2}}\right),$ (24) $$

which defines a view-consistent diffusion model, whose score function is a weighted average of two Zero123 models:

$$ $\begin{split}\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}| \boldsymbol{y}_{1},\phi_{1},\boldsymbol{y}_{2},\phi_{2}})&=(1-\gamma_{t}) \nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t}|\boldsymbol{y}_{1}, \phi_{1}})+\gamma_{t}\nabla_{\boldsymbol{z}_{t}}\log p_{t}({\boldsymbol{z}_{t} |\boldsymbol{y}_{2},\phi_{2}}),\\ \gamma_{t}&=\frac{\beta_{t}}{2\beta_{t}+\nu_{t}^{2}}.\end{split}$ (25) $$

This derivation can be readily extended to scenarios with multiple unposed images.

#### Annealing of View-consistent Diffusion.

An annealing strategy is essential for the view-consistent diffusion process in pose-free 3D reconstruction due to initial inaccuracies in estimated camera poses. In a two-image-based 3D reconstruction problem, for instance, model errors are quantified by $\nu_{t}$ in Eq. 25. This error term starts large and gradually decreases to zero as the camera pose $\phi_{2}$ becomes increasingly accurate. Consequently, $\gamma_{t}$ progressively increases from nearly 0 to 0.5. During early stages, the diffusion process primarily relies on a single reference view $\boldsymbol{y}_{1}$, and the intermediate generative images are utilized to calibrate camera poses of other images. As camera poses gain accuracy, the other images exert growing influence on the diffusion process, ultimately achieving view-consistent results.

### C.3 M-step

The M-step estimates unknown camera poses by aligning unposed images to synthetic and reference views:

$$ $\phi_{2}=\arg\min_{\phi}\mathbb{E}_{\hat{\boldsymbol{x}}_{0}}\left[\lambda\| \boldsymbol{z}_{t}(\boldsymbol{y}_{2},\phi_{2})-\boldsymbol{z}_{t}(\hat{ \boldsymbol{x}}_{0},\mathbf{0})\|_{2}^{2}+\delta\|\boldsymbol{z}_{t}( \boldsymbol{y}_{2},\phi_{2})-\boldsymbol{z}_{t}(\boldsymbol{y}_{1},\phi_{1})\| _{2}^{2}\right],$ (26) $$

where $\hat{\boldsymbol{x}}_{0}$ are the posterior samples from the E-step, and $\lambda$ and $\delta$ balance the calibration loss on the synthetic and reference images, respectively. $\boldsymbol{z}_{t}(\cdot,\cdot)$ is the time-dependent latent variable representing semantic information of the transformed input image. As suggested by Eq. 12, this optimization problem can be efficiently solved using gradient-based optimization. We dynamically adjust the ratio of the two balancing factors, $\lambda/\delta$, throughout the diffusion process. In early stages, we set $\lambda/\delta$ to a small value, primarily relying on the reference image for pose calibration. As the synthetic image becomes more realistic during the diffusion process, we gradually increase $\lambda/\delta$ until it converges to 1, balancing the influence of both synthetic and reference images in the final stages.

Figure: Figure 8: 3D reconstruction of an apple using different numbers of unposed images. We evaluate our method’s performance with varying numbers of input sparse views. Left: One view (Output 1 is from Input view 1 which is 1equivalent to Zero123) generates an unrealistic model that is hollow inside. Middle: Output 2 is from Input view 1 & 2. Two views improve results but still exhibit hallucinations in the 3D geometry of the bitten apple, *e.g*. the apple handle is missing. Right: Output 3 is from all three input views. Three views successfully recover all details.

Figure: Figure 9: 3D mesh reconstruction with One-2-3-45 []. We compare our method’s performance on 3D mesh reconstruction with One-2-3-45. Both texture and textureless meshes are shown. The baseline sometimes fails to recover fine details as they can only take one input view, while our method shows better mesh reconstruction with two input views.

### C.4 3D Reconstruction from 2D Novel Views

Previous sections primarily focused on synthesizing novel views from unposed input images, however, this technique can be extended to render 3D objects by generating multiple random novel views. Following the 3D reconstruction process described in One-2-3-45[], we input these synthetic images and their corresponding poses into an SDF-based neural surface reconstruction module to achieve $360^{\circ}$ mesh reconstruction. Figure 9 demonstrates the results of this process, showing 3D reconstructions of both textured and textureless meshes derived from two unposed input images.

### C.5 More Input Views Yield Better Reconstruction

In Sec. C.2, Sec. C.3 and Sec. C.4, we primarily explore 3D reconstruction based on images from two unposed views. This approach is readily adaptable to scenarios involving an arbitrary number of views. We evaluate the quality of 3D reconstructions using one, two, and three unposed images, as shown in Fig. 8. The results indicate that adding more views significantly improves the fidelity of the 3D reconstruction. The LatentDEM framework facilitates consistent 3D reconstruction across different images. Specifically, 3D reconstruction from a single view (equivalent to Zero123) results in a hollow reconstruction, whereas incorporating additional views progressively yields more realistic 3D models.

## Appendix D More Ablations and Experimental Results

#### LDMVQ-4 v.s. Stable Diffusion-V1.5.

We evaluate LatentDEM’s performance on 2D blind deblurring using two widely adopted foundational latent diffusion models: LDMVQ-4 and Stable Diffusion V1.5 []. Fig. 10 presents the results of this comparison. Both models achieve satisfactory reconstruction outcomes, demonstrating LatentDEM’s ability to generalize across various LDM priors when solving blind inverse problems. Notably, LatentDEM exhibits superior performance with Stable Diffusion compared to LDMVQ-4. This difference can be attributed to Stable Diffusion’s more recent release and its reputation as a more advanced diffusion model.

#### Vanilla EM v.s. Diffusion EM.

Traditional EM algorithms perform both E-step and M-step until convergence at each iteration before alternating. This approach guarantees a local optimal solution, as demonstrated by numerous EM studies. However, in the context of diffusion posterior sampling, which involves multiple reverse diffusion steps, this paradigm proves inefficient and computationally expensive.

Our implementation of the EMDiffusion algorithm for blind inverse tasks deviates from this conventional approach. Instead of waiting for the diffusion process to converge (typically 1,000 reverse steps with a DDIM scheduler), we perform model parameter estimation (M-step) after each single diffusion reverse step, except during the initial stage where we employ the skip gradient technique, as detailed in Sec. 4.2. Table 3 compares the performance of vanilla EM and our method on the blind deblurring task. Our approach, LatentDEM, requires only 1.5 minutes compared to vanilla EM’s 120 minutes, while achieving superior reconstruction quality and kernel estimation.

These results demonstrate that performing the M-step after each reverse step is both effective and efficient for blind inverse tasks. Moreover, this strategy offers improved escape from local minima and convergence to better solutions compared to vanilla EM, which completes the entire diffusion reverse process in each EM iteration.

**Table 3: Vanilla EM v.s. Diffusion EM in blind deblurring.**
| Method | Image | Kernel | Time |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PSNR | SSIM | LPIPS | MSE | MNC | Minute |  |
| Vanilla EM | 20.43 | 0.561 | 0.419 | 0.024 | 0.124 | 120 |
| LatentDEM | 22.23 | 0.695 | 0.183 | 0.023 | 0.502 | 1.5 |

Figure: Figure 10: LDMVQ-4 v.s. Stable Diffusion-V1.5. Stable Diffusion V1.5 generates results with more detailed textures due to its more powerful priors.

#### Additional Results of Pose-free Sparse-view Novel-view Synthesis.

Figure 11 presents additional results of novel-view synthesis using LatentDEM.

Figure: Figure 11: Pose-free sparse-view novel-view synthesis.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: the following issues and ambiguities require further clarification or validation:

1. While (Chung et al. 2022b) formulates the score of the data likelihood as $\nabla_{z_t} \log p_t(y|z_t)$ in Eq.(14), this paper opts for Eq.(9), citing “annealing consistency” for enhanced stability and performance. The authors should provide empirical evidence or theoretical justification for this choice. How does “annealing consistency” compare to the “vanilla” latent DPS term? Also, clarify the behavior of Eq.(9) and Eq.(18) when measurements are noiseless, i.e.  $\sigma = 0$.

2. For the 2D blind deblurring task, Eq.(16) includes a “gluing” regularization term (in addition to “vanilla” latent DPS-based term) as per (Rout et al. 2024). It is unclear whether the reported improvements are due to this regularization or the proposed “annealing consistency.” An experiment isolating these factors in 2D deblurring would be helpful.

3. The paper should explain why BlindDPS performs much better on FFHQ than ImageNet, and detail the training setup. Additionally, comparing FID metrics between this work, FastEM (Rombach et al. 2022), and BlindDPS (Chung et al. 2022a) would improve clarity [as used in the aforementioned papers as well]. Why does FastEM show negligible improvement over initial observations (e.g., see the estimated blurring kernel which is close to Dirac delta)?

4. Clarify the parameter $\phi$ for widely studied image inverse tasks (e.g., compressed sensing, tomography, super-resolution, inpainting, spectral imaging) to enhance understanding of its role across applications.

5. The paper aims to tackle [general] ‘blind’ inverse problems through estimating the parameters of forward measurement operator in the M-step of the EM framework as eq.(10).  To this aim, the parameter $\phi$ (e.g., blurring kernel for the task of 2D deblurring) is estimated through a PnP denoising step as eq.(19). Please elucidate on whether a denoiser pre-trained on Gaussian noise can generalize to other inverse tasks like compressed sensing, and explain what signal regularities it might exploit in estimating $\phi$.

6. A comprehensive list of trainable and tunable hyperparameters for the proposed method, along with those used in Fast Diffuse EM (Laroche et al. 2024) and BlindDPS (Chung et al. 2022a), would facilitate fair comparisons.

7. Please specify the network used in the E-step and provide experimental setup details for Table 1.

Minor Comments:

- In the context “these priors cannot capture the complex natural image distributions, limiting the solvers’ ability to produce high-quality reconstructions”, please consider replacing the (Candes & Romberg, 2007) citation with the following, which better aligns with the stated limitations of image priors:
https://doi.org/10.1109/TIP.2005.863057
https://doi.org/10.1109/ICIP.2007.4379013
https://doi.org/10.1109/GlobalSIP.2013.6737048
https://doi.org/10.1137/16M1102884

- Please clarify initialization of $\mathcal{A}_{\phi^{(t)}}$ in Algorithm 1 and specify corresponding M-step equations (perhaps Appendix C.2 ?)

- Differences between Eq.(11) of this paper and Eq.(24)/Eq.(B.9) in (Laroche et al. 2024) require clarification.
    
- Please correct discrepancies in Section 4.3 regarding the noiseless assumption in Eq.(1).

- Minor typographical and grammatical errors include:
  * Typo corrections: “a intermediate” and “an one-to-many.”
  * Redundant citations: “Liu et al. (2023b)” and “Liu et al. (2023c)” refer to the same work.
  * Misuse of “While” and “However” following Eq.(4).
  * Replace “introduce” with “utilize” or “employ” when referring to “gluing” regularization as it is not part of the contribution of this work.
  * The paper overlooks the following relevant work:
        Murata et al., “Gibbsddrm: A partially collapsed Gibbs sampler for solving blind inverse problems with denoising diffusion restoration.” ICML, 2023.
soundness: 2
strengths: This problem holds broad interest across the machine learning and signal processing communities.
The paper is generally well-structured and somewhat clear.
summary: This paper aims to tackle the challenging problem of blind inverse tasks by employing latent diffusion priors within an expectation-maximization (EM) framework, which iteratively estimates both the unknown measurement operator (within M-step) and the underlying image (within E-step). The proposed approach has been validated for tasks of 2D blind deburring and 3D pose-free sparse-view reconstruction. The parameters of the forward model are estimated in the M-step through using a pre-trained denoiser of Gaussian noise within a PnP framework. In the E-step, a so called "annealing consistency" approach is proposed through introducing a tunable hyper-parameter in the score of likelihood term to stabilize the training. In addition to the score of likelihood term,  a "gluing" regularization (Rout et al, 2024) is employed within diffusion posterior sampling for 2D deblurring task.
weaknesses: Several technical ambiguities require clarification. These include the choice of likelihood formulation, the role of “gluing” regularization versus annealing consistency, and the performance discrepancies across datasets. More experimental validation, particularly isolating the effects of different regularization techniques, would strengthen the claims. Additionally, a detailed comparison of hyperparameters with competing methods would improve transparency and reproducibility. For further information of the detected ambiguities please refer to the section "Questions".

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - The presentation of EM in Section 3 differs from the plain EM formulation (as in Dempster et al., 1977 and textbooks).
For example, what are the unobserved/missing data? Where in the E-step do you compute (conditional) expectation of the log likelihood/posterior with respect to the missing data?
The authors may refer to some variational Bayes modification of EM but not to the actual EM. Make the discussion more clear and point to the reference that justifies your formulation.

- Consequently, the presentation of the method in Section 4 needs to be improved. The derivation of the method does not match the EM paradigm. It looks like some alternating optimization scheme that lacks formal connection to EM.

- Lack of theoretical analysis for the proposed modifications to the baseline alternating optimization approach.  
For "Technique 1" there is some motivation in the appendix, but no formal mathematical argument that actually considers the effect of error in phi. The other modifications are quite heuristic.  

- Regarding the experiments, you need to provide more details on the kernels, noise levels, etc., that you use in training and in testing.  
Also, state in some table the hyperparameters that you use. Do you use the same hyperparameters for all the settings and for both 2D and 3D?

-  The reported results of the competitors for blind deblurring on FFHQ are lower than those reported in previous works  (Chung et al., 2022a), (Laroche et al., 2024) (e.g., worse kernel estimation for BlindDPS and huge deterioration for FastEM).  
How can you explain it? Do you use the same setup as these works? If not, then explain why.

- Please compare also with GibbsDDRM:  
Murata, Naoki, et al. "Gibbsddrm: A partially collapsed gibbs sampler for solving blind inverse problems with denoising diffusion restoration." ICML, 2023.

- Present also the results for the non-blind methods on which BlindDPS and FastEM, and your approach, are based. They can be interpreted as the upper bound on the results, as done in GibbsDDRM and FastEM.
soundness: 2
strengths: - As far as I know, this is the first work that uses a pretrained LDM for blind restoration.

- The proposed method appears to have empirical advantages over competitors.
summary: The paper proposed a method for solving blind restoration problems (examined mostly on blind deblurring) using a pretrained latent diffusion model (LDM). It is based on adding to the iterations of unconditional LDM sampling steps (which are sometimes skipped) where the the observation/forward model parameters (e.g., blur kernel) are updated and being used for guiding the diffusion (I do not see a clear relation to expectation maximization). The proposed method is quite heuristic and shares similar ideas with existing works that use guided LDM for non-blind restoration and works that use guided pixel-space DMs for blind restoration. In the reported experiments it outperforms alternative methods.
weaknesses: - The contribution seems incremental in terms of the techniques being used and the lack of theoretical motivation and analysis of the proposed method. 

- The presentation is not good in terms of explaining why the authors refer to the proposed method as an expectation maximization (EM) technique.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: Have the authors observed any instances of non-converging patterns in their algorithm? Specifically, does a particular choice of annealing or tunable parameters (Equations 11 or 12) lead to divergence?
soundness: 3
strengths: The paper is well-organized and clearly communicates its results. 

The authors provide extensive experimental results, comparing their method to prominent existing methods for blind inverse problems, such as BlindDPS and FastEM. 

They provide an ablation study examining the influence of gradient estimation skipping and annealing consistency on performance.
summary: This paper introduces LatentDEM, a novel method for addressing blind inverse problems through latent diffusion models. The proposed method leverages an Expectation-Maximization (EM) framework: in the expectation step, a clean image is recovered using a latent diffusion prior, and in the maximization step, the forward model is estimated based on the recovered clean image. The authors demonstrate the method’s application to blind deblurring and 3D pose-free sparse-view reconstruction.
weaknesses: 1. A primary concern is the paper’s original contribution. The methodology appears to combine elements previously explored in the literature. For instance, latent diffusion models for inverse problems have been investigated in [1], while Expectation-Maximization approaches for blind inverse problems have been studied in [2]. Simply substituting a latent diffusion model as the prior does not, in itself, seem to constitute a novel contribution. Explicitly listing the contributions could improve clarity.

2. Another concern is the runtime of the proposed method. As seen in Table 2, the method appears to trade time for performance, with comparable results to BlindDPS but with similar computational demands. In other words, we would not gain any performance advantage compared to BlindDPS, if are limited by the runtime of the method. 

3. The reported performance metrics for FastEM and BlindDPS appear inconsistent with previous results. For instance, in the FastEM paper, performance on the FFHQ dataset is as follows: MPRNet: 19.52, BlindDPS: 24.05, and FastEM: 25.75, demonstrating comparability to DPS given the GT kernel. However, in this paper, the results are MPRNet: 21.60, BlindDPS: 22.58, and FastEM: 17.46. Could the authors provide an explanation for this performance discrepancy? Is the comparison of FastEM accurate? Can they reproduce FastEM’s results by strictly following its settings for a more direct comparison?

[1] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.

[2] Charles Laroche, Andrés Almansa, and Eva Coupete. Fast diffusion em: a diffusion model for blind inverse problems with application to deconvolution. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5271–5281, 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - What is the denoiser used for estimating the blur kernels? how was that denoiser trained? 
- Is there an overlap between training kernels and test kernels? How were the test kernels generated/chosen? 
- Was the Stable Diffusion model was trained on the FFHQ and ImageNet images used for testing the models?
soundness: 2
strengths: - introduces an approach that leverages pretrained encoder/decoders which reduces the computational burden of working in the pixel space.
- good experimental results in blind deblurring and sparse view 3D reconstruction.
summary: The paper presents a new diffusion-based model for blind inverse problems, which alternates between a diffusion step with a fixed parameter (eg a fixed blur kernel), and a step estimating the kernel given an estimate of the image. The paper motivates this algorithm as an instance of the expectation-maximization (EM) framework. In particular, the approach leverages powerful latent diffusion models such as Stable Diffusion v1.5, and performs the bulk of the diffusion in the latent space. 

The method is demonstrated in blind image deblurring and sparse view 3D reconstruction, obtaining a better performance in terms of both distortion and perceptual quality than the competing baselines.
weaknesses: - I believe the link to the EM framework is not sufficiently well founded: the MAP kernel estimation step uses \hat{x_0} instead of a sample from the posterior p(x|y, \phi). It is not clear to me what is the role of q(x|y) in equation 5, and more generally the equation does not add much to the explanation in my opinion.
- The experimental setup is not sufficiently well described in the main text, i) raises doubts about testing on a training set (see related questions below) and ii) the papers seems to use circular padding which is not a very realistic setting.
- The presentation of the paper could be improved:
    - the citation style is incorrect in many parts of the paper (use citep{} for references that are not part of a sentence)
    - the mathematical notation is not consistent: equation 11 uses Z uppercase for denoting a vector, whereas the rest uses lowercase, the letter 'z' is used for both latent space vectors and auxiliary variables in 11, the letter D is used for a denoiser and a decoder. What is the  'F^2' operator in equation 11? Acronyms are defined multiple times throughout the text (eg. EM). The method Zero123 is not followed by a reference when introduced on page 7.

Meta-review: This paper proposes LatentDEM, a novel method for addressing blind inverse problems through latent diffusion models within an Expectation-Maximization (EM) framework. The authors claim that their method outperforms existing methods for 2D blind deblurring and enables new capabilities in non-linear 3D inverse rendering problems, such as pose-free sparse-view 3D reconstruction.  

Strengths:
- The paper is generally well-written and easy to follow.  
- It proposes a novel method for addressing blind inverse problems using latent diffusion models.  
- The method is shown to be effective for both 2D and 3D inverse problems.  

Weaknesses:
- Some reviewers have raised concerns about the clarity and accuracy of the EM formulation.  
- There are ambiguities regarding the likelihood formulation, the role of "gluing" regularization versus annealing consistency, and the handling of performance discrepancies across datasets.  
- The novelty of the "annealing consistency" technique is questioned.  
- More experimental validation is needed, particularly isolating the effects of different regularization techniques and providing a detailed hyperparameter comparison for reproducibility.  

The decision to reject this paper is based on the following reasons:
- The concerns about the EM formulation are not fully addressed. This hinders the understanding of the proposed formulation.
- The technical ambiguities and the lack of novelty of the "annealing consistency" technique raise concerns about the overall soundness of the method.
- The experimental validation is not sufficient to support the claims.

This paper presents interesting ideas but their potential impact remains unclear due to the current presentation.  A significant revision is necessary to address the reviewers' concerns and fully articulate the paper's contributions. In particular, the following aspects could help make the work clearer and better:

- Clearly respond to concerns regarding the E and M steps.
- Discuss the drawbacks of the proposed methods transparently.
- Clarify that annealing consistency is an existing technique, not a novelty of the paper.
- Provide a comparison between annealing and gluing performances.
- Review, cite, and compare overlooked related works.
- Detail the experimental setup and configurations for all methods.
- Elaborate on the poor performance of comparative methods.

Decision: Reject