Solving Video Inverse Problems Using Image Diffusion Models

## Abstract

Abstract Recently, diffusion model-based inverse problem solvers (DIS) have emerged as state-of-the-art approaches for addressing inverse problems, including image super-resolution, deblurring, inpainting, etc.
However, their application to video inverse problems arising from spatio-temporal degradation remains largely unexplored due to the challenges in training video diffusion models.
To address this issue, here we introduce an innovative video inverse solver that leverages only image diffusion models.
Specifically, by
drawing inspiration from the success of the recent decomposed diffusion sampler (DDS),
our method treats the time dimension of a video as the batch dimension of image diffusion models and solves spatio-temporal optimization problems within denoised spatio-temporal batches derived from each image diffusion model.
Moreover, we introduce a batch-consistent diffusion sampling strategy that encourages consistency across batches by synchronizing the stochastic noise components in image diffusion models.
Our approach synergistically combines batch-consistent sampling with simultaneous optimization of denoised spatio-temporal batches at each reverse diffusion step, resulting in a novel and efficient diffusion sampling strategy for video inverse problems.
Experimental results demonstrate that our method effectively addresses various spatio-temporal degradations in video inverse problems, achieving state-of-the-art reconstructions.
Project page: https://solving-video-inverse.github.io/main/

## 1 Introduction

Diffusion models (Ho et al., ; Song et al., ) represent the state-of-the-art generative modeling by learning the underlying data distribution $p({\bm{x}})$ to produce realistic and coherent data samples from the learned distribution $p_{\theta}({\bm{x}})$.
In the context of Bayesian inference, the parameterized prior distribution $p_{\theta}({\bm{x}})$ can be disentangled from the likelihood $p({\bm{y}}|{\bm{x}})$, which denotes the probability of observing ${\bm{y}}$ given ${\bm{x}}$. This seperation facilitates the derivation of the posterior distribution $p_{\theta}({\bm{x}}|{\bm{y}})\propto p_{\theta}({\bm{x}})p({\bm{y}}|{\bm{x}})$.

Diffusion model-based inverse problem solvers (DIS) (Kawar et al., ; Chung et al., ; Song et al., ; Wang et al., ; Chung et al., ) leverage this property, enabling the unconditional diffusion models to solve a wide range of inverse problems.
They achieve this by conditional sampling from the posterior distribution $p_{\theta}({\bm{x}}|{\bm{y}})$, effectively integrating information from both the forward physics model and the measurement ${\bm{y}}$.
This approach allows for sophisticated and precise solutions to complex inverse problems, introducing the power and flexibility of diffusion models in practical applications.

Despite extensive DIS research on a wide range of image inverse problems such as super-resolution, colorization, inpainting, compressed sensing, deblurring, and so on (Jalal et al., ; Kawar et al., ; Chung et al., ; Song et al., ; Wang et al., ; Chung et al., ), the application of these approaches to video inverse problems, particularly those involving spatio-temporal degradation, has received relatively less attention.
Specifically,
in time-varying data acquisition systems, various forms of motion blur often arise due to the camera or object motions (Potmesil & Chakravarty, ), which can be modeled as a temporal PSF convolution of motion dynamics. These are often associated with spatial degradation caused by noise, camera defocus, and other factors.
Specifically, the spatio-temporal degradation process can be formulated as:

$$ $\displaystyle{\bm{Y}}={\mathcal{A}}({\bm{X}})+{\bm{W}}$ (1) $$

with

$$ $\displaystyle{\bm{X}}=\begin{bmatrix}{\bm{x}}[1]&\cdots&{\bm{x}}[N]\end{ bmatrix},\quad{\bm{Y}}=\begin{bmatrix}{\bm{y}}[1]&\cdots&{\bm{y}}[N]\end{ bmatrix},\quad{\bm{W}}=\begin{bmatrix}{\bm{w}}[1]&\cdots&{\bm{w}}[N]\end{ bmatrix},$ (2) $$

where ${\bm{x}}[n],{\bm{y}}[n]$ and ${\bm{w}}[n]$ denote the $n$-th frame ground-truth image,
measurement, and additive noise, respectively; $N$ is the number of temporal frames, and
${\mathcal{A}}$ refers to the operator that describes the spatio-temporal degradation process.
The spatio-temporal degradation introduces complexities that image diffusion priors cannot fully capture, as image diffusion priors are primarily designed to handle spatial features rather than temporal dynamics.
Employing video diffusion models (Ho et al., ) could address these issues, but
poses significant implementation challenges for video inverse problems, due to the difficulty
of training video diffusion models for various applications.

Contrary to the common belief that a pre-trained video diffusion model is necessary for solving video inverse problems, here we propose a radically different method that addresses video inverse problems using only image diffusion models.
Inspired by the success of the decomposed diffusion sampler (DDS) (Chung et al., ), which simplifies DIS by formulating it as a Krylov subspace-based optimization problem for denoised images via Tweedie’s formula at each reverse sampling step,
we treat the time dimension of a video as the batch dimension of image diffusion models
and solve spatio-temporal optimization problems using the batch of denoised temporal frames from image diffusion models.
However, treating each frame of the video as a separate sample in the batch dimension can lead to inconsistencies between temporal frames. To mitigate this, we introduce the batch-consistent sampling strategy
that controls the stochastic directional component (e.g., initial noise or additive noise) of each image diffusion model during the reverse sampling process, encouraging the temporal consistency along the batch dimension.
By synergistically combining batch-consistent sampling with the simultaneous optimization of the spatio-temporal denoised batch, our approach effectively addresses a range of spatio-temporal inverse problems, including spatial deblurring, super-resolution, and inpainting.
Our contribution can be summarized as follows.

- •
We introduce an innovative video inverse problem solver using pre-trained image diffusion models by solving spatio-temporal optimization problems within the batch of denoised frames.
- •
We develop a batch-consistent sampling strategy to ensure temporal consistency by synchronizing stochastic noise components in image diffusion models.
- •
Extensive experiments confirm that our method generates state-of-the-art results for various video inverse problems.

## 2 Background

Diffusion models.
Diffusion models (Ho et al., ) attempt to model the data distribution $p_{\text{data}}({\bm{x}})$ based on a latent variable model

$$ $\displaystyle p_{\theta}({\bm{x}}_{0})=\int p_{\theta}({\bm{x}}_{0:T})d{\bm{x} }_{1:T},\quad\text{where}\quad p_{\theta}({\bm{x}}_{0:T}):=p_{\theta}({\bm{x}} _{T})\prod_{t=1}^{T}p_{\theta}^{(t)}({\bm{x}}_{t-1}|{\bm{x}}_{t})$ (3) $$

where the ${\bm{x}}_{1:T}$ are noisy latent variables defined by the Markov chain with Gaussian transitions

$$ $\displaystyle q({\bm{x}}_{t}|{\bm{x}}_{t-1})=\mathcal{N}({\bm{x}}_{t}|\sqrt{ \beta_{t}}{\bm{x}}_{t-1},(1-\beta_{t})I),\quad q({\bm{x}}_{t}|{\bm{x}}_{0})= \mathcal{N}({\bm{x}}_{t}|\sqrt{\bar{\alpha}_{t}}{\bm{x}}_{0},(1-\bar{\alpha}_{ t})I).$ (4) $$

Here, the noise schedule $\beta_{t}$ is an increasing sequence of $t$, with $\bar{\alpha}_{t}:=\prod_{i=1}^{t}\alpha_{i},\,\alpha_{i}:=1-\beta_{i}$.
Training of diffusion models amounts to training a multi-noise level residual denoiser:

$$ $\displaystyle\min_{\theta}\mathbb{E}_{{\bm{x}}_{t}\sim q({\bm{x}}_{t}|{\bm{x}} _{0}),{\bm{x}}_{0}\sim p_{\text{data}}({\bm{x}}_{0}),\bm{\epsilon}\sim\mathcal {N}({\bm{0},I})}\left[\|\bm{\epsilon}_{\theta}^{(t)}({\bm{x}}_{t})-\bm{ \epsilon}\|_{2}^{2}\right].$ (5) $$

Then, sampling from (3) can be implemented by ancestral sampling, which iteratively performs

$$ $\displaystyle{\bm{x}}_{t-1}=$ $\displaystyle\frac{1}{\sqrt{\alpha_{t}}}\left({\bm{x}}_{t}-\frac{1-\alpha_{t}} {\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta^{*}}^{(t)}({\bm{x}}_{t}) \right)+\tilde{\beta}_{t}\bm{\epsilon}$ (6) $$

where $\tilde{\beta}_{t}:=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t}$
and
$\theta^{*}$ refers to the optimized parameter from Eq. (5).
On the other hand, DDIM (Song et al., ) accelerates the sampling based on
non-Markovian assumption. Specifically, the sampling iteratively performs

$$ $\displaystyle{\bm{x}}_{t-1}=$ $\displaystyle\sqrt{\bar{\alpha}_{t-1}}\hat{{\bm{x}}}_{t}+\sqrt{1-\bar{\alpha}_ {t-1}}\hat{\bm{\epsilon}}_{t}$ (7) $$

where

$$ $\displaystyle\hat{{\bm{x}}}_{t}:=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left({\bm{x }}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}_{\theta^{*}}^{(t)}({\bm{x}}_{t}) \right),\quad\hat{\bm{\epsilon}}_{t}:=\frac{\sqrt{1-\bar{\alpha}_{t-1}-\eta^{2 }\tilde{\beta}^{2}_{t}}\bm{\epsilon}_{\theta^{*}}^{(t)}({\bm{x}}_{t})+\eta \tilde{\beta}_{t}\bm{\epsilon}}{\sqrt{1-\bar{\alpha}_{t-1}}}$ (8) $$

Here, $\hat{{\bm{x}}}_{t}$ is the denoised estimate of ${\bm{x}}_{t}$ that is derived from Tweedie’s formula (Efron, ).
Accordingly, DDIM sampling can be expressed as a two-step manifold transition: (i) the noisy sample ${\bm{x}}_{t}\in\mathcal{M}_{t}$ transits to clean manifold $\mathcal{M}$ by deterministic estimation using Tweedie’s formula, (ii) a subsequent transition from clean manifold to next noisy manifold $\mathcal{M}_{t-1}$ occurs by adding noise $\hat{\bm{\epsilon}}_{t}$,
which is composed of the deterministic noise $\bm{\epsilon}_{\theta^{*}}^{(t)}({\bm{x}}_{t})$ and the stochastic noise $\bm{\epsilon}$.

Diffusion model-based inverse problem solvers.
For a given loss function $\ell({\bm{x}})$ which often stems from the likelihood
for
measurement consistency, the goal of DIS is to address the following optimization problem

$$ $\displaystyle\min_{{\bm{x}}\in{\mathcal{M}}}\ell({\bm{x}})$ (9) $$

where
${\mathcal{M}}$ represents the clean data manifold sampled from unconditional distribution $p_{0}({\bm{x}})$.
Consequently, it is essential to find a way that minimizes cost while also identifying the correct manifold.

Recently, Chung et al. () proposed a general technique called diffusion posterior sampling (DPS), where the updated estimate from the noisy sample ${\bm{x}}_{t}\in{\mathcal{M}}_{t}$ is constrained to stay on the same noisy manifold ${\mathcal{M}}_{t}$.
This is achieved by computing the manifold constrained gradient (MCG) (Chung et al., ) on a noisy sample ${\bm{x}}_{t}\in{\mathcal{M}}_{t}$.
The resulting algorithm can be stated as follows:

$$ $\displaystyle{\bm{x}}_{t-1}$ $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}\left(\hat{\bm{x}}_{t}-\gamma_{t}\nabla _{{\bm{x}}_{t}}\ell(\hat{\bm{x}}_{t})\right)+\sqrt{1-\bar{\alpha}_{t-1}}\hat{ \bm{\epsilon}}_{t},$ (10) $$

where $\gamma_{t}>0$ denotes the step size.
Under the linear manifold assumption (Chung et al., ; ), this allows precise transition to ${\mathcal{M}}_{t-1}$.
Unfortunately, the computation of MCG requires computationally expensive backpropagation and is often unstable.

In a subsequent work, Chung et al. () shows that
under the same linear manifold assumption in DPS,
the one step update by $\hat{\bm{x}}_{t}-\gamma_{t}\nabla_{\hat{\bm{x}}_{t}}\ell(\hat{\bm{x}}_{t})$
are guaranteed to remain within a linear subspace, thus obviating the need for explicit computation of the MCG and
leading to a simpler approximation:

$$ $\displaystyle{\bm{x}}_{t-1}$ $\displaystyle\simeq\sqrt{\bar{\alpha}_{t-1}}\left(\hat{\bm{x}}_{t}-\gamma_{t} \nabla_{\hat{\bm{x}}_{t}}\ell(\hat{\bm{x}}_{t})\right)+\sqrt{1-\bar{\alpha}_{t -1}}\hat{\bm{\epsilon}}_{t}.$ (11) $$

Furthermore, instead of using a one-step gradient update, Chung et al. () demonstrated that multi-step update using Krylov subspace methods, such as the conjugate gradient (CG) method, guarantees that the intermediate steps lie in the linear subspace. This approach improves the convergence of the optimization problem without incurring additional neural function evaluations (NFE). This method, often referred to as decomposed diffusion sampling (DDS), bypasses the computation of the MCG and improves the convergence speed, making it stable and suitable for large-scale medical imaging inverse problems (Chung et al., ).

## 3 Video Inverse Solver using Image Diffusion Models

### 3.1 Problem formulation

Using the forward model Eq. (1) and the optimization framework in Eq. (9), the video inverse problem can be formulated as

$$ $\displaystyle\min_{{\bm{X}}\in\bm{\mathcal{M}}}\ell({\bm{X}}):=\|{\bm{Y}}-{ \mathcal{A}}({\bm{X}})\|^{2}$ (12) $$

where ${\bm{X}}$ denotes the spatio-temporal volume of the clean image composed of $N$ temporal frames as defined in Eq. (2), and
$\bm{\mathcal{M}}$ represents the clean video manifold sampled from unconditional distribution $p_{0}({\bm{X}})$.
Then, a naive application of the one-step gradient within the DDS framework can be formulated by

$$ $\displaystyle{\bm{X}}_{t-1}$ $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}\left(\hat{\bm{X}}_{t}-\gamma_{t}\nabla _{\hat{\bm{X}}_{t}}\ell(\hat{\bm{X}}_{t})\right)+\sqrt{1-\bar{\alpha}_{t-1}} \hat{\bm{\mathcal{E}}}_{t}.$ (13) $$

where $\hat{\bm{X}}_{t}$ and $\hat{\bm{\mathcal{E}}}_{t}$ refer to Tweedie’s formula and noise in the spatio-temporal volume, respectively, which are defined by

$$ $\displaystyle\hat{\bm{X}}_{t}:=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left({\bm{X}} _{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\mathcal{E}}_{\theta^{*}}^{(t)}({\bm{X}}_{t} )\right),\quad\hat{\bm{\mathcal{E}}}_{t}:=\frac{\sqrt{1-\bar{\alpha}_{t-1}- \eta^{2}\tilde{\beta}^{2}_{t}}\bm{\mathcal{E}}_{\theta^{*}}^{(t)}({\bm{X}}_{t} )+\eta\tilde{\beta}_{t}\bm{\mathcal{E}}}{\sqrt{1-\bar{\alpha}_{t-1}}}$ (14) $$

Here, ${\bm{X}}_{t}$ refers to the spatio-temporal volume at the $t$-th reverse diffusion step and
$\bm{\mathcal{E}}\sim\prod_{i=1}^{N}\mathcal{N}(\bm{0},{\bm{I}})$.
Although the formula Eq. (14) is a direct extension of the image-domain counterpart Eq. (8),
the main technical challenge lies in training the video diffusion model $\bm{\mathcal{E}}_{\theta}^{(t)}$, which is required for the formula Eq. (14). Specifically, the video diffusion model is trained by

$$ $\displaystyle\min_{\theta}\mathbb{E}_{{\bm{X}}_{t}\sim q({\bm{X}}_{t}|{\bm{X}} _{0}),{\bm{X}}_{0}\sim p_{\text{data}}({\bm{X}}_{0}),\bm{\mathcal{E}}\sim\prod _{i=1}^{N}\mathcal{N}(\bm{0},{\bm{I}})}\left[\|\bm{\mathcal{E}}_{\theta}^{(t)} ({\bm{X}}_{t})-\bm{\mathcal{E}}\|_{2}^{2}\right],$ (15) $$

which requires large-scale video training data and computational resources beyond the scale of training image diffusion models.
Therefore, the main research motivation is to propose an innovative method that can bypass the need for computationally extensive video diffusion models.

### 3.2 Batch-consistent reconstruction with DDS

Consider
a batch of 2D diffusion models along the temporal direction:

$$ $\displaystyle\bm{\tilde{\mathcal{E}}}_{\theta}^{(t)}({\bm{X}}_{t}):=\begin{ bmatrix}\bm{\epsilon}_{\theta^{*}}^{(t)}({\bm{X}}_{t}[1])&\cdots&\bm{\epsilon} _{\theta^{*}}^{(t)}({\bm{X}}_{t}[N])\end{bmatrix}$ (16) $$

where $\bm{\epsilon}_{\theta^{*}}^{(t)}$ represents an image diffusion model.
Suppose that $\bm{\tilde{\mathcal{E}}}_{\theta}^{(t)}({\bm{X}}_{t})$ is used for Eq. (14).
Since unconditional reverse diffusion is entirely determined by Eq. (14), the
generated video
is then
fully controlled by the behavior of the image diffusion models.
Thus, we investigate the limitations of using a batch of image diffusion models compared to using a video diffusion model
and explore ways to mitigate these limitations.

Recall that for the reverse sampling of each image diffusion model, the stochastic transitions occur from two sources: (i) the initialization and (ii) re-noising.
Accordingly, in batch-independent sampling, where each image diffusion model is initialized with independent random noise and re-noised with independent additive noise, it is difficult to impose any temporal consistency in video generation so that
each generated temporal frame may represent different content from each other (see Fig. 2(a)).
Conversely, in batch-consistent sampling, where each image diffusion model is initialized with the same noise and re-noised with the same additive noise, the generated frames from the unconditional diffusion model should be trivially reduced to identical images (see Fig. 2(b)). This dilemma is why separate video diffusion model training using Eq. (15) was considered necessary for effective video generation.

Figure: Figure 2: Geometric illustration of the sampling path evolution. (a) Batch-independent sampling produces independent frames. (b) Batch-consistent sampling produces identical frames. (c) Batch-consistent sampling combined with frame-dependent perturbation through multi-step CG generates distinct frame satisfying spatio-temporal data consistency.
Refer to caption: extracted/5832688/figures/figure_2.jpg

One of the most important contributions of this paper is demonstrating that the aforementioned dilemma can be readily mitigated in conditional diffusion sampling originated from inverse problems. Specifically, inspired by the DDS formulation in Eq. (13), we propose a method that employs a batch-consistent sampling scheme to ensure temporal consistency and introduces temporal diversity from the conditioning steps.
More specifically, the denoised image for each frame is computed individually using Tweedie’s formula via image diffusion models:

$$ $\displaystyle\hat{\bm{X}}_{t}^{b}:=\frac{1}{\sqrt{\bar{\alpha}_{t}}}\left({\bm {X}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{\tilde{\mathcal{E}}}_{\theta^{*}}^{(t)}( {\bm{X}}_{t})\right)$ (17) $$

where we use the superscript ^b to represent the batch-consistency and
$\bm{\tilde{\mathcal{E}}}_{\theta^{*}}^{(t)}$ is a batch of image diffusion models defined by Eq. (16).
Here, the image diffusion models are initialized with the same random noises to ensure temporal consistency.
Subsequently, the denoised spatio-temporal batch is perturbed as a whole by applying the $l$-step
conjugate gradient (CG) to optimize
the data consistency term from the spatio-temporal degradation.
This can be formally represented by

$$ $\displaystyle\bar{\bm{X}}_{t}:=\operatorname*{arg\,min}_{{\bm{X}}\in\hat{\bm{X }}_{t}^{b}+\bm{\mathcal{K}}_{l}}\|{\bm{Y}}-{\mathcal{A}}({\bm{X}})\|^{2}$ (18) $$

where $\bm{\mathcal{K}}_{l}$ denotes the $l$-dimensional Kyrlov subspace
associated with the given inverse problem (Chung et al., ).
The multistep CG can
diversify each temporal frame according to the condition and achieve faster convergence than a single gradient step.
The resulting solution ensures that the loss function from the spatio-temporal degradation process can be minimized with coherent but frame-by-frame distinct reconstructions.
Finally, the reconstructed spatio-temporal volume from the CG is renoised with additive noise as:

$$ $\displaystyle{\bm{X}}_{t-1}$ $\displaystyle=\sqrt{\bar{\alpha}_{t-1}}\bar{\bm{X}}_{t}+\sqrt{1-\bar{\alpha}_{ t-1}}\hat{\bm{\mathcal{E}}}_{t}^{b}.$ (19) $$

where

$$ $\displaystyle\hat{\bm{\mathcal{E}}}_{t}^{b}:=\frac{\sqrt{1-\bar{\alpha}_{t-1}- \eta^{2}\tilde{\beta}^{2}_{t}}\bm{\tilde{\mathcal{E}}}_{\theta^{*}}^{(t)}({\bm {X}}_{t})+\eta\tilde{\beta}_{t}\bm{\mathcal{E}}^{b}}{\sqrt{1-\bar{\alpha}_{t-1 }}}$ (20) $$

Here, $\bm{\mathcal{E}}^{b}$ denotes the additive random noise from $\mathcal{N}(\bm{0},{\bm{I}})$.
In contrast
to $\bm{\mathcal{E}}$ in Eq. (14), which is composed of frame-independent random noises,
we impose batch consistency by adding the same random noises to each temporal frame to ensure temporal consistency.
In summary, the proposed batch-consistent sampling and frame-dependent perturbation through multistep CG ensure that the sampling trajectory of each frame, starting from the same noise initialization, gradually diverges from each other during reverse sampling to meet the spatio-temporal
data consistency. The
geometric illustration of the sampling path evolution is shown in Fig. 2(c). The detailed illustration of the intermediate sampling process of our method is shown in Fig. 3.
Additionally, the pseudocode implementation is given in Algorithm 1.

Figure: Algorithm 1 Video inverse problem solver using 2D diffusion models

Figure: Figure 3: Sampling process in our video inverse problem solver. ${\bm{X}}_{t}$ is denoised to produce $\hat{{\bm{X}}}^{b}_{t}$ using 2D Tweedie formula, then reshaped into a video tensor. Multi-step CG in the video space, satisfying Eq. (18), is applied to obtain $\bar{{\bm{X}}}_{t}$, which is then reshaped back into an image batch. Finally, ${\bm{X}}_{t-1}$ is sampled by adding noise $\hat{\bm{\mathcal{E}}}^{b}_{t}$.
Refer to caption: extracted/5832688/figures/figure_3.jpg

## 4 Experiments

In this section, we conduct thorough comparison studies to demonstrate the efficacy of the proposed method
in addressing spatio-temporal degradations.
Specifically, we consider two types of loss functions for video inverse problems:

$$ $\displaystyle\ell({\bm{X}}):=\|{\bm{Y}}-{\mathcal{A}}({\bm{X}})\|^{2},\quad \ell_{TV}({\bm{X}}):=\|{\bm{Y}}-{\mathcal{A}}({\bm{X}})\|^{2}+\lambda\ TV({\bm {X}})$ (21) $$

where the first loss is from Eq. (12)
and $TV({\bm{X}})$ denotes the total variation loss along the temporal direction.

Then, classical optimization methods are used as the baselines for comparison to minimize each loss function. Specifically, the stand-alone Conjugate Gradient (CG) method is employed to minimize $\ell({\bm{X}})$, while the Alternating Direction Method of Multipliers (ADMM) is used to minimize $\ell_{TV}({\bm{X}})$. Additionally, diffusion-based methods are utilized as baselines to minimize the loss functions in
Eq. (21).
Specifically, DPS (Chung et al., ) is used to minimize $\ell({\bm{X}})$. However, instead of relying on 3D diffusion models, we use 2D image diffusion models, similar to our proposed methods, to ensure that backpropagation for MCG computation can be performed through 2D diffusion models.
Second, we employ DiffusionMBIR (Chung et al., ) to minimize $\ell_{TV}({\bm{X}})$, also using 2D image diffusion models. Unlike the original DiffusionMBIR, which applies TV along the $z$-direction, we apply TV along the temporal direction.

To test various spatio-temporal degradations, we select the temporal degradation in time-varying data acquisition systems, which is represented as PSF convolution along temporal dimension (Potmesil & Chakravarty, ).
We select three types of PSFs: (i) uniform PSF with widths of 7, (ii) uniform PSF with widths of 13, and (iii) Gaussian PSF with a standard deviation of 1.0. Each kernel is convolved along the temporal dimension with the ground truth video to produce the measurements. Note that convolving uniform PSF with widths of 7 and 13 correspond to averaging 7 and 13 frames, respectively.
Furthermore, we combined temporal degradation and various spatial degradations to demonstrate various combinations of spatio-temporal degradations.
For spatio-temporal degradations, we fix a temporal degradation as a convolving uniform PSF with a width of 7 and add various spatial degradations to the video.
These spatial degradations include (i) deblurring using a Gaussian blur kernel with a standard deviation $\sigma$ of 2.0, (ii) super-resolution through a 4$\times$ average pooling, and (iii) inpainting with random masking at a ratio $r$ of 0.5 (For specific implementation details of degradations, see Appendix A).

Figure: Figure 4: Qualitative evaluation of temporal degradation tasks. 1${}^{\text{st}}$ row: temporal ${\mathcal{A}}$ with uniform PSF with kernel width $k$ = 7. 2${}^{\text{nd}}$ row: temporal ${\mathcal{A}}$ with Gaussian PSF with $\sigma$=1. Red and blue boxes indicate the enlarged views of the previous and next frames, respectively.
Refer to caption: extracted/5832688/figures/figure_4.jpg

Table: Table 1: Quantitative evaluation of temporal degradation tasks on the DAVIS dataset. Bold indicates the best results. FVD is displayed scaled by $10^{-3}$ for easy comparison.

We conduct our experiments on the DAVIS dataset (Perazzi et al., ; Pont-Tuset et al., ), which includes a wide variety of videos covering multiple scenarios. The pre-trained unconditional 256$\times$256 image diffusion model from ADM (Dhariwal & Nichol, ) is used directly without fine-tuning and additional networks. All videos were normalized to the range [0, 1] and split into 16-frame samples of size 256$\times$256. A total of 338 video samples were used for evaluation. More preprocessing details are described in the Appendix A.

For quantitative comparison, we focus on the following two widely used standard metrics: peak signal-to-noise-ratio (PSNR) and structural similarity index (SSIM) (Wang et al., ) with further evaluations with two perceptual metrics - Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., ) and Fréchet Video Distance (FVD) (Unterthiner et al., ). FVD results are displayed scaled by $10^{-3}$ for easy comparison.
For all proposed methods, we employ $l$ = 5, $\eta$ = 0.15 for 20 NFE in temporal degradation tasks, and $l$ = 5, $\eta$ = 0.8 for 100 NFE in spatio-temporal degradation tasks unless specified otherwise.

### 4.1 Results

We present the quantitative results of the temporal degradation tasks in Table. 1. The table shows that the proposed method outperforms the baseline methods by large margins in all metrics.
The large margin improvements in FVD indicate that the proposed method successfully solves inverse problems with temporally consistent reconstruction.
Fig. 4 shows the qualitative reconstruction results for temporal degradations ${\mathcal{A}}$. The proposed method restores much finer details compared to the baselines and demonstrates robustness across various temporal PSFs.
In contrast, as shown in Fig. 4, while DPS performs well in reconstructing uniform PSFs with a kernel width of 7, it fails to accurately reconstruct frame intensities as the kernel becomes wider or more complex as shown in the bottom figures, leading to significant drops in Table 1.
DiffusionMBIR ensures temporal consistency and performs well for static scenes, but it struggles with dynamic scenes in the video.
In the same context, ADMM-TV produces unsatisfactory results for dynamic scenes.

Figure: Figure 5: Qualitative evaluation of spatio-temporal degradation tasks. Each spatio-temporal degradation is combined with various spatial degradation tasks. 1${}^{\text{st}}$ row: Deblurring ($\sigma$ = 2.0). 2${}^{\text{nd}}$ row: SR ($\times$ 4). 3${}^{\text{rd}}$ row: Inpainting ($r$ = 0.5).
Refer to caption: extracted/5832688/figures/figure_5.jpg

Table: Table 2: Quantitative evaluation of spatio-temporal degradation tasks on the DAVIS dataset. Bold indicates the best results. FVD is displayed scaled by $10^{-3}$ for easy comparison.

The results of the spatio-temporal degradations are presented in Table 2 and Fig. 5.
Even with additional spatial degradations, the proposed method consistently outperforms baseline methods.
On the other hand, DPS often produces undesired details, as shown in Fig. 5.
DiffusionMBIR fails to restore fine details in dynamic scenes.
Specifically, in the 3${}^{\text{rd}}$ row of Fig. 5, DiffusionMBIR restores the static mural painting but fails to capture the motion of the person.
This is because
TV regularizer often disrupts the restoration of dynamic scenes.
In this context, our method ensures temporal consistency without the need for a TV regularizer.
Furthermore, thanks to the consistent performance even at low NFE, the proposed method achieves a dramatic 10$\times$ to 50$\times$ acceleration in reconstruction time.
For handling temporal degradation with 20 NFE, the proposed diffusion model-based inverse problem solver can now achieve speeds exceeding 1 FPS.

### 4.2 Ablation study

Figure: Figure 6: Reconstruction results of (left) stand-alone CG method and (right) the proposed method.
Refer to caption: extracted/5832688/figures/figure_6.jpg

Effect of CG updates. Experimental results demonstrate the tangential CG updates in video space on the denoised manifold are key elements in solving spatio-temporal degradations.
Here, we compare the proposed method with a stand-alone CG method to demonstrate its impact within the solver.
We applied the same CG iterations as in the proposed method but excluded the diffusion updates. As shown in Fig. 6, while the stand-alone CG method nearly solves the video inverse problem, it leaves residual artifacts, as seen in the first row, or fails to fully resolve spatial degradation, as shown in the second row.
In contrast, the proposed method generates natural and fully resolved frames.
This indicates that the diffusion update in the proposed method refines the unnatural aspects of the CG updates.

Effect of batch-consistent sampling.
Fig. 8 illustrates the inter-batch difference within the denoised manifold $\mathcal{M}$ during the reverse diffusion process. The blue plot shows results from our full method, while the green and orange plots represent results without stochasticity control and with gradient descent (GD) updates instead of conjugate gradient (CG) updates, respectively. Notably, GD converges more slowly than CG.
Our method consistently achieves low inter-batch difference (i.e., high inter-batch similarity), ensuring batch-consistent reconstruction and precise reconstructions. In contrast, the absence of stochasticity control or the use of GD updates results in higher difference (i.e., lower similarity), leading to less consistent sampling. The intermediate samples $\hat{\bm{X}}_{t}$ in Fig. 8 and reconstruction results in Table 8 further confirm that our method outperforms the others in producing batch-consistent results.
Further experimental results and ablation studies are illustrated in Appendix B.

Figure: Figure 7: The inter-batch difference within the denoised manifold $\mathcal{M}$, quantified as $\sum_{i=1}^{N-1}\|\hat{\bm{X}}_{t}[i+1]-\hat{\bm{X}}_{t}[i]\|/(N-1)$, throughout the reverse diffusion sampling process. \raisebox{-.9pt} {1}⃝ indicates stochasticity control and \raisebox{-.9pt} {2}⃝ indicates using CG updates within the denoised manifold $\mathcal{M}$.
Refer to caption: extracted/5832688/figures/figure_7.jpg

## 5 Conclusion

In this work, we introduce an innovative video inverse problem solver that utilizes only image diffusion models. Our method leverages the time dimension of video as the batch dimension in image diffusion models, integrating video inverse optimization within the Tweedie denoised manifold. We combine batch-consistent sampling with video inverse optimization at each reverse diffusion step, resulting in a novel and efficient solution for video inverse problems. Extensive experiments on temporal and spatio-temporal degradations demonstrate that the proposed method achieves superior quality while being faster than previous DIS methods, even reaching speeds exceeding 1 FPS.

## Appendix A Experimental details

### A.1 Implementation of Degradations

For spatio-temporal degradations, we applied temporal degradation followed by spatial degradation sequentially.
We utilize spatial degradation operations for super-resolution, inpainting, and deblurring as specified in the official implementation from Wang et al. () and Chung et al. (). For super-resolution, we employ 4$\times$ average pooling as the forward operator ${\mathcal{A}}$. For inpainting, we use a random mask to eliminate half of the pixels for both the forward operator ${\mathcal{A}}$. In deblurring, we apply a Gaussian blur with a standard deviation ($\sigma$) of 2.0 and a kernel width of 13 as the forward operator ${\mathcal{A}}$.

### A.2 Data preprocessing details

We conducted every experiment using train/val sets of DAVIS 2017 dataset (Perazzi et al., ; Pont-Tuset et al., ). 480p resolution dataset has a spatial resolution of 480$\times$640. Therefore, to avoid spatial distortion, the frames were first center cropped to 480$\times$480, then resized to a resolution of 256$\times$256. The resizing was performed using the ‘resize’ function from the ‘cv2’ library. After that, all videos were normalized to the range [0, 1].
In the temporal dimension, the video was segmented into chunks of 16 frames starting from the first frame. Any remaining frames that did not form a complete set of 16 were dropped. Through this process, a total of 338 video samples were obtained. The detailed data preprocessing code and the preprocessed Numpy files have all been open-sourced.

### A.3 Comparative methods

DiffusionMBIR (Chung et al., ).
For DiffusionMBIR,
we use the same pre-trained image diffusion model (Dhariwal & Nichol, ) with 1000 NFE sampling.
The optimal $\rho$ and $\lambda$ values are obtained
through grid search within the ranges [0.001, 10] and [0.0001, 1], respectively. The values are set to ($\rho$, $\lambda$) = (0.1, 0.001) for temporal degradation, and ($\rho$, $\lambda$) = (0.01, 0.01) for spatio-temporal degradation.

DPS (Chung et al., ).
For DPS,
we use the same pre-trained image diffusion model (Dhariwal & Nichol, ) with 1000 NFE sampling.
The optimal step size $\zeta$ is obtained through grid search within the range [0.01, 100].
The value is set to $\zeta$ = 30 for both temporal degradation and spatio-temporal degradation.
Memory issues exist when performing DPS sampling more than 5 batch sizes in NVIDIA GeForce RTX 4090 GPU with VRAM 24GB. Therefore, we divide 16-frame videos into 4-frame videos and use them for all DPS experiments.

ADMM-TV. Following the protocol of Chung et al. (), we optimize the following objective

$$ $\displaystyle{\bm{X}}^{\ast}=\underset{{\bm{X}}}{\mathrm{argmin}}\frac{1}{2}\| {\mathcal{A}}{\bm{X}}-{\bm{Y}}\|^{2}_{2}+\lambda\|{\bm{D}}{\bm{X}}\|_{1}$ (22) $$

where ${\bm{D}}=[{\bm{D}}_{t},{\bm{D}}_{h},{\bm{D}}_{w}]$, which corresponds to the classical TV. $t$, $h$, and $w$ represent temporal, height, and width directions, respectively. The outer iterations of ADMM are solved with 30 iterations and the inner iterations of CG are solved with 20 iterations, which are identical settings to Chung et al. (). We perform a grid search to find the optical parameter values that produce the most visually pleasing solution. The parameter is set to ($\rho$, $\lambda$) = (1, 0.001). We set initial ${\bm{X}}$ as zeros.

## Appendix B Further Experimental results

### B.1 VRAM-efficient Sampling

The proposed method is VRAM-efficient, treating video frames as batches in the image diffusion model for sampling. As shown in Table 3, the method can reconstruct an 8-frame video at 256x256 resolution using less than 11GB of VRAM, which is feasible on GPUs like the GTX 1080Ti or RTX 2080Ti (11GB VRAM). With a single RTX 4090 GPU (24GB VRAM), it can reconstruct a 32-frame video at the same resolution.

Table: Table 3: Our VRAM usage for 256$\times$256 video.

### B.2 Ablation study of stochasticity

Experimental results show that synchronizing stochastic noise along batch direction enables batch-consistent reconstruction, offering an effective solution for video inverse problems.
While it is theoretically possible to achieve batch-consistent sampling with $\eta$ set to 0 (by eliminating stochastic noise), our empirical findings, as shown in Table. 4, indicate that incorporating stochastic noise is beneficial for video reconstruction, particularly in cases involving spatio-temporal degradations.
Consequently, in our experiments, the optimal $\eta$ value was determined through a grid search.

Table: Table 4: Ablation study on the selection of $\eta$ for spatio-temporal degradation ($\times$4 SR). Bold indicates the best and underline indicates the second best results.

### B.3 Detailed visualizations of experimental results

Figure: Figure 9: Ablation study results showing eight consecutive Tweedie denoised frames at different diffusion timesteps: the first row displays the 1${}^{\text{st}}$ of 20 DDIM sampling steps, the second row displays the 5${}^{\text{th}}$ step, and the third row displays the 10${}^{\text{th}}$ step.
Refer to caption: extracted/5832688/figures/appendix_1.jpg

Figure: Figure 10: Detailed qualitative comparison in temporal degradation using a uniform PSF with $k$=7 on the DAVIS dataset, shown with a 2-frame skip.
Refer to caption: extracted/5832688/figures/appendix_2.jpg

Figure: Figure 11: Detailed qualitative comparison in temporal degradation using a uniform PSF with $k$=13 on the DAVIS dataset, shown with a 2-frame skip.
Refer to caption: extracted/5832688/figures/appendix_3.jpg

Figure: Figure 12: Detailed qualitative comparison in temporal degradation using a Gaussian PSF with $\sigma$=1 on the DAVIS dataset, shown with a 2-frame skip.
Refer to caption: extracted/5832688/figures/appendix_4.jpg

Figure: Figure 13: Detailed qualitative comparison in spatio-temporal degradation, including the spatial deblurring ($\sigma$=2.0) task on the DAVIS dataset, shown with a 2-frame skip.
Refer to caption: extracted/5832688/figures/appendix_5.jpg

Figure: Figure 14: Detailed qualitative comparison in spatio-temporal degradation, including the spatial super-resolution ($\times$ 4) task on the DAVIS dataset, shown with a 2-frame skip.
Refer to caption: extracted/5832688/figures/appendix_6.jpg

Figure: Figure 15: Detailed qualitative comparison in spatio-temporal degradation, including the spatial inpainting ($r$=0.5) task on the DAVIS dataset, shown with a 2-frame skip.
Refer to caption: extracted/5832688/figures/appendix_7.jpg

Figure: Figure 16: Additional reconstruction results for temporal degradations: (top) uniform PSF with $k$=7, (bottom) uniform PSF with $k$=13.
Refer to caption: extracted/5832688/figures/appendix_8.jpg

Figure: Figure 17: Additional reconstruction results for spatio-temporal degradations. The spatial degradations are: (top) deblurring ($\sigma$=2.0), (mid) super-resolution ($\times$ 4), and (bottom) inpainting ($r$=0.5).
Refer to caption: extracted/5832688/figures/appendix_9.jpg

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How does the model perform on high frame rate video datasets? Could it be adapted to conduct video frame interpolation task?
soundness: 3
strengths: 1. This paper offers an innovative approach to video inverse problems by leveraging pre-trained image diffusion models for video tasks, eliminating the need for computationally intensive video diffusion model training.
2. The method is computationally efficient, with VRAM savings that make it feasible for deployment in lower-resource environments.
summary: This paper proposes a video inverse problem solver using pre-trained image diffusion models. Specifically, the method treats the time dimension of a video as the batch dimension of image diffusion modles and solves spatial-temporal optimization problems. Additionally, a batch-consistent sampling strategy is developed to ensure temporal consistency. Experimental results demonstrate the model’s effectiveness in handling various spatio-temporal degradations with significant speed improvements over existing methods.
weaknesses: 1. The paper lacks comparison with recent state-of-the-art video processing methods beyond diffusion-based approaches, which would provide a more comprehensive benchmark.
2. The evaluation is limited to the DAVIS dataset, potentially restricting insights into the model’s performance on a broader range of video types and characteristics, such as high-frame-rate and highly dynamic videos.
3. The degradations used in the experiments are synthesized through point spread functions (PSF), which may not fully capture real-world scenarios. Further testing on established video restoration datasets, such as GoPro for deblurring and other widely-used datasets for denoising and decompression, would help more thoroughly assess the model’s performance across diverse degradation types.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Can the authors provide specific examples of failure cases where your proposed method does not perform as expected? 
Additionally, it is well-known that metric evaluations, even subjective ones, may not accurately reflect people's actual viewing experiences. Can the authors provide a user study to further compare the different methods?
soundness: 3
strengths: 1. The paper utilizes image diffusion models to reduce the need for extensive video model training, leading to faster processing times.

2. The paper proposes a batch-consistent sampling strategy that ensures coherent frame generation, enhancing the visual quality of reconstructed videos.

3. The paper conducts experiments on various video inverse problems, such as deblurring and super-resolution, with improved reconstruction accuracy.
summary: This paper introduces a method to solve video inverse problems using image diffusion models instead of complex video models. By treating video frames as image batches, the authors achieve temporal consistency across frames through a batch-consistent sampling strategy. They also apply a multi-step optimization to improve accuracy, allowing for efficient and coherent video reconstruction without needing specialized video model training.
weaknesses: 1. The ablation experiments in this paper could be more comprehensive. For instance, it remains unclear whether the choice of different image pre-trained models affects the final results and whether the proposed algorithm yields consistent conclusions across various pre-trained models.

2. The paper lacks testing on more complex real-world datasets, such as videos with low bitrate compression or older films.

3. The image quality in the manuscript is low, please provide a high-quality version.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Why is the sampling approach different between the proposed method and other methods (e.g., 20 steps for DDIM vs. 1000 steps for NFE)? Would baseline methods with 20-step DDIM show a notable performance drop?
soundness: 3
strengths: 1. The presentation is clear and intuitive, with well-defined formulations and symbols.
2. The proposed method is efficient and effective for video inverse problems. Experimental settings are clearly explained, and the results validate the effectiveness of the sampling strategy in improving temporal coherence.
3. The ablation study is comprehensive and provides a clear understanding of the role of each component.
summary: This paper tackles video inverse problems by leveraging pretrained image diffusion models. The authors introduce a batch-consistent sampling scheme that maintains temporal consistency while introducing temporal diversity from the conditioning steps. Experimental results demonstrate the effectiveness of this approach in preserving quality across frames.
weaknesses: Incremental Novelty: The main contribution of this paper lies in addressing the batch consistency problem by applying a 2D pretrained diffusion model. The paper highly dependent on established techniques for inverse problems, including Tweedie denoising and multi-step conjugate gradient (CG) for frame-dependent perturbation. While the results are impressive, the technical contribution appears relatively incremental.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Based on the weaknesses above I have the following questions:
1) Is your method based on complete knowledge of the degradation operator?
2) If yes to 1) could you provide insights on how well your proposed video prior would work in the blind restoration case?
3) If your proposed video prior has other applications other than in the more practical blind restoration case, could you present these applications/motivations?
4) Could you clarify how your method would compare to prior blind video restoration work and why it is a better framework?
soundness: 3
strengths: The presentation is clear and self-contained. The proposed method is original as far as I know in the context of video restoration. The main originality is in the use of the same additional noise across all the frames at each step of the inverse diffusion process. This change combined with the data term constraint (and the use of conjugate gradients from prior work) seems to be sufficient to restore videos and yield sota performance on different video restoration tasks.
summary: The authors introduce a method to restore videos by using an pretrained image-based diffusion model.
The videos are degraded by spatio-temporal blurring, downsampling and missing data, and as far as I understand the degradation model A(X) is assumed known. The focus of the paper is on the prior on X obtained via a pre-trained diffusion model. However, in the case of videos one would need a corresponding pre-trained video diffusion model. Such models are today available, albeit very challenging to train due to the required large model, data dimensionality and compute resources. The authors suggest to use a pre-trained image diffusion model, which is much easier to train, and propose a method to adapt it to videos. The essential idea is to use the same noise used during the diffusion steps across all frames so that their denoising is more consistent. The authors show that this simple change combined with the gradients to match the data term (specifically, the conjugate gradients -- which are inspired by prior work) is sufficient to obtain temporally consistent restored videos.
weaknesses: This method has been demonstrated for the case where the degradation operator is given, ie, we are
working in the non-blind video restoration case.  If I misunderstood this point, please clarify this in the rebuttal.
I am assuming that A(X) is fully known and used to compute the conjugate gradients.
My main critique is that in practice it is very difficult to have access to such operator in the case of real videos. One could argue that having these operators is at the same level of complexity of solving the restoration task. Typically, one argues that focusing on the prior term by working in the non-blind case of the restoration task is still quite useful. However, this holds only if one can show that the prior could be 
used effectively also in the blind case. My main concern is that this may not hold in the proposed method.

The proposed novelty is that one should use the same additional noise samples for all the restored video frames. I wonder if this would work also in the blind case, ie, when also one needs to simultaneously recover the degradation operator. Basically, if the proposed method would only work in the non-blind case, then I would say that this is not a very significant contribution. If instead we could know that the method would also hold in the blind case, then it would be more significant (ie, the implicit temporal constraint through the same-noise sampling would be strong enough to work in the general case).

Intuitively, my concern is that the noise sampling choice implicitly guides the frame restorations to be very similar to each other, but does not provide any hint on how the temporal dynamics should be. In fact, an image diffusion model cannot impose this type of constraint. The component that imposes some temporal constraint seems to be just the degradation operator. So if this operator is unknown/to be estimated, I wonder how well all the rest would work.

In the literature there is quite a bit of past work on dealing with unknown degradation operators (the so-called _blind_ case). These methods differ from the one proposed here, as they typically use a dataset to train a model in a supervised fashion to map a blurry/degraded video to a sharp one. The advantage is that they learn to generalise to new degradation operators at test time. It would be important to understand how this approach compares to such methods and to highlight their pros and cons.
I think it would be more important to have such comparisons than to compare to even older ADMM methods.
If needed I can provide a few references on such recent blind video restoration methods, but it is straightforward to look them up.

Meta-review: This paper addresses video inverse problems, or video restoration tasks, using a pretrained image-based diffusion model. Reviewers find the proposed extension to the spatio-temporal domain novel, and the experimental results confirm its effectiveness and efficiency. During the discussion stage, the authors addressed key concerns raised by the reviewers, including extending the approach to blind restoration and conducting more extensive evaluations on real-world datasets. The reviewers are largely satisfied with the authors’ responses and unanimously agree that this paper surpasses the acceptance threshold. The AC found no strong reasons to disagree and recommends accepting this paper at ICLR.

Decision: Accept (Poster)