Solving Video Inverse Problems Using Image Diffusion Models
Authors: Taesung Kwon
Jong Chul Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Image diffusion models
Video inverse problems
Batch-consistent sampling
Primary area: generative models
TL;DR: We propose a video inverse problem solver using an image diffusion model with a batch-consistent sampling strategy.

Abstract: Recently, diffusion model-based inverse problem solvers (DIS) have emerged as state-of-the-art approaches for addressing inverse problems, including image super-resolution, deblurring, inpainting, etc. 
However, their application to video inverse problems arising from spatio-temporal degradation remains largely unexplored due to the challenges in training video diffusion models.
To address this issue, here we introduce an innovative video inverse solver that leverages only image diffusion models.
Specifically, by
drawing inspiration from the success of the recent decomposed diffusion sampler (DDS), 
our method treats the time dimension of a video as the batch dimension of image diffusion models and solves spatio-temporal optimization problems within denoised spatio-temporal batches derived from each image diffusion model.
Moreover, we introduce a batch-consistent diffusion sampling strategy that encourages consistency across batches by synchronizing the stochastic noise components in image diffusion models. 
Our approach synergistically combines batch-consistent sampling with simultaneous optimization of denoised spatio-temporal batches at each reverse diffusion step, resulting in a novel and efficient diffusion sampling strategy for video inverse problems.
Experimental results demonstrate that our method effectively addresses various spatio-temporal degradations in video inverse problems, achieving state-of-the-art reconstructions.
Project page: https://svi-diffusion.github.io/

Published as a conference paper at ICLR 2025
SOLVING VIDEO INVERSE PROBLEMS USING IMAGE DIFFUSION MODELS
Taesung Kwon', Jong Chul Ye?
1 Dept. of Bio & Brain Engincering, KAIST (star kwon, jong aeasstetttaaaeee..
2 Kim Jac Chul Graduate School of AI, KAIST
(o) Temporal degradation
Temporal degradation deblur
Mesurmment
Mesoumment
Reconsmaaton resutts
Reconctruciion realts
(c) Temporal degradation Supar-recolution
(d) Temporal degradation Inpainting (random)
Mossuroment
Mcssurement
Recomotradion
Recornbnccton
Figure 1: Representative video reconstruction results for (a) Temporal degradation, (b) Temporal degradation Deblurring combination, (c) Temporal degradation + Super-resolution combination, and (d) Temporal degradation Inpainting combination.
ABSTRACT
Recently, diffusion model-based inverse problem solvers (DIS) have emerged as state-of-the-art approaches for addressing inverse problems, including image super-resolution, deblurring, inpainting, etc. However, their application to video inverse problems arising from spatio-temporal degradation remains largely explored due the challenges training video diffusion models. To address this issue, here we introduce an innovative video inverse solver that leverages only image diffusion models. Specifically, by drawing inspiration from the suc- cess of the recent decomposed diffusion sampler (DDS), our method treats the time dimension of video the batch dimension of image diffusion mod- els and solves spatio-temporal optimization problems within denoised spatio- temporal batches derived from each image diffusion model. Moreover, we in- troduce a batch-consistent diffusion sampling strategy that encourages consis- tency across batches by synchronizing the stochastic noise components in im- age diffusion models. Our approach synergistically combines batch-consistent sampling with simultancous optimization of denoised spatio-temporal batches at each reverse diffusion step. resulting in a novel and efficient diffusion sam- pling strategy for video inverse problems. Experimental results demonstrate that method effectively addresses various spatio-temporal degradations in

---

Published as a conference paper at ICLR 2025
video inverse problems, achieving state-of-the- -hr- reconstructions. Project page: hc.pps://svd-diEEuussooo... giw.iiin..ttuu.oooo.
INTRODUCTION
Diffusion models (Ho et al. 2020 Song et al. 2020) represent the state-of-the-art generative model- ing by learning the underlying data distribution pl x) to produce realistic and coherent data samples from the learned distribution pe (z)). In the context of Bayesian inference, the parameterized prior distribution pe (x) can be disentangled from the likelihood p(y)æ)) which denotes the probabil- ity of observing given æ. This seperation facilitates the derivation of the posterior distribution po(y) & po (z)))p()))z)
Diffusion model- basedl-based inverse problem solvers (DIS) Kawar et al.
2022 Chung et al. 2022a
Song et al. 2023 Wang et al. 2023 Chung et al., 2024 leverage this property, enabling the uncon- ditional diffusion models to solve wide range of inverse problems. They achieve this by conditional sampling from the posterior distribution ps (dly) effectively integrating information from both the forward physics model and the measurement y. This approach allows for sophisticated and precise solutions to complex inverse problems, introducing the power and flexibility of diffusion models in practical applications.
Despite extensive DIS research on a wide range of image inverse problems such as super-rexolution, colorization, inpainting, compressed sensing, deblurring, and (Jalal et al,, 2021 Kawar al.
2022 Chung et al., 2022a [Song et al., 2023: Wang et al., 2023 Chung et al. 2024). the appli- video inverse problems, particularly those involving spatio-temporal
cation of these approaches degradation, has received relatively less attention. Specifically, in time-varying data acquisition systems, various forms of motion blur often arise due to the camera or object motions (Potmesil & Chakravarty 1983), which can be modeled as a temporal PSF convolution of motion dynam- ics. These are often associated with spatial degradation caused by noise, camera defocus, and other factors. Specifically, the spatio-tempe temioral degradation process can be formulated as:
Y = A(X) + W
with
X X= [x[]]]
x[N]],
Y Y- [1
y[V],, W=
w [[]]], (2)
where xfn]. y [n] and win] denote the n-th frame ground-truth image, measurement, and additive noise, respectively; N is the number of temporal frames, and A refers the operator that describes the spatio-temporal degradation process. The spatio-temporal degradation introduces complexities that image diffusion priors cannot fully capture, image diffusion priors are primarily designed to handle spatial features rather than temporal dynamics. Employing video diffusion models (Ho et al.) 2022) could address these issues, but poses significant implementation challenges for video inverse problems, due to the difficulty of training video diffusion models for various applications.
Contrary the common belief that pre-trained video diffusion model is necessary for solving video inverse problems, here we propose radically different method that addresses video inverse problems using only image diffusion models. Inspired by the success of the decomposed diffusion sampler (DDS) (Chung et al. [2024). which simplifies DIS by formulating it as a Krylov subspace- based optimization problem for denoised images via Tweedie's ' formula at each reverse sampling step, treat the time dimension of video the batch dimension of image diffusion models and solve spatio-temporal optimization problems using the batch of denoised temporal frames from im- age diffusion models. However, treating each frame of the video as a separate sample in the batch dimension can lead to inconsistencies between temporal frames. To mitigate this, we introduce the batch-consistent sampling strategy that controls the stochastic directional component (e.g., initial noise or additive noise) of each image diffusion model during the reverse sampling process, en- couraging the temporal consistency along the batch dimension. By synergistically combining batch- consistent sampling with the simultaneous optimization of the spatio-temporal denoised batch, our approach effectively addresses a range of spatio-temporal inverse problems, including spatial de- blurring, ruseurres,lution, and inpainting. Our contribution can be summarized as follows.
We introduce innovative video inverse problem solver using pre-trained image diffusion models by solving spatio-temporal optimization problems within the batch of denoised frames.
2

---

Published as a conference paper at ICLR 2025
We develop batch-consistent sampling strategy to ensure temporal consistency by syn- chronizing stochastic noise components in image diffusion models.
Extensive experiments confirm that our method achieves state-of-the-art performance on various video inverse problems and can also be extended to blind restoration problems.
BACKGROUND
Diffusion models. Diffusion models (Ho al. 2020) attempt model the data distribution Pdata (a)) based on a latent variable model
T Pg (
/
po( =
po (a daar,
where po(oor)) == (o
II
(3)
where the 21:7 are noisy latent variables defined by the Markov chain with Gaussian transitions
( = (Vx: B, (1-))))), g(>,|<u)) = (ZzlVvaa (1 =)))).
(4)
Here, the noise schedule Be is an increasing sequence of t. with ār == I as a; == Bi.
Training of diffusion models amounts training multi-noise level residual denoiser:
[IE (zx) e1
min E Bgg (me (mr) an pman-pau e-
(5)
Then, sampling from 3 can implemented by ancestral sampling, which iteratively performs
at
(t) (xx)) + B,e
æt
Eg- ai
(6)
VI
where 3. = 1- Bg and refers to the optimized parameter from Eq. On the other hand, T-
DDIM (Song et al., 202T) accelerates the sampling based on Mar-Maroovian assumption. Specifi- cally, the sampling iteratively performs
+ /1 a.-
Ft-1
where
V ūi-1 112 33 (
(xx))
n/.66
2.
x [(1 Vāz
v1 o 10 (zi)
&
(8)
-1 ā--1
Here, L; is the denoised estimate of æ; that is derived from Tweedie's formula (Efron 2011 1). Ac- cordingly, DDIM sampling can be expressed as a two-step manifold transition: (i) the noisy sample æ, E M, transits clean manifold M by deterministic estimation using Tweedie's formula, (ii) subsequent transition from clean manifold to next noisy manifold M,- occurs by adding noise é.. which is composed of the deterministic noise c/22 (ax)) and the stochastic noise c.
Diffusion mosel-based inverse problem solvers. For a given loss function E(x) which often stems from the likelihood for measurement consistency, the goal of DIS is address the following opti- mization problem
min ((x) MEM
where M represents the clean data manifold sampled from unconditional distribution Po (xx). Conse- quently, it is essential to find a way that minimizes cost while also identifying the correct manifold.
Recently, Chung al. (2023a) proposed general technique called diffusion posterior sampling (DPS), where the updated estimate from the noisy sample E .M is constrained to stay on the same noisy manifold Me. This is achieved by computing the manifold constrained gradient
(MCG) (Chung et al. 2022b) noisy sample E MM. The resulting algorithm can be stated as follows:
at-1 Va- (+. 7t Va, ((),)) /1 ā
(10)
3

---

Published as a conference paper at ICLR 2025
where "/ 0 denotes the step size. Under the linear manifold assumption (Chung al. 2022b 2023a), this allows precise transition to MM-1- Unfortunately, the computation of MCG requires computationally expensive backpropagation and is often unstable.
In a subsequent work, Chung et al. (2024) shows that under the same linear manifold assumption in DPS, the one step update by ă; - V aa obviating the need for explicit computation of the MCG and leading to a simpler approximation:
{(tt) are guaranteed remain within linear subspace, thus
Lt-
V 1 ( 74 V. ((x))) + / 16-1
(11)
Furthermore, instead of using one-step gradient update, Chung et al. (2024) demonstrated that multi-step update using Krylov subspace methods, such as the conjugate gradient (CG) method, guarantees that the intermediate steps lie in the linear subspace. This approach improves the conver- gence of the optimization problem without incurring additional neural function evaluations (NFE). This method, often referred to decomposed diffusion sampling (DDS), bypasses the computa- tion of the MCG and improves the convergence speed, making it stable and suitable for large-scale medical imaging inverse problems (Chung et al. 2024).
VIDEO INVERSE SOLVER USING IMAGE DIFFUSION MODELS
3.1 PROBLEM FORMULATION
Using the forward model Eq. ① and the optimization framework in Eq. (9. the video inverse prob- lem can be formulated
min ((X) = A(X)1 XEM
(12)
where X denotes the spatio-temporal volume of the clean image composed of N temporal frames as defined in Eq. (2), and M represents the clean video manifold sampled from unconditional distribution Po(X (X). Then, naive application of the one-step gradient within the DDS framework can be formulated by
X, 0 (x
(Xx) + 1-1-0--1
(13)
where X, and & refer to Tweedie's formula and noise in the spatio-temporal volume, respectively, which are defined by
1238 (X,) + n1,E ane (x:)) ε, = v1 ē, -178117888E7
X. 1 (x, VI-VI-- (x.-vI- Vot,
(14)
01-06-1
Here, X, refers to the spatio-temporal volume at the f-th reverse diffusion step and E [I,, Although the formula Eq. 14 is direct extension of the image-domain coun-
terpart Eq. 8), the main technical challenge lies in training the video diffusion model e which is required for the formula Eq. [44. Specifically, the video diffusion model is trained by
min E x (X, X, (Xo) --), ||E (1E(((())
(X,) E13
(15)
which requires lcrge video training data and computational resources beyond the scale of train- ing image diffusion models. Therefore, the main research motivation is propose innovative method that can bypass the need for computationally extensive video diffusion models.
3.2 CONSISCONTITTNTT RECONSTRUCTION WITH DDS
Consider batch of 2D diffusion models along the temporal direction:
(t)
Xi) = Eg-
(1)
6g- (x, [N]]
(16)
where e represents an image diffusion model. Suppose that (() (X,) is used for Eq. (14). Since
unconditional reverse diffusion entirely determined by Eq. (4). the generated video is then fully
4

---

Published as a conference paper at ICLR 2025
Y == A(x)
Mr
:
M2
Me
Ma-1
M
Mr-1
M
M
M
(a) Betch-independent sampling
(b) Batch-consistent sampling
(c) Ours
Figure 2:
Geometric illustration of the sampling path evolution. (a) Batch-independent sampling
produces independent frames. (b) Batch-consistent sampling produces identical frames. (c) Batch-
consistent sampling combined with frame- crameddependent perturbation through multi-step CG generates distinct frame satisfying spatio-temporal data consistency.
controlled by the behavior of the image diffusion models. Thus, we investigate the limitations of using batch of image diffusion models compared to using video diffusion model and explore ways to mitigate these limitations.
Recall that for the reverse sampling of each image diffusion model, the stochastic transitions occur from two sources: (i) the initialization and (ii) re-noising. Accordingly, in batch-independent sam- pling, where each image diffusion model initialized with independent random noise and re-noised with independent additive noise, it is difficult to impose any temporal consistency in video gener- ation so that each generated temporal frame may represent different content from each other (see Fig. 2())). Conversely, in consiseent sampling, where each image diffusion model is initial- ized with the same noise and re-noised with the same additive noise, the generated frames from the unconditional diffusion model should be trivially reduced to identical images (see Fig. (()).. This dilemma is why separate video diffusion model training using Eq. (5) was considered necessary for effective video generation.
One of the most important contributions of this paper is demonstrating that the aforementioned dilemma can be readily mitigated in conditional diffusion sampling originated from inverse prob- lems. Specifically, inspired by the DDS formulation in Eq. (33). we propose a method that employs batch-consistent sampling scheme to ensure temporal consistency and introduces temporal diver- sity from the conditioning steps. More specifically, the denoised image for each frame is computed individually using Tweedie's 's formula via image diffusion models:
X
X: - /1 a (X.)
(17)
where we use the superscript to represent the batch-consistency and {(4 is a batch of image diffusion models defined by Eq. (6). Here, the image diffusion models are initialized with the same random noises to ensure temporal consistency. Subsequently, the denoised spatio-temporal batch is perturbed as whole by applying the l-step conjugate gradient (CG) to optimize the data consistency term from the spatio-temporal degradation. This can be formally represented by
X arg ||Y
||2
(18)
XXX
where 1C, denotes the I-dimensional Kyrlov subspace associated with the given inverse problem (Chung et al. 2024). The multistep CG can diversify each temporal frame according to the condition and achieve faster convergence than a single gradient step. The resulting solution ensures that the loss function from the spatio-temporal degradation process can be minimized with coherent but frame-by-frame distinct reconstructions. Finally, the reconstructed spatio-temporal volume from the CG renoised with additive noise as:
X = / Vor X1+ V1 11- E.
(19)
5

---

Published as a conference paper at ICLR 2025
J|Y A(X))"
(& N,A,,, Vinmo termcc
Image Video Transition
Figure 3: Sampling process in our video inverse problem solver. X2 is denoised to produce X using 2D Tweedie formula, then reshaped into video tensor. Multi-step CG the video space, satisfying Eq. (18), is applied to obtain X which is then reshaped back into an image batch. Finally, X is sampled by adding noise E..
where
ex 1 āx-1 - 112 E (1) (X,) + +
(20)
vI 11-a--1
Here, Ed denotes the additive random noise from N( I). In contrast E in Eq. (14. which composed of frame-independen random noises, we impose batch consistency by adding the same random noises to each temporal frame to ensure temporal consistency. In summary, the proposed batch-consistent sampling and frame-dependent perturbation through multistep CG ensure that the sampling trajectory of each frame, starting from the same noise initialization, gradually diverges data consistency. geometric illustration of the sampling path evolution is shown in Fig. 2[c). The detailed illustration of the
from each other during reverse sampling meet the apatio-temper
intermediate sampling process of our method is shown in Fig. 3 The pseudocode implementation is
given in Algorithm T. Furthermore, our method can be extended to blind inverse problems, with the
corresponding algorithm and experimental results provided in Appendix
B
Algorithm Video inverse problem solver using 2D diffusion models
) Require: _.., [o, [xx) A.A.Y. Y. 1: 8b 1),
Controlled stochasticity
2: for = :
3: x [x VI - ar 200 (X:) /vāc
> Tweedie denoising
4: X, + arg min x x K, Y A(X) 12
Imposing frame deme-deenddent data consistency
5: Ea - (V مم-1 10 11 (X.) + m),E)) /V- ãi-1
D. Controlled stochasticity
6: X. lbt 12: VI - E
> Remoising
7: end for
8: Xo
[X1
(1) x13)/aa1
9:
X
EXPERIMENTS
In this section, conduct thorough comparison studies demonstrate the efficacy of the proposed method addressing spatio-tempord. degradations. Specifically, we consider two types of loss functions for video inverse problems:
((X)
=
(x) = |Y =
(21)
where the first loss is from Eq. 12 and TV(X) denotes the total variation loss along the temporal direction.
Then, classical optimization methods used as the baselines for comparison to minimize each loss function. Specifically, the stand-alone Conjugate Gradient (CG) method is employed to min- imize f(X), while the Alternating Direction Method of Multipliers (ADMM) is used to minimize

---

Published as a conference paper at ICLR 2025
Weasurement
ADMM-TV
DPS (1003)
DiffusioMMBR (1000)
Ohrrs (20)
Ground Truth
Uniform PSF (k 7]
Temporal A
MULO
MULO
MULO
Goussia PSF (a 1]
Temporal
Prer
NNed
Prev
Nast
Prey
Nest
Prwy
Niast
Pray
Herd
Nart
Figure 4: Qualitative evaluation of temporal degradation tasks. 1st row: temporal A with uniform PSF with kernel width k = 7. 2nd row: temporal A with Gaussian PSF with o=1. Red and blue boxes indicate the enlarged views of the previous and next frames, respectively.
Uniform PSF (A =
7)
Uniform PSF (F
13)
Gausiian PSF (o 1.00
Method
Time (s)
PSNR
SSIM
1PIPS
FVD.
PSNR
SSIM
1.PIPS
FVD
PSNR 1
551M
1.PIPS
FVD
Churs (20)
12
43,16
0.992
0.004
0,008
39.69
0.984
0.008
0.035
34.25
0.9999
0.829
0.068
DiflusiemMBIR (1000)
29:13
0,865
0.096
0.430
26.15
0.794
0.157
0.836
29.29
0.875
0.322
DPS (2000)
1244
33.42
0.914
0.071
12,225
20.61
0.666
0,30
2005
12:21
0.610
0.222
2.210
ADMM-TV
24.46
0.744
0.345
1.297
23.57
0.697
0.904
1.500
26.76
0.836
0.155
0.656
Table 1: Quantitative evaluation of temporal degradation tasks on the DAVIS dataset. Bold indicates the best results. FVD is displayed scaled by 10-3 for easy comparison.
Crv (X). Additionally, diffusion-bassee methods utilized as baselines to minimize the loss func- tions in Eq. (2I). Specifically, DPS (Chung et al. 2022a) is used to minimize ((X). However, instead of relying on 3D) diffusion models, we use 2D image diffusion models, similar our pro- posed methods, to ensure that backpropagation for MCG computation can be performed through 21) diffusion models. Second, we employ DiffusionMBII (Chung et al. (023b) to minimize (Tv(X). also using 2D image diffusion models. Unlike the original DiflusionMBIRR, which applies TV along the z-direction, we apply TV along the temporal direction.
To test various data acquisition systems, which is represented as PSF convolution along temporal dimen- sion (Potmesil & Chakravarty] 1983). We select three types of PSFs: (i) uniform PSF with widths of 7, (ii) uniform PSF with widths of T3, and (iii) Gaussian PSF with a standard deviation of 1 1.0. Each kernel is convolved along the temporal dimension with the ground truth video to produce the mea- surements. Note that convolving uniform PSF with widths 7 and 13 correspond to averaging 7 and 13 frames, respectively. Furthermore, we combined temporal degradation and various spatial degra- dations to demonstrate various combinations of degradations, fix temporal degradation as convolving uniform PSF with width of 7 and add various spatial degradations to the video. These spatial degradations include (i) deblurring using a Gaussian blur kernel with standard deviation of 2.0, (ii) super-resolution through a 4x average pooling, and (iii) inpainting with random masking at a ratio r of 0.5 (For specific implementation details of degradations, see Appendix A).
degradations, we select the temporal degradation in time-varying
degradations. For ceatio-temporal
We conduct our experiments on the DAVIS dataset (Perazzi et al. 2016 Pont- et al. 2017 which includes wide variety of videos covering multiple scenarios. The pre-trained unconditional 256x256 image diffusion model from ADM (Dhariwal & Nichol, 202T) is used directly without fine-tuning and additional networks. All videos were normalized to the range [0. 1] and split into 16-frame samples of size 256. A total of 338 video samples were used for evaluation. More preprocessing details are described the Appendix(A A
For quantitative comparison, we focus on the following two widely used standard metrics: peak signal-to-noise-ratio (PSNR) and structural similarity index (SSIM) (Wang et al. 2004) with further evaluations with two perceptual metrics Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al. 2018) and Fréchet Video Distance (FVD) (Unterthiner et al., 2019). FVD results are displayed scaled by 10-3 for easy comparison. For all proposed methods, we employ I = 5, = 0.15 for 20 NFE in temporal degradation tasks, and = 5, 17 = 0.8 for 100 NEE in spatio- temporal degradation tasks unless specified otherwise.
7

---

Published as a conference paper at ICLR 2025
Measurement
ADMM-TV
DPS (1000)
MnfussioMBBR (1006)
Ours (10D)
Ground Truth
(T=0] Buwngan
(olodue--ogg oooddee--odds ogeds
PX
055 Co
Figure 5: Qualitative evaluation of spatio-tempon degradation tasks. Each spatio-temporal degra- dation combined with various spatial degradation tasks. 1st row: Deblurring (a 2.0). row: SR (x 4). 3rd row: Inpainting (r 0.5).
Deblur (r 2.0)
Super-rcoolution (x4)
Inpainting (+ 0.5)
Method
Time
PSNR
SSIM
LPIPS
FVD
PSNR
SSIM
LPIPS
FVD
PSNR
SSIM
1.PIPS
FVD
(100)
27.27
DiflusiemMBIR (1000)
611
21.79
0.583
0.304
1,809
21.41
0.552
0.418
2.085
19:46
0.535
0.509
2.899
(1000)
18.19
21.39
0.532
1.722
0.817
ADMM-TV
2.4
22.76
0.638
0.462
1
22.09
0.592
0
1.739
22.53
0.63
0.336
1892
Table 2: Quantitative evaluation of spatio-temporal degradation tasks on the DAVIS dataset. Bold indicates the best results. FVD is displayed scaled by 10-3 for easy comparison.
4.1 RESULTS
We present the quantitative results of the temporal degradation tasks in Table [ The table shows that the proposed method outperforms the baseline methods by large margins in all metrics. The large margin improvements in FVD indicate that the proposed method successfully solves inverse problems with temporally consistent reconstruction. Fig. shows the qualitative reconstruction results for temporal degradations A. The proposed method restores much finer details compared to the baselines and demonstrates robustness across various temporal PSFs. contrast, as shown in Fig. while DPS performs well in reconstructing uniform PSFs with kernel width of 7, it fails to accurately reconstruct frame intensities as the kernel becomes wider or more complex as shown in the bottom figures, leading significant drops in Table I DiffusionMBIR ensures temporal consistency and performs well for static scenes, but it struggles with dynamic scenes in the video. In the same context, ADMM-TV produces unsatisfactory results for dynamic scenes.
The results of the spatio-temporal degradations are presented in Table 2 and Fig. 5 Even with additional spatial degradations, the proposed method consistently outperforms baseline methods. On the other hand, DPS often produces undesired details, as shown in Fig. 5 DiffusionMBIR fails to restore fine details in dynamic scenes. Specifically, restores the static mural painting but fails to capture the motion of the person. This is because TV regularizer often disrupts the restoration of dynamic scenes. In this context, our method ensures temporal consistency without the need for TV regularizer. Furthermore, thanks to the consistent performance even at low NFE, the proposed method achieves a dramatic 10x to 50x acceleration in reconstruction time. For handling temporal degradation with 20 NFE, the proposed diffusion model-based inverse problem solver can now achieve speeds exceeding I FPS.
the 3rd row Fig. 5 DiffusionMBIR
4.2 ABLATION STUDY
Effeet of CG updates. Experimental results demonstrate the tangential CG updates in video space on the denoised manifold are key elements in solving spatio-temports degradations. Here, we com- pare the proposed method with stand-alone CG method to demonstrate its impact within the solver. We applied the same CG iterations as in the proposed method but excluded the diffusion updates. As shown in Fig. 6 while the stand-alone CG method nearly solves the video inverse problem, it leaves
8

---

Published as a conference paper at ICLR 2025
Stand-alome CG
Ours
Goussian PSF (o 1)
conto-tomoorll.. Temporal Debbiring (0=2)
SAGA
SAR
SADA
Figure Reconstruction results of (left) stand-alone CG method and (right) the proposed method.
residual artifacts, as seen in the first row, or fails to fully resolve spatial degradation, as shown in the second row. In contrast, the proposed method generates natural and fully resolved frames. This indicates that the diffusion update in the proposed method refines the unnatural aspects of the CG updates.
Effeet of batch-consistent sampling. Fig. illustrates the inter-batch difference within the de- noised manifold M during the reverse diffusion process. The blue plot shows results from our full method, while the green and orange plots represent results without stochasticity control and with gradient descent (GD) updates instead of conjugate gradient (CG) updates, respectively. Notably, GD converges more slowly than CG. Our method consistently achieves low inter-batch difference (i.e.. high inter-batch similarity), ensuring batch-consistent reconstruction and precise reconstruc- tions. contrast, the absence of stochasticity control or the use of GD updates results in higher difference (i.e., lower similarity), leading to less consistent sampling. The intermediate samples X, in Fig. - and reconstruction results in Table 3 others producing batch-consistent results. Further experimental results and ablation studies are illustrated in Appendix C
further confirm that our method outperforms the
Method
PSNR
Full
39.69
wlo T
30.86
(eee Satch dfferree
w/o ( 2
23.22
Method
FVD
Full
0.035
wlo C
0.567
w/o C. (
1.275
X
00
x
Tin
Difialoo tim
Figure 7: The inter-batch difference within the denoised manifold M. quanti- Table 3: Recon-
fied as W [X X, 1), throughout the reverse diffusion struction results of sampling process. ! indicates stochasticity control and 2 indicates using CG the ablation study. updates within the denoised manifold M.
CONCLUSION
In this work, we introduce an innovative video inverse problem solver that utilizes only image dif- fusion models. Our method leverages the time dimension of video as the batch dimension in image diffusion models, integrating video inverse optimization within the Tweedie denoised manifold. We combine batch- catch-consistent sampling with video inverse optimization at each reverse diffusion step, resulting in novel and efficient solution for video inverse problems. Extensive experiments on temporal and spatio-temporal degradations demonstrate that the proposed method achieves superior quality while being faster than previous DIS methods, even reaching speeds exceeding FPS.

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Research Foundation of Korea under Grant RS-2024- 00336454 and by the Institute of Information & Communications Technology Planning & Evalu- ation (IITP) grant funded by the Korea government (MSIT) (RS-2019-1119007) Artificial Intelli- gence Graduate School Program, KAIST).
REFERENCES
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh Interna- tional Conference on Learning Representations, 2022a.
Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 25:25683-25666, 2022b.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In International Confer- on Learning Representations, 2023a. URL htttt::://ppereeeeeeeeee net http://wwwwwee/oooummo.. forum?id=
OnD9zGAGTOK
Hyungjin Chung, Dohoon Ryu, Michael T McCann, Mare L Klasky, and Jong Chul Ye. Solving 3d inverse problems using pre-trained 2d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22542-22551, 2023b.
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Decomposed diffusion sampler for accelerating large-scale inverse problems. In The Twelfth International Conference on Learning Representa- fiony, 2024. hnttss:tt///peetee
net wwt//fouuume id=DsEha
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in nearal information processing systems, 34:8780-8794, 2021.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- tion, 106(4966::6022-1614 2011.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey A Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In ICLR Workshop on Deep Generative Models for Highly Struc- tured Data, 2022.
Ajil Jalal, Marius Arvinte, Giannis Daras, Fric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems, 111441999544 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration
models. Advances in Neural Information Processing Systems, 35:23593-23606, 2022.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattemn recognition, pp. 3883-3891, 2017.
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Yorkine-Hornung A benchmark dataset and evaluation methodology for video object segmen- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp- 724-732, 2016.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint (Xiv:/704.0000 2017.
10

---

Published as a conference paper at ICLR 2025
Michael Potmesil and Indranil Chakravarty. Modeling motion blur in computer-generatee images. ACM SIGGRAPH Computer Graphics, 17(3):389-333 399, 1983.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021. URL httpps://eeenreveew..ccc.. net/
Forum?id-StllgaarCHi
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse-guided diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
URL URL/Iittpss//cppeeevveew net/ 1 _
Yang Song. Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar. Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020.
Zachary Teed and Jia Deng. Raft: Recurrent ppl-s field transforms for optical flow. Computer 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part II 16, pp. 402-419. Springer, 2020.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023. URL httpttttt///ppereeee
net wwt//eerurriii.. forum id=RR 1
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Chang-Han Yeh, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Ting-Hsuan Chen, Hau-Shiang Shiu, and Yu-Lun Liu. Diffir2vr-zero: Zero-shot video restoration with diffusion-based image restoration models. arXiv preprint arXiv:2407.01559, 2024.
Geunhyuk Youk, Jihyong Oh, and Munchurl Kim. Fma-net: Flow-guided dynamic filtering and iterative feature refinement with multi-attention for joint video super-resolution and deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 44-55,22 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
A. IMPLEMENTATION OF DEGRADATIONS
For spatio-temporal degradations, we applied temporal degradation followed by spatial degradation sequentially. We utilize spatial degradation operations for super-resolution, inpainting, and deblur- ring as specified in the official implementation from Wang et al. (2023) and Chung et al. (2022a)- For super-resolution, we employ 4x average pooling as the forward operator A. For inpainting, we use a random mask to eliminate half of the pixels for both the forward operator A. deblurring, we apply Gaussian blur with a standard deviation (or) of 2.0 and kernel width of 13 as the forward operator A.
A.2 DATA PREPROCESSING DETAILS
We conducted every experiment using train/val sets of DAVIS 2017 dataset (Perazzi et al. 2016
Pont- et al. 2017). 480p resolution dataset has spatial resolution of 640. Therefore, to avoid spatial distortion, the frames were first center cropped 480x480, then resized a resolution of 256 The resizing was performed using the 'resize" function from the 'cv2' library. After that, all videos were normalized to the range [0, 1]. In the temporal dimension, the video was segmented into chunks of 16 frames starting from the first frame. Any remaining frames that did not form a complete set of 16 were dropped. Through this process, a total of 338 video samples were obtained. The detailed data preprocessing code and the preprocessed Numpy files have all been open-sourced.
A.3 COMPARATIVE METHODS
DiflusionMBIR Chung al. 2023b). For DiffusionMBIR, we use the same pre-trained image diffusion model (Dhariwal & Nichol (2021) with 1000 NFE sampling. The optimal and À values are obtained through grid search within the ranges [0.001, 10] and [0.0001, 1], respectively. The values are set to (p, x) (0.1, 0.001) for temporal degradation, and (p. A) (0.01, 0.01) for spatio- temporal degradation.
DPS (Chung et al. 2022a). For DPS, we use the same pre-trained image diffusion model (Dhariwal & Nichol, 2021) with 1000 NFE sampling. The optimal step size Ç is obtained through grid search within the range [0.01, 100]. The value is set to 30 for both temporal degradation and spatio- temporal degradation. Memory issues exist when performing DPS sampling more than 5 batch sizes in NVIDIA GeForee RTX 4090 GPU with VRAM 24GB. Therefore, we divide 16-frame videos into 4- I-frame videos and use them for all DPS experiments.
ADMM-TV. Following the protocol of Chung et al. (2023b), optimize the following objective
X argmin
|AX 2
DX
(22)
where D [D., D;,, Dw), which corresponds to the classical TV. f, h, and w represent temporal, height, and width directions, respectively. The outer iterations of ADMM are solved with 30 iter- ations and the inner iterations of CG are solved with 20 iterations, which are identical settings to Chung al. (2023b). We perform grid search to find the optical parameter values that produce the most visually pleasing solution. The parameter is set to (p. A) = (1, 0.001). We set initial X as zeros.
12

---

Published as a conference paper at ICLR 2025
B EXTENSION TO BLIND INVERSE PROBLEMS
B.1 BLIND VIDEO DEBLURRING
Algorithm Ours (blind) Extension to blind video deblurring
Require: Eg-, T, [oe) So
1: Xmm + (x(X)
Di PSF estimation using pre-restoration module
2: be + arg argming, Y
Xm hrl
3: Xr + E² NO(11)
4:
ddo
5: x (x, vT m. (x:) lvoL
Stage with estimated
6:
X, arg x
X
7: E - (V1 a1-1 a1-1 8 (0) (Xi) + ndre)) /V1 ac-1
8: X,-1 + 1 v 12
9: end for
10: Xo + (X1 v1 6. (x1))\ v
11: ho arp min,, 1| No + ho
estimation using stage
resalt
12: XT + =E" .)
13: 22:2du
14: X x (xx- v1 2(1) (X+) /vā.
Stage with refined
15: 8, arg arg x + Y X Xoh,1
16:
E. (v 1-11 ā1-1 1f 8 11) (X,) mdre)) //- a
17: X.-1 v X, vI 1 :
18: end
19: Xo +- (X1 VI o (x.))/ooT
20:
return No
Our method can be extended to blind video inverse problems. the standard approach address blind deconvolution, alternating between PSF estimation and deconvolution is intuitive and effective. Since initial PSF estimation challenging, we first use a light-weight video deblurring module, DeepDeblur (Nah et al. 2017). for pre-reconstruction and estimate the initial PSF from it. Using this PSF, we perform Stage reconstruction using our method, then refine the PSF based on the resulting video. Finally, the refined PSF is used for the final (Stage 2) reconstruction. In summary, for blind deconvoltuion, our method can leverage a lightweight pre-restoration module to estimate the initial PSF and achieves the final reconstruction using the refined PSF. The detailed algorithm is given in Algorithm 2
The GoPro dataset consists of 240 fps videos captured using GoPro camera, with blur strengths created by averaging 7 to 13 consecutive frames (Nah et al. 2017). All experiments are conducted on the GoPro test dataset. In our method, Ours (blind) refers blind reconstruction applied to randomly selected blur strengths between 7 and 13, while Ours (known, k = 13) corresponds to reconstruction the maximum blur strength of 13 with known degradation. To highlight the effectiveness of our approach, we compared our results with the reconstructions obtained from the pre-restoration module using the GoPro test dataset (Nah et al. 2017). As shown in Fig. 18 and Table our method provides further refinements, resulting improved performance and yielding highly satisfactory results. Notably, in blind cases, inaccuracies the initial and second PSF estimations may result in suboptimal performance compared to cases with known temporal degradation.
Method (GoPr)
FVD.L
LPIPS 1
PSNR 1
SSIM
DeepDeblur
0.119
0.116
30.93
0.904
Ours (blind)
0.017
38.98
0.974
Ours (knowa, l==1)
0.024
0.012
38.05
0.981
Table 4: Quantitative evaluations on video deblurring using the GoPro test dataset (Nah et al. 2017 Bold indicates the best and underline indicates the second best results.
13

---

Published as a conference paper at ICLR 2025
GOPR0881_11, _1___1
6OPRO410_ _1__00
DeepDeblur
Ours (blind)
28392
Figure 8: Reconstruction results on the GoPro test dataset Nah et al. 2017). (Top) DeepDeblur Naa et al., 2017) and (bottom) the proposed extension. Enlarged views of the sample included for detailed comparison.
B.2 BLIND VIDEO SUPER-RENT RESOLUTION AND VIDEO FRAME INTERPOLATION
In blind video super-resolution, information about the degradation type can be inferred from the degraded measurement. For instance, a 64x64 measurement is provided as input to 256x256 restoration module, it is straightforward to estimate that the degradation corresponds to a 4x super- resolution (SR). Since the spatial resolution of the measurement can be directly determined, the corresponding SR process be applied based on this estimation, enabling simple implementation of blind video SR.
For video frame interpolation, flow estimation module like RAFT (Teed & Deng 2020 can be employed to generate warped estimations. Subsequently, our method can serve as an inpainting solver fill in the gaps within the warped estimations, leveraging the explicit temporal constraints provided by batch-consistent sampling. This adaptability may allow our method to effectively tackle video interpolation tasks.
14

---

Published as a conference paper at ICLR 2025
C FURTHER EXPERIMENTAL RESULTS AND DISCUSSIONS
C.I VRAM-EFFICIENT SAMPLING
The proposed method is VRAM-efficient, treating video frames as batches in the image diffusion model for sampling. As shown in Table the method can reconstruct 8-frame video at 256x256 resolution using less than I1GB of VRAM, which is feasible on GPUs like the GTX 1080Ti or RTX 2080Ti (IGGB VRAM). With single RTX 4090 GPU (24GB VRAM), it reconstruct
32-frame video at the same resolution.
Frame
VRAM (GB)
2.73
2
3,36
4
4.90
8
7.33
16
13.33
32
23.65
4090 (24 GB)
Table 5: Our VRAM usage for 256x256 video.
C.2 ABLATION STUDY OF STOCHASTICITY
Experimental results show that synchronizing stochastic noise along batch direction enables batch- consistent reconstruction, offering an effective solution for video inverse problems. While it is theoretically possible to achieve batch-consistent sampling with r) set to (by eliminating stochastic noise), our empirical findings, shown in Table [6] indicate that incorporating stochastic noise beneficial for video reconstruction, particularly in cases involving spatio-temporal degradations. Consequently, in our experiments, the optimal value was determined through grid search.
PSNR
SSIM
LPIPS
FVD
0.677
Table Ablation study on the selection of v) for spatio-temporal degradation (x4 SR). Bold indi- cates the best and underline indicates the second best results.
C.3 COMPARISON WITH ADDITIONAL VIDEO RESTORATION METHOD
To evaluate reconstruction performance in comparison the latest video restoration methods, we conducted additional experiments with the recently proposed DiffIR2VR (Yeh et al. 2024), which has shown superior video processing performance over both supervised video processing method (You et al. (2024) and diffusion-based method SDx4 upscaler (Rombach et al. 2022 While DI (Ych al. 2024) supports video super-resolution tasks, we conducted exper- ments on video super-resolution (x4) task. To ensure fair comparisons with identical resolutions, we used patch reconstruction. As shown Table 7 our method outperforms DiffIR2VR, achieving superior reconstruction performance.
Method (DAVIS)
PSNR
FVD
LPIPS
DiffRR2VR Yeh et al. 2024
30,51
0.212
0.061
Ours
32,88
0.166
0.089
Table 7: Quantitative evaluations on video super-resolution (x4) using the DAVIS dataset. Bold indicates the best results.
15

---

Published as a conference paper at ICLR 2025
C.4 TEST ON ADDITIONAL VIDEO DATASETS
To further evaluate the adaptability of our method across diverse datasets, we conducted additional experiments high-frame-rate dataset (collected from Pexels") For the high-frame-rate dataset from Pexels, we compared our method with DiffIRVVR (Yeh et al. (024) on video super-resolution (x4) task. As shown Table our method maintains superior performance even on high-frame- data.
Method (Pexels)
PSNR
FVD
LPIPS
DifiRRVVR Ych et al.
2024
31,31
0.301
0.956
Ours
33.79
0.205
0.104
Table Quantitative evaluations on video super-resolution (x4) using the high frame rate (Pexels) dataset. Bold indicates the best results.
C.5 HUMAN PERCEPTUAL STUDY
We conducted a perceptual human evaluation comparing our method with bascline methods used in the paper. Specifically, we collected total of 36 votes from computer vision researchers. Recon- struction results were displayed side-by-side, and researchers were asked to vote on the method that best addressed each of the following questions: (Q1) Which video has better reconstruction quality? (Q2) Which video has better temporal consistency? As shown in Table 9 our method outperformed the baseline methods in both aspects according to human perceptual evaluations.
Method (DAVIS)
Q1 (votes total votes) 1
(votes total votes)
ADMM-TV
0.056
0.056
DiffusiouMBIR
0
Ours
0.944
0.944
Table 9: Human perceptual study on various video inverse problems using the DAVIS dataset. Bold indicates the best results.
C.6 LIMITATIONS AND FUTURE WORKS
Our method employed the unconditional pixel-space diffuuson model (Dhariwal & Nichol 2021 which supports maximum resolution of 256x256. Consequently, the current approach is limited by this spatial resolution. Extending the framework latent diffusion models (Rombach et al. 2022) offers promising direction for improving both the supported resolution and reconstruction quality, thereby enabling broader applications. In scenarios with severe temporal degradation, such as video frame interpolation, our method may become less reliable. However, as discussed in Appendix B.2 our framework is flexible enough to incorporate additional modules to address these challenges. For utilize two-round sampling process, which results in a doubling of the
blind video deblurring, sampling time. Future research to reduce this additional sampling time could enhance the efficiency of blind inverse problem solvers.
httpp::/ wttp//wwwwpwwwwwpwww/e//pp/ com/
16

---

Published as a conference paper at ICLR 2025
C.7
DETAILED VISUALIZATIONS EXPERIMENTAL RESULTS
Full method
Sampling steps
No stochasticity control
Sampling steps
No stochasticity control updates
Sampling steps
Figure 9: Ablation study results showing eight consecutive Tweedie denoised frames at different diffusion timesteps: the first row displays the 1s of 20 DDIM sampling steps, the second row dis- plays the 5th step, and the third row displays the 10th step.
17

---

Published as a conference paper at ICLR 2025
Measurement
ADMM-TV
DPS
DiffusioMMBIR
Ours
Ground Truth
Figure 10: Detailed gualitative comparison in temporal degradation using uniform PSF with k=7 on the DAVIS dataset, shown with a 2-frame skip.
18

---

Published as a conference paper at ICLR 2025
Measurement
ADMM-TV
DPS
DiffusioMMBIR
Ours
Ground Truth
Figure 11: Detailed qualitative comparison in temporal degradation using uniform PSF with k=13 on the DAVIS dataset, shown with a 2-frame skip.
19

---

Published as a conference paper at ICLR 2025
Measurement
ADMM-TV
DPS
DiffusionMBIRR
Ours
Ground Truth
Figure 12: Detailed qualitative comparison in temporal degradation using Gaussian PSF with o=1 on the DAVIS dataset, shown with a 2- 2-frame skip.
20

---

Published as a conference paper at ICLR 2025
Measurement
ADMM-TV
DPS
DiffusionMBIR
Ours
Ground Truth
Figure 13: Detailed qualitative comparison in ppatio-tempore degradation, including the spatial
deblurring (o=2.0) task on the DAVIS dataset, shown with a 2-frame skip.
21

---

Published as a conference paper at ICLR 2025
Measurement
ADMM-TV
DPS
DiffusioNMBIR
Ours
Ground Truth
Figure 14: Detailed qualitative comparison in ppatio-tempont degradation, including the spatial super-resolution (x 4) task on the DAVIS dataset, shown with 2-frame skip.
22

---

Published as a conference paper at ICLR 2025
Measurement
ADMM-TV
DPS
DiffusionMBIR
Ours
Ground Truth
Figure 15: Detailed qualitative comparison in spatio-temports degradation, including the spatial
inpainting (r=0.5) task on the DAVIS dataset, shown with a 2-frame skip.
23

---

Published as a conference paper at ICLR 2025
Measurement
Ours
Ground Truth
Measurement
Ours
Ground Truth
Figure 16: Additional reconstruction results for temporal degradations: (top) uniform PSF with k=7,
(bottom) uniform PSF with k=13.
24

---

Published as a conference paper at ICLR 2025
Measurement
Ours
Ground Truth
Measurement
Ours
Ground Truth
Measurement
Ours
Ground Truth
Figure 17: Additional reconstruction results for apatio-temporal degradations. The spatial degrada-
tions are: (top) deblurring (a=2.0), (mid) super-resolution (x 4), and (bottom) inpainting (r=0.5).
25


Reviews:
[{"review_id": "V23nX4za2G", "reviewer": "Reviewer_Cj3w", "summary": "This paper proposes a video inverse problem solver using pre-trained image diffusion models. Specifically, the method treats the time dimension of a video as the batch dimension of image diffusion modles and solves spatial-temporal optimization problems. Additionally, a batch-consistent sampling strategy is developed to ensure temporal consistency. Experimental results demonstrate the model’s effectiveness in handling various spatio-temporal degradations with significant speed improvements over existing methods.", "questions": "1. How does the model perform on high frame rate video datasets? Could it be adapted to conduct video frame interpolation task?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper offers an innovative approach to video inverse problems by leveraging pre-trained image diffusion models for video tasks, eliminating the need for computationally intensive video diffusion model training.\n2. The method is computationally efficient, with VRAM savings that make it feasible for deployment in lower-resource environments.", "weaknesses": "1. The paper lacks comparison with recent state-of-the-art video processing methods beyond diffusion-based approaches, which would provide a more comprehensive benchmark.\n2. The evaluation is limited to the DAVIS dataset, potentially restricting insights into the model’s performance on a broader range of video types and characteristics, such as high-frame-rate and highly dynamic videos.\n3. The degradations used in the experiments are synthesized through point spread functions (PSF), which may not fully capture real-world scenarios. Further testing on established video restoration datasets, such as GoPro for deblurring and other widely-used datasets for denoising and decompression, would help more thoroughly assess the model’s performance across diverse degradation types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mRPcSej1QO", "reviewer": "Reviewer_imTd", "summary": "This paper introduces a method to solve video inverse problems using image diffusion models instead of complex video models. By treating video frames as image batches, the authors achieve temporal consistency across frames through a batch-consistent sampling strategy. They also apply a multi-step optimization to improve accuracy, allowing for efficient and coherent video reconstruction without needing specialized video model training.", "questions": "Can the authors provide specific examples of failure cases where your proposed method does not perform as expected? \nAdditionally, it is well-known that metric evaluations, even subjective ones, may not accurately reflect people's actual viewing experiences. Can the authors provide a user study to further compare the different methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper utilizes image diffusion models to reduce the need for extensive video model training, leading to faster processing times.\n\n2. The paper proposes a batch-consistent sampling strategy that ensures coherent frame generation, enhancing the visual quality of reconstructed videos.\n\n3. The paper conducts experiments on various video inverse problems, such as deblurring and super-resolution, with improved reconstruction accuracy.", "weaknesses": "1. The ablation experiments in this paper could be more comprehensive. For instance, it remains unclear whether the choice of different image pre-trained models affects the final results and whether the proposed algorithm yields consistent conclusions across various pre-trained models.\n\n2. The paper lacks testing on more complex real-world datasets, such as videos with low bitrate compression or older films.\n\n3. The image quality in the manuscript is low, please provide a high-quality version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5477Qh4Qxg", "reviewer": "Reviewer_cLVE", "summary": "This paper tackles video inverse problems by leveraging pretrained image diffusion models. The authors introduce a batch-consistent sampling scheme that maintains temporal consistency while introducing temporal diversity from the conditioning steps. Experimental results demonstrate the effectiveness of this approach in preserving quality across frames.", "questions": "Why is the sampling approach different between the proposed method and other methods (e.g., 20 steps for DDIM vs. 1000 steps for NFE)? Would baseline methods with 20-step DDIM show a notable performance drop?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is clear and intuitive, with well-defined formulations and symbols.\n2. The proposed method is efficient and effective for video inverse problems. Experimental settings are clearly explained, and the results validate the effectiveness of the sampling strategy in improving temporal coherence.\n3. The ablation study is comprehensive and provides a clear understanding of the role of each component.", "weaknesses": "Incremental Novelty: The main contribution of this paper lies in addressing the batch consistency problem by applying a 2D pretrained diffusion model. The paper highly dependent on established techniques for inverse problems, including Tweedie denoising and multi-step conjugate gradient (CG) for frame-dependent perturbation. While the results are impressive, the technical contribution appears relatively incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XJXPgpXS4Q", "reviewer": "Reviewer_v7hT", "summary": "The authors introduce a method to restore videos by using an pretrained image-based diffusion model.\nThe videos are degraded by spatio-temporal blurring, downsampling and missing data, and as far as I understand the degradation model A(X) is assumed known. The focus of the paper is on the prior on X obtained via a pre-trained diffusion model. However, in the case of videos one would need a corresponding pre-trained video diffusion model. Such models are today available, albeit very challenging to train due to the required large model, data dimensionality and compute resources. The authors suggest to use a pre-trained image diffusion model, which is much easier to train, and propose a method to adapt it to videos. The essential idea is to use the same noise used during the diffusion steps across all frames so that their denoising is more consistent. The authors show that this simple change combined with the gradients to match the data term (specifically, the conjugate gradients -- which are inspired by prior work) is sufficient to obtain temporally consistent restored videos.", "questions": "Based on the weaknesses above I have the following questions:\n1) Is your method based on complete knowledge of the degradation operator?\n2) If yes to 1) could you provide insights on how well your proposed video prior would work in the blind restoration case?\n3) If your proposed video prior has other applications other than in the more practical blind restoration case, could you present these applications/motivations?\n4) Could you clarify how your method would compare to prior blind video restoration work and why it is a better framework?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The presentation is clear and self-contained. The proposed method is original as far as I know in the context of video restoration. The main originality is in the use of the same additional noise across all the frames at each step of the inverse diffusion process. This change combined with the data term constraint (and the use of conjugate gradients from prior work) seems to be sufficient to restore videos and yield sota performance on different video restoration tasks.", "weaknesses": "This method has been demonstrated for the case where the degradation operator is given, ie, we are\nworking in the non-blind video restoration case.  If I misunderstood this point, please clarify this in the rebuttal.\nI am assuming that A(X) is fully known and used to compute the conjugate gradients.\nMy main critique is that in practice it is very difficult to have access to such operator in the case of real videos. One could argue that having these operators is at the same level of complexity of solving the restoration task. Typically, one argues that focusing on the prior term by working in the non-blind case of the restoration task is still quite useful. However, this holds only if one can show that the prior could be \nused effectively also in the blind case. My main concern is that this may not hold in the proposed method.\n\nThe proposed novelty is that one should use the same additional noise samples for all the restored video frames. I wonder if this would work also in the blind case, ie, when also one needs to simultaneously recover the degradation operator. Basically, if the proposed method would only work in the non-blind case, then I would say that this is not a very significant contribution. If instead we could know that the method would also hold in the blind case, then it would be more significant (ie, the implicit temporal constraint through the same-noise sampling would be strong enough to work in the general case).\n\nIntuitively, my concern is that the noise sampling choice implicitly guides the frame restorations to be very similar to each other, but does not provide any hint on how the temporal dynamics should be. In fact, an image diffusion model cannot impose this type of constraint. The component that imposes some temporal constraint seems to be just the degradation operator. So if this operator is unknown/to be estimated, I wonder how well all the rest would work.\n\nIn the literature there is quite a bit of past work on dealing with unknown degradation operators (the so-called _blind_ case). These methods differ from the one proposed here, as they typically use a dataset to train a model in a supervised fashion to map a blurry/degraded video to a sharp one. The advantage is that they learn to generalise to new degradation operators at test time. It would be important to understand how this approach compares to such methods and to highlight their pros and cons.\nI think it would be more important to have such comparisons than to compare to even older ADMM methods.\nIf needed I can provide a few references on such recent blind video restoration methods, but it is straightforward to look them up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)