DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

Authors: Changhan Yeh, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao, Ting-Hsuan Chen, Hau-Shiang Shiu, Yu-Lun Liu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video restoration, Zero-shot, Training-free, Diffusion models

Abstract:
This paper introduces a method for zero-shot video restoration using pre-trained image restoration diffusion models. Traditional video restoration methods often need retraining for different settings and struggle with limited generalization across various degradation types and datasets. Our approach uses a hierarchical latent warping strategy for keyframes and local frames, combined with token merging that uses a hybrid correspondence mechanism that integrates spatial information, optical flow, and feature-based matching. We show that our method not only achieves top performance in zero-shot video restoration but also significantly surpasses trained models in generalization across diverse datasets and extreme degradations (8$\times$ super-resolution and high-standard deviation video denoising). We present evidence through quantitative metrics and visual comparisons on various challenging datasets. Additionally, our technique works with any 2D restoration diffusion model, offering a versatile and powerful tool for video enhancement tasks without extensive retraining.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730339258279

thread_type:
official_review

<thread_object>
creator: reviewer_ttiq

<content>
summary:
This paper proposes a training-free method to leverage pre-trained image restoration diffusion models for zero-shot video restoration. Specifically, the proposed method employs hierarchical latent warping and an enhanced token merging strategy to maintain temporal consistency and restore details across video frames. Experimental results demonstrate the versatility of the method across various tasks, including denoising, super-resolution, and depth estimation.

strengths:
1. The proposed method is pioneering in achieving zero-shot video restoration by leveraging pre-trained image restoration diffusion models, enabling multiple video restoration tasks without additional training.
2. The method presents strong quantitative and visual results compared with state-of-the-art methods, balancing temporal consistency and detail generation.

weaknesses:
1. In line 267, the authors claim that the combination of hierarchical latent warping and hybrid flow-guided spatial-aware token merging could achieve adaptation to various degradation types. However, it is not sufficiently discussed that why the combination could handle various degradation types.
2. While the use of optical flow guidance aligns with previous video restoration works [1,2], the paper introduces similarity-based guidance to capture correspondences distinct from optical flow. However, the specific benefits of similarity-based guidance over optical flow are not thoroughly discussed.
3. Although recent video restoration methods, such as Shift-Net and FMA-Net, are included as baselines, some classic methods like BasicVSR++ and RVRT are not compared in the experiments.

[1] Chan K C K, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 5972-5981.

[2] Liang J, Fan Y, Xiang X, et al. Recurrent video restoration transformer with guided deformable attention[J]. Advances in Neural Information Processing Systems, 2022, 35: 378-393.

questions:
1. Is there a comparison of inference efficiency with baselines?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595195410

thread_type:
official_review

<thread_object>
creator: reviewer_jkq8

<content>
summary:
This paper introduces a versatile zero-shot video restoration approach leveraging pre-trained image restoration diffusion models. Unlike conventional methods that require retraining and often struggle with generalization, this method uses a hierarchical latent warping strategy for keyframes and local frames, along with a hybrid correspondence mechanism that merges tokens based on optical flow and token similarity. This approach demonstrates outstanding zero-shot performance, effectively generalizing across varied datasets and managing extreme degradations, including video super-resolution, denoising, and depth completion. Compatible with numerous 2D restoration diffusion models, the technique is validated through quantitative and visual benchmarks on challenging datasets, removing the need for extensive retraining.

strengths:
The primary contribution of this approach is its ability to leverage conventional image generation models directly, without requiring modifications to network architecture or the need for retraining or fine-tuning. This is achieved through a straightforward yet effective technique: hierarchical token merging within the latent space, which ensures temporal consistency across generated video frames. By using this token merging strategy, the method successfully adapts static image models for dynamic video generation, maintaining coherence between frames without compromising quality. Furthermore, this approach reflects a well-engineered integration of recent advances in token merging, incorporating techniques from VidToMe and Upscale-A-Video. These advancement enhances the network’s capacity to address various video restoration challenges, effectively overcoming the limitations of traditional methods that struggle to maintain temporal consistency across frames.

weaknesses:
First, the paper’s structure requires improvement, as the current organization makes it challenging to follow. 
A major concern is the lack of a comprehensive comparison between this approach and conventional methods, such as VidToMe and Upscale-A-Video. VidToMe introduces local and global token merging techniques, while Upscale-A-Video presents a flow-based merging approach—both key contributions relevant to this work. Please clarify how this approach differentiates itself from these methods, and refer to the “Questions” section for further details.

questions:
1. VidToMe is implemented on top of Stable Diffusion. In Table 1, the proposed approach (based on Stable Diffusion (SD x4)) is compared with VidToMe, but it does not seem to demonstrate performance gains over VidToMe in terms of objective image quality (PSNR/SSIM) or temporal consistency (E_warp). On the other hand, perceptual quality metrics, such as LPIPS, show improved results. Could you clarify if there is a specific reason why perceptual quality metrics like LPIPS perform better? If there is a trade-off involved, would it be possible to show how performance varies with adjustments to the hyperparameters?
2. Although the proposed method emphasizes compatibility with any 2D image restoration diffusion model, the comparison results in Table 1—before and after applying the proposed method on SD x4—show only modest performance gains, whereas the improvements over the DiffBIR baseline are more substantial. Could you please clarify if there is a specific reason for this difference or if there is any dependence on specific network architecture?
3. It appears that the code for Upscale-A-Video is now available. Since this approach also aims to improve temporal consistency across video frames generated by image generation models, it would be beneficial to include a comparison of the results. If the code is still unavailable, could you instead provide results on the datasets used in Upscale-A-Video and compare them with the figures reported in the Upscale-A-Video manuscript?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712121051

thread_type:
official_review

<thread_object>
creator: reviewer_12wh

<content>
summary:
This paper presents a zero-shot video restoration method using pre-trained image restoration diffusion models. The proposed approach introduces a hierarchical latent warping strategy and a hybrid flow-guided token merging approach to enhance both temporal consistency and visual detail. Experimental results demonstrate competitive performance in video super-resolution, denoising, showing advantages over state-of-the-art methods. An ablation study further highlights the importance of each proposed component.

strengths:
- The paper is well-structured, making the methodology and findings easy to understand.
- The method achieves competitive results without requiring additional training.

weaknesses:
- Both the latent warping and hybrid flow-guided token merging approaches rely heavily on optical flow information, which could be a limitation in cases where optical flow estimation is challenging or inaccurate.
- The paper's novelty is somewhat limited, as it primarily combines two existing methodologies with minor modifications. Specifically, the contributions include adjusting the range of warping frames at global and local levels and introducing a flow-guided confidence criterion for token merging.

questions:
- Degradations that are not encountered during optical flow training may introduce significant errors. How does the proposed method address this issue, and what motivated the choice of the GMFlow network for this approach?
- In Equation (5), is there a specific reason for including f_{src -> tar} (X(T_{src})) twice? 
- Including additional real-world datasets (e.g., [1, 2]) in the evaluation would help assess the method's generalization and robustness. 
- Could you provide a comparison of inference times with other methods?

Reference

[1] Towards Real-world Video Face Restoration: A New Benchmark, cvprw 2024

[2] Toward convolutional blind denoising of real photographs, cvpr 2019
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716287876

thread_type:
official_review

<thread_object>
creator: reviewer_nzxb

<content>
summary:
This paper proposes a method for zero-shot video restoration using pre-trained image restoration diffusion models. It combines a hierarchical latent warping strategy for keyframes and local frames and token merging that integrates spatial information, optical flow, and feature-based matching.  The method acheives competitive results on zero-shot video restoration tasks such as super-resolution and denoising.

strengths:
1. The method is training-free which makes it computationally practical.

weaknesses:
1. To my understanding, the method relies on certain architectural heuristics inspired by the video-editing literature rather than on a solid mathematical framework. Since the method involves no training, there is no theoretical guarantee that it won't fail under certain conditions.

2. The method combines existing ideas from video editing, which on my opinion is still acceptable; however, this limits its novelty.

3. The improvement over the baseline on some datasets, particularly in the case of denoising, is not significant.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732173201284

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response Summary

comment:
We sincerely thank all reviewers for their thorough and constructive feedback. We are pleased that the reviewers recognized our method's **training-free nature** (Reviewer NZxb), **well-structured presentation** (Reviewer 12WH), **strong performance** across various restoration tasks (Reviewer ttiq), and our ability to **leverage conventional image models directly without architectural modifications** through an effective **hierarchical token merging** strategy (Reviewer JKQ8).

Our main responses include:
1. Clarified our method's **theoretical limitations** and pointed **failure cases** in L537-539 and Appendix Figure 17
2. Expanded comparisons with **BasicVSR++**, **RVRT** (Figure 6 and 13 and the tables below), and **Upscale-A-Video** (Figure 7), demonstrating our superior perceptual quality
3. Added qualitative comparisons on **video face restoration** (Figure 14)
4. Clarified technical details about the **hybrid correspondence mechanism** and **degradation handling**
5. Provided comprehensive **runtime analysis** and detailed **ablation studies** in the tables below

Notably, our experiments show that our **training-free approach** achieves competitive or superior results compared to methods requiring extensive resources (e.g., Upscale-A-Video needs **32 A100 GPUs**). The revised manuscript incorporates all these improvements while maintaining clarity and technical depth.

Thank you for helping us improve the paper's quality and completeness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732524598802

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Please let us know if you have additional questions after reading our response.

comment:
Dear Reviewers,

We sincerely thank you for your thorough reviews and constructive feedback. We have made our best efforts to address all your concerns through detailed responses and revisions to our manuscript. As the **discussion phase ends on Nov. 27 (AOE)**, we welcome any additional questions or concerns you may have regarding our responses.

If you find our responses and revisions satisfactory, particularly our added comparisons with classical methods (BasicVSR++/RVRT) and Upscale-A-Video, additional qualitative comparisons on video face restoration, comprehensive runtime analysis, and clarified technical details, we would greatly appreciate your consideration in updating your rating of our paper.

Thank you again for helping us improve this work on training-free video restoration.

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a zero-shot video restoration method with a pre-trained image-restoration diffusion model without additional training. A hierarchical latent warping strategy is used to improve temporal consistency. The token merging technique is used with the hybrid correspondence information that aggregates the spatial information, optical flow, and features. 

The reviewers agreed on the benefit of the training-free video restoration method as most other methods require additional training.

On the other hand, there were several concerns.
* The reviewers were concerned about the robustness against the severe input video degradation as the optical flow prediction error could cause inaccuracies. The authors argue that for optical flow prediction, GMFlow network was chosen due to its robustness and the hybrid correspondences handle the error by merging the confidently matched tokens pairs. As the authors point out, this is a combination of existing techniques and do not bring much novelty. Also, such hybrid correspondence family of techniques have been widely explored in the computer vision literature such as object tracking and video recognition.
* While the perceptual measure LPIPS is good, the fidelity measures PSNR/SSIM do not excel compared with VidToMe or more classical supervised methods. The authors did not explain why only LPIPS is emphasized by the proposed method other than the general perception-fidelity trade-off. While the token merging ratio can affect the preference on LPIPS over PSNR and SSIM, it is still unknown how the hyperparameter can be determined for users.

This paper can be improved by better presenting how the correspondence can be improved with a thorough survey and analysis of the existing works and by providing the understanding of the output characteristics.