DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

## Abstract

Abstract This paper introduces a method for zero-shot video restoration using pre-trained image restoration diffusion models. Traditional video restoration methods often need retraining for different settings and struggle with limited generalization across various degradation types and datasets.
Our approach uses a hierarchical latent warping strategy for keyframes and local frames, combined with token merging that uses a hybrid correspondence mechanism that integrates spatial information, optical flow, and feature-based matching.
We show that our method not only achieves top performance in zero-shot video restoration but also significantly surpasses trained models in generalization across diverse datasets and extreme degradations (8 × \times × super-resolution and high-standard deviation video denoising). We present evidence through quantitative metrics and visual comparisons on various challenging datasets. Additionally, our technique works with any 2D restoration diffusion model, offering a versatile and powerful tool for video enhancement tasks without extensive retraining.
See our project page for video results and source code: jimmycv07.github.io/DiffIR2VR_web .

## 1 Introduction

Figure: Figure 1: Zero-shot temporal-consistent diffusion model for video restoration and beyond. Given a pre-trained diffusion model for *single-image* restoration, our method generates temporally consistent restored video with fine details *without* any further training. Our method applies to other video applications, such as depth estimation.
Refer to caption: x1.png

Diffusion models have recently achieved remarkable success in image restoration tasks (Xia et al., ; Lin et al., ). These models can generate realistic details, overcoming the limitations of traditional regression-based methods that often produce blurry outputs without fine details (Fig. 2 (a)). The state-of-the-art methods employing convolutional neural networks (CNNs) (Albawi et al., ; Kalchbrenner et al., ; O’shea & Nash, ) or transformers (Dosovitskiy et al., ; Liu et al., ; Vaswani et al., ) trained on large-scale data have shown incredible effectiveness in image restoration.

Given the success of diffusion models in image restoration, a natural extension is to apply them to video restoration. Video restoration, which typically involves denoising, super-resolution, and deblurring, is a valuable field that transforms low-quality videos into high-quality ones. However, directly applying image-based diffusion models to video restoration presents significant challenges. Notably, performing per-frame inference on videos using these models often results in severe flickering (Fig. 2 (b)), especially when using Latent Diffusion Models (LDMs).

Surprisingly, the application of image restoration diffusion models to video restoration remains largely unexplored. While some attempts have been made to adapt these models for video tasks, they typically involve fine-tuning with 3D convolution and temporal attention layers. However, such approaches require extensive computational resources (*e.g.*, 32 A100-80G GPUs for video upscaling (Zhou et al., )) and task-specific retraining, limiting their practicality and generalizability.

In this paper, we present a novel, training-free approach to leverage pre-trained image restoration diffusion models for video restoration. Our method introduces two key modules: hierarchical latent warping and hybrid flow-guided spatial-aware token merging. These modules work in tandem to enforce temporal consistency in both latent and token (feature from the attention layer) spaces, enabling high-quality video restoration without any additional training or fine-tuning. Our method (Fig. 2 (c)) achieves both realistic and temporally consistent results without any additional training, leveraging an image diffusion model to restore videos effectively.

Fig. 1 illustrates our method’s capability to generate temporally consistent restored videos across various tasks, including denoising, super-resolution, and depth estimation, *without* any further training. Our zero-shot video restoration framework can be applied to any pre-trained image diffusion model, offering a versatile solution that can adapt to various restoration tasks.

While inspired by recent advances in diffusion-based generation models like VidToMe (Li et al., ) and TokenFlow (Geyer et al., ), our work goes beyond combining existing techniques. Our main contributions are:

- •
First zero-shot video restoration using diffusion models, balancing temporal consistency and detail generation across various image-based models.
- •
Training-free framework manipulating latent and token spaces with hierarchical latent warping and improved token merging.
- •
State-of-the-art results in extreme scenarios, surpassing traditional methods in generalizability and robustness.

Figure: Figure 2: 4$\times$ video super-resolution results. (a) Traditional regression-based methods such as FMA-Net (Youk et al., ) are limited to the training data domain and tend to produce blurry results when encountering out-of-domain inputs. (b) Although applying image-based diffusion models such as DiffBIR (Lin et al., ) to individual frames can generate realistic details, these details often lack consistency across frames. (c) Our method leverages an image diffusion model to restore videos, achieving both realistic and consistent results *without* any additional training.
Refer to caption: x2.png

## 2 Related Work

#### Video Restoration.

Video restoration aims to restore high-quality frames from degraded videos, addressing issues such as noise, blur, and low resolution (Chan et al., ; ; Isobe et al., ; Li et al., ; Youk et al., ; Zhang et al., ; Liu et al., ; ).
This task is more challenging than image restoration due to the need for temporal consistency.
Learning-based approaches employ architectures like optical flow warping (Huang et al., ; Pan et al., ; Shi et al., ; ; Xue et al., ), deformable convolutions (Chan et al., ; ; Dai et al., ; Tian et al., ; Wang et al., ; ; Zhu et al., ), and attention mechanisms to handle temporal dependencies (Cao et al., ; Li et al., ; Liang et al., ; Zamir et al., ).
Major limitations include dependency on paired HQ-LQ data (Chan et al., ; Xie et al., ; Yang et al., ), assumptions of predefined degradation processes (Kim et al., ; ; Kong et al., ; Li et al., ; Liang et al., ), and the need for retraining for different degradation levels (Liu & Sun, ; Nah et al., ; Yi et al., ; Youk et al., ).
These factors reduce effectiveness in real-world applications and lead to poor generalization. Additionally, these methods often lose significant detail, similar to image restoration  (Chen et al., ; Liang et al., ; Wang et al., ; Zhang et al., ).

#### Diffusion Models for Image Restoration.

With significant advancements in diffusion models (Choi et al., ; Dhariwal & Nichol, ; Hertz et al., ; Ho et al., ; Rombach et al., ), many diffusion-based approaches have been proposed for image restoration (Fei et al., ; Ho et al., ; Nichol et al., ; Sohl-Dickstein et al., ; Song et al., ; Wang et al., ; Yang et al., ). These methods include training diffusion models from scratch (Rombach et al., ; Saharia et al., ; Xia et al., ; Yue et al., ), introducing constraints into the reverse diffusion process of pre-trained models (Kawar et al., ), and fine-tuning frozen pre-trained diffusion models with additional trainable layers (Wang et al., ; Yang et al., ; Zhang et al., ), as seen in StableSR (Wang et al., ) and DiffBIR (Lin et al., ). Despite their effectiveness in image restoration, these methods face challenges in video restoration due to temporal inconsistencies caused by the diffusion process’s randomness. In contrast, our method allows these approaches to work on video without any training, addressing the temporal consistency issue while leveraging the strengths of image restoration diffusion models.

#### Video Editing Methods for Temporal Consistency.

Recent research has extended pre-trained image diffusion models to video tasks (Esser et al., ; Ho et al., ; ; Hu et al., ; Lu et al., ; Luo et al., ; Mei & Patel, ; Kara et al., ). Various methods have been proposed to enhance temporal consistency in video editing, which can be categorized based on the level at which they operate:

- •
Latent Space Level: Approaches working at the latent space level include Rerender-A-Video (Yang et al., ), which employs latent warping (Teed & Deng, ; Xu et al., ) and frame interpolation. While these methods aim to maintain consistency in the latent representations of consecutive frames, they may struggle with semantic consistency in demanding restoration tasks. Our method introduces a novel hierarchical latent warping technique that addresses these limitations.
- •
Token Level: Methods operating at the token level include VidToMe (Li et al., ) and TokenFlow (Geyer et al., ), which enhance temporal consistency by merging attention tokens across frames. Token merging (Bolya et al., ) is another technique used at this level. However, these techniques often produce blurry outputs in restoration tasks. Our approach improves upon these methods by introducing a hybrid flow-guided spatial-aware token merging technique that maintains sharpness while ensuring temporal consistency.

While these video editing techniques generate impressive results with minimal effort, they often struggle with semantic consistency and detail preservation in demanding restoration tasks. Our work draws inspiration from these approaches but introduces novel elements specifically designed to address the challenges of video restoration, combining the strengths of latent and token-level methods while mitigating their individual weaknesses.

Figure: Figure 3: Pipeline of our proposed zero-shot video restoration method. We process low-quality (LQ) videos in batches using a diffusion model, with a keyframe randomly sampled within each batch. (a) At the beginning of the diffusion denoising process, hierarchical latent warping provides rough shape guidance both globally, through latent warping between keyframes, and locally, by propagating these latents within the batch. (b) Throughout most of the denoising process, tokens are merged before the self-attention layer. For the downsample blocks, optical flow is used to find the correspondence between tokens, and for the upsample blocks, cosine similarity is utilized. This hybrid flow-guided, spatial-aware token merging accurately identifies correspondences between tokens by leveraging both flow and spatial information, thereby enhancing overall consistency at the token level.
Refer to caption: extracted/5898068/figures/pipeline.png

## 3 Method

Given a low-quality video with $n$ frames $\left\{y^{1},y^{2},\ldots,y^{n}\right\}$, we aim to restore it to high-quality $\left\{x^{1},x^{2},\ldots,x^{n}\right\}$ using image-based diffusion models. Directly applying these models frame-by-frame causes temporal inconsistency due to inherent stochasticity, especially in extreme degradation (Fig. 2 and Fig. 6). Our method (Fig. 3) addresses this by enforcing temporal stability in latent and token spaces through Hierarchical Latent Warping (Sec. 3.2) and Hybrid Flow-guided Spatial-aware Token Merging (Sec. 3.3). We introduce diffusion models and video token merging (Sec. 3.1), then detail our key components (Sec. 3.2-Sec. 3.4).

Figure: Figure 4: An illustration of our key modules. Without requiring any training, these modules can achieve coherence across frames by enforcing temporal stability in both latent and token space. Hierarchical latent warping provides global and local shape guidance; Hybrid spatial-aware token merging before the self-attention layer improves temporal consistency by matching similar tokens using optical flow in the down blocks and cosine similarity in the up blocks of the UNet.
Refer to caption: x3.png

### 3.1 Diffusion Models for Video Editing

Diffusion models have been successfully applied to video editing tasks by extending image-based models. These models typically operate as follows:

#### Diffusion Process.

The forward process adds noise to a clean image $x_{0}$ over $T$ steps:

$$ $x_{t}=\sqrt{\alpha_{t}}x_{t-1}+\sqrt{1-\alpha_{t}}\epsilon_{t-1}\Rightarrow x_ {t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,$ (1) $$

where $t\sim[1,T]$, $\epsilon_{t},\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$,
and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$.
A UNet-based denoiser $\epsilon_{\theta}$ is trained to estimate and remove this noise. During inference, the inverse process gradually denoises $x_{t}$ to produce $x_{0}$ (Ho et al., ; Song et al., ; ).
These models can be enhanced with additional guidance signals for controlled generation (Zhang et al., ; Kawar et al., ).

#### Video Token Merging.

To maintain temporal consistency, techniques like Video Token Merging (VidToMe) (Li et al., ) are employed. This process merges similar tokens within frame chunks in attention blocks:
Given a token chunk $\textbf{T}\in\mathbb{R}^{B\times A\times C}$, where $A=w*h$, the algorithm first separates the tokens into source tokens $\textbf{T}_{\text{src}}\in\mathbb{R}^{B\times A-1\times C}$ and a target token $\textbf{T}_{\text{tar}}\in\mathbb{R}^{B\times 1\times C}$. It then calculates the cosine between each source and target token, determining their corresponding similarity levels, denoted $score\in\mathbb{R}^{((B-1)*A)\times A}$. The algorithm then identifies the most similar target token for each source token by taking the maximum value in the last column.

$$ $\displaystyle s(\textbf{T}_{\text{src}},\textbf{T}_{\text{tar}})=\frac{\textbf {T}_{\text{src}}\cdot\textbf{T}_{\text{tar}}}{\left\|\textbf{T}_{\text{src}} \right\|\left\|\textbf{T}_{\text{tar}}\right\|}\,,\,\,\,c=\max_{\{\textbf{t} \in\textbf{T}_{\text{tar}}\}}(s(\textbf{T}_{\text{src}},\textbf{t})),$ (2) $$

where $s(\cdot,\cdot)$ is the cosine similarity score and $c$ indicates the correspondences.
Next, the $r$ most similar paired source-target tokens are merged, and the remaining tokens are concatenated as the output. Merged tokens are subsequently unmerged after self-attention to preserve the original shape by simply assigning the merged source-target tokens the exact same value. The token merging and unmerging are defined as follows:

$$ $\displaystyle\textbf{T}_{\text{merge}}=\mathcal{M}(\textbf{T}_{\text{src}}, \textbf{T}_{\text{tar}},\;c,\;r)\,,\,\,\,\textbf{T}_{\text{unmerge}}=\mathcal{ U}(\textbf{T}_{\text{merge}},\;c)\,,$ (3) $$

where $\mathcal{M}$ and $\mathcal{U}$ denote the merging and unmerging operations, respectively.

#### Latent Warping.

Some methods (Zhou et al., ) perform warping in the latent space to maintain consistency between frames. This is done by warping the latent representations of adjacent frames.

#### Limitations in video restoration.

Existing video editing techniques face challenges in video restoration, often prioritizing temporal consistency over detail preservation. Early-stage denoising produces noisy latents, making traditional similarity measures unreliable, especially in UNet’s downsample blocks (Fig. 5, top). Most methods focus on frame-to-frame consistency and missing global-local coherence, while high merging ratios can lead to over-smoothing.
Our approach combines hierarchical latent warping with hybrid flow-guided spatial-aware token merging to address these limitations. This combination provides multi-scale temporal consistency, balances detail preservation with consistency, and adapts to various degradation types. Latent warping handles large-scale inconsistencies in early stages, while token merging ensures fine detail consistency as features become more meaningful. By leveraging both optical flow and similarity measures, our method aims for superior zero-shot video restoration without task-specific training or computational resources.

### 3.2 Hierarchical Latent Warping

We introduce a hierarchical latent warping module operating in latent space, with a two-level approach: (1) Global level: Warping between keyframes, and (2) Local level: Propagating warped latents within each batch.
As shown in Fig. 5 (upper part), this provides rough shape guidance on global and local scales.
Let $\hat{x}^{i}_{t\rightarrow 0}$ be the predicted $\hat{x}_{0}$ latent for the $i^{th}$ keyframe at denoising step $t$. We first perform global-level warping between keyframes:

$$ $\hat{x}^{i}_{t\rightarrow 0}\leftarrow M_{ji}\cdot\hat{x}^{i}_{t\rightarrow 0} +\left(1-M_{ji}\right)\cdot\mathcal{W}(\hat{x}^{j}_{t\rightarrow 0},f_{ji}),$ (4) $$

where $j=i-1$ and $f_{ji}$, $M_{ji}$ denotes the optical flow and the occlusion mask from $lq_{j}$ to $lq_{i}$ estimated by GMFlow (Xu et al., ).
We then perform local-level warping by propagating these latents to remaining frames within each batch. This approach ensures corresponding points share similar latents globally across keyframes and locally within batches from the start of denoising, providing a more comprehensive approach to maintaining consistency compared to simple frame-to-frame warping.

### 3.3 Hybrid Flow-guided Spatial-aware Token Merging

While latent manipulation can achieve a certain degree of consistency, manipulating latents during the later stages of the denoising process would result in blurry outcomes. Additionally, the token space is highly semantically related to the image. Therefore, we propose hybrid flow-guided spatial-aware token merging to achieve consistency in the token space.

#### Flow-guided.

Our hybrid correspondence mechanism integrates spatial information, optical flow, and feature-based similarity. In early denoising stages, latents are noisy, making cosine similarity unreliable, especially in UNet’s downsample blocks (Fig. 5, top). However, optical flow from low-resolution inputs provides better guidance. As denoising progresses (*e.g.*, steps 30-40), flow-based and similarity-based methods often identify different matches (Fig. 5, bottom), suggesting the benefit of a hybrid approach.
Even with low-quality video, we can identify correspondences between frames based on color. We use flow for correspondences in UNet downsample blocks and employ forward-backward consistency check as a criterion to determine $r$ most similar paired source token $\textbf{T}_{\text{src}}$ and target token $\textbf{T}_{\text{tar}}$:

$$ $\sigma=\exp(-\left\|f_{\text{src}\rightarrow\text{tar}}(X(\textbf{T}_{\text{ src}}))+f_{\text{tar}\rightarrow\text{src}}\left(X(\textbf{T}_{\text{src}})+f_ {\text{src}\rightarrow\text{tar}}(X(\textbf{T}_{\text{src}}))\right)\right\|_{ 2}^{2}),$ (5) $$

where $\sigma$ is the confidence, $X(\textbf{T}_{\text{src}})$ is the spatial location of $\textbf{T}_{\text{src}}$, and $f_{src\rightarrow tar}$, $f_{tar\rightarrow src}$ denotes the forward and backward flow between $\textbf{T}_{\text{src}}$ and $\textbf{T}_{\text{tar}}$. The proposed flow-guided token merging is:

$$ $\textbf{T}_{\text{merge}}=\mathcal{M}(\textbf{T}_{\text{src}},\;\textbf{T}_{ \text{tar}},\;f_{\text{src}\rightarrow\text{tar}},\;\sigma,\;r).$ (6) $$

Fig. 5 provides a clearer illustration of our proposed component.
While optical flow can be challenging in certain conditions (*e.g.*, fast motion, textureless regions), our method incorporates several safeguards. We use forward-backward consistency checks, merge only the $r$ most similar token pairs, and combine flow-based correspondence with spatial information and similarity matching. This multi-faceted approach ensures robust performance in challenging conditions. Additionally, as shown in Fig. 5 (bottom), flow and cosine similarity identify different correspondences, providing comprehensive guidance. Tab. 1 demonstrates that using flow correspondence in downblocks and similarity in upblocks yields the best visual quality and temporal consistency.

#### Spatial-awareness and Padding Removal.

Directly finding correspondences using cosine similarity can lead to mismatches in areas with uniform textures, especially in video backgrounds (*e.g.*, sky, sand, grass; Fig. 5, bottom), resulting in blurrier outcomes. Given that corresponding points in adjacent frames are typically spatially close, we leverage this information by weighting cosine similarity scores with tokens’ spatial distances:

$$ $s_{ij}^{\prime}=s_{ij}\cdot e^{-\tau}\,,\,\,\text{with}\,\,\tau=\left\lfloor \left[{\|X(i)-X(j)\|_{2}^{2}}\right]/{R}\right\rfloor\,,$ (7) $$

where $X(i)$, $X(j)$ are spatial locations of the $i^{th}$ source and $j^{th}$ target token; $R$ is a hyperparameter defining the radius of the uniform weight region.

This spatial awareness primarily applies to cosine similarity correspondences in UNet upsample blocks. For flow correspondences in downsample blocks, we rely on forward-backward consistency checks as described in Eq. 5, since optical flow models inherently consider spatial information. This combination ensures effective utilization of spatial information throughout our token merging process.
Another point to consider is that images are often padded to pass through the UNet, which can significantly impact token correspondences by causing cosine similarity to mistakenly align padding with actual content, even in later denoising stages. To mitigate this, we remove padding before merging and reapply it after unmerging. See the appendix for visual ablation results.

#### Token Unmerging.

After the self-attention operation, tokens need to be unmerged to restore the original shape. We adopt a replacement-based unmerging process where tokens are restored to their original shape using the identified correspondences. This approach is similar to VidToMe (Li et al., ), but our method’s primary innovation lies in enhancing the correspondence identification process during the merging stage, which leads to more accurate and effective token matching.

#### Merging Ratio Annealing.

To prevent over-smoothing in later denoising stages, we employ ratio annealing to gradually reduce the merging ratio. The merging ratio of the $i^{th}$ denoising step is:

$$ $r_{i}=r\cdot\cos\left(\frac{\pi}{2}\cdot\text{max}\left(\text{min}\left(\delta \cdot\frac{i-i_{\text{beg}}}{i_{\text{end}}-i_{\text{beg}}},1\right),0\right) \right),$ (8) $$

where $i_{\text{beg}}$, $i_{\text{end}}$ are predefined steps indicating the beginning and end of the merging process, and $\delta$ controls annealing speed.
This technique balances smoothness and temporal consistency, achieving a compromise between regression-based methods (temporally consistent but overly smooth) and per-frame inferencing (detailed but inconsistent). As shown in Fig. 2 and Fig. 6, our approach preserves fine details while maintaining temporal coherence, proving effective in severe degradation scenarios. Visual comparisons for 8$\times$ super-resolution are provided in supplementary materials.

### 3.4 Scheduling

As depicted in Fig. 3, at the initial stage of the diffusion denoising process, hierarchical latent warping offers rough shape guidance on a global scale by warping latents between keyframes and on a local scale by propagating these latents within the batch. During the majority of the denoising process, tokens are processed with our hybrid spatial-aware token merging before entering the attention layer. This component further improves temporal consistency by matching similar tokens, utilizing both flow and spatial information.

**Table 1: Quantitative comparisons. (*Left*) Video super-resolution on the DAVIS (Perazzi et al., ), Vid4 (Liu & Sun, ) and REDS4 (Nah et al., ) datasets. (*Right*) video denoising of various noise levels on the REDS30 and Set8 (Tassano et al., ) dataset. The best and second performances are marked in red and blue, respectively. $E_{\text{warp}}^{*}$ denotes $E_{\text{warp}}(\times 10^{-3})$ and $E_{\text{inter}}$, LPIPS${}_{\text{inter}}$ denotes interpolation error and LPIPS. - indicates out-of-memory.**
|  |  |  |  | SD $\times$4 | DiffBIR |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Metrics | VidToMe | FMA-Net | Frame | Ours | Frame | Ours |
| DAVIS $\times$4 | PSNR $\uparrow$ | 23.014 | 25.215 | 23.504 | 23.843 | 23.780 | 24.182 |
| SSIM $\uparrow$ | 0.566 | 0.727 | 0.584 | 0.618 | 0.601 | 0.621 |  |
| LPIPS $\downarrow$ | 0.405 | 0.347 | 0.277 | 0.272 | 0.264 | 0.262 |  |
| $E_{\text{warp}}^{*}\downarrow$ | 0.520 | 0.186 | 0.912 | 0.745 | 0.654 | 0.474 |  |
| $E_{\text{inter}}\downarrow$ | 13.676 | 11.558 | 18.125 | 17.431 | 16.529 | 14.666 |  |
| LPIPS${}_{\text{inter}}\downarrow$ | 0.329 | 0.078 | 0.292 | 0.274 | 0.266 | 0.232 |  |
| DAVIS $\times$8 | PSNR $\uparrow$ | 22.097 | 22.690 | 20.268 | 20.519 | 21.964 | 22.331 |
| SSIM $\uparrow$ | 0.513 | 0.594 | 0.446 | 0.424 | 0.502 | 0.519 |  |
| LPIPS $\downarrow$ | 0.554 | 0.528 | 0.470 | 0.434 | 0.362 | 0.367 |  |
| $E_{\text{warp}}^{*}\downarrow$ | 0.440 | 0.351 | 2.199 | 1.759 | 0.964 | 0.699 |  |
| $E_{\text{inter}}\downarrow$ | 12.624 | 13.978 | 24.496 | 21.746 | 17.981 | 15.853 |  |
| LPIPS${}_{\text{inter}}\downarrow$ | 0.388 | 0.132 | 0.457 | 0.442 | 0.372 | 0.333 |  |
| REDS4 $\times$4 | PSNR $\uparrow$ | 23.134 | 25.829 | 24.189 | 24.226 | 24.679 | 25.118 |
| SSIM $\uparrow$ | 0.589 | 0.761 | 0.638 | 0.641 | 0.657 | 0.683 |  |
| LPIPS $\downarrow$ | 0.357 | 0.327 | 0.247 | 0.242 | 0.211 | 0.222 |  |
| $E_{\text{warp}}^{*}\downarrow$ | 0.579 | 0.392 | 0.817 | 0.811 | 0.704 | 0.499 |  |
| $E_{\text{inter}}\downarrow$ | 17.869 | 19.014 | 22.906 | 22.889 | 22.305 | 20.130 |  |
| LPIPS${}_{\text{inter}}\downarrow$ | 0.356 | 0.133 | 0.295 | 0.281 | 0.271 | 0.221 |  |
| REDS4 $\times$8 | PSNR $\uparrow$ | 21.894 | 22.842 | - | - | 22.479 | 22.961 |
| SSIM $\uparrow$ | 0.532 | 0.644 | - | - | 0.559 | 0.59 |  |
| LPIPS $\downarrow$ | 0.538 | 0.423 | - | - | 0.311 | 0.306 |  |
| $E_{\text{warp}}^{*}\downarrow$ | 0.423 | 0.753 | - | - | 0.828 | 0.551 |  |
| $E_{\text{inter}}\downarrow$ | 15.502 | 21.519 | - | - | 21.76 | 19.382 |  |
| LPIPS${}_{\text{inter}}\downarrow$ | 0.412 | 0.159 | - | - | 0.351 | 0.287 |  |
| REDS4 $\times$16 | PSNR $\uparrow$ | 20.520 | 21.569 | 18.706 | 18.858 | 20.124 | 20.712 |
| SSIM $\uparrow$ | 0.483 | 0.570 | 0.461 | 0.410 | 0.461 | 0.509 |  |
| LPIPS $\downarrow$ | 0.697 | 0.565 | 0.612 | 0.562 | 0.446 | 0.438 |  |
| $E_{\text{warp}}^{*}\downarrow$ | 0.296 | 0.619 | 2.664 | 2.030 | 1.168 | 0.665 |  |
| $E_{\text{inter}}\downarrow$ | 12.945 | 18.758 | 28.478 | 24.000 | 21.33 | 17.731 |  |
| LPIPS${}_{\text{inter}}\downarrow$ | 0.417 | 0.139 | 0.559 | 0.493 | 0.444 | 0.358 |  |
| Vid4 $\times$4 | PSNR $\uparrow$ | 19.622 | 23.209 | 20.047 | 20.134 | 20.687 | 21.226 |
| SSIM $\uparrow$ | 0.425 | 0.679 | 0.478 | 0.473 | 0.497 | 0.525 |  |
| LPIPS $\downarrow$ | 0.491 | 0.375 | 0.343 | 0.331 | 0.329 | 0.326 |  |
| $E_{\text{warp}}^{*}\downarrow$ | 0.687 | 0.203 | 1.502 | 1.397 | 1.156 | 0.677 |  |
| $E_{\text{inter}}\downarrow$ | 11.754 | 4.442 | 17.234 | 16.921 | 15.478 | 11.316 |  |
| LPIPS${}_{\text{inter}}\downarrow$ | 0.337 | 0.026 | 0.275 | 0.271 | 0.265 | 0.198 |  |
| Vid4 $\times$8 | PSNR $\uparrow$ | 18.811 | 21.033 | 17.813 | 17.992 | 18.636 | 19.304 |
| SSIM $\uparrow$ | 0.372 | 0.521 | 0.345 | 0.307 | 0.367 | 0.406 |  |
| LPIPS $\downarrow$ | 0.654 | 0.514 | 0.507 | 0.484 | 0.440 | 0.435 |  |
| $E_{\text{warp}}^{*}\downarrow$ | 0.477 | 0.221 | 2.523 | 1.972 | 1.524 | 0.767 |  |
| $E_{\text{inter}}\downarrow$ | 9.942 | 5.269 | 22.881 | 19.970 | 18.112 | 12.281 |  |
| LPIPS${}_{\text{inter}}\downarrow$ | 0.393 | 0.032 | 0.423 | 0.419 | 0.395 | 0.294 |  |

**Table 2: Ablation studies for 8$\times$ VSR on DAVIS (Perazzi et al., ) test sets. (*Left*) different correspondence matching methods. (*Right*) the proposed components applied at different stages of the denoising process. We apply our two proposed components, hierarchical latent warping (HLW) and hybrid spatial-aware token merging (HS-ToMe), at the early, mid, and late denoising stages.**
| Down<br>blocks | Up<br>blocks | Spatial-<br>aware | LPIPS $\downarrow$ | $E_{\text{warp}}^{*}\downarrow$ | LPIPS${}_{\text{inter}}\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| Flow | Flow | – | 0.518 | 1.214 | 0.563 |
| Cos | Cos | – | 0.390 | 0.736 | 0.350 |
| Cos | Flow | – | 0.507 | 1.049 | 0.545 |
| Flow | Cos | – | 0.375 | 0.677 | 0.347 |
| Flow | Cos | ✓ | 0.367 | 0.699 | 0.333 |

Figure: Figure 6: Qualitative comparisons on 4$\times$ video super-resolution. As shown in the first row, the low-quality input lacks almost all details. In the zoomed-in patches, our method produces clearer and more consistent results.
Refer to caption: x5.png

Figure: Figure 7: Video denoising comparisons on the REDS30 (Nah et al., ) dataset. Our method effectively denoises and generates detailed results while maintaining temporal coherence.
Refer to caption: x6.png

## 4 Experiments

#### Testing Dataset.

For video super-resolution, we evaluate on REDS4 (Nah et al., ), Vid4 (Liu & Sun, ) and DAVIS (Perazzi et al., ) testing sets, with downsample scales $\times$4 and $\times$8, following the degradation pipeline of RealBasicVSR (Chan et al., ).
For video denoising, we evaluate on REDS30 (Nah et al., ) and Set8 (Tassano et al., ) with different noise levels (std. $=$ 50, 75, 100, 150 and randomly sampled from the range [50, 100]).

#### Evaluation Metrics.

We assess (1) image quality via LPIPS, SSIM, and PSNR; (2) temporal consistency, using warping error $E_{\text{warp}}$, interpolation error, and interpolation LPIPS. Since LPIPS better reflects visual quality, we propose interpolation LPIPS, based on the interpolation error used in a previous study  (Li et al., ), to more accurately measure video continuity from a visual perspective. This involves interpolating a target frame from its previous and next frames and computing the LPIPS between the estimated and target frames.

#### Implementation Details.

The experiment is conducted on an NVIDIA RTX 4090 GPU. We apply our method to DiffBIR (Lin et al., ) and SDx4 upscaler (sdx, ), both image-based diffusion models, to demonstrate the proposed method’s compatibility with different models. Note that for models that are restricted to a super-resolution scale of 4$\times$, we will apply the process twice and then use bicubic downsampling to achieve 8$\times$ results. However, this will can lead to out-of-memory issues for SDx4 upscaler in REDS.

### 4.1 Comparisons with State-of-the-Art Methods

To verify the effectiveness of our approach, we compare it with several state-of-the-art methods, including FMA-Net (Youk et al., ) for video super-resolution, and Shift-Net (Li et al., ) for video denoising. We also compare our method to per-frame restoration and the application of VidToMe (Li et al., ), a zero-shot video editing method, onto the same model as ours. We also try to compare with Upscale-A-Video (Zhou et al., ), which is a diffusion-based video restoration model fine-tuned from an image-based diffusion model. However, due to the lack of released implementation, pre-trained weights, or inference results, we cannot include Upscale-A-Video as a competing method in our comparison.

Our zero-shot video restoration framework is designed to be highly adaptable and capable of leveraging a wide range of pre-trained image diffusion models. This flexibility allows easy adaptation from image to video models without extensive retraining, enabling the application of various restoration tasks by simply switching the underlying image diffusion model.

#### Video Super-resolution.

As shown in Tab. 1, regression-based methods like FMA-Net (Youk et al., ) struggle with large motion or severe degradation.
VidToMe (Li et al., ) can generate highly consistent results, but they are often very blurry, leading to poor visual quality. In contrast, our method enhances temporal consistency while maintaining the generation quality of the original diffusion model, making it the most competitive approach.
Fig. 6 provides visualizations of two challenging VSR cases.
FMA-Net fails to produce sharp results due to domain gaps between training and testing. Diffusion-based image restoration method DiffBIR (Lin et al., ) and SD$\times$4 upscaler (sdx, ) can generate sharp results with details, while per-frame processing makes the result video temporal inconsistent and jitters across frames.
On the contrary, our zero-shot video restoration framework restores a low-quality input video into a temporally consistent high-quality video.

#### Video Denoising.

Video denoising, compared to VSR, is a simpler task for regression models, as they can often find the correct pixel value given a sufficiently large batch size. However,
our method consistently outperforms others in terms of visual quality (LPIPS) and remains highly robust even as degradation becomes severe.
Fig. 7 visualizes the denoising results on the REDS30 dataset. Shift-Net (Li et al., ) fails to remove all noise, likely due to the out-of-domain noise level; VidToMe (Li et al., ) produces smooth results but lacks fine details. Although DiffBIR (Lin et al., ) generates highly detailed images, it suffers from poor temporal consistency, as evident in the changes to the pedestrian’s head and the statue’s face. In contrast, our method preserves both fine details and temporal consistency, effectively balancing these two aspects.

#### Other Video Tasks: Consistent Video Depth.

Our zero-shot framework is applicable to any pre-trained image-based diffusion models and could improve the predicted video consistency. Therefore, we integrate our proposed zero-shot framework into a state-of-the-art latent diffusion-based monocular depth estimator: Marigold (Ke et al., ). Fig. 8 shows that integrating our proposed framework into Marigold helps improve the temporal consistency of video depth estimation. We provide more visual comparisons in the supplementary materials.

Figure: Figure 8: Applying our techniques to consistent video depth. Integrating our proposed framework into Marigold (Ke et al., ) helps improve the temporal consistency of video depth estimation.
Refer to caption: extracted/5898068/figures/depth/input_goat_00043.jpg

This adaptability to various tasks (super-resolution, denoising, depth estimation) showcases the broad applicability of our approach. As more powerful or specialized image models emerge, our framework can quickly adapt to leverage these improvements for video restoration tasks. We provide computational complexity evaluations in the supplementary materials.

### 4.2 Ablation Study

#### Ways of Identifying Correspondence.

Tab. 2 presents an ablation study comparing different approaches (optical flow and cosine similarity) for finding correspondences and their order in the UNet. As detailed in Sec. 3.3, the hybrid approach of using optical flow at the downsample blocks and cosine similarity at the upsample blocks achieves the best performance. Additionally, our proposed spatial-aware token merging further enhances performance by utilizing spatial information to guide correspondences. See supplementary materials for temporal profile comparisons.

#### Applied Stages in the Denoising Process.

Tab. 2 presents an ablation study evaluating the application of our two proposed components, hierarchical latent warping (HLW, Sec. 3.2) and hybrid spatial-aware token merging (HS-ToMe, Sec. 3.3), at the early, mid, and late stages of the denoising process. The results indicate that applying latent warping in the mid or late stages can significantly degrade the generated outcomes. Furthermore, ensuring consistency in the token space is crucial for achieving coherent and high-quality results.

## 5 Conclusion

We introduce a novel zero-shot video restoration framework utilizing pre-trained image-based diffusion models, eliminating the need for extensive retraining. Our approach integrates hierarchical latent warping and hybrid flow-guided, spatial-aware token merging, significantly enhancing temporal consistency and video quality under various degradation conditions.
Experimental results demonstrate that our framework surpasses existing methods both in quality and consistency.

#### Limitations.

Our framework has two main limitations:
(1) LDM decoder sensitivity can cause flickering in dynamic scenes. (2) Extreme degradation may yield unsatisfactory results. Future work will address these issues by stabilizing decoder output, and enhancing extreme degradation handling. Our framework’s adaptability allows for the integration of future, more powerful diffusion models.

## Appendix A Appendix / supplemental material

In this supplementary material, we first provide additional details on the testing datasets and evaluation metrics. Subsequently, we present more visual comparisons of various methods.

### A.1 Ablation studies on correspondences identified by cosine similarity

Fig. 9 The figure shows the correspondences at denoising step 40 for three scenarios: without spatial awareness and padding removal, without spatial awareness, and with both spatial awareness and padding removal (ours). It is evident that padding values significantly affect the matching quality. However, even after removing padding, many mismatched diagonal lines remain, leading to blurry results. In contrast, our method effectively finds accurate correspondences by leveraging spatial information from the video.

Figure: (a) Correspondences without spatial-awareness and padding removal
Refer to caption: extracted/5898068/figures/correspondences/padding_no_distance.png

### A.2 Severe Degradation Scenarios.

Our balanced approach proves particularly effective in severe degradation scenarios. For instance, in 8$\times$ super-resolution tasks, our method not only avoids artifacts but can even improve visual quality compared to per-frame approaches (Fig. 10). This underscores the effectiveness of our ratio annealing technique in addressing the over-smoothing tendency while maintaining the benefits of our token merging approach. Additional comparisons on video super-resolution can be found at Fig. 11 and Fig. 12.

Figure: Figure 10: Applying our method on DiffBIR and SD $\times$4 upscaler for 8$\times$SR task. In this case of severe degradation, our method avoids artifacts and outperforms perframe inference in terms of visual quality.
Refer to caption: x7.png

Figure: Figure 11: Additional qualitative comparisons on 4$\times$ video super-resolution. In the zoomed-in patches, our method produces clearer and more consistent results.
Refer to caption: x8.png

Figure: Figure 12: Additional qualitative comparisons on 8$\times$ video super-resolution. As shown in the first row, the low-quality input lacks almost all details. In the zoomed-in patches, our method produces clearer and more consistent results.
Refer to caption: x9.png

#### Other Video Tasks: Consistent Video Depth.

Our zero-shot framework is applicable to any pre-trained image-based diffusion models and could improve the predicted video consistency. Therefore, we integrate our proposed zero-shot framework into a state-of-the-art latent diffusion-based monocular depth estimator: Marigold (Ke et al., ). Fig. 13 shows that integrating our proposed framework into Marigold helps improve the temporal consistency of video depth estimation.

Figure: Figure 13: Applying our techniques to consistent video depth. Integrating our proposed framework into Marigold (Ke et al., ) helps improve the temporal consistency of video depth estimation.
Refer to caption: extracted/5898068/figures/depth/input_dogs-jump_00031.jpg

### A.3 Computational Complexity

While our method focuses on zero-shot video restoration without additional training, it’s important to consider the computational requirements in comparison to other approaches. Tab. 3 provides an overview of the training time and GPU specifications for different methods, including ours.

As shown in the table, our method stands out by not requiring any training or fine-tuning, which significantly reduces the computational resources needed. This is in stark contrast to other methods that require multiple high-end GPUs and several days of training time.
For inference, our method introduces some computational overhead due to the hierarchical latent warping and hybrid token merging processes. However, this overhead is relatively small compared to the resources required for training or fine-tuning video models. Specifically, our method adds only approximately 6 seconds to the inference time of the base image diffusion model per frame.

**Table 3: Training time and used devices for different methods.**
| Method | Training time | GPU specs |
| --- | --- | --- |
| Shift-Net (Yan et al., ) | Not reported | 8 NVIDIA A100-32G GPUs |
| FMA-Net (Youk et al., ) | Not reported | Not reported |
| Upscale-A-Video (Zhou et al., ) | Not reported | 32 NVIDIA A100-80G GPUs |
| Ours | No training needed | - |

### A.4 Additional Ablation Studies

#### Comparison of temporal profiles.

The comparisons in Fig. 14 also indicate that our results are smoother, demonstrating better temporal stability.

Figure: Figure 14: Comparison of temporal profile. We examine a row of pixels and track changes over time. The profiles from Flow + Flow and Cosine + Flow methods exhibit noise, indicating flickering artifacts. The Cosine + Cosine method shows smoother profiles but contains some discontinuities. Flow + Cosine demonstrates improved consistency but retains some distortions. Utilizing flow, cosine, and spatial-aware techniques, our method achieves the most seamless and consistent transitions, effectively minimizing artifacts.
Refer to caption: x10.png

#### Token Unmerging Strategies.

We experimented with two unmerging strategies: averaging paired tokens and direct replacement with keyframe tokens. Tab. 4 shows the results of these experiments on the Vid4 x4 SR task.
As shown in the table, the replacement method outperforms averaging in terms of LPIPS, indicating better perceptual quality. Our experiments consistently showed that averaging tends to produce blurrier outputs in restoration tasks. Based on these results, we adopted the replacement-based unmerging process in our final model, as it preserves more details and leads to sharper outputs.

**Table 4: Quantitative comparisons of different unmerging methods on Vid4 x4 SR task.**
| Unmerging Method | LPIPS $\downarrow$ |
| --- | --- |
| Averaging | 0.337 |
| Replacement | 0.329 |

#### Limitations: Extreme Degradation

Extreme degradation (*e.g.*, 32$\times$ super-resolution) or overly detailed facial features may yield unsatisfactory results (Fig. 15).
However, our framework’s adaptability allows the incorporation of future, more powerful image-based diffusion models. Future improvements will focus on refining keyframe selection, stabilizing decoder output across LDM architectures, and enhancing extreme degradation handling. These aim to improve practical application and mitigate flickering issues inherent in LDM decoders.

Figure: Figure 15: Failure case under 32x SR. Most methods fail under this extreme degradation. However, if more powerful image-based diffusion models emerge in the future, our method can be easily adapted, offering greater potential to achieve this task.
Refer to caption: extracted/5898068/figures/rebuttal/failure/DDNM_00000.jpg

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: 1. The method is training-free which makes it computationally practical.
summary: This paper proposes a method for zero-shot video restoration using pre-trained image restoration diffusion models. It combines a hierarchical latent warping strategy for keyframes and local frames and token merging that integrates spatial information, optical flow, and feature-based matching.  The method acheives competitive results on zero-shot video restoration tasks such as super-resolution and denoising.
weaknesses: 1. To my understanding, the method relies on certain architectural heuristics inspired by the video-editing literature rather than on a solid mathematical framework. Since the method involves no training, there is no theoretical guarantee that it won't fail under certain conditions.

2. The method combines existing ideas from video editing, which on my opinion is still acceptable; however, this limits its novelty.

3. The improvement over the baseline on some datasets, particularly in the case of denoising, is not significant.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Degradations that are not encountered during optical flow training may introduce significant errors. How does the proposed method address this issue, and what motivated the choice of the GMFlow network for this approach?
- In Equation (5), is there a specific reason for including f_{src -> tar} (X(T_{src})) twice? 
- Including additional real-world datasets (e.g., [1, 2]) in the evaluation would help assess the method's generalization and robustness. 
- Could you provide a comparison of inference times with other methods?

Reference

[1] Towards Real-world Video Face Restoration: A New Benchmark, cvprw 2024

[2] Toward convolutional blind denoising of real photographs, cvpr 2019
soundness: 3
strengths: - The paper is well-structured, making the methodology and findings easy to understand.
- The method achieves competitive results without requiring additional training.
summary: This paper presents a zero-shot video restoration method using pre-trained image restoration diffusion models. The proposed approach introduces a hierarchical latent warping strategy and a hybrid flow-guided token merging approach to enhance both temporal consistency and visual detail. Experimental results demonstrate competitive performance in video super-resolution, denoising, showing advantages over state-of-the-art methods. An ablation study further highlights the importance of each proposed component.
weaknesses: - Both the latent warping and hybrid flow-guided token merging approaches rely heavily on optical flow information, which could be a limitation in cases where optical flow estimation is challenging or inaccurate.
- The paper's novelty is somewhat limited, as it primarily combines two existing methodologies with minor modifications. Specifically, the contributions include adjusting the range of warping frames at global and local levels and introducing a flow-guided confidence criterion for token merging.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. VidToMe is implemented on top of Stable Diffusion. In Table 1, the proposed approach (based on Stable Diffusion (SD x4)) is compared with VidToMe, but it does not seem to demonstrate performance gains over VidToMe in terms of objective image quality (PSNR/SSIM) or temporal consistency (E_warp). On the other hand, perceptual quality metrics, such as LPIPS, show improved results. Could you clarify if there is a specific reason why perceptual quality metrics like LPIPS perform better? If there is a trade-off involved, would it be possible to show how performance varies with adjustments to the hyperparameters?
2. Although the proposed method emphasizes compatibility with any 2D image restoration diffusion model, the comparison results in Table 1—before and after applying the proposed method on SD x4—show only modest performance gains, whereas the improvements over the DiffBIR baseline are more substantial. Could you please clarify if there is a specific reason for this difference or if there is any dependence on specific network architecture?
3. It appears that the code for Upscale-A-Video is now available. Since this approach also aims to improve temporal consistency across video frames generated by image generation models, it would be beneficial to include a comparison of the results. If the code is still unavailable, could you instead provide results on the datasets used in Upscale-A-Video and compare them with the figures reported in the Upscale-A-Video manuscript?
soundness: 3
strengths: The primary contribution of this approach is its ability to leverage conventional image generation models directly, without requiring modifications to network architecture or the need for retraining or fine-tuning. This is achieved through a straightforward yet effective technique: hierarchical token merging within the latent space, which ensures temporal consistency across generated video frames. By using this token merging strategy, the method successfully adapts static image models for dynamic video generation, maintaining coherence between frames without compromising quality. Furthermore, this approach reflects a well-engineered integration of recent advances in token merging, incorporating techniques from VidToMe and Upscale-A-Video. These advancement enhances the network’s capacity to address various video restoration challenges, effectively overcoming the limitations of traditional methods that struggle to maintain temporal consistency across frames.
summary: This paper introduces a versatile zero-shot video restoration approach leveraging pre-trained image restoration diffusion models. Unlike conventional methods that require retraining and often struggle with generalization, this method uses a hierarchical latent warping strategy for keyframes and local frames, along with a hybrid correspondence mechanism that merges tokens based on optical flow and token similarity. This approach demonstrates outstanding zero-shot performance, effectively generalizing across varied datasets and managing extreme degradations, including video super-resolution, denoising, and depth completion. Compatible with numerous 2D restoration diffusion models, the technique is validated through quantitative and visual benchmarks on challenging datasets, removing the need for extensive retraining.
weaknesses: First, the paper’s structure requires improvement, as the current organization makes it challenging to follow. 
A major concern is the lack of a comprehensive comparison between this approach and conventional methods, such as VidToMe and Upscale-A-Video. VidToMe introduces local and global token merging techniques, while Upscale-A-Video presents a flow-based merging approach—both key contributions relevant to this work. Please clarify how this approach differentiates itself from these methods, and refer to the “Questions” section for further details.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Is there a comparison of inference efficiency with baselines?
soundness: 3
strengths: 1. The proposed method is pioneering in achieving zero-shot video restoration by leveraging pre-trained image restoration diffusion models, enabling multiple video restoration tasks without additional training.
2. The method presents strong quantitative and visual results compared with state-of-the-art methods, balancing temporal consistency and detail generation.
summary: This paper proposes a training-free method to leverage pre-trained image restoration diffusion models for zero-shot video restoration. Specifically, the proposed method employs hierarchical latent warping and an enhanced token merging strategy to maintain temporal consistency and restore details across video frames. Experimental results demonstrate the versatility of the method across various tasks, including denoising, super-resolution, and depth estimation.
weaknesses: 1. In line 267, the authors claim that the combination of hierarchical latent warping and hybrid flow-guided spatial-aware token merging could achieve adaptation to various degradation types. However, it is not sufficiently discussed that why the combination could handle various degradation types.
2. While the use of optical flow guidance aligns with previous video restoration works [1,2], the paper introduces similarity-based guidance to capture correspondences distinct from optical flow. However, the specific benefits of similarity-based guidance over optical flow are not thoroughly discussed.
3. Although recent video restoration methods, such as Shift-Net and FMA-Net, are included as baselines, some classic methods like BasicVSR++ and RVRT are not compared in the experiments.

[1] Chan K C K, Zhou S, Xu X, et al. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 5972-5981.

[2] Liang J, Fan Y, Xiang X, et al. Recurrent video restoration transformer with guided deformable attention[J]. Advances in Neural Information Processing Systems, 2022, 35: 378-393.

Meta-review: This paper presents a zero-shot video restoration method with a pre-trained image-restoration diffusion model without additional training. A hierarchical latent warping strategy is used to improve temporal consistency. The token merging technique is used with the hybrid correspondence information that aggregates the spatial information, optical flow, and features. 

The reviewers agreed on the benefit of the training-free video restoration method as most other methods require additional training.

On the other hand, there were several concerns.
* The reviewers were concerned about the robustness against the severe input video degradation as the optical flow prediction error could cause inaccuracies. The authors argue that for optical flow prediction, GMFlow network was chosen due to its robustness and the hybrid correspondences handle the error by merging the confidently matched tokens pairs. As the authors point out, this is a combination of existing techniques and do not bring much novelty. Also, such hybrid correspondence family of techniques have been widely explored in the computer vision literature such as object tracking and video recognition.
* While the perceptual measure LPIPS is good, the fidelity measures PSNR/SSIM do not excel compared with VidToMe or more classical supervised methods. The authors did not explain why only LPIPS is emphasized by the proposed method other than the general perception-fidelity trade-off. While the token merging ratio can affect the preference on LPIPS over PSNR and SSIM, it is still unknown how the hyperparameter can be determined for users.

This paper can be improved by better presenting how the correspondence can be improved with a thorough survey and analysis of the existing works and by providing the understanding of the output characteristics.

Decision: Reject