ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler

Authors: Serin Yang, Taesung Kwon, Jong Chul Ye
Primary area: generative models
Keywords: Keyframe interpolation, Stable video diffusion, Bidirectional diffusion sampling

Abstract:
Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models has greatly enhanced video generation, especially in terms of keyframe interpolation. However, current image-to-video diffusion models, while powerful in generating videos from a single conditioning frame, need adaptation for two-frame (start \& end) conditioned generation, which is essential for effective bounded interpolation. Unfortunately, existing approaches that fuse temporally forward and backward paths in parallel often suffer from off-manifold issues, leading to artifacts or requiring multiple iterative re-noising steps. In this work, we introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames. Additionally, we incorporate advanced guidance techniques, CFG++ and DDS, to further enhance the interpolation process. By integrating these, our method achieves state-of-the-art performance, efficiently generating high-quality, smooth videos between keyframes. On a single 3090 GPU, our method can interpolate 25 frames at 1024$\times$576 resolution in just 195 seconds, establishing it as a leading solution for keyframe interpolation.
Project page: https://vibidsampler.github.io/

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730173720086

thread_type:
official_review

<thread_object>
creator: reviewer_hsxn

<content>
summary:
This paper proposes ViBiDSampler, a training-free method utilizing pre-trained image-to-video diffusion models for keyframe interpolation. They aim to tackle the off-manifold problem of previous studies, which leads to artifacts or costly processes. They propose bidirectional sampling, with a set of denoising-renoising-denoising with alternating conditioning on start and end frames. This way, they manage to keep the generated results stay on the original manifold. The additionally integrate advanced guidance methods like CFG++ and DDS to reach state-of-the-art performance.

strengths:
Strengths
- Overall, the paper is easy to follow, with coherent and clear problem definition and solution.
- Simple methodology. The proposed method, Bidirectional sampling seems to be easy to use.
- Efficiency. The proposed method does not require additional training and is fast in inference.

weaknesses:
1. Notations.
- The notations are confusing. For instance, $x_t$ seems to be used as a variable and a function. 
- The definitions could be further clarified. 
- Lines 247 - 256 was kind of confusing at first, with many similar looking notations without predefinitions, and mixed use of notations as functions and variables.


2. Limited contribution.

Although the proposed method is intuitive, simple, and quantitatively effective, some uncertainties remain.
- the 'Vanilla' setting at Table 1 and Figure 5, seems to show the effectiveness of bidirectional sampling solely, without guidance such as CFG++ and DDS. However, I am not fully convinced with its effectiveness, as Generative Inbetweening seems to outperform the vanilla version in Table 1. Also, in the qualitative results of Figure 5, the results of bidirectional sampling (vanilla) itself does not seem to be better than that of TRF.
- The concern above makes me wonder, if the superior performance of the entire framework proposed comes from the good guidance methods, CFG++ and DDS.
- In addition, CFG++ and DDS are manifold guidance methods proposed in other studies, and seems like this work simply took advantage of existing methods. I think it is hard to say that simply adoption of these guidance methods is a contribution.

In short, I am not fully convinced if the proposed bidirectional sampling itself is a powerful method compared to existing works, and suspect the superior performance could have come from the advanced manifold guidance methods, which, is hard to say to be a contribution.

questions:
1. To my understanding, CFG++ and DDS could possibly be applied to existing methods, such as Generative Inbetweening. If so, could the authors provide experiments on this, to ablate and show the importance of bidirectional sampling?

2. In bidirectional sampling, in some aspects, I think it is possible for this to work, but in some aspects, I don't. For example, taking the example of Fig.3(b), there are three steps, (1) denoising (w/ $c_{start}$), (2) renoising, (3) denoising (w/ $c_{end}$). I am curious how the generation process goes through these steps. Maybe a visualization would be helpful, showing how the start / end conditions are really affecting in those steps. I am specifically curious of this process in that the (2) renoising step could possibly remove the information of $c_{start}$ injected from (1) denoising, and the conditioning could have possibly be injected only at (3), making the steps (1) and (2) redundant. A further analysis / clarification on this part, that the proposed method surely takes both start / end frame conditioning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730353796446

thread_type:
official_review

<thread_object>
creator: reviewer_hjqi

<content>
summary:
This paper proposes a inference strategy to achieve frame interpolation w/o training. Its key idea is to use the pre-trained stable video diffusion model as a prior and complete the frame interpolation given the first and the last frame as a condition. The main improvement seems from the proposed bidirectional sampling combining with SOTA sampling approaches, i.e., DDS and CFG++.

strengths:
+The paper is easy to follow .
+The visualization results seem impressive comparing with other baselines provided in the paper.

weaknesses:
The contribution of this paper seems marginal. It seems the main contribution comes from eq. (9). However, the improvement of such modification seems not that obvious comparing with using SOTA sampling approaches including DDS and CFG++ from Table. 1, i.e., only "Ours (full)" shows obvious improvement comparing with other baselines but if my understanding is correct, other baselines seem not using DDS and CFG++.

questions:
My main concerns are mainly as follows:

1. The contribution of this paper seems marginal. It seems the main contribution comes from eq. (9). However, the improvement of such modification seems not that obvious comparing with using SOTA sampling approaches including DDS and CFG++ from Table. 1, i.e., only "Ours (full)" shows obvious improvement comparing with other baselines but if my understanding is correct, other baselines seem not using DDS and CFG++. The author should further verify the improvement of each technology used in the paper, i.e., bidirectional sampling, DDS and CFG++, e.g., providing more visualization as well as adding DDS and CFG++ to other baselines and show the improvement.

2. The practical use the proposed approach seems not convincing. The proposed approach still requires 192s to generate 25 frames, which is at least two times longer than directly inferencing a video generation model with similar computational cost. Moreover, I also wonder how to keep the consistency between frames when the required length of the output video is beyond 25.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692763420

thread_type:
official_review

<thread_object>
creator: reviewer_wfk9

<content>
summary:
This paper presented a bidirectional sampling strategy, sequential sampling along both forward and backward paths, for video interpolation. Additional manifold guidance such as DDS and CFG++ are further introduced to enhance the interpolation results. Experiments are conducted to evaluate the proposed ViBiDSampler.

strengths:
+ A bidirectional sampling strategy, sequential sampling along both forward and backward paths, for video interpolation. 
+ Additional manifold guidance such as DDS and CFG++ are further introduced to enhance the interpolation results. 
+ Training-free, convenient for use.
+ Experiments are conducted to evaluate the proposed ViBiDSampler.

weaknesses:
- The method is training-free. In general, training-based method can be more effective.
- More ablation study is suggested to assess the effect of pre-trained video generation models. Will better video generation models result in better performance?
- Discussion on the limitation and future work.

questions:
- The method is training-free. In general, training-based method can be more effective.
- More ablation study is suggested to assess the effect of pre-trained video generation models. Will better video generation models result in better performance?
- Discussion on the limitation and future work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731086296167

thread_type:
official_review

<thread_object>
creator: reviewer_vmqd

<content>
summary:
This work introduces a novel, bidirectional sampling strategy to address off-manifold issues without requiring extensive re-noising or fine-tuning. The proposed method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.

strengths:
- Unlike previous fusing strategies, which compute two conditioned outputs in parallel and then fuse them, the proposed bidirectional diffusion sampling strategy samples two conditioned outputs sequentially, which mitigates the off-manifold issue.
- The proposed method can further work with advanced on-manifold guidance techniques (CFG++, DDS) to produce more reliable interpolation results that have better alignment of the input frames.
- This is a training-free method which can efficiently interpolate between two keyframes to generate a 25-frame video at 1024×576 resolution in just 195 seconds on a single 3090 GPU.

weaknesses:
- The experimental results presented lack PSNR scores. While accurate recovery of intermediate frames may not be the primary objective of this study, and high PSNR values might not necessarily indicate superior human preferences, it is still beneficial to include the score to provide readers with a clearer understanding of the work’s potential.
- The proposed bidirectional sampling addresses the off-manifold issue, but it comes with the potential for deviation from the start or end frame, depending on the order of sequential sampling. To mitigate this issue, the authors have incorporated previous guidance techniques. However, I would appreciate a more comprehensive video comparison in the ablation study, in addition to the provided static comparison on a single scene.

questions:
- How to incorporate text conditioning into your framework so users can control the action in the generated frames?
- The results and presentation are sound, except for some minor issues (see weaknesses). I will increase my score once the authors address my concerns.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732077386664

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We would like to thank the reviewers for their constructive and thorough reviews.

We are encouraged by the positive reception, as highlighted by the reviewers: **the soundness of our results and presentation** (VMQD), **the convenient, training-free approach of our method** (wFk9), **the impressive results compared to other baselines** (hjqi), and **the simplicity and efficiency of our methodology** (hsxN).


We have made the following major revisions to address the concerns raised from the reviewers.

**1. Additional ablation studies on the guidance methods, CFG++ and DDS.**

In response to the reviewers' suggestions to validate the effectiveness of bidirectional sampling, we conducted ablation studies by incorporating CFG++ and DDS to various baseline models. Please refer to the anonymous webpage (https://vibid.github.io/supple) and the revised Appendix B.2 for experimental details. As shown in the webpage, our vanilla bidirectional sampling shows superior perceptual quality compared to TRF. Furthermore, as shown in the revised Appendix B.2, the videos generated by our proposed method consistently outperform all baseline methods with CFG++ and DDS. The results demonstrate the effectiveness of bidirectional sampling in video interpolation.

**2. More comprehensive video comparisons.**

In response to the suggestions from reviewers VMQD and hjqi, we included more visualization results regarding ablation study on the building blocks. The videos are available at the anonymous webpage (https://vibid.github.io/supple). The roles of each building blocks can be verified from the visualizations and Appendix B.3.

For point-to-point response, please refer to below.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: Proposes a frame interpolation method that introduces a novel bi-directional sampling approach, that avoids off-manifold issues, in a training-free manner. It sequentially samples along the forward and backward paths, each conditioning on the start and end frames. Additionally, manifold guidance such as DDS and CFG++ are employed to synergically enhance interpolation results. 

Strength:  The bi-directional sampling strategy is novel, and it’s also simple and shown to be effective in qualitative results. Its training-free nature makes it an alternative approach to high-quality frame interpolation without resorting to expensive training or fine-tuning approaches. The paper is written very well, the problem is well motivated, which makes it easy to follow design choices and experimental results. Authors present extensive experimental comparisons that demonstrate the effectiveness of their approach. 

Weakness: It currently is not controllable via text. However, authors have identified this as a future work. 

Acceptance Reason: See strength. It is a novel approach that shows strong results using a simple and efficient technique. Future works and limitations sections outline a promising avenue for future work.