ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler

## Abstract

Abstract Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models has greatly enhanced video generation, especially in terms of keyframe interpolation.
However, current image-to-video diffusion models, while powerful in generating videos from a single conditioning frame, need adaptation for two-frame (start & end) conditioned generation, which is essential for effective bounded interpolation.
Unfortunately, existing approaches that fuse temporally forward and backward paths in parallel often suffer from off-manifold issues, leading to artifacts or requiring multiple iterative re-noising steps.
In this work, we introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning.
Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively,
ensuring more coherent and on-manifold generation of intermediate frames.
Additionally, we incorporate advanced guidance techniques, CFG++ and DDS, to further enhance the interpolation process.
By integrating these, our method achieves state-of-the-art performance, efficiently generating high-quality, smooth videos between keyframes.
On a single 3090 GPU, our method can interpolate 25 frames at 1024 × \times × 576 resolution in just 195 seconds, establishing it as a leading solution for keyframe interpolation.
Project page: https://vibid.github.io/

## 1 Introduction

Recent advancements in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models (Blattmann et al., ; ; Wu et al., ; Xing et al., ; Bar-Tal et al., ) have made it possible to generate high-quality videos that closely match a given text or image conditions.
Various efforts have been made to leverage the powerful generative capabilities of these video diffusion models, especially in the context of keyframe interpolation, to improve perceptual quality significantly.
Specifically, diffusion-based keyframe interpolation (Voleti et al., ; Danier et al., ; Huang et al., ; Feng et al., ; Wang et al., ) focuses on generating intermediate frames between two keyframes, aiming to create smooth and natural motion dynamics while preserving the keyframes’ visual fidelity and appearance.
Image-to-video diffusion models are particularly well-suited for this task because they are designed to maintain the visual quality and consistency of the initial conditioning frame.

While image-to-video diffusion models are designed for start-frame conditioned video generation, they need to be adapted for start and end frame conditioned video generation for keyframe interpolation.
One line of works (Feng et al., ; Wang et al., ) addresses this issue by introducing a new sampling strategy that fuses the intermediate samples of the temporally forward path, conditioned on the start frame, and the temporally backward path, conditioned on the end frame.
The fusing strategy ensures smooth and coherent frame generation in-between two keyframes using image-to-video diffusion models in a training-free (Feng et al., ) or a lightweight fine-tuning manner (Wang et al., ).

In the geometric view of diffusion models (Chung et al., ), the sampling process is typically described as iterative transitions $\mathcal{M}_{t}\to\mathcal{M}_{t-1},\>t=T,\cdots,1$, moving from the noisy manifold $\mathcal{M}_{T}$ to the clean manifold $\mathcal{M}_{0}$.
From this perspective, fusing two intermediate sample points through linear interpolation on a noisy manifold can lead to an undesirable off-manifold issue, where the generated samples deviate from the learned data distribution.
TRF (Feng et al., ) reported that this fusion strategy often results in undesired artifacts.
To address these discrepancies, they apply multiple rounds of re-noising and denoising to the fused samples, which may help correct the off-manifold deviations.

Unlike the prior works, here we introduce a simple yet effective sampling strategy to address off-manifold issues.
Specifically, at timestep $t$, we first denoise ${\bm{x}}_{t}$ to obtain ${\bm{x}}_{t-1}$ along the temporally forward path, conditioned on the start frame ($I_{\text{start}}$). Then, we re-noise ${\bm{x}}_{t-1}$ back to ${\bm{x}}_{t}$ using stochastic noise. After that, we denoise ${\bm{x}}^{\prime}_{t}$ to get ${\bm{x}}^{\prime}_{t-1}$ along the temporally backward path, conditioned on the end frame ($I_{\text{end}}$), where the ^′ notation indicates that the sample has been flipped along the time dimension. Unlike the fusing strategy, which computes two conditioned outputs in parallel and then fuses them, our bidirectional diffusion sampling strategy samples two conditioned outputs sequentially, which mitigates the off-manifold issue.

Furthermore, we incorporate advanced on-manifold guidance techniques to produce more reliable interpolation results. First, we employ the recently proposed CFG++ (Chung et al., ), which addresses the off-manifold issues inherent in Classifier-Free Guidance (CFG) (Ho & Salimans, ).
Second, we incorporate DDS guidance (Chung et al., ) to ensure proper alignment of the last frame of the generated samples with the given frames, as the ground-truth start and end frames are already provided.
By combining bidirectional sampling with these guidance techniques, our method achieves stable, state-of-the-art keyframe interpolation performance without requiring fine-tuning or multiple re-noising steps. Thanks to its efficient sampling strategy, our method can interpolate between two keyframes to generate a 25-frame video at 1024$\times$576 resolution in just 195 seconds on a single 3090 GPU.
Since our method is designed for high-quality and vivid video keyframe interpolation using bidirectional diffusion sampling, we refer to it as Video Interpolation using BIdirectional Diffusion (ViBiD) Sampler.

## 2 Related Works

Video interpolation.
Video interpolation is a task that generates the intermediate frames based on two bounding frames. Conventional interpolation methods have utilized convolutional neural networks (Kong et al., ; Li et al., ; Lu et al., ; Huang et al., ; Zhang et al., ; Reda et al., ), which are typically trained in a supervised manner to estimate the optical flows for synthesizing an intermediate frame. However, they primarily focus on minimizing $L_{1}$ or $L_{2}$ distances between the output and target frames, emphasizing high PSNR values at the expense of perceptual quality. Furthermore, the train datasets generally consist of high frame rate videos, limiting the model’s ability to learn extreme motion effectively.

Diffusion-based methods and time reversal sampling.
Diffusion-based methods have been proposed (Danier et al., ; Huang et al., ; Voleti et al., ) to leverage the generative priors of diffusion models to produce high-quality perceptual intermediate frames. Although these methods demonstrate improved perceptual performance, they still struggle with interpolating frames that contain significant motion. However, video keyframe interpolation methods that build on the robust performance of video diffusion models have been more successful in handling ambiguous and non-linear motion (Xing et al., ; Jain et al., ), largely due to the incorporation of the temporal attention layers in these models (Blattmann et al., ; Ho et al., ; Chen et al., ; Zhang et al., ).

Recent advancements in video diffusion models, particularly for image-to-video diffusion, have introduced new sampling techniques that leverage temporal and perceptual priors. These techniques reverse video frames in parallel during inference and fuse bidirectional motion from both the temporally forward and backward directions. TRF (Feng et al., ) proposed a method that combines forward and backward denoising processes, each conditioned on the start and end frames. Similarly, Generative Inbetweening (Wang et al., ) introduced a method that extracts temporal self-attention maps and rotates them to sample reversed frames, enhancing video quality by fine-tuning diffusion models for reversed motion.
However, these methods rely on a fusion strategy that often results in an off-manifold issue.
Moreover, although methods such as multiple noise injections and model fine-tuning have been employed to address these challenges, they continue to exhibit off-manifold issues and substantially increase computational costs.
In contrast, we introduce a simple yet effective sampling strategy that eliminates the need for multiple noise injections or model fine-tuning.

## 3 Video Interpolation using Bidirectional Diffusion

Figure: Figure 2: Comparison of denoising processes. (a) Time Reversal Fusion method and (b) bidirectional sampling (Ours).
Refer to caption: extracted/5906086/figures/fig_method.jpg

Although our method is applicable to general video diffusion models, we employ Stable Video Diffusion (SVD) (Blattmann et al., ) as a proof of concept in this paper. By introducing SVD, we aim to provide a clearer understanding of our approach.
SVD is a latent video diffusion model employed in EDM-framework (Karras et al., ) with micro-conditioning (Podell et al., ) on frame rate (fps). For the image-to-video model, SVD replaces text embeddings with the CLIP image embedding (Radford et al., ) of the conditioning.

In EDM-framework, the denoiser ${\bm{D}}_{\theta}$ computes the denoised estimate from the U-Net ${\bm{\epsilon}}_{\theta}$:

$$ $\displaystyle{\bm{D}}_{\theta}({\bm{x}};\sigma,{\bm{c}})=c_{\text{skip}}( \sigma){\bm{x}}+c_{\text{out}}(\sigma){\bm{\epsilon}}_{\theta}\left(c_{\text{ in}}(\sigma){\bm{x}};c_{\text{noise}}(\sigma),{\bm{c}}\right),$ (1) $$

where $c_{\text{skip}}$, $c_{\text{out}}$, $c_{\text{in}}$, and $c_{\text{noise}}$ are $\sigma$-dependent preconditioning parameters and ${\bm{c}}$ is the condition.
In practice, the denoiser ${\bm{D}}_{\theta}$ takes concatenated inputs [${\bm{x}}$, ${\bm{x}}$] to return ${\bm{c}}$-conditioned estimate and null-conditioned estimate [$\hat{{\bm{x}}}_{c}({\bm{x}})$, $\hat{{\bm{x}}}_{\varnothing}({\bm{x}})$], where $\hat{{\bm{x}}}_{c}$ is then updated using $\omega$-scale classifier-free guidance (CFG) (Ho & Salimans, ):

$$ $\displaystyle\hat{{\bm{x}}}_{{\bm{c}}}({\bm{x}})\leftarrow\hat{{\bm{x}}}_{ \varnothing}({\bm{x}})+\omega[\hat{{\bm{x}}}_{{\bm{c}}}({\bm{x}})-\hat{{\bm{x} }}_{\varnothing}({\bm{x}})].$ (2) $$

For sampling, SVD employs Euler-step to gradually denoise from Gaussian noise ${\bm{x}}_{T}$ to get ${\bm{x}}_{0}$:

$$ $\displaystyle{\bm{x}}_{t-1}({\bm{x}}_{t};\sigma_{t},{\bm{c}}):=\hat{{\bm{x}}}_ {{\bm{c}}}({\bm{x}}_{t})+\frac{\sigma_{t-1}}{\sigma_{t}}({\bm{x}}_{t}-\hat{{ \bm{x}}}_{{\bm{c}}}({\bm{x}}_{t})),$ (3) $$

where $\hat{{\bm{x}}}_{{\bm{c}}}({\bm{x}}_{t})$ is the denoised estimate from (2) and $\sigma_{t}$ is the discretized noise level for each timestep $t\in[0,T]$.

### 3.1 Bidirectional Sampling

Prior approaches such as TRF (Feng et al., ) and Generative Inbetweening (Wang et al., ) have employed a fusion strategy that linearly interpolates between samples from the temporally forward path, conditioned on the start frame ($I_{\text{start}}$), and the temporally backward path, conditioned on the end frame ($I_{\text{end}}$):

$$ $\displaystyle{\bm{x}}_{t-1,{\bm{c}}_{\text{start}}}$ $\displaystyle={\bm{x}}_{t-1}({\bm{x}}_{t};\sigma_{t},{\bm{c}}_{\text{start}}),$ (4) $\displaystyle{\bm{x}}^{\prime}_{t-1,{\bm{c}}_{\text{end}}}$ $\displaystyle={\bm{x}}_{t-1}({\bm{x}}^{\prime}_{t};\sigma_{t},{\bm{c}}_{\text{ end}}),$ (5) $\displaystyle{\bm{x}}_{t-1}$ $\displaystyle=\lambda{\bm{x}}_{t-1,{\bm{c}}_{\text{start}}}+(1-\lambda)({\bm{x }}^{\prime}_{t-1,{\bm{c}}_{\text{end}}})^{\prime},$ (6) $$

where the ^′ notation indicates that the sample has been flipped along the time dimension, $\lambda$ denotes interpolation ratio, ${\bm{c}}_{\text{start}}$ and ${\bm{c}}_{\text{end}}$ denotes the encoded latent condition of $I_{\text{start}}$ and $I_{\text{end}}$, respectively.
However, as the authors in TRF (Feng et al., ) reported, the vanilla implementation of this fusion strategy suffers from random dynamics and unsmooth transitions.
This occurs because linearly interpolating between two distinct sample points in the noisy manifold $\mathcal{M}_{t}$ can cause the deviation from the original manifold, as illustrated in Fig. 3 (a).

In this work, we aim to leverage the image-to-video diffusion model (SVD) for keyframe interpolation tasks, eliminating the multiple noise injections or model fine-tuning.
Notably,
our key innovation lies in the sequential sampling of the temporally forward path of ${\bm{x}}_{t}$ and the temporally backward path of ${\bm{x}}^{\prime}_{t}:=\text{flip}({\bm{x}}_{t})$ by integrating a single re-noising step between them:

$$ $\displaystyle{\bm{x}}_{t-1,{\bm{c}}_{\text{start}}}$ $\displaystyle={\bm{x}}_{t-1}({\bm{x}}_{t};\sigma_{t},{\bm{c}}_{\text{start}}),$ (7) $\displaystyle{\bm{x}}_{t,{\bm{c}}_{\text{start}}}$ $\displaystyle={\bm{x}}_{t-1,{\bm{c}}_{\text{start}}}+\sqrt{\sigma_{t}^{2}- \sigma_{t-1}^{2}}\epsilon,$ (8) $\displaystyle{\bm{x}}^{\prime}_{t-1}$ $\displaystyle={\bm{x}}_{t-1}({\bm{x}}^{\prime}_{t,{\bm{c}}_{\text{start}}}; \sigma_{t},{\bm{c}}_{\text{end}}),$ (9) $\displaystyle{\bm{x}}_{t-1}$ $\displaystyle=\left({\bm{x}}^{\prime}_{t-1}\right)^{\prime}.$ (10) $$

This approach effectively constrains the sampling process for bounded generation between the start frame ($I_{\text{start}}$) and the end frame ($I_{\text{end}}$).
As depicted in Fig. 3 (b), our method seamlessly connects the temporally forward and backward paths so that the sampling trajectory stays within the SVD manifold, resulting in smooth and coherent transitions throughout the interpolation process.

Figure: Figure 3: Comparison of diffusion sampling paths. (a) Existing methods encounter off-manifold issues due to the averaging of two sample points. (b) In contrast, our bidirectional sampling sequentially connects the temporally forward and backward paths, ensuring that the process remains within the manifold.
Refer to caption: extracted/5906086/figures/fig_manifold.jpg

### 3.2 Additional manifold guidances

We further employ recent advanced manifold guidance techniques to enhance the interpolation performance of the bidirectional sampling. First, we introduce additional frame guidance using DDS (Chung et al., ). Then, we replace traditional CFG (Ho & Salimans, ) with CFG++ (Chung et al., ) to mitigate the off-manifold issue of CFG in the original implementation of SVD (Blattmann et al., ).

Last frame guidance with DDS.
DDS (Chung et al., ) synergistically combines the diffusion sampling and Krylov subspace methods (Liesen & Strakos, ) such as the conjugate gradient (CG) method, guaranteeing the on-manifold solution of the following optimization problem:

$$ $\displaystyle\min_{{\bm{x}}\in\mathcal{M}}\ell({\bm{x}}):=\|{\bm{y}}-{\mathcal {A}}({\bm{x}})\|^{2},$ (11) $$

where ${\mathcal{A}}$ is the linear mapping, ${\bm{y}}$ is the condition, and $\mathcal{M}$ represents the clean manifold of the diffusion sampling path.

Here, we leverage the DDS framework to guide a start-frame-conditioned sampling path of SVD to a start- and end-frame-conditioned sampling path. Specifically, for the temporally forward path, conditioned on the start frame ($I_{\text{start}}$), we take the DDS step on denoised estimate $\hat{{\bm{x}}}_{{\bm{c}}_{\text{start}}}({\bm{x}}_{t})$ to guide the last frame of $\hat{{\bm{x}}}_{{\bm{c}}_{\text{start}}}({\bm{x}}_{t})$ to align with ${\bm{c}}_{\text{end}}$. For the temporally backward path, conditioned on the end frame ($I_{\text{end}}$), we take the DDS step on denoised estimate $\hat{{\bm{x}}}^{\prime}_{{\bm{c}}_{\text{end}}}({\bm{x}}^{\prime}_{t})$ to guide the last frame of $\hat{{\bm{x}}}^{\prime}_{{\bm{c}}_{\text{end}}}({\bm{x}}^{\prime}_{t})$ to align with ${\bm{c}}_{\text{start}}$. In practice, we set ${\mathcal{A}}({\bm{x}}):={\bm{x}}_{\text{last}}$ as last frame extractor, ${\bm{y}}$ as matched condition which are ${\bm{c}}_{\text{end}}$ for temporally forward path and ${\bm{c}}_{\text{start}}$ for temporally backward path:

$$ $\displaystyle\bar{\bm{x}}_{{\bm{c}}_{\text{start}}}:=\operatorname*{arg\,min}_ {{\bm{x}}\in\hat{{\bm{x}}}_{{\bm{c}}_{\text{start}}}+\bm{\mathcal{K}}_{l}}\|{ \bm{c}}_{\text{end}}-{\mathcal{A}}({\bm{x}})\|^{2},\quad\bar{\bm{x}}^{\prime}_ {{\bm{c}}_{\text{end}}}:=\operatorname*{arg\,min}_{{\bm{x}}\in\hat{{\bm{x}}}^{ \prime}_{{\bm{c}}_{\text{end}}}+\bm{\mathcal{K}}_{l}}\|{\bm{c}}_{\text{start}} -{\mathcal{A}}({\bm{x}})\|^{2},$ (12) $$

where $\bm{\mathcal{K}}_{l}$ is the $l$-th order Krylov subspace, in which Krylov subspace methods seek an approximate solution.
By leveraging this DDS framework, we effectively guide the sampling process toward a path conditioned by both the start and end frames, which is particularly effective for keyframe interpolation.

Better Image-Video alignment with CFG++. Recent advances of CFG++ (Chung et al., ) tackles the inherent off-manifold issue in CFG (Ho & Salimans, ). Specifically, CFG++ mitigates this undesirable off-manifold issue using the unconditional score instead of the conditional score in a re-noising process of CFG.
By using the unconditional score, CFG++ can overcome the off-manifold phenomena in CFG-generated samples,
resulting in better text-image alignment for text-to-image generation tasks.

While SVD replaces text embedding with CLIP image embedding, we can still use CFG++ for image-to-video diffusion models to ensure better image-video alignment. Specifically, after applying CFG++ into SVD sampling framework, the Euler-step of SVD (3) now reads:

$$ $\displaystyle{\bm{x}}_{t-1}({\bm{x}}_{t};\sigma_{t},{\bm{c}}):=\hat{{\bm{x}}}_ {{\bm{c}}}({\bm{x}}_{t})+\frac{\sigma_{t-1}}{\sigma_{t}}({\bm{x}}_{t}-\hat{{ \bm{x}}}_{\varnothing}({\bm{x}}_{t})),$ (13) $$

where the last term in (3) is replaced by $\hat{{\bm{x}}}_{\varnothing}({\bm{x}}_{t})$.
In practice, we apply DDS guidance before CFG++ update, so $\hat{{\bm{x}}}_{{\bm{c}}}({\bm{x}}_{t})$ in (13) should be replaced with $\bar{{\bm{x}}}_{{\bm{c}}}({\bm{x}}_{t})$ as in (12).
We experimentally found that incorporating DDS and CFG++ guidance improves the interpolation performance of bidirectional sampling.
The overall sampling method effectively steers the SVD sampling path to perform keyframe interpolation in an on-manifold manner, fully leveraging the generation capabilities of SVD. The detailed algorithm is provided in Algorithm 1. The vanilla bidirectional sampling can be implemented by removing DDS guidance (orange) and replacing the CFG++ update (blue) with a traditional CFG update. The detailed algorithm of the vanilla bidirectional sampling is provided in Appendix A.

Figure: Algorithm 1 Bidirectional sampling (Full)

Figure: Figure 4: Qualitative evaluation compared to three baselines: FILM, TRF, and Generative Inbetweening. The start and end frames ($I_{0}$, $I_{24}$) are used as keyframes. While FILM encounters difficulties in capturing motion when there is a significant discrepancy between the two keyframes, and TRF and Generative Inbetweening experience a decline in perceptual quality due to the blurring of objects within the image, our method successfully captures motion while maintaining high fidelity in the generated images.
Refer to caption: extracted/5906086/figures/fig_qualitative.jpg

## 4 Experimental Results

### 4.1 Experimental setting

Dataset.
The high-resolution (1080p) video datasets used for evaluation are sourced from the DAVIS dataset (Pont-Tuset et al., ) and the Pexels dataset(^1^11[https://www.pexels.com/](https://www.pexels.com/)). For the DAVIS dataset, we preprocessed 100 videos into 100 video-keyframe pairs, with each video consisting of 25 frames. This dataset includes a wide range of large and varied motions, such as surfing, dancing, driving, and airplane flying. For the Pexels dataset, we collected 45 videos, primarily featuring scene motions, natural movements, directional animal movements, and sports actions. We used the first and last frames from each video as keyframes for our evaluation.

Implementation Details.
For the sampling process, we used the Euler scheduler with 25 timesteps for both forward and backward sampling.
The motion bucket ID was fixed at 127, and the decoding frame number was set to 4 due to memory limitations on an NVIDIA RTX 3090 GPU. All other parameters followed the default settings from SVD.
Since micro-condition fps is sensitive to the data, we applied a lower fps for cases with large motion and a higher fps for cases with smaller motion. While both DDS and CFG++ generally improve the results, the choice between them depends on the specific use case.
All evaluations were performed on a single NVIDIA RTX 3090.

### 4.2 Comparative studies

We conducted a comparative study with four different keyframe interpolation baselines, including FILM (Reda et al., ), a conventional flow-based frame interpolation method, and three frame interpolation methods based on video diffusion models: TRF (Feng et al., ), DynamiCrafter (Xing et al., ), and Generative Inbetweening (Wang et al., ). We conducted these studies using the official implementations with default values, except for TRF, which has not been open-sourced yet.

Qualitative evaluation.
As illustrated in Fig. 4, our model clearly outperforms the other methods in terms of motion consistency and identity preservation. Other baselines struggle to accurately predict the motion between the two keyframes when there is a significant difference in content. For example, in Fig. 4, the first frame shows the tip of an airplane, while the last frame reveals the airplane’s body. In this case, FILM fails to produce a linear motion path, instead showing the airplane’s shape converging toward the middle frame from both end frames, resulting in the airplane’s body being disconnected by the 18th frame. While TRF and Generative Inbetweening show sequential movement, the airplane’s shape becomes distorted. In contrast, our method preserves the airplane’s shape while effectively capturing its gradual motion. Furthermore, it can be observed from the second and third cases from Fig. 4 that our method generates temporally coherent results while semantically adhering to the input frames. In TRF, the shapes of the robot and the dog become blurred due to the denoising paths deviating from the manifold during the fusion process, leading to artifacts in the image. While Generative Inbetweening mitigated this off-manifold issue through temporal attention rotation and model fine-tuning, artifacts still persist. In contrast, our method preserves the shapes of both the robot and the dog, generating frames with strong temporal consistency.

Table: Table 1: Quantitative evaluation on DAVIS and Pexels datasets. We compared our method against four different baselines and conducted ablation studies to assess the impact of CFG++ and DDS. Ours (Vanilla) refers to the bidirectional sampling method utilizing traditional CFG update without DDS guidance. Ours (Vanilla w/ CFG++) refers to the bidirectional sampling method with CFG++ update, also without DDS guidance. Bold and underline refer to the best and the second best, respectively.

Figure: Figure 5: Ablation study on the effects of CFG++ and DDS. The inclusion of CFG++ and DDS results in improved perceptual quality in the generated frames.
Refer to caption: extracted/5906086/figures/fig_ablation_golf.jpg

Quantitative evaluation.
For quantitative evaluation, we used LPIPS (Zhang et al., ) and FID (Heusel et al., ) to assess the quality of the generated frames, and FVD (Unterthiner et al., ) to evaluate the overall quality of the generated videos.
As shown in Table 1, our method surpasses the other baselines in terms of fidelity. Moreover, it achieves superior perceptual quality, particularly in scenarios involving dynamic motions (DAVIS), indicating that our approach effectively addresses the issue of deviations from the diffusion manifold, resulting in improved video generation quality.

Table: Table 2: A comprehensive comparison of our method with other diffusion-based approaches.

Figure: Figure 6: Effect of CFG++ guidance scale. The rows, from top to bottom, correspond to the CFG++ scales of $0.6$, $0.8$, and $1.0$.
Refer to caption: extracted/5906086/figures/fig_cfg++.jpg

Table: Table 3: Quantitative analysis on CFG++ guidance scale $\omega$. Effective results are obtained at the scale of $1.0$.

### 4.3 Computational efficiency

We performed comparative studies on the computational cost of diffusion models, as presented in Table 2. In the training stage, DynamiCrafter undergoes additional training with a large-scale image-to-video model for the frame interpolation task, while Generative Inbetweening also necessitates SVD model fine-tuning, both of which demands significant computational resources. During the inference stage, both TRF and Generative Inbetweening generate videos in $25\sim 50$ steps for each forward and backward direction, with additional noise injection steps that further increase the number of function evaluations (NFE) and inference time. However, our method does not require additional training or fine-tuning and completes the process in just $25$ steps per direction, without requiring multiple re-noising.

### 4.4 Ablation studies

Bidirectional sampling and conditional guidance.
The effectiveness of bidirectional sampling can be validated in the vanilla version without any conditional guidance, such as CFG++ or DDS. As demonstrated in Table 1, our vanilla model outperforms TRF across all three metrics, supporting the claim that fusing time-reversed denoising paths leads to off-manifold issues, which our method addresses through bidirectional sampling. In addition, with conditional guidance from CFG++ and DDS, we could achieve even better results and outperform DynamicCrafter and Generative Inbetweening which further train the image-to-video models.
This is consistent with Fig. 5, which illustrates that frames generated by TRF exhibit blurry shapes of the golfer and unnecessary camera movement. In contrast, the body shape of the golfer and the golf club are progressively better preserved as additional conditional guidance is incorporated.

CFG++ guidance scale.
As shown in Fig. 6, at higher CFG++ scales, the semantic information of the input frames is better preserved in the generated intermediate frames, resulting in improved fidelity. For instance, while the small person in the first input frame disappears in the intermediate frames at CFG++ scales of $0.6$ and $0.8$, the person remains visible across all the intermediate frames at a scale of $1.0$. Additionally, as the CFG++ scale decreases, the blurriness of the chairlift in the output frames gradually worsens.
This aligns with the findings presented in Table 3. The LPIPS, FID, and FVD values are lowest at a CFG++ scale of $1.0$ and highest at a scale of $0.6$, indicating that CFG++ contributes to improving the perceptual quality of the generated videos.

Figure: Figure 7: Application to keyframe interpolation with various boundary conditions. The end image a is identical to the start image. End images b-c represent dynamic boundaries sampled from different time points.
Refer to caption: extracted/5906086/figures/fig_identical2.jpg

### 4.5 Identical and dynamic bounds

Our method is applicable not only to dynamic bounds, where the start and end frames are different, but also to static bounds, where the start and end frames are identical. As illustrated in Fig. 7, our method successfully generates temporally coherent videos with identical start and end images (a). For example, the wave line also consistently fluctuates with the progression of time. Furthermore, as seen in the fifth and sixth rows of Fig. 7, our method effectively generates intermediate frames based on varying end frames (b and c). Given that the end images of the two rows differ, the resulting intermediate frames are generated accordingly.

## 5 Conclusion

We present Video Interpolation using Bidirectional Diffusion Sampler (ViBiDSampler), a novel approach for keyframe interpolation that leverages bidirectional sampling and advanced manifold guidance techniques to address off-manifold issues inherent in time-reversal-fusion-based methods. By performing denoising sequentially in both forward and backward directions and incorporating CFG++ and DDS guidance, ViBiDSampler offers a reliable and efficient framework for generating high-quality, temporally coherent, and vivid video frames without requiring fine-tuning or repeated re-noising steps. Our method achieves state-of-the-art performance in keyframe interpolation, as evidenced by its ability to generate 25-frame video at high resolution in a short processing time.

## Appendix A Algorithm

Figure: Algorithm 2 Bidirectional sampling (Vanilla)

## Appendix B Additional experimental results

Figure: Figure 8: Additional comparison with baseline methods.
Refer to caption: extracted/5906086/figures/suppl_qualitative1.jpg

Figure: Figure 9: Additional comparison with baseline methods.
Refer to caption: extracted/5906086/figures/suppl_qualitative2.jpg

Figure: Figure 10: Additional comparison with baseline methods.
Refer to caption: extracted/5906086/figures/suppl_qualitative3.jpg

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: - How to incorporate text conditioning into your framework so users can control the action in the generated frames?
- The results and presentation are sound, except for some minor issues (see weaknesses). I will increase my score once the authors address my concerns.
soundness: 3
strengths: - Unlike previous fusing strategies, which compute two conditioned outputs in parallel and then fuse them, the proposed bidirectional diffusion sampling strategy samples two conditioned outputs sequentially, which mitigates the off-manifold issue.
- The proposed method can further work with advanced on-manifold guidance techniques (CFG++, DDS) to produce more reliable interpolation results that have better alignment of the input frames.
- This is a training-free method which can efficiently interpolate between two keyframes to generate a 25-frame video at 1024×576 resolution in just 195 seconds on a single 3090 GPU.
summary: This work introduces a novel, bidirectional sampling strategy to address off-manifold issues without requiring extensive re-noising or fine-tuning. The proposed method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.
weaknesses: - The experimental results presented lack PSNR scores. While accurate recovery of intermediate frames may not be the primary objective of this study, and high PSNR values might not necessarily indicate superior human preferences, it is still beneficial to include the score to provide readers with a clearer understanding of the work’s potential.
- The proposed bidirectional sampling addresses the off-manifold issue, but it comes with the potential for deviation from the start or end frame, depending on the order of sequential sampling. To mitigate this issue, the authors have incorporated previous guidance techniques. However, I would appreciate a more comprehensive video comparison in the ablation study, in addition to the provided static comparison on a single scene.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - The method is training-free. In general, training-based method can be more effective.
- More ablation study is suggested to assess the effect of pre-trained video generation models. Will better video generation models result in better performance?
- Discussion on the limitation and future work.
soundness: 3
strengths: + A bidirectional sampling strategy, sequential sampling along both forward and backward paths, for video interpolation. 
+ Additional manifold guidance such as DDS and CFG++ are further introduced to enhance the interpolation results. 
+ Training-free, convenient for use.
+ Experiments are conducted to evaluate the proposed ViBiDSampler.
summary: This paper presented a bidirectional sampling strategy, sequential sampling along both forward and backward paths, for video interpolation. Additional manifold guidance such as DDS and CFG++ are further introduced to enhance the interpolation results. Experiments are conducted to evaluate the proposed ViBiDSampler.
weaknesses: - The method is training-free. In general, training-based method can be more effective.
- More ablation study is suggested to assess the effect of pre-trained video generation models. Will better video generation models result in better performance?
- Discussion on the limitation and future work.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: My main concerns are mainly as follows:

1. The contribution of this paper seems marginal. It seems the main contribution comes from eq. (9). However, the improvement of such modification seems not that obvious comparing with using SOTA sampling approaches including DDS and CFG++ from Table. 1, i.e., only "Ours (full)" shows obvious improvement comparing with other baselines but if my understanding is correct, other baselines seem not using DDS and CFG++. The author should further verify the improvement of each technology used in the paper, i.e., bidirectional sampling, DDS and CFG++, e.g., providing more visualization as well as adding DDS and CFG++ to other baselines and show the improvement.

2. The practical use the proposed approach seems not convincing. The proposed approach still requires 192s to generate 25 frames, which is at least two times longer than directly inferencing a video generation model with similar computational cost. Moreover, I also wonder how to keep the consistency between frames when the required length of the output video is beyond 25.
soundness: 3
strengths: +The paper is easy to follow .
+The visualization results seem impressive comparing with other baselines provided in the paper.
summary: This paper proposes a inference strategy to achieve frame interpolation w/o training. Its key idea is to use the pre-trained stable video diffusion model as a prior and complete the frame interpolation given the first and the last frame as a condition. The main improvement seems from the proposed bidirectional sampling combining with SOTA sampling approaches, i.e., DDS and CFG++.
weaknesses: The contribution of this paper seems marginal. It seems the main contribution comes from eq. (9). However, the improvement of such modification seems not that obvious comparing with using SOTA sampling approaches including DDS and CFG++ from Table. 1, i.e., only "Ours (full)" shows obvious improvement comparing with other baselines but if my understanding is correct, other baselines seem not using DDS and CFG++.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. To my understanding, CFG++ and DDS could possibly be applied to existing methods, such as Generative Inbetweening. If so, could the authors provide experiments on this, to ablate and show the importance of bidirectional sampling?

2. In bidirectional sampling, in some aspects, I think it is possible for this to work, but in some aspects, I don't. For example, taking the example of Fig.3(b), there are three steps, (1) denoising (w/ $c_{start}$), (2) renoising, (3) denoising (w/ $c_{end}$). I am curious how the generation process goes through these steps. Maybe a visualization would be helpful, showing how the start / end conditions are really affecting in those steps. I am specifically curious of this process in that the (2) renoising step could possibly remove the information of $c_{start}$ injected from (1) denoising, and the conditioning could have possibly be injected only at (3), making the steps (1) and (2) redundant. A further analysis / clarification on this part, that the proposed method surely takes both start / end frame conditioning.
soundness: 3
strengths: Strengths
- Overall, the paper is easy to follow, with coherent and clear problem definition and solution.
- Simple methodology. The proposed method, Bidirectional sampling seems to be easy to use.
- Efficiency. The proposed method does not require additional training and is fast in inference.
summary: This paper proposes ViBiDSampler, a training-free method utilizing pre-trained image-to-video diffusion models for keyframe interpolation. They aim to tackle the off-manifold problem of previous studies, which leads to artifacts or costly processes. They propose bidirectional sampling, with a set of denoising-renoising-denoising with alternating conditioning on start and end frames. This way, they manage to keep the generated results stay on the original manifold. The additionally integrate advanced guidance methods like CFG++ and DDS to reach state-of-the-art performance.
weaknesses: 1. Notations.
- The notations are confusing. For instance, $x_t$ seems to be used as a variable and a function. 
- The definitions could be further clarified. 
- Lines 247 - 256 was kind of confusing at first, with many similar looking notations without predefinitions, and mixed use of notations as functions and variables.


2. Limited contribution.

Although the proposed method is intuitive, simple, and quantitatively effective, some uncertainties remain.
- the 'Vanilla' setting at Table 1 and Figure 5, seems to show the effectiveness of bidirectional sampling solely, without guidance such as CFG++ and DDS. However, I am not fully convinced with its effectiveness, as Generative Inbetweening seems to outperform the vanilla version in Table 1. Also, in the qualitative results of Figure 5, the results of bidirectional sampling (vanilla) itself does not seem to be better than that of TRF.
- The concern above makes me wonder, if the superior performance of the entire framework proposed comes from the good guidance methods, CFG++ and DDS.
- In addition, CFG++ and DDS are manifold guidance methods proposed in other studies, and seems like this work simply took advantage of existing methods. I think it is hard to say that simply adoption of these guidance methods is a contribution.

In short, I am not fully convinced if the proposed bidirectional sampling itself is a powerful method compared to existing works, and suspect the superior performance could have come from the advanced manifold guidance methods, which, is hard to say to be a contribution.

Meta-review: Summary: Proposes a frame interpolation method that introduces a novel bi-directional sampling approach, that avoids off-manifold issues, in a training-free manner. It sequentially samples along the forward and backward paths, each conditioning on the start and end frames. Additionally, manifold guidance such as DDS and CFG++ are employed to synergically enhance interpolation results. 

Strength:  The bi-directional sampling strategy is novel, and it’s also simple and shown to be effective in qualitative results. Its training-free nature makes it an alternative approach to high-quality frame interpolation without resorting to expensive training or fine-tuning approaches. The paper is written very well, the problem is well motivated, which makes it easy to follow design choices and experimental results. Authors present extensive experimental comparisons that demonstrate the effectiveness of their approach. 

Weakness: It currently is not controllable via text. However, authors have identified this as a future work. 

Acceptance Reason: See strength. It is a novel approach that shows strong results using a simple and efficient technique. Future works and limitations sections outline a promising avenue for future work.

Decision: Accept (Poster)