Framer: Interactive Frame Interpolation

## Abstract

Abstract We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity.
Concretely, besides taking the start and end frames as inputs, our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints.
Such a design enjoys two clear benefits.
First, incorporating human interaction mitigates the issue arising from numerous possibilities of transforming one image to another, and in turn enables finer control of local motions.
Second, as the most basic form of interaction, keypoints help establish the correspondence across frames, enhancing the model to handle challenging cases ( e.g. , objects on the start and end frames are of different shapes and styles).
It is noteworthy that our system also offers an “autopilot” mode, where we introduce a module to estimate the keypoints and refine the trajectory automatically, to simplify the usage in practice.
Extensive experimental results demonstrate the appealing performance of Framer on various applications, such as image morphing, time-lapse video generation, cartoon interpolation, etc. The code, the model, and the interface will be released to facilitate further research. Project page: 𝚊𝚒𝚖 𝚊𝚒𝚖 \tt aim typewriter_aim - 𝚞𝚘𝚏𝚊 . 𝚐𝚒𝚝𝚑𝚞𝚋 . 𝚒𝚘 / 𝙵𝚛𝚊𝚖𝚎𝚛 formulae-sequence 𝚞𝚘𝚏𝚊 𝚐𝚒𝚝𝚑𝚞𝚋 𝚒𝚘 𝙵𝚛𝚊𝚖𝚎𝚛 \tt uofa.github.io/Framer typewriter_uofa . typewriter_github . typewriter_io / typewriter_Framer

## 1 Introduction

The creation of seamless and visually appealing transitions between frames (Dong et al., ) is a fundamental requirement in various applications, including image morphing (Aloraibi, ), slow-motion video generation (Reda et al., ), and cartoon interpolation (Xing et al., ). Users often need to control the motion trajectories, deformation dynamics, and temporal coherence of interpolated frames to achieve specific outcomes. Therefore, incorporating interactive capabilities into frame interpolation frameworks is crucial for expanding the practical applicability.

Traditional video frame interpolation methods (Jiang et al., ; Xu et al., ; Liu et al., ; Niklaus & Liu, ; Sim et al., ; Lee et al., ; Ding et al., ) often rely on estimating optical flow or motion to predict intermediate frames deterministically.
While significant progress has been made in this area, these approaches struggle in scenarios involving large motion or substantial changes in object appearance, due to an inaccurate flow estimation.
What’s more, when transforming one image to another, there can be numerous plausible ways objects and scenes can transition. A deterministic result may not align with user expectations or creative intent.

Orthogonal to existing methods, we propose Framer, an interactive frame interpolation framework designed to produce smoothly transitioning frames between two images.
Our approach allows users to customize the transition process by tailoring the trajectories of selected keypoints, thus directly influencing the motion and deformation of objects within the scene.
Such design offers two significant benefits.
First, the incorporation of keypoint-based interaction resolves the ambiguity inherent in transforming one image into another, allowing for precise control over how specific regions of the image move and change. As shown in Fig. 1a, users can control the movements of the dog’s paw and head through simple and intuitive interactions.
Second, keypoint trajectories establish explicit correspondences across frames, which is especially beneficial in challenging cases where objects change in shape, style, or even semantic meaning.
As shown in Fig. 1b, the keypoint trajectories establish the correspondences between keypoints from Pokémon in varying forms and help produce a smooth “evolution” process of Pokémon.

Concretely, we view video frame interpolation from a generative perspective and finetune a large-scale pre-trained image-to-video diffusion model (Blattmann et al., ) on open-domain video datasets (Nan et al., ) to facilitate video frame interpolation.
The additional last-frame conditioning is introduced during the fine-tuning process.
Afterward, a point trajectory controlling branch is introduced to take the additional point trajectory inputs, thus guiding the video interpolation process.
During inference, Framer supports the “interactive” mode for customized video frame interpolation, following user-input point trajectories.

Understanding that manual keypoint annotation may not always be desirable, we offer an “autopilot” mode for Framer.
Technically, we propose a novel bi-directional point-tracking method that estimates the trajectories of matched points over the entire video sequence, by analyzing both forward and backward motions between frames.
It automates the process of obtaining keypoint trajectories, enabling Framer to generate motion-natural and temporally coherent interpolation results without requiring extensive user input.
The “autopilot” mode simplifies the workflow while still benefiting from the enhanced correspondence provided by the points trajectories.

We conduct extensive experiments to evaluate the performance of Framer across various applications, including image morphing, time-lapse video generation, and cartoon interpolation. The results demonstrate that Framer produces smooth and visually appealing transitions, outperforming existing methods, particularly in cases involving complex motions and significant appearance changes. By combining the strengths of generative models with user-guided interactions, Framer improves both the quality and controllability of the interpolated frames.

## 2 Related Work

### 2.1 Video Frame Interpolation

Video frame interpolation (VFI) aims to synthesize intermediate frames from two successive video frames. Most previous methods view VFI as a low-level task, assuming a moderate motion between frames.
These methods can roughly be categorized as flow-based methods and kernel-based methods. Specifically, the flow-based methods leverage estimated optical flow for frame synthesis (Jiang et al., ; Xu et al., ; Liu et al., ; Niklaus & Liu, ; ; Sim et al., ; Huang et al., ; Jin et al., ; Xue et al., ; Park et al., ; ; Kong et al., ).
By contrast, the kernel-based methods rely on spatially adaptive kernels to synthesize the interpolated pixels (Lee et al., ; Cheng & Chen, ; Ding et al., ; Niklaus et al., ; Cheng & Chen, ; Gui et al., ; Lu et al., ).
While the former potentially suffers from inaccurate flow estimation, the latter are often constrained by kernel size.
To obtain the best of both worlds, some methods combine the flow- and kernel-based methods for end-to-end video frame interpolation (Bao et al., ; ; Danier et al., ; Li et al., ).
We refer readers to (Dong et al., ) for a more comprehensive survey on these methods.

Recently, inspired by the generative capacity of large-scale pre-trained video diffusion models, some methods attempt to tackle VFI from a generation perspective (Danier et al., ; Feng et al., ; Jain et al., ; Xing et al., ; Wang et al., ).
For example, LDMVFI (Danier et al., ) formulates VFI as a conditional generation problem and utilizes a latent diffusion model for perceptually oriented video frame interpolation.
Similarly, VIDIM (Jain et al., ) leverages cascaded diffusion models to generate high-fidelity interpolated videos with nonlinear motions.
Though progress has been made, these methods still have difficulties in tackling large differences between the starting and ending frames.
Moreover, they focus on generating a single deterministic solution for video frame interpolation, without controllability.
Differently, we can generate multiple plausible solutions under large motion changes, and allow simple and intuitive drag interaction for user-intended interpolation results.

### 2.2 Video Diffusion Models

Large-scale pre-trained video diffusion models (Brooks et al., ; Blattmann et al., ; Ge et al., ; Chen et al., ; ; Wang et al., ; Blattmann et al., ) have shown unprecedented generation results in visual quality, diversity, and realism.
These methods leverage text or starting image controls, which are often insufficient in precision and interactiveness.
Inspired by the success in controllable image generation (Zhang et al., ; Mou et al., ), several works attempt to add additional controls to video diffusion models.
Early explorations (Wang et al., ; Guo et al., ) utilize structural controls, like sketch and depth maps, for video generation. However, these control signals are often difficult to obtain during sampling, limiting their practical applications.
Differently, recent works focus on motion control and introduce trajectory control for object motion (Wu et al., ; Mou et al., ; Yin et al., ) and camera pose control for camera motion (Wang et al., ; He et al., ; Bahmani et al., ).
Both control signals can be obtained through easy and intuitive user interactions.
In this paper, we enhance the creative potential and flexibility of the video framer interpolation process, allowing users to produce plausible frame interpolation results following their control.

## 3 Method

Given two frames, $I^{0}$ and $I^{n}$, indicating the start and end frame in a video, our goal is to generate the plausible contiguous video $I=\{I^{i}\}_{i=0}^{n}$ by sampling from the conditional distribution $p\left(I\mid I^{0},I^{n}\right)$.
Here, $n$ is the number of frames in the video.
Our method, termed Framer, supports a user-interactive mode for customized point trajectories and an “autopilot” mode for video frame interpolation without trajectory inputs, as shown in Fig. 2a and Fig. 2b.
In the following, we will introduce how we add frame conditions to the video diffusion model to achieve video interpolation in Sec. 3.1.
To support user-interactive drag control, we introduce a control branch in Sec. 3.2 for point trajectory guidance, which also enhances point correspondences across frames.
In the “autopilot” mode, we estimate trajectories of matched points in the video with our novel bi-directional point tracking method, as illustrated in Sec. 3.3.

Figure: Figure 2: Framer supports (a) a user-interactive mode for customized point trajectories and (b) an “autopilot” mode for video frame interpolation without trajectory inputs. During training, (d) we fine-tune the 3D-UNet of a pre-trained video diffusion model for video frame interpolation. Afterward, (c) we introduce point trajectory control by freezing the 3D-UNet and fine-tuning the controlling branch.
Refer to caption: x2.png

### 3.1 Model Architecture

Large-scale pre-trained video diffusion models have a strong visual prior on the appearance, structure, and movement of open-world objects (Brooks et al., ). Our approach builds on the video diffusion model to exploit this prior. Considering that the Image-to-Video (I2V) diffusion model naturally supports first-frame conditioning, we choose the representative I2V diffusion model, Stable Video Diffusion (SVD) (Blattmann et al., ), as our base model, as shown in Fig. 2d.

Based on the I2V model, we need to introduce additional end-frame conditioning to realize video interpolation. To preserve the visual prior of the pre-trained SVD as much as possible, we follow the conditioning paradigm of SVD and inject end-frame conditions in the latent space and semantic space, respectively.
Specifically, we concatenate the VAE-encoded latent feature of the first frame, denoted as $z^{0}$, with the noisy latent of the first frame, as did in SVD. Additionally, we concatenate the latent feature of the last frame, $z^{n}$, with the noisy latent of the end frame, considering that the conditions and the corresponding noisy latents are spatially aligned.
In addition, we extract the CLIP image embedding of the first and last frames separately and concatenate them for cross-attention feature injection.
The U-Net $\epsilon_{\theta}$ is trained using the denoising score matching objective:

$$ $\mathcal{L}=\mathbb{E}_{z_{t},z^{0},z^{n},t,\epsilon\sim\mathcal{N}(0,\mathbf{ I})}\left[\left\|\epsilon-\epsilon_{\theta}\left(z_{t};t,z^{0},z^{n}\right) \right\|^{2}\right].$ (1) $$

### 3.2 Interactive Frame Interpolation

Ambiguity remains given the start and end frames, especially when the distinction between the two frames is large.
The reason is that multiple plausible interpolation results can be obtained by sampling video from the conditional distribution $P\left(I\mid I^{0},I^{n}\right)$ for the same input pair.
To better align with the user intention, we introduce a control branch for custmized point trajectory guidance.

Technically, we train a point trajectory-based control branch for correspondence modeling, as shown in Fig. 2c. During training, we use the following steps to obtain the point trajectory as control signals. Firstly, we randomly initialize some sampled points around a fixed sparse grid in the first frame, and use Co-Tracker (Karaev et al., ) to obtain the trajectories of these points in the whole video. Secondly, we remove trajectories that are not visible in more than half of the video frames. Lastly, we sample the point trajectories with larger motions with greater probability. Considering that the users usually only input a small number of point trajectories, we keep only 1 to 10 trajectories during training.
Please refer to the App. A for more details.

After obtaining the sampled point trajectories, we follow DragNUWA (Yin et al., ) and DragAnything (Wu et al., ) to transform the point coordinates into a Gaussian heatmap, denoted as $c_{traj}$, which is used as input to the control module.
We follow the conditioning mechanism in ControlNet (Zhang et al., ) to incorporate the trajectory control. Specifically, we copy the encoder of 3D-UNet to encode the trajectory map and add it into the decoder of U-Net after zero-convolution (Zhang et al., ). This training process can be represented as:

$$ $\mathcal{L}=\mathbb{E}_{z_{t},z^{0},z^{n},t,\epsilon\sim\mathcal{N}(0,\mathbf{ I})}\left[\left\|\epsilon-\epsilon^{c}_{\theta}\left(z_{t};t,z^{0},z^{n},c_{ traj}\right)\right\|^{2}\right].$ (2) $$

Here, $\epsilon^{c}_{\theta}$ is the combination of the denoising U-Net and the ControlNet branch.

Discussion.
The introduction of point trajectory control not only facilitates user interaction, but also enhances the correspondence among points from different frames.
As demonstrated in experiments, this approach enables the model to effectively tackle challenging cases, such as when the start and end frames differ significantly.

### 3.3 “Autopilot” Mode for Frame Interpolation

In practical applications, users may not always prefer manual drag controls. For this reason, we propose an “autopilot” mode to enhance the ease of use of our Framer.
It mainly contains a trajectory initialization and a trajectory updating process, as illustrated in Fig. 2b.

Trajectory Initialization.
Given the start and end frames of the input video, we can obtain the matching points between the two frames by applying feature-matching algorithms.
The matched points are denoted as $\{\bm{p}_{i}\}^{m}_{i=1}$, where $m$ is the number of matching points. $\bm{p}_{i}$ denotes the known anchor points on the trajectory. At initialization, it contains the matched points on the first and last frames, i.e., $\bm{p}_{i}=[p^{0}_{i},p^{n}_{i}]$.
Although varying feature matching algorithms are feasible, we use the classical SIFT feature matching (Lowe, ) here for its simplicity and effectiveness.
Subsequently, we can obtain the $i$-th trajectory $\hat{c}_{i}$ by interpolating the anchor points $\bm{p}_{i}$.
The estimated trajectory for all $m$ matched points, denoted as $\hat{c}_{traj}=\{\hat{c}_{i}\}^{m}_{i=1}$, are used as the input condition in Eq. 2.

Figure: Figure 3: Point trajectory estimation. The point trajectory is initialized by interpolating the coordinates of matched keypoints. In each de-noising step, we perform point tracking by finding the nearest neighbor of keypoints in the start and end frames, respectively. Lastly, We check the bi-directional tracking consistency before updating the point coordinate.
Refer to caption: x3.png

Trajectory Updating.
Although the initial trajectory provides temporally consistent point correspondence, the trajectory obtained by connecting points in the first and last frames may not be accurate.
Inspired by DragGAN (Pan et al., ) and DragDiffusion (Shi et al., ), we perform point tracking using the intermediate feature in U-Net to update the trajectories.
Specifically, in each denoising step, we interpolate the U-Net features to the image resolution, denoted as $\mathbf{F}$.
Here we use the feature of the penultimate upsampled block in U-Net, since it enjoys a good trade-off between feature resolution and discriminativeness.
We use $\mathbf{F}(p)$ to represent the feature of the point $p$, which is obtained via bilinear interpolation, since the coordinates may not be integers.

In each denoising step, we apply point tracking to update the coordinates of the middle frame points.
We use nearest neighbor search in a feature patch around the point.
The feature patch represents a set of points whose distance to point $p$ is less than $r$, and is denoted as
$\Omega\left({p},r\right)=\left\{(x,y)||x-x_{p}\left|<r,\right|y-y_{p}\mid<r\right\}$.
For a middle frame point ${p}^{k}_{i}$ in the $k$-th frame, we find the nearest point relative to the anchor point ${p}^{0}_{i}$ via:

$$ ${p}^{k,0}_{i}:=\underset{{q}^{k}_{i}\in\Omega\left({p}^{k}_{i},r_{1}\right)}{ \arg\min}\left\|\mathbf{F}\left({q}^{k}_{i}\right)-\mathbf{F}\left({p}^{0}_{i} \right)\right\|_{1}.$ (3) $$

Similarly, we can obtain the nearest point relative to the last anchor point ${p}^{n}_{i}$:

$$ ${p}^{k,n}_{i}:=\underset{{q}^{k}_{i}\in\Omega\left({p}^{k}_{i},r_{1}\right)}{ \arg\min}\left\|\mathbf{F}\left({q}^{k}_{i}\right)-\mathbf{F}\left({p}^{n}_{i} \right)\right\|_{1}.$ (4) $$

As shown in Fig. 3, to further ensure the accuracy of the coordinates of the updated points, we check the consistency of the two nearest points obtained by matching with ${p}^{0}_{i}$ and ${p}^{n}_{i}$.
When the distance between the two is less than a threshold $r_{2}$, i.e., ${p}^{k,n}_{i}\in\Omega\left({p}^{k,0}_{i},r_{2}\right)$, we update the point coordinates by setting ${p}^{k}_{i}=({p}^{k,0}_{i}+{p}^{k,n}_{i})/2$.
Then, we add the point to the anchor points list $\bm{p}_{i}$ and interpolate $\bm{p}_{i}$ to get the updated trajectory $c_{i}$, which is used as the input condition to the next denoising step.

## 4 Experiments

### 4.1 Implementation Details

Our method is built on SVD and trained on the high-quality OpenVidHD-0.4M dataset (Nan et al., ).
During the training of U-Net, we fixed the spatial attention and residual blocks, and only fine-tuned the input convolutional and temporal attention layers. The model is trained for 100K iterations using the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of 1e-5.
We obtained the point trajectories by pre-processing the video using the Co-Tracker (Karaev et al., ).
When training the control module, we fixed the U-Net and optimized the control module for 10K steps using the AdamW optimizer, with a learning rate of 1e-5.
All training is performed on 16 NVIDIA A100 GPUs, and the total batch size is 16.
During “autopilot” mode sampling, we keep $m=5$ best matching keypoints for trajectory guidance, and the distance thresholds for point tracking are set as $r_{1}=5$ and $r_{2}=3$.
Please refer to App. A for more details.

### 4.2 Comparison

Figure: Figure 4: Qualitative comparison. ‘GT” strands for ground truth. For each method, we only present the middle frame of 7 interpolated frames. The full results can be seen in Fig. S4 and Fig. S5 in the Appendix.
Refer to caption: x4.png

Figure: Figure 5: Reults on human preference.
Refer to caption: x5.png

Existing methods do not support drag-user interaction. Thus, we use the “autopilot” mode of Framer to make fair comparisons.
We select baselines from two distinct categories. The first category includes the latest general diffusion-based video interpolation models, including LDMVFI (Danier et al., ), DynamicCrafter (Xing et al., ), and SVDKFI (Wang et al., ). The second category encompasses traditional video interpolation methods, such as AMT (Li et al., ), RIFE (Huang et al., ), FLAVR (Kalluri et al., ), and FILM (Reda et al., ).
We conduct quantitative and qualitative analyses, as well as user studies, on two publicly available datasets: DAVIS (Pont-Tuset et al., ) and UCF101 (Soomro et al., ).

Qualitative Comparison.
As shown in Fig. 4, our method produces significantly clearer textures and natural motion compared to existing interpolation techniques.
It performs especially well in scenarios with substantial differences between the input frames, where traditional methods often fail to interpolate content accurately. Compared to other diffusion-based methods like LDMVFI and SVDKFI, Framer demonstrates superior adaptability to challenging cases and offers better control.

Quantitative Comparison.
As discussed in VIDIM (Jain et al., ), reconstruction metrics like PSNR, SSIM, and LPIPS fail to capture the quality of interpolated frames accurately, since they penalize other plausible interpolation results that are not pixel-aligned with the original video.
While generation metrics such as FID offer some improvement, they still fall short as they do not account for temporal consistency and evaluate frames in isolation.
Despite this, we present the quantitative metrics for various settings on both datasets, where our method achieves the best FVD score among all baselines as in Tab. 1.
We also evaluate Framer with 5 random point trajectories from ground-truth videos, estimated using Co-Tracker. As can be seen, “Framer with Co-Tracker” achieves superior performance even in reconstruction metric.
For a more comprehensive assessment of quality, we recommend reviewing the supplementary comparison videos.

**Table 1: Quantitative comparison with existing video interpolation methods on reconstruction and generative metrics, evaluated on all 7 generated frames.**
|  | DAVIS-7 | UCF101-7 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ | FVD$\downarrow$ |
| AMT (Li et al., ) | 21.66 | 0.7229 | 0.2860 | 39.17 | 245.25 | 26.64 | 0.9000 | 0.1878 | 37.80 | 270.98 |
| RIFE (Huang et al., ) | 22.00 | 0.7216 | 0.2663 | 39.16 | 319.79 | 27.04 | 0.9020 | 0.1575 | 27.96 | 300.40 |
| FLAVR (Kalluri et al., ) | 20.94 | 0.6880 | 0.3305 | 52.23 | 296.37 | 26.50 | 0.8982 | 0.1836 | 37.79 | 279.58 |
| FILM (Reda et al., ) | 21.67 | 0.7121 | 0.2191 | 17.20 | 162.86 | 26.74 | 0.8983 | 0.1378 | 16.22 | 239.48 |
| LDMVFI (Danier et al., ) | 21.11 | 0.6900 | 0.2535 | 21.96 | 269.72 | 26.68 | 0.8955 | 0.1446 | 17.55 | 270.33 |
| DynamicCrafter (Xing et al., ) | 15.48 | 0.4668 | 0.4628 | 35.95 | 468.78 | 17.62 | 0.7082 | 0.3361 | 61.71 | 646.91 |
| SVDKFI (Wang et al., ) | 16.71 | 0.5274 | 0.3440 | 26.59 | 382.19 | 21.04 | 0.7991 | 0.2146 | 44.81 | 301.33 |
| Framer (Ours) | 21.23 | 0.7218 | 0.2525 | 27.13 | 115.65 | 25.04 | 0.8806 | 0.1714 | 31.69 | 181.55 |
| Framer with Co-Tracker (Ours) | 22.75 | 0.7931 | 0.2199 | 27.43 | 102.31 | 27.08 | 0.9024 | 0.1714 | 32.37 | 159.87 |

Figure: Figure 6: Results on user interaction. The first row is generated without drag input, while the other two are generated with different drag controls. Customized trajectories ares overlaid on frames.
Refer to caption: x6.png

Figure: Figure 7: Novel view synthesis on both static (1st row) and dynamic scenes (2nd row).
Refer to caption: x7.png

User Study.
Since quantitative metrics fall short in reflecting video quality, we further assessed our method’s performance through a user study. In this study, participants reviewed video sets generated from the same input frame pair by both existing methods and our Framer. Participants assessed up to 100 randomly ordered video sets and selected the one they found most realistic. In total, 20 participants provided 1,000 ratings across these video sets. As illustrated in Fig. 5, the results demonstrate a strong preference among human raters for the outputs produced by our method.

### 4.3 Applications

Optional drag control.
Given the same input start and end frames, multiple plausible results can satisfy the goal of video interpolation.
With Framer, users can direct the motion of the entities in input frames with simple drags for their intention, or simply obtain a default interpolation result without drags.
As shown in Fig. 6, the seal moves in varying directions given the same input frames.

Figure: Figure 8: Applications on cartoon (1st row) and sketch (2nd row) interpolation.
Refer to caption: x8.png

Figure: Figure 9: Applications on time-lapsing video generation.
Refer to caption: x9.png

Novel view synthesis (NVS) is a classical 3D vision task, with a wide range of applications.
Using images of different viewpoints as the start and end frames of the video respectively, we can realize the NVS from sparse viewpoint input by performing video interpolation.
As shown in Fig. 7, our method achieves pleasing NVS in both static scenes (first row) and dynamic scenes (second and third rows).
Taking the second row as an example, the house gradually moves out of the scene as the camera keeps moving forward. In the meantime, the car moves in the opposite direction to the camera and gradually takes up a larger proportion in the frame.

Figure: Figure 10: Applications on slow-motion video generation. The y-t slice highlighted in red on video frames is visualized on the right.
Refer to caption: x10.png

Cartoon and sketch interpolation.
We can dramatically simplify the process of cartoon video production, by interpolating manually created cartoon images. To this end, we tested our method on cartoon data.
Although our method is not specifically trained on cartoon videos, it produces appealing cartoon video results and supports both color images and sktech drawing frame interpolation, as shown in Fig. 8.
For example, our method successfully models the motion of two objects, i.e., the front vehicle pulls sideways while the rear vehicle follows, as shown in the first row.
In the third row, Framer produces a smooth motion of the hand lifting in sketch drawings.

Time-lapsing video generation.
Time-lapse photography can vividly demonstrate slow changes that are difficult to detect with the naked eye.
Typically, it requires sufficient storage space to hold a large amount of image data and a complex post-processing procedure to organize and edit the images.
Video interpolation provides a simple and effective way to obtain time-lapse videos by interpolating frames with only a few images of key moments.
As shown in Fig. 9, Framer produces the smooth change of moon waxing and waning.

Slow-motion video generation enhances visual effects by highlighting fine details and allows closer examination of fast phenomena. Our Framer inherently supports fast frame interpolation, as demonstrated in Fig. 10, enabling smooth slow-motion effects suitable for films and animations.

Figure: Figure 11: Applications on image morphing. Customized trajectories ares overlaid on end frames.
Refer to caption: x11.png

Image morphing (Aloraibi, ) is a popular image transformation technique with many applications in computer vision and computer graphics. Given two topologically similar images, it aims to generate a series of reasonable intermediate images. Using tue two images as the start and end frames, Framer can produce natural and smooth image morphing results.
For example, in Fig. 1, we show the “evolution” process of Pokemon. More cases can be found in Fig. S13.

### 4.4 Ablations Studies

We conducted ablation studies on the individual components of Framer to validate their effectiveness. The results are illustrated in Fig. 12.
Our observations are as follows.
First, when the trajectory guidance is removed (denoted as “w/o traj.”), the foreground motorcycle exhibits significant distortion, as shown in the 1st row of Fig. 12.
Conversely, with the inclusion of trajectory guidance, the temporal consistency of the video is notably enhanced, as depicted in the 2nd row.
We believe this is due to the enhancement of point correspondence modeling across frames.
Second, removing trajectory updates (denoted as “w/o traj. update”) or updating the trajectory without bi-directional consistency checks (denoted as “w/o bi-directional”) results in blurring in the wheel regions of the output video.
We suspect the blurring is caused by the guidance of unnatural motion from inaccurate trajectories, which conflicts with the generation prior in the pre-trained diffusion model, leading to local blurring.
In contrast, our method produces video frame interpolation results with natural motion and smooth temporal coherence.
The quantitative results in Tabs. S1 and S2 in App. B further support these findings, showing a similar trend to the qualitative ablation experiments.

Figure: Figure 12: Ablations on each component. “w/o trajectory” denotes inference without guidance from point trajectory, “w/o traj. update” indicates inference without trajectory updates, and “w/o bi” suggests trajectory updating without bi-directional consistency verification.
Refer to caption: x12.png

## 5 Conclusion and Future Work

In this paper, we introduce Framer, an interactive frame interpolation pipeline designed to produce smoothly transitioning frames between two images, guided by user-defined point trajectories.
By harnessing user input point controls from the start and end frames, we effectively guide the video interpolation process.
Moreover, our method offers an “autopilot” mode that introduces a module to automatically estimate keypoints and refine trajectories without manual input.
Through extensive experiments and user studies, we demonstrate the superiority of our method in achieving promising results in terms of both the quality and controllability of the interpolated frames.
However, challenges remain, particularly in transitioning between different clips.
A potential solution involves splitting the clips into several keyframes and then interpolating these keyframes sequentially.
Future work will focus on addressing these challenges.

## Appendix

## Appendix A More Implementation Details

During training, we sample 14 consecutive frames from videos, with a spatial resolution of 512$\times$320.
Specifically, we center-crop the video to an aspect ratio of $512/320$, then resize the video frames to the resolution of $512\times 320$.
Random horizontal flip is utilized for data augmentation.
We sample the video in temporal dimension, with a frame interval of 2.
For the training of the point trajectory-based ControlNet, we sample 1 to 10 trajectories with larger motions for training.
Specifically, we follow ReVideo (Mou et al., ) and sample the trajectories by setting the normalized lengths of the trajectories as sampling probabilities.
During “autopilot” mode sampling, we use the Euler sampler with 30 diffusion steps in total.
For point tracking in Sec. 3.3, we use the output feature of the second decoder block in the 3D-UNet.
We resize the shorter side of the video to the length of 512, then center crop the video to the resolution of $512\times 320$.

## Appendix B More Detailed Ablation Results

#### Qualitative results for ablation studies.

In Fig. 12, we show the qualitative results for ablation studies. We supplement these results with the quantitative results in Tab. S1 and Tab. S2, which show a similar trend to the qualitative ablation experiments.

**Table S1: Ablations on each component, evaluating all 7 generated frames. “w/o trajectory” denotes inference without guidance from point trajectory, “w/o traj. updating” indicates inference without trajectory updating, and “w/o bi” suggests trajectory updating without bi-directional consistency verification.**
|  | DAVIS-7 | UCF101-7 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ | FVD$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ | FVD$\downarrow$ |
| w/o trajectory | 20.19 | 0.6831 | 0.2787 | 28.25 | 128.71 | 24.16 | 0.8677 | 0.1798 | 32.64 | 195.54 |
| w/o traj. updating | 20.82 | 0.7054 | 0.2621 | 27.33 | 120.73 | 24.69 | 0.8748 | 0.1842 | 31.95 | 187.37 |
| w/o bi-directional | 20.94 | 0.7102 | 0.2602 | 27.23 | 116.81 | 24.73 | 0.8746 | 0.1845 | 31.66 | 183.74 |
| Framer (Ours) | 21.23 | 0.7218 | 0.2525 | 27.13 | 115.65 | 25.04 | 0.8806 | 0.1714 | 31.69 | 181.55 |

**Table S2: Ablations on each component, evaluating only the middle frame out of all 7 generated frames. “w/o trajectory” denotes inference without guidance from point trajectory, “w/o traj. updating” indicates inference without trajectory updating, and “w/o bi” suggests trajectory updating without bi-directional consistency verification.**
|  | DAVIS-7 (mid-frame) | UCF101-7 (mid-frame) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ |
| w/o trajectory | 19.30 | 0.6504 | 0.3093 | 57.10 | 23.14 | 0.8523 | 0.1967 | 54.98 |
| w/o traj. updating | 19.84 | 0.6700 | 0.2935 | 55.37 | 23.60 | 0.8590 | 0.2009 | 53.83 |
| w/o bi-directional | 19.95 | 0.6739 | 0.2919 | 54.75 | 23.65 | 0.8586 | 0.2016 | 53.54 |
| Framer (Ours) | 20.18 | 0.6850 | 0.2845 | 55.13 | 23.92 | 0.8646 | 0.1889 | 53.33 |

#### Ablations on diffusion feature for point tracking.

As detailed in Sec. 3.3, we perform point tracking using the diffusion feature for point trajectory updating. Here we perform ablated experiments on the selection of the diffusion feature. The results are shown in Fig. S1. It can be seen that in both DAVIS-7 and UCF-7, point tracking with the output feature from the second diffusion block gives rise to the best-performing results in FVD.

Figure: Figure S1: Ablations on diffusion feature for point tracking at test time, experiments conducted on DAVIS-7 (left) and UCF101-7 (right).
Refer to caption: x13.png

#### Ablations on diffusion steps for correspondence guidance.

We ablate the diffusion steps for correspondence guidance by only applying the guidance at the early steps or late steps in diffusion sampling. The results are shown in Fig. S2.
As can be seen, the early steps are often more important than the late steps for correspondence modeling.
For example, on DAVIS-7, a pleasing FVD can be obtained when performing guidance only on 0-18 diffusion steps. By contrast, performing guidance only on 18-30 diffusion steps brings litter improvements.
We speculate that this is because the early diffusion steps focus on the structural information of the video, while the late diffusion steps focus on the texture and details (Xue et al., ).
The correspondence guidance at early steps already helps the model obtain a reasonable video structure.
In the implementation, we simply apply correspondence guidance in all diffusion steps, without detailed searches on the hyper-parameter.

Figure: Figure S2: Ablations on the start and end diffusion steps for correspondence guidance, experiments conducted on DAVIS-7 (left) and UCF101-7 (right). We use a total sampling step of 30.
Refer to caption: x14.png

#### Ablations on the number of trajectories for correspondence guidance.

As described in Sec. 3.3, we use $m$ trajectories for correspondence guidance during sampling.
Here we perform ablated experiments on this hyper-parameter, and the result is shown in Fig. S3.
It can be seen that sampling with the 5 trajectories leads to the best performance. Thus we set $m=5$ by default.

Figure: Figure S3: Ablations on the number of trajectories for guidance during sampling, experiments conducted on DAVIS-7 (left) and UCF101-7 (right).
Refer to caption: x15.png

## Appendix C More Details on Comparison with Previous Methods

#### Benchmark.

We follow the practice of VIDIM (Jain et al., ) and perform the quantitative evaluation on DAVIS-7 and UCF101-7 datasets using both reconstruction and generative metrics.
Both DAVIS-7 and UCF101-7 are obtained by sampling 7 consecutive video frames from the corresponding datasets.
We use all videos in the DAVIS dataset and a subset of 400 videos in the UCF101 dataset.

#### More results on comparisons.

In Tab. S3 we provide the quantitative comparison based on the middle frame of the 7 interpolated video frames.
Besides, in Fig. S4, Fig. S5, Fig. S6, and Fig. S7, we show more qualitatively comparisons with exiting methods.

**Table S3: Quantitative comparison with existing video interpolation methods on reconstruction and generative metrics, evaluated only on the middle frame out of all 7 generated frames.**
|  | DAVIS-7 (mid-frame) | UCF101-7 (mid-frame) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | FID$\downarrow$ |
| AMT (Li et al., ) | 20.59 | 0.6834 | 0.3564 | 100.36 | 25.24 | 0.8837 | 0.2237 | 75.97 |
| RIFE (Huang et al., ) | 20.74 | 0.6813 | 0.3102 | 80.78 | 25.68 | 0.8842 | 0.1835 | 59.33 |
| FLAVR (Kalluri et al., ) | 19.93 | 0.6514 | 0.4074 | 118.45 | 24.93 | 0.8796 | 0.2164 | 79.86 |
| FILM (Reda et al., ) | 20.28 | 0.6671 | 0.2620 | 48.70 | 25.31 | 0.8818 | 0.1623 | 41.23 |
| LDMVFI (Danier et al., ) | 19.87 | 0.6435 | 0.2985 | 56.46 | 25.16 | 0.8789 | 0.1695 | 43.01 |
| DynamicCrafter (Xing et al., ) | 14.61 | 0.4280 | 0.5082 | 77.65 | 17.05 | 0.6935 | 0.3502 | 97.01 |
| SVDKFI (Wang et al., ) | 16.06 | 0.4974 | 0.3719 | 53.49 | 20.03 | 0.7775 | 0.2326 | 69.26 |
| Framer (Ours) | 20.18 | 0.6850 | 0.2845 | 55.13 | 23.92 | 0.8646 | 0.1889 | 53.33 |
| Framer with Co-Tracker (Ours) | 21.94 | 0.7693 | 0.2437 | 55.77 | 25.86 | 0.8868 | 0.1873 | 54.64 |

## Appendix D More Qualitative Results

We provide more qualitative results on drag control, novel view synthesis, cartoon and sketch interpolation, time-lapsing video generation, slow-motion video generation, and image morphing in Fig. S8, Fig. S9, Fig. S10, Fig. S11, Fig. S12, and Fig. S13, respectively.

## Appendix E Discussions on Limitations

Framer is built on top of the large-scale pre-trained video diffusion model, thus it inherits the limitations of the pre-trained model.
Moreover, the point trajectories in Framer rely on the matching points between the input image pair for interpolating complex motions.
While this is a step forward compared with current models that can only simply motions, our method still faces difficulties when the differences between the front and back frames are so large that no matched points can be found at all.
Thus, we will explore more powerful pre-trained video diffusion models, as well as training video interpolation models on larger-scale video data in the future.
Lastly, our approach currently only supports drag control and does not explore other interaction methods.
In the future, we will continue to explore other user-friendly controls such as text control and camera pose control.

Figure: Figure S4: More qualitative comparison with existing methods. “GT” strands for ground truth.
Refer to caption: x16.png

Figure: Figure S5: More qualitative comparison with existing methods. “GT” strands for ground truth.
Refer to caption: x17.png

Figure: Figure S6: More qualitative comparison with existing methods. “GT” strands for ground truth.
Refer to caption: x18.png

Figure: Figure S7: More qualitative comparison with existing methods. “GT” strands for ground truth.
Refer to caption: x19.png

Figure: Figure S8: More results on user interaction. We show the results of two trajectory controls with the same input image pair.
Refer to caption: x20.png

Figure: Figure S9: More results on novel view synthesis. The first and second rows show results on static and dynamic scenes, respectively.
Refer to caption: x21.png

Figure: Figure S10: More results on (a) cartoon and (b) sketch interpolation.
Refer to caption: x22.png

Figure: Figure S11: More results on time-lapsing video generation.
Refer to caption: x23.png

Figure: Figure S12: More results on slow-motion video generation. The x-t slice highlighted in red on video frames is visualized on the right.
Refer to caption: x24.png

Figure: Figure S13: More results on image morphing.
Refer to caption: x25.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - What are the differences in training / inference methodology between the autopilot and user-guided mode?
- Are there categories of videos where the results aren't as high quality as other categories?
- Could you explain a bit more about "considering that the conditions and the corresponding noisy latents are spatially aligned" near line 183?
- Can you explain how you transform 2D points into gaussian heatmaps? Are any other modalities like text used here? References to DragNUWA or DragAnything might be too recent. Consider also citing an older source such as from 2014 "Stacked Hourglass Networks for Human Pose Estimation" if it makes sense.
- Can the autopilot mode re-use the randomly sampled points ~line 197 instead of using feature matching to find new correspondences over time?
- Have you considered simpler alternatives for the Trajectory updating section?
- How do you handle camera motion in your framework?
soundness: 3
strengths: The paper is mostly well written and easy to follow. The novelty here lies in the control branch for guiding the interpolation process that allows for fine-grained user-friendly control over the interpolation process.. The authors evaluate several existing methods on a variety of input types like real-world, sketch, cartoon, etc both qualitatively and quantitatively and produce high quality results.  Also included in a technique to produce the control signals automatically using bidirectional consistency checks. Several key design choices have been evaluated thoroughly, such as number of trajectories, bidirectional consistency.
summary: The paper proposes a new technique for generating frame interpolations that can incorporate user input to drive the interpolation process between two given frames. The method uses Stable Video Diffusion as the base model with the inputs modified to include both the first and last frame. Additionally, a control branch consisting of gaussian heatmaps is encoded and included in the model. The method is evaluated on a variety of image categories and on publicly available datasets.
weaknesses: - There is no mention of inference speed, training speed or overall training methodology (end-to-end, multi stage, pretraining, etc)
- There appears to be no underlying camera model for handling the "novel view synthesis" examples. Having such a model could potentially improve quality while providing another user-control.
- The paper isn't very easy to follow specifically around the trajectory update for autopilot mode section.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1)	Can the frame interpolation function support arbitrary number of frames generation between the start and end inputs?
2)	how to process the case when the number of corresponding points between frames is too few?
soundness: 3
strengths: 1）	The method is straight forward but effective, which can be used for many applications;
2）	Most results shown in this paper are impressive.
3）	Ablation experiments are sufficient to evaluate the method.
summary: This paper presents an interactive frame interpolation framework to produce smoothly transitioning frames between two images, interactively or automatically. By harnessing user input point controls from the start and end frames, this method can effectively generate the video frames for different applications including drag control, novel view synthesis, cartoon and sketch interpolation, time-lapsing video generation, slow-motion video generation, and image morphing.
weaknesses: 1)	The component of the Framer includes Trajectory preprocessing, Trajectory control and Interpolation. Although the method is effective, the novelty is limited. Trajectory control is designed by following DragNUWA, and the Interpolation is mainly based on SVD that determines the performance of this method.
2)	The examples shown in this paper are a bit simple. The motion and the difference between two key frames (first and last frames) are not complex. More complex scenarios, as cases shown in the paper “Training Weakly Supervised Video Frame Interpolation with Events”, should be considered to demonstrate the superiority of this method.
3)	The image quality of some results shown in this paper degrades compared to the first input frame. For example, the IQ of frames in “laugh.mp4” is worse than that of the start and end frames;
4)	The reconstruction metric of this method is not best when comparing it to related methods.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 4
questions: Please address the issues raised in the weaknesses section. This reviewer is willing to increase the rating if the issues are well addressed.
soundness: 3
strengths: The method’s introduction of keypoint trajectories addresses motion ambiguity, leveraging the stable video diffusion model to improve interpolation fidelity.   
The qualitative outcomes are notably strong.
summary: This paper presents a diffusion-based architecture, *Framer*, for interactive frame interpolation, introducing last-frame conditioning and keypoint trajectories to achieve controllable transitions. While the qualitative results are impressive, the practical application, related work discussion, and unique technical contributions need further improvement.
weaknesses: 1. **Related Work**: The paper does not sufficiently discuss previous work tackling motion ambiguity and interactive frame interpolation, such as [1] and [2], which address motion/velocity ambiguities. And [2] is the early work in interactive VFI. A more comprehensive discussion would clarify the novelty and contributions of *Framer*.
2. **Real-World Application**: The practical applicability of this method is unclear, especially given that real-world videos typically run at frame rates like 24–30fps, making extremely large-motion interpolation less common. There is also no analysis of inference speed in autopilot mode.
3. **Robustness and Scope**: The method’s effectiveness under complex trajectories remains uncertain, as the learned keypoint trajectory distribution is unlikely to generalize to all cases. Additionally, autopilot mode’s reliance on feature-matching algorithms may limit performance under conditions like blur or low resolution. Further experiments are needed to clarify performance under these situations.
4. **Benchmarking and Quantitative Analysis**: The paper lacks evaluations on commonly used VFI datasets, such as Vimeo90K and X4K1000FPS, which would offer valuable comparison. In addition, considering that the proposed method achieves superior performance only in terms of FVD, the inclusion of non-reference metrics such as NIQE will help to further understand the image quality.
5. **Technical Innovation**: While the method yields high-quality visuals, the technical novelty appears limited. Techniques such as conditioning diffusion models with start and end frames and using keypoint trajectories (DragGAN) already exist. Emphasizing unique contributions would strengthen the paper’s impact.

[1] "Exploring motion ambiguity and alignment for high-quality video frame interpolation." CVPR2023  
[2] "Clearer Frames, Anytime: Resolving Velocity Ambiguity in Video Frame Interpolation." ECCV2024.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Overall, I think this is a solid piece of work. Aside from the minor points mentioned in the weaknesses section, I don’t have significant concerns. The approach is interesting and well-presented, with promising results. I would lean toward recommending acceptance of this paper.
soundness: 4
strengths: 1. The motivation of the paper is clear and well-founded. The authors address the inherent variability in intermediate frame generation and propose an effective solution by integrating keypoint control, which allows for a more tailored interpolation process that better aligns with user preferences.
2. The paper is well-written and easy to follow. The use of figures and tables is excellent, with ample illustrative examples demonstrating the model’s application.
3. The results presented are noticeably better than existing methods, particularly in challenging scenarios. Additionally, the inclusion of an automated mode ensures fairness in comparison, which further strengthens the validity of the results.
summary: The paper proposes Framer, an interactive frame interpolation model designed to generate smooth frame transitions between two images. The approach allows users to customize the interpolation process by adjusting the motion trajectories of keypoints. This interactive feature provides precise control over local motions, improving the model’s ability to handle challenging scenarios. The model also includes an autopilot mode that automatically estimates keypoints and refines trajectories, simplifying the interpolation process for users.
weaknesses: 1. It would be helpful to see examples of failure cases, if any, to better understand the model’s limitations. Displaying such cases could provide insight into potential improvements or areas where the method might struggle in real-world applications.
2. The title might be slightly misleading. The approach seems more aligned with image morphing rather than traditional frame interpolation. A title that reflects this focus on image-to-image transitions could better represent the paper’s core contribution.

Meta-review: The paper introduces Framer, an method for using user-provided trajectories as guidance in frame interpolation or morphing. Reviewers praised the paper's clear motivation and impressive performance, but some questioned its practical applicability and technical innovation.  Reviews were unanimously positive, and thus the paper has been chosen for acceptance.

Decision: Accept (Poster)