Framer: Interactive Frame Interpolation
Authors: Wen Wang
Qiuyu Wang
Kecheng Zheng
Hao OUYANG
Zhekai Chen
Biao Gong
Hao Chen
Yujun Shen
Chunhua Shen
Venue: ICLR 2025 Poster (2025)
Keywords: Video Frame Interpolation; Interactive; Diffusion Model; Correspondence Modeling
Primary area: generative models
TL;DR: We propose Framer, an interactive frame interpolation method that allows users to produce smoothly transitioning frames between two images by customizing the trajectory of selected keypoints, enhancing control and handling challenging cases.

Abstract: We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity. Concretely, besides taking the start and end frames as inputs, our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. Such a design enjoys two clear benefits. First, incorporating human interaction mitigates the issue arising from numerous possibilities of transforming one image to another, and in turn enables finer control of local motions. Second, as the most basic form of interaction, keypoints help establish the correspondence across frames, enhancing the model to handle challenging cases (e.g., objects on the start and end frames are of different shapes and styles). It is noteworthy that our system also offers an "autopilot" mode, where we introduce a module to estimate the keypoints and refine the trajectory automatically, to simplify the usage in practice. Extensive experimental results demonstrate the appealing performance of Framer on various applications, such as image morphing, time-lapse video generation, cartoon interpolation, etc. The code, model, and interface are publicly accessible at https://github.com/aim-uofa/Framer.

Published as a conference paper at ICLR 2025
Framer: INTERACTIVE FRAME INTERPOLATION
Wen Wang 1.2 Qiuyu Wang",
Kecheng Zheng', Hao Ouyang,,
Zhekai Chen',
Biao Gong", Hao Chen', Yujun Shen", Chunhua Shen
1 Zhejiang University
Ant Group
3 Zhejiang University Technology
Start Frame
Generated Frames
End Frame
Same input image with varying drags
(b) Image morphing (irputs highlighted dashed boxes)
Showcases produced by our Framer. facilitates fine-grained customization of local motions and generates varying interpolation results given the same input start and end frame pair (first 3 rows). Moreover, Framer handles challenging cases and can realize smooth image morphing (last 2 rows). The input trajectories are overlayed on the frames.
Figure 1:
ABSTRACT
We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity. Con- cretely, besides taking the start and end frames as inputs, our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. Such design enjoys two clear benefits. First, incorporating human interaction mitigates the issue arising from numerous possibilities of transforming image another, and in turn enables finer control of local motions. Second, as the most basic form of interaction, keypoints help establish the correspondence across frames, enhancing the model handle challenging cases (e.g. objects on the start and end frames are of different shapes and styles). It is noteworthy that our system also offers an "autopilot" mode, where we introduce module to estimate the keypoints and refine the trajectory automatically, to simplify the usage practice. Extensive experimental results demonstrate the appealing performance of Framer on various applications, such image morphing, time- lapse video generation, cartoon interpolation, etc. The code, model, and interface are publicly accessible at https: /gi thub. com/ aimm sim-uoff a/ Framer.
INTRODUCTION
The creation of seamless and visually appealing transitions between frames (Dong et 2023) is fundamental requirement in various applications, including image morphing (Aloraibi, 2023), slow- motion video generation (Reda al., 2022), and cartoon interpolation (Xing et al., 2024). Users

---

Published as a conference paper at ICLR 2025
often need to control the motion trajectories, deformation dynamics, and temporal coherence of interpolated frames to achieve specifie outcomes. Therefore, incorporating interactive capabilities into frame interpolation frameworks is crucial for expanding the practical applicabilitty.
Traditional video frame interpolation methods (Jiang et al., 2018; Xu et al., 2019; Liu et al., 2020; Niklaus & Liu, 2020; Sim et al., 2021; Lee et al., 2020; Ding et al., 2021) often rely on estimating optical flow or motion to predict intermediate frames deterministically. While significant progress has been made in this area, these approaches struggle in scenarios involving large motion or substantial changes in object appearance, due to an inaccurate flow estimation. What's more, when transforming one image to another, there can be numerous plausible ways objects and scenes can transition. A deterministic result may not align with user expectations or creative intent.
Orthogonal to existing methods, we propose Framer, an interactive frame interpolation framework designed to produce smoothly transitioning frames between two images. Our approach allows users customize the transition process by tailoring the trajectories of selected keypoints, thus directly influencing the motion and deformation of objects within the scene. Such design offers two significant benefits. First, the incorporation of keypoint-based interaction resolves the ambiguity inherent in transforming one image into another, allowing for precise control over how specific regions of the image move and change. As shown in Fig. la, users can control the movements of the dog's 's paw and head through simple and intuitive interactions. Second, keypoint trajectories establish explicit correspondences across frames, which is especially beneficial in challenging cases where objects change in shape, style, even semantic meaning. As shown in Fig. 1b, the keypoint trajectories establish the correspondences between keypoints from Pokémon in varying forms and help produce smooth "evolution" process of Pokémon.
Concretely, we view video frame interpolation from a generative perspective and finetune a large-scale pre-trained image-to-video diffusion model (Blattmann et al., 2023a) on open-domain video datasets (Nan et al., 2024) to facilitate video frame interpolation. The additional last- frame conditioning is introduced during the fine-tuning process. Afterward, a point trajectory controlling branch is introduced to take the additional point trajectory inputs, thus guiding the video interpolation process. During inference, Framer supports the "interactive" mode for customized video frame interpolation, following user-input point trajectories.
Understanding that manual keypoint annotation may not always be desirable, we offer an "autopilot" mode for Framer. Technically, we propose novel bi-directional point-tracking method that estimates the trajectories of matched points over the entire video sequence. by analyzing both forward and backward motions between frames. It automates the process of obtaining keypoint trajectories, enabling Framer to generate motion-natural and temporally coherent interpolation results without requiring extensive user input. The "autopilot" mode simplifies the workflow while still benefiting from the enhanced correspondence provided by the points trajectories.
We conduct extensive experiments to evaluate the performance of Framer across various appli- cations, including image morphing, time-lapse video generation, and cartoon interpolation. The results demonstrate that Framer produces smooth and visually appealing transitions, outperforming existing methods, particularly in cases involving complex motions and significant appearance changes. By combining the strengths of generative models with user-guided interactions, Framer improves both the quality and controllability of the interpolated frames.
RELATED WORK
2.1 VIDEO FRAME INTERPOLATION
Video frame interpolation (VFI) aims to synthesize intermediate frames from two successive video frames. Most previous methods view VFI as a low-level task, assuming moderate motion between frames. These methods can roughly be categorized as flow-based methods and kernel- based methods. Specifically, the flow-based methods leverage estimated optical flow for frame synthesis (Jiang et al., 2018; Xu et al., 2019; Liu et al., 2020; Niklaus & Liu, 2020; 2018; Sim et al., 2021; Huang et al., 2020; Jin et al., 2023; Xue et al., 2019; Park et al., 2020; 2021; Kong et al., 2022; Yu et al., 2021). By contrast, the kernel-based methods rely on spatially adaptive kernels to synthesize the interpolated pixels (Leee et al., 2020; Cheng & Chen, 2022: Ding al., 2021; Niklaus
2

---

Published as a conference paper at ICLR 2025
et al., 2017; Cheng & Chen, 2020; Gui al., 2020; Lu et al., 2022). While the former potentially suffers from inaccurate flow estimation, the latter are often constrained by kernel size. To obtain the best of both worlds, some methods combine the flow- and kernel-based methods for end-to-end video frame interpolation (Bao et al., 2019; 2021; Danier et al., 2022; Li et al., 2022). Realizing that motion ambiguity remains given the start and end frames (Zhou et al., 2023; Zhong et al,, 2024), Zhong et al. (2024) resolves the ambiguities by providing explicit hints on how far objects have traveled between start and end frames, termed "distance indexing". While Zhong et al. (2024) supports user-interacti interretrat,,,, the distance-inden
interaction requires setting detailed distance values for middle frames, making them less intuitive.
Recently, inspired by the generative capacity of large-scale pre-trained video diffusion models, some methods attempt to tackle VFI from a generation perspective (Danier et al., 2024; Feng et al., 2024; Jain et al., 2024; Xing et al., 2023; Wang et al., 2024a). For example, LDMVFI (Danier et al., 2024) formulates VFI as a conditional generation problem and utilizes latent diffusion model for perceptually oriented video frame interpolation. Similarly, VIDIM (Jain et al., 2024) leverages cascaded diffusion models to generate high-fidelity interpolated videos with nonlinear motions. Though progress has been made, these methods still have difficulties in tackling large differences between input frames. Moreover, they generate a single deterministic solution for video frame interpolation, without controllability. Differently, we can generate multiple plausible solutions under large motion changes, and allow simple and intuitive drag interaction for user-intended results.
Video frame interpolation has a wide range of applications in many fields. While traditional interpolation methods focus on improving the frame rate of the input video (Li et al., 2023; Huang al., 2020; Kalluri et al., 2023; Reda et al., 2022), generative frame interpolation methods take advantage of large-scale pre-trained diffusion models, and are more concerned with dealing with situations where the input frames have large differences (Xing al., 2023; Wang et al., 2024a). In addition, some works train tailored video frame interpolation models for specific application scenarios, such as cartoon interpolation (Siyao et al., 2021; Chen & Zwicker, 2022; Xing et al., 2024), sketch interpolation (Siyao et al., 2023; Shen et al., 2024), etc. In this paper, we show that Framer can handle all of the above tasks under a unified framework, and allow users to achieve fine-grained control of the interpolation process through simple interactions.
2.2 VIDEO DIFFUSION MODELS
Large-scale pre-trained video diffusion models (Brooks et al., 2024; Blattmann et al., 2023b; Chen et al., 2024; Blattmann et al., 2023a) have shown unprecedented generation results in visual quality, diversity, and realism. These methods leverage text starting image controls, which are often insufficient in precision. Inspired by the success in controllable image generation (Zhang et al., 2023b; Mou et al., 2024b), several works attempt to add additional controls to video diffusion models. Early explorations (Wang et al., 2023; Guo et al., 2023) utilize structural controls, like sketch and depth maps, for video generation. However, these control signals are difficult to obtain during sampling, limiting their practical applications. Differently, recent works focus on motion control and introduce trajectory control for object motion (Wu et al., 2024; Mou et al., 2024a; Yin et al., 2023) and camera pose control for camera motion (Wang et al., 2024b; He et al., 2024; Bahmani et al., 2024). In this paper, we enhance the creative potential and flexibility of the video frame interpolation process, allowing users to produce plausible results following their control.
3 METHOD
Given two frames, 1 and 1", indicating the start and end frame in video, our goal is to generate the plausible contiguous video / [ by sampling from the conditional distribution p (7) 70, ). Here, n the number of frames in the video. Our method, termed Framer, supports user-interactive mode for customized point trajectories and an "autopilot" mode for video frame interpolation without trajectory inputs, as shown in Fig. 2a and Fig. 2b. In the following, we will introduce how we add frame conditions to the video diffusion model to achieve video interpolation in Sec. 3.1. To support user-interactivn drag control, we introduce a control branch in Sec. 3.2 for point trajectory guidance, which also enhances point correspondences across frames. In the "autopilot" mode, we estimate trajectories of matched points in the video with our novel bi-directional point tracking method, as illustrated in 3.3.
3

---

Published as a conference paper at ICLR 2025
(c) User Interactive Mode
(c) Trajectory Controlling
Condition
Trajestory Ciral
zaro
UNer Encader
User custemizes knypuints
Outpunt Video
Start
Video
Notse
Menching
Point
Trujectwry
Trojectery Uptoting
Ourpon
Bed Frome
(b) Autopliot Mode
(d) Video Frame Interpolation Fine-Tunke
Figure 2: Framer supports (a) user-interactive mode for eustomized point trajectories and (b) "autopilot" mode for frame interpolation without trajectory inputs. During training, (d) we fine- tune the 3D-UNet of pre-trained video diffusion model for frame interpolation. Afterward, (c) introduce point trajectory control by freezing the 3D-UNet and fine-tuning the controlling branch.
3.1 MODEL ARCHITECTURE
Large-scale pre-trained video diffusion models have strong visual prior the appearance, structure, and movement of open-world objects (Brooks et al., 2024). Our approach builds on the video diffusion model to exploit this prior. Considering that the Image-to- Video (I2V) diffusion model naturally supports first-frame conditioning, we choose the representative 12V diffusion model, Stable Video Diffusion (SVD) (Blattmann al., 2023a),
base model, as shown in Fig. 2d.
Based on the 12V model, we need to introduce additional end-frame conditioning to realize video interpolation. To preserve the visual prior of the pre-trained SVD) much possible, follow the conditioning paradigm of SVD and inject end-frame conditions in the latent space and semantic space, respectively. Specifically, we concatenate the VAE-encoded latent feature of the first frame, denoted as 22, with the noisy latent of the first frame, as did in SVD. Additionally, the latent feature of the last frame, 2", with the noisy latent of the end frame, 2,", considering that = is derived by adding noise to z" In addition, we extract the CLIP image embedding of the first and last frames separately and concatenate them for cross-attention feature injection. The U-Net ep matching objective:
concatenate
trained using the denoising
[||e co (zntt22,,2))
L Egg,e 2n N(O.)
3.2 INTERACTIVE FRAME INTERPOLATION
Ambiguity remains given the start and end frames, especially when the distinction between the two frames is large. The reason that multiple plausible interpolation results can obtained by sampling video from the conditional distribution / (1 1 (","") for the same input pair. To better align with the user intention, we introduce a control branch for custmized point trajectory guidance.
Technically, we train point trajectory- based control branch for correspondence modeling, as shown in Fig. 2c. During training, we use the following steps to obtain the point trajectory as control signals. Firstly, we randomly initialize some sampled points around fixed sparse grid in the first frame, and use Co- Co-Tracker (Karaev al., 2023) to obtain the trajectories of these points in the whole video. Secondly, we remove trajectories that Lastly, we sample the point trajectories with larger motions with greater probability. Considering that the users usually only input a small number of point trajectories, keep only to 10 trajectories during training. Please refer to the Appendix A for more details.
visible in more than half of the video frames.
After obtaining the sampled point trajectories, Dragnny (Wu et al., 2024) to transform the point coordinates into a Gaussian heatmap, denoted Ctraj, which is used input to the control module. We follow the conditioning mechanism in
follow DragNUWA (Yin al., 2023) and
4

---

Published as a conference paper at ICLR 2025
SIFT Fecture Motching
(o) Forward Point Tracking
Start Frome Feoture
Start Prens
Necrest Neighbar
Middle Frome Feature
(
F
*F (pt')
At
(v(-+1)
Neanest Neighbor
1
Unpdating
directional
Tracking
Figure 3: Point trajectory estimation. The point trajectory initialized by interpolating the coordinates of matched keypoints. In each de-noising step, we perform point tracking by finding the nearest neighbor of keypoints in the start and end frames, respectively. Lastly, We check the bi-directional tracking consistency before updating the point coordinate.
ControlNet (Zhang et al., 2023b) to incorporate the trajectory control. Specifically, we copy the encoder of 3D-UNet encode the trajectory map and add into the decoder of U-Net after zero- convolution (Zhang et al., 2023b). This training process can be represented as:
[||e - cg teit,,2,20 Ctras) )"
L E E., 2,
Here, eß is the combination of the denoising U-Net and the ControlNet branch.
Discussion. The introduction of point trajectory control not only facilitates user interaction, but also enhances the correspondence among points from diffferent frames. As demonstrated in experiments, this approach enables the model to effectively tackle challenging cases, such as when the start and end frames differ significantly.
3.3 "AUTOPILOT" MODE FOR FRAME INTERPOLATION
In practical applications, users may not always prefer manual drag controls. For this reason, we propose "autopilot" mode to enhance the case of use of our Framer. It mainly contains trajectory initialization and a trajectory updating process, as illustrated in Fig. 2b.
Trajectory Initialization. Given the start and end frames of the input video, can obtain the matching points between the two frames by applying points denoted as {p, ) where is the number of matching points. p. denotes the known anchor points on the trajectory. At initialization, it contains the matched points on the first and last the
algorithms. The matched
frames, i.e., P; [pg p. Although varying feature matching algorithms feasible, classical SIFT feature matching (Lowe, 2004) here for its simplicity and effectiveness, Subsequently, we can obtain the i-th trajectory by interpolating the anchor points p;. The estimated trajectory for all matched points, denoted as Graj = {& ) are used as the input condition in Eq. (2).
Trajectory Updating. Although the initial trajectory provides temporally consistent point correspondence, the trajectory obtained by connecting points in the first and last frames may not be accurate. Inspired by DragGAN (Pan al., 2023) and DragDiffusion (Shi al., 2023), we perform point tracking using the intermediate feature in U-Net to update the trajectories. Specifically, in each denoising step, we interpolate the U-Net features to the image resolution, denoted F. Here we use the feature of the penultimate upsampled block in U-Net, since it enjoys good trade-off between feature resolution and discriminativeness. We use (()) to represent the feature of the point p. which obtained via bilinear interpolation, since the coordinates may not be integers.
In each denoising step, we apply point tracking to update the coordinates of the middle frame points. We use nearest neighbor search in feature patch around the point. The feature patch represents a set of points whose distance to point is less than r, and is denoted as (p,r) = ((x, v)ll xp K r, yp < r). For middle frame point p in the k-th frame, find the
5

---

Published as a conference paper at ICLR 2025
Blended Inputs
6T
AMT
RIFE
FLAVR
FILM
LDMVFI
DynamiCrafter
SVDKFI
Ours
Figure 4: Qualitative comparison. For each method, we only present the middle frame of interpolated frames. The full results be seen in Fig. S7 and Fig. S8 in the Appendix.
nearest point relative to the anchor point pf via:
P1 O
arg min
|| (4.) -F (1))
(3)
of m))
Similarly, obtain the nearest point relative the last anchor point p!:
P1
arg min
| (a F (w))
(4)
As shown in Fig. 3, to further ensure the accuracy of the coordinates the updated points, we check the consistency of the two nearest points obtained by matching with pt and p²" When the distance between the two is less than a threshold r2, L..., P; k.m E Z (p.
.r2) we update the point coordinates R.0
by setting of = k.0 + p. 1/2. Then, we add the point to the anchor points list pa and interpolate p, get the updated trajectory a. which is used the input condition the next denoising step. The point trajectory estimation process also illustrated Alg. the Appendix.
4
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Our method is built on SVD and trained on the high-quality Open VienD-DDN- dataset (Nan et al., 2024). The model is trained two stages. Specifically, we first fine-tune the U-Net to accept the end frame conditions. Then, we train the controlling branch for point trajectory guidance. During the training of U-Net, we fixed the spatial attention and residual blocks, and only fine-tuned the input convolutional and temporal attention layers. The model is trained for 100K iterations using the AdamW optimizer Loshchilov & Hutter (2019) with learning rate of le-5. When training the control module, we fixed the U-Net and optimized the control module for 10K steps using the AdamW optimizer, with a learning rate of le-5. We obtained the point trajectories by pre- processing the video using the Co-Tracke (Karaev et al., 2023). All training is performed NVIDIA A100 GPUs, and the total batch size is 16. The training takes about 2 days. During sampling, it takes about 4.64 seconds generate 7 interpolated frames the DAVIS-7 dataset. On average, it takes 0.67 seconds to produce single interpolated frame. During "autopilot" mode sampling, we keep m 5 best matching keypoints for trajectory guidance, and the distance thresholds for point tracking are set as rı 5 and 3. Please refer to Appendix A for more details.
16
4.2 COMPARISON
Existing methods do not support drag-user interaction. Thus, we use the "autopilot" mode of Framer to make fair comparisons. We select baselines from two distinct categories. The first category includes the latest general diffusion based video interpolation models, including LDMVFI (Danier et al., 2024), DynamicCrafter (Xing al., 2023), and SVDKFI (Wang et al., 2024a). The second category encompasses traditional video interpolation methods, such as AMT (Li et al., 2023), RIFE (Huang et al., 2020), FLAVR (Kalluri al., 2023), and FILM (Reda al., 2022). We conduct

---

Published as a conference paper at ICLR 2025
DAVIS-7
UCF101-7
PSNKT
SSIM
LPIPSL
FIDL
FVDL
PSNR1
SSIM
[PPSS]
RID.
FVD.
Latency
AMT(Li et. nl., 2023)
21.66
0.7229
0.2860
39.17
245.25
26.64
0.9000
0.1878
37.80
270.98
0.165
RIFE (Huung 2020)
22.00
0.7216
0.2663
39.16
319.79
27.04
0,9020
0.1575
27.96
300.40
0.072
FLAVR (Kalluri et al.,.0200000 2023)
20.94
0.6890
0.5305
52.23
296.37
26.50
0.8982
0.1836
37.79
279.58
0.028
FILM (Reda et nl., 2022)
21.67
0.7121
0.2191
17.20
162.86
26.74
0.8983
0.1378
16.22
239.48
0.291
LDMVET (Dunier al., 2024)
21.11
269.72
0.8955
0.1446
270.33
9.340
DynamiCratterr (Xing et 2023)
15.48
0.4668
0.4628
35.95
468.78
17.62
0.7082
0,3361
61.71
646.91
13.166
SVDKFI (Wang al., 20246)
0.5274
3,4400
0.7991
0,2146
44.81
301.33
42,923
Franer (Ours)
21.23
0.7218
0:2525
27.13
1T5.65
25.04
0,8806
0.1714
31.69
181.35
4.644
Franar with Co Tracker (Ours)
22.75
0.7931
0.2199
27.43
102.31
27.08
0.9024
0.1714
32.37
159,87
4.644
Table 1: Quantitative comparison with existing video interpolation methods on reconstruction and generative metrics, evaluated on all 7 generated frames. The latency for generating 7 intermediate frames is assessed on the NVIDIA A6000 GPU, using seconds as the measurement metric.
Strat Frame
Generated Frames
End Frame
13
13
13
Goup o/m
13
13
13
B1
13
13
13
13
13
2
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
13
Figure 6: Results on user interaction. The first row is generated without drag input, while the other two are generated with different drag controls. Customized trajectories ares overlaid on frames.
quantitative and qualitative analyses, well as user studies, on two publicly available datasets: DAVIS (Pont-Tuset et al., 2017) and UCF101 (Soomro et al., 2012).
Qualitative Comparison. As shown in Fig. 4, our method produces significantly clearer textures and natural motion compared to existing interpolation techniques. It performs especially well in scenarios with substantial differences between the input frames, where traditional methods often fail to interpolate content accurately. Compared to other diffusion-based methods like LDMVFI and SVDKFI, Framer demonstrates superior adaptability to challenging cases and offers better control.
Quantitative Comparison. As discussed in VIDIM (Jain et al., 2024), reconstruction metrics like PSNR, SSIM, and LPIPS fail to capture the quality of interpolated frames accurately, since they penalize other plausible interpolation results that are not pixel-aligned with the original video. While generation metrics such as FID offer some improvement, they still fall short as they do not account for temporal consistency and evaluate frames in isolation. Despite this, we present the quantitative metrics for various settings on both datasets, where our method achieves the best FVD score among all baselines as in Table 1. We also evaluate Framer with 5 random point trajectories from ground-truth videos, estimated using Co- Tracke... As can be seen, "Framer with Co- Torarrceer" achieves superior performance even in reconstruction metric. For more comprehensive assessment of quality, we recommend reviewing the supplementary comparison videos.
User Study. Since quantitative metrics fall short in reflecting video quality, we further assessed our method's performance through a user study. In this study, participants reviewed video sets generated from the same input frame pair by both existing methods and our Framer. Participants assessed up to 100 randomly ordered video sets and selected the one they found most realistic. In total, 20 participants provided 1,000 ratings across these video sets. As illustrated in Fig. 5, the results demonstrate strong preference among human raters for the outputs produced by our method.
LONATE 075
4.4%
07%
AMT717%
Freser900 03
Figure Reults on human preference.
7

---

Published as a conference paper at ICLR 2025
Strat Frome
Generated Fromes
End Frome
Figure 7: Novel view synthesis both static (Ist row) and dynamic scenes (2nd row).
Strat
Generated Fromes
Frome
Figure 8: Applications on cartoon (Ist row) and sketch (2nd row) interpolation.
4.3 APPLICATIONS
Optional drag control. Given the same input start and end frames, multiple plausible results can satisfy the goal of video interpolation. With Framer, users can direct the motion of the entities in input frames with simple drags for their intention, simply obtain default interpolation result without drags. As shown in Fig. 6, the seal moves in varying directions given the same input frames.
Novel view synthesis (NVS) classical 3D vision task, with wide range of applications. Using images of different viewpoints as the start and end frames of the video respectively, we can realize the NVS from sparse viewpoint input by performing video interpolation. As shown in Fig. 7, our method achieves pleasing NVS in both static scenes (first row) and dynamic scenes (second and third rows). Taking the second row as an example, the house gradually moves out the scene as the camera keeps moving forward. In the meantime, the moves in the opposite direction to the camera and gradually takes up a larger proportion the frame.
Cartoon and sketch interpolation. We can dramatically simplify the process of cartoon video production, by interpolating manually created cartoon images. To this end, we tested our method on cartoon data. Although our method is not specifically trained cartoon videos, it produces appealing cartoon video results and supports both color images and sktech drawing frame interpolation, as shown in Fig. 8. For example, our method successfully models the motion of two objects, i.e., the front vehicle pulls sideways while the rear vehicle follows, as shown in the first row. In the third row, Framer produces a smooth motion of the hand lifting in sketch drawings.
Time-lapsing video generation. Tipe-lapse photography can vividly demonstrate slow changes that are difficult detect with the naked eye. Typically, it requires sufficient storage space hold a large amount of image data. Video interpolation provides a simple and effective way to obtain time-lapse videos by interpolating frames with only few images key moments. As shown in Fig. 9, Framer produces the smooth change of moon waxing and waning.
Frames
End
Figure Applications on time-lapsing video generation.
8

---

Published as a conference paper at ICLR 2025
Stret Frome
Generated Fromes
End Frome
Figure 10: Applications slow-motion video generation. The y-t slice highlighted in red on video
frames is visualized on the right.
Strat Frame
Generated Fromes
End Frome
Figure 11: Applications on image morphing. Customized trajectories overlaid end frames.
Slow-motion video generation enhances visual effects by highlighting fine details and allows closer examination of fast phenomena. Our Framer inherently supports fast frame interpolation, as demonstrated in Fig. 10, enabling smooth slow-motion effects suitable for films and animations.
Image morphing (Aloraibi, 2023) is popular image transformation technique with many applications in computer vision and computer graphics. Given two topologically similar images, aims generate series of reasonable intermediate images. Using tue two images the start and end frames, Framer can produce natural and smooth image morphing results. For example, in Fig. 1, we show the "evolution" process of Pokemon. More cases can be found in Fig. S16.
4.4 ABLATIONS STUDIES
We conducted ablation studies on the individual components of Framer to validate their effec- tiveness. The results are illustrated in Fig. 12. Our observations are follows. First, when the trajectory guidance removed (denoted as "w/o traj."), the foreground motorcycle exhibits significant distortion, shown in the 1st row of Fig. 12. Conversely, with the inclusion of trajectory guidance, the temporal consistency of the video is notably enhanced, as depicted the 2nd row. We believe this is due the enhancement of point correspondence modeling across frames. Second, removing trajectory updates (denoted as "w/o traj. update") or updating the trajectory without bi- directional consistency checks (denoted "w/o bi-directional") results in blurring in the wheel regions of the output video. We suspect the blurring is caused by the guidance of unnatural motion from inaccurate trajectories, which conflicts with the generation prior in the pre-trained diffusion model, leading to local blurring. In contrast, our method produces video frame interpolation results with natural motion and smooth temporal coherence. The quantitative results in Tables S1 and $2 in Appendix B further support these findings, showing a similar trend to the qualitative ablation experiments.
4.5 FAILURE CASES
Though Framer achieves superior performance of video frame interpolation, it still faces several limitations. Here examine potential scenarios where the model may underperform encounter failures, particularly in cases where it fails to capture object semantics, or no suitable correspon- dences between the input frames be found. The failure cases presented in Fig. 13.
Failure to capture object semantics. When a moving object appears blurred in the first and last frames of sequence, it's sometimes difficult for the model to accurately interpret the object's semantics, which can lead to unrealistic generation results. As highlighted DynamiCrafter (Xing al., 2023), incorporating text guidance during video frame interpolation enhances the model's

---

Published as a conference paper at ICLR 2025
wefo troj
troj upditte
w/o
bi- bi-diracti ional
ional
w/o
Inputs with SIFT matching
Fromer (Ours)
Figure 12: Ablations each component, "w/o trajectory" denotes inference without guidance from point trajectory, "w/o traj. update" indicates inference without trajectory updates, and "w/o bi" suggests trajectory updating without bi-directional consistency verification.
Strat Frome
Genenated Fromes
End Frome
C
Figure 13: Examples of failure cases.
comprehension of moving objects and helps resolve ambiguities associated with unclear objects. For this reason, we plan to introduce textual guidance to mitigate this problem in future work.
Lack of suitable matching points between the first and last frames. When there matching points between the first and last frames, Framer struggles to utilize trajectory guidance effectively, resulting in sub-optimal video interpolation results. The model faces challenges when generating a scene where a character present in the initial frame exits while another character emerges from the background. As shown in Fig. 13, while Framer can produce reasonable video interpolation results, there is still noticeable distortion in the image. To address this problem, we are exploring the use text guidance, as well as leveraging more advanced video models like Mochi (Team, 2024) and Cog Video-X (Yang et al., 2024), to improve our handling of such scenes.
suitable
5 CONCLUSION AND FUTURE WORK
In this paper, we introduce Framer, an interactive frame interpolation pipeline designed to produce smoothly transitioning frames between two images, guided by user-defined point trajectories. By harnessing user input point controls from the start and end frames, effectively guide the video interpolation process. Morcover, our method offers an "autopilot" mode that introduces a module automatically estimate keypoints and refine trajectories without manual input. Through extensive experiments and user studies, we demonstrate the superiority of our method in achieving promising results in terms of both the quality and controllability of the interpolated frames. However, challenges remain, particularly in transitioning between different clips. A potential solution involves splitting the clips into several keyframes and then interpolating these keyframes sequentially. Future work will focus on addressing these challenges.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partially supported by the National Key R&D Program of China (NO.20 N.0.20222D01001001000 Ningbo Science and Technology Bureau (Grant Number 20247291) and the National Natural Science Foundation of China (No. 62206244). CS is the corresponding author.
REFERENCES
Alyaa Aloraibi. Image morphing techniques; A review. Technium: Romanian Journal of Applied Sciences and Technology, 2023.
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace. Guocheng Qian, Michael Vasilkovsky, Hsin- Hsin-Ying Lee, Chaoyang Wang. Jiaxu Zou, Andrea Tagliasacchi, David B. Lindell, and Sergey Tulyakov. VD3D: taming large video diffusion transformers for 3d camera control. arXiv: Computing Research Repo., abs/2407.1278.. 2024.
Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Meme-net: Motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE Trans. Pattern Anal. Mach. Intell., 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv: Computing Research Repo., abss2231..151 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., 2023b.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue. Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. OpenAl technical reports, 2024.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv: Computing Research Repo., abs/2401 000047977 2024.
Shuhong Chen and Matthias Zwicker. Improving the perceptual quality of 2d animation interpola- tion. In European Conference Computer Vision, pp. 271-2877 Springer, 2022.
Xianhang Cheng and Zhenzhong Chen. Video frame interpolation via deformable separable convolution. In Assoc. Adv. Artif. Intell., 2020.
Xianhang Cheng and Zhenzhong Chen. Multiple video frame interpolation via enhanced deformable separable convolution. IEEE Trans. Pattern Anal. Mach. Intell., 2022.
Duolikun Danier, Fan Zhang, and David Bull. St-mfnet: A frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
multi-flow network for
Duolikun Danier, Fan Zhang, and David Bull. LDMVFI: video frame interpolation with latent diffusion models. In Assoc. Adv. Artif. Intell. 2024.
Tianyu Ding, Luming Liang, Zhihui Zhu, and llya Zharkov. CDFI: compression-drine driven network design for frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog,, 2021.
Jiong Dong, Kaoru Ota, and Mianxiong Dong. Video frame interpolation: A comprehensive survey. ACM Trans. Multim. Comput. Commun, Appl., 2023.

---

Published as a conference paper at ICLR 2025
Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Fernández Abrevaya, Michael J. Black, and Xuaner Zhang. Explorative inbetweening of time and space. arXiv: Computing Research Repo., abs/2403.14611, 2024.
Shurui Gui, Chaoyue Wang. Qihua Chen, and Dacheng Tao. Featureftow: Robust video interpolation via structure-t-texxtur generation. In JEEE Conf. Comput. Vis. Pattern Recog., 2020.
Yuwei Guo, Ceyuan Yang. Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls text-to-video diffusion models. arXiv: Computing Research Repo., abs/2311.1699) 2023.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv: Computing Research Repo., abs/2444.021011 2024.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. RIFE: real-time intermediate flow estimation for video frame interpolation. arXiv: Computing Research Repo., abs/2011.06294 2020.
Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Holynski, Ben Poole, and Janne Kontkanen. Video interpolation with diffusion models. arXiv: Computing Research Repo., abs/2404.01203,, 2024.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsual Yang, Erik G. Learned-Millrr, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
Xin Jin, Longhai Wu, Guotao Shen, Youxin Chen, Jie Chen, Jayoon Koo, and Cheul-Hee Hahm. Enhanced bi-directional motion estimation for video frame interpolation. In JEEE Winter Conf. Appl. Comput. Vis., 2023.
Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran. FLAVR: flow-agnostic video representations for fast frame interpolation. In JEEE Winter Conf. Appl. Comput. Vis., 2023.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: is better to track together. arXiv: Computing Research Repo.,
foss/3077.06335 2023.
Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang, and Jie Yang. Ifrnet: Intermediate feature refine network for efficient frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Hyeongmin Lee, Taeoh Kim, Yoe-Young Chung, Daehyun Pak, Yuseok Ban, and Sangyoun Lee. Adacof: Adaptive collaboration of flows for video frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Changlin Li, Guangyang Wu, Yanan Sun, Xin Tao, Chi-Keung Tang, and Yu-Wing Tai. H-VFI: hierarchical frame interpolation for videos with large motions. arXiv: Computing Research Repo., abs/22 1. 1309, 2022.
Zhen Li, Zuo-Liang Zhu, Linghao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. AMT: all-s multi-field transforms for efficient frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
Yihao Liu, Liangbin Xie, Siyao Li, Wenxiu Sun, Yu Qiao, and Chao Dong. Enhanced quadratic video interpolation. In Fur. Conf. Comput. Vis. Worksh. 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Int. Conf. Learn. Represent., 2019.
David G. Lowe. Distinctive image features from scale-invariant keypoints. Int. Comput. Vis., 2004.
12

---

Published as a conference paper at ICLR 2025
Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang. Ying Shan, and Jian Zhang. Revideo: Remake video with motion and content control. arXiv: Computing Research Repo., 0bs/24005138665 2024a.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Assoc. Adv. Artif. Intell., 2024b.
Kepan Nan, Rui Xie, Penghao Zhou, Tiehan Fan, Zhenheng Yang. Zhijie Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-Im: A large-scale high-quality dataset for text-to-video generation. arXiv: Computing Research Repo., abs/2407.02371, 2024.
Alejandro Newell, Kaiyu Yang, and Jia Deng- Stacked hourglass networks for human pose estimation. In Eur. Conf. Comput. Vis., 2016.
Simon Niklaus and Feng Liu. Context-aware synthesis for video frame interpolation. IEEE Conf. Comput. Vis. Puttern Recog., 2018.
Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution. In Int. Conf. Comput. Vis., 2017.
Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, and Christian Theobalt. Drag your GAN: interactive point-based manipulation on the generative image manifold. In Erik Brunvand, Alla Sheffer, and Michael Wimmer (eds.), SIGGRAPHI, 2023.
Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim. BMBC: bilateral motion estimation with bilateral cost volume for video interpolation. In Eur. Conf. Comput. Vis., 2020.
Junheum Park. Chul Lee, and Chang-Su Kim. Asymmetric bilateral motion estimation for video frame interpolation. Int. Conf. Comput. Vis., 2021.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Homunng, and Luc Van Gool. The 2017 DAVIS challenge video object segmentation. arXiv: Computing Research Repo., abs/1704.00675, 2017.
Fitsum A. Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. FILM: frame interpolation for large motion. Eur. Conf. Comput. Vis., 2022.
Jiaming Shen, Kun Hu, Wei Bao, Chang Wen Chen, and Zhiyong Wang. Bridging the gap: Sketch- aware interpolation network for high-quality animation sketch inbetweening. In ACM Int. Conf. Multimedia, 2024.
Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent Y. F. Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based based image editing. arXiv: Computing Research Repo., abs/2306. 14435, 2023.
Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. XVFI: extreme video frame interpolation. In Int. Conf. Comput. Vis., 2021.
Li Siyao, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris Metaxas, Chen Change Loy, and Ziwei Liu. Deep animation video interpolation in the wild. In IEEE Conf. Comput. Vis. Pattern Recog.. 2021.
Li Siyao, Tianpei Gu, Weiye Xiao, Henghui Ding, Ziwei Liu, and Chen Change Loy. Deep geometrized cartoon line inbetweening. In Int. Conf. Comput. Vis., 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arxiv: Computing Research Repo., abs/1212.0402. 2012.
13

---

Published as a conference paper at ICLR 2025
Shuochen Su, Mauricio Delbracio, Jue Wang. Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang. Deep video deblurring for hand-held cameras. In IEEE Conf. Comput. Vis. Pattern Recog,, 2017.
Genmo Team. Mochi 1. httpss///gggttbb....//g/////////
2024.
Xiang Wang, Hangjie Yuan, Shiwei Zhang. Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. In Adv. Neural Inform. Process. Syst., 2023.
Xiaojuan Wang. Boyang Zhou, Brian Curless, Ira Kemelmacher Shanzeerre-,liiiemnn, Aleksander Holynski, and Steven M Seitz. Generative inbetweening: Adapting image-to-vide models for keyframe interpolation. arXiv: Computing Research Repo., abs/2408. 2024a.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. In SIGGRAPH, 2024b.
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. arXiv: Computing Research Repo., abs/2403.07420, 2024.
Xiang Xiaoyu, Liu Ding, Yang Xiao, Zhu Yiheng, and Shen Xiaohui. Anime2sketch. https:
//github. com/ genmoai
satmmmaiimodess... /mw.els.dlsssoo.//mm/...m 2021.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv: Computing Research Repo., abs/2310. 12190, 2023.
Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Tooncrafter: Generative cartoon interpolation. arXiv: Computing Research Repo., abs/2405.17933, 2024.
Xiangyu Xu, Li Si-Yao, Wenxiu Sun, Qian Yin, and Ming Hsuan Yang. interpolation. In Adv. Neural Inform. Process. Syst., 2019.
Quadratic video
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T. Freeman. Video enhancement with task-oriented flow. Int. J. Comput. Vis., 2019.
Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. RAPHAEL: text-to-image generation via large mixture of diffusion paths. In Adv. Neural Inform. Process. Syst., 2023.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.060) 2024.
Shengming Yin, Chenfei Wu, Jian Liang Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control video generation by integrating text, image, and trajectory. arXiv: Computing Research Repo., abs/2308.08088 2023.
Zhiyang Yu, Yu Zhang, Deyuan Liu, Dongqing Zou, Xijun Chen, Yebin Liu, and Jimmy S Ren. Training weakly supervised video frame interpolation with events. In Int. Conf. Comput. Vis., 2021.
Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, and Limin Wang. Extracting motion and appearance via frter- attention for efficient video frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2023a.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Inf. Conf. Comput. Vis., 2023b.
Zhihang Zhong, Gurunandan Krishnan, Xiao Sun, Yu Qiao, Sizhuo Ma, and Jian Wang. Clearer frames, anytime: Resolving velocity ambiguity in video frame interpolation. In Eur. Conf. Comput. Vis., 2024.
14

---

Published as a conference paper at ICLR 2025
Kun Zhou, Wenbo Li, Xiaoguang Han, and Jiangbo Lu. Exploring motion ambiguity and alignment for high-quality vidco frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A MORE IMPLEMENTATION DETAILS
During training, we sample 14 consecutive frames from videos, with a spatial resolution of 512x320. Specifically, we center-crop the video an aspect ratio of 512/320, then resize the video frames to the resolution of 512 x 320. Random horizontal flip is utilized for data augmentation. We sample the video in temporal dimension, with a frame interval of 2. For the training of the point trajectory- Specifically, we follow Re ReVideo (Mou al., 2024a) and sample the trajectories by setting the normalized lengths of the trajectories as sampling probabilities. During "autopilot" mode sampling, we use the Euler sampler with 30 diffusion steps total. For point tracking Sec. 3.3, we use the output feature of the second decoder block in the 3D-UNet. We provide Algorithm in Alg. to illustrate the point trajectory estimation method. We resize the shorter side of the video to the length of 512, then center crop the video to the resolution of 512 320.
ControlNet, we sample 1 to 10 trajectories with larger motions for training.
We transform 2D points into gaussian heatmaps, following the practice Stacked Hour- glass (Newell et al., 2016), DragNUWA (Yin et al., 2023), and DragAnthing (Wu al., 2024). Specifically, we initialize a canvas map with the same height and width of the input video, setting all values to zero. Subsequently, for each trajectory point the coordinate position p, create grid region centered on this point with pixel area of 41x41. The center of this area (coordinate p) assigned a value of 1, while the values decrease in accordance with Gaussian distribution the distance from increases. The variance of this Gaussian distribution is set to 8 in both the horizontal and vertical directions.
Algorithm Point trajectory estimation algorithm in "Autopilot mode".
Input:
Io: start image, I,: end image
mt number of matching points
distance threshold for nearest neighbor search rı
distance threshold for the bi-directional consistency check 72
Output: {p: = Anchor point list for m trajectories Ctraj: Esitimated point trajectory
Apply SIFT point matching
(p.)r 1= (Ip) p² pi SIFT((
p. denotes the known anchor points the trajectory
> Point trajectory updating
for i = 1...m do
Interpolate the anchor points to obtain the current trajectory Ceray Interpolaae(p.)
for = 1...n do
> Forward point tracking
p arg min
||F (q.*) - (1
ot *ER(pf.r))
> Backward point tracking
= arg min
|F (at") - F (pP))
> Bi-directional consistency check and point update
p. k.n E pp. .2)
then p: + p. (v. k.0 + p, 1/2 Add updated point to anchor point list
end for end for
16

---

Published as a conference paper at ICLR 2025
34
FID on DAVIS-7
FID on UCF101-7
26-
UCF10
FVD on
24-
26
3D-UNet Decoder Feature Index for Point Tracking
3D-UNet Decoder Feature Index for Point Tracking
Figure S1: Ablations on diffusion feature for point tracking at test time, experiments conducted on
DAVIS-7 (left) and UCF101-7 (right).
B MORE DETAILED ABLATION RESULTS
B. QUALITATIVE RESULTS FOR ABLATION STUDIES.
In Fig. 12, we show the qualitative results for ablation studies. We supplement these results with the quantitative results in Table S1 and Table S2, which show a similar trend to the qualitative ablation experiments.
DAVIS-7
UCF101-7
PSNR
SSIMÍ
LPIPS]
FID.
FVD.
PSNRT
SSIM
LPIPS.
FDD.
FVDJ
w/o trujectory
20.19
0.6831
0.2787
28.25
128.71
24.16
0.8677
0.1798
32.64
195.54
w/o traj. updating
20.82
0.7054
0.2621
27.33
120.73
24.69
0.8748
0.1842
31.95
187.37
w/o bi-directional
20.94
0.7102
0,2602
27.23
116.81
24.73
0.8746
0.1845
31.66
183.74
Framer (Ours) 21.23
0.7218
0.2525
27.13
115.65
25.04
0,8806
0.1714
31.69
181.55
Table S1: Ablations on each component, evaluating all generated frames. "w/o trajectory" denotes inference without guidance from point trajectory, "w/o traj. updating" indicates inference without trajectory updating, and "w/o bi" suggests trajectory updating without bi-directional consistency verification.
DAVIS-7 (mid-frame)
UCF101-7 (mid-frame)
PSNRT
SSIM
LPIPS4
FID
PSNRT
SSIM
IPIPST.
FID.
w/o trajectory
19.30
0.6504
0.3093
57.10
23.14
0.8523
0.1967
54.98
w/o traj. updating
19.84
0,6700
0.2935
55.37
23.60
0,8590
0.2009
53.83
w/o bi-directional
19.95
0.6739
0.2919
54.75
23.65
0.8586
0.2016
53.54
Framer (Ours)
20.18
0.6850
0.2845
55.13
23.92
0.8646
0.1889
53.33
Table S2: Ablations each component, evaluating only the middle frame out of all 7 generated frames. "w/o trajectory" denotes inference without guidance from point trajectory, "w/o traj. updating" indicates inference without trajectory updating, and "w/o bi" suggests trajectory updating without bi-directional consistency verification.
B.2 ABLATIONS ON DIFFUSION FEATURE FOR POINT TRACKING.
As detailed in Sec. 3.3, we perform point tracking using the diffusion feature for point trajectory updating. Here perform ablated experiments on the selection of the diffusion feature. The results are shown in Fig. S1. can be seen that in both DAVIS-7 and UCF-7, point tracking with the output feature from the second diffusion block gives rise to the best-performing results in FVD.
17

---

Published as a conference paper at ICLR 2025
on DAVIS-7
on UCF101-7
UCF10A
26-
FVD
26
26
0-22
0:18
D.34
0-30
6:30
12:30
18:30
24-30
0.8
2-12
0-18
634
0:30
6:30
12:30
10:10
24-30
Start-End Steps for Guidance
Start-End Steps Steps for Guidance
Figure S2: Ablations on the start and end diffusion steps for correspondence guidance, experiments conducted on DAVIS-7 (left) and UCF101-7 (right). We use total sampling step of 30.
130
200
325
195
FID on DAVIS-7
120
on UCF101-7
190
UCF101
3155
185
on
310
180
305
175
100
170
15
20
Number of Trajectories for Guidance
Number Trajectories for Guidance
Ablations on the number of trajectories for guidance during sampling, experiments Figure S3:
conducted on DAVIS-7 (left) and UCF101-7 (rightt...
B.3 ABLATIONS ON DIFFUSION STEPS FOR CORRESPONDENCE GUIDANCE.
We ablate the diffusion steps for correspondence guidance by only applying the guidance at the early steps late steps in diffusion sampling. The results are shown in Fig. S2. As can be seen, the early steps are often more important than the steps for correspondence modeling. For example, DAVIS-7, pleasing FVD be obtained when performing guidance only on 0-18 diffusion steps. By contrast, performing guidance only on 18-30 diffusion steps brings litter improvements. We speculate that this is because the early diffusion steps focus the structural information of the video, while the late diffusion steps focus on the texture and details (Xue et al., 2023). The correspondence guidance at early steps already helps the model obtain reasonable video structure. In the implementation, we simply apply correspondence guidance in all diffusion steps, without detailed searches the hyper-parameter.
B.4 ABLATIONS ON THE NUMBER OF TRAJECTORIES FOR CORRESPONDENCE GUIDANCE.
As described in Sec. 3.3, we use m trajectories for correspondence guidance during sampling. Here perform ablated experiments this syper-parameter, and the result is shown in Fig. S3. It can be seen that sampling with the trajectories leads to the best performance. Thus we set m by default.
18

---

Published as a conference paper at ICLR 2025
DAVIS-7 (mid-frame)
UCF101-7 (mid-frame)
PSNRT
SSIMT
LPIPS
FIDI
PSNRT
SSIMI
LPIPS,
FIDL
AMT cLi al., 2023)
20.59
0.6834
0.3564
100.36
25.24
0.8837
0.2237
75.97
RIFE (Huang et al., 2020)
20.74
0.6813
0.3102
80.78
25.68
0.8842
0.1835
59.33
FLAVR (Kalluri et al., 2023)
19.93
0.6514
0.4074
118.45
24.93
0.8796
0.2164
79.86
FILM (Reda al., 2022)
20.28
0.6671
0.2620
48.70
25.31
0.8818
0.1623
41.23
LDMVFI (Danier al., 2024)
19.87
0.6435
0.2985
56.46
25.16
0.8789
0.1695
43.01
DynamicCrafter (Xing et al., 2023)
14.61
0.4280
0.5082
77.65
17.05
0.6935
0.3502
97.01
SVDKFI (Wang al., 2024a)
16.06
0.4974
0.3719
53,49
20.03
0.7775
0.2326
69.26
Framer (Ours)
20.18
0.6850
0.2845
55.13
23.92
0.8646
0.1889
53.33
Framer with Co-Tracker (Ours)
21.94
0.7693
0.2437
55.77
25.86
0.8868
0.1873
54.64
Table S3: Quantitative comparison with existing video interpolation methods on reconstruction and generative metrics, evaluated only on the middle frame out of all 7 generated frames.
PSNR
SSIM
LPIPS
FIDI
FVD1
NIQEL
2x
23.60
0.8203
0.1992
24.12
N/A
5.0753
4x
23.08
0.7899
0.2091
25.92
93.42
4.9948
8x
21.23
0.7218
0.2525
27.13
115.65
5,0598
Table S4: Evaluation on generating varying numbers of middle frames.
C MORE QUANTITATIVE RESULTS
C.I MORE DETAILS ON COMPARISON WITH PREVIOUS METHODS
We follow the practice of VIDIM (Jain et al., 2024) and perform the quantitative evaluation on DAVIS-7 and UCF101-7 datasets using both reconstruction and generative metrics. Both DAVIS- 7 and UCF101-7 are obtained by sampling 7 consecutive video frames from the corresponding datasets. We use all videos in the DAVIS dataset and a subset 400 videos in the UCF101 dataset.
In Table S3 we provide the quantitative comparison based on the middle frame of the 7 interpolated video frames. Besides, in Fig. S7, Fig. S8, Fig. S9, and Fig. S10, we show more qualitatively comparisons with exiting methods.
C.2 GENERALIZATION TO VARYING NUMBER OF MIDDLE FRAMES
Framer supports generating a variable number of frames between the start and end frames. Video interpolation at different frame rates can be achieved by adjusting the number of temporal channels in the initial noise during sampling. Although the model is fine-tuned using 14 consecutive frames sampled from the training videos, it generalizes effectively to frame interpolation with varying numbers of intermediate frames. In the main text, we evaluated frame interpolation with 7 intermediate frames (8x in the temporal dimension) to align with the evaluation settings in VIDIM (Jain et al., 2024). Here, extend analysis to explore different settings for video frame interpolation, including 2x and 4x. The results of these experiments are presented Table S4. As shown in the table, Framer delivers pleasing results for video frame interpolation across different numbers of intermediate frames, highlighting its robustness.
C.3 QUANTITATIVE RESULTS ON MORE BENCHMARKS
Following the practice of Zhong al. (2024), conduct experiments on the Vimeo90K septuplet dataset (Xue et al., 2019), X4K 10000SS (Sim et al., 2021), and Adobe240 (Su et al., 2017) to evaluate the performance of Framer. The results are shown Table S5 and Table S6. It can be seen that Framer achieves competitive results on these datasets, especially on the NIQE metric, since it does not require video results be pixel-aligned with the ground truth.
C.4 RESULTS ON CARTOON INTERPOLATION
We collected 500 cartoon videos from the Internet make comparisons with existing cartoon interpolation methods, including Animelnterp (Siyao et al., 2021), EISAI (Chen & Zwicker, 2022), and TronCraft (Xing al., 2024). Following the practice of ToonCraft (Xing al., 2024),
19

---

Published as a conference paper at ICLR 2025
PSNRT
SSIMT
LPIPS.
NIQF]
RIFE (Huang al, 2020)
28.22
0.912
0.105
6.663
IFRNet (Kong et 2022)
28.26
0.915
0,088
6.422
AMT (Li et 20,,)
28.52
0,920
0.920
6.866
EMA-VFI (Zhang et al., 2023a)
29.41
0,928
0.086
6.736
AnterpAny-Cleareee [D] (Zhong et al, 2024)
29.20
0.929
0.092
6.475
InterpAny-Clcen Clccrerr [D, RJ (Zhong ct al,, 2024)
28.84
0,926
0.081
6.286
LDMVFI (Danier et al., 2024)
27.43
0,912
0.092
6.279
DynamiCrafted (Xing al., 2023)
26.51
0.891
0.128
6.912
SVDKFT (Wang et al., 2024a)
28.01
0.903
0.082
5,969
Franer (Ours)
28.32
0.918
0.072
5.623
Table S5: Quantitative results Vimeo90K (Xue et al., 2019) septuplet dataset.
X4K1000FPS7
Adobe240
PSNR
SSIMT
LPIPS]
NIQE,
PSNK1
SSIM
LPIPST
NIQEL
RIFE (Huang et al. 2020)
36.30
0,967
0.040
7.130
30.24
0.939
0.073
3.206
InterpAny-Clese Clearer [D] (Zhong 2024)
36.80
0.964
0.032
6.936
30.47
0.938
0.057
4.974
InterpAny-Cleares [D. RJ (Zhong et ul., 2024)
36.26
0.964
0.032
6.924
30.30
0.937
0.054
4,907
LDMVH (Danier al., 2024)
29.95
DynamiCrafier (Xing al., 2023)
35.42
0.925
0.051
7.116
27.54
0.883
0.084
5.824
SVDKFI (Wang et al., 2024g)
36,31
0.938
0.046
6.621
28.43
0.903
0.069
5.695
Framer (Ours)
36.38
0.955
0.033
5.632
29.89
0.914
0.068
5.045
Table $6: Quantitative results on X4K1000FPS (Sim et al., 2021) and Adobe240 (Su et al., 2017)
dataset.
tested 512x320 resolution and each video contains 16 frames. The results are follows. As can be seen, even though Framer is not specifically trained on cartoon data, it achieves superior be seen in Fig. S4. We found that when the
results this task. The qualitative comparison difference between the start and end frames is large, existing methods often fail to connect the first and last frames, producing blurred results (Siyao al., 2021; Chen & Zwicker, 2022), or directly jumping from the start frame to the end frame (Xing et al., 2024). In contrast, Framer can utilize the correspondence between input frames and the guidance of the trajectory to connect input frames, thus producing coherent videos.
C.5 RESULTS ON SKETCH INTERPOLATION
Comparisons on the sketch interpolation task. We extracted the sketch of these cartoons using Anime2Sketch (Xiaoyu et al., 2021) make comparisons with different methods. It can be seen that Framer achieves competitive performance, demonstrating its superiority in sketch interpolation. The qualitative comparison can be seen in Fig. S5.
lle
Aninsctnterp
ll
EISAI
He
TounGrafter
lle
Framer
Figure S4: Qualitative comparison on cartoon interpolation.
20

---

Published as a conference paper at ICLR 2025
PSNRT
SSIMT
LPIPS
FID.
FVD1
NIQEL
Animelnterp (Syya el al,, 2021)
14.51
0.6196
0,4521
55.13
322.14
6.91
EISAI (Chen & Zwicker, 2022)
13.86
0,5142
0.4132
62.41
483.09
7.24
ToonCrafter (Xing et al., 2024)
16.34
0.5988
0.3576
34.74
208.34
(Ours)
19.33
0.6591
0.2280
24.72
142.85
Table S7:
Quantitative results on cartoon interpolation.
PSNRT
SSIMF
LPIPS.
FID.
FVD.
NIQEL
Animeinterp (Siyao et al., 2021)
16.77
0.4374
0.3263
55.13
281.54
12.13
EISAI (Chen & Zwicker, 2022)
15.25
0.3872
0.3513
73.08
352.72
12.75
ToonCrafter (Xing et al., 2024)
17.07
0.4767
0.2978
48.89
135.53
12.00
Animelnbet (Siyao ct al., 2023)
21.42
0.5829
0.2413
31.82
136.07
11.51
Framer (Ours)
20.88
0.6489
0.1237
28.63
106.48
12.53
Table S8:
Quantitative results on sketch interpolation.
D MORE QUALITATIVE RESULTS
D. MORE QUALITATIVE RESULTS APPLICATIONS.
We provide more qualitative results drag control, novel view synthesis, cartoon and sketch interpolation, time-lapsing video generation, slow-motion video generation, and image morphing in Fig. S11, Fig. S12, Fig. $13, Fig. S14, Fig. S15, and Fig. $16, respectively.
D.2 QUALITATIVE RESULTS INTERPOLATING COMPLEX MOTIONS.
We additionally provide qualitative results interpolation frames in complex scenarios with large motions, shown in Fig. $6.
E DISCUSSIONS ON LIMITATIONS
Framer is built top of the large-scale pre-trained video diffusion model, thus it inherits the limitations of the pre-trained model. Moreover, the point trajectories in Framer rely on the matching points between the input image pair for interpolating complex motions. While this is step forward compared with current models that can only simply motions, our method still faces difficulties when the differences between the front and back frames are so large that no matched points can found at all. Thus, we will explore more powerful pre-trained video diffusion models, as well as training video interpolation models on larger-scale video data in the future. Lastly, our approach currently only supports drag control and does not explore other interaction methods. In the future, we will continue to explore other user-fite friendly controls such as text control and camera pose control.
EISAT Aninctittrp
TounGraftee
Frumer
Figure S5: Qualitative comparison on sketch interpolation.
21

---

Published as a conference paper at ICLR 2025
Strat Frame
Genercted Frames
End Frome
Figure S6: Results on input frames with large differences.
22

---

Published as a conference paper at ICLR 2025
GT
AMT
RIFE
FLAVR
FILM
LDMVFI
DynamiCrafter
SVDKFI
Ours
Ours w/ tracker
Frame
Frame
Frame 3
Frame 4
Frame 5
Frame
Frame 7
Figure S7: More qualitative comparison with existing methods. "GT" strands for ground truth.
23

---

Published as a conference paper at ICLR 2025
GT
AMT
RIFE
FLAVR
FILM
LDMVFI
DynamiCrafter
SVDKFI
Ours
Ours w/ tracker
Frame
Frame
Frame 3
Frame 4
Frame 5
Frame
Frame 7
Figure S8: More qualitative comparison with existing methods. "GT" strands for ground truth.
24

---

Published as a conference paper at ICLR 2025
Siegerseite. ewz Siegerselle. ewz Siegerselte. ewar Siegerselte. Siegerselte. Siegerselte. Slegerselle.
Siegerseite. ewa Siegerseite. ewa Siegerseite. ewa Siegerseite. ewar Siegerselte. ewa Siegerselte. ewart Siegerseite. ewa
AMT
Siegerseite. ewz Siegerseile. ewz Siegerselte. ewa Siegerseite. ewarr Siegerselte. ewa Siegerselte. ewart Siegerselle. ewa
RIFE
Siegerseite. ewz Siegerseite. ewz Siegerselte. ewz Siegerselte. ewa Siegerseite. ewa Siegerselte. ewa Siegerselle. ew
FLAVR
Siegerseite. ewz Siegerseile. ewz Siegerseite. BWZ Siegerseite. Siegerseite. Siegerselte. Siegerseile.
FILM
Siegerseite. ewz Siegerseite. ewa Siegerseite. ewar Siegerseite. Siegerselte. ewart Siegerselte. ewart Slegerseite. ewa
LDMVFI
er Siegerselne. ewer Siegerselte. ever Siegerseite. ever Siegerselte. ever Siegerselte. ewer Siegerseite. ew Siegerselte. ewa
DynamiCrafter
Siegerseite awa Siegerseite. ewz Siegerseite. ewa Siegerseite. ewz Siegerseite. ewa Siegerseife. ewa Siegerseife. ew
SVDKFI
Siegerseite. ewz Siegerseite. ewz Siegerseite. ewz Siegerseite. Siegerseite. Siegerseite. ewar Siegerselte.
Ours
Siegerseite. ewz Siegerseite. ewa Siegerseite. ewar Siegerseite. ewarr Siegerseite. ewart Siegerselte. ewarr Siegerselte. ew
Ours w/ tracker
Frame
Frame
Frame
Frame 4
Frame
Frame
Frame 7
Figure S9: More qualitative comparison with existing methods. "GT" strands for ground truth.
25

---

Published as a conference paper at ICLR 2025
GT
AMT
RIFE
FLAVR
FILM
LDMVFI
DrnamiCrafter
SVDKFI
Ours
Ours w/ tracker
Frame
Frame
Frame 3
Frame 4
Frame 5
Frame
Frame 7
Figure S10: More qualitative comparison with existing methods. "GT" strands for ground truth.
26

---

Published as a conference paper at ICLR 2025
Strat Frome
Generated Frames
End Frome
Figure S11: More results interaction. We show the results of two trajectory controls with the same input image pair.
Strat Frame
Generated Fromes
End Frome
Figure S12: More results on novel view synthesis. The first and second rows show results on static
and dynamic scenes, respectively.
Strat Frame
Generated Frames
End Frame
(a) Cartoon Interpolation
(b) Interpolation
Figure S13: More results on (a) cartoon and (b) sketch interpolation.
27

---

Published as a conference paper at ICLR 2025
Strat Frome
Generated Fromes
End Frome
Figure S14: More results on time-lapsing video generation.
Strat Frome
Generated Fromes
End Frome
Figure $15: More results slow-motion video generation. The x-t slice highlighted in red on video
frames is visualized on the right.
Strat Frame
Generated Frames
End Frame
Figure S16: More results on image morphing.
28


Reviews:
[{"review_id": "MwTwKdGOr3", "reviewer": "Reviewer_urco", "summary": "The paper proposes a new technique for generating frame interpolations that can incorporate user input to drive the interpolation process between two given frames. The method uses Stable Video Diffusion as the base model with the inputs modified to include both the first and last frame. Additionally, a control branch consisting of gaussian heatmaps is encoded and included in the model. The method is evaluated on a variety of image categories and on publicly available datasets.", "questions": "- What are the differences in training / inference methodology between the autopilot and user-guided mode?\n- Are there categories of videos where the results aren't as high quality as other categories?\n- Could you explain a bit more about \"considering that the conditions and the corresponding noisy latents are spatially aligned\" near line 183?\n- Can you explain how you transform 2D points into gaussian heatmaps? Are any other modalities like text used here? References to DragNUWA or DragAnything might be too recent. Consider also citing an older source such as from 2014 \"Stacked Hourglass Networks for Human Pose Estimation\" if it makes sense.\n- Can the autopilot mode re-use the randomly sampled points ~line 197 instead of using feature matching to find new correspondences over time?\n- Have you considered simpler alternatives for the Trajectory updating section?\n- How do you handle camera motion in your framework?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is mostly well written and easy to follow. The novelty here lies in the control branch for guiding the interpolation process that allows for fine-grained user-friendly control over the interpolation process.. The authors evaluate several existing methods on a variety of input types like real-world, sketch, cartoon, etc both qualitatively and quantitatively and produce high quality results.  Also included in a technique to produce the control signals automatically using bidirectional consistency checks. Several key design choices have been evaluated thoroughly, such as number of trajectories, bidirectional consistency.", "weaknesses": "- There is no mention of inference speed, training speed or overall training methodology (end-to-end, multi stage, pretraining, etc)\n- There appears to be no underlying camera model for handling the \"novel view synthesis\" examples. Having such a model could potentially improve quality while providing another user-control.\n- The paper isn't very easy to follow specifically around the trajectory update for autopilot mode section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5OPhykrepe", "reviewer": "Reviewer_SBQc", "summary": "This paper presents an interactive frame interpolation framework to produce smoothly transitioning frames between two images, interactively or automatically. By harnessing user input point controls from the start and end frames, this method can effectively generate the video frames for different applications including drag control, novel view synthesis, cartoon and sketch interpolation, time-lapsing video generation, slow-motion video generation, and image morphing.", "questions": "1)\tCan the frame interpolation function support arbitrary number of frames generation between the start and end inputs?\n2)\thow to process the case when the number of corresponding points between frames is too few?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1）\tThe method is straight forward but effective, which can be used for many applications;\n2）\tMost results shown in this paper are impressive.\n3）\tAblation experiments are sufficient to evaluate the method.", "weaknesses": "1)\tThe component of the Framer includes Trajectory preprocessing, Trajectory control and Interpolation. Although the method is effective, the novelty is limited. Trajectory control is designed by following DragNUWA, and the Interpolation is mainly based on SVD that determines the performance of this method.\n2)\tThe examples shown in this paper are a bit simple. The motion and the difference between two key frames (first and last frames) are not complex. More complex scenarios, as cases shown in the paper “Training Weakly Supervised Video Frame Interpolation with Events”, should be considered to demonstrate the superiority of this method.\n3)\tThe image quality of some results shown in this paper degrades compared to the first input frame. For example, the IQ of frames in “laugh.mp4” is worse than that of the start and end frames;\n4)\tThe reconstruction metric of this method is not best when comparing it to related methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U68CdzY1jp", "reviewer": "Reviewer_4487", "summary": "This paper presents a diffusion-based architecture, *Framer*, for interactive frame interpolation, introducing last-frame conditioning and keypoint trajectories to achieve controllable transitions. While the qualitative results are impressive, the practical application, related work discussion, and unique technical contributions need further improvement.", "questions": "Please address the issues raised in the weaknesses section. This reviewer is willing to increase the rating if the issues are well addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The method’s introduction of keypoint trajectories addresses motion ambiguity, leveraging the stable video diffusion model to improve interpolation fidelity.   \nThe qualitative outcomes are notably strong.", "weaknesses": "1. **Related Work**: The paper does not sufficiently discuss previous work tackling motion ambiguity and interactive frame interpolation, such as [1] and [2], which address motion/velocity ambiguities. And [2] is the early work in interactive VFI. A more comprehensive discussion would clarify the novelty and contributions of *Framer*.\n2. **Real-World Application**: The practical applicability of this method is unclear, especially given that real-world videos typically run at frame rates like 24–30fps, making extremely large-motion interpolation less common. There is also no analysis of inference speed in autopilot mode.\n3. **Robustness and Scope**: The method’s effectiveness under complex trajectories remains uncertain, as the learned keypoint trajectory distribution is unlikely to generalize to all cases. Additionally, autopilot mode’s reliance on feature-matching algorithms may limit performance under conditions like blur or low resolution. Further experiments are needed to clarify performance under these situations.\n4. **Benchmarking and Quantitative Analysis**: The paper lacks evaluations on commonly used VFI datasets, such as Vimeo90K and X4K1000FPS, which would offer valuable comparison. In addition, considering that the proposed method achieves superior performance only in terms of FVD, the inclusion of non-reference metrics such as NIQE will help to further understand the image quality.\n5. **Technical Innovation**: While the method yields high-quality visuals, the technical novelty appears limited. Techniques such as conditioning diffusion models with start and end frames and using keypoint trajectories (DragGAN) already exist. Emphasizing unique contributions would strengthen the paper’s impact.\n\n[1] \"Exploring motion ambiguity and alignment for high-quality video frame interpolation.\" CVPR2023  \n[2] \"Clearer Frames, Anytime: Resolving Velocity Ambiguity in Video Frame Interpolation.\" ECCV2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjbA2uwIdI", "reviewer": "Reviewer_hifJ", "summary": "The paper proposes Framer, an interactive frame interpolation model designed to generate smooth frame transitions between two images. The approach allows users to customize the interpolation process by adjusting the motion trajectories of keypoints. This interactive feature provides precise control over local motions, improving the model’s ability to handle challenging scenarios. The model also includes an autopilot mode that automatically estimates keypoints and refines trajectories, simplifying the interpolation process for users.", "questions": "Overall, I think this is a solid piece of work. Aside from the minor points mentioned in the weaknesses section, I don’t have significant concerns. The approach is interesting and well-presented, with promising results. I would lean toward recommending acceptance of this paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The motivation of the paper is clear and well-founded. The authors address the inherent variability in intermediate frame generation and propose an effective solution by integrating keypoint control, which allows for a more tailored interpolation process that better aligns with user preferences.\n2. The paper is well-written and easy to follow. The use of figures and tables is excellent, with ample illustrative examples demonstrating the model’s application.\n3. The results presented are noticeably better than existing methods, particularly in challenging scenarios. Additionally, the inclusion of an automated mode ensures fairness in comparison, which further strengthens the validity of the results.", "weaknesses": "1. It would be helpful to see examples of failure cases, if any, to better understand the model’s limitations. Displaying such cases could provide insight into potential improvements or areas where the method might struggle in real-world applications.\n2. The title might be slightly misleading. The approach seems more aligned with image morphing rather than traditional frame interpolation. A title that reflects this focus on image-to-image transitions could better represent the paper’s core contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)