Framer: Interactive Frame Interpolation

Authors: Wen Wang, Qiuyu Wang, Kecheng Zheng, Hao OUYANG, Zhekai Chen, Biao Gong, Hao Chen, Yujun Shen, Chunhua Shen
Primary area: generative models
Keywords: Video Frame Interpolation; Interactive; Diffusion Model; Correspondence Modeling

Abstract:
We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity. Concretely, besides taking the start and end frames as inputs, our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. Such a design enjoys two clear benefits. First, incorporating human interaction mitigates the issue arising from numerous possibilities of transforming one image to another, and in turn enables finer control of local motions. Second, as the most basic form of interaction, keypoints help establish the correspondence across frames, enhancing the model to handle challenging cases (e.g., objects on the start and end frames are of different shapes and styles). It is noteworthy that our system also offers an "autopilot" mode, where we introduce a module to estimate the keypoints and refine the trajectory automatically, to simplify the usage in practice. Extensive experimental results demonstrate the appealing performance of Framer on various applications, such as image morphing, time-lapse video generation, cartoon interpolation, etc. The code, model, and interface are publicly accessible at https://github.com/aim-uofa/Framer.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729260460309

thread_type:
official_review

<thread_object>
creator: reviewer_hifj

<content>
summary:
The paper proposes Framer, an interactive frame interpolation model designed to generate smooth frame transitions between two images. The approach allows users to customize the interpolation process by adjusting the motion trajectories of keypoints. This interactive feature provides precise control over local motions, improving the model’s ability to handle challenging scenarios. The model also includes an autopilot mode that automatically estimates keypoints and refines trajectories, simplifying the interpolation process for users.

strengths:
1. The motivation of the paper is clear and well-founded. The authors address the inherent variability in intermediate frame generation and propose an effective solution by integrating keypoint control, which allows for a more tailored interpolation process that better aligns with user preferences.
2. The paper is well-written and easy to follow. The use of figures and tables is excellent, with ample illustrative examples demonstrating the model’s application.
3. The results presented are noticeably better than existing methods, particularly in challenging scenarios. Additionally, the inclusion of an automated mode ensures fairness in comparison, which further strengthens the validity of the results.

weaknesses:
1. It would be helpful to see examples of failure cases, if any, to better understand the model’s limitations. Displaying such cases could provide insight into potential improvements or areas where the method might struggle in real-world applications.
2. The title might be slightly misleading. The approach seems more aligned with image morphing rather than traditional frame interpolation. A title that reflects this focus on image-to-image transitions could better represent the paper’s core contribution.

questions:
Overall, I think this is a solid piece of work. Aside from the minor points mentioned in the weaknesses section, I don’t have significant concerns. The approach is interesting and well-presented, with promising results. I would lean toward recommending acceptance of this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729850162208

thread_type:
official_review

<thread_object>
creator: reviewer_4487

<content>
summary:
This paper presents a diffusion-based architecture, *Framer*, for interactive frame interpolation, introducing last-frame conditioning and keypoint trajectories to achieve controllable transitions. While the qualitative results are impressive, the practical application, related work discussion, and unique technical contributions need further improvement.

strengths:
The method’s introduction of keypoint trajectories addresses motion ambiguity, leveraging the stable video diffusion model to improve interpolation fidelity.   
The qualitative outcomes are notably strong.

weaknesses:
1. **Related Work**: The paper does not sufficiently discuss previous work tackling motion ambiguity and interactive frame interpolation, such as [1] and [2], which address motion/velocity ambiguities. And [2] is the early work in interactive VFI. A more comprehensive discussion would clarify the novelty and contributions of *Framer*.
2. **Real-World Application**: The practical applicability of this method is unclear, especially given that real-world videos typically run at frame rates like 24–30fps, making extremely large-motion interpolation less common. There is also no analysis of inference speed in autopilot mode.
3. **Robustness and Scope**: The method’s effectiveness under complex trajectories remains uncertain, as the learned keypoint trajectory distribution is unlikely to generalize to all cases. Additionally, autopilot mode’s reliance on feature-matching algorithms may limit performance under conditions like blur or low resolution. Further experiments are needed to clarify performance under these situations.
4. **Benchmarking and Quantitative Analysis**: The paper lacks evaluations on commonly used VFI datasets, such as Vimeo90K and X4K1000FPS, which would offer valuable comparison. In addition, considering that the proposed method achieves superior performance only in terms of FVD, the inclusion of non-reference metrics such as NIQE will help to further understand the image quality.
5. **Technical Innovation**: While the method yields high-quality visuals, the technical novelty appears limited. Techniques such as conditioning diffusion models with start and end frames and using keypoint trajectories (DragGAN) already exist. Emphasizing unique contributions would strengthen the paper’s impact.

[1] "Exploring motion ambiguity and alignment for high-quality video frame interpolation." CVPR2023  
[2] "Clearer Frames, Anytime: Resolving Velocity Ambiguity in Video Frame Interpolation." ECCV2024.

questions:
Please address the issues raised in the weaknesses section. This reviewer is willing to increase the rating if the issues are well addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730529326256

thread_type:
official_review

<thread_object>
creator: reviewer_sbqc

<content>
summary:
This paper presents an interactive frame interpolation framework to produce smoothly transitioning frames between two images, interactively or automatically. By harnessing user input point controls from the start and end frames, this method can effectively generate the video frames for different applications including drag control, novel view synthesis, cartoon and sketch interpolation, time-lapsing video generation, slow-motion video generation, and image morphing.

strengths:
1）	The method is straight forward but effective, which can be used for many applications;
2）	Most results shown in this paper are impressive.
3）	Ablation experiments are sufficient to evaluate the method.

weaknesses:
1)	The component of the Framer includes Trajectory preprocessing, Trajectory control and Interpolation. Although the method is effective, the novelty is limited. Trajectory control is designed by following DragNUWA, and the Interpolation is mainly based on SVD that determines the performance of this method.
2)	The examples shown in this paper are a bit simple. The motion and the difference between two key frames (first and last frames) are not complex. More complex scenarios, as cases shown in the paper “Training Weakly Supervised Video Frame Interpolation with Events”, should be considered to demonstrate the superiority of this method.
3)	The image quality of some results shown in this paper degrades compared to the first input frame. For example, the IQ of frames in “laugh.mp4” is worse than that of the start and end frames;
4)	The reconstruction metric of this method is not best when comparing it to related methods.

questions:
1)	Can the frame interpolation function support arbitrary number of frames generation between the start and end inputs?
2)	how to process the case when the number of corresponding points between frames is too few?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716858613

thread_type:
official_review

<thread_object>
creator: reviewer_urco

<content>
summary:
The paper proposes a new technique for generating frame interpolations that can incorporate user input to drive the interpolation process between two given frames. The method uses Stable Video Diffusion as the base model with the inputs modified to include both the first and last frame. Additionally, a control branch consisting of gaussian heatmaps is encoded and included in the model. The method is evaluated on a variety of image categories and on publicly available datasets.

strengths:
The paper is mostly well written and easy to follow. The novelty here lies in the control branch for guiding the interpolation process that allows for fine-grained user-friendly control over the interpolation process.. The authors evaluate several existing methods on a variety of input types like real-world, sketch, cartoon, etc both qualitatively and quantitatively and produce high quality results.  Also included in a technique to produce the control signals automatically using bidirectional consistency checks. Several key design choices have been evaluated thoroughly, such as number of trajectories, bidirectional consistency.

weaknesses:
- There is no mention of inference speed, training speed or overall training methodology (end-to-end, multi stage, pretraining, etc)
- There appears to be no underlying camera model for handling the "novel view synthesis" examples. Having such a model could potentially improve quality while providing another user-control.
- The paper isn't very easy to follow specifically around the trajectory update for autopilot mode section.

questions:
- What are the differences in training / inference methodology between the autopilot and user-guided mode?
- Are there categories of videos where the results aren't as high quality as other categories?
- Could you explain a bit more about "considering that the conditions and the corresponding noisy latents are spatially aligned" near line 183?
- Can you explain how you transform 2D points into gaussian heatmaps? Are any other modalities like text used here? References to DragNUWA or DragAnything might be too recent. Consider also citing an older source such as from 2014 "Stacked Hourglass Networks for Human Pose Estimation" if it makes sense.
- Can the autopilot mode re-use the randomly sampled points ~line 197 instead of using feature matching to find new correspondences over time?
- Have you considered simpler alternatives for the Trajectory updating section?
- How do you handle camera motion in your framework?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732558637797

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Responses to Reviewers

comment:
We thank all reviewers for their valuable feedback, and are encouraged that the reviewers appreciate our clear motivation (**hifJ**), novelty (**urco**), detailed evaluation (**urco**, **SBQc**, **hifJ**), and impressive performance (**urco**, **SBQc**, **4487**, **hifJ**). Here, we provide a high-level summary of the changes we've made to address your concerns, and conclude with an overview of our key contributions.

---
Here is the summary of updates that we've made to address the reviewers' concerns:
- We discuss failure cases under the following scenarios, and discuss potential improvements. (**urco**, **hifJ**)
  - Lack of suitable matching points between the first and last frames.
  - Failure to capture object semantics.
- We clarify the novelty and contribution of this paper. (**SBQc**, **4487**)
- We evaluate and discuss the inference speed of Framer. (**urco**, **4487**)
- We conduct experiments to demonstrate the robustness and generalization capability of Framer, including
  - Generating varying numbers of frames between the start and end inputs. (**SBQc**)
  - Handling large motion and differences between input frames. (**SBQc**)
  - Effectiveness under complex trajectories, measured by video quality and trajectory accuracy. (**4487**)
  - Effectiveness under low-resolution inputs. (**4487**)
  - Benchmarking results on Vimeo90K, X4K1000FPS, and Adobe240, and evaluation with NIQE metric. (**4487**)
  - Benchmarking results on the task of cartoon interpolation and sketch interpolation. (**4487**)
- We improve paper writing from the following aspects.
  - We discuss previous work on tackling motion ambiguity and interactive frame interpolation. (**4487**)
  - We discuss the applications of video frame interpolation methods. (**4487**)
  - We add an Algorithm to better illustrate the point trajectory estimation method. (**urco**)
  - We add more details on transforming 2D points into Gaussian heatmaps. (**urco**)
  - We discuss camera motion control in Framer. (**urco**)

---
The contribution of our work is summarized as follows.
- Framer introduces **a novel way of interaction in video frame interpolation to align with user intentions**. Users can unleash their creativity through simple and intuitive click-and-drag operations on input images. 
- In Framer, we utilize **explicit long-term correspondences between start and end frames** to enhance video frame interpolation (VFI). Previous methods struggle with input frames that exhibit significant differences. In contrast, our approach leverages **long-term temporal guidance derived from user drag inputs or estimated point trajectories**, resulting in improved quality and temporal consistency in the output videos.
- we present **a novel point trajectory estimation method that integrates feature matching with a new bi-directional point tracking technique** to achieve accurate point trajectory guidance. Although a uni-directional point tracking method has been proposed in DragGAN, directly applying it to video frame interpolation leads to sub-optimal results, as demonstrated in Section 4.4 and Table S1. To tackle this, our motivation is that matched keypoints in both start and end frames offer distinctive features for effective point tracking. Based on this observation, we implement both forward and backward point tracking and introduce a bi-directional consistency check process to enhance tracking accuracy.
- **Framer supports novel downstream applications, such as image morphing and cartoon or sketch interpolation**, which require user customization and the ability to manage large differences between input frames.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces Framer, an method for using user-provided trajectories as guidance in frame interpolation or morphing. Reviewers praised the paper's clear motivation and impressive performance, but some questioned its practical applicability and technical innovation.  Reviews were unanimously positive, and thus the paper has been chosen for acceptance.