Realtime Video Frame Interpolation using One-Step Diffusion Sampling
Authors: Yongrui Ma
Shijie Zhao
Mingde Yao
Junlin Li
Li zhang
Xiaohong Liu
Qi Dou
Jinwei Gu
Tianfan Xue
Venue: ICLR 2026 Poster (2026)
Keywords: Video Frame Interpolation; Diffusion Models; Realtime Processing
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Video Frame Interpolation (VFI) involving large, complex motions remains a significant challenge due to the difficulty of modeling diverse pixel trajectories from limited inputs. Traditional methods struggle with low-order approximations, and recent Latent Video Diffusion Models (LVDM) improve it through a conditional generation modeling. Still, current LVDMs often prioritize pixel fidelity over motion coherence in their reconstruction objective, leading to artifacts in extreme motion scenarios. To address this, we propose RDVFI, a novel approach that leverages an LVDM to generate sparse latent keyframes which define high-order, continuous pixel trajectories. The estimated continuous pixel trajectories accurately index pixel movements from inputs to arbitrary timestamps, generating optical flows to warp input pixels into the target frame. By decoupling sequence motion generation from high-resolution rendering, RDVFI operates on a fixed, lower resolution, and fewer diffusion sampling steps, introducing significant efficiency gains. Extensive experiments demonstrate that RDVFI achieves state-of-the-art visual and numerical performance, with over 75\% of viewers selecting it as the best method in terms of motion and frame quality compared to leading baselines. Furthermore, RDVFI is the first LVDM-based VFI method to achieve real-time performance (17 FPS at $1024\times 576$), offering a $\times 44$ acceleration over the current state-of-the-art and also robustly handling challenging motions.

Published as a conference paper at ICLR 2026
REALTIME VIDEO FRAME INTERPOLATION USING ONE-STE DIFFUSION SAMPLING
Yongrui Ma Shijie Zhao', Mingde Yao Junlin Li Li Zhang', Xiaohong Liu?, Qi Dou', Jinwei Gu Tianfan 2 DayteDance Inc. CUHK MMLab
Janlin
CUHK CCIII under InnoHK
STTU
Input Frames
Runtime (sec//rame)
FVD
84.37
400x fasler
Direct Generation
Linear Motion
Continuous Pixel Trajectories
Wan
FCVG
RDVELUU
RDVELD
Video frame interpolation results the proposed Unet-based RDVFI-U and DiT-based RDVFI-D with comparisons state-of-the-ans methods. RDVFI produces the most visually pleas- ant and the best numeric results with x00x acceleration, owing to more proper modeling of the intermediate motions using the high-order continuous pixel trajectories, compared with previous di- rect frame-generation methods, and linear motion controls.
Figure I:
ABSTRACT
Video Frame Interpolation (VFI) involving large, complex motions remains sig- nificant challenge due to the difficulty of modeling diverse pixel trajectories from limited inputs. Traditional methods struggle with low-order approximations, and recent Latent Video Diffusion Models (LVDM) improve it through conditional generation modeling. Still, current LVDMs often prioritize pixel fidelity over mo- tion coherence in their reconstruction objective, leading artifacts in extreme motion scenarios. To address this, we propose RDVFI, a novel approach that leverages an LVDM to generate sparse latent keyframes which define high-order, continuous pixel trajectories. The estimated continuous pixel trajectories accu- rately index pixel movements from inputs to arbitrary timestamps, generating op- tical flows to warp input pixels into the target frame. By decoupling sequence mo- tion generation from high-resolution rendering, RDVFI operates on a fixed, lower resolution, and fewer diffusion sampling steps, introducing significant efficiency gains. Extensive experiments demonstrate that RDVFI achieves state-of-the-arr visual and numerical performance, with over 75% of viewers selecting it the best method in terms of motion and frame quality compared to leading basclines. Furthermore, RDVFI is the first LVDM-based VFI method to achieve real-time performance (17 FPS at 1024 576), offering x44 acceleration over the current state-of-the-ard and also robustly handling challenging motions. Please visit the project page for code and more visualization samples.
INTRODUCTION
Video Frame Interpolation (VFI) aims to predict intermediate frames between two input ending frames. It serves foundational component in diverse fields, including frame-rate photogra-
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Non-parametric Estimation
Estimation by Low-arder Functions
Our Continuous Pixel Trajectories
Quadratic function
Latent keyframes
Linear fanction
T-05 0.5
Figure 2: Comparisons between motion estimation methods for VFI. Previous non- non-parmettii meth- ods (Huang et al., 2022; Reda et al., 2022; Li et al., 2023; Danier et al., 2024; Lew et al., 2025) and low-order approximations (linear (Jiang al., 2018; Bao et al., 2ua9/aarrrai (Xu al., 2019; Liu et al., 2020)) struggle to capture large and nonlinear motions like the above rotating windmill. The proposed RDVFI solves continuous pixel trajectories via high-order functions determined by sparse latent keyframes ([k,) from the LVDM. Therefore, RDVFI can interpolate to more diverse and complex real-world motions that previous methods fail to capture.
phy (Xiang et al., 2020; Gao et al., 2022; Ma al., 2024), einematic post- post-production (Kokaram et al,, 2020; Briedis et al., 2021), and scientific visualization (Priessner et al., 2024; Joshi et al., 2025). Cur- rent methods often interpolate by estimating optical flows between intermediate and input frames to warp input pixels to each interpolated frame. As illustrated in Fig. 2, standard con-aaraettric meth- ods (Huang et al., 2022; Reda et al., 2022; Li al., 2023; Guo et al., 2024; Danier et al., 2024; Lew et al., 2025) predict flows between one middle and input frames directly in data-driven manner, optionally assisted by an image diffusion model. While straightforward, these approaches struggle to resolve nonlinear, non-rigid motions. Some methods approximate pixel trajectories using rigid motion priors, such as linear (Jiang et al., 2018; Bao et al., 2019) or quadratic (Xu et al., 2019; Liu et al., 2020) functions. However, these low-order approximations lack the ability model the complex and ambiguous real-world dynamics.
Recently, Latent Video Diffusion Models (LVDM) have revolutionized generative tasks by their ability to produce high-fidelity video content (Blattmann et al., 2023; Guo et al., 2023; Wang et al., 2025). Inspired by this, state-of-the-art approaches also (Zhu al., 2024; Wang et al., 2024; 2025) reformulate VFI as conditional generation problem using LVDMs. Still, despite their ability to synthesize plausible content, these models face two critical limitations. First, they rely on standard pixel-reconstruction objectives that prioritize local texture approximation over global motion coher- ence. This often leads to severe structural artifacts, such as ghosting and object deformation, as seen in Fig. 1. Second, the iterative diffusion sampling required for high-frequency detail imposes heavy computational burden. Although current distillation techniques can accelerate these methods by reducing the number of sampling steps, they may aggravate the structural artifacts in interpolated frames, especially when dealing with nonlinear and non non-rigid motions.
In this paper, we propose RDVFI, novel VFI framework that formulates large and complex motions as high-order functions, with coefficients determined by an LVDM. Previous methods model pixel movements based solely input frames, limiting them to linear (Jiang al., 2018; Bao et al., 2019), quadratic (Xu et al., 2019; Liu et al., 2020), or nonparametric (Huang et al., 2022; Sim et al., 2021; Li et al., 2023; Danier et al., 2024; Lew et al., 2025) formulations, as shown in the first two examples in Fig. 2. Instead, by incorporating an LVDM to generate additional latent keyframes, RDVFI enables high-order pixel trajectories, which capture real-world dynamics more accurately than previous approaches (see Fig. 2), as shown in the third example of Fig. 2. RDVFI then samples the estimated continuous pixel trajectories for optical flows warp input pixels for interpolation. Because our LVDM primarily focuses generating accurate intermediate motions that lack high- frequency details, the proposed RDVFI can interpolate to complex motions with superior temporal stability via one-step diffusion sampling, introducing significant efficiency gains in Fig. 1.
RDVFI offers three primary advantages. First, by shifting the LVDM's objective from frame syn- thesis to motion modeling, we achieve superior temporal coherence and accuracy, shown in Fig. 1. Second, because the predicted trajectories represent underlying structural motion rather than fine-grat fine-grained texture, one-step sampling is sufficient capture the necessary dynamics, sig- nificantly reducing inference latency. Third, as motion can be computed from sparse and low- resolution keyframes, RDVFI further accelerates the inference by offloading major computation on
2

---

Published as a conference paper at ICLR 2026
low-resolution and sparse input, and uses relatively lightweight module for -rame and full- resolution computation. This design makes RDVFI extremely efficient for high-resolution input.
Extensive evaluations demonstrate that RDVFI successfully interpolates large-scale motions with over 50x acceleration of the leading baseline, Wan (Wang ct al., 2025). On a single A100 GPU, RDVFI processes 1024 x 576 resolution at 17 FPS. Our core contributions are:
We present RDVFI, a VFI framework that solves large, complex motions using high-order continuous pixel trajectories and coefficients determined by a one-step LVDM. To our knowledge, RDVFI is the first work to solve motion in VFI using high-order functions
We Mennsec LVDM.
implicit motion regularization term that enforces physical and temporal consistency, allowing for accurate internolation of roualtty non-Jinear trajectories We demonstratte sigmiecnt gains
perceptual quality and computational efficiency, effectively bridging the gap between diffusion- generative power and practical de- ployment requirements.
2 RELATED WORK
2.1 VIDEO FRAME INTERPOLATION
Non-generative Methods Conventional VFI methods (Jiang et al., 2018; Bao et al., 2019; Xu et al., 2019; Liu et al., 2020; Huang et al., 2022; Reda et al., 2022; Li et al., 2023) primarily rely on regression-
objectives. These are categorized by their underlying motion priors. Early approaches assume pixels follow rigid, predefined trajectories, such as linear (Jiang et al., 2018; Bao et al., 2019) or quadratic (Xu et al., 2019; Liu et al., 2020) models. However, these parametric assumptions fail to capture the stochastic and non-linear nature of real-world dynamics. Subsequent data-driven strategies (Huang et al., 2022; Reda et al., 2022; Li et al., 2023; Guo et al., 2024) predict optical flows between specific timestamps and input frames. While effective reducing warping artifacts, these methods remain brittle when confronted with the large and complex motions.
Generative-based Methods Diffusion Models (DMs) (Ho et al., 2020) have demonstrated supe- rior performance in solving ill- ill-posed inverse problems for image (Dhariwal & Nichol, 2021; Ramesh et al., 2022; Saharia et al., 2022) and video synthesis (Hong et al., 2023; Blattmann et al., 2023; Yang al, 2024; Lin et al., 2024; Wang et al., 2025). Recent studies (Danier et al., 2024; Lew et al., 2025; Zhu et al., 2024; Wang et al., 2024; 2025) leverage DMs to improve interpolation fidelity. For in- stance, LDMVFI (Danier et al., 2024) and MoMo (Lew et al., 2025) use image-based based DMs to refine flow estimation. However, they remain optimized for relatively simple motion. More recent state- of-the-art frameworks like GI (Wang et al., 2024), FCVG (Zhu et al., 2024), and Wan (Wang al., 2025) treat VFI as a conditional generation task using Latent Video Diffusion Models (LVDMss. De- spite the progress, these methods suffer from a "pixel-fidelity bias": their training objectives focus on per-pixe) reconstruction rather than explicit motion consistency and accuracy. Consequently, they often produce frames with rich frame details but poor temporal coherence and unnatural motions.
2.2 ENHANCING MOTION IN LVDMS
Several attempts (Zhu et al., 2024; Chefer et al., 2025; Shi et al., 2024) have been made to improve motion modeling within LVDMs. FCVG (Zhu et al., 2024) utilizes linearly interpolated sparse matching to guide motions, yet it remains limited by the reliability of the matching algorithm and the simplicity of its motion model. VideoJAM (Chefer et al., 2025) forces LVDMs to jointly pre- dict frames and flow visualizations; however, this multitask approach often leads to frame degrada- tion due to a distribution mismatch between natural imagery and flow visualizations. Alternatively, Motion-12y (Shi et al., 2024) decouples flow from frame generation using separate LVDMs. How- ever, it still fails on complex motions because flow estimation degrades with increasing temporal distance, making it impossible
generate accurate ground truth for training. Unlike these methods, RDVFI introduces a robust motion representation by predicting coefficients for high-order continu- ous pixel trajectories, thereby enabling a more expressive, physically consistent motion space.
2.3 DIFFUSION ACCELERATION
Accelerating diffusion inference is critical for practical VFI. Current techniques include rectified flow (Liu et al., 2022; 2023), adversarial training (Lin et al., 2025; Zhang al., 2024), and score
3

---

Published as a conference paper at ICLR 2026
Phase Continuous Pixel Trajectories Estimationn Lon-rciolation Input
Plase 2: High-resolution Frame Synthesis
repu fores
Dowssarple insuh
Optiaal Bloa for intep..
Inaepet furres
Frame synthess (Algorithm
Latent
Encoder
Dilluskon
Figure 3: Overall framework of the proposed RDVFI. Our RDVFI interpolates intermediate frames Jufi in two phases: continuous pixel trajectories estimation and frame synthesis. Phase 1 determines the coefficients of high-order continuous pixel trajectories, which index pixel offsets at arbitrary intermediate timestamps, using an LVDM. Phase samples the estimated trajectories for optical flows that warp input pixels to each interpolation timestamp. RDVFI then fuses the warped frames for interpolated frames with fine- une-grained details using the frame synthesis module.
distillation (Wang al., 2023; Yin al., 2024), guide lightweight, fewer-step student network to behave like a robust, multi-step teacher model. While one-step generation has been achieved in text-to-image (Liu al., 2023; Yin al., 2024) and video synthesis (Lin al., 2025; Zhang et al., 2024), these distillation-base methods are difficult to adapt to VFI for two reasons. First, existing conditional gentreegnnrttionn LVDMs may suffer from fewer diffusion sampling steps, ag- gravating their artifacts for large and complex motions. Second, the auxiliary networks required for distillation introduce significant training overhead. To this end, RDVFI does not involve distillation techniques. We demonstrate that by re-tasking the LVDM to predict motion trajectories rather than raw pixels, one-step sampling is inherently sufficient to resolve motion ambiguity while maintaining high-fidelityy results through warping.
METHOD
We introduce RDVFI, framework designed bridge the gap between generative fidelity and mo- tion coherence in VFI. We first outline the overall framework Section 3.1. This is followed by our continuous pixel trajectory modeling in Section 3.2, the one-step LVDM in Section 3.3, the synthesis network in Section 3.4, and finally our two-stage training strategy in Section 3.5.
3.1 OVERALL FRAMEWORK
Given two endpoint frames fo, I1, RDVFI synthesizes a sequence of intermediate frames (ir, 3 arbitrary timestamps Ty (0, 1). The pipeline operates in distinct phases: continuous pixel trajectory estimation at a fixed, relatively lower resolution, such as 448 x 256, and high-resolution frame synthesis (e.g., at 1024 X 576), as shown in Fig. 3.
In the first phase, a Latent Video Diffusion Model (LVDM) generates set of sparse latent keyframes (2x. 1 via one-step sampling. These keyframes serve as temporal anchors to determine the coeffi cients of continuous pixel trajectories. Because the LVDM operates at reduced olution and models motion instead of texture, achieves significant computational efficiency. the second phase, RDVFI samples the estimated trajectories at each interpolation timestamp {Ti) y=1- and we derive dense optical flows for each intermediate frame. These generated optical flows are used to warp the input frames, following frame synthesis module to fuse and produce the final high-resolution output Left-
tettioa-tmmraall res-
3.2 CONTINUOUS PIXEI TRAJECTORIES
We define continuous pixel trajectories as functions foo-r: that map pixel offsets from Io or I any (0, 1). To ensure accuracy across large displacements, we decompose complex mo- tions between input frames into small and easy-to-estimate components and solve them iteratively, shown in Fig. 4. To this end, first estimate motions across consecutive frames by:
Skitkket 22 (āt,, EkgT1).
4

---

Published as a conference paper at ICLR 2026
Lakat Sayfirno
Mation Acreas Consecative frore
Modion between laydone tepute
Contensora Inaeettoiio
Optial
Figure 4: The proposed continuous pixel trajectories estimator. The proposed RDVFI decomposes the large and complex motions between intermediate and input frames into small and simple motions across consecutive frames guided by latent keyframes generated by the LVDM. RDVFI then recur- sively fuses these motion components to compute optical flows between sparse latent keyframes and input frames. RDVFI fits high-order continuous pixel trajectory functions using estimated optical flows, which can flexibly sample optical flows to warp input pixels to any intermediate time.
The estimated motion components are fused recursively to compute optical flows between each keyframe and input frames, following:
fo- = JRa 20, 2k(41)
(2)
In this formulation, 6 and Φ2 are convolutional neural networks in the supplementary Section B.2. We estimate the backward continuous pixel trajectories using the same network and weights, but in reversed direction from 7 = 1 to O. In addition to latent keyframes (k, J.V. we also define ko + to be encoded latent input frames and calculate the continuous pixel trajectories from i = and = N + 1 to N and i 1, respectively. This iterative refinement only updates small and casy-to-estimate pixel offsets to previous flow results, allowing RDVFI to resolve large-scale mo- tions more effectively and efficiently than direct-estimation baselines. Finally, we fit these keyframe optical flows using cubic convolution (Keys, 2003) to provide optical flows for all interpolation.
3.3 ONE-STEP LATENT VIDEO DIFFUSION MODEL
The continuous pixel trajectories introduced above computed from keyframes, and we use the Latent Video Diffusion Model (LVDM) to generate those keyframes.
To illustrate that, first revisit the preliminaries of LVDM. LVDMs generate video frames from noise based on input conditions in a low-resolution latent space, following a VAE decoder ((-) to synthesize high-resolution video frames. In VFI, the LVDM first encodes the two input frames, To and I, into the latent frames zo and 21. These latent frames are then used as generation conditions. To learn the generation process, the LVDM degrades ground truth latent frames z with noise e (0, I), generating noisy latent frames by z a'z + o'e'. Here, a' and o² are parameters determining how much noise is added. The diffusion timestep E (0, T), with T, corresponds to pure noise. The LVDM uses denoising network Folz"; 20, 21) to estimate the added noise. The training objective is:
(t) = Fo 20, 21) v'1l.
(3)
where n' = a'c'' a'z is called the velocity of diffusion. After convergence, the LVDM recursively noise to generate noise-free intermediate latent frames é. Because these
denoises vandomly-samplee LVDMs directly generate intermediate frames, they require tedious diffusion sampling capture the fine-grained details of each frame, resulting in a high computation cost that is difficult to accelerate.
Unlike previous methods Zhu al. (2024); Wang et al. (2024; 2025) that utilize the multi-step LVDM to generate all interpolated frames, our RDVFI only uses one-step LVDM to generate a few keyframes in a continuous pixel trajectory, resulting in significant acceleration. Our design is based on the fact that input frames in VFI have already provided the most desired intermediate appear- ances, and the core task is determining how extract pixels from these input frames to generate in- termediate frames. Because our LVDM only determines motions, it does not require dense diffusion sampling for fine-grained frame details, where one-step diffusion sampling can generate reasonable results. Furthermore, because the number of keyframes N is often much smaller than the number of interpolated frames L, and generated motions can be flexibly resized to different resolutions. our LVDM can sample at fixed and relatively smaller spatial- petiial-temporal frame resolution. This yields further acceleration and facilitates diffusion training. Please find the supplementary Section B. for the network details.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: RDVEI Frame synthesis.
Require: Input frames To and I1, interpolation time {Ty f= estimated continuous pixel trajectories :
Ensure: Interpolated frames (4,}
1: for j=1,
do
2: Sample optical flows (fo- fi 1-, from {fo- [1+T]
3: Warp imput frames To-+r, = W arpp To: fu-rr, 11-,, = W (arp(I1,fi-a
4: Find the nearest keyframe timestep, satisfying k; < Tj kit11, ki kit1 =
5: Sample optical flows [ST--rz STE11-7, from (Jo-r, p1-73
6: Warp latent keyframes =k, = W Warr((2k: fk. fki-t,, 2k41 W Warp(zk--11 fkmI)1)))
7: 1 = 00o 41, lo- 11 11-,,, fo-r, f1 - 2k141-T7,)
8: end for
3.4 FRAME SYNTHESIS
Given the estimated pixel trajectory introduced above, RDVFI finally generates all high-resolution dense intermediate frames. As shown in Algorithm 1, RDVFI first samples the estimated continu- ous pixel trajectories (Ja-7, fi-r) ) different interpolation times 7 {r) to generate optical flows /1-+, D RDVFI then warps input frames lo, /1 with sampled optical flows, pro- ducing bi-directional warped frames (To-ry: I ]--,;]- J- Unlike previous -on-generaeive methods that directly fuse the warped frames, further incorporate latent keyframes generated through one- step LVDM sampling to provide appearance changes that cannot be modeled by optical flows, such dynamic textures, lightning changes, and occlusions. For this target, between the latent keyframes 2k,, 2k,4, that satisfy k; S Tj S ki+1- We further warp these latent frames to time Ty. producing warped latent frames { 2k,41 RD- VFI then fuses warped frames {To /I- } and latent keyframes with neural network (-) to produce the interpolated frames. Please find the supplementary Section B.3 for network details.
sample optical flows
3.5 TRAINING PROCESS
As shown in Fig. 8, we train our RDVFI in two stages: motion-guided latent-to-frame generation and one-step diffusion training.
Motion-guided Decoding RDVFI trains the continuous pixel trajectories estimator to predict ac- curate optical flows for interpolation using given latent keyframes. Due to the lack of accurate ground truth, RDVFI learns this process an unsupervised manner with the loss on interpolated frames. Existing research (Xue et al., 2019) has demonstrated that these task-oriented optical flows can enhance downstream tasks and have been widely used in recent VFI methods (Jiang et al., 2018; Bao et al., 2019; Xu et al., 2019; Liu et al., 2020; Huang et al., 2022; Reda et al., 2022; Li et al., 2023; Danier et al., 2024). Specifically, RDVFI first selects N keyframes [4(t) by frame skipping and then downsamples selected keyframes to relatively low resolution I((x.), RDVFI uses motion-guided decoding network to synthesize high-resolution frames free from encoded ground truth latent keyframes z. The training objective is formulated as follows:
Crea I) = wi Tipipp(T. Trec) - T Freel12.
(4)
In this formulation, W1 and We denote the loss weights, which are set 0.2 and 1, respectively.
One-step Diffusion Training In this stage, RDVFI trains the LVDM generate sparse latent keyframes for interpolation. To achieve this, the continuous pixel trajectories estimator and synthesis network are frozen, and only the parameters in the LVDM are updated. Rather than requiring the LVDM to generate noise-free latent keyframes directly from pure noise, we train the LVDM with randomly sampled E (O,T') for higher stability. We linearly increase the diffusion timestep by adjusting the lower bound for random sampling according to the number of training steps. In addition to the original pixel reconstruction loss in the latent space (Equation 3), the reconstruction loss on the interpolated frames (Equation 4) is also employed, resulting in the following:
LofilT, fofs) X1 Lo) + Xe T).
(5)

---

Published as a conference paper at ICLR 2026
Our Figure 5: Visual comparison results DAVIS-7 (Jain et al., 2024) (the top row), FCVG (Zhu et al., 2024) (the middle row), and our self-collected evaluation dataset (the bottom row). Our RDVFI-D method consistently outperforms existing diffusion-based VFI methods with less blurring, ghosting effects, color shifts, and fractions.
Imput D
Ingitt
LDMVFI
MoMe
GI
FCVG
Wan
In this context, A and A2 are coefficients that balance each loss term, set to 1.0 and 0.5, respectively. Since our nottion-guided decoding process involves continuous pixel trajectories estimation, the reconstruction loss term can serve as an implicit motion regularization term in the diffusion training and is essential for generating high-quality frames.
4 EXPERIMENTAL RESULTS
4.1 IMPLEMENTATION DETAILS
To verify that our RDVFI can adopt different pretrained LVDMs, we trained two versions: the Unet-based RDVFI-U, which builds upon 1.5B SVD (Blattmann et al., 2023), and the DiT-based (Wang et al., 2025). Both them use seven keyframes. During the training, we randomly resize images to one of four resolutions: (448 256, 576 x 320, 1024 x 576, 1280 x 720). The LVDM and the continuous pixel trajectories estimator operate the 448 x 256 resolution by resizing input images. The motion-guided decoding module then generates intermediate frames at the input resolutions with up-scaled optical flows. We use fixed learning rate of 2c-5 for all experiments. We adopt eight A A100-80G GPUs and total batch size of 8. We train for 400K iterations in the first stage and 800K in the second. Please find the supplementary Section C for more details.
RDVFI-D based on Wan2.1-Fun-1.3B-N
4.2 BENCHMARKING
Selected Methods and Setting We compare the proposed RDVFI with several state-of-the- art methods, including conventional non-generative ones, such as RIFE (Huang et al., 2022), FILM (Reda et al., 2022), and AMT (Li et al., 2023), as well as the diffusion-based ones, includ- ing LDMVFI (Danier et al., 2024), MoMo (Lew et al., 2025), GI (Wang al., 2024), FCVG (Zhu et al., 2024). and Wan 1.3B InP (Wang et al., 2025). Because methods may require different training strategies for optimal performance, authors have already best optimized them. We evaluate Wan (Wang et al., 2025) with blank text in- put. We do not compare with any distillation-basee VFI method because current methods still focus on solving large and complex motions, and we have not found a baseline method that specifically involves VFI distillation.
utilize the released weights without further tuning, as the
Results We report comparison results between the proposed RDVFI and baseline methods in Ta- ble 1, Fig. 6, and Fig. 5. The results show that our RDVFI outperforms existing baseline methods across the reported benchmark datasets. Our DiT-based method RDVFI-D outperforms the SVD- based version, RDVFI-U, benefiting from more advanced backbone network. Image diffusion- based interpolation methods (Danier et al., 2024; Lew et al., 2025) cannot create complex motions between intermediate and input frames, thus producing severe ghosting effects. Directly generating
7

---

Published as a conference paper at ICLR 2026
FCVG
Ours
End time Figure 6: Sequence interpolation comparison. Our RDVFI-D can correctly estimate the continuous pixel trajectories between input frames from diffusion outputs and thus interpolate with correct motion and superior visual quality, compared with the direct generation method Wan (Wang et al., 2025) and the linearly-controlledd FCVG (Zhu et al., 2024).
Start time
Table I: Numeric comparison results on three benchmark datasets :DAVI-77 (Jain et al., 2024), FCVG (Zhu al., 2024) and Pixels evaluation set. The best and the second-best results are highlighted by bold and underline. The proposed RDVFI outperforms existing baseline methods on most metrics.
DAVIS-7 (Jain et al., 2024)
FCVG et 2024)
Pixels
Methods
LPIPS} FIDI
FVDI
LPIPST
FID,
FVDI
LPIPS,
FID.
FVD.
Non-Gencrative Methods
AMT AMT at,,, 2023)
0.254
34.65
234.50
0.224
44.74
375.00
0.361
41.36
378.86
RIFE (Huang et al., 2022)
0.258
23.98
240.04
0.247
39.01
366.14
0.278
31.10
207.82
FILM (Reda et al., 2022)
0.271
30.16
214.80
0,241
39.82
279.08
0.251
27.06
158.68
GIMM-VFI (Guo et al., 2024)
0.276
24.10
245.02
0,231
38.71
292.08
0.269
29.21
192.37
Image Diffusion-
Methods
LDMVFI (Danier et al., 2024)
0.276
22.10
245.02
0.228
37.74
371.49
0.287
31.36
211.70
MoMo (Lew et al., 2025)
0.268
23.67
240.09
0,207
33.59
261.37
0.269
27.31
230.19
Video
Diffusion-basse based Methods
TRF (Feng et al., 2024)
0.270
29.12
230.12
0.331
45.37
305.88
0.301
35.31
279.65
GI (Wang et al,, 2024)
0.267
27.71
211.47
0.334
43.08
282.22
0.273
33.27
251.38
FCVG (Zhu al., 2024)
0.266
25.96
207.17
0.266
31.24
225.48
0.257
24.51
137.57
Wan (Wang al., 2025)
0.323
26.97
248.14
0.223
28.52
214.45
0.261
22.48
131.22
Our RDVFI-U
0.260
23.65
201.49
0.220
27.03
217.72
0.261
23.71
129.33
Our RDVFI-D
0.251
21.17
189.37
0.201
19.98
197.86
0.253
19.48
119.21
intermediate frames with Latent Video Diffusion Models (LVDM) may introduce severe degrada- tions (Wang et al., 2024), due to the mismatched interpolation results from separate forward and backward interpolation from two image-to-video SVD (Blattmann et al., 2023) models; FCVG (Zhu et al., 2024) attempts to mitigate such misalignment by involving linear motion controls, however, sacrificing motion generation ability of VDMs and utilizing them as shaders. As shown Fig. 6, FCVG (Zhu et al., 2024) produces severe degradations when the matching based linear motion con- troller does not distinguish the falling boy. Sharing the same backbone, however, our RDVFI-D significantly outperforms Wan (Wang et al., 2025). Our motion-guided decoding pipeline generates intermediate frames by warping, which forces the diffusion model to create latents that can accu- rately restore motions across frames. Thus, our interpolation is smoother and more stable, producing the fewest artifacts compared to other LVDM-baset solutions. Please find our supplementary video demo and Section for more dynamic and static visual comparisons.
4.3 EFFICIENCY COMPARISONS
We report detailed efficiency metrics in Table 2. As observe, RDVFI is the fastest diffusion- interpolation method, which can interpolate at real time (17 FPS) at the resolution of 1024 576. Although non-generative baseline method, RIFE (Huang al., 2022), is slightly
8

---

Published as a conference paper at ICLR 2026
Table 2: We compare the inference efficiency between our RDVFI and existing interpolation base- line methods, including AMT (Li et al., 2023), RIFE (Huang et 2022), FILM (Reda et al., 2022), LDMVFI (Danier al., 2024), MoMo (Lew al., 2025), GI (Wang al., 2024), FCVG (Zhu al., 2024), and Wan (Wang et al., 2025), on the FCVG (Zhu et al., 2024) dataset for x interpolation at the 1024 x 576 resolution. We bold and underline the best and the second-best results, respec- tively. Our RDVFI is the fastest and most memory-efficient diffusion-based interpolation method. Although RIFE (Huang et al., 2022) is slightly faster than our RDVFI, our RDVFI outperforms with clear margin for challenging motions.
Method
GPU Mem. (GB)
Runtime (sec.)
FID
FVD1
Non- Gon-Generative Methods
AMT L. al. (2023)
13.5
0..11)
44.74
375.00
RIFF Huang et al. (2022)
1.4
0.025
39.01
366.14 14
FILM Reda al. (2022)
8.0
0.830
39.82
279.08
Image Diffusion- based Methods
LDMVFI Danier et al (2024)
21.7
1.563
37.74
371.49
MoMo Lew al. (2025)
3.9
0.157
33.59
261.37
Video Diftusion- based Methods
GI Wang et al. (2024)
23.5
29.613
45.37
282.22
FCVG Zhu et al. (2024)
27.6
14.381
31.24
225, 448.48
Wan Wang et all (2025)
18.0
2.579
28.52
214.45
Our RDVFI-U
14.2
0.137
27.03
217.72
Our RDVFI-D
13.1
0.057
19.98
197.86
Table 3: Ablation study on different resolutions (3a) and network designs (3b).
(a) Because perform the diffusion sampling fixed and relatively smaller resolution than the inter- polated frame, the runtime the algorithm does not change a lot with the resolution increase.
(b) Our RDVFI-D consistently outperforms all abla- tion experiments. showing the best interpolation ac- curacy in terms of both reconstruction and perceptual metrics.
Runtime (ms)
Setting
LPIPS
FID
FVD4
Resolution
One-step Wan RDVFI-D
Direct Warping
0.287
42.33
327.11
576 320
103.3
55.3
L loss
0.269
33.29
281.37
1024 576
267.7
58,01
L+P-L2 loss
0.251
29.37
267.42
1280 x 720
314.3
63.49
RDVEI-D
0.224
23.71
209.38
faster than our RDVFI, it cannot deal with large complex motions and introduces severe visual degra- dations. In contrast, our RDVFI efficiently and effectively interpolate large complex motions, outperforming RIFE with a clear margin, as shown in Table 1, Fig. 5, and Fig. 6.
4.4 ABLATION STUDY
Motion-guided Decoding Pipeline We compare the runtime of one-step Wan (Wang al., 2025), which directly reduces the sampling step of the pretrained Wan model, and our RDVFI-D, These two models share the same diffusion backbone and sampling step with two core differences. First, Wan's resolution changes with interpolated frames, whereas our RDVFI-D operates at fixed, relatively lower resolution. Second, Wan generates high-resolution frames with the VAE decoder and our RDVFI-D scales up the estimated optical flows and warps high-resolution inputs for interpolation. In addition, we claim that iterative motion fusion for continuous pixel trajectories estimation is one of our key contributions to accurate motion estimation. Thus, we degrade our motion-guided decoding pipeline by removing iterative flow fusion and estimating pixel offsets between each keyframe and the input frames from scratch via regression. We define this setting "Direct Warping".
Training Strategy To validate our latent-pixel training strategy, we degrade our training objective by removing the pixel-space perceptual loss only, formatting LLP-LL loss", and all pixel-wise loss, defined as "L loss".
As shown in Table 3a and Table 3b, our RDVFI-D consistently outperforms all degraded versions on efficiency or accuracy. Benefiting from our motioon-guiden decoding, RDVFI-D can interpolate at different resolutions the same diffusion sampling cost, resulting in significant efficiency gain compared with traditional diffusion networks, as shown in Table 3a. Table 3b also shows the effec- tiveness of our flow fusion strategy and training objective design. For fair comparison, fine-tune

---

Published as a conference paper at ICLR 2026
Table 4: Human evaluation results. Most viewers select our RDVFI-D as the best-performed inter- polation method when compared with existing diffusion-based state-of-the-an methods in terms of both generated motion, frame details, and overall quality. Our RDVFI-D surpasses the second-bes method, Wan (Wang et al., 2025), for 60% more.
Direction
LDMVFI
MoMo
GI
FCVG
Wan
Ours
Motion 0.9%
0.8%
1.1%
4.1%
16.2%
76,9 %
Details
0.3%
0.9%
1.5%
9.2%
13.7%
74.4 %
Overall
0.5%
0.6%
1.0%
8.0%
14.2%
75.7 %
Frame Skip
Frame Skip
Frame Skip
O
Results
Input
Input
Results
Inpot
Figure 7: Our RDVFI can robustly interpolate when input frames have consistent objects and enough overlapping regions, such as the I-frame and 3-frame skip cases. However, when the overlapped re- gions become minimal, the ending input frame is produced by frames skipped, the algorithm cannot generate coherent and accurate intermediate optical flows to warp input pixels for interpola- tion, resulting in severe degradations.
each ablation experiment and report our results the same iterations. Please find Section A for the ablation study on the keyframe number.
4.5 HUMAN EVALUATION
To better illustrate the visual gains the proposed RDVFI over existing solutions, we also include human evaluation study involving 25 participants. We form questionnaire with ten groups of videos, each group contains six video clips from the proposed RDVFI-D and five state-of-the-ar methods, including Wan (Wang al., 2025), FCVG (Zhu et al., 2024), GI (Wang et al., 2024), MoMo (Lew et al., 2025), and LDMVFI (Danier et al., 2024). The respondents are expected to select the best video clip from each group based the generated motion quality, frame details, and overall quality. As shown in the Table 4, the proposed RDVFI-D significantly outperforms all state-of-the the-trr methods, with more than 75% of respondents selecting our RDVFI-D as the best- performing method, surpassing the second- highhst method, Wan (Wang et al., 2025), by around 5 times.
4.6 FAILURE CASE AND LIMITATIONS
Our method warps input frames with generated intermediate optical flows for interpolation, thus requiring the input frames to contain consistent objects and enough overlapping regions. As shown in Fig. 7, with the same starting frame, our method robustly interpolate with the ending input frame produced by frame and frames skipped. However, when the overlapped regions become minimal, with changed main object in the 7-frame-skipped situation, the algorithm struggles to generate coherent and accurate optical flows to move pixels from the inputs to each intermediate frame, resulting in severe degradations through interpolation.
5 CONCLUSIONS
this paper, we propose the first real-time video diffusion-based Video Frame Interpolation (VFI) pipeline that can run 17FPS at 1024 x 576 resolution with even superior interpolation quality than current multi-step solutions. Our work is advancing diffusion VFI to more practical and chal- lenging scenarios, such super slow motion and video compression. This work also hopes to sug- gest that solving ambiguous components with diffusion models, rather than end-to-end generation, may lead superior accuracy and efficiency.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work is partially supported by the RGC Early Career Scheme (ECS) No. 24209224. This work was done Yongrui Ma's internship at the Bytedance Inc.
REFERENCES
Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 3703- 3712, 2019.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15117 2023.
Karlis Martins Briedis, Abdelaziz Djelouah, Mark Meyer, lan McGonigal, Markus Gross, and Christopher Schroers. Neural frame interpolation for rendered content. ACM Trans. Graph., 40(6):1-13, 2021.
Hila Chefer, Uriel Singer. Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. Videojam: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXiv:2502.02249) 2025.
Duolikun Danier, Fan Zhang, and David Bull. Ldmvfi: Video frame interpolation with latent diffu- sion models. In AAAI, volume 38, pp. 1472-1480, 2024.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Adv Neural Inform. Process. Syst., 34:8780-8794, 2021.
Haiwen Feng, Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Abrevaya, Michael Black, and Xuaner Zhang. Explorative inbetweening of time and space. In Eur: Conf. Comput. Vis., pp- 378-395. Springer, 2024.
Yue Gao, Siqi Li, Yipeng Li, Yandong Guo, and Qionghai Dai. Superfast: 200x video frame interpolation via event camera. IEEE Trans. Pattern Anal. Mach. Intell., 45(6):7764 2022.
Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 7277- 7288, 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuming. arXiv preprint arXth:2307.047233 2023.
Zujin Guo, Wei Li, and Chen Change Loy. Generalizable implicit motion modeling for video frame interpolation. Adv. Neural Inform. Process. Syst., 37:63747-63770, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Adv. Neural Inform. Process. Syst., 30, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Adv. Neural Inform. Process. Syst., 33:6840-6853 2020.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. In Int. Conf. Learn. Represent. 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Int. Conf. Learn. Represent., 1(2):3, 2022.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In Eur. Conf. Comput. Vis., pp. 624-642. Springer, 2022.
11

---

Published as a conference paper at ICLR 2026
Siddhant Jain, Daniel Watson, Eric Tabellion, Ben Poole, Janne Kontkanen, et al. Video interpola- tion with diffusion models. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 7341-7351, 2024.
Huaizu Jiang, Deqing Sun, Varun Jampani. Ming-Hsuan Hsuan Yang, Erik Learned-Min -Mille,,,,, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 9000-9008, 2018.
Saurabh Joshi, André Forjaz, Kyu Sang Han, Yu Shen, Vasco Queiroga, Florin A Selaru, Marie Gérard, Daniel Xenes, Jordan Matelsky, Brock Wester, et al. Interpolai: deep gearning-basee optical flow interpolation and restoration of biomedical images for improved 3d tissue mapping. Nature Methods, pp. 1-12, 2025.
Robert Keys. Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing. (9(6):1153-11 2003.
Anil Kokaram, Davinder Singh, Simon Robinson, Damien Kelly, Bill Collis, and Kim Libreri. Motion-based frame interpolation for film and television effects. IET Computer Vision, 14(6): 323-338, 2020.
Jaihyun Lew, Jooyoung Choi, Chachun Shin. Dahuin Jung, and Sungroh Yoon. Disentangled motion modeling for video frame interpolation. In AAAI, volume 39, pp. 4607-4615, 2025.
Zhen Li, Zuo- Zuo-Liang Zhu, Ling-Har Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In IEEE Conf. Comput. Vis. Pattern Recog,, pp. 9801-9810, 2023.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint 11 2024.
Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, and Lu Jiang. Diffusion adversar- ial post-training for one-step video generation. In Int. Conf. Learn. Represent, 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2000000009000, 2022.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step enough for high-quality diffusion-barte based text-to-image generation. In Int. Conf. Learn. Represent. 2023.
Yihao Liu, Liangbin Xie, Li Siyao, Wenxiu Sun, Yu Qiao, and Chao Dong. Enhanced quadratic video interpolation. In Eur. Conf. Comput. Vis. Workshop. pp. 41-56. Springer, 2020.
Yongrui Ma, Shi Guo, Yutian Chen, Tianfan Xue, and Jinwei Gu. Timelens-xl: Real-time event- based video frame interpolation with large motion. In Eur. Conf. Comput. Vis., pp. 178-194. Springer, 2024.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine Horning, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 2017.
Martin Priessner, David CA Gaboriau, Arlo Sheridan, Tchern Lenn, Carlos Garzon-Coral, Alexan- der R Dunn, Jonathan Chubb, Aidan M Tousley, Robbie G Majzner, Uri Manor, et al. Content- aware frame interpolation (cafi): deep learning-basse based temporal super-resolution for fast bioimag- ing. Nature Methods, 21(2):322-330 2024.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. urXiv preprint 221:22204400125, 1(2)3, 2022.
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. Eur. Conf. Comput. Vis., pp. 250-266. Springer, 2022.
12

---

Published as a conference paper at ICLR 2026
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, Adv. Neural Inform. Process. Syst, 35:36479-36494, 2022.
Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-vider generation with explicit motion modeling. In Int. Conf. Comput. Graph. Interact. Tech., pp. 1-11, 2024.
Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. Int. Conf. Comput. Vis., pp. 14489-14498, 2021.
Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint TrXiv:2503.203/11 2025.
Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmacher-Shlizemmen Aleksander Holynski, and Steven M Seitz. Generative inbetweening: Adapting amage-to-videe models for keyframe interpolation. Int. Conf. Learn. Represent. 2024.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Adv. Neural Inform. Process. Syst., 36:8406 8441, 2023.
Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P Allebach, and Chenliang Xu. Zooming slow-mo: Fast and accurate one-stage space-time video super-resolution. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 3370-3379, 2020.
Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpola- tion. Ady Neural Inform. Process. Syst., 32, 2019.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. Int. J. Comput. Vis., 127(8):1 (06-1125, 2019.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-vider diffusion models with an expert transformer. arXiv preprint arXiv:2408.0600 2024.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 6613-6623, 2024.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog.. 2018.
Zhixing Zhang, Yanyu Li, Yushu Wa, Anil Kag, Ivan Skorokhodov, Willi Menapace, Aliaksandr Siarohin, Junli Cao, Dimitris Metaxas, Sergey Tulyakov, al. Sf-v: Single forward video gener- ation model. Adv. Neural Inform. Process. Syst., 37:103599-1036188 2024.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint /805../805.000 2018.
Tianyi Zhu, Dongwei Ren, Qilong Wang, Xiaohe Wu, and Wangmeng Zuo Generative inbetween- ing through frame-wise conditions-driven video generation. arXiv preprint arXiv:2412.11755 2024.
13


Reviews:
[{"review_id": "ETH88oYLDL", "reviewer": "Reviewer_Ba4o", "summary": "This paper proposes a video frame interpolation method using a one-step diffusion model, aiming to improve inference efficiency by eliminating the need for multi-step denoising. The approach decomposes intermediate frame generation into two stages: the first estimates a continuous motion field between the input frames, and the second synthesizes intermediate frames by warping the inputs according to the predicted motion field. Experimental results are compared against conventional video frame interpolation methods as well as diffusion-based approaches.", "questions": "1. How does  the number of keyframes in the continuous motion representation affect the inbetweening results especially in 24x interpolation?\n2. Directly fine-tuning the original diffusion denoiser to perform full noise removal in a single step seems rather ambitious. Incorporating a distillation-based loss might help improve stability and performance.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper aims to improve the efficiency of diffusion models for video frame interpolation by disentangling in-between frame synthesis into two stages: continuous motion prediction and warping. The continuous motion (which use a spline interpolation curve) prediction stage is lightweight, operating on latent features from spatially and temporally downsampled video representations, which reduces memory consumption. In addition, the method employs a one-step diffusion model to predict the latent features during inference time further increasing inference speed.", "weaknesses": "The main weakness is the experiment does not convince me of the effectiveness of the method:\n1. There are only six qualitative video comparisons in the supplementary material, which are insufficient to demonstrate the superior quality of the proposed method. Moreover, I did not observe a clear visual difference between Wan and the proposed approach.\n2.  Lacks baselines on direct one/few-step distillation of video in-betweening diffusion models, rather than decomposing the process into two stages as done in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nqiZOMBluc", "reviewer": "Reviewer_R4Dp", "summary": "1. The core idea of RDVFI is to disentangle VFI into two stages: motion prediction and appearance generation. Generating new frames based on low-resolution motion information enables RDVFI to perform effectively in both inference speed and generation authenticity.\n2. This is the first diffusion-based VFI method with one-step inference, which achieves 50× acceleration compared to SOTA with also better results.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. In the diffusion stage, this method generates low-resolution optical flow as an intermediate result to both accelerate the diffusion process and improve the stability of interpolation.\n2. Bi-directional interpolation significantly improves generation authenticity, making RDVFI get SOTA performance on DAVIS and FCVG benchmarks.\n3. The video demo shows significantly better results than existing methods.", "weaknesses": "1. Optical flow is the key information for the entire pipeline, as it is utilized for both image warping and feature warping in a bidirectional manner. However, in the first training stage, optical flow results are trained in an unsupervised way. The rationality of this setting requires further verification.\n2. The number of testsets is limited and lacks diversity in their sources. The authors evaluate their method on DAVIS-7 and FCVG, however FCVG is sampled from DAVIS and RealEstate10K.\n3. For the RDVFI-U and RDVFI-D models, the authors set different numbers of key frames, but did not provide a reasonable explanation and lacked corresponding ablation experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wg0SxYEsOz", "reviewer": "Reviewer_G3iW", "summary": "The authors propose a novel one-step diffusion method for video frame interpolation. The authors present a two-stage strategy. Specifically, it first decomposes the interpolation process into motion modeling and frame synthesis. The motion modeling and frame synthesis module are trained together in an end-to-end format, with the ground truth frame latents as the additional input. In the second training stage, they train the diffusion model for denoising the pseudo ground truth frame latents for motion prediction. In their experiments, the proposed method outperforms the baselines with fast speed.", "questions": "- Is it necessary to use the VAE encoder? Given the recent progress in the academy, it would be interesting to replace the VAE encoder with other encoders, such as DINOv2.\n- Please add more visualizations for the cases indicated in the weakness section, including more challenging scenarios and continuous temporal results, to support the claimed interpolation ability.\n- It is necessary to have discussions with previous closely related work to enhance the clarity of the paper's motivation and contribution. \n- Please add more detailed descriptions for the core motion modeling part, especially the flow sampling operation, for better presentation.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors propose a novel framework for video frame interpolation, achieving fast speed and competitive performance.\n- The proposed continuous motion field representation enables more flexible motion modeling and generates plausible flow samples, as evidenced by the visualizations.", "weaknesses": "- The paper lacks visualizations on more challenging scenarios, such as the “breaking dance” case in the DAVIS dataset.\n- The flow sampling mechanism is not sufficiently explained; more details on how samples are generated and utilized would strengthen the paper.\n- The paper is missing discussions of closely related works that also integrate motion or optical flow modeling in video generation/interpolation, such as *VideoJAM* [1], *Motion-I2V* [2], and *GIMM-VFI* [3].\n- Continuous temporal results. More visualizations like Figure 5 to present the continuous motion modeling and interpolation ability.\n\n[1] VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models.\n\n[2] Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling.\n\n[3] Generalizable Implicit Motion Modeling for Video Frame Interpolation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3qnirnJAz", "reviewer": "Reviewer_1Tgm", "summary": "This work introduces an efficient video interpolation approach based on a video diffusion model, enabling real-time frame interpolation. The proposed pipeline consists of three main stages: (1) estimating sparse, low-resolution keyframes through a one-step video diffusion process; (2) extracting a complex motion field from these keyframes using a continuous motion estimator; and (3) synthesizing full-resolution intermediate frames from the motion field and the input frames ($I_0$ and $I_1$) using a frame synthesis network. In addition, the authors propose a two-stage training framework to address the unstable convergence may observed in simple end-to-end training.", "questions": "1) Please include explicit efficiency metrics — such as inference time (s/frame), memory usage (GB), and training compute (e.g., approximate FLOPs or other comparable measures) — in the main manuscript to better support the claimed efficiency of the proposed framework.\n\n2) Please clearly categorize the baselines into zero-shot, fine-tuned, and fully trained settings for a fair comparison. Additionally, consider including more zero-shot video diffusion interpolation works such as TRF [1] and ViBiDSampler [2], which are missing from the current submission.\n\n3) If possible, add a brief ablation study on the number of keyframes ($k$). This would significantly improve the clarity and completeness of the paper by illustrating how $k$ influences both interpolation quality and computational efficiency.\n\n4) It would be helpful to include failure case analysis. Presenting representative failure cases and describing in which situations such samples frequently occur would provide deeper insight into the limitations and behavior of the proposed method.\n\n[1] Explorative Inbetweening of Time and Space, ECCV 2024.\n\n[2] ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler, ICLR 2025.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Traditional non-generative methods preserve the appearance of input frames well but struggle to interpolate dynamic motions. In contrast, diffusion-based methods effectively model and interpolate complex motions but are computationally expensive and often fail to maintain the appearance consistency of input frames.\n\nThis paper proposes a novel approach that disentangles motion from the video diffusion model (VDM) and applies the estimated sparse motion to achieve efficient and effective video interpolation.\nThe main strengths of the paper can be summarized as follows:\n\n1) Disentangling motion field from a one-step VDM is both novel and effective.\n\n2) Integrating the motion field extracted from the VDM into a frame synthesis framework improves efficiency compared to full VDM-based methods.\n\n3) The qualitative and quantitative results are impressive and demonstrate the effectiveness of the proposed method.", "weaknesses": "1) Insufficient experimental analysis on efficiency:\nAlthough the proposed framework emphasizes efficiency as a key contribution, the paper lacks detailed quantitative evidence. The authors should include explicit efficiency metrics in Table 1, such as inference time (s/frame) and memory usage (VRAM in GB), to substantiate their claims.\n\n2) Lack of detailed baseline categorization:\nFor video interpolation methods based on video diffusion models (VDMs), it is crucial to clearly categorize baselines into zero-shot, fine-tuned, and fully trained settings. Without this distinction, the fairness and clarity of the comparison become ambiguous, potentially undermining the paper’s original motivation. Reporting the total training compute—e.g., approximate FLOPs or other comparable measures—would further improve the transparency and fairness of the evaluation.\n\n3) Limited ablation studies:\nThe initial estimation of $k$ keyframes using the VDM plays a central role in determining the motion field and thus in the overall effectiveness of the proposed method. However, the paper lacks sufficient analysis of how the number of keyframes ($k$) affects both efficiency and interpolation quality. A detailed ablation study on this factor would greatly enhance the understanding of the method’s underlying principles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)