Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation
Authors: Xiaojuan Wang
Boyang Zhou
Brian Curless
Ira Kemelmacher-Shlizerman
Aleksander Holynski
Steve Seitz
Venue: ICLR 2025 Poster (2025)
Keywords: generative keyframe interpolation
image-to-video diffusion models
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We present a method for generating video sequences with coherent motion between a pair of input keyframes. We adapt a pretrained large-scale image-to-video diffusion model (originally trained to generate videos moving forward in time from a single input image) for keyframe interpolation, i.e., to produce a video between two input frames. We accomplish this adaptation through a lightweight fine-tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single input image. This model (along with the original forward-moving model) is subsequently used in a dual-directional diffusion sampling process that combines the overlapping model estimates starting from each of the two keyframes. Our experiments shows that our method outperforms both existing diffusion-based methods and traditional frame interpolation techniques.

Published as a conference paper at ICLR 2025
GENERATIVE INBETWEENING: ADAPTING IMAGE-TO- VIDEO DIFFUSION MODELS FOR KEYFRAME INTERPO- LATION
Xiaojuan Wang' Boyang Zhou'
Brian Curless
Ira Kemnemacher--Shizzemaan
Aleksander Holynski2.3 Steven M. Seitz University of Washington, ?Google DeepMind, SUC Berkeley
ABSTRACT
We present method for generating video sequences with coherent motion be- tween a pair of input keyframes. We adapt pretrained large-scale image-to-video diffusion model (originally trained generate videos moving forward in time from a single input image) for keyframe interpolation, i.c., to produce a vidco be- tween two input frames. We accomplish this adaptation through lightweight fine- tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single imput image. This model (along with the original forward-moving fusion sampling process that combines the overlapping model estimates starting from each of the two keyframes. Our experiments shows that our method outper- forms both existing diffusion- baassd methods and traditional frame interpolation techniques.
model) is subsequently used in dual-directional dif-
INTRODUCTION
Recent advances of large-scale text-to-video and image-to-video models (Blattmann al. 2023b a Wu et al. 2023 Xing et al., 2023 Bar et al., 2024; Zeng et al. (024) have shown the ability to generate high resolution videos with dynamic motion. While these models can accept vari- ety of input conditioning signals, such as text captions or single images, most available models remain unsuitable for an obvious application: keyframe interpolation. Interpolating between a pair of keyframes- that is, producing video that simulates coherent motion between two input frames, one defining the starting frame of the video, and one defining the ending frame-is certainly possi- ble if a large-scale model has been trained to accept these particular two conditioning signals, but most spercsource models have not. Despite the task's 's similarity to existing conditioning signals, creating interpolation model requires further training, and therefore both large amounts of data and substantial computational resources beyond what most researchers have access to.
Given the similarity between the input signals needed for keyframe interpolation (i.e., two-frame conditioning) and the input signals to existing models (e.g., one-frame conditioning), an interesting alternative solution is to instead adapt an existing pre-trained image-to-video model, without train- ing a specialized model from scratch. In this paper, we propose an approach for enabling key keyframe interpolation by doing precisely this, Our approach is founded upon the observation that keyframe interpolation model needs to know how to accomplish three objectives: (1) given starting frame, it needs to predict coherent motion starting from that frame and advancing into the future, (2) given an ending frame, it needs to predict coherent motion starting from that frame and advancing backwards into the past, and (3) given these two predictions, produce video that has a coherent combination of the two. Since existing image-to- to-vidoo models can already accomplish the first of these three ob- jectives, we focus our efforts on the the latter two, i.c., producing a Lingle-rrame conditioned model that can generate motion backwards in time, and mechanism for combining forward and backward motion predictions into coherent videos.
One may imagine that producing such single-image conditioned model that produces backwards motion should be trivial: simply pass an image into a regular image-to-video model, and reverse the output. Unfortunately, real-world motion is inherently asymmetric, and reversed motion into the

---

Published as a conference paper at ICLR 2025
future is notably different from motion into the past. As such, we first propose novel, lightweight fine-tuning mechanism that reverses the arrow of time by rotating the temporal self-attention maps (i.e., reversing the temporal interactions) within the diffusion U-Net. This enables the reuse of the existing learned motion statistics in the pretrained model, and enables generalization while only requiring small number of training videos.
Given both the original image-to-vides model and this adapted reverse model, we also propose a sampling mechanism that merges the scores of both produce a single consistent sample. These two sampling paths are synchronized through shared rotated temporal self-attention maps, ensur- ing they generate exactly opposite motions, an effect which we term "forward backward motion consistency". At each sampling step, their intermediate noise predictions are fused, resulting in a generated video with coherent motion that starts and ends with the provided frames. We compare our work qualitatively and quantitatively to related methods on two curated difficult datasets targeted for generative inbetweening: Davis (Pont- Tuste et al. 2017) and Pexel=] and our method produces notably higher quality videos with more coherent dynamics given distant keyframes.
"forward-backward
RELATED WORKS
Frame interpolation Frame interpolation (Dong et al. tween two frames by taking pair of input frames or multiple adjacent frames in the context of video frame interpolation, and has been a long-standing research area in computer vision. Example applications include temporal up-sampling to increase refresh rate, create slow-motion videos, or interpolating between near-duplicate photos. Much of the research in this field (Jiang et al., 2018 Niklaus & Liu 2020 Huang al. 2020; Park et al. 2020 Lee et al., 2020 Park et al. 2021 employs flow- methods, which estimate optical flow between the frames and then synthesize the middle images guided by the flow via either warping splatting techniques. There are also works (Kalluri et al. 2023] Shi et al. 2022) that use CNNs or transformers to learn to extract fea- tures and directly output the middle frames. Traditionally, this task assumes unambiguous motion and the input frames are usually closely spaced (< 1/30s) samples in the video. Recent studies have begun to address large motions (Sim et al., 2021 Reda et al.] 2022), or quadratic motion (Xu et al. 2019 Liu et al., 2020), though these still involve single motion interpolation and cannot ad- dress distant input frames. In contrast, we aim to generate in-between frames that capture dynamic motions across distant input keyframes (2 1s apart) with a generative model, a challenge that goes beyond the capability of current frame interpolation techniques.
2023 synthesizes intermediate images be-
Diffusion models for in-between video generation Diffusion models have shown remarkable ca- pabilities for generative modeling of images (Ho et al.] 2020
Dhariwal & Nichol 2021 Song & 2015) and videos (Ho et al. 2022b Wu
Ermon 2019; [Song et al. 2020a by [Sohl-Dickstein et al.,
cct al. 2023 Blattmann et al. 2023b). Early work MCVD (Voleti et al.) 2022) devises a general- purpose diffusion model for a range of video generative modeling tasks including in-between video generation. More recent works (Guo et al. 2023) Jain et al. 2024| Xing et al. 2023) explicitly train diffusion models to accept two end frames with conditioning to generate 7 or 16 intermediate frames at maximum resolution of 320 x 512 at once, and achieved superior results in generating dynamic motions. In this work, we focus on adapting pre-trained large-scale image-to-video model to do keyframe inbetweening without having to train or fine-tune from scratch. Exposed to millions of videos, these models have demonstrated remarkable capabilities in generating high-resolution (up to 1080p) and long (up to 4s) videos with rich motion priors.
Diffusion sampling for consistent generation In diffusion-based image generation tasks, novel joint diffusion sampling techniques (Bar-Ta Tal et al., 2023) Zhang et al.) [0033] Tang et al. 2023 Lec al., 2023) for consistent generation are usually employed in generating arbitrary-sized images or panoramas from smaller pieces. These methods involve concurrently generating these multiple pieces and merging their intermediate results in the overlapping areas within the sampling process. For example, MultiDiffusion (Bar- Tal et al.) 2023) averages the diffusion model predictions to rec- oncile the different denoising processes. Recent work, TRF (Feng et al., 2024) extends this joint generation approach to the bounded video generation taking two end frames as input. By running two parallel image-to-vin -ivid generations guided by the start and end frames, it merge their outputs
hhtps//wwww.peeck.com//
2

---

Published as a conference paper at ICLR 2025
by averaging in each denoising step. However, significant drawback of this method is that it can- not generate coherent motion in-between: simply fusing a forward video generation from the first end frame and the reversed forward video starting from the second end frame using single image- to-video model designed for forward motion only causes the generated videos to oscillate between moving forward and then reversing, rather than continuously progress forward as method does.
3 BACKGROUND
We introduce some background on Stable Video Diffusion (Blattmann et al. 2023a). ) the base image- to-video diffusion model used in our work, and then specifically explain the temporal self-attention layers within its architecture, which are key to modeling motion within the generated video.
3.1 STABLE VIDEO DIFFUSION
Diffusion models trained to convert random noise into high-resolution images/videos via an iterative sampling process DDhariwal & Nichol 2021] Sohl-Dickstein et al.] 2015 Song & Ermon
2019 Song al. 2020a Liu et al. (2022). This sampling process aims reverse fixed, time- dependent destructive process (forward process) that gradually corrupts data by adding Gaussian noise. In particular, Stable Video Diffusion (SVD) is latent diffusion model where the diffusion process operates in the latent space of pre-trained autoencoder with encoder E.) and decoder D()).
In the forward process, video sample x { [N.,...11. composed of N frames, is first encoded in the latent space z = {x). then the intermediate noisy video at time step t is created as Z, a, + o,E, where e N is Gaussian noise, and a, and o, define a fixed noise schedule. The denoising network fo receives this noisy video latent and the conditioning e computed from the input image, i.e., the first frame To in the video, and trained by minimizing the loss:
((0) =
(2,; t, c) y1
where the target vector here is v = 0,6 - 0,4,, referred to as v-prediction.
Once the denoising network is trained, starting from pure noise Zr No., I), the sampling process iteratively denoises the noisy latent by predicting the noise in the input and then applying an update step remove portion of the estimated noise from the noisy latent
Z - update(zz fo(z;; t, c):
until we get clean latent Zp. followed by decoding D(zo) to get the generated video. The exact implementation of the update(-. .) function depends on the specifics of the sampling method; SVD
uses EDM sampler (Karras et al. 2022).
3.2 TEMPORAL SELF-ATTENTION
The denoising network fe in SVD is 3D U-Net, composed of "down", "mid", and "up" blocks. Each block contains spatial layers interleaved with temporal layers, with the temporal self-attention layers responsible for modeling motion in the generated video. This layer takes spatio-temporal tensor X x xxxxx XX input, where N is the number of frames, and is the number of channels. Here we use batch size of 1 for simplicity. The tensor is reshaped by moving the spatial dimensions (H, W) into the batch dimension. This creates X' E R xX'C'' where self-attention operates solely on the temporal axis. More specifically, X' is projected through three separate matrices Wg. WR, W. E d (d is the dimensionality of the projected space.), resulting in the corresponding query (Q = W X'), key (K WxX') and value (V W, features. Then the scale-dot product attention is applied:
Attention(( K. V) softmax(QK /Vay
The attention output fed through another linear layer W, to get the final output. We refer to A = QK" E R "W' NxN as the temporal self-attentio map, which models the inter-frame correlations per spatial location. This temporal attention mechanism allows each frame's 's updated feature to gather information from other frames.
3

---

Published as a conference paper at ICLR 2025
Iterative denoising
Cp
Train Plaaa))) 3
Fuse Update
Inference
2r ((
Zt
Vr,0
Reverse
Reverse
rotation
W
fo
Z
V1,1
Figure 1: Method overview. In the lightweight backward motion fine-tuning stage, an input video = (To, JI,... IN-1} is encoded into the latent space by E(x), and noise is added to create noisy latent Zy; during inference, Z, is created by iterative denoising starting from Zr N I). (1) Forward motion prediction: we first take the conditioning of the first input image (inference stage) or the first frame in the video (training stage) Jo. along with the noisy latent 2, to feed into the pre-trained 3D U-Net fa to get the noise predictions Vi.0, as well as the temporal self attention maps {..}. (2) Backward motion prediction: We reverse the noisy latent z, along temporal axis to get z/. Then take the conditioning CN-1 of the second input image, or the last frame in the video IN-1. along with the 180-degree rotated temporal self-attention maps {A,), and feed them through the fine-tuned 3D U-Net for for backward motion prediction Vi,1. (3) Fuse and update: The predicted backward motion noise is reversed again to fuse with the forward motion noise to create consistent motion path. Note that only the value and output projection matrices W (v.o) in the temporal self-attention layers (green) are fine-tuned; see Fig. for more details.
METHOD
Given a pair of keyframes To and -N- our goal to generate video {To, J1, 72, IN-1) that begins with frame To and ends with frame Jx-1, leveraging the pre-trained image-to-video Stable Video Diffusion (SVD) model. The generated video should exhibit a natural and consistent motion path, such car traveling person walking in steady direction.
Image-to-videe video models typically generate video with motions that run forward time. is primarily the temporal self-attention layers that learn this motion-time association. In Sec Sec|4.1, discuss how this forward motion can be reversed by rotating the temporal self-attention maps by 180 degrees. Then introduce efficient lightweight fine-tuning technique to reverse this association and enable SVD to generate backward motion videos from the input image in Sec. 4.2 Finally we present dual-directional sampling approach that fuses the forward motion generation starting with frame to and backward motion video generation starting with frame IN-1 in consistent manner in Sec. 4.3 An overview of our method is shown in Fig.
4.1 REVERSE MOTION-TIME ASSOCIATION BY SELF-ATTENTION MAP ROTATION
The temporal self-attention maps {A,} in the network fo feature the forward motion trajectory in video {Lo, I,,.,,,.,, ..., IN- IN-1 }. By rotating these attention maps by 180 degrees, obtain new set {A( that depicts the opposite backward motion, corresponding to the reversed one { (Iv-1, N To) starting from the last frame T1-
Specifically, rotating the temporal self-attention maps by 180 begrees-flippnng them vertically and horizontally-yieldss a backward motion opposite the original forward motion. For example, consider attention map A; the rotated map A'y A'- = Aj,., where A,, indicates the attention score between the j-th and k-th frames (I, and Ie). In the corresponding reversed video, the reverse frame indices N -- j and N k maintain the same relative response.
4

---

Published as a conference paper at ICLR 2025
4.2 LIGHTWEIGHT BACKWARD MOTION FINE-TUNING
We introduce lightweight fine-tuning framework that specifically fine-tunes the value and output projection matrix W., ,, W. in the temporal self-attention layers, using the 180-degree rotated atten- tion map from the forward video as additional input (see Fig. 2]. We use (z L, c, (4()) to denote the backward motion generation network. This fine-tuning approach offers two key advantages:
First, by utilizzig existing forward motion statistics from the pre-trained SVD model, fine-tuning W (v.of simplifies the model's 's task to focus on learning how to synthesize reasonable content when operating in reverse. This strat- egy requires significantly less data and fewer parameters compared full model fine-tuning. Second, enables the control for the model to generate backward motion trajectory corresponding to the opposite of the forward trajectory described by the attention map. This feature is particularly beneficial when planning to merge forward and backward motions converging towards each other, and thus achieving forward-backward consistency.
X EERT N
A
180 rotation
Linear Wv
softmax (arimax(AA//// (A'/Va)V
A'
process is shown in Alg. [ For la- The detailed - tent video z E xgxxXxCxx*XX we denote flip(z) specif- ically by the second dimension, i.e., reversing the latent video along the time axis. In every training iteration, we sample an input video of N frames, and random time step f, then the noisy video latent z, is created by adding the noise in that time step. The noisy video latent along with the input conditioning the pre-trained 3D U-Net Se to extract the self attention maps {A,} from the temporal attention layers. Then we reverse the noisy video latent, along with the last frame conditioning CN-1, feed them into the backward motion 3D-U-Net for The loss function is computed by taking the predictions of the network and the ground truth re- verse video.
Linear Wo
Temporal self-attention module in the backward motion gen- eration. Given input tensor X, our attention mechanism additionally takes the respective attention map A from the pre-trained SVD featuring forward mo- tion, rotating it by 180 degrees to create a reverse motion-time association A'. Note that W (v.o) are the only trainable parameters in this module.
Figure
(computed from (o) is fed into
ALGORITHM 1: Light-weight backward motion fine-tuning
Input: fe. Paum (
while not converged do
Sample x Pam. (x), x = )i .2 E(x):
Compute conditioning cu from Iu;
f ~ Uniform(1 ()), NO( I);
0:2 d:C;
{A:} cxtract..ttentioo.aapp((aada
z! = flip(z.);
Compute conditioning ex from IN-
Take gradient descent step on Wwq.. |I Lx (41) - -y1i.y <x:lip(e)) az2i;
end
Return: W W(v.o}
4.3 DIRETIORRC SAMPLING
WITH FORWARD- FORWARD-BACKWARD MOTION CONSISTENCY
Our complete dual- direecirrtioonl sampling process is detailed in Alg 2 Given pair of keyframes lo and IN-1, their corresponding conditioning cu and cx-1 are pre-computed. Then each sampling step (illustrated in Figure D works as follows:
(1) Forward motion denoising with Io input: The noisy video latent 2 along with the conditioning Cp fed into the pre-trained 3D U-Net fo in SVD to predict the noise volume Vr,o. Additionally, the temporal aelf-attention maps (A,) in the 3D U-Net are extracted.
5

---

Published as a conference paper at ICLR 2025
(2) Backward motion denoising with IN-1 as input: The noisy video z, is flipped along the tem- poral dimension to create the reverse video latent z corresponding to the backward motion. This backward video, along with the conditioning cN-1, well as the 180-degree rotated attention maps {4{}, are fed into our fine-tuned 3D U-Net for This step predict the noise volume Vr,1 representing reverse motion from Ix-1-
(3) Finally, the predicted noise volumes from both forward and reverse motion paths are fused and then denoised using the update(-.-) function create less noisy video Zt-1. In this way, we ensure forward-backward consistency and thus consistent moving direction in the generated video. The fuse( function performs simple average.
practice, we also adopt per-step recurrence to enhance the fusion as seen in (Bansal et al.) 2023 Feng et al. (024), by re-injecting Gaussian noise into the update 7 and repeating the denoising times before continuing the sampling for the next step.
ALGORITHM 2: Dual-directions diffusion sampling
Input: Io. IN-1, So. Se. for D())
Compute condition ca, ex-1 from Io, IN-1;
Set žr (0, I);
for +
do
0. = fo(zs;t, co);
[4:]
co));
z; = flip(z);;
V (or(z LN-11, (A(});
V. flip(fr.);;
fuse((vr,o, Vi,1);
7;-1 = update(Zs, ne, Vett))
end
Return: D(z.)
4.4
IMPLEMENTATION DETAILS
Our lightweight fine-tuning technique fine-tunes less than 2% of the U-Net parameters, and does not rely on large collection of training videos. So we collected 100 high quality videos which are originally generated from SVD from community website as training data. Our experimental results show that our method generalizes well to the real image data. We select the ones with large object motion such as animal running, vehicle moving, people walking, and on. We use the Adam optimizer with learning rate of le - 4, 81 0.9, .2 0.999, and weight decay of le - 2. The training takes around 15K iterations with batch size of 4. We trained 4 A100 GPUs. For sampling, we apply 50 sampling steps. For other parameters in SVD, we use the default values: motion bucket 127, noise aug strength 0.02.
EXPERIMENTS
we demonstrate that our approach successfully generates high quality videos with consistent motion given distant keyframes. We highly recommend viewing the videos in the supplementary to see the results more clearly. Sec. [.1 describes the data we used to evaluate our method and the baselines. Sec. 5.2 demonstrates how method outperforms traditional frame interpolation method FILM, and the recent work TRF (Feng et al. 2024) that also leverages SVD for video generation. Sec. 3.33 justiffee our design decisions with ablation study. Sec. d.c..s.4isscuusess the optimal scenarios where our method excels and sub-optimal ones where it outperforms the basclines but remains limited by SVD itself. Sec. [5.5 discusses our failure cases.
In Figs.
5.1 EVALUATION DATASET
We high-resolution (1080p) datasets for evaluations: (1) The Davis dataset (Pont- Pront-Tuset et al., 2017), where we create a total of 117 input pairs from all of the videos. This dataset mostly features
https//wwww.abbeevitec.oom

---

Published as a conference paper at ICLR 2025
GT
FILM
TRF
Ours
6
12
18
24
GT
FILM
TRF
Ours
i=2 21
1= 22
1= 23
i= 24
GT
FILM
Ours TRF
22
24
Figure 3: Qualitative baseline comparisons. Leftmost (i 0) and rightmost columns 24): start and end frames. TRF generates back-and-forth motions, such as vehicles moving forward and then reversing. FILM struggles to find correspondences when the input frames distant and morphs from the first frame to the last. The red arrow indicates the direction of motion. We recom- mend viewing the supplementary videos.
subject articulated motions, such as animal or human motions. (2) The Pexels dataset, where we collect total of 106 input keyframe pairs from a compiled collection of high resolution videos on F featuring directional dynamic scene motions such as vehicles moving, animals, or people running, surfing, wave movements, and time-lapse videos. All input pairs are at least 25 frames apart and have the corresponding ground truth video clips.
hhaps//wwww.peeck.com//

---

Published as a conference paper at ICLR 2025
5.2 BASELINE COMPARISONS
We mainly compare approach FILM (Reda et al.; 2022), the current state-of-the-art frame interpolation method for large motion, and TRF (Feng et al., 2024) which also adapts SVD for bounded generation. We show representative qualitative results in Figs. 3 5 In addition, also include results for the keyframe interpolation feature from the recent work DynamiCrafter (Xing) al., 2023) large-scale image-to-video model. The keyframing feature modified from it and specially trained to accept two end frames as conditions, while we focus on how to adapt a pretrained image-to-video model in lightweight way with small collection of training videos and much less computational resources. This feature generates videos at resolution 512 x 320, while ours generates resolution 1024 x 576. Nonetheless, we present its results for reference.
Quantitative evaluation For each dataset, we evaluate the generated in-between videos using FID (Heusel al., 2017) and FVD (Ge et al.] 2024), widely used metrics for evaluating gen- erative models. These two metrics measure the distance between the distributions of generated frames/videos and actual ones. The results are shown in Tab. [ and our method outperforms all of the baselines by significant margin.
Pexels
Davis
FID +
FVD L
FID
FVD.
FILM (Reda et al. TRF
2019
25.16
371.83
41.85
1048.65
Feng el al. 2024)
31.43
563.16
36.79
563.07
DynamiCralter (Xin el al.
2023
32.06
393.12
38.32
439.74
Ours w/o RA
26.42
458.76
36,70
549.98
Ours w/o FT
37.68
555.10
47.23
604.76
Ours
22.99
306.84
32.68
424.69
Table 1: Comparisons with baselines and our ablation variants. Ours w/o RA: full pipeline with fine- tuning all parameters (g.. without using the 180-degree rotated temporal attention map. Ours w/o FT: full pipeline using rotated attention map only in the "up" blocks and without fine-tuning W [evo) for backward motion.
Comparison to FILM The flow-based frame interpolation method FILM suffers from two prob- lems. First, it struggles to find correspondences in scenes with large motions. For example, in the second row of Fig. 3] in a highway where vehicles moving in both directions, FILM fails to find the correspondence between the moving cars across the input keyframes, resulting in implausible intermediate motions. For example, some cars in the first frame disappear in the middle and reap- pear at the end. Second, it generates undesirable unambiguous motion which takes the shortest path between the end frames. In the example in Fig. [ given two Amilar-looking frames that captures different states of a person running, FILM produces motion that merely translates the person across the frames, losing the natural kinematic motions of the legs.
Comparison to TRF TRF fuses the forward video generation starting from the first frame and the reversed forward video starting from the second frame, both using the original SVD. The reversed forward video from the second frame creates a backward motion video that ends at the second frame. Fusing these generation paths results in back-and-forth motion in the generated videos. One notable effect we observe with TRF is that the generated videos exhibit a pattern of progressing forward first and then reversing to the end frame. For example, in the third row of Fig. 3 we can see the red truck moving backward over time; the seventh row, the dog's 's legs are moving backwards, leading to unnatural motions. In contrast, our approach fine-tunes SVD to generate a backward video starting from the second frame in the opposite direction to the forward video from the first frame. This forward- video.
forward-backward backward motion consistency leads to the generation of motion-consistent
5.3 ABLATIONS
In Fig. and Tab. we show visual and quantitative comparisons to simpler versions of our method to evaluate the effect of the key components in method.
8

---

Published as a conference paper at ICLR 2025
RA
w/O THT
Ours
i=0
1=2
24
Figure 4: Ablation study. We evaluate other options for generating in-between motion consistency. (1) Ours wo RA: full pipeline with fine-tuning all parameters (g..k.oo) in the temporal attention layers but without using 180-degree rotated temporal self-attention maps extra input (top row). (2) Ours w/o FT: full pipeline without fine-tuning W (o,oa) for backward motion (second row). The differences are highlighted in the red rectangle.
Fine-tuning without rotated attention map (Ours w/o RA) We compare with variant that fine- tunes all parameters in the temporal self-attention layers, namely, W (n,k,oo)) but without using the 180-degree rotated temporal self-attention map from the forward video as extra input. Though fine-tuning all parameters can generate backward motion from the second input image, there is no guarantee that the backward motion will mirror the forward motion from the first input image. This discrepancy makes hard for the model to reconcile the two motion paths, often resulting in blending artifacts, as shown in the top row of Fig. In contrast, fine-tuning W (v.o) with the rotated attention maps generates coherent and high-fidelity in- in-between videos.
Fine-tuning Wtv.o3 vs. no fine-tuning (Ours w/o FT) Sec. 4.1 we show that rotating the temporal attention maps by 180 degrees reverses the motion-time association, creating backward motion trajectory. Here we show that fine-tuning the value and output projection matrices We necessary for the model to synthesize high-fidelity content given the input backward motion- time association. We run our full pipeline without any fine-tuning, and our attention map rotation operation only applied to the "up" blocks this variant. As shown in the second row Fig. and Tab. D without fine-tuning these parameters, the model can create consistent motion but suffers from poor frame quality due to the low frame quality the backward video generation. For example, the person is disfigured in the generated video. Note that applying the attention map rotation operation to the "down" and "mid" blocks this variant worsens visual fidelity even further; thus, we show the best-case scenario without fine-tuning (i.e., applying rotated attention maps the "up" blocks only).
5.4 OPTIMAL AND SUB-OPTIMAL SCENARIOS
Our method is limited by the motion quality and priors learned by SVD. Firstly, our empirical ex- periments indicate that SVD works well with generating rigid motions, but struggles with non-rigid, articulated movements. has difficulty accurately rendering the limb movements of animal/people. In Fig. 5 though method significantly improves upon FILM and TRF, it still appears unnatural compared to the ground truth movements. The bottom row, showing the sequence generated by SVD using only the first input frame, confirms that SVD itself struggles to generate natural running movements in between.
5.5 FAILURES
When the input pairs are captured at such distant intervals that they have sparse correspondences, as shown in Fig. 6, where only small portion of cars appear in both input frames, becomes difficult for our method to fuse the forward and backward motions. This situation, where the overlapping areas minimal, leads artifacts in the intermediate frames.

---

Published as a conference paper at ICLR 2025
GT
FILM
TRF
Ours
SVD
i= 6
i i= 12
18
i= 24
Figure 5: Our method outperforms FILM and TRF in generating articulated movements inbetween, but still struggles to create natural kinematic motions because of the limitation SVD itself failing generated complex kinematics (bottom row). Note that the input image serve conditioning to SVD, so generated first frame might differ from the input image if SVD struggles to create plausible videos from that input.
input frame
input frame 2
mid frame
Figure 6: Failure case. Our method fails to work well in the cases where input pairs have sparse correspondences.
DISCUSSIONS & LIMITATIONS
Our method is limited by the motion quality of the underlying base model, Stable Video Diffusion (SVD), as discussed in Sec. 5.4, Another limitation that SVD has strong motion priors derived from the input image, tending to generate only specific motions for given input. As result, the actual motion required to connect the input key frames may not be represented within SVD's motion space, making it challenging to synthesize plausible intermediate videos. However, with advancements in large scale image-to-video models like SoRAF) we are optimistic that these limita- tions can be addressed in the future. Including better motion datasets and incorporating articulated motion/physical movement priors may also help. Another potential improvement involves using motion heuristics between the input key frames to prompt the image-to-vice model to generate more accurate nn-between motions.
Acknowledgements We thank Meng-Li Shih, Bowei Chen, and Dor Verbin for helpful discussions and feedback. This work was supported by the UW Reality Lab and Google.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta. Micah Goldblum. Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 843- 852, 2023.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08//1 13,00/11111000011 2023.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint 00l:2401.12445. 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:231..15127, 2023a.
Andreas Blattmann, Robin Rombach, Huan I Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 22563-22575, 2023b.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Jiong Dong, Kaoru Ota, and Mianxiong Dong. Video frame interpolation: A comprehensive survey, ACM Transactions on Multimedia Computing, Communications and Applications, 19(2s):1-11, 2023.
Haiwen Feng. Zheng Ding, Zhihao Xia, Simon Niklaus, Victoria Abrevaya, Michael J Black, and Xuaner Zhang. Explorative inbetweening of time and space. arXiv preprint 14X.::2403.16111 2024.
songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yan Zhu, and Jia-Bin Huang. On the content bias fréchet video distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7277-7288, 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-video diffusion models. arXiv preprint arXiv:2311. 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 6851,
2020. 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 2rxiv:2210.022 023033 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-86466 2022b.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou Rife: Real-time intermediate flow estimation for video frame interpolation. arXiv preprint arXiv:201 1.06294, 2020.
Siddhant Jain, Daniel Watson, Erie Tabellion, Aleksander Holyáski, Ben Poole, and Janne Kontkanen. Video interpolation with diffusion models. arXiv preprint arXiv:2404.01200: 2024.
Huaizu Jiang, Deqing Sun. Varun Jampani, Ming -suan Yang, Erik Learned-Minn and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9000-9008, 2018
Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Tran. Flavr: Flow-agnostic video represen- tations for fast frame interpolation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2071 2082, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-hased generative models. Advances Neural Information Processing Systems, 35:26565-26577, 2022.
11

---

Published as a conference paper at ICLR 2025
Hyeongmin Lee, Taeoh Kim, Tae-young Chung, Dachyun Pak, Yuseok Ban, and Sangyoun Lee. Adacof: Adaptive collaboration of flows for video frame interpolation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5316-5325, 2020.
Yuseung Lee, Kunho Kim, Hyunjin Kim, and Minhyuk Sung. Syncdiffusion: Coherent montage via synchro- nized joint diffusions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arX:v:2202. 09778, 2022.
Yihao Liu, Liangbin Xie, Li Siyao, Wenxiu Sun, Yu Qiao, and Chao Dong. Enhanced quadratic video interpo- lation. Computer Vision-ECCV 2020 Workshops: Glasgow; UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 41-56. Springer, 2020.
Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5437-5446, 2020.
Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim. Bmbc: Bilateral motion estimation with bilateral cost volume for video interpolation. European Conference on Computer Vision, pp. 109-125. Springer, 2020.
Junheum Park, Chul Lee, and Chang-Su Kim. Asymmetric bilateral motion estimation for video frame inter- polation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14539-14548, 2021.
Jordi Pont-Tuset, Federico Perazzi. Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint urXiv:1704.0065,,200 2017.
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. In European Conference on Computer Vision, pp. 250-266. Springer, 2022.
Fitsum A Reda, Deging Sun, Aysegul Dundar, Mohammad Shoeybi, Guilin Liu, Kevin J Shih, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Unsupervised video interpolation using cycle consistency. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 892-900, 2019.
Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming-Hsuan Yang. Video frame interpolation trans- former. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 17482- 17491, 2022.
Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 14489-14498, 2021.
Jascha Sohl- Dicksiics,,,,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.0255 2020a.
arXiv preprint
Yang Song and Stefano Ermon. Generative modeling estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-basee generative modeling through stochastic differential equations. arXiv preprint 1rX1v:2011.13456, 2020h.
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mydiffusion: Enabling holis- tic multi-view image generation with correspondence-awaan diffusion. arXiv preprint arXiv:2307.01099, 2023.
Vikram Voleti. Alexia Martinee,,, and Chris Pal. Mevd-masked conditional video diffusion for pre- diction, generation, and interpolation. Advances in neural information processing prucessinn systems, 35:23371-23385, 2022.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi. Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-vides One-shot tuning of image diffusson models for text- to-video generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7623-7633, 2023.
12

---

Published as a conference paper at ICLR 2025
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang. Tien-Tsin Wong, and Ying Shan. Dy- namicrafter: Animating open-domaie images with video diffusion priors. arXiv preprint arXi::2310.12190 2023.
Xiangyu Xu, Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Hsuan Yang. Quadratic video interpolation. Advances in Neural Information Processing Systems, 32, 2019.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8850 8860, 2024.
Qinsheng Zhang, Jiaming Song, Xun Huang, Yongxin Chen, and Ming- Yu Liu, Diffcollage: Parallel generation of large content with diffusion models. arXiv preprint arXiv:2303. 17076, 2023.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 SENSITIVITY TO THE SCALE OF TRAINING SET
As stated in Sec. 4.4 our method fine-tunes fewer than 2% parameters of the original model by using the attention map from the pretrained model, and thus we reduce the need for extensive training data. We use 100 synthetic training videos in our experiments. Here we we conduct an ablation by varying the training dataset size to be 50 and 150 videos, and evaluate the performance as done in Tab. I Our method still outperforms the baselines even with a training size 50, and its performance increases slowly as more data is added (see Fig.
Training data keyframe interpolation performance
Pexels
FVD
34
450
32
30
400
28
375
26
350
325
24
300
22
100
140
100
140
Training Data Size
Training Data Size
Figure 7: Ablation on how the scales of the training dataset affect our model's performance.
Pexels
Davis
FID J
FVD |
FID |
FVD
FILM
25.16
371.83
41.85
1048.65
TRF
31.43
563.16
36.79
563.07
Ours (motion bucket id 255)
22.18
306.61
34.06
426.53
Ours (motion bucket id 65)
23.33
270.27
32.75
491.85
Ours (motion bucket 127)
22.99
806.84
32.68
424.69
Ablation how the conditioning parameters motion bucket id in Stable Video Diffusion affect our model's performance. Our method uses 127 as default in the paper.
Table 2:
A.2 THE INFLUENCE OF MOTION BUCKET ID IN STABLE VIDEO DIFFUSION
Stable Video Diffusior Blattmann al. 2023a) takes motion bucket íd as micro conditioning parameter in the video generation process, which is expected to affect the motion magnitude the generated video: higher values result more dynamic video and vice versa. However, keyframe interpolation is more constrained task where the second end frame provides additional guidance for the generation (Feng et al. 2024).. Here we experiment with different motion bucket id values in Tab. [ and our method still outperforms TRF and FILM. On the Pexels dataset, motion bucket id of 65 results in better FVD score (generated motion closer to the ground truth vidcos). However, the same value results in worse FVD score the Davis dataset. This discrepancy is likely due motion difference between the two datasets.
A.3 PSEUDO CODE FOR LIGHTWEIGHT BACKWARD MOTION FINE-TUNING AND DUAL DIRECTIONAL SAMPLING
'e use the public available model weights fin
https://hhugginggfcccc///tt lityai/
14

---

Published as a conference paper at ICLR 2025
def get_trainnale__araee (rev_UNet)
# Only finetune WV and temporal self attention layers rev_ zev_unet_train__arat for name, param in rew__umett_aameeeerrateeeees
if te oot.::/ss...aatttooo.....eeeee ttanppaa__trassffcere.l.oio wtw.:/ccc.....allttoooo.oaa. atttnl.too_outt weeight in rey_uuet_irainnparrmm....pp appp:peeaaaaaaaaapadddddo append
in name
(param) (parnn
param. rerriresrquiiss_grraa True
return rev_unst_train_patt
Backward motion fine-tuning
rev_UNet copy hopy.deepoppyyooo (ori
ori oNi_ requires_grad (Falssee)) (False) & pretrained 30UNet in SVD
rev www._uett.require__gtaea.. (False) # backward motion UNet to be fine-tuned opt omiimizer for epech in (0n nun_train_epochs) rev UNet.train()
(gt_i (get_trainaale e_pe params (rev_iNet)
for batch_vides in crain_dataloade
I_ batch_videol:, 0] # get the first frame from the video
I = batch_videol:: ;;, -1] & get the last frame from the video
muapute_image_coddiiionnn
c_N compuuee_iims__ccdddtttooonnn
batca_video rearrange
"b feolwwe -> (b E) a h w")
2 vae encode (batch_video)
rearrange (z_0, (b f) h -> b b w", f-num_frames)
z forch.flip
(1,s=(1,)) GT reverse latent video
noise torch. (zk
t torch. randint (0, num www__trainttmmmeeee
z_t = saise_scheduler add (zd_ noise, t)
rev torch.flip(2. dims=
noise_,, attenticn_man maps o (z_t, t, e_0)
8 rotate attention maps by 180 degree
rotated_attention_.ee. [torch. flip (attention_mapp dims- (-2, -1))
for attention map in attention [appatioon_map
map
pred_noise_2 cov_uNet(2_t_rew t, c_N, rotatesLattentioon_aaa.
pred_z_rev = noise ptw.heeddieer..eedee_deeeeeeeee sacddssapleeeeeeeeeeeeeeeeeeeeee loss += mse_less (pred_a_rev, z_rev) loss backward
optimizer.aten
aptimizer.zers (erd_grad()
return rev_UNe:
Figure 8: Pytorch pseudocode for lightweight backward motion fine-tuning.
15

---

Published as a conference paper at ICLR 2025
Dual ricetional diffusion sampl ing for generating in-between video ori Uiet: pretrained 3DUNet in sVD
rev UNet:
Fine-tuned backward motion UNet
c_0
(T_0)
C_N compute pepputee_immg___ondiiiioooo (I_N)
7_t torch. randn (laamt((aatet___hhaaaa) initialize video latent variable
timesteps scheduler..st__iimetee (num_steps)
for i, t in enumerateetthnestesss)
predicted noise forward motion start from T_0 gred_roiss__i attention_maps maps = ori_ (z_t, E, c_0)
if ffrr_fessiierr_fee_____iii ori ori_UNet (2_t, t, null_conditionie pred_noiss_uunenee
[red_noise_l quidance_scale prrd__oiis___ pred
+
[torch. flip (attention map, dims= (-2, -1)) aotated for attention_map in attention_maps]
# predicted noise in backward motion start from I_N 2_t_rev torch. flrch.flippz_t, dims=(1,)) pred_ = rev_UNet (2_t_rev, t, c_N, rotatedd if do_classi htr_r_frreee_ddneeeee
geed_noise_uncond_
rev_UNet (z_t_rev, t, aull_conditooning rotated_attention__nm
pred_noise__ pred poiseeei____oodd__ guuidancessaale scale (pred_noise__ pred_noisee_nccee
2 2 torch. (frch.
dims= (1,))
pred_ roise (pred_ pered_noise_1 pres_noise) /2. fuse
2_t scheduler update (z_t, pred_ t) denoise satput_video vae decode (2_t) return outdut_video
Figure 9: Pytorch pseudocode for dual directional diffusion sampling.
16


Reviews:
[{"review_id": "v53A4UBwOS", "reviewer": "Reviewer_Ytxq", "summary": "The paper presents Generative Inbetweening, a method for creating intermediate frames between two keyframes by adapting a pre-trained image-to-video diffusion model. This model adapts Stable Video Diffusion with dual-directional diffusion: generating video frames that interpolate both forwards and backwards in time. \nThis approach achieves motion-coherent inbetween frames through a technique that involves reversing the temporal self-attention maps within the U-Net model to generate backward motion from the endpoint keyframe, then combining this with forward-motion frames to produce smooth video sequences.\nEvaluations on the Davis and Pexels datasets show the method’s performance against the existing techniques, including TRF and FILM, in terms of frame coherence and motion fidelity for larger motion gaps.", "questions": "Given the model's limitations with non-rigid motions, did the authors explore any alternative solutions, such as enforcing additional temporal consistency constraints or incorporating motion priors for articulated objects?\n\nWhile the quantitative results are promising, did the authors consider conducting a user study to assess perceived motion realism, as subjective assessments might capture nuances that FID/FVD cannot?\n\nCould the authors elaborate on how sensitive the model's performance is to the choice of the 180-degree rotation in the self-attention map? Did they experiment with other configurations for reversing the temporal interaction?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Strengths\nThe paper’s fine-tuning approach makes effective use of a pre-trained model (SVD) to generate backward motion without requiring extensive additional data or full retraining. This demonstrates an efficient approach to model adaptation.\nBy developing forward-backward motion consistency through temporal self-attention, the method generates smooth and coherent transitions, especially in scenarios with long differences between keyframes. \nThe paper provides good experimental results, using both qualitative comparisons and metrics like FID and FVD to validate performance improvements over established baselines (FILM, TRF, etc.).\nAblations explore the impact of various components and the paper transparently discusses limitations, providing clarity on the model's boundaries, especially with non-rigid motion types.", "weaknesses": "While the paper includes comparisons with baseline models, it lacks an in-depth discussion on the unique metrics or benchmarks used to capture differences between models, particularly in subjective aspects like motion realism. Including a more detailed discussion on why certain metrics (e.g., FID or FVD) were selected over others could clarify the relevance of the performance gains.\n\nThe model relies heavily on SVD’s motion priors, which, as the authors note, can struggle with non-rigid or complex kinematic movements. \nWhile the paper acknowledges this, further discussion on how future models might address such limitations, possibly by incorporating other motion datasets or additional temporal constraints, would add depth to the future directions.\n\nAlthough the fine-tuning approach is a strength, it may be challenging for readers unfamiliar with diffusion models to follow the model adaptation process fully. More visual aids or pseudocode detailing the fine-tuning and dual-directional sampling steps would enhance clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Y3OvDlrj5", "reviewer": "Reviewer_aMmk", "summary": "The paper focuses on the keyframe interpolation problem, which has been overlooked in existing large-scale video generation models. The article proposes a solution to this task by treating keyframe interpolation as a forward video generation from the first frame and a backward generation from the last frame, followed by a coherent fusion of the generated frames. Based on this, the paper reuses existing large-scale image-to-video models to obtain a video generation model for backward motion by reversing temporal interactions. Additionally, it uses sampling techniques to blend paired frames generated by the forward and backward temporal directions with synchronized paths, producing intermediate frames.", "questions": "1. As noted in the weaknesses, I observed that in the cases provided, the camera and object movements between keyframes are slight. Can this method still perform well when there is a significant difference between the given keyframes? Additionally, when the keyframe difference is large, the backward generation may be unable to reuse the rotated attention matrix from the forward generation, potentially causing large discrepancies in frames generated at the same time step. In such cases, can fusion still work effectively?\n2. This generation pipeline seems to require a substantial number of corresponding points between keyframes. Beyond the issue of low overlap mentioned earlier, I’m also curious whether the method could still generate smooth transitions if, for example, one object in the keyframes—such as a fish in the ocean—undergoes a mirrored flip, meaning every point has a mapped counterpart but with an orientation change.\n3. The paper adopts simple averaging for intermediate frame fusion (line 281), but intuitively, frames generated closer to the initial keyframe might exhibit higher quality. Why not use weighted averaging instead? For example, linearly blending frames based on their proportional distance from each keyframe might yield smoother transitions and higher quality.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper fills a gap in the field of large-scale video generation, specifically keyframe interpolation, at a related lower cost. As summarized earlier, this paper presents a novel pipeline to generate synchronized frames and targeted frame fusion techniques to achieve smooth transitional videos.", "weaknesses": "The keyframes shown in the paper have relatively small motion ranges and require extensive pixel mapping; otherwise, obvious artifacts occur (as mentioned in the limitations), making this approach unsuitable for large-scale object or camera movements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fWzgc1hYDx", "reviewer": "Reviewer_VhZE", "summary": "The author proposes a novel method for distant keyframe interpolation, leveraging pretrained image-to-video diffusion models. This approach generates intermediate frames by predicting and fusing forward and backward video sequences, conditioned respectively on the given start and end frames. The author introduces a lightweight fine-tuning technique to tackle the key challenge of predicting backward video sequences from the end frame. Additionally, a dual-directional diffusion sampling strategy is employed to effectively fuse noise in both forward and backward directions.", "questions": "Please kindly refer to the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clear and easy to understand, with well-presented motivation and methodology.\n- The proposed method is novel, straightforward, and effective, demonstrating improvements over the selected baseline interpolation methods.", "weaknesses": "Further ablation studies on the proposed method could explore:\n1. **Training Dataset Scale**: In the paper, the model is fine-tuned with only 100 videos. It would be interesting to investigate how the scale of the training dataset affects the model’s performance.\n2. **Fine-tuning Modules**: The paper fine-tunes only the value and output projection matrices in the self-attention layers of the backward framework. Since there might be a gap for the forward motion in the context of the image-to-video task and the interpolation task, it would be worth exploring whether the interpolation performance could be further improved by fine-tuning both the forward and backward framework matrices while preserving the attention map rotation operation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wvwG4TO1ff", "reviewer": "Reviewer_MnYK", "summary": "This paper addresses the keyframe interpolation problem by leveraging the large-scale image-to-video diffusion model, Stable Video Diffusion, to generate frames between a pair of keyframes. \n\nUnlike traditional image-to-video models that generate frames in a forward-moving manner, this paper proposes finetuning the model for backward-moving videos and utilizing both the original and finetuned models together during inference. \n\nTo leverage the knowledge from the forward-moving model, only the value and output projection matrices of the 3D self-attention layers are trained, and the attention maps from the forward-moving videos are rotated by 180 degrees and inserted into the finetuned backward-moving model. \n\nDuring inference, the attention maps generated by the forward-moving model are rotated and applied to the finetuned backward-moving model, and the predictions from both models are fused. \n\nThis approach demonstrates superior performance over FILM and TRF on the Davis and Pexels datasets, despite being trained on only 100 videos.", "questions": "- There are three publicly available weights for Stable Video Diffusion (img2vid, img2vid-xt, img2vid-xt-1-1). Which of these weights did the authors use?\n\n- Stable Video Diffusion applies different classifier-free guidance scales to each frame. Did the authors use the same approach in this paper?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This method is both parameter-efficient and data-efficient, making it highly effective even with limited resources.\n\n- It leverages an open-source model, which enhances its accessibility and contributes to the broader video interpolation research community.\n\n- It demonstrates superior qualitative and quantitative performance compared to FILM, a well-known method for large motion interpolation, as well as TRF, which also uses Stable Video Diffusion.\n\n- In Section 5, the qualitative results are thoroughly explained, clearly highlighting the strengths of this approach in various aspects.", "weaknesses": "- Video interpolation performance can vary significantly based on FPS and the magnitude of motion, but the paper does not provide any analysis of these factors. Besides the motion bucket ID mentioned in the paper, Stable Video Diffusion also takes FPS as a condition. The paper would benefit from demonstrating whether the method still outperforms FILM and TRF when varying the motion bucket ID and FPS during finetuning and inference.\n\n- The proposed method requires both the base forward-moving model and the finetuned backward-moving model during both training and inference, making it more computationally intensive compared to a baseline of  \n fine-tuning on video interpolation dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)