Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation

## Abstract

Abstract We present a method for generating video sequences with coherent motion between a pair of input key frames. We adapt a pretrained large-scale image-to-video diffusion model (originally trained to generate videos moving forward in time from a single input image) for key frame interpolation, i.e., to produce a video in between two input frames. We accomplish this adaptation through a lightweight fine-tuning technique that produces a version of the model that instead predicts videos moving backwards in time from a single input image. This model (along with the original forward-moving model) is subsequently used in a dual-directional diffusion sampling process that combines the overlapping model estimates starting from each of the two keyframes.
Our experiments shows that our method outperforms both existing diffusion-based methods and traditional frame interpolation techniques. Please see our project page at svd-keyframe-interpolation.github.io .

## 1 Introduction

Recent advances of large-scale text-to-video and image-to-video models [, , , , , ] have shown the ability to generate high resolution videos with dynamic motion. While these models can accept a variety of input conditioning signals, such as text captions or single images, most available models remain unsuitable for an obvious application: keyframe interpolation. Interpolating between a pair of keyframes—that is, producing a video that simulates coherent motion between two input frames, one defining the starting frame of the video, and one defining the ending frame—is certainly possible if a large-scale model has been trained to accept these particular two conditioning signals, but most open-source models have not. Despite the task’s similarity to existing conditioning signals, creating an interpolation model requires further training, and therefore both large amounts of data and substantial computational resources beyond what most researchers have access to.

Given the similarity between the input signals needed for keyframe interpolation (i.e., two-frame conditioning) and the input signals to existing models (e.g., one-frame conditioning), an interesting alternative solution is to instead *adapt* an existing pre-trained image-to-video model, without training a specialized model from scratch. In this paper, we propose an approach for enabling key frame interpolation by doing precisely this. Our approach is founded upon the observation that a keyframe interpolation model needs to know how to accomplish three objectives: (1) given a starting frame, it needs to predict coherent motion starting from that frame and advancing into the future, (2) given an ending frame, it needs to predict coherent motion starting from that frame and advancing backwards into the *past*, and (3) given these two predictions, produce a video that has a coherent combination of the two. Since existing image-to-video models can already accomplish the first of these three objectives, we focus our efforts on the the latter two, i.e., producing a single-frame conditioned model that can generate motion *backwards* in time, and a mechanism for combining forward and backward motion predictions into coherent videos.

One may imagine that producing such a single-image conditioned model that produces backwards motion should be trivial: simply pass an image into a regular image-to-video model, and reverse the output. Unfortunately, real-world motion is inherently asymmetric, and reversed motion into the future is notably different from motion into the past. As such, we first propose a novel, lightweight fine-tuning mechanism that reverses the arrow of time by rotating the temporal self-attention maps (i.e., reversing the temporal interactions) within the diffusion U-Net. This enables the reuse of the existing learned motion statistics in the pretrained model, and enables generalization while only requiring a small number of training videos.

Given both the original image-to-video model and this adapted reverse model, we also propose a sampling mechanism that merges the scores of both to produce a single consistent sample. These two sampling paths are synchronized through shared rotated temporal self-attention maps, ensuring they generate exactly opposite motions, an effect which we term “forward-backward motion consistency”. At each sampling step, their intermediate noise predictions are fused, resulting in a generated video with coherent motion that starts and ends with the provided frames.
We compare our work qualitatively and quantitatively to related methods in keyframe interpolation, and find that our method produces notably higher quality videos with more coherent dynamics given temporally distant key frames. Please see our webpage for a complete collection of video results: [svd-keyframe-interpolation.github.io](https://svd-keyframe-interpolation.github.io/).

## 2 Related Works

Frame interpolation Frame interpolation [] synthesizes intermediate images between a pair of input frames, and has been a long-standing research area in computer vision. Example applications include temporal up-sampling to increase refresh rate, create slow-motion videos, or interpolating between near-duplicate photos. Much of the research in this field [, , , , , ] employs flow-based methods, which estimate optical flow between the frames and then synthesize the middle images guided by the flow via either warping or splatting techniques. Traditionally, this task assumes unambiguous motion and the input frames are usually closely spaced ($\leq 1/30$s) samples in the video. Recent studies have begun to address large motions [, ], or quadratic motion [, ], though these still involve a single motion interpolation and cannot address distant input frames. In contrast, we aim to generate in-between frames that capture dynamic motions across distant input keyframes ($\geq 1$s apart), a challenge that goes beyond the capability of current frame interpolation technique.

Diffusion models for video interpolation
Diffusion models have shown remarkable capabilities for generative modeling of images  [, , , , , ] and videos [, , , ]. In earlier work, MCVD [] devises a general-purpose diffusion model for a range of video generative modeling tasks including video interpolation. More recent works  [, , ] explicitly train diffusion models to accept two end frames with conditioning to generate $7$ or $16$ intermediate frames at maximum resolution of $320\times 512$ at once, and achieved superior results in generating dynamic, ambiguous motions compared with traditional methods. In this work, we focus on enabling pre-trained large-scale image-to-video models to do key frame interpolation. Exposed to millions of videos, these models have demonstrated remarkable capabilities in generating high-resolution (up to $1080$p) and long (up to 4$s$) videos with rich motion priors.

Diffusion sampling for consistent generation In diffusion-based image generation tasks, novel joint diffusion sampling techniques [, , , ] for consistent generation are usually employed in generating arbitrary-sized images or panoramas from smaller pieces. These methods involve concurrently generating these multiple pieces and merging their intermediate results in the overlapping areas within the sampling process. For example, MultiDiffusion [] averages the diffusion model predictions to reconcile the different denoising processes. Recent work, TRF [] extends this joint generation approach to the bounded video generation taking two end frames as input. By running two parallel image-to-video generations guided by the start and end
frames, it merge their outputs by averaging in each denoising step.
However, a significant drawback of this method is that it cannot generate coherent motion in-between: simply fusing a forward video generation from the first end frame and the reversed forward video starting from the second end frame using a single image-to-video model designed for forward motion only causes the generated videos to oscillate between moving forward and then reversing, rather than continuously progress forward as our method does.

## 3 Background

Figure: Figure 1: Method overview. In the lightweight backward motion fine-tuning stage, an input video $\mathbf{x}=\{I_{0},I_{1},...,I_{N-1}\}$ is encoded into the latent space by $\mathcal{E}(\mathbf{x})$, and noise is added to create noisy latent $\mathbf{z}_{t}$; during inference, $\mathbf{z}_{t}$ is created by iterative denoising starting from $\mathbf{z}_{T}\sim\mathcal{N}(\mathbf{0,I})$. (1) Forward motion prediction: we first take the conditioning $\mathbf{c}_{0}$ of the first input image (inference stage) or the first frame in the video (training stage) $I_{0}$, along with the noisy latent $\mathbf{z}_{t}$ to feed into the pre-trained 3D UNet $f_{\theta}$ to get the noise predictions $\hat{\mathbf{v}}_{t,0}$, as well as the temporal self attention maps $\{A_{i}\}$. (2) Backward motion prediction: We reverse the noisy latent $\mathbf{z}_{t}$ along temporal axis to get $\mathbf{z}^{\prime}_{t}$. Then we take the conditioning $\mathbf{c}_{N-1}$ of the second input image, or the last frame in the video $I_{N-1}$, along with the 180-degree rotated temporal self-attention maps $\{A^{\prime}_{i}\}$, and feed them through the fine-tuned 3D UNet $f_{\theta^{\prime}}$ for backward motion prediction $\hat{\mathbf{v}}_{t,1}$. (3) Fuse and update: The predicted backward motion noise is reversed again to fuse with the forward motion noise to create consistent motion path. Note that only the value and output projection matrices $W_{\{v,o\}}$ in the temporal self-attention layers (green) are fine-tuned; see Fig. 2 for more details.
Refer to caption: x1.png

We introduce some background on Stable Video Diffusion [], the base image-to-video diffusion model used in our work, and then specifically explain the temporal self-attention layers within its architecture, which are key to modeling motion within the generated video.

### 3.1 Stable Video Diffusion

Diffusion models are trained to convert random noise into high-resolution images/videos via an iterative sampling process [, , , , , ]. This sampling process aims to reverse a fixed, time-dependent destructive process (forward process) that gradually corrupts data by adding Gaussian noise. In particular, Stable Video Diffusion (SVD) is a latent diffusion model where the diffusion process operates in the latent space of a pre-trained autoencoder with encoder $\mathcal{E}(\cdot)$ and decoder $\mathcal{D}(\cdot)$.

In the forward process, a video sample $\mathbf{x}=\{I_{0},I_{1},...,I_{N-1}\}$ composed of $N$ frames, is first encoded in the latent space $\mathbf{z}=\mathcal{E}(\mathbf{x})$, then the intermediate noisy video at time step $t$ is created as $\mathbf{z}_{t}=\alpha_{t}\mathbf{z}+\sigma_{t}\bm{\epsilon}$,
where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0,I})$ is Gaussian noise, and $\alpha_{t}$ and $\sigma_{t}$ define a fixed noise schedule. The denoising network $f_{\theta}$ receives this noisy video latent $\mathbf{z}_{t}$ and the conditioning $\mathbf{c}$ computed from the input image, i.e., the first frame $I_{0}$ in the video, and is trained by minimizing the loss:

$$ $\mathcal{L}(\theta)=\mathbb{E}_{t\sim U[1,T],\bm{\epsilon}\sim\mathcal{N}( \mathbf{0},\mathbf{I})}[\|f_{\theta}(\mathbf{z}_{t};t,\mathbf{c})-\mathbf{y}\| _{2}^{2}]$ $$

where the target vector $\mathbf{y}$ here is $\mathbf{v}=\alpha_{t}\bm{\epsilon}-\sigma_{t}\mathbf{z}_{t}$, referred to as v-prediction.

Once the denoising network is trained, starting from pure noise $\mathbf{z}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, the sampling process iteratively denoises the noisy latent by predicting the noise in the input and then applying an update step to remove a portion of the estimated noise from the noisy latent

$$ $\mathbf{z}_{t-1}=\text{update}(\mathbf{z}_{t},f_{\theta}(\mathbf{z}_{t};t, \mathbf{c});t)$ $$

until we get clean latent $\mathbf{z}_{0}$, followed by decoding $\mathcal{D}(\mathbf{z}_{0})$ to get the generated video. The exact implementation of the $\text{update}(\cdot,\cdot)$ function depends on the specifics of the sampling method; SVD uses EDM sampler [].

### 3.2 Temporal self-attention

The denoising network $f_{\theta}$ in SVD is a 3D UNet, composed of “down”, “mid”, and “up” blocks. Each block contains spatial layers interleaved with temporal layers, with the temporal self-attention layers responsible for modeling motion in the generated video. This layer takes a spatio-temporal tensor $X\in\mathbb{R}^{1\times N\times H\times W\times C}$ as input, where $N$ is the number of frames, and $C$ is the number of channels. Here we use batch size of $1$ for simplicity. The tensor is reshaped by moving the spatial dimensions $(H,W)$ into the batch dimension. This creates $X^{\prime}\in\mathbb{R}^{HW\times N\times C}$, where self-attention operates solely on the temporal axis. More specifically, $X^{\prime}$ is projected through three separate matrices $W_{q}$, $W_{k},W_{v}\in\mathbb{R}^{d\times C}$ ($d$ is the dimensionality of the projected space.), resulting in the corresponding query ($Q=W_{q}X^{\prime}$), key ($K=W_{k}X^{\prime}$) and value ($V=W_{v}X^{\prime}$) features. Then the scale-dot product attention is applied:

$$ $\text{Attention}(Q,K,V)=\text{softmax}(QK^{T}/{\sqrt{d}})V$ $$

The attention output is fed through another linear layer $W_{o}$ to get the final output. We refer to $A=QK^{T}\in\mathbb{R}^{HW\times N\times N}$ as the temporal self-attention map, which models the inter-frame correlations per spatial location. This temporal
attention mechanism allows each frame’s updated feature to gather information from other frames.

## 4 Method

Given a pair of key frames $I_{0}$ and $I_{N-1}$, our goal is to generate a video $\{I_{0},I_{1},I_{2},....,I_{N-1}\}$ that begins with frame $I_{0}$ and ends with frame $I_{N-1}$, leveraging the pre-trained image-to-video Stable Video Diffusion (SVD) model. The generated video should exhibit a natural and consistent motion path, such as a car traveling or a person walking in a steady direction.

Image-to-video models typically generate video with motions that run forward in time. It is primarily the temporal self-attention layers that learn this motion-time association. In Sec 4.1, we discuss how this forward motion can be reversed by rotating the temporal self-attention maps by 180 degrees. Then we introduce an efficient lightweight fine-tuning technique to reverse this association and enable SVD to generate backward motion videos from the input image in Sec. 4.2. Finally we present our dual-directional sampling approach that fuses the forward motion generation starting with frame $I_{0}$ and backward motion video generation starting with frame $I_{N-1}$ in a consistent manner in Sec. 4.3. An overview of our method is shown in Fig. 1.

### 4.1 Reverse motion-time association by self-attention map rotation

The temporal self-attention maps $\{A_{i}\}$ in the network $f_{\theta}$ feature the forward motion trajectory in video $\{I_{0},I_{1},...,I_{N-1}\}$. By rotating these attention maps by 180 degrees, we obtain a new set $\{A_{i}^{\prime}\}$ that depicts the opposite backward motion, corresponding to the reversed one $\{I_{N-1},I_{N-2},...,I_{0}\}$ starting from the last frame $I_{N-1}$.

Specifically, rotating the temporal self-attention maps by 180 degrees—flipping them vertically and horizontally—yields a backward motion opposite to the original forward motion. For example, consider attention map $A$; the rotated map $A^{\prime}_{N-j,N-k}=A_{j,k}$, where $A_{j,k}$ indicates the attention score between the j-th and k-th frames ($I_{j}$ and $I_{k}$). In the corresponding reversed video, the reverse frame indices $N-j$ and $N-k$ maintain the same relative response.

Figure: Figure 2: Temporal self-attention module in the backward motion generation. Given input tensor $X$, our attention mechanism additionally takes the respective attention map $A$ from the pre-trained SVD featuring forward motion, rotating it by 180 degrees to create a reverse motion-time association $A^{\prime}$. Note that $W_{\{v,o\}}$ are the only trainable parameters in this module.
Refer to caption: x2.png

### 4.2 Lightweight backward motion fine-tuning

We introduce a lightweight fine-tuning framework that specifically fine-tunes the value and output projection matrix $W_{v},W_{o}$ in the temporal self-attention layers, using the 180-degree rotated attention map from the forward video as additional input (see Fig. 2). We use $f_{\theta^{\prime}}(\mathbf{z}_{t};t,\mathbf{c},\{A_{i}^{\prime}\})$ to denote the backward motion generation network. This fine-tuning approach offers two key advantages.
First, by utilizing existing forward motion statistics from the pre-trained SVD model, fine-tuning $W_{\{v,o\}}$ simplifies the model’s task to focus on learning how to synthesize reasonable content when operating in reverse. This strategy requires significantly less data and fewer parameters compared to full model fine-tuning. Second, it enables the control for the model to generate a backward motion trajectory corresponding to the opposite of the forward trajectory described by the attention map. This feature is particularly beneficial when planning to merge forward and backward motions converging towards each other, and thus achieving forward-backward consistency.

The detailed training process is shown in Alg. 1. For latent video $\mathbf{z}\in\mathbb{R}^{1\times N\times C\times H\times W}$, we denote $\text{flip}(\mathbf{z})$ specifically by the second dimension, i.e., reversing the latent video along the time axis. In every training iteration, we sample an input video of $N$ frames, and random time step $t$, then the noisy video latent $\mathbf{z}_{t}$ is created by adding the noise in that time step. The noisy video latent along with the input conditioning $\mathbf{c}_{0}$ (computed from $I_{0}$) is fed into the pre-trained 3D UNet $f_{\theta}$ to extract the self attention maps $\{A_{i}\}$ from the temporal attention layers. Then we reverse the noisy video latent, along with the last frame conditioning $\mathbf{c}_{N-1}$, feed them into the backward motion 3D-UNet $f_{\theta^{\prime}}$. The loss function is computed by taking the predictions of the network and the ground truth reverse video.

Figure: ALGORITHM 1 Light-weight backward motion fine-tuning

### 4.3 Dual-directional sampling with forward-backward consistency

Our complete dual-directional sampling process is detailed in Alg. 2. Given a pair of key frames $I_{0}$ and $I_{N-1}$, their corresponding conditioning $\mathbf{c}_{0}$ and $\mathbf{c}_{N-1}$
are pre-computed. Then each sampling step (illustrated in Figure 1) works as follows:

(1) Forward motion denoising with $I_{0}$ as input: The noisy video latent $z_{t}$ along with the conditioning $\mathbf{c}_{0}$ is fed into the pre-trained 3D UNet $f_{\theta}$ in SVD to predict the noise volume $\hat{\mathbf{v}}_{t,0}$. Additionally, the temporal self-attention maps $\{A_{i}\}$ in the 3D UNet are extracted.

(2) Backward motion denoising with $I_{N-1}$ as input: The noisy video $\mathbf{z}_{t}$ is flipped along the temporal dimension to create the reverse video latent $\mathbf{z}^{\prime}_{t}$ corresponding to the backward motion. This backward video, along with the conditioning $\mathbf{c}_{N-1}$, as well as the 180-degree rotated attention maps $\{A_{i}^{\prime}\}$, are fed into our fine-tuned 3D UNet $f_{\theta^{\prime}}$. This step predict the noise volume $\hat{\mathbf{v}}_{t,1}$ representing a reverse motion from $I_{N-1}$.

(3) Finally, the predicted noise volumes from both forward and reverse motion paths are fused and then denoised using the $\text{update}(\cdot,\cdot)$ function to create less noisy video $\mathbf{z}_{t-1}$. In this way, we ensure forward-backward consistency and thus a consistent moving direction in the generated video. The $\text{fuse}(\cdot,\cdot)$ function performs a simple average. In practice, we also adopt per-step recurrence to enhance the fusion as seen in [, ], by re-injecting Gaussian noise into the update $\mathbf{z}_{t-1}$ and repeating the denoising $5$ times before continuing the sampling for the next step.

Figure: ALGORITHM 2 Dual-directional diffusion sampling

### 4.4 Implementation Details

As presented in Sec. 4.2, our lightweight fine-tuning does not rely on large collection of videos. So we collected $100$ high quality videos which are originally generated from SVD from a community website(^1^11https://www.stablevideo.com/) as our training data. Our experimental results show that our method generalizes well to the real image data. We select the ones with large object motion such as animal running, vehicle moving, people walking, and so on. We use the Adam optimizer with learning rate of $1e-4$, $\beta_{1}=0.9,\beta_{2}=0.999$, and weight decay of $1e-2$. The training takes around $15K$ iterations with batch size of $4$. We trained on $4$ A100 GPUs. For sampling, we apply $50$ sampling steps. For other parameters in SVD, we use the default values: $\emph{ motion bucket id}=127$, $\emph{ noise aug strength}=0.02$.

## 5 Experiments

Figure: Figure 3: Qualitative baseline comparisons. Leftmost ($i=0$) and rightmost columns ($i=24$): start and end frames. TRF generates back-and-forth motions, such as vehicles moving forward and then reversing. FILM struggles to find correspondences when the input frames are distant and morphs from the first frame to the last. The red arrow indicates the direction of motion. We recommend viewing the video results on the project webpage.
Refer to caption: extracted/5808897/figs/Results/GT/2053100-uhd_3840_2160_30fps/0001.png

In Figs. 3,  4,  5,  7, we demonstrate that our approach successfully generates high quality videos with consistent motion given key frames capturing a dynamic scene at different moments. We highly recommend viewing the video results on the project page to see the results more clearly.
Sec. 5.1 describes the data we used to evaluate our method and the baselines. Sec. 5.2 demonstrates how our method outperforms traditional frame interpolation method FILM, and the recent work TRF [] that also leverages SVD for video generation.
Sec. 5.3 justifies our design decisions with an
ablation study. Sec. 5.4 discusses the optimal scenarios where our method excels and sub-optimal ones where it outperforms the baselines but remains limited by SVD itself. Sec. 5.5 demonstrates the adaptability of our method by fixing the first key frame and varying the second.
Sec. 5.6 discusses our failure cases.

### 5.1 Evaluation Dataset

We use two high-resolution (1080p) datasets for evaluations: (1) The Davis [] dataset, where we create a total of $117$ input pairs from all of the videos. This dataset mostly features subject articulated motions, such as animal or human motions. (2) The Pexels dataset, where we collect a total of $45$ input key frame pairs from a compiled collection of high resolution videos on Pexels(^2^22https://www.pexels.com/), featuring directional dynamic scene motions such as vehicles moving, animals, or people running, surfing, wave movements, and time-lapse videos. All input pairs are at least $25$ frames apart and have the corresponding ground truth video clips.

### 5.2 Baseline Comparisons

We mainly compare our approach to FILM [], the current state-of-the-art frame interpolation method for large motion, and TRF [] which also adapts SVD for bounded generation.
We show representative qualitative results in Figs. 3, 5. In addition, we also include results for the keyframe interpolation feature from the recent work DynamiCrafter []—a large-scale image-to-video model.
The keyframing feature is modified from it and specially trained to accept two end frames as conditions, while we focus on how to *adapt* a pretrained image-to-video model in a lightweight way with small collection of training videos and much less computational resources. This feature generates $16$ frames at resolution $512\times 320$, while ours generates at resolution $1024\times 576$.

Quantitative evaluation For each dataset, we evaluate the generated in-between videos using FID [] and FVD [], widely used metrics for evaluating generative models. These two metrics measure the distance between the distributions of generated frames/videos and actual ones. The results are shown in Tab. 1, and our method outperforms all of the baselines by a significant margin.

**Table 1: Comparisons with baselines and our ablation variants. *Ours w/o RA*: full pipeline with fine-tuning all parameters $W_{\{q,k,v,o\}}$ without using the 180-degree rotated temporal attention map. *Ours w/o FT*: full pipeline with using rotated attention map only in the “up” blocks and without fine-tuning $W_{\{v,o\}}$ for backward motion.**
|  | Pexels | Davis |  |  |
| --- | --- | --- | --- | --- |
|  | FID $\downarrow$ | FVD $\downarrow$ | FID $\downarrow$ | FVD$\downarrow$ |
| FILM [] | 36.03 | 460.89 | 41.85 | 1048.65 |
| TRF [] | 41.90 | 788.07 | 36.79 | 563.07 |
| DynamiCrafter [] | 47.60 | 410.67 | 38.32 | 439.74 |
| Ours w/o RA | 39.11 | 644.47 | 36.70 | 549.98 |
| Ours w/o FT | 53.44 | 633.51 | 47.23 | 604.76 |
| Ours | 34.58 | 366.53 | 32.68 | 424.69 |

Comparison to FILM
The flow-based frame interpolation method FILM suffers from two problems. First, it struggles to find correspondences in scenes with large motions. For example, in the second row of Fig. 3, in a highway where vehicles moving in both directions, FILM fails to find the correspondence between the moving cars across the input keyframes, resulting in implausible intermediate motions. For example, some cars in the first frame disappear in the middle and reappear at the end. Second, it generates undesirable unambiguous motion which takes the shortest path between the end frames. In the example in Fig. 5, given two similar-looking frames that captures different states of a person running, FILM produces a motion that merely translates the person across the frames, losing the natural kinematic motions of the legs.

Comparison to TRF
TRF fuses the forward video generation starting from the first frame and the reversed forward video starting from the second frame, both using the original SVD. The reversed forward video from the second frame creates a backward motion video that ends at the second frame. Fusing these generation paths results in a back-and-forth motion in the generated videos. One notable effect we observe with TRF is that the generated videos exhibit a pattern of progressing forward first and then reversing to the end frame. For example, in the third row of Fig. 3, we can see the red truck moving backward over time; in the seventh row, the dog’s legs are moving backwards, leading to unnatural motions. In contrast, our approach fine-tunes SVD to generate a backward video starting from the second frame in the opposite direction to the forward video from the first frame.This forward-backward motion consistency leads to the generation of a motion-consistent video.

Figure: Figure 4: Ablation study. We evaluate other options for generating in-between motion consistency. (1) *Ours w/o RA*: full pipeline with fine-tuning all parameters $W_{\{q,k,v,o\}}$ in the temporal attention layers but without using 180-degree rotated temporal self-attention maps as extra input (top row). (2) *Ours w/o FT*: full pipeline without fine-tuning $W_{\{v,o\}}$ for backward motion (second row).
Refer to caption: extracted/5808897/figs/Ablations/full_finetuning/0000.png

### 5.3 Ablations

Figure: Figure 5: Our method outperforms FILM and TRF in generating articulated movements inbetween, but still struggles to create natural kinematic motions because of the limitation of SVD itself failing to generated complex kinematics (bottom row). Note that the input image serve as conditioning to SVD, so generated first frame might differ from the input image if SVD struggles to create plausible videos from that input.
Refer to caption: extracted/5808897/figs/Results/GT/3125907-uhd_3840_2160_25fps/0001.png

In Fig. 4 and Tab. 1, we show visual and quantitative comparisons to simpler versions of our method to evaluate the effect of the key components in our method.

Fine-tuning without rotated attention map (Ours w/o RA) We compare with a variant that fine-tunes all parameters in the temporal self-attention layers, namely, $W_{\{q,k,v,o\}}$, but without using the 180-degree rotated temporal self-attention map from the forward video as extra input. Though fine-tuning all parameters can generate backward motion from the second input image, there is no guarantee that the backward motion will mirror the forward motion from the first input image. This discrepancy makes it hard for the model to reconcile the two motion paths, often resulting in blending artifacts, as shown in the top row of Fig. 4. In contrast, fine-tuning $W_{\{v,o\}}$ with the rotated attention maps generates coherent and high-fidelity in-between videos. in forward-backward consistency.

Fine-tuning $W_{\{v,o\}}$ vs. no fine-tuning (Ours w/o FT)
In Sec. 4.1, we show that rotating the temporal attention maps by 180 degrees reverses the motion-time association, creating a backward motion trajectory. Here we show that fine-tuning the value and output projection matrices $W_{v,o}$ is necessary for the model to synthesize high-fidelity content given the input backward motion-time association. We run our full pipeline without any fine-tuning, and our attention map rotation operation is only applied to the “up” blocks in this variant. As shown in the second row of Fig. 4 and Tab. 1, without fine-tuning these parameters, the model can create consistent motion but suffers from poor frame quality due to the low frame quality of the backward video generation. For example, the person is disfigured in the generated video. Moreover, applying attention map rotation operation in the “down” and “mid” blocks in this variant worsens the visual fidelity even further.

### 5.4 Optimal and sub-optimal scenarios

Our method is limited by the motion quality and priors learned by SVD. Firstly, our empirical experiments indicate that SVD works well with generating rigid motions, but struggles with non-rigid, articulated movements. It has difficulty accurately rendering the limb movements of animal/people. In Fig. 5, though our method significantly improves upon FILM and TRF, it still appears unnatural compared to the ground truth movements. The bottom row, showing the sequence generated by SVD using only the first input frame, confirms that SVD itself struggles to generate natural running movements in between.

### 5.5 Adaptive In-between generation

Given the same first keyframe, and varying the second keyframe, our method can still generate plausible videos accordingly
due to the rich motion space of large-scale image-to-video models (see Fig. 7 ).

### 5.6 Failures

When the input pairs are captured at such distant intervals that they have sparse correspondences, as shown in Fig. 6, where only a small portion of cars appear in both input frames, it becomes difficult for our method to fuse the forward and backward motions. This situation, where the overlapping areas are minimal, leads to artifacts in the intermediate frames.

Figure: Figure 6: Failure case. Our method fails to work well in the cases where input pairs have sparse correspondences.
Refer to caption: extracted/5808897/figs/Results/Failures/frame1.png

Figure: Figure 7: Leftmost ($i=0$) and rightmost columns ($i=24$): key frames input. Middle columns: generated frames. By fixing the first key frame and varying the second, our method adapts to different ending poses of the dog, generating motion kinematics that naturally depict a walking dog.
Refer to caption: extracted/5808897/figs/Results/Variations/Ours/8746813-uhd_4096_2160_30fps/0000.png

## 6 Discussion & Limitations

Our method is limited by the motion quality in Stable Video Diffusion (SVD), as discussed in Sec. 5.4. Another limitation is that SVD has strong motion priors derived from the input image, tending to generate only specific motions for a given input. As a result, the actual motion required to connect the input key frames may not be represented within SVD’s motion space, making it challenging to synthesize plausible intermediate videos. However, with advancements in large scale image-to-video models like SoRA(^3^33https://openai.com/index/sora/), we are optimistic that these limitations can be addressed in the future. Another potential improvement involves using motion heuristics between the input key frames to prompt the image-to-video model to generate more accurate in-between motions.

Acknowledgements We thank Meng-Li Shih, Bowei Chen, and Dor Verbin for helpful discussions and feedback. This work was supported by the UW Reality Lab and Google.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: Given the model's limitations with non-rigid motions, did the authors explore any alternative solutions, such as enforcing additional temporal consistency constraints or incorporating motion priors for articulated objects?

While the quantitative results are promising, did the authors consider conducting a user study to assess perceived motion realism, as subjective assessments might capture nuances that FID/FVD cannot?

Could the authors elaborate on how sensitive the model's performance is to the choice of the 180-degree rotation in the self-attention map? Did they experiment with other configurations for reversing the temporal interaction?
soundness: 3
strengths: Strengths
The paper’s fine-tuning approach makes effective use of a pre-trained model (SVD) to generate backward motion without requiring extensive additional data or full retraining. This demonstrates an efficient approach to model adaptation.
By developing forward-backward motion consistency through temporal self-attention, the method generates smooth and coherent transitions, especially in scenarios with long differences between keyframes. 
The paper provides good experimental results, using both qualitative comparisons and metrics like FID and FVD to validate performance improvements over established baselines (FILM, TRF, etc.).
Ablations explore the impact of various components and the paper transparently discusses limitations, providing clarity on the model's boundaries, especially with non-rigid motion types.
summary: The paper presents Generative Inbetweening, a method for creating intermediate frames between two keyframes by adapting a pre-trained image-to-video diffusion model. This model adapts Stable Video Diffusion with dual-directional diffusion: generating video frames that interpolate both forwards and backwards in time. 
This approach achieves motion-coherent inbetween frames through a technique that involves reversing the temporal self-attention maps within the U-Net model to generate backward motion from the endpoint keyframe, then combining this with forward-motion frames to produce smooth video sequences.
Evaluations on the Davis and Pexels datasets show the method’s performance against the existing techniques, including TRF and FILM, in terms of frame coherence and motion fidelity for larger motion gaps.
weaknesses: While the paper includes comparisons with baseline models, it lacks an in-depth discussion on the unique metrics or benchmarks used to capture differences between models, particularly in subjective aspects like motion realism. Including a more detailed discussion on why certain metrics (e.g., FID or FVD) were selected over others could clarify the relevance of the performance gains.

The model relies heavily on SVD’s motion priors, which, as the authors note, can struggle with non-rigid or complex kinematic movements. 
While the paper acknowledges this, further discussion on how future models might address such limitations, possibly by incorporating other motion datasets or additional temporal constraints, would add depth to the future directions.

Although the fine-tuning approach is a strength, it may be challenging for readers unfamiliar with diffusion models to follow the model adaptation process fully. More visual aids or pseudocode detailing the fine-tuning and dual-directional sampling steps would enhance clarity.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. As noted in the weaknesses, I observed that in the cases provided, the camera and object movements between keyframes are slight. Can this method still perform well when there is a significant difference between the given keyframes? Additionally, when the keyframe difference is large, the backward generation may be unable to reuse the rotated attention matrix from the forward generation, potentially causing large discrepancies in frames generated at the same time step. In such cases, can fusion still work effectively?
2. This generation pipeline seems to require a substantial number of corresponding points between keyframes. Beyond the issue of low overlap mentioned earlier, I’m also curious whether the method could still generate smooth transitions if, for example, one object in the keyframes—such as a fish in the ocean—undergoes a mirrored flip, meaning every point has a mapped counterpart but with an orientation change.
3. The paper adopts simple averaging for intermediate frame fusion (line 281), but intuitively, frames generated closer to the initial keyframe might exhibit higher quality. Why not use weighted averaging instead? For example, linearly blending frames based on their proportional distance from each keyframe might yield smoother transitions and higher quality.
soundness: 4
strengths: This paper fills a gap in the field of large-scale video generation, specifically keyframe interpolation, at a related lower cost. As summarized earlier, this paper presents a novel pipeline to generate synchronized frames and targeted frame fusion techniques to achieve smooth transitional videos.
summary: The paper focuses on the keyframe interpolation problem, which has been overlooked in existing large-scale video generation models. The article proposes a solution to this task by treating keyframe interpolation as a forward video generation from the first frame and a backward generation from the last frame, followed by a coherent fusion of the generated frames. Based on this, the paper reuses existing large-scale image-to-video models to obtain a video generation model for backward motion by reversing temporal interactions. Additionally, it uses sampling techniques to blend paired frames generated by the forward and backward temporal directions with synchronized paths, producing intermediate frames.
weaknesses: The keyframes shown in the paper have relatively small motion ranges and require extensive pixel mapping; otherwise, obvious artifacts occur (as mentioned in the limitations), making this approach unsuitable for large-scale object or camera movements.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please kindly refer to the Weaknesses.
soundness: 3
strengths: - The paper is clear and easy to understand, with well-presented motivation and methodology.
- The proposed method is novel, straightforward, and effective, demonstrating improvements over the selected baseline interpolation methods.
summary: The author proposes a novel method for distant keyframe interpolation, leveraging pretrained image-to-video diffusion models. This approach generates intermediate frames by predicting and fusing forward and backward video sequences, conditioned respectively on the given start and end frames. The author introduces a lightweight fine-tuning technique to tackle the key challenge of predicting backward video sequences from the end frame. Additionally, a dual-directional diffusion sampling strategy is employed to effectively fuse noise in both forward and backward directions.
weaknesses: Further ablation studies on the proposed method could explore:
1. **Training Dataset Scale**: In the paper, the model is fine-tuned with only 100 videos. It would be interesting to investigate how the scale of the training dataset affects the model’s performance.
2. **Fine-tuning Modules**: The paper fine-tunes only the value and output projection matrices in the self-attention layers of the backward framework. Since there might be a gap for the forward motion in the context of the image-to-video task and the interpolation task, it would be worth exploring whether the interpolation performance could be further improved by fine-tuning both the forward and backward framework matrices while preserving the attention map rotation operation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - There are three publicly available weights for Stable Video Diffusion (img2vid, img2vid-xt, img2vid-xt-1-1). Which of these weights did the authors use?

- Stable Video Diffusion applies different classifier-free guidance scales to each frame. Did the authors use the same approach in this paper?
soundness: 3
strengths: - This method is both parameter-efficient and data-efficient, making it highly effective even with limited resources.

- It leverages an open-source model, which enhances its accessibility and contributes to the broader video interpolation research community.

- It demonstrates superior qualitative and quantitative performance compared to FILM, a well-known method for large motion interpolation, as well as TRF, which also uses Stable Video Diffusion.

- In Section 5, the qualitative results are thoroughly explained, clearly highlighting the strengths of this approach in various aspects.
summary: This paper addresses the keyframe interpolation problem by leveraging the large-scale image-to-video diffusion model, Stable Video Diffusion, to generate frames between a pair of keyframes. 

Unlike traditional image-to-video models that generate frames in a forward-moving manner, this paper proposes finetuning the model for backward-moving videos and utilizing both the original and finetuned models together during inference. 

To leverage the knowledge from the forward-moving model, only the value and output projection matrices of the 3D self-attention layers are trained, and the attention maps from the forward-moving videos are rotated by 180 degrees and inserted into the finetuned backward-moving model. 

During inference, the attention maps generated by the forward-moving model are rotated and applied to the finetuned backward-moving model, and the predictions from both models are fused. 

This approach demonstrates superior performance over FILM and TRF on the Davis and Pexels datasets, despite being trained on only 100 videos.
weaknesses: - Video interpolation performance can vary significantly based on FPS and the magnitude of motion, but the paper does not provide any analysis of these factors. Besides the motion bucket ID mentioned in the paper, Stable Video Diffusion also takes FPS as a condition. The paper would benefit from demonstrating whether the method still outperforms FILM and TRF when varying the motion bucket ID and FPS during finetuning and inference.

- The proposed method requires both the base forward-moving model and the finetuned backward-moving model during both training and inference, making it more computationally intensive compared to a baseline of  
 fine-tuning on video interpolation dataset.

Meta-review: The paper presents a method to in-between the frames of a video located between two quite distant keyframes provided as an input. The reviewers acknowledged superior results compared to several recent works. At the same time, the reviewers also listed a bunch of weaknesses, such as lack of in-depth analysis of metrics to capture motion realism, lack of support for articulated motions (which the authors attributed to the use of SVD, which is known to have troubles with complex motions) and others. There was only a shallow discussion between the authors and reviewers, in which, in AC's opinion, the authors addressed only a portion of concerns. Hence, the reviewers put the paper just marginally above the borderline.

Decision: Accept (Poster)