MAVIN: Multi-Action Video Generation with Diffusion Models via Transition Video Infilling

## Abstract

Abstract Diffusion-based video generation has achieved significant progress, yet generating multiple actions that occur sequentially remains a formidable task.
Directly generating a video with sequential actions can be extremely challenging due to the scarcity of fine-grained action annotations and the difficulty in establishing temporal semantic correspondences and maintaining long-term consistency.
To tackle this, we propose an intuitive and straightforward solution: splicing multiple single-action video segments sequentially. The core challenge lies in generating smooth and natural transitions between these segments given the inherent complexity and variability of action transitions.
We introduce MAVIN (Multi-Action Video INfilling model), designed to generate transition videos that seamlessly connect two given videos, forming a cohesive integrated sequence. MAVIN incorporates several innovative techniques to address challenges in the transition video infilling task. Firstly, a consecutive noising strategy coupled with variable-length sampling is employed to handle large infilling gaps and varied generation lengths. Secondly, boundary frame guidance (BFG) is proposed to address the lack of semantic guidance during transition generation. Lastly, a Gaussian filter mixer (GFM) dynamically manages noise initialization during inference, mitigating train-test discrepancy while preserving generation flexibility. Additionally, we introduce a new metric, CLIP-RS (CLIP Relative Smoothness), to evaluate temporal coherence and smoothness, complementing traditional quality-based metrics. Experimental results on horse and tiger scenarios demonstrate MAVIN’s superior performance in generating smooth and coherent video transitions compared to existing methods. Codes will be available at https://github.com/18445864529/MAVIN .

## 1 Introduction

The evolution of video generation models has been significantly shaped by the advent of diffusion-based techniques, offering unprecedented fidelity and temporal coherence in video synthesis [, , , , ]. However, these models often struggle to generate videos that encompass multiple actions or adhere to complex instructions, and typically produce relatively short clips, limiting their use in scenarios requiring longer, multi-action sequences.

Generating multi-action videos directly presents numerous unresolved challenges. Firstly, the lack of fine-grained action-level annotations in existing large-scale video datasets hampers model training. Secondly, multi-action sequences, involving extended durations and significant motion ranges, challenge models to maintain spatiotemporal consistency throughout the video. The structural characteristics of video U-Net models further complicate complex temporal semantic correspondence modeling.
To circumvent these challenges, in this paper, we propose an innovative approach for generating multi-action videos by integrating several single-action video clips. This process entails two fundamental steps: first, the production of various video clips featuring the same subject engaging in distinct actions; second, the concatenation of these clips through action transitions. While the first step has been facilitated by recent advancements in text-conditioned image-to-video (TI2V) generation [, , , , , ], the second step remains understudied.
To this end, we introduce MAVIN (Multi-Action Video INfilling model), a transition model designed to infill an intermediate video clip between two adjoining clips, ensuring a fluid and seamless transition.

This task requires meticulous attention to overall motion consistency and smoothness. Therefore, MAVIN is trained with consistent conditioning on the reference videos. To manage the potential for substantial motion gaps and the requirement for flexible infilling lengths, we utilize a variable-length sampling strategy. The performance of MAVIN is further enhanced by boundary frame guidance (BFG) and a Gaussian filter mixer (GFM). BFG leverages high-level semantic features from the boundary frames of input videos to guide the video infilling process, ensuring visual coherence throughout the transition. Meanwhile, GFM dynamically manages the introduction and modulation of noise during inference, improving generation fidelity while maintaining flexibility. Our method is trained in a self-supervised manner, eliminating the need for finely annotated video-text transcriptions.

Moreover, existing metrics for evaluating video generation primarily focus on visual quality and often overlook temporal coherence, which is crucial for assessing action transitions. To address this gap, we introduce a new metric, CLIP-RS (CLIP Relative Smoothness), specifically designed to evaluate the temporal consistency and smoothness of transition videos. This metric complements traditional quality-based metrics and provides a comprehensive evaluation of our model’s performance.
Experimental results conducted on two distinct animal scenarios—horses and tigers—demonstrate our method’s superior performance in generating smooth and natural video transitions over existing methods, both in qualitative and quantitative assessments.

## 2 Related Work

Text-to-Video Generation. Text-to-Video (T2V) studies have shifted their focus from GAN-based models [, , , ] and auto-regressive models [, , , ] to diffusion models [, , , , , , , ], attributed to their superiority in generation quality, training stability, and condition flexibility. Foundation T2V models such as ModelScopeT2V [] and VideoCrafter [, ] are trained on large-scale captioned datasets, possessing rich motion priors and text-motion correspondences. Nevertheless, challenges persist in generating actions that fully adhere to complex text descriptions.

Image-to-Video Generation. Generating videos solely from text prompts leads to a high degree of randomness in the appearance of each generation, thereby limiting its range of applications. Image-to-Video (I2V) generations, on the other hand, animate a user-input image by leveraging the motion priors learned from video-only datasets [, , , ] and have demonstrated the ability to generate high-fidelity and aesthetically pleasing videos. However, they often exhibit limitations in the form of minor and uncontrollable motion patterns. Considering these issues, many studies have begun to focus on text-conditioned image-to-video generation (TI2V) synthesis, which involves generating videos from a reference image, coupled with a text prompt indicating how the image should be animated. Videos generated in this manner typically use the provided image as the initial frame [, , , , ] or retain its appearance identity and characteristics [, , ], while performing the motion described in the text. There has also been a stream of works that further specialize in motion controllability by integrating extra controlling signals [, , , ].

Generative Video Interpolation.
Diffusion models have also gained momentum in video interpolation, challenging traditional methods that rely on optical flow computation and frame blending techniques.
MCVD [] and RaMViD [] adopt diffusion-based models with random frame masking, making it capable of handling a range of video generative modeling tasks, including video prediction and interpolation.
LDMVFI [] claims to be the first effort solving video interpolation using latent diffusion models and has achieved superior perceptual quality compared to traditional models.
However, these works are primarily centered on standard video frame interpolation tasks, where the motions are less ambiguous and straightforward.
A concurrent work [] delves into large and challenging motions by interpolating 7 frames with an approximate stride of 3. It utilizes a cascaded framework where the interpolation occurs at a low-resolution pixel level and is subsequently upsampled with a super-resolution model. However, the absence of open-source availability of the model and data renders further evaluation under our task scenario unfeasible. SEINE [] explores diffusion-based scene transition, where the model can generate a smooth transition from the start image depicting one scene to the end image representing another.

## 3 Methodology

### 3.1 Preliminaries

Latent diffusion model (LDM) [] is a diffusion model (DM) [] variant that operates on the compressed latent space instead of the pixel space, and has exhibited its strong efficacy in image generation. LDM first encodes an input image sample $x_{0}$ into a clean latent code $z_{0}=\mathcal{E}(x_{0})$ using a VAE [, ] encoder $\mathcal{E}(\cdot)$. The latent code then undergoes a forward diffusion process, where it is incrementally perturbed with Gaussian noise following a Markov chain

$$ $q(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I),$ (1) $$

where $t\in\{1,\ldots,T\}$, and $T$ is the number of total forward diffusion steps. $\beta_{t}$ controls the noise strength at each step. By rewriting $\bar{\alpha}_{t}:=\prod_{i=1}^{t}(1-\beta_{i})$, this formula can be simplified as

$$ $z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\;\; \epsilon\sim\mathcal{N}(0,I).$ (2) $$

A U-Net [] model parameterized with $\theta$ works as a noise prediction function $\epsilon_{\theta}(\cdot)$ to predict the added noise $\epsilon$ given the time step $t$ and condition $c$ (e.g. text prompt). The training objective can be formulated as

$$ $\arg\min_{\theta}\mathbb{E}_{z_{0},\epsilon,t,c}\|\epsilon-\epsilon_{\theta}(z _{t},t,c)\|^{2}_{2}\ .$ (3) $$

Video latent diffusion model (VLDM) [, , , ] inflates the U-Net model into a 3D architecture by inserting temporal modules, making it capable of handling video data. Given an encoded video latent representation $z\in\mathbb{R}^{n\times h\times w\times c}$ where $n$ is the number of frames in the video; $h$ and $w$ denote the height and width of the latent code; and $c$ is the dimension of the latent space, the model performs spatial operations over the $h\times w$ space and temporal operations along the $n$ axis. The spatiotemporal structure empowers the model to manage spatial and temporal dependencies in a coordinated manner, facilitating the generation of coherent and high-quality video sequences.

### 3.2 Problem Formulation and Challenges

Problem formulation. The proposed transition video infilling task is a specialized form of video interpolation that deals with long ranges and large motions, with the input being two videos. The objective of this task is to generate a transition video given two videos, one preceding and one following, thereby seamlessly connecting the two. Given an encoded preceding video latent $z_{0}^{\mathcal{P}}=\{z_{0}^{0},\ldots,z_{0}^{s}\}$ and a following video latent $z_{0}^{\mathcal{F}}=\{z_{0}^{e},\ldots,z_{0}^{L-1}\}$, the model aims to generate an intermediate latent $z_{0}^{\mathcal{I}}=\{z_{0}^{s+1},\ldots,z_{0}^{e-1}\}$, where $s$ is the end frame index of the preceding video and $e$ is the start frame index of the following video. We term these two frames as the *boundary frames* for simplicity and clarity. $L$ is the length of the integrated video after infilling.

Challenges and remedies.
The novel nature of this task presents new challenges. In this section, we briefly outline the challenges and our solutions, with a detailed elaboration to follow in the next section.

The first challenge lies in temporal dependency modeling, which should support generating a transition video with potentially large motion gaps while maintaining motion consistency. Existing works [, ] typically adopt a BERT-like masking strategy for conditional modeling. However, such approaches are not effective for learning long-span motion patterns as prediction targets and references appear alternately on the temporal axis. To address this issue, we propose to consistently apply noise to *a consecutive subsequence* of training data. This method allows for natural conditioning on reference videos in temporal modules while creating large motion gaps in the middle, enforcing the capture of long-term temporal dependencies. Nevertheless, denoising only the middle part of training data can restrict data utilization and model robustness. To overcome this, we implement a variable-length sampling strategy to optimize data usage and simultaneously enhance the flexibility in generation length.

Furthermore, in text-conditioned generations, text prompts guide the generation direction in spatial modules, where each frame is processed independently, and the temporal modules align them, as vividly illustrated in []. However, in this task, the model operates in a self-supervised fashion and there is no text providing content or semantic guidance to the model. Without such guidance, spatial modules can generate incoherent images, placing extreme burdens on temporal modules to align them. We propose boundary frame guidance (BFG) in spatial modules to mitigate this issue.

Lastly, as revealed in [, ], a train-test noise initialization discrepancy hinders VLDM from generating high-quality videos. While previous solutions in I2V generation [, ] generally involve a shared noise strategy, it is not optimal for this task because the preserved condition frame signal throughout the generation sequence can limit motion range, discouraging synthesis of distinct transition states. To better serve the transition video infilling scenario, we propose a Gaussian filter mixer (GFM) module to balance initialization discrepancy and generation flexibility.

### 3.3 Model Architecture

Figure: Figure 1: Model architecture. The input sequence is divided into three clips using variable-length sampling. Noise is added exclusively to the latent of the intermediate clip, with length embedded as extra information. Boundary frames are encoded with a CLIP vision encoder as content guidance for spatial transformers. During inference, a Gaussian filter mixer (GFM) is used for noise initialization.
Refer to caption: x1.png

The overall model architecture is depicted in Figure 1.
In the model training stage, we simulate transition video infilling by dividing a training video into three segments. An entire video sample is first encoded into $z_{0}=\{z_{0}^{0},\ldots,z_{0}^{L-1}\}$, and *only the intermediate clip is corrupted* by $t$-step Gaussian noise according to Eq. 2 resulting in $z^{\mathcal{I}}_{t}$. The input to the U-Net model hence becomes $z_{t}=\{z^{\mathcal{P}}_{0},z^{\mathcal{I}}_{t},z^{\mathcal{F}}_{0}\}$, and the model is optimized to predict $\{\epsilon_{t}^{s+1},\ldots,\epsilon_{t}^{e-1}\}$ as per Eq. 3. Loss is computed only on noised frames. Since this approach does not involve extra mask or condition frame concatenation to the channel dimension, it enables the utilization of most pre-trained foundation models that accept 3-channel RGB inputs.

Variable-length sampling.
To improve data utilization and generation flexibility, we employ variable-length sampling by randomly shifting the start and end points of the infilling clip. Concretely, at each training step, we draw the boundary frame indices $s$ and $e$ randomly from two independent uniform distributions: $s\sim\mathcal{U}(a_{s},b_{s})$; $e\sim\mathcal{U}(a_{e},b_{e})$, where $0<a_{s}<b_{s}<a_{e}<b_{e}<L-1$. The resulting length of the noised clip $l:=e-s-1$ thereby follows a triangular distribution with the upper and lower limits $l_{{upper}}=b_{e}-a_{s}$, $l_{{lower}}=a_{e}-b_{s}$. Particularly, when $b_{s}-a_{s}=b_{e}-a_{e}$, the PDF of the distribution is symmetric and has the mode $l_{mode}=(l_{lower}+l_{upper})/2$.
To avoid confusion stemming from variable-length sampling, we equip the model with an awareness of the generation length it is handling. This is achieved by incorporating a length embedding using sinusoidal encoding followed by an MLP. The length embedding is subsequently added to the timestep encoding and collectively processed through another MLP into the spatial convolution module. This approach improves the model’s capacity to leverage training samples by accommodating predictions at varying positions and lengths. It effectively allows for generating at various lengths and accepting reference videos of diverse durations, thereby bolstering the model’s robustness and flexibility.

Dynamic boundary frame guidance.
Boundary frames play a pivotal role in guiding the model’s generation as they provide explicit information about the gap the model is tasked to bridge. Therefore, we propose boundary frame guidance (BFG) to compensate for the lack of guidance in transition video generation.
Most popular frame conditioning strategies entail extending the keys and values of spatial self-attention layers to include those of the condition frames [, , ]. However, empirical experiments did not prove these approaches effective for this task, and the low-level visual information sometimes restricted the freedom of generation, resulting in synthesized frames copying too much information from existing ones. Instead, we inject the guidance signal into the cross-attention layers via a higher-level CLIP representation. Concretely, we encode the pixel-level boundary frames $x_{0}^{s}$ and $x_{0}^{e}$ using a CLIP vision encoder and concatenate the representations along the sequence dimension. This integrated representation serves both as content and semantic guidance due to the nature of CLIP representations, informing the model about the generation direction.
A short text prompt briefly describing the subject, such as “horse movement”, can be optionally leveraged to help classify the action subject or extract knowledge from a pre-trained foundation model. The combined use of the CLIP encoders and the concatenation operation provides the model with a consistent understanding of the integrated condition signal.

Gaussian filter mixer for inference-time noise initialization.
We propose a dynamic inference-time noise mixing strategy tailored for the transition video infilling task. Since the infilling video functions as a bridge, its first few frames should thereby resemble the preceding video, while the last few frames approach the following video. The frames in the middle should be granted the flexibility to display transition states that are significantly distinct from any reference frames. Inspired by FreeInit [], we propose a Gaussian filter mixer (GFM) module that dynamically retains a certain amount of information from the closest boundary frame latent. This is accomplished by keeping the low-frequency component of the diffused boundary latent, which offers a rough layout guidance to the denoising process. The preserved information gradually diminishes as the frame position moves away from the boundaries, allowing for greater freedom in generation. It is then mixed with individual Gaussian noise at each frame, resulting in the mixed inference-time noise initialization $\tilde{z}_{t}^{n}$ at frame $n$ as

$$ $\displaystyle\mathcal{F}^{low}(n)$ $\displaystyle=\begin{dcases}\mathcal{FFT}_{3D}(z_{t}^{s})\odot\mathcal{G}(f_{S }(n),f_{T}(n))&\text{if}\;n\leq\frac{s+e}{2},\\ \mathcal{FFT}_{3D}(z_{t}^{e})\odot\mathcal{G}(f_{S}(n),f_{T}(n))&\text{if}\;n> \frac{s+e}{2},\end{dcases}$ (4) $\displaystyle\mathcal{F}^{high}(n)$ $\displaystyle=\mathcal{FFT}_{3D}(\epsilon_{t}^{n})\odot\left(1-\mathcal{G}(f_{ S}(n),f_{T}(n))\right),$ (5) $\displaystyle\tilde{z}_{t}^{n}$ $\displaystyle=GFM(n)=\mathcal{IFFT}_{3D}(\mathcal{F}^{low}(n)+\mathcal{F}^{ high}(n)),$ (6) $$

where $s$ and $e$ are the indices of the boundary frames; $\mathcal{FFT}_{3D}(\cdot)$ and $\mathcal{IFFT}_{3D}(\cdot)$ represent discrete fast Fourier transform and its inverse operation, performing in 3D dimensions;
$f_{S}(\cdot)$ and $f_{T}(\cdot)$ are functions that adjust the spatial and temporal stop frequencies, respectively; and $\mathcal{G}(\cdot,\cdot)$ is a 3D Gaussian low-pass filter taking both spatial and temporal stop frequencies as control parameters.

Eq. 4 first ensures that each intermediate frame refers to its closest boundary frame. Subsequently, the adjusting functions progressively reduce the stop frequency values as the distance to the selected boundary increases. We opt for a straightforward linear decreasing function with a scaling coefficient $\lambda$ to realize such control. The stop frequency for both $f_{S}(\cdot)$ and $f_{T}(\cdot)$ is computed as

$$ $f(n)=\max(0,f_{0}-\lambda\cdot\min(|n-s|,|n-e|)\cdot f_{0}),$ (7) $$

where $f_{0}$ is the initial stop frequency of the low-pass filter. Here, $f_{0}$ determines the maximum layout information we aim to retain from the boundary frames, and $\lambda$ regulates the rate at which such information decreases as the synthesis target moves away from the referred boundary.

## 4 Experiments

### 4.1 Experimental Setup

Table: Table 1: Quantitative comparison with other generative transition models.

Datasets. For our experiments, we focus on two distinct animal species to verify the effectiveness of the proposed method: horses and tigers. We use the AnimalKingdom dataset [] for training the horse model and the TigDog dataset [] for the tiger model. The AnimalKingdom dataset encompasses a diverse range of species and tasks, but we only utilize the videos labeled as “Horse” from the *action_recognition* task for training. However, we noticed that the video clips in *action_recognition* are generally too short and correspond to only single actions, resulting in insufficient action transition patterns for model training. Therefore, we supplemented the horse training data with additional long-take web videos that capture horse movements. Consequently, the total duration of the training data for each dataset is approximately 45 minutes under 30 FPS.

Testing clips for the transition video infilling task should ideally contain action transitions or large motions to effectively evaluate the model’s efficacy. Such data, however, is challenging to source from existing datasets, prompting us to construct our own. We collect videos from the Internet and generate the test data in two ways: manual cutting, which yields high-quality samples, and automatic generation, which produces a large number of test clips. We refer to the test sets generated in these ways as *test-manual* and *test-auto*, respectively. For *test-manual*, we meticulously cut web videos into 32-frame clips by ensuring the occurrence of significant movements or posture changes (e.g., transitioning from grazing to standing upright) in the intermediate clips. We curated 34 such test samples for each animal class. For *test-auto*, we employ an optical flow estimator, RAFT [], to estimate the motion intensity between the two reference clips. Concretely, we select video clips based on the average optical flow magnitudes of the boundary frames. Since small magnitude values suggest minor motions and excessively large values typically result from dramatic camera movements, only those with values falling within a certain range are leveraged. To formalize this, the boundary frame indices $s$ and $e$ for *test-auto* are selected using the following equation:

$$ $\left\{(s,e)\right\}=\left\{(s,e)\,|\,\frac{1}{h\cdot w}\sum_{i=1}^{h}\sum_{j= 1}^{w}\left\|\mathcal{E}_{flow}(x_{s},x_{e})_{i,j}\right\|_{2}\in\left(T_{ lower},T_{upper}\right),e\!-\!s\!-\!1\!=\!l_{test}\right\},$ (8) $$

where $T_{lower}$ and $T_{upper}$ are the lower and upper thresholds; $h$ and $w$ are the height and width of estimated optical flows; and $l_{test}$ is the length of the generation sequence we want to test. We tuned the thresholds and obtained 113 visually satisfactory test samples for the horse class and 104 for the tiger class by setting $l_{test}$ to 12.

Implementation details. We initialize our model from ModelScopeT2V-1.7b [] and fine-tune it with the proposed framework for 40K steps. The optimization is carried out using an AdamW optimizer [], with a constant learning rate of 5e-6 and a batch size of 1. Training videos are randomly sampled into 32-frame clips at a sample rate of 2, and pre-processed to eliminate potential shot transitions by excluding clips where any SSIM [] value between consecutive frames falls below 0.1. Videos shorter than 32 frames are discarded. Experiments are conducted at a resolution of 256$\times$256. Training and inference require around 40 and 12 GB vRAM, respectively. All training is performed on a single NVIDIA L40 GPU, with each trial taking approximately one day. The length range of random intermediate clips is set to $l_{lower}=2,l_{upper}=22$. The GFM parameters employed are $f_{0}=0.6,\lambda=0.1$.

Evaluation Metrics.
We present the following evaluation metrics: multi-scale structural similarity (MS-SSIM)[], peak signal-to-noise ratio (PSNR), LPIPS[], FVD [], and CLIP similarity []. However, most of these metrics primarily assess reconstruction quality and similarity between generated and ground-truth videos, considering each frame independently. They do not adequately account for temporal coherence, which reflects the smoothness of generated motions. To address this gap, we propose a CLIP-similarity-based inner-frame consistency measurement to quantify the *relative smoothness* with respect to the ground truth video clip. We term this measure CLIP Relative Smoothness score (CLIP-RS), computed as follows:

$$ $\text{CLIP-RS}=\frac{1}{L-1}\sum_{i=1}^{L-1}{\frac{\min(\text{CLIPSIM}(p_{i-1} ,p_{i}),\text{CLIPSIM}(q_{i-1},q_{i}))}{\max(\text{CLIPSIM}(p_{i-1},p_{i}), \text{CLIPSIM}(q_{i-1},q_{i}))}},$ (9) $$

where $p_{i}$ is the $i$-th generated frame and $q_{i}$ is the corresponding ground truth frame. $L$ is the length of the generated video. $\text{CLIPSIM}(p_{i},p_{j})$ denotes the cosine similarity between the CLIP representations of images $p_{i}$ and $p_{j}$. Each summation term quantifies the relative frame change in the generated video compared to the actual change in the ground truth. Either a relatively drastic or subtle change results in a low score. For example, if the oracle transition occurs at a steady rate while the synthesized video initially remains stationary and then abruptly changes to complete the transition, the differences in transition pace will be captured, leading to low relative smoothness values.

**Table 2: CLIP-RS responds to temporal changes and is not sensitive to visual aesthetics.**
|  | SSIM↑ | PSNR↑ | LPIPS↓ | CLIPSIM↑ | CLIP-RS↑ |
| --- | --- | --- | --- | --- | --- |
| Original (self-comparison) | 1.00 | inf | 0.00 | 1.00 | 1.00 |
| Decrease luminance by 50% | 0.71 | 13.6 | 0.21 | 0.66 | 0.97 |
| Increase contrast by 50% | 0.69 | 19.9 | 0.08 | 0.82 | 0.97 |
| Zeroing-out red channel | 0.67 | 11.3 | 0.29 | 0.66 | 0.96 |
| Zeroing-out red&green channels | 0.33 | 8.44 | 0.60 | 0.49 | 0.95 |
| Replicating 1^st frame as video | 0.79 | 20.7 | 0.06 | 0.77 | 0.87 |

CLIP-RS is a metric calculated along the frame axis, measuring the degree of changes between adjacent frames. Although it references the ground truth video, it does not engage in any direct frame-to-frame comparisons between the two videos. This characteristic renders this metric indifferent to the quality of the generated images or their resemblance to the original video.
We demonstrate this by manipulating a 12-frame video clip and computing the metrics with the original video. As shown in Table 2, when the video’s visual aesthetics are perturbed (rows 2-5), metrics based on predicted-actual similarity are significantly impacted despite the structural content and motion effect of the video remaining unchanged, whereas CLIP-RS maintains a score close to 1. In contrast, when the video’s temporal property is altered (the last row, where the new video is comprised of a 12-time repetition of the original video’s first frame), similarity-based and quality-based metrics yield superior results compared to when visual aesthetics were disturbed, while CLIP-RS can identify such smoothness discrepancies. This is in direct contrast to the use of absolute smoothness measurement [], where a static video can achieve a perfect smoothness score of 1, which contradicts our objective. The $\text{CLIPSIM}(\cdot,\cdot)$ function in CLIP-RS can also be substituted with other similarity measurements such as SSIM, averaged optical flow momentum, etc.

### 4.2 Results

Figure: Figure 2: Qualitative comparison of MAVIN with baseline models. The top two rows are input reference videos, with glowing frames marking the boundaries. MAVIN demonstrates smoother and more natural transitions and superior spatiotemporal consistency compared to baseline models.
Refer to caption: x2.png

Comparison with existing methods. For our comparative analysis, we selected two open-source diffusion-based generative models: DynamicCrafter [] and SEINE []. Both models are capable of generating transition videos from two condition images. However, to ensure a more fair and relevant comparison to our work, we also conducted experiments where these models were conditioned on video inputs. We refer to these modified versions as DynamicCrafter-Vid and SEINE-Vid.

We conducted experiments with two infilling length settings: (i) generating 8 frames given 12-frame condition clips on each side, and (ii) generating 12 frames given 10-frame references. The total input length is 32, matching our test set samples. Particularly, we found that DynamicCrafter, which was trained to generate fixed 16-frame videos, performed poorly when this length was altered. Therefore, for DynamicCrafter, we use a 4-frame reference on each side for 8-frame infilling, and 2 for 12-frame infilling, maintaining a total length of 16. For image-conditioned generations, where DynamicCrafter generates 14 frames (16 minus 2 reference images), we evenly sampled 8 and 12 frames from the 14 for metric computations. All metrics were calculated only on intermediate clips, except for FVD, which was compared with the entire input sequence.

We show quantitative results in Table 1 and qualitative results in Figure 2. MAVIN substantially outperforms other generative baseline methods, especially when the motion is difficult.
Specifically, SEINE is the most competitive transition generation model, but as the number of infilling frames increases, the gap between MAVIN and SEINE becomes obvious. *test-auto* is generated at a sample rate of 4, which is rather challenging. It is equivalent to bridging a 48-frame gap when infilling 12 frames on *test-auto*. The performance gap further increases under this setting, showing the effectiveness of the proposed method in infilling videos with large and complex motions.

As the only existing generative model trained for transition purposes, SEINE adopts a BERT-like masking strategy for masked modeling, where each frame is corrupted by chance independently, resulting in an intermittent corruption pattern. Although this method enhances data utilization and robustness, it falls short in generating long-term temporally cohesive videos because the corruption pattern allows the model to rely on nearby clean frames for predictions. In contrast, our method consistently applies continuous corruption up to a maximum length of 22 frames, compelling the model to capture long-term motion dependencies.

Ablation Study. We ablate the two key components of the proposed framework and present the qualitative results in Table 3. Results were obtained on *Horse test-manual* by predicting 12 frames under the same experimental setup. Boundary frame guidance (BFG) offers important content direction during model training, and the Gaussian filter mixer (GFM) helps stabilize the generation by providing essential information to address the discrepancy between training and inference phases. By ablating either BFG or GFM, the performance deteriorates across all metrics. When both components are removed together, the model experiences severe degradation, demonstrating the effectiveness and necessity of these components for high-quality video generation. (See supplementary materials)

Table: Table 3: Ablation study on boundary frame guidance (BFG) during training and Gaussian filter mixer (GFM) noise initialization during inference.

Figure: Figure 3: Application of the transition video infilling model. It connects multiple TI2V-generated single-action video clips into a cohesive extended video with smooth and natural action transitions.
Refer to caption: x3.png

### 4.3 Application for Multi-Action Generation

We achieve multi-action generations by connecting single-action videos with MAVIN. This work does not focus on optimizing the single-action models; instead, we employ existing TI2V models that animate an input image through text control, as discussed in Section 2.
In our empirical experiments, directly generating large motions or non-continuous actions using pre-trained TI2V models [, ] led to failure. Therefore, we fine-tune these models for the single-action generation purpose.
To integrate the synthesized single-action videos, we insert noise of the desired length between two videos and use MAVIN to infill a transition video. Alternatively, instead of inserting noise, we can concatenate single-action videos and replace the junction frames with noise to regenerate the transition parts.

We initialize the action model from AnimateAnything []. The training data for single actions is also derived from the AnimalKingdom and TigDog datasets, except that we collect additional data for training the action “horse jumping”. We train one model per animal species. A fixed action prompt, such as “horse is jumping”, is tied to each action and serves as the text condition during training.

To create a multi-action video, we first use a single image, controlled by multiple action prompts, to generate multiple single-action videos separately. These videos are then concatenated into a longer sequence, arranged in the desired order. We regenerate the junction frames using the infilling model for smooth action transitions. Figure 3 illustrates an example of such an application. In this example, we generate 20 frames for each single action and refine 12 frames centered around each junction, resulting in a 60-frame-long video that contains three actions: jump, stand, and run. The first, third, and fifth rows depict the single-action videos generated by the action model, while the second and fourth rows are generated by the infilling model to connect them. This approach generates highly temporally cohesive examples with great flexibility.

## 5 Conclusion

In conclusion, this study has presented a novel approach to generative video infilling, specifically targeting the generation of transition clips in multi-action sequences, leveraging the capabilities of diffusion models. Our model, MAVIN, demonstrates a significant improvement over existing methods by generating smoother and more natural transition videos across complex motion sequences. This research lays the groundwork for future advancements in unsupervised motion pre-training, large-motion video interpolation, and multi-action video generation.
While this technique enables new applications, it is also crucial to establish guidelines and implement safeguards to prevent its potential misuse in creating fake content, which raises ethical and security concerns.

Limitations.
Due to computational limitations and the proprietary nature of the widely used video training dataset WebVid-10M [], our experiments were conducted only under specific scenarios and initialized from existing foundation models. Further exploration of the task might require training at scale.
Moreover, while we did not concentrate on optimizing the single-action (TI2V) models, a notable trade-off between visual quality and motion intensity persists even after fine-tuning, highlighting an area for further research. The failure cases include the single-action model’s inability to follow the action prompt and the inconsistency in appearance in later frames for actions involving large motions.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: I expect the experimental results on other scenarios.
soundness: 2
strengths: This paper focuses on the transition video infilling with diffusion models, which is an important research topic that is relatively under-explored compared to the Text-to-Video or Video-to-Video setting. It is significant due to its potential applications in fields like digital art making. Generating a transition video with large motion gaps while maintaining motion consistency  is a challenge problem, for which this paper contributes interesting training pipeline and system design for offering guided information and stabilizing the inference. In particular, the use of GFM to balance initialization discrepancy and generation flexibility is technically sound and interesting. The generated results seem to be more smooth and natural than other methods by leveraging the proposed methods.
summary: This paper introduces MAVIN, a framework for the transition video infilling task, which aims to generate an intermediate video clip between two adjoining clips, ensuring a fluid and seamless transition. To enhance the performance of the video model, the authors propose several methods. First, they introduce a self-supervised training pipeline that allows transition video infilling without the need for text prompt guidance. To address large infilling gaps and accommodate varied generation lengths, they employ a consecutive noising strategy combined with variable-length sampling. Second, boundary frame guidance is utilized to mitigate issues caused by the lack of semantic guidance. Third, a Gaussian Filter Mixer is proposed to balance initialization discrepancies and generation flexibility. Additionally, they introduce a metric to evaluate temporal coherence and smoothness. This approach is significant for its potential in fields like digital art making, where smooth and natural transitions are crucial. Extensive qualitative experiments demonstrate the effectiveness of the proposed method. However, I have concerns regarding the experimental setup (see weaknesses section).
weaknesses: Lack of discussion and comparison with most relevant works on the  transition video infilling setting (e.g., [1], [2]).
Lack of ablation studies.
The experiment results on horses and tigers data set is not convinced, since it is too simple. I recommend the author to examine the proposed method on open-domain test set. I also expect the video result on digital art or cartoon scenarios.

[1] Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation, 2024.

[2]  Explorative inbetweening of time and space, 2024.

Overall, while I appreciate the important problem that this paper proposes to solve. However, the unsatisfactory performance could come from pre-mature experiments. I believe this paper can benefit from more thorough experiments and in-depth study.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: See the above weaknesses.
soundness: 3
strengths: 1. The motivation of proposed methods is clear introduced.
2. The paper writing is good and easy to follow.
summary: This paper proposes a diffusion-model based method to generate transition video frames for two given videos. It first designs a consecutive noising strategy coupled with variable-length sampling for handling large infilling gaps and varied generation lengths. To address the lack of semantic guidance during transition generation, it utilizes boundary frames as guidance. During inference, Gaussian filter mixer (GFM) dynamically manages noise initialization, mitigating train-test discrepancy while preserving generation flexibility. Finally, it introduces a new metric, CLIP-RS (CLIP Relative Smoothness), to evaluate temporal coherence and smoothness. Experiments are conducted on two animal datasets and demonstrate good results.
weaknesses: 1. It needs a clear definition for multiple actions in the Abstract and Introduction. And how existing method processes this scenario. 
2. In the related work, please state how different between the proposed method and existing works, which helps reader to decide the novelty.
3. Are the two datasets are multiple action videos?
4. More quality results are needed and it is better to show some failure cases.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1- MAVIN performs relatively better under the `test-auto` compared to the baselines. What could be the reason?
2- I don't have a clear understanding of how Eq.5 & 6 outputs are being used. Can you define each term and also explain more of where each is being used? Also, it would be beneficial if their usage be visualized in Fig.1.
3- Why not use a simple (like linear) interpolation between (z^s and z^e)? How much performance advantage is gained by the FFT and IFFT operations and doing the low pass filtering in the frequency domain?
4- Motion (and its smoothness) is very important in temporal coherence. How CLIP-RS address that? If the CLIP-RS can just measure the contextual coherence and not the motion, which other metric do you observe for that?
soundness: 3
strengths: The proposed method has enough novelty.
The authors touched on the problem of evaluation of the multi-action video generation.
summary: This video proposes a method for filling in between two videos to build multi-action videos, while most video generation models are focused on generating videos containing one atomic action. For this goal, a training paradigm has been proposed to ensure the smoothness and preserving context in the transition state of a video. Authors also proposed a new evaluation metric.
weaknesses: The main weakness of this manuscript is the small scope of its train/test data. Also, the few examples make it very hard to draw a conclusion.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors argue that the BERT-like training in Chen et al. is less effective due to the alternating appearance of target and reference frames. To address this, they propose applying noise to a consecutive subsequence of training data. However, this approach is relatively standard, as seen in methods like MAGVIT (Yu et al.). Providing additional background and discussion would help contextualize this choice and highlight its novelty.
soundness: 3
strengths: - Among the proposed components, the GFM module is an especially interesting idea.
- The authors specifically introduced the CLIP-RS metric to quantify relative smoothness, and Table 2 provides valuable insights on this metric.
- The videos in the supplementary materials and results in Table 1 demonstrate that MAVIN achieves superior results compared to prior work using the proposed benchmark.
- The ablation study effectively shows the impact of the proposed BFG and GFM components.
summary: The paper addresses the task of generating smooth and natural transitions between two video segments. A consecutive denoising strategy with variable-length sampling is adopted for infilling training. Boundary frame guidance is proposed to ensure semantically coherent frames, and the Gaussian filter mixer method is applied to balance the noise discrepancy between training and testing. A novel CLIP-relative smoothness metric is introduced to evaluate temporal coherence and smoothness. Experimental results on horse and tiger videos show that the proposed method achieves superior performance compared to SEINE and DynamiCrafter.
weaknesses: - The dataset used in the experiments is limited to tiger and horse videos, restricted domains with only 45 minutes of video available for training. Furthermore, the method requires fine-tuning an existing T2V model for 40k steps with a batch size of 1. This setup is limited in both scope and scale, making it insufficient to fully demonstrate the proposed method’s effectiveness. Larger-scale experiments across multiple domains would strengthen the evaluation.
- The compared models, SEINE and DynamiCrafter, are trained on open-domain data, enabling them to handle a broader range of scenarios. However, they are not specifically trained on animals, which makes the comparison to MAVIN (focused on animal videos) less direct. Training SEINE and DynamiCrafter on the proposed benchmark could enable a fairer comparison.
- The proposed components rely on standard approaches in video generation. For example: 1) the infilling objective in L201–L208, 2) variable-length sampling with a length embedding added to the timestep encoding, and 3) BFG uses CLIP embedding rather than a low-level embedding. These contributions, while effective, are not technically new.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What does "PDF" mean in line 215?

2. In line 427, the roles of BFG and GFM are discussed, but the data in Table 3 does not convincingly support this viewpoint. For instance, while BFG provides contextual guidance, removing BFG does not result in significant changes to the Clipscore; only when both are removed do substantial changes occur. Does this prove the combined effect of BFG and GFM?

3. There are many ways to transition between the two input videos; is it reasonable to use SSIM to evaluate the structural similarity of generated videos and GT frame by frame? For example, in Figure 2, it is reasonable for the tiger to rise in any frame, but using SSIM requires the tiger to rise at a specific frame.
soundness: 3
strengths: 1. The MAVIN model is proposed to generate transition videos between two independent video segments, and it is capable of performing this task.

2. BFG is introduced to guide the Diffusion process in generating transition videos, and GFM is proposed to smooth noise for better quality generation.

3. A new metric, CLIP-RS, is introduced to evaluate whether the smoothness of generated videos aligns with the original videos.

4. The paper presents clear illustrations, with Figure 1 effectively conveying the entire model's flow.

5. The authors provide code to enhance reproducibility.

6. The method has also been applied in extended contexts.
summary: This paper proposes a video prediction method that outputs intermediate transition frames given varying numbers of preceding and succeeding frames. Additionally, a CLIP-based evaluation method is introduced to verify inter-frame coherence.
weaknesses: 1. This task has some overlap with video prediction or frame interpolation, making it not particularly novel, and there are numerous models that have completed this work.

2. I believe the effectiveness of the proposed method lies in the use of Dynamic Boundary Frame guidance, which is fundamentally about injecting image features into cross-attention using CLIP—this technique has been widely applied in Image to Video generation.

3. Regarding GFM, many papers have discussed the influence of initialization noise on generation [1].

4. The testing data in this paper is somewhat limited; for example, it only uses two self-created datasets and does not test on publicly available datasets.

5. For the ICLR conference, this paper lacks explanatory depth.

[1] Wu T, Si C, Jiang Y, et al. Freeinit: Bridging initialization gap in video diffusion models[J]. arXiv preprint arXiv:2312.07537, 2023.

Meta-review: Despite the interesting problem context and the proposed MAVIN model, the reviewers' concerns about limited novelty, insufficient experimental validation and lack of clarity. Therefore the paper's contributions are considered insufficient for acceptance on ICLR, and a rejection decision is recommended. The reviewers encourage the authors to consider expanding their experimental evaluation, providing clearer explanations and analyses, and exploring more innovative approaches to further strengthen their work.

Decision: Reject