SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction
Authors: Xinyuan Chen
Yaohui Wang
Lingjun Zhang
Shaobin Zhuang
Xin Ma
Jiashuo Yu
Yali Wang
Dahua Lin
Yu Qiao
Ziwei Liu
Venue: ICLR 2024 poster (2024)
Keywords: generative model; video generation; diffusion model
Primary area: generative models

Abstract: Recently video generation has achieved substantial progress with realistic results. Nevertheless, existing AI-generated videos are usually very short clips ("shot-level'') depicting a single scene. To deliver a coherent long video ("story-level''), it is desirable to have creative transition and prediction effects across different clips. This paper presents a short-to-long video diffusion model, SEINE, that focuses on generative transition and prediction. The goal is to generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos. Specifically, we propose a random-mask video diffusion model to automatically generate transitions based on textual descriptions. By providing the images of different scenes as inputs, combined with text-based control, our model generates transition videos that ensure coherence and visual quality. Furthermore, the model can be readily extended to various tasks such as image-to-video animation and autoregressive video prediction. To conduct a comprehensive evaluation of this new generative task, we propose three assessing criteria for smooth and creative transition: temporal consistency, semantic similarity, and video-text semantic alignment. Extensive experiments validate the effectiveness of our approach over existing methods for generative transition and prediction, enabling the creation of story-level long videos.

Published as a conference paper at ICLR 2024
SEINE: SHORT-TO-LONG VIDEO DIFFUSION MODEL FOR GENERATIVE TRANSITION AND PREDICTION
Xinyuan Chen' Yaohui Wang Lingjun Zhang Shaobin Zhuang 1.
Xin Ma Jiashuo Yu! Yali Wang Dahua Lin Yu Qiao't Ziwei Liu
Shanghai Artificial Intelligence Laboratory. 2 East China Normal University
Shanghai Jiao Tong University, Dept of Data Science & AI, Monash University
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
6 S-Lab, Nanyang Technological University
ABSTRACT
Recently video generation has achieved substantial progress with realistic results. Nevertheless, existing Al-generated videos are usually very short clips ("shot- level") depicting single scene. To deliver coherent long video ("story-level"", it is desirable to have creative transition and prediction effects across different clips. This paper presents short-to-long (S2L) video diffusion model, SEINE, that focuses on generative transition and prediction. The goal is to generate high- quality long videos with smooth and creative transitions between scenes and vary- ing lengths of shot-level videos. Specifically, we propose random-mane video diffusion model automatically generate transitions based on textual descrip- tions. By providing the images of different scenes as inputs, combined with text- based control, our model generates transition videos that ensure coherence and visual quality. Furthermore, the model can be readily extended to various tasks such as image-to-video animation and auto-regressive video prediction. duct comprehensive evaluation of this new generative task, we propose three assessing criteria for smooth and creative transition: temporal consistency, seman- tie similarity, and video-text semantic alignment. Extensive experiments validate the effectiveness of our approach over existing methods for generative transition and prediction, enabling the creation of story-level long videos. Project page: https: //wp:ttetctttt...eeeoe. github. io/ SEINE- SEINE-Pro-eene project/.
con-
A Animation: a dog in spacesuit.
The panda is working in the office Transition: Spider-man and reading a paper
Transition:
Prediction: Iron Man flying in the sky.
comes a sand sculpture.
Figure 1: Generated samples. Our S2L model is able to synthesize high-definition transition
(shown in the left two columns) and prediction videos (shown in the right column) by giving textual descriptions. Best view with Acrobat Reader. Click the images to play the videos.
"Equal contribution. Correspondence. Work done interns Shanghai AT Laboratory.

---

Published as a conference paper at ICLR 2024
acare
soent 03
cOmte Rt
Gonerated short videos
Stoly Video
Figure 2: Short-to-long (S2L) pipeline. Our S2L video generation model enables the seamless connection of shots from different scenes with transitions and generates videos of varying lengths through autoregressive prediction, facilitating the creation of story-level videos. The red box repre- sents the transition video, while the blue box represents long video generated through prediction.
INTRODUCTION
Currently, with the success of diffusion model-based (Ho et al., 2020; Song et al., 2021a;b) text- to-image generation models (Ramesh et al., 2021; 2022; Saharia et al., 2022; Balaji et al., 2022; Rombach et al., 2022), a series of video generation works (Singer et al., 2023; Ho et al., 2022; Blattmann et al., 2023; Zhou et al., 2022; He et al., 2022; Wang et al., 2023b) have emerged and demonstrated impressive results. However, current video generation methods typically only yield "shot-level" video generation, which only consists of around few seconds and depicts single scene. Such shot-level short videos fall short of the demands for cinematic and film productions.
In cinematic industrial-level video productions, "story-level" long video is typically character- ized by the creation of distinct shots with different scenes. These individual shots of various lengths are interconnected through techniques like transitions and editing, providing a way for longer video and more intricate visual storytelling. The process of combining scenes or shots in film and video editing, known as transition, plays a crucial role in post- productto..... Traditional transition meth- ods, including dissolves, fades, and wipes, rely on predefined algorithms or established interfaces. However, these methods lack flexibility and are often limited in their capabilities. An alternative approach to achieve seamless and smooth transitions involves the use of diverse and imaginative shots to smoothly connect between scenes. This technique, commonly employed in films, cannot be directly generated using predefined programs. In this work. we present model address the less common problem of generating scamless and smooth transitions by focusing on generating interme- diate frames between
different scenes. We establish three criteria that the generated transition frames should meet: 1) semantic relevance to the given scene image; 2) coherence and smoothness within frames; and 3) consistency with the provided text.
In this work, we present a short-to-long (S2L) video diffusion model, SEINE, for generative tran- sition and prediction. The goal is generate high-quality long videos with smooth and creative transitions between scenes and varying lengths of shot-level videos. The pipeline is shown in Fig. We delve into the study of new task, "generative transition", which aims at producing diverse and creative transition video segments to connect two different scenes. Alongside, we propose an approach tackle this task and establish criteria for evaluating the efficacy of methods. We ap- proach the problem of transitions and final frames a given scene as inputs, we employ a text-based and video conditioning method to generate smooth transition video in between. This approach allows to effectively control the transition process and ensure the coherence and visual quality of the generated videos. To this end, design flexible random-mask diffusion model. The model is of capable generating transition shots by giving the first frame of one scene and the last frame of another scene, the scene transition is generated by given textual description. Our model can be extended to image-to-viee video animation
conditional video generation task. By providing the initial
2

---

Published as a conference paper at ICLR 2024
and autoregressive video prediction, enabling the generation of long-shot videos and dynamic video creation in film production.
We summarize our contributions as follows: 1) We propose new problem of generative transition and prediction, aiming at coherent "story-level" long video generation with smooth and creative scene transition and varying lengths of videos; 2) We present short-to-long video diffusion model, SEINE, focusing on generative transition and prediction; 3) we propose three assessing criteria for transition and extensive experiments demonstrate our method the superior performance on the metrics, as well as its ability to be applied across versatile applications.
2 RELATED WORKS
Text-to- Video Generation. In recent years, diffusion models (Ho et al., 2020; Song et al., 1021a;b) have significantly advanced the field of text-to-image (T2I) generation. Existing methods (Ramesh et 2021; He et. 2022; Ramesh et al., 2022; Saharia et al, 2022) have made remarkable progress in generating realistic images that are closely related to textual prompts. Recent advancements in T21 generation have led to the expansion of diffusion models to develop video generation from domain-specific models (Tulyakov et al., 2018; Wang et al., 2020a;b; Tian et al., 2021; Skorokhodov et al., 2022) general text-to-video (T2V) models. This extension involves adapting the conven- tional 2D UNet architecture into a spatial-temporal 3D network to establish temporal correlations between video frames. Make-A- Video (Singer al., 2023) and Imagen Video et (2022) lever- age text-to-image diffusion models of DALL-E2 (Ramesh et al., 2022) and Imagen (Saharia et al., 2022) respectively for large-scale T2V generation. PYoCo (Ge et al., 2023) presented a noise prior approach and utilized a pre-trained eDiff-I (Balaji et al., 2022) as initialization. An alternative ap- proach involves constructing a T2V model based on pre-trained Stable Diffusion and subsequently fine-tuning the model either entirely (Zhou et al., 2022; He et al., 2022) partially (Blattmann et al., 2023) on video data. In addition, LaVie (Wang et al., 2023b) fine-tuned the entire T2V model on both image and video datasets. While these methods demonstrate the potential of text-to-video synthesis, they are mostly limited to generating short videos.
Transition Generation. Scene transition is less common problem but is crucial in storytelling. serving as the bridge that connects different narrative moments. They facilitate the smooth pro- gression of story through shifts in time, place, or perspective. Several techniques are employed to achieve effective scene transitions, such as "fade", "dissolves", "wipes". "IRIS" or simply cut where a scene abruptly transitions to another. These transitions can be achieved using pre-defined algorithms with fixed patterns. Morphing (Wolberg, 1998; Shechtman et al., 2010) enables smooth transitions involving finding pixel-level similarities and estimating their transitional offsets. Existing generative models (Van Den Oord et al., 2017) can leverage linear interpolation at the latent code level to capture semantic similarities. This approach has been widely explored in various works, e.g.. style transfer (Chen et al., 2018), object transfiguration (Sauer et al., 2022; Kang et al., 2023).
Long Video Generation. Long video generation is a long-standing challenge in the field. Earlier works (Skorokhodov et al., 2022; Yu et al., 2022; Chen et al., 2020) employing generative adver- sarial networks (GANs) or variational auto-encoder (VAE) to model video distributions. VideoGPT (Yan et al., 2021) employs combination of VQVAE (Van Den Oord et al., 2017) and transformer architecture to sequentially produce tokens within a discrete latent space. TATS (Ge et al., 2022) trains time-agnostic VQGAN and subsequently learns a time-sensitive transformer based on the latent features. Phenaki (Villegas et al., 2023) uses a transformer-bassd model to compress videos into discrete tokens and generate longer continuous videos based on a sequence of text prompts. Recently, Diffusion Models (DMs) (Ho et al., 2020; Song et al., 202 la; Nichol & Dhariwal, 2021) have made significant advancements in video generation. LEO (Wang et al., 2023c) proposed to produce long human videos in latent motion space. LVDM (He et al., 2022) focuses on generating shot-level long videos by employing a hierarchical latent diffuusion model. NUWA-XL (Yin et al., 2023) employs a hierarchical diffusion model to generate long videos, but it has only been trained on the cartoon FDDntstonesHD dataset due to the lack of datasets on natural videos. On the other hand, Gen-L-Vided (Wang al., 2023a) generates long videos by utilizing existing short text-to-video generation models and treating them as overlapping short video clips. In contrast, our approach views long videos as compositions of various scenes and shot-level videos of different lengths. Our
3

---

Published as a conference paper at ICLR 2024
Temporal Coherence
Scene 2: 8g
Prompt: spiderman becomes sand sculpture.
Video-text alingment
Similarity
Figure 3: Notations and schematic representation of generative transition objective. Generative transition exhibits semantic similarity between each frame I" with the two scene images; each frame x² and its neighboring frames x=~1 and x*+11 should be temporal coherence; the semantics of generated frames should be alignment to the provided textual description.
short-to- tonttto--ong model aims to generate "story-level" videos containing continuous multi-scene videos with varying shot lengths.
METHODOLOGY
3.1 PRELIMINARIES
Text-to-Image Diffusion Model. Stable Diffusion (Rombach et al., 2022) performs image diffusion in a compressed latent space an autoencoder (i.e., VQ-GAN and VQ-VAB During training, given image sample zo is encoded into the latent code Exxo) and corrupted by Gaussian noise from a pre-defined Markov chain:
(( -N(zi 1,
for = 1. ..... T. with T being the number of steps in the forward diffusion process. The sequence of hyperparameters B, determines the noise strength at each step. The above iterative process can be reformulated in a dosed-form manner as follows:
0, + v1 - ōge, ~ 1),
(2)
where ar = - az; 02 co is minimized by:
3r The model adopts e-prediction and DDPM to learn a function
(3)
where e indicates the condition of the textual description, is drawn from diagonal Gaussian dis- tribution. The Eg is commonly implemented by Unet, where is parameterized neural network.
Text-to- Video Diffusion Model. Our framework is constructed upon pre-trained diffusion-based T2V model, LaVie (Wang et al., 2023b), which is a cascaded framework that includes a base T2V model. a temporal interpolation model, and video super-resolution model. LaVibbaase, which is part of the Vie framework, is developed using Stable Diffusion vve-trained model and incorporates temporal attention module and mage-video joint fine-tuning. To benefit from its ability generate visually and temporally consistent videos with diverse content, we utilize the pre-trained LaVis LaVie-base model the initialization for our framework.
3.2 GENERATIVE TRANSITION
The objective of the transition task is to generate sequence of intermediate frames, denoted as r", between two given images S₁ and S2, using a transition description caption c. We define z as Si and N S2, the sequence consists of N - 2 target (in-between) frames. The generated sequence should satisfy the following three properties, as shown in Fig. 3: 1) Temporal Coherence: The changes in visual content across the sequence should be smooth and coherent over time. This property ensures that the generated video appears natural and visually consistent. 2) Semantic Similarity: The semantic content of each generated intermediate frame should resemble either of
4

---

Published as a conference paper at ICLR 2024
Training proceess
concat
Encoder
Diffusion
E0
Random Mask
mig
Dociabe
Inference process
Transition nevlls
Easie
Prodecsion
Figure 4: Overview of our proposed method. We present S2L generation model for generating transition video and long video prediction. Our model supports text as input, which processed through clip encoder and injected into the model (blue part). For brevity, we omit the details of the text injection, which is identical to the stable diffusion model.
the source images to some extent. This prevents the appearance of the sequence from deviating too much from the source scene images. Additionally, the similarity should gradually change from one source image to the other. 3) Video-text Alignment: The generated frames should align with the provided text description. This alignment allows us control the transition between the two source images using the text description.
3.3 SHORT-TO-LONG VIDEO DIFFUSION MODEL
We develop a diffusion- based model for S2L videos via transition generation and video prediction. In order to generate unseen frames of transition and prediction, based visible conditional images or videos, our S2L video diffusion model incorporates a random mask module, as shown in Fig. Given the video dataset Priden- we take N-frame original video denoted xo R Xx3xxx where zo follows the distribution Pvideo- The frames of video To are first encoded by the pre-trained variational auto-encoder simultaneously as 20 E Raxxxxx" where c, h and w indicate the channel, height, and width of the latent code, represents the frame number. To achieve more controllable transition videos and leverage the capability of short text-to-vide generation, our model also takes a textual description as an input. The goal of our model is to learn a dual condition diffusion model: pe po(2olc, āg). the training stage, the latent code video is corrupted as Eq. 2t za v1- a₁e, where the initial noise (N0 7) whose size is same zn: E Bnxexxxw To capture an intermediate representation of the motion between frames, we introduce a random- mask condition layer at the input stage. This layer applies binary mask R" broadcasting to m E gnxxxxxx as the size of the latent code, resulting in a masked latent code:
20 ① m, E P Baxexxxx
(4)
The binary masks, represented by m. serve as mechanism to selectively preserve or suppress information from the original latent code. In order to identify which frames are masked and which ones are visible conditional images, our model takes the masked latent code Eo concatenated with the mask m as conditional input:
Σ = [R m. ēo], & E (3xx))))
(5)
where ž, represents the final input to the U-Net. The model is trained to predict the noise of the entire corrupted latent code:
- - t.
(6)
This involves learning the underlying distribution of the noise that affects the unmasked frames and textual description. By modeling and predicting the noise, the model aims to generate realistic and visually coherent transition frames that seamlessly blend the visible frames with the unmasked frames. The mask has n elements indicating the index of frames, each of which can take on the values 0 1. The probability of the element taking the value is denoted p. Bernouli(p). Given that our model primarily focuses on transitions and predictions, where typically less than frames are involved. In our model, set p 0.15 considering the probability 2 p < This
5

---

Published as a conference paper at ICLR 2024
Morphing
Morphing
VQVAF-based Methad
POVAE-boved Methou
SD-bared Method
Doseed Method
Oury
Onrs
Figure 5: Qualitative comparisons with existing methods. Left: Spiderman becomes into sand sculpture. Right: A cat from sitting on the coach transfer lying on the sand.
choice of probability ensures that, on average, approximately two or three out every n frames will be visible. It aligns with the assumption that the mask is primarily used for capturing transitions between frames, where adjacent frames are more likely to influenced by the mask. Our random- mask based model is capable of generating frames for any given frames at arbitrary positions within the sequence. Transition can be obtained by providing the first and last frames of sequence and utilizing prompts control the transition style and content, resulting in intermediate transition frames that depict the transition or progression within the video sequence. Long video involves recursive using the last few frames of the generated video as input and utilizing masks to predict the subsequent frames. By recursively iterating this process, the model can generate longer video sequences. Image-to-video animation can be achieved by treating the reference image as the first frame. The first binary mask is set as ones and the remaining is set as zeros. The animated video can be obtained through a denoising process and subsequently decoded from the encoder.
EXPERIMENTS
Implementation Details. SEINE is initialized from LaVie-base (Wang al., 2023b), with the addition of a random mask layer being randomly initialized. We first utilize the Web Vid dataset (Bain al., 2021) as the main training set, and during the later stages of training, we use few internal watermark-free datasets to mitigate the presence of watermarks in the generated output. Our model trained on videos of 320 x 512 resolution with 16 frames. During inference, SEINE is able to generate videos of arbitrary aspect ratios. Comparison Methods. Transition video gen- eration a relatively understudied and novel problem within the field. While traditional methods have explored smooth transitions through image morphing techniques, there is lack of specific approaches addressing generative transition in the context of video generation. In this part, we con- duct analyses between our proposed approach and existing methods that achieve similar effects: diffusion model-bel based method (i.e., Stable Diffusion). VQGAN- model (Esser et al., 2021)). Details description and implementation for the compared methods described in App. A.1.

---

Published as a conference paper at ICLR 2024
4.1 QUALITATIVE COMPARISON
Fig. 5 shows the visual comparison with the existing methods. We observe that using interpolation methods for VQVAE to generate intermediate transition scenes often results transparent blend- ing of two images. Morphing techniques, on the other hand, can achieve object deformation by matching key points (such as changes cat's pose), ensuring that objects do not undergo transpar- ent blending. However, for scenes without significant object deformation, such as transitioning from a cityscape to the sky in a Spiderman scene, simple blending may still occur. Stable Diffusion-base methods do not suffer from simple blending issues, but their lack of sufficient continuity in hidden codes leads to a lack of consistency in the generated intermediate images. In contrast, our method ensures smooth object motion while incorporating plausible phenomena during transitions.
4.2 QUANTITATIVE COMPARISON
We assess the generative transition from three aspects: temporal coherence, semantic similarity, and video-text alignment. To evaluate the semantic similarity between generated frames and given scene images, we use the clip similarity (CLIPSIM) metric. To compute "CI.IPSMM-xcenes abbr. "CLIPSIM-s", we calculate the clip image similarity for each frame and the given scene images. To compute "CLIPSIM-frames" abbr. "CLIPSIM--", we calculate the clip score in between generated frames. Likewise, we utilize the clip text-image similarity "CLIPSIM-1exI" abbr. "CLIPSIM-"", to quantify the semantic correlation between the generated videos and their corresponding descrip- tive text. In experiments, employ the MSR-VTT dataset due to its open-domain nature with textual annotations. We randomly selected one caption per video from the official test set (a total of 2,990 videos). In order evaluate the quality of the generative transition, we considered the given scene images as two frames of a video with a larger interval timestep. This allows us to analyze and compare the smoothness and coherence of the transition between the two images. Specifically, we utilize the first and 64th frames as the reference scene images in our evaluation. Tab, presents the comparative results of our analysis. The results demonstrate that our method outperforms the com- parison algorithms. It is worth noting that our method does not achieve significantly higher scores than CLIPSIM-Scee Scene and tions, we calculate FVD and tramewwise FID score. The outcomes reveal that our model consistently delivers superior video quality compared to other methods examined in our study.
To further scrutinize the quality our video transi-
Table 1: Comparison with methods w.r.t. CLIPSIM-1, CLIPSIM-S, CLIPSIM-J, FVD and FID.
Methods
CLIPSIM-r CLIPSIM-s() CLIPSIM- (1)
FVD
FID(L)
Morphing
0,2535
0.7707
0.9569
573.9
35.3
VQGAN-basee Transition
0.2528
0.7389
0,9542
445.6
36.7
SD-based Transition
0.2665
0.6080
0,8809
502.0
62.0
Ours
0.2726
0.7740
0.9675
181.6
13.4
4.3 DIVERSE AND CONTROLLABLE TRANSITION GENERATION
We showcase the advantages of our method in handling generative transitions: 1) diverse results by sampling multiple times; and 2) controllable results of our method by employing textural descrip- tions such camera movement control. Diverse Transition Generation. We present visual results demonstrating the variety of transitions generated under identical prompts within two equivalent scenes, as depicted in Fig. 6. We can observe that as the raccoon transitions from being in the sea to the stage scene, the first two outcomes depict a form of similarity transition. However, the sequence of transitioning between the objects and backgrounds varies. The final transition, indeed, resembles the unfurling of a curtain, akin to wave sweeping in from the right side. Controllable Transition Generation. Our method can achieve transitions by controlling the prompt with camera movements. For instance, by employing camera zoom-in and zoom-out control, we create visual occlusions, and during the zoom-out phase, we generate the subsequent scene to achieve a seamless scene transition. This occlusion technique is widely employed transition method in filmmaking. Fig. 7 illustrates our example of this technique through the prompt, transitioning a scene from an office to a library with the presence of a panda.
7

---

Published as a conference paper at ICLR 2024
Figure 6: Diverse transition results. Each row illustrates frames of our transition results by "the scene transition from the sea into playing trumpet raccoon".
1s prompt: A panda working in the office, camera zoom-in.
2d prompt: panda working the library, camera zoom-out.
Figure 7: Text-controllable transition. Examples our scene transition are achieved by control- ling them through textual descriptions, i.c., namera-motion control.
4.4 COMPARISON WITH STATE-OF-THE-ART METHOD FOR VIDEO PREDICTION
We conducted a comprehensive comparative analysis between our proposed method and the state-of- the-art method TATS for the task of generating long videos consisting of 1024 frames, on the UCF- 101 dataset (Soomro et al., 2012). Qualitative and quantitative comparisons are illustrated in Fig. 8. Fig. 8 (a) visually presents the comparative results, showing that TATS tends to degrade over time, leading to the emergence of grid-like artifacts and visual collapse. In contrast, our approach avoids such collapsing phenomena, maintaining stable and visually coherent video generation throughout the sequence. Fig. 8 (b) depicts the quantitative evaluation by FVD. In our study, we compared TATS with the outcomes obtained from model's zero-shot generation and the results achieved after fine-tuning our model on the UCF-101 dataset. All models were evaluated under the auto- regressvee setting. Our results generated by the DDIM sampling of 100 steps. Our approach demonstrates slower degradation of video quality over time compared to TATS. In the first few frames, due to the disparity of data distribution between the training and UCF-101, our model's scores higher than TATS. However, as TATS degrades over time, our model's FVD score becomes lower than that of TATS. Moreover, by performing fine-tuning on the UCF-101 dataset, our method consistently outperforms TATS.
4.5 MORE APPLICATIONS
Long Video Generation. Fig. 9 demonstrates the generation of long videos through auto-regressive prediction. Each video consists of tens of frames by auto-regressive prediction for 4 times. Notably,
8

---

Published as a conference paper at ICLR 2024
FVD
TATS
FUCUDID
Oues
105
(3) Quiliative Resuli
Quantintive
Figure 8: Comparison for auto-regressive long video prediction a) Qualitative Results. Each frame is selected with frame step 128 from the 256th frame; b) Qualitative results. The lower value indicates better quality.
A raccoon dressed a suit playing teddy washing the dishex. the trumpet, stage background.
Two pandas discussing an acu- demic paper.
Figure 9: Long video results via auto-regressive prediction. Best view with Acrobat Reader. Click the images to play the videos.
Ours
Gen-2
Ours
Gen-2
Figure 10: Image animation results. In each case, our result is the left and the result of Gen-2 is on the right. Best view with Acrobat Reader. Click the images to play the videos.
our method ensures that the visual quality of the generated content remains pristine while preserving consistent semantic coherence throughout the synthesized results. Image-to- Videeo Animation. We demonstrate the outcomes of video generation by utilizing an image as the initial frame. This approach can be seamlessly integrated with other T2I models, such Midjourney'. Here, compare our results with those obtained from the online API image-to-vides generation model, Gen-2 (Esser et al., 2023). From the comparison, it can be observed that despite not specifically training our model for this task, our model still achieves comparable results.
CONCLUSION
In this paper, introduce SEINE, S2L video diffusion model that focuses smooth and creative transitions and auto-regressive predictions. SEINE utilizes a random-mask video diffusion model automatically generate transitions based on textual descriptions, ensuring coherence and visual quality. The model can also be extended to image-to-vides animation and auto-regressive video prediction tasks, allowing for the creation of coherent "story-level" long videos. The evaluation of transitions is conducted based on temporal consistency, semantic similarity, and video-text seman- tic alignment. The experimental results demonstrate the superiority of our approach over existing methods in terms of transition and prediction.
hhttps: /www.www.. djourney.com

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
We acknowledge the ethical concerns that are shared with other text-to-image and text-to-video diffusion models (Ramesh al., 2021; 2022; Rombach al., 2022; Saharia al., 2022; Ho et al,, 2022; He et al., 2022; Wang et al., 2023b). Furthermore, our optimization is based on diffusion model (Ho et al., 2020; Song et al., 2021b), which has the potential to introduce unintended bias as result of the training data. On the other hand, our approach presents short-to-long video diffusion model that aims create a coherent story-level long video, which may have implications for various domains including filmmaking, video games, and artistic creation.
ACKNOWLEDGEMENT
This work is supported by the National Key R&D Program of China under Grand NO. 20.20222D01601000, the National Natural Science Foundation of China under Grant No. 62102150, the Ministry of Education Singapore under its MOE AcRF Tier 2 (MOE-T2EP20221- 0012) and NTU NAP, and the Science and Technology Commission of Shanghai Municipality under Grant No. 23QDI HOD114000 4008000, No. 2251 105800, and the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. ICCV, 2021.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: -mage diffusion models with an ensemble of expert denoisers. arXiv preprint 2rXiv:2211.01322 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint ar3.v:2300. 08465, 2023.
Xinyuan Chen, Chang Xu, Xiaokang Yang, Li Song, and Dacheng Tao. Gated-gan: Adversarial gated networks for multi-collection style transfer. IEEE Transactions on Image Processing, 28 (2):546-560, 2018.
Xinyuan Chen, Chang Xu, Xiaokang Yang, and Dacheng Tao. Long-term video prediction via criticization and retrospection. IEEE Transactions on Image Processing, 29:7090-7103, 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.030/1, 2023.
songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In ECCV, 2022.
Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305. 10474, 2023.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. 2022.
10

---

Published as a conference paper at ICLR 2024
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, $3:6840-68511 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 2X.i::220000200000 2022.
MinGuk Kang, Joonghyuk Shin, and Jaesik Park. StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICMI, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICMI,, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-36494, 2022.
Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large di- verse datasets. In Munkhtsetseg Nandigjav, Niloy Mitra, and Aaron Hertzmann (eds.), SIG- GRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Con- ference, Vancouver, BC, Canada, August 11, 2022, pp. 49:1 49:10. ACM, 2022. doi: 10.1145//52823333537773
https: //o..ii../// wttp///.....///32333..... 35337733
Eli Shechtman, Alex Rav-Acha, Michal Irani, and Steve Seitz. Regenerative morphing. In IEEE Conference Computer VIsion and Pattern Recognition (CVPR), San-Francisco, CA, June 2010.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without text-video data. ICLR, 2023.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. CVPR, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffuusion implicit models. In Interna- tional Conference on Learning Representations, 2021a. URL httpps://ppeeeeeeeeeeee .ee//
Forum?id=Stlqiaaee
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLRs, 2021b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A Dataset of 101 Human Action Classes From Videos in The Wild. Technical report, CRCV-TR-12-01, November 2012.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator what you need for high-resolution video synthesis. In ICLR, 2021.
Sergey Tulyakov. Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In CVPR, 2018.

---

Published as a conference paper at ICLR 2024
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to tome translation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. NeurIPS, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2023.
Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. 2023a.
Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. G3AN: Disentangling appearance and motion for video generation. In CVPR, 2020a.
Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. Imaginator: Conditional spatio-temporal gan for video generation. In WACV, 2020b.
Yaohui Wang. Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Yu Jiashuo, Yang Peiqing, Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen. Change Loy Chen, Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.5 1353777 2023b.
Yaohui Wang, Xin Ma, Xinyuan Chen, Antitza Dantcheva, Bo Dai, and Yu Qiao. Leo: Generative latent image animator for human video synthesis. arXiv preprint (0t::2335503999, 2023c.
George Wolberg. Image morphing: a survey. The visual computer, 14(8-9):360-37,, 1998.
Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Vidcogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2 10157, 2021.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li. Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303. 12346, 2023.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In ICLR, 2022.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
COMPARED METHODS FOR TRANSITION.
Here we describe the details and implementation details for the compared methods achieving tran- sition.
Morphing. Morphing is special effect in motion pictures and animations that changes (or morphs) one image or shape into another through a seamless transition. Traditionally such a depiction would be achieved through dissolving techniques film. The transition is achieved by using the first and the last frames and generating the intermediate frames.
SD-based Method. We adopt the SD as our baseline model. Since SD cannot directly generate intermediate frames, first compute the latent code of the given two images reverse diffusion process (i.e., DDIM inversion (Song et al., 2021a)), then we use the interpolation of Euclidean distance of the latent codes to denoising and obtain the intermediate frames.
VQGAN-based Method. VQGAN serves the latent code layer for SD. We employ the inter- polated representation of the codes from the encoder of VQGAN output (prior to quantization) as a reference method.
A.2 FAILURE CASES
Watermarks, Since part of our training data (WebVid10M) contains watermarks, there is chance that generated videos may also exhibit such signals (see Fig. 11 and 12). In the prediction task, we observed that in the autoregressive process, watermarks gradually emerge. Once they appear, they persist throughout the entire video. In the transition task, if the watermarks are present in the given images, the generated intermediate frames will also be affected. However, if watermarks appear in the given images, watermarks are less likely to appear, and even if they do, they are usually faint.
There is table by window with sunlight streaming through illumi- nating a pile of books.
view of lake autumn.
panda playing on swing set.
Figure 11: Failure cases for prediction with watermarks. Best view with Acrobat Reader. Click the images to play the videos.
Abrupt Transition. While our method excels transition generation, it does have limitations regarding the requirement for the similarity of the source and target scenes. A natural and smooth transition relies on finding meaningful correspondences between the two scenes. If the scenes are dissimilar lack common elements, the transition may appear abrupt or visually inconsistent. An alternative solution to address this limitation is through multi-step approach. We utilize image editing methods (Tumanyan et al., 2023; Cao al., 2023: Zhang & Agrawala, 2023) transform the source image in the scenes and actions into the target image. This method is particularly beneficial scene generation as operates the image level, naturally preserving similarities such as the structure, background, and objects of the image. As result, the transition effects produced using this approach often appear highly natural.
Text-video Unalignment. In our S2L model, aim to optimize po(zole, ēo) which io represents the latent code of the unmasked condition image. However, our training dataset, videos 20. unmasked frames og. and captions/prompts (c) not completely independent and exhibit strong correlations. Consequently, when there is no inherent connection between the given unmasked im- age and caption, it can lead discontinuities or difficulties in aligning with video-text alignment.
13

---

Published as a conference paper at ICLR 2024
A panda in a hamboo forest, The scene has changed from the sea The scene has changed from the shifting from sketch-like to realistic into playing trumper raccoon. panda watching television the style.
panda working in the office.
Figure 12: Failure cases for transition with watermarks. Best view with Acrobat Reader. Click the images to play the videos.
Additionally, we have observed that the quality of generated videos would be influenced to some extent by the given prompt. Providing prompts that align well with the realistic scenario and have a resemblance to unmasked videos can yield better results. Ensuring the provision more suit- able prompts in the context of generating long videos is topic we will explore in our subsequent research.
Scenes
Transition Video 1
Transition Video
Transition Video 3
Transition samples. each case, transitions are generated by the same prompt with different random seeds. Best view with Acrobat Reader. Click the images to play videos.
Figure 13:
14

---

Published as a conference paper at ICLR 2024
A.2.1 IMAGE-TO- VIDEO ANIMATION
We present additional outcomes of image-to-video animation, as depicted in Fig. 14. For each instance, the reference image is displayed on the left, while the corresponding animation video is showcased on the right. Notably, our model demonstrates the ability to accommodate images of various sizes. The figure showcases the animation results obtained from both square and rectangular images. The reference images the first three rows are generated from Midjourney. The reference images in the last two rows are generated from Stable Diffusion XI..
(a) First Frame is generated from Midjourney
(b) First Frame is generated from Stable Diffusion XL.
Figure 14: Image-to-vided video animation results, Best view with Acrobat Reader. Click the images to play the transition videos.
15


Reviews:
[{"review_id": "Uik1TO8KOa", "reviewer": "Reviewer_GChA", "summary": "The paper presents the SEINE model, for generating \"story-level\" long videos from short clips. It introduces a unique problem in generative transition and prediction. Using a random-mask video diffusion based on textual descriptions, the model shows smooth transitions between scenes. To evaluate its efficacy, the authors provide three new criteria: temporal consistency, semantic similarity, and video-text alignment. Results show its potential for generating coherent long videos.", "questions": "How long does the inference take? Is it capable of handling transitions with multiple objects across more than two scenes?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method of using masks was proposed in [1] and [2], but as far as I know, this is the first time it has been used in video transition. It could be novel.\n\n- The proposed method shows better performance on the metric compared to the baseline.\n\n- The proposed method can be applied in various areas such as long video generation and image-to-video animation.\n\n\nReferences\n\n[1] Voleti, Vikram, Alexia Jolicoeur-Martineau, and Chris Pal. \"MCVD-masked conditional video diffusion for prediction, generation, and interpolation.\" Advances in Neural Information Processing Systems 35 (2022): 23371-23385.\n\n[2] Fu, Tsu-Jui, et al. \"Tell me what happened: Unifying text-guided video completion via multimodal masked video generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "weaknesses": "- [Major] My main concern is that there is not enough quantitative evaluation of video transitions. This paper conducted quantitative experiments by randomly selecting one caption from MSRVTT and determining CLIP-TEXT. However, since video transitions occur when scenes change, it does not seem appropriate to evaluate video semantic correlation. Also, no video quality evaluation metrics (such as FVD etc.) have been considered. This makes it difficult to quantify the exact quality of generation.\n\n- [Major] Several details related to the human evaluation are missing. (such as number of frames in the generated video, the dataset used, and the questions posed in the user study.)\nWas the user study appropriately reflective of temporal coherence, text-video alignment, and semantic similarity?\n\n- [Minor] For transitions to be applied in the real-world, it would require generating more than 16 frames. Would the quality be maintained if more frames are generated?\n\n- [Minor] In Figure 5 related to video transition, the frame numbers and details are omitted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T4GH6cFaQc", "reviewer": "Reviewer_Tv5a", "summary": "In this paper, the author focus on a new task, \"generative transition\", which aims at smooth and creative transitions between scenes. Specifically, this paper proposes SEINE, a short-to-long video diffusion model with random masks to generate transitions frames based on textual prompts that describe transitions. Given a few unmasked frames, the proposed random-mask based diffusion model is able to generate frames at arbitrary positions. Therefore, their model can be used for tasks including generative transition, long video generation, and image-to-video animation by giving unmasked frames at different positions. \n\nIn the experiments, the authors compare their model with other baselines including morphing, VQGAN-based transition, and SD-based transition. Quantitative and qualitative results show that SEINE has better transition temporal coherence, semantic similarity across frames, and better video-text alignment.", "questions": "- For controllable transition generation, do we give the first and last frames unmasked to the modl for each prompt? If this is the case, I'm wondering maybe the model can also generate smooth zoom-in/out transitions without explicitly adding \"camera zoom-in/out\" in the prompt. It would be nice to provide ablation study that removes \"camera zoom-in/out\" in the prompts and see if the generation quality deteriorates.\n\n- As mentioned in the above part, it would be nice if the author can provide some discussions about what kinds of scene transitions (small transition vs large transition, same object vs different objects) their model is good at.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The task of generative transition is novel and rarely explored, which I believe is one of the main novelty of this paper. As current text-to-video generation models are mostly tacking short video clips, a smooth and creative transition between these short clips is of increasing importance.\n\n- The proposed random-mask based model seems a good solution for this task. In addition to generating transition frames, the random-mask based model can also deal with long-video generation and image-to-video generation by giving unmasked frames at different positions. \n\n- The quantitative and qualitative experiments demonstrate the effectiveness of the proposed model.", "weaknesses": "- From the qualitative result shown in Figure 6, it seems that the transition is more like a \"interpolation\" between two scenes. For example, the frames in (row1, col4). (row2, col2), and (row2, col3) are not very natural. \n\n- In Figure 5 right part (the cat example), it seems that morphing also provides a descent transition. So for two frames with small transitions needed in between, it seems that the proposed method might add unnecessary variety/creativity. \n\n- In general, it's hard to see if the proposed method provides a good solution to this new task. The paper also lacks enough ablation study of the model architecture design. More discussions and intuitions about this task would be helpful for future works.\n\n- Some minor things: \n1. In Sec. 2, the citation for PYoCo is missing. \n2. In the last paragraph of Sec. 3, the sentence describing \"Long video\" is incomplete.\n3. Figure 4 is not easy to understand at first glance. It would be nice to add more descriptions for better readability.\n4. In Figure 10, the image on the left part has red-green-blue watermarks. Is that example from Gen-2 instead of SEINE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kec6CJ0T1S", "reviewer": "Reviewer_Yzaz", "summary": "The paper proposes a new problem of generative transition and prediction, which can help generate story-level videos through different shot transitions. The author also proposed a short-to-long video diffusion model, which utilizes a random mask strategy for training. To evaluate the task, this paper proposes three assessing criteria. Both objective and subjective evaluation prove their proposed method’s effectiveness.", "questions": "I wondered how many GPUs they used and how long it takes for training. Besides, as far as I know, FID is used to evaluate video generation quality in many papers, can they provide this to make their paper more solid?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This article addresses the limitation of existing models that can only generate shot-level videos and proposes a method to generate story-level videos using transitions. They extend an existing video generation framework and achieve impressive results in generating long videos. They also propose a reasonable evaluation framework to assess the proposed model, and a large number of demos and quantitative evaluations demonstrate the effectiveness of their approach. The contribution of this work is significant.", "weaknesses": "The author should provide a more detailed description of the model for reproducibility, including training resources, training parameters, and so on. Additionally, the author should also report scores on commonly used evaluation metrics such as FID.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RM12egHW4E", "reviewer": "Reviewer_4xUi", "summary": "This paper proposes a method to train a diffusion model with random masking on the frame level to enable video generation, prediction, and interpolation. They demonstrate that their method is able to generate longer video and create smooth transitions between two different frames. The authors demonstrate that their method outperforms baseline methods.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written, and clear\n- The paper shows good long video generations, and is able to generate complex transitions between semantically different frames", "weaknesses": "My primary concerns center around the lack of baselines and novelty. Particularly, the authors fail to cite a few very related works, that accomplish similar tasks that enable frame prediction and interpolation.\n\n- MaskViT [1], MAGVIT [2]: MaskGit-like models trained on tokenized video frames. Given the masked learning object, these models can also usually generalize to enable generation, prediction, and interpolation. MAGVIT is trained explicitly to do this. \n- MCVD [3], RaMViD [4]: These two methods seem nearly identical to the proposed method, where a video diffusion model is trained with masked latents. An exception is a lack of text-conditioning and scale in [3,4], however, I do not believe that meets the bar as a point of novelty.\n\nCould the authors please clarify on how their method is novel over the prior work mentioned above? In addition, it would be necessary to compare against a subset of these methods as baselines (or a similar model), as currently there are no baselines explicitly trained for the prediction / interpolation tasks.\n\n[1] Gupta, Agrim, et al. \"Maskvit: Masked visual pre-training for video prediction.\" arXiv preprint arXiv:2206.11894 (2022).\n\n[2] Yu, Lijun, et al. \"Magvit: Masked generative video transformer.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Voleti, Vikram, Alexia Jolicoeur-Martineau, and Chris Pal. \"MCVD-masked conditional video diffusion for prediction, generation, and interpolation.\" Advances in Neural Information Processing Systems 35 (2022): 23371-23385.\n\n[4] Höppe, Tobias, et al. \"Diffusion models for video prediction and infilling.\" arXiv preprint arXiv:2206.07696 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)