ExVideo: Extending Video Diffusion Models via Parameter-Efficient Post-Tuning

## Abstract

Abstract. Recently, advancements in video synthesis have attracted significant attention. Video synthesis models such as AnimateDiff and Stable Video Diffusion have demonstrated the practical applicability of diffusion models in creating dynamic visual content. The emergence of SORA has further spotlighted the potential of video generation technologies. Nonetheless, the extension of video lengths has been constrained by the limitations in computational resources. Most existing video synthesis models can only generate short video clips. In this paper, we propose a novel post-tuning methodology for video synthesis models, called ExVideo. This approach is designed to enhance the capability of current video synthesis models, allowing them to produce content over extended temporal durations while incurring lower training expenditures. In particular, we design extension strategies across common temporal model architectures respectively, including 3D convolution, temporal attention, and positional embedding. To evaluate the efficacy of our proposed post-tuning approach, we conduct extension training on the Stable Video Diffusion model. Our approach augments the model’s capacity to generate up to 5 × 5\times 5 × its original number of frames, requiring only 1.5k GPU hours of training on a dataset comprising 40k videos. Importantly, the substantial increase in video length doesn’t compromise the model’s innate generalization capabilities, and the model showcases its advantages in generating videos of diverse styles and resolutions. We will release the source code and the enhanced model publicly 1 1 1 Project page: https://ecnu-cilab.github.io/ExVideoProjectPage Github repo: https://github.com/modelscope/DiffSynth-Studio .

## 1. Introduction

Over recent years, diffusion models (Sohl-Dickstein et al., ; Ho et al., ) have achieved outstanding results in image synthesis, significantly surpassing previous GANs (Dhariwal and Nichol, ). These achievements have subsequently fostered a burgeoning interest in the adaptation of diffusion models for video synthesis. Models such as Stable Video Diffusion (Blattmann et al., ), AnimateDiff (Guo et al., ), and VideoCrafter (Chen et al., ) epitomize this research trajectory, showcasing the ability to produce frames that are not only coherent but also of high visual quality. These achievements underscore the practicality and potential of employing diffusion models in the field of video synthesis. With the groundbreaking results of SORA (Liu et al., ) reported at the beginning of 2024, the research direction of video synthesis has once again attracted widespread attention.

Although current video synthesis models are capable of producing video clips of satisfactory quality, there remains a considerable gap in their ability to extend the duration of the videos generated. To generate longer videos, current methodologies can be categorized into three types. 1) Training with long video datasets (Chen et al., ; Wang et al., ; Bain et al., ): Given the computational constraints, present video generation models are predominantly trained on short video clips. The use of longer videos for training would result in prohibitively escalated costs. 2) Generating videos in a streaming (Kodaira et al., ) or sliding window (Duan et al., ) manner. Without further training, we can process the video by dividing it into multiple segments. However, this approach leads to lower video coherence. In addition, existing video generation models lack the capability for long-term video understanding, hence the accumulation of errors becomes inevitable. As a result, during the generation of long videos, the visual quality is prone to deterioration, manifesting as a breakdown in the imagery. 3) Frame Interpolation (Huang et al., ; Wu et al., ): Video frame interpolation models offer a method to augment the frame count of generated videos. However, this approach is inadequate for extending the narrative timeframe of the video. While it increases the number of frames, retaining the original frame rate would give the video an unnatural slow-motion effect, failing to prolong the narrative span of the video content. These outlined challenges underscore the necessity for innovative solutions capable of overcoming the existing hurdles associated with video duration extension, without compromising video quality or coherence.

Recent breakthroughs in the development of LLMs (large language models) (Xiong et al., ; Xiao et al., ; Chen et al., ) have inspired us. Notably, LLMs, despite being trained on fixed-length data, exhibit remarkable proficiency in understanding contexts of variable lengths. This flexibility is further enhanced through the integration of supplementary components and the application of lightweight training procedures, enabling the processing of exceptionally lengthy texts. Such innovations have motivated us to explore analogous methodologies within video synthesis models. In this paper, we introduce a novel post-tuning strategy, called ExVideo, specifically designed to empower existing video synthesis models to produce extended-duration videos within the constraints of limited computational resources. We have designed an extension structure for mainstream video synthesis model architectures. This framework incorporates adapter components, meticulously engineered to maintain the intrinsic generalization capabilities of the base model. Through post-tuning, we enhance the temporal modules of the model, thereby facilitating the processing of content across longer temporal spans.

In theory, ExVideo is designed to be compatible with the majority of existing video synthesis models. To empirically validate the efficacy of our post-tuning methodology, we applied it to the Stable Video Diffusion model (Blattmann et al., ), a popular open-source image-to-video model. Through ExVideo, we can extend the original frame production capacity from a limit of 25 frames to 128 frames. Importantly, this expansion was achieved without compromising the model’s distinguished generative capabilities. Additionally, the enhanced model exhibits the versatility to be seamlessly integrated with text-to-image models (Podell et al., ; Li et al., ; Chen et al., ). This synergistic amalgamation establishes robust and versatile pipelines. This adaptability underscores the potential of our post-training technique, the source code and the extended model will be released publicly.

In summary, the contributions of this paper include:

- •
We present ExVideo, a post-tuning technique for video synthesis models that can extend the temporal scale of existing video synthesis models to facilitate the generation of long videos.
- •
We have trained an extended video generation model based on Stable Video Diffusion, capable of generating coherent videos of up to 128 frames while preserving the generative capabilities of the original model.

## 2. Related Work

### 2.1. Diffusion Models

Diffusion models (Sohl-Dickstein et al., ; Ho et al., ) are a category of generative models that model the generation process as a Markov random process. Unlike GANs (Goodfellow et al., ), diffusion models do not require adversarial training, hence their training process is more stable. Moreover, through an iterative generation process, diffusion models are capable of producing images of exceptionally high quality. In recent years, image synthesis models based on diffusion, including Pixart (Chen et al., ), Imagen (Saharia et al., ), Hunyuan DiT (Li et al., ), and the Stable Diffusion series (Rombach et al., ; Podell et al., ; Kang et al., ), have achieved impressive success. Diffusion models have given rise to a vast open-source technology ecosystem. Technologies such as LoRA (Hu et al., ), ControlNet (Zhang et al., ), DreamBooth (Ruiz et al., ), Textual Inversion (Gal et al., ), and IP-Adapter (Ye et al., ) have endowed the generation process of diffusion models with a high degree of controllability, thereby meeting the needs of various application scenarios.

### 2.2. Video Synthesis

Given the remarkable success of diffusion models in image synthesis, video synthesis approaches based on diffusion have also been proposed in recent years. For example, by adding motion modules to the UNet model (Ronneberger et al., ) in Stable Diffusion (Rombach et al., ), AnimateDiff (Guo et al., ) transfers the capabilities of image synthesis to video synthesis. Stable Video Diffusion (Blattmann et al., ) is an image-to-video model architecture and can synthesize video clips after end-to-end video synthesis training. Unlike image synthesis models, video synthesis models require substantial computational resources since the model needs to process multiple frames simultaneously. As a result, most current video generation models can only produce very short video clips. For instance, AnimateDiff can generate up to 32 frames, while Stable Video Diffusion can generate a maximum of 25 frames. This limitation prompts us to explore how to construct video synthesis models over longer temporal scales.

### 2.3. Extending Generative Models

Although the existing diffusion models are trained with a fixed scale, such as Stable Diffusion being trained at a fixed resolution of $512\times 512$, some approaches can extend them to larger scales. For instance, in image synthesis, approaches like Mixture of Diffusers (Jiménez, ), MultiDiffusion (Bar-Tal et al., ), and ScaleCrafter (He et al., ) can increase the resolution of generated images by altering the inference process of the UNet model in Stable Diffusion. Similar techniques have also emerged in large language models. With the help of positional encoding technologies such as RoPE (Su et al., ) and ALiBi (Press et al., ), large language models can extrapolate to longer text processing tasks under the premise of training with limited-length texts. Post-tuning can further help language models achieve super-long text comprehension and generation (Xiong et al., ; Chen et al., ). These research findings have inspired us, prompting us to explore the extension of video synthesis models. We aim to endow existing video synthesis models with the capability to generate longer videos.

## 3. Methodology

Figure: Figure 1. The architecture of extended temporal blocks in Stable Video Diffusion. We replace the static positional embedding with trainable positional embedding and add an adaptive identity 3D convolution layer to learn long-term video features. The modifications are adaptive, preserving the original generalization abilities of the pre-trained model. All parameters outside the temporal block are fixed while training for lower memory usage.
Refer to caption: x1.png

In this section, we first review the architectures of mainstream video diffusion models, then discuss how we extend the temporal modules for long video synthesis, and finally introduce the post-tuning strategy.

### 3.1. Preliminaries

The huge demands for computational resources in training video synthesis models lead to a prevalent practice of adapting existing image synthesis models for video generation. This adaptation is typically achieved by incorporating temporal modules into the model to enable the synthesis of dynamic content. We provide a comprehensive overview of temporal module architectures, which can be categorized as follows:

- •
3D convolution (Li et al., ): Convolution layers form the foundational blocks in computer vision. 2D convolution layers have been employed in the UNet (Ronneberger et al., ) architecture, which is widely used in diffusion models. By extending 2D convolutions into the third dimension, these layers are seamlessly adapted for use in video synthesis models. Research indicates that convolution layers in diffusion models exhibit a high degree of adaptability across various resolutions (Bar-Tal et al., ), which is a testament to their capacity for generalization.
- •
Temporal attention (Vaswani et al., ): In image synthesis, the importance of attention mechanisms is underscored by their contribution to the generation of images with remarkable fidelity, as evidenced by the ablation studies in latent diffusion (Rombach et al., ). Transferring spatial attention mechanisms to the video domain raises concerns regarding computational efficiency due to the quadratic time complexity of the attention operators. To circumvent this computational bottleneck, advanced video synthesis models typically adopt additional temporal attention layers (Guo et al., ; Blattmann et al., ) that optimize efficiency by curtailing the volume of embeddings processed by each attention operator.
- •
Positional embedding (Su et al., ): The native attention layers cannot model the positional information in videos. Therefore, video synthesis models typically incorporate positional embeddings to enrich the embedding space with positional information. Positional embeddings can be instantiated through diverse methodologies. For example, AnimateDiff (Guo et al., ) opts for learnable parameters to establish positional embeddings, whereas Stable Video Diffusion (Blattmann et al., ) utilizes trigonometric functions to generate static positional embeddings.

### 3.2. Extending Temporal Modules

Most video synthesis models are pre-trained on videos comprising only a constrained number of frames due to limited computational resources. For instance, Stable Video Diffusion is capable of generating a maximum of 25 frames, while AnimateDiff is limited to synthesizing sequences of up to 32 frames. To augment the capacity of these models to produce extended videos, we propose enhancements to the temporal modules within these models.

Firstly, the inherent functionality of 3D convolution layers to adaptively accommodate various scales has been previously validated through empirical studies (Jiménez, ; Bar-Tal et al., ; He et al., ), even without necessitating fine-tuning. Consequently, we opt to retain the 3D convolution layers in their original form to preserve these capabilities. Secondly, regarding the temporal attention modules, research on large language models has demonstrated the potential for scaling existing models to accommodate longer contextual sequences (Xiong et al., ; Chen et al., ). Inspired by these findings, we fine-tune the parameters within the temporal attention layers during the training process to enhance their efficacy over extended frame sequences. Thirdly, for the positional embedding layers, either static or trainable embeddings cannot be directly applied to longer videos. To circumvent this pitfall while ensuring compatibility with a wide array of existing video models, we use extended trainable parameters to replace the original positional embeddings. These extended trainable positional embeddings are initialized in a cyclic pattern, drawing upon the configurations of the pre-existing embeddings. Further, drawing inspiration from various adapter models (Hu et al., ; Zhang et al., ), we incorporate an additional identity 3D convolution layer subsequent to the positional embedding layer, aimed at learning long-term information. The central unit of this 3D convolution kernel is initialized as an identity matrix, and the remaining parameters are initialized to zero. The identity 3D convolution layer ensures that, before training, there is no alteration to the video representation, thereby maintaining consistency with the original computational process.

We apply our devised extending approach to Stable Video Diffusion (Blattmann et al., ), which is a popular model within open-source communities for video synthesis. The comparative architectures, both pre and post-extension, are illustrated in Figure 1. Because of the fundamental similarities that underpin the construction of temporal blocks within video synthesis models, our extending approach can also be applied to various video synthesis models.

### 3.3. Post-Tuning

Figure: Figure 2. Examples in different styles generated by our Extended Stable Video Diffusion, where the first frame is generated by Stale Diffusion 3. The prompt is “A beautiful coastal beach in spring, waves lapping on sand”, followed by the description of style.
Refer to caption: x2.png

After extending the temporal blocks in the video synthesis models, we enhance the model’s abilities to generate extended videos via post-tuning. To circumvent potential copyright concerns with video content, we employed a publicly available dataset OpenSoraPlan(^2^22[https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0](https://huggingface.co/datasets/LanguageBind/Open-Sora-Plan-v1.0.0)), which comprises 40,258 videos. These videos were sourced from copyright-free platforms, including Mixkit(^3^33[https://mixkit.co/](https://mixkit.co/)), Pexels(^4^44[https://www.pexels.com/](https://www.pexels.com/)), and Pixabay(^5^55[https://pixabay.com/](https://pixabay.com/)). The videos in this dataset maintain a resolution of $512\times 512$, inconsistent with the original resolution supported by Stable Video Diffusion. Given the model’s design to accommodate varying resolutions, we opt to conduct the training at this resolution. ExVideo expands this capacity to 128 frames. Over such extended sequences, full training is deemed impractical because of the substantial computational requirements. Instead, we employed several engineering optimizations aimed at optimizing GPU memory usage. These optimizations are crucial for managing the increased computational load and facilitating efficient training within limited hardware resources:

- •
Parameter freezing: All parameters except the temporal blocks are frozen.
- •
Mixed precision training (Micikevicius et al., ): We deploy a mixed precision training program by converting a subset of the model’s parameters to 16-bit floating-point format.
- •
Gradient checkpointing (Feng and Huang, ): Gradient checkpointing is enabled in the model. By storing intermediate states during forward passes and recomputing gradients on-demand during the backward pass, this technique effectively decreases memory usage.
- •
Flash Attention (Dao, ): We integrate Flash Attention to enhance the computational efficiency of attention mechanisms.
- •
Shard optimizer states and gradients: We leverage DeepSpeed (Rasley et al., ), a library optimized for distributed training, to enable shard optimizer states and gradients across multiple GPUs.

The loss function and the noise scheduler are consistent with the original model. The learning rate is $10^{-5}$ and the batch size on each GPU is 1. The training was conducted using only 8 NVIDIA A100 GPUs over one week. In order to ensure the stability of the training process, exponential moving averages are employed for the update of weights.

## 4. Case Studies

### 4.1. Text-to-Video Synthesis

By integrating existing text-to-image models, we can easily develop integrated pipelines capable of converting textual descriptions into videos. In this pipeline, the outputs from a text-to-image model are utilized as the foundational frames for the subsequent image-to-video model. Illustrative examples are shown in Figure 2, wherein the foundational frames are synthesized by Stable Diffusion 3. The prompts direct the text-to-image model to create images across various styles. Our model subsequently generates fluent motion transitions from these high-quality images, even if styles such as flat anime and pixel art are not included in the training dataset. The Extended Stable Video Diffusion model preserves and extends the generalization capabilities in the original model.

### 4.2. Visualization of Training Process

Figure: Figure 3. Video examples in different training phases. The first frame is generated by Hunyuan DiT, and the prompt is “sunset, mountains, clouds”. We present the optical flow to visualize the motion, where pixels with similar colors are moving in similar directions.
Refer to caption: x3.png

We investigated the evolution of the model’s capabilities during the training process. Figure 3 presents the generated videos that exemplify the model’s performance at three distinct phases of training. It is difficult to present the dynamics using still images, thus we present the optical flow, computed by RAFT (Teed and Deng, ), to the right of each example for a clearer demonstration of motion. Initially, before training, the extended model architecture was solely capable of guaranteeing the structural integrity of the video frames, which suffered from pronounced jittering artifacts. Progressing through the training, after 32,000 steps, the model began to produce videos displaying smooth camera movements. With continued training up to 64,000 steps, the model further advanced to create complex motions, such as clouds and mountains moving with nuanced, layered speed. The model effectively understands the depth and spatial relationships within the scene. This example intuitively illustrates the process of the model learning long-term information.

### 4.3. Adaptive Resolution

Figure: Figure 4. Video examples in various resolutions. The first frame is generated by Stable Diffusion 3, and the prompt is “bonfire, on the stone”.
Refer to caption: x4.png

We also ascertain the performance of the extended Stable Video Diffusion across various resolutions. Several video examples are illustrated in Figure 4. These examples demonstrate that the model, with common aspect ratios, adeptly generates videos in higher resolutions. This capability not only highlights the robustness and generalizability of the base model but also underscores the effectiveness of our post-tuning methodologies in enhancing model performance.

### 4.4. Comparison with Other Models

Figure: Figure 5. Visual comparisons of text-to-video results from several existing video synthesis models and our Extended model. The prompts are “a boat sailing smoothly on a calm lake” and “an astronaut flying in space, Van Gogh style”. In our pipeline, the first frame is generated by Hunyuan DiT, and our extended Stable Video Diffusion generates the video according to the first frame.
Refer to caption: x5.png

To evaluate the performance of our model, we conducted comparative analyses against several existing video synthesis models. Illustrative outcomes from these models, including AnimateDiff (Guo et al., ), LaVie (Wang et al., ), ModelScopeT2V (Wang et al., ), OpenSora(^6^66[https://github.com/hpcaitech/Open-Sora](https://github.com/hpcaitech/Open-Sora)), T2VTurbo (Li et al., ), VideoCrafter2 (Chen et al., ), alongside our extended Stable Video Diffusion, are displayed in Figure 5. The videos generated by the baseline models are collected from GenAI-Arena(^7^77[https://github.com/ChromAIca/VideoGenMuseum](https://github.com/ChromAIca/VideoGenMuseum)) (Ku et al., ). The text-to-image model utilized in our pipeline is Hunyuan DiT (Li et al., ). A critical observation from this comparison is that the majority of existing video synthesis models usually generate videos with minimal motion dynamics. In contrast, owing to the post-tuning processes applied over extensive temporal duration, our extended model demonstrates a superior capability to generate videos with significant movements. This differential outcome underscores the advanced generative performance of our model.

## 5. Limitations

While ExVideo can enhance the capabilities of video diffusion models, the post-tuned version continues to be constrained by the inherent limitations of its foundational model. Notably, the extended Stable Video Diffusion struggles to accurately synthesize human portraits, leading to frequent instances of truncated frames. To develop a model capable of synthesizing high-quality long videos, it is imperative to train a robust base model. Nevertheless, due to limitations in resources, we are unable to independently pre-train a large video synthesis model. Consequently, we eagerly anticipate the release of open-source models in the future to advance our research endeavors.

## 6. Conclusions and Future Work

In this paper, we delve into the enhancement of video diffusion models through post-tuning. Specifically, we propose a post-tuning approach called ExVideo, which can extend the duration of generated videos and release the potential of video synthesis models. Based on Stable Video Diffusion, our approach achieves a quintupling in the number of frames, while preserving the original generalization abilities. ExVideo is designed within the constraints of limited computational resources, thus it is exceptionally memory-efficient. By integrating this method with other open-source technologies, we facilitate pipelines conducive to the production of high-quality videos. However, the post-tuned models still face the limitations of the base model. To further improve the performance, we will try to train the model on larger datasets in the future.

###### Acknowledgements.

###### Acknowledgements.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: 1. The writing is excellent—clean, clear, and concise.

2. Most current video generation methods demand substantial computational resources for training. However, this post-tuning method requires significantly fewer resources, such as 8 A100 GPUs for just one week.
summary: This paper introduces a post-tuning method to enhance current video generation models, allowing them to produce longer videos with lower training costs. It leverages 3D convolution layers and temporal attention, enabling the video generation up to 5 times longer than the original length without compromising quality.
weaknesses: 1. My major concern is the novelty. The proposed methods extends the original SVD by adding an identity 3D convolution layers in the temporal block. The technique novelty is limited. 

2. The experiments are not comprehensive. 
(a) For the user study. The authors only compared with SVD. Authors should compare with more methods.
(b) I am a little confused of the user study setting. SVD generate videos with 25 frames, while ExSVD generates videos with 128 frames. Users can easily find the video generated by the new method. Is it a fair comparison?
(c) I have the same concerns of the results on VBench. How many frames does the ExSVD generate? Other methods, e.g. SVD, Kling, generate short videos. Are these evaluation metrics also sentensive to the number of frames?


Minor.
1. Table 2 only labels the best result in the total score column as bolden. The authors should label more results.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: Scalability: How does ExVideo perform as the model scales up in terms of complexity and the number of parameters? Are there any limitations to the model's scalability?

Model Limitations: The paper mentions that the extended Stable Video Diffusion struggles with accurately synthesizing human portraits. Could the authors elaborate on the nature of these issues and any potential solutions or workarounds?

Dataset Diversity: Are there plans to train ExVideo on other datasets to evaluate its robustness across different video content?
soundness: 2
strengths: Originality: The paper presents an effective approach to extend the capabilities of video synthesis models to generate longer videos, which is an original contribution in the field of video generation. 

Quality: The methodology appears to be well-thought-out, with a clear description of the technical details and the design of the extension strategies for different temporal model architectures. The paper also provides a comprehensive evaluation of the proposed method, including both automatic metrics and human evaluations.

Clarity: The paper is well-organized and clearly written. The introduction sets the stage for the problem, the methodology section explains the approach in detail, and the experiments section provides a clear demonstration of the effectiveness of ExVideo. The figures and tables are well-designed and aid in understanding the content.

Significance: The work is relatively significant as it addresses a critical challenge in video synthesis, which is the ability to generate longer, high-quality videos. The proposed solution has the potential to impact various applications, including content creation, entertainment, and beyond.
summary: The paper titled "ExVideo: Extending Video Diffusion Models via Parameter-Efficient Post-Tuning" introduces a novel post-tuning methodology aimed at enhancing the capabilities of current video synthesis models to generate longer videos with lower training costs. The authors propose ExVideo, which is designed to extend the temporal scale of existing video synthesis models, allowing them to produce content over extended durations without compromising video quality or coherence. The paper demonstrates the efficacy of ExVideo by training an extended model, ExSVD, based on the Stable Video Diffusion model. The authors claim that ExSVD can generate up to 5 times its original number of frames with only 1.5k GPU hours of training on a dataset of 40k videos. The paper also discusses the compatibility of ExVideo with various temporal model architectures, including 3D convolution, temporal attention, and positional embedding.
weaknesses: Generalization to Other Models: While the paper claims that ExVideo is designed to be compatible with the majority of existing video synthesis models, the evaluation is primarily based on the Stable Video Diffusion model. More evidence is needed to support the generalization of ExVideo to other models.

Comparison with Other Long Video Generation Models: The paper does not introduce or compare with other existing models that focus on long video generation, particularly in terms of frame consistency over extended durations. Such comparisons are crucial to establish the relative advantages and limitations of ExVideo, especially concerning the maintenance of coherence and quality in long videos.

Lack of Ablation Study: An ablation study is missing in the paper, which would be valuable to understand the contribution of each component of the ExVideo methodology. Such a study could provide insights into the effectiveness of the proposed extensions to the temporal modules and the impact of post-tuning on the overall performance.

Novelty of the Methodology: The paper positions ExVideo as an innovative approach to extend video generation models. However, the methodology described does not appear to introduce any novel techniques but rather builds upon existing model architectures. The paper could benefit from a more detailed discussion of how ExVideo differs from and improves upon current state-of-the-art methods.

Clarity on Training and Inference Algorithm: The paper lacks a clear and detailed explanation of the exact algorithms used for training and inference of longer videos. The process by which the extended temporal modules are integrated and the specifics of the post-tuning process are not explicitly outlined. The authors should provide a more detailed description of these processes to ensure that the paper is accessible to readers who may wish to build upon this work.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: My main concern is the unclear motivation and insufficient comparison, please refer it at point 2 in Weaknesses.
soundness: 1
strengths: The paper writing is clear and easy to understand. The visual results are pleasing.
summary: This paper propose ExVideo, a post-tuning methodology for video synthesis models. Particularly, it designs extension strategies across common temporal model architectures respectively, including 3D convolution, temporal attention, and positional embedding. Experimental results demonstrates the effectiveness of proposed method.
weaknesses: 1. Lack of novelty. This paper proposes extending the pre-trained SVD to generate more frames by adding an additional 3D convolution and a trainable positional embedding. These techniques are commonly used in modern video generation model architectures. Additionally, Section 3.3 includes practical training settings and parameters with no algorithm-level contribution, which should be moved to the appendix.
2. Unclear Motivation and Insufficient Comparison. The motivation of this paper is not clear, and it lacks corresponding comparisons. The authors mention that the original SVD can only generate 25 frames, and this paper proposes extending the generated videos to 128 frames. However, what is the problem with directly fine-tuning the original model on 128 frames? Is the proposed additional 3D convolution necessary? Does it save much memory or make the training procedure converge faster? If so, please provide corresponding numerical comparison results.
3. Lack of Implementation Details. Since the core contributions of this paper are the proposed "Identity 3D Convolution" and "Trainable Positional Embedding," please illustrate and describe the implementation details of these components, preferably with formulas and detailed illustrations.
4. Paper Layout Issues. Figures 3 and 4 are placed at the end of the paper, which is not visually pleasing.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Please see the Weaknesses above.
soundness: 2
strengths: 1. The generated videos seem to have a good quality.
2. Easy to use, applicable to existing models.
summary: This paper presents ExVideo, a post-tuning method for video synthesis models, to extend the video length in generation with limited resources. The authors extend the temporal modules of SVD and adopt efficient tuning methods for training at limited costs,
weaknesses: 1. Limited Contribution

The contribution of the proposed method is limited; the novel part of this work is two components, using additional trainable parameters for positional embeddings, and using an additional 3D convolutional layer with a tailored initialization. Simply adding learnable parameters cannot be a contribution, without theoretical / experimental justification of its necessity. Specifically, I cannot understand why the additional (identity) 3D convolutional layer is necessary, which I believe the merit of using it would simply come from additional parameters. Likewise, I think adding trainable positional embeddings could be further explained on its necessity and efficacy, with comparison to existing methods.

In the intro, the authors address the difficulty of tuning video synthesis models for extended length generation, in terms of computational constraints. However, their proposed (temporal) modules are irrelevant to this matter, but rather introduces additional parameters, which would increase the overall cost. What makes the proposed method efficient is, to my understanding, the methods used for post-tuning, described in Section 3.3. Yet, the techniques used here are from prior research, and not a novel contribution of this paper. If the authors were to argue that their method is efficient, I believe there should have been a novel design for computational savings, which cannot be found in the current paper.


2. Experimental results

- The experiments do not reflect the main argument of this paper. In Fig.3, the generation results with comparison to existing T2V methods are presented. However, I believe this does not show how well ExVideo successfully extends the generation length of videos. The baseline methods all show good temporal coherency and excellent visual quality. The most noticeable difference between the baselines and the proposed model is text-alignment, which is not the main argument of the paper. The better text-alignment may be possibly coming from a stronger baseline, SVD. To highlight the effectiveness of the proposed method, experiments on generation quality when extending the video length with comparison to SVD would have been necessary.
In addition, in order to claim the main argument of the paper, better extension ability of video sequence, more experiments on how existing methods fail when the generation length gets longer should have been presented. (e.g., Generation quality of latter timestep frames collapsing) However, according to the qualitative results in Fig.3, existing models do not seem to be struggling with video generation of extended length, and the only difference I can find with comparison to baseline methods is text-alignment, which is presumed to be due to use of advanced baseline. 

- Quantitatively, the results do not seem to be strong enough. Using 8 A100 GPUs over a week of training does not seem to be cost-efficient, but considering the cost, the improvement seems to be marginal. 

- Lack of ablation studies. The paper currently does not discuss the ablation study of the proposed components, which makes it hard to verify the effectiveness of the method. For instance, the authors could have done an ablation study on how the initialization scheme in identity 3D convolutional layer affects the training, or on how switching the positional embeddings affect the performance.

Meta-review: Reviewers unanimously recommend rejection of the paper, citing lack of novelty, clarity in presentation, lack of comparisons, among other things. 

Authors did not participate in discussion phase, and as such, the initial scores persist. 

There are a handful of good bits of feedback for improving the current manuscript in the provided reviews, we encourage the authors to revise their manuscript and re-submit.

Decision: Reject