VersVideo: Leveraging Enhanced Temporal Diffusion Models for Versatile Video Generation
Authors: Jinxi Xiang
Ricong Huang
Jun Zhang
Guanbin Li
Xiao Han
Yang Wei
Venue: ICLR 2024 poster (2024)
Keywords: video generation
diffusion model
temproal modeling
Primary area: generative models
TL;DR: We introduce a video generation model for stable and controllable videos across various conditions.

Abstract: Creating stable, controllable videos is a complex task due to the need for significant variation in temporal dynamics and cross-frame temporal consistency. To address this, we enhance the spatial-temporal capability and introduce a versatile video generation model, VersVideo, which leverages textual, visual, and stylistic conditions. Current video diffusion models typically extend image diffusion architectures by supplementing 2D operations (such as convolutions and attentions) with temporal operations. While this approach is efficient, it often restricts spatial-temporal performance due to the oversimplification of standard 3D operations. To counter this, we incorporate two key elements: (1) multi-excitation paths for spatial-temporal convolutions with dimension pooling across different axes, and (2) multi-expert spatial-temporal attention blocks. These enhancements boost the model's spatial-temporal performance without significantly escalating training and inference costs. We also tackle the issue of information loss that arises when a variational autoencoder is used to transform pixel space into latent features and then back into pixel frames. To mitigate this, we incorporate temporal modules into the decoder to maintain inter-frame consistency. Lastly, by utilizing the innovative denoising UNet and decoder, we develop a unified ControlNet model suitable for various conditions, including image, Canny, HED, depth, and style. Examples of the videos generated by our model can be found at https://jinxixiang.github.io/versvideo/.

Published as a conference paper at ICLR 2024
VERSVIDEO: LEVERAGING ENHANCED TEMPORAL DIFFUSION MODELS FOR VERSATILE VIDEO GENER- ATION
Jinxi Xiang', Ricong Huang", Jun Zhang'", Guanbin Li², Xiao Han', Yang Wei*
Teeneett AI Lab, Shenzhen, China
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China Corresponding author:
junejzhang @ tencent.com
ABSTRACT
Creating stable, controllable videos is a complex task due the need for signif- icant variation in temporal dynamics and cross-frame temporal consistency. To address this, enhance the spatial-temporal capability and introduce versa- tile video generation model, VersVideo, which leverages textual, visual, and stylistic conditions. Current video diffusion models typically extend image dif- fusion architectures by supplementing 2D operations (such as convolutions and attentions) with temporal operations. While this approach is efficient, it often re- stricts spatial-temporal performance due to the oversimplification of standard 3D operations. To counter this, we incorporate two key elements: (1) multi-excitation paths for spatial-temporal convolutions with dimension pooling across different axes, and (2) multi-expert spatial spatial-temporee temporal attention blocks. These enhancements boost the model's ppatial-tempond performance without significantly escalating training and inference costs. We also tackle the issue of information loss that arises when variational autoencoder is used to transform pixel space into la- tent features and then back into pixel frames. To mitigate this, we incorporate temporal modules into the decoder maintain inter-frame consistency. Lastly, by utilizing the innovative denoising UNet and decoder, we develop a unified ControlNet model suitable for various conditions, including image, Canny, HED, depth, and style. Examples of the videos generated by our model can be found at https: nixiiiiiiii
nxi iang.. github i i tt/versvideo/
INTRODUCTION
Diffusion models have transformed automated content creation, empowering designers to generate highly realistic images videos from textual prompts as inputs (Ho et al. 2022a b Rombach al., 2021 Singer et al.] 2022 Huang et al.) 2024). However, despite significant strides in image gener- ation, the quality of generated videos often fails match real-world footage. This discrepancy can be attributed to the inherent high dimensionality and complexity of videos, which encapsulate in- tricate spatiotemporal dynamics within high-resolution frames. In this paper, leverage diffusion models for video generation, specifically addressing three key challenges; (i) the fundamental build- ing block in UNet for handling spatial-temporal dynamics; (ii) the issues of a variation autoencoder to transform a video into latent and subsequently reconstruct it back into pixel space, and (iii) the capacity to control video generation under diverse conditions.
UNet architecture. The creation of open-domain text-to-video models is widely acknowledged as a significant challenge, primarily due to the scarcity of large-scale text-video paired data and the complexity of constructing space-time models from the ground up. To mitigate these challenges, most current methodologies are built upon pre-trained image generation models (Ho et al. 2022b) Wu et al. 2022 Chen et al.] 2023a Xing et al. 2023a). These methodologies typically adopt space- time separable architectures, where spatial operations are derived from the image generation model. To incorporate temporal modeling, various strategies have been employed, including pseudo-3D modules (Singer et al., 2022) Zhou et al., 2022). serial 2D and ID blocks (Saharia et al. (2022] Xing
al. 2023a), and parameter-free techniques such temporal shift (An al. 2023a Zhang al.

---

Published as a conference paper at ICLR 2024
custom spatiotemporal attention (Wu et al. 2022 Wang et al. 2023a). However, these methodologies often overlook the critical interaction between time and space, which is essential for creating visually engaging text-to-video content.
2023)
Specifically, parameter-free approaches (An et al,, 2023b; Wu ct al., 2022) rely on manually de- signed rules that often fail capture the intrinsic nature of videos, resulting in the generation of unnatural motions. Learnable 2D+1D modules and blocks (Blattmann et al. 2023] Singer et al. 2022 Xing al. (023a) primarily concentrate on temporal modeling, either by directly feeding temporal features to spatial features or combining them through simplistic element-wise additions. This limited interaction often results in temporal distortions and mismatches between the input texts and the generated videos, adversely affecting the overall quality of the generated content. Recent advancements in convolution designs for video classification (Wang et al., 2021; Li et al., 2020 Jiang et al., 2019 Wang et al. 2021: Liu et al., (023a), as well as attention mechanisms for vision transformers (Puigcerver et al.
2023 Huang et al., a 2023b), have demonstrated significant potential for efficient and robust spatial -emporal modeling. We aim to address these challenges by advancing our approach with a highly efficient design of spatial-temporee convolution and attention blocks.
Instead of training the model raw pixels, the Latent Diffusion Model (LDM) first uses an autoencoder to learn a low- low-dimensional latent space that succinctly parameterizes images, and then models this latent distribution. Building on this concept, numerous video diffusion models (Blattmann et al. 2023 An et al.) 2023a Wang et al,, 2023d He et al. 2022b; Liu et al. (023a) also employ autoencoder project videos into latent space, significantly redueing the computational load when training on high-resolution videos. To leverage the pre-trained image diffusion model, many of these models (Liu et al. 2023a) repurpose the image autoencoder, focusing primarily on the UNet model design. However, while these models have demonstrated promising results in im- age reconstruction, the time- indeeenndeend image autoencoder can distort temporal information in the latent space (Zhou et al.] (022). The pre-trained image autoencoder has also been reported to cause image distortions due to the high compression rate (Lin et al. 2023 Zhu et al. (023). Lin et al.) 2023) suggests using frequency-compensatee challenges, we enhance the spatial-temporet reconstruction capability of the autoencoder by train- ing new decoder counteract information loss that occurs during projection and reconstruction with spectrum constraints.
Autoencoder.
modules to alleviate this issue. To address these
Controllable generation. The ability to control generation is highly sought-after feature in gen- erative foundation models. In the realm of image generation, models such as ControlNet (Zhang & Agrawala 2023). T21 Adapter (Mou al., 2023), and Composer (Huang al., 2023a) have been developed to manage visual conditions like edge maps and depth maps. When it comes to video generation, VideoComposer (Wang et al., 2023c) enhances video synthesis by providing superior control over both spatial and temporal dimensions. This is accomplished by breaking down a video into three primary factors: textual conditions, spatial conditions, and crucial temporal conditions. A latent diffusion model then uses these conditions to reconstruct the input video. Another method, similar the image ControlNet, Control-A Video (Chen al.) (0023)). This model also has the ability to handle visual conditions. However, each ControlNet model is restricted to managing specific control modality it was trained on, such as an edge map. To handle different visual condi- tion modality, it becomes necessary to train a separate model, leading to significant time and spatial complexity costs. Inspired by the Uni-C ControlNet (Zhao et al. 2023) used in image generation, we propose unified and scalable video control model. This model can be smoothly integrated with our diffusion foundation model, resulting in a less complex training process.
In summary, we present a versatile video generation model that incorporates innovative concepts in (i) The development of a novel UNet video diffusion model: By combining more robust multi- excitation spatial-temporal convolution and Mixture of Experts attention blocks, we enhance the model without significantly increasing training and inference complexity. (ii) The implementation of a spatial-temporal compensated decoder for video reconstruction: We design a VAE decoder with temporal modules and utilize a refinement network to improve reconstruction quality. (iii) The establishment of unified control for video generation: By extending the proposed diffusion model with shared ControlNet and diverse adapters, we achieve comprehensive video control generation.
2

---

Published as a conference paper at ICLR 2024
Training
Inference
HED
crystal elear fish swints of Tre evader
T.D
Dessarg Like
(2c.
Skift
Skill
Skil
Chasad poclica
Spocil parkey
Bucaaion
JD Cleen
10 Come
Multi-excitation conv
Figure 1: The VersVideo builds multi-excitation convolution and MoEs spatial-temporal attention blocks. Unified condition adapters and a shared ControlNet are used for controllable gen- eration. A temporal compensated video decoder addresses potential inconsistencies during video reconstruction. 'T.' and 'S.' denote temporal and spatial, respectively.
METHODS
2.1 DIFFUSION PROBABILISTIC MODELS
DPM, first introduced in (Sohl-Dickstein et al. 2015). is a powerful generative model that has been successfully applied to various tasks, such as image synthesis and video generation. It consists of diffusion process and denoising process. the diffusion process, random noises are gradually added to the data via a T-step Markov chain (Kong & Ping 2021). The noised latent variable at step & can be expressed as:
Z₁ = Vár + v1 âeet, with â = II as a ~~~i N(0,1)
(1)
where aa E (0, 1) is the corresponding coefficient. For I" that is large enough, c.g., T = 1000, we have vār ~ and áī = 1. And zr approximates random Gaussian noise. Then the generation of can be modeled as iterative denoising.
Ho et al. 2020) connect DPM with denoising score matching and propose e e-prediction form for the denoising process:
L. = ||+ zo (zz. t)
(2)
where Zg is a denoising neural network parameterized by 0, and L. loss function.
2.2 DENOISING UNET
As shown in Fig. The diffusion UNet employed in VersVideo is built upon Multi-Excitation Convolution (MEC) and Mfxture-of-Experts Opatial-Temport Attention (MoE-STA) blocks. A semporal-compensteee
decoder is utilized reconstruct videos from the latent space back to the pixel domain. By incorporating auxiliary ControlNet, VersVi deo is capable of handling vari- ous input conditions, such as text, images, HED, depth, and Canny maps.
Folti-Excitttion Convolution. In contrast to existing video diffusion models that often limit spatial-temporal performance due to over-simplification of standard 3D operations, we introduce multi-excitation paths for spatial-temporal convolutions with dimension pooling across different axes. Our design fundamentally inspired by squeeze-and gqueezz-and-exciingg and-excitation (SE) block (Hu et al., 2017] Liet 2020) by explicitly modeling channel/temporal dependencies. The excitation paths consist of (i) factorized 2D spatial ID temporal convolution; (ii) spatial-tempont excitation; (iii) channel excitation; and (iv) motion excitation.

---

Published as a conference paper at ICLR 2024
Given batch of video latents E RX Notations used in this section are B (batch size), C' (channels), T (number of frames), H (height), and W (width). The factorized 2D + 1D branch first reshapes into (BT, C, H, W) for spatial convolution, and then processes it with temporal pseudo-3D kernel (3, 1, 1) in the shape of (B,C. C, H, W)
For the other three branches, the imput tensors are shifted temporally to following TSM (Lin et al., 2018) before processing, which is computationally free but has strong spatiotemporal modeling ability. The motivation behind this is that both the channel and spatial axes have high dimensions, driving us to pool over these axes and apply convolution separately. For spatial-temporal excitation, we average å along channel axis into F R R We apply 3D convolution F to get the excitation mask Ms. The output of this branch is:
ZSPE 2 + 2 + Ms- 2 + 2 o sigmoid (ConvaD (F)).
For channel excitation, we reduce 2 through spatial averaging into C₁ E RBXCXTXIXI, and then squeezing the channels into C₂ E BXC through 1 x convolution (the channel and temporal should swap for convolution and revert back). Pseudo 3D convolution (3, 1, 1) is applied for temporal reasoning. after that we extend the channel back to the original size and get C3 e RxxxxxxxX Using C3 as the excitation, we gate the with the excitation mask Mc to get:
cc 2 Mc = 2 + ? sigmoid (Ca).
Motion excitation has been explored in previous studies (Li et al. 2020 Jiang et al. 2019 Wang et al. 2021 Liu et al. (223a),, shown in Fig. 2 The motion information is extracted by differen- tiating the adjacent frames. The input frames are squeezed along the channel axis and the motion of 2 and 2, is expressed as:
m = K20 * 24 24-1-
(5)
Motion results {mi, ... are stacked along the tempo- ral axis to reconstruct the video tensor. Then, the tensor pooled on the spatial space and unsqueeze along the chan- nel axis. Finally, the motion tensor is activated with sig- moid to modulate 2.
MoEs Spatial- Teeiioall Attention The UNet incorporates series of convolutional blocks with attention blocks. The 3D attention has a prohibitive complexity of ((HWT)) for tensor (B, HWT, d), where HW denotes the spatial scale, T is the number of frames, and d represents the fea- ture dimension.
1xl, 2D Cone
2--1
3x3, 20 Cony
We use factorized attention to conduct spatial attention in the shape of (BT, HW, d) and subsequently apply tempo- ral attention (BHW,T, d). To enhance this attention, we augment it with MoE design without significant increases in computational cost. In specific, we create MoE layers with N spatial experts (i.e., N FFNs) (Es,1, Es,2, Es.N) and M temporal experts {ET,1, ET.,, ..., ΣT,M) (Huang eet al., 2023b). Without loss of generality, we denote input tokens as (B, d) where L HW for spatial attention and L = ' for temporal attention. Assuming L is divisible by N (or M), we randomly split the tokens into N (or M) groups and then process them with experts:
concate
18.7
Pooling
(R.7
10 ki)
1xl, 2D Cone
(8..6.113
Tranqpus::
signcid
JR.CT.UU
Figure 2: Motion Excitation.
(21, .... FL}
shutlle,
{21, i2. tr} split (X1, XN) feedforward Y, Es.((X))),
(6)
where X1 (2 L X2
(#L/N+1, ... } etc. We unshuffle
(Y, YN, to recover the output of the MoE layer. The temporal MoE layer follows the same operation.
After each training iteration, we update experts by averaging all of them: F (1 E), AFss Fr. = (1 FT, + XFoT
4

---

Published as a conference paper at ICLR 2024
VQ-GAN
Consittenyy Joss
SM
Image Eacoder E.
Tempural Componated Decader
Lasa temporal information
Compenoate temparal Information
TCM
tempural comppensatc module
imput video
reconstructed
Figure 3: The proposed TCM-VAE. Many current video diffusion models rely image VAEs for video encoding and reconstruction, which often neglects crucial temporal information (Wu et al.) 2022 Wang al. 2023d). The proposed TCM-VAE builds image VAE by introducing TCMs and refinement UNet to ensure temporal consistency and video quality.
with:
N. 1 fjf
N F.s.. ETT. D.M 1 257
Es.:
M Ery:
(8)
where Es,; and Er.; the i-th spatial/temporal expert; controls the information exchange rate among experts. Since a large dropout ل (or _ is applied for every expert, we have to com- municate the weights among all of them to avoid data insufficiency and underfit problem. By default, we set = 0.001 as a small value, ensuring experts gradually evolve together, while in the mean- time, avoiding weights collapse where all experts are identical unexpectedly.
During inference, each MoE layer is transformed into an FFN layer by averaging FFNs LEN and FFNT attention does not increase inference computation cost. Merging weights of models for better perfor- mance is widely acknowledged in large language models (Jin et (022) and image classification
- S.M Er,i In this way, introducing MoE enhance spatial-temporal
3
WWotsman et al. 2022
Ainsworth et al., 2022).
2.3
TEMPORAL COMPENSATED DECODER
Most existing video latent diffusion models utilize image VAEs to reconstruct videos from latents (Wu et al. 2022 Chen et al. 2023b Xing et al.) 2023b Wang et al.) 2023d). However, an image VAE without temporal modules fails to capture the dependencies between frames. This leads to a loss of temporal information in videos, which in turn results in flickering artifacts and temporal
inconsistency Blattmann et al. 2023).
We introduce Temporal Compensation Modules (TCMs) enhance temporal consistency, and refinement UNet for quality enhancement, building upon an image pretrained VAE. To optimize training efficiency, porate TCMs into the decoder blocks to infuse temporal information into multi-level intermediate features. The implementation of TCMs leverages the multi-excitatii module in Section 2.2 More details about the TCM-VAE can be found in the Appendix.
freeze the encoder and concentrate training the video decoder. We incor-
We fine-tune the decoder with the added TCMs using video data and incorporate path tem- poral discriminator constructed from 3D convolutions. The optimization loss on the reconstructed image X aligns with that of the VQ-GAN (Esser et al. 2020). For intermediate features derived from TCMs, we impose a consistency loss constraint on the feature, aligning with features of the same level. The underlying principle is that temporal information is gradually lost during encoding; this process can be counteracted through consistency constraints. The final optimization loss training the video decoder is formulated as follows:
Lass LVO-GAN(X,X) + M CF.
(9)
where À is a weight coefficient (10- 4 by default); LVo-GAN is the reconstruction loss in VQ-GAN, with LI, perceptual and discriminator loss; LTCM constrains the similarity between the encoder features and TCM features, for instance, through the 1.2 loss.
5

---

Published as a conference paper at ICLR 2024
A turtle crawling the bettom the
Art Noureeu pointing of a female botanist surrounded by exotic plants In greenhouse
man in riding motorcycle on the read, cyberpunh-styll
A cloumfish ruimming coral reaft.
Tiny plant sprout coming of ground.
Figure 5: Sample text-to-video outputs generated by VersVi deo-H, H, with resolution of 576 x 320 pixels, displayed. Frames selected at rate of every four frames for analysis.
2.4 VERSATILE CONDITIONS GENERATION
For training versatile conditions that guide video generation, freeze the denoising UNet and train the unified ControlNet for visual conditions. Additionally, we train the linear adapters of CLIP image for style as illustrated in Fig-
The extension of the image ControlNet for video generation builds upon previous studies (Zhang et al. 2023; Chen et al. 2023b). Un-
like previous models that train separate video ControlNets for specific conditions like canny, depth, HED, etc., we propose unified ap- proach to handle visual conditions using shared ControlNet because diverse visual con- ditions share common low-level visual features. Input visual conditions are preprocessed using lightweight adapters and the shared ControlNet to introduce controllability into the UNet.
Doysh
Trainable Encole. Denzaing UNet
Coney Adgverr
vieual condition
etyle orsdition
Image
Conca:
lest grorgt
ecceder
To adapt various low-level visual conditions, we create adapter modules to capture features before applying the shared Non Mathe- matically, the adapter module can be expressed as:
Figure 4: Versatile conditions strategy.
FAdapter (I+) = EK 10=== = F) (I
(10)
where 1 is the indicator function; Ih the k-th visual condition; F the adapter convolution.
For the style condition, which serves as global guidance similar to the text prompt, we propose train- ing a linear adapter in conjunction with the frozen CLIP image encoder Ye et al. 2023). Typically, the linear adapter for text prompts functions control and text embeddings are subsequently concatenated and fed into the cross attentioo modules within both the denoising UNet and the ControlNet.
identity mapping. The image embeddings for style

---

Published as a conference paper at ICLR 2024
Table 1: Text-to-Videe generation on MSR- VTT (Xu et 2016 & denotes the lower the better.
denotes the higher the better.
Method
Training Data
FVD (1)
CLIPSIM
GODIVA (Wu et al.
2021a
MSR-VE VTT
0.2402
NOWA (Wu et al. 202Th1
MSR-VTT
0.2439
Make-A Video (inger et al. 2022)
Web JOM + HD- VILAA-OMM
0.3049
VideoFactory Wang et al. 2023b
Web 10M + HD-V VG-130M
0,3005
LVDM (He et al. 202
Za
WebVid-2M
742
0.2381
MMVG (u at. 2024
WebVid-2.5M
0,2644
CogVideo Hong
2022
WcbVid-5.4M
1294
0.2631
ED-T2V (Luu ct al.
2023b)
Web Vid-1OM
0,2763
MagicVideo (Zhou el all 2022)
Web WebVid-10M
998
Video-LDM [Baatmmann el al.
2023
Wed-Vd-IOM.
0.2929
VideoComposer Wang
202.d
WebVid-IOM
580
0.2932
Latent-Shift (An al. 20236
Web RebbVid-100 Vid-IOM
0,2773
VidcoFusion (Lu ct all
2023
Web SabVid-ONN
581
0,2795
SimDA (Xing et al. (025b)
WebVid-10M.
456
0.2945
VezVVicco-1150MM
Web
620
0,2796
VersVidec-H
WebVid-10M
421
0.3014
EXPERIMENTS
3.1 IMPLEMENTATION DETAILS
Training. The training of VersVi deo consists of two stages. In the first stage, we train the text- to-video video diffusion model using the WebVid-10M. video-text dataset (Bain et al.) (021] and the 300M image-text datasets from LAION-5B (Sohl-Dickstein et al. 2015). We allocate 1 of the GPUs for updating image-text data, while the remaining GPUs are used for video-text data. We develop VersVideo-L, which has 500 million parameters for 16 x 256 x 256 generation, and VersVideo-H, which has 2 billion parameters for 24 576 320 generation. The training details and network specifications are defined in the Appendix. In the second stage, we freeze the UNet and train the visual conditions ControlNet and style conditions using the Web Vid-10M dataset.
Inference. Video generation combines text, visual conditions, and style are presented. We generate the video using the target prompt via DDIM sampling (Song al., 2020).
2016) and UCF-101 datasets using the FVD (Unterthiner al., 2018). CLIPSIM (Radford et al. ablation study, we further employ frame consistency (FC) to assess video continuity by calculating the average CLIP similarity between two successive frames (Wang al. 2023d
Evaluation. We evaluated our model on the MSR-VTL (Xu et al.
2021), and IS metrics. In the
Esser al. 2023).
3.2 TEXT-TO-VIDES GENERATION
The MSR-VTT test consists 2,990 examples, each containing 20 descriptions or prompts. To generate videos, we randomly select one prompt from each example, resulting in 2,990 videos. Each video consists of 16 frames at frame rate of 30 fps. For VersVide deo-H,,d,eH, we extract 16 consecutive frames from a total of 24 frames. These frames are then resized and center-cropped to dimensions of 256 x 256. To evaluate the quality of the generated videos, we calculate the CLIPSIM score by averaging the scores of the 47, 47,840 frames. For comparative FVD statistics, we randomly select video sequences from the dataset, ensuring that each sequence contains a minimum of 16 frames.
In Table[] our approach achieves average CLIPSIM score of 0.3014, even in zero-shot scenario, outperforming the majority of competing methods. This indicates a robust semantic correlation between the generated videos and the input text. It is worth noting that Make-A- A-Video (Singer
2023b) achieve higher CLIP scores, but they leverage et al. 2022) and VideoFactory (Wang et al.
additional large-scale HD-VILA (Xue al.
2021) datasets during their training process.
We also provide visualizations of the generated videos for various text prompts in Fig. 5 These examples demonstrate deo's ability to generate diverse videos with high frame consistency.
7

---

Published as a conference paper at ICLR 2024
Original video
Cybrrpunk
audnming
glase
Mounteln river in spring
robut walking tha fluld
Locus Rosuerr,
Figure 6: Conditioned video generation with image, depth, canny, HED, and style.
3.3 CONDITIONS GENERATION
Visual conditions. We categorize elements such images, HED, Canny, depth, etc., as visual conditions, as they provide spatial information that guides the video generation process. In VersVideo model, these conditions processed using individual adapters, which then introduce controllable features into the denoising UNet. Fig. showcases the results of incorporating diverse text prompts with visual conditions, demonstrating the effectiveness of approach.
Style condition. The style condition serves as a global guide. is extracted using a multi-modal CLIP image encoder and subsequently concatenated with text embeddings. The last two rows in Fig. 6 show two distinct painting styles for video generation, using depth maps as the spatial control.
3.4 ABLATION STUDIES
UNet model design. We carried ablation study validate the design of the denoising UNet model, incorporating both the MEC and MoE-STA components. We trained various versions of the VersVideo-L model, and VersVideo-BP and subsequently evaluated their class-conditional generation using the UCF-101 dataset. Table 2 presents the study's 's findings. Both the MEC and MoE-STA have demonstrated their effectiveness design elements in improving the performance of the video diffusion model. These elements consistently contribute to performance enhancements. Although the VersVideo-I model does not surpass state-of-the-art models like Make-a- Video (Singer et al.) 2022) and VideoFusion (Luo al. 2023), which boast billions of parameters, it is worth noting that the VersVideo-2 model has a significantly smaller parameter count of 500 M. Notably, the Vers VersVideo-H model achieves an IS score of 81.3 and an FVD score of 1 119, making the second-be best performing model after Make-A Video (9.728 (9.72B parameters). The FC consistently improves with our designed modules.
8

---

Published as a conference paper at ICLR 2024
Table 2: Ablation of model designs UCF101. FC means frame consistency.
Method
Resolution
IS
FVD
FC
MoCoGAN-HD (Tulyakov et al.
2017)
128 x 128
12.42
CigVideo (Hong
2022)
160 160
50.46
626
DVD-GAN (Clark et
2019)
128 x 128
32.97
TATS (Ge et al. 2022)
128 x 128
79.28
332
VideoFusion [uo al.
2023
128 x 128
80,03
173
Make-A-Video [Singer el al.
2022)
256 256
82.55
81.25
VersVideo-1, (baseline)
256 x 256
60.2
355
0.843
VeraVideo-L (+ MEC)
256 x 256
68.8
287
0,862
VersVideo-I. (+ MoE-STA)
256 x 256
63.5
302
0.861
VersVideo-I MEC MoE-STA)
256 256
72.9
207
0.880
VersVideo-H
320
81.3
0.905
Table 3: TCM-VAE for video reconstruction and generation on UCF-101.
Video Reconstruction
Text-to-video Generation
PSNR
SSIM
1St
FVD1
FCt
vanilla image decoder
32.376 dB
0.9129
43.23
598
0.865
3D convolution decoder
32.692 dB
0.9181
49.40
508
0.868
decoder
32.254 dB
0.9194
50.39
490
0.871
TCM-VAI (Ours)
33.019 dB
0.9253
60.18
443
0.880
TCM-VAE. To evaluate the effectiveness of TCM-VVE, we compare VAE decoder variants: the vanilla image decoder (Esser et al., 2020), the 3D convolution decoder, and the temporally fine- tuned decoder (Blattmann al.] 2023). Models
trained with subset of 100,000 videos from the HD-100M (Xue et al. 2021) dataset to avoid watermarks. We assess video reconstruction using PSNR and SSIM in the YUV color space the UCF-101 test dataset comprising 3,783 videos. We evaluate text-to-vided generation with VersVideo using FVD, CLIPSIM, and FC.
In Table the vanilla image decoder overlooks temporal inter-frame dependencies, leading only moderate reconstruction performance. Previous studies have demonstrated that incorporating 3D convolution or adding temporal layers to the standard image decoder can significantly improve video reconstruction and generation. Our designs of the proposed TCM-VAE collectively contribute to a superior overall performance. Specifically, our TCM-VAE achieves an improvement in video reconstruction, with an increase of 0.643 dB in PSNR and 0.0124 in SSIM. Furthermore, the text- to-video generation achieves an improvement of 16.95 IS, an improvement 0.015 in FC, and reduction of 155 in FVD. These results highlight the effecttivnees of our modifications in enhancing both the fidelity of video reconstruction and the quality of text-to-vided generation.
4 CONCLUSION
In response the oversimplified spatial-temporal modeling prevalent in most video diffusion mod- els, we introduce a temporal-enhanced diffusion model that incorporates multi-excitation convolu- tion and MoE attention, without significantly increasing the computation costs. Additionally, we augment the VAE decoder to reconstruct the latent space into pixel space and employ unified control model accommodate diverse conditions.
Limitation and future work. (i) The generated videos sometimes contain watermarks due to the training dataset WebVid-10M. To ensure high-quality generation, it necessary to have watermark- free videos with textual descriptions. (ii) Our primary focus is on generating videos of a fixed length (e.g., L = 24). However, an interesting future direction would be to extend our method for long video synthesis, considering clip-by-clip generation. (iii) VersVideo inherits certain common challenges from image diffusion, including hand generation and text alignment. Solutions employed in image generation could serve inspiration for video generation. (iv) The stability of image-to- video generation is not perfect. The input image cannot consistently maintained throughout the videos. It requires additional control preserve the spatial appearance.

---

Published as a conference paper at ICLR 2024
REPRODUCTIVITY STATEMENT
We provide details (e.g., hyperparameter, model, and optimizer) and experiment setups (c.g., datasets, metrics) in Section 3.1 and Appendix A, B, and C.
REFERENCES
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging mod- els modulo permutation symmetries. ArXiv, https/2299.488345 2022. URL https://api.
semanticscholie wwt.ooogggcoooo.o.
2521 99400]
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent- shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint
arXiv:2304.08477, 2023a.
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xiaoyue Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video genera- tion. ArXiv, abs/2304.08477, 2023b. URL
CorpusID:2581880222
Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. 2021 IEEE/CVF International Conference on Com- puter Vision (ICCV), pp. 1708-1718, 2021. URLhttps: /tt///aapp.p semanticschollar
CorpusiD:2324780
A. Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 22563- 22575, 2023, UtL/itttess//pp...eemnn...ooooo. wwtp:oopoo/ooooooo/oooo 255818877
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023a.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang-Jin Jin Lin. Control-a-video: Controllable text-to-video generation with diffusion models. ArXiv, abs/2305. 13840, 2023b. URL hhtpps://aat 1588416451
aarcchola.... wwwpllaa..rrrrroeeeeeeeeeeeeeeee wwwp//rrr///....eeeee ID:
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation complex datasets. arXiv: Computer Vision and Pattern Recognition, 2019. URL https://api.
isemantcsscholrr org. Corpus ID: 200:200086665 749
Nan Du, Yanping Huang, Andrew M. Dai. Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellsterm, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Z. Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of- experts. ArXiv, abs/21 12.06905, 2021. URL hitps::/appp www.steisccoooolee.o....cccoo org/ CorpusiD:2451241:
Patrick Esser, Robin Rombach, and Björ Ommer. Taming transformers for high-resolution image synthesis. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12868-12878, 2020. URL https: /stpss//aap... iemattischolaa.
229297973
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthesis with diffusion models. ArXiv, abs/2302.03011, 2023. URL petpp::tp///.m api semant iewraanttssciie icscholar. wtt//oorpssaaooooooooooooopoe :::::::665555
William Fedus, Barret Zoph. and Noam M. Shazeer. Switch transformers: Scaling to trillion pa- rameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:39 2021.
URL https::///........... /appi. semant cschol org/ otrp/Corpuus1::211573335
10

---

Published as a conference paper at ICLR 2024
Tsu-Jui Fu, Licheng Yu, N. Zhang, Cheng- Yang Fu, Jong-Chyl Su, William Yang Wang, and Sean Bell. Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 10681-10692, 2022. URL hatps://api.emmeee iesemanticccooo::: wwsccccchllass.... org/ CorpuSID:253801977
Songwei Ge, Thomas Hayes, Harry Yang. Xiaoyue Yin, Guan Pang. David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference on Computer Vision 2022. URL https://api. semant 1 cscholaaae aaroooar... httphoolll...ro/ooootottii T0:
Yin- He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelitg video generation with arbitrary lengths. ArXiv, abs/2211. 2022a. URL
ar. wtt/CorpussaaoCoooooooo ID: 253802030
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- cls for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13222 2022b.
Jonathan Ho, Ajay Jain, and Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006.1122 11239, 2020. 219955663
URL http:://pp....ennaanooooooooorrp
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole. Mohammad Norouzi, David J. Fleet, and Tim Salimans. Im- agen video: High definition video generation with diffusion models. ArXiv, abs/2210.02303, 2022a.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. ArXiv, fass/2244034558, 2022b. URL https://api.
semant wwchoischhlloo ar wrg. www./oogg/oooroee Corpus ID: 24800016
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to- -iieoo generation via transformers. arXiv preprint arXiv:2205. 2022.
Jie Hu, Li Shen, Samuel Albanic, Gang Sun, and Enhua Wu. Squeeze-and-excinee networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2017. URL https: //api. www.mttiisccoo...ooo ar org/ Corpus 140309863
Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero- shot text-to-video generator with Ilm director and Idm animator. Advances in Neural Information Processing Systems, 36, 2024.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. In International Conference on Machine Learning, 2023a. URL httpss:/app www.ntiescholla.ttt....
attp:s:/aa...smmmniccctooo..rrr /trg/Corppss::
257038979
Yongqian Huang, Peng Ye, Xiaoshui Huang, Sheng Li, Tao Chen, and Wanli Ouyang. Ex- perts weights averaging: new general training scheme for vision transformers. ArXiv,
abs/2308.0600 066993, 2023b. URL https::/app. Seemmnnttcsccooorrrr//CCooo oug.a//aoooo/////////aaaaaaaaaan
260865986
Hugging Face. Zeroscope v2 576w Model, 2023. URL httpps:/hugginngaace.
v7666 Accessed: Nov. 16, 2023.
Boyuan Jiang, Mengmeng Wang. Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. 2019 IEEF/CVF International Conference Com- puter Vision (ICCV), pp. 2000-2009, 2019. htt.hhttp::/appp
Coxpus1D:1994721
Xisen Jin, Xiang Ren, Daniel Preotiuc- Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. ArXiv, abs/2212.09849,, 2022. URL hhtps:: ://pp:::/p.... semant cschol ar. arr.org/corpusa1:2557777
11

---

Published as a conference paper at ICLR 2024
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion mod- els are zero-shot video generators. ArXiv, abs/2303.13439, 2023. URL https://api,
semant i cschol ar. org/ Ctte/orruuuuuu ID: 257 687280
Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. ArXiv, 00122 2021. URL http:s//app....mmniiiccchho
235265701
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam M. Shazeer, and Z. Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. ArXiv, 166620066.6668, 2020. URL https://api.
semant. i cscho 1 ar. wttg/oorp.u..ooooooooooooooootep ID: 2202
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 906-915, 2020. URL http://///app.ssmaiitiee.ee
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. 2019 JEEE/CVF International Conference on Computer Vision (ICCV), pp. 7082-7092, 2018. URL https:://pp..
www.nt..cccooaa..coomm www.orr//rorroooo.ooo : 85542740
Xinmiao Lin, Yikang Li, Jenhao Hsiao, Chiu Man Ho, and Yu Kong. Catch missing details: Image reconstruction with frequency augmented variational autoencoder. 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1736-1745, 2023. URL https:
/app..smanticschollaa http://c......o.a.oooooooooaa//.
Binhui Liu. Xin Liu, Anbo Dai, Zhiyong Zeng, Zhen Cui, and Jian Yang. Dual-stream diffu- sion net for text-to-video generation. ArXiv, abs/2308.0833 2023a. URL https::/apii semanticschon ort.oooruro//oouusppppa : 2 260926302
Jiawei Liu, Weining Wang, Wei Liu, Qian Hc, and J. Liu. Ed-12v: An efficient training frame- work for diffusion-based text-to-video generation. 2023 International Joint Conference Neu- ral Networks (IJCNN), pp. 1-8, 2023b. URL https:/ /ttts:///pp..sematttsihhtt
Cox3usID:260387646
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liangsheng Wang, Yujun Shen, Deli Zhao, Jinren Zhou, and Tien-Ping Tan. Videofusion: Decomposed diffusion models for high- quality video generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 10209-10218, 2023. URL https:///pppp http://ppp...mmatt.cccccooo..... .org/
SorpusID:2575:35
Chong Mou, Xintao Wang, Liangbin Xie, Jing Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. ArXiv, abs/2302.08.. 08453, 2023. URL https: //ap::/apppp... www.sseecccccccooar..ccomo
CorpusID:256900
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. ArXiv, abs/2308.000 00951,, 2023. URL https: h/aps:://ap..../// api. www.nticschooar .orgo/rrrrrrrooo//ooooooo
Corpus
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021. URL https::/aaagp http:/ppp..emmmiiicccccccccocooa
2:315914455
Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021. URL httph:tt//////
semant cscho ar. org. httpo/rr...orooooooo112233555333
12

---

Published as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205. 1487, 2022. ar orr. wtt..oorr/roooot:::22302.0
URL htRl:hhtps:///ee
Uriel Singer, Adam Polyak, Thomas Hayes, Xiaoyue Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make- a-video: Text-to-video generation without text-video data, ArXiv, abs/2209. 14792, 2022. URL
wwtp/orrggsoiir. https::/app..emmntiiico o :52595992222999
Jascha Narain Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep un- supervised learning using nonequilibrium thermodynamics. ArXiv, fbbs/1503.03533 2015, URL
httpss:/sap..eemaarccccco stw.nnitcceemmcnccccccccoco ar.
14888175
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv
preprint arXiv:2010.02502: 2020.
S. Tulyakov, Ming- Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan:
Decomposing motion and content for video generation. 2018 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 1526-1535, 2017. URL https:/ aapppp.... www.stemsmccooooo...o.m https://ap.semmantccchhlllll. org/
CorpusID:4675365
Thomas Unterthiner. Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards accurate generative models of video: A new metric chal- lenges. ArXiv, abs/1812.01777, 2018. URI. http:s://pp...emmttttioo www.setiiccccolar...ooo
/aap:
DD:
Wen Wang, Kan Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot video editing using off- the-lf image diffusion models. ArXiv, abs/2303.17599 2023a. URL
https://api wwwant
Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention spatiotemporal diffusions for text-to-video genera- tion. ArXiv, abs/2305, 10874, 2023b. URL https:/ :aaps://app... sewanticscoolso......... org/
CorpusID:2557624113
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. ArXiv, abs/2306.022 020110 2023c. URL /api. semant ccschola...cc..r....... org/
CDrpus1D:259075722
Xiang Wang, Hangjee Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollability. arXiv preprint arXiv:2306.02011 02018. 2023d.
Zhengwei Wang, Qi She, and Aljoscha Smolic. Action-net: Multipath excitation for action recog- nition. 2021 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR),
pp. 13209-13218, 2021. URL https::/app. |rg.0//0ooooooooooo//.oo.ooo...0
232223075
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ArXiv, abs/2203.054p 055422, 2022. URL https:
7/api. www.ataaaannchhhaceeo www.arorg/oooruueeeeoe 24:362688267
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain vidcos from natural descriptions. ArXiv, 2021a.
abs/2104, 14806. 26314
URL https: //aps.//ap.... swwanticscholeseee.... h wttpsscchho.....ooooooooooooaaaa
13

---

Published as a conference paper at ICLR 2024
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre- pre-training for neural visual world creation. ArXiv, abs/2111.12417, 2021b. URL
htttps://api www.eteiicccoooeeeeee. www.:at..roooooocceeaaaa. 24:457772
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11565, 2022.
Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang. Yong Zhang, Yingqing He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, et al. Make-your-video: Customized video genera- tion using textual and structural guidance. arXiv preprint arXiv:2306. 00943, 2023a.
Zhen Xing, Qi Dai, Hang-Rui Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation. ArXiv, abs/2308.09710, 2023b. URL https://api. Semanticscholess org/ org.
261 030352
Jun Xu, Tao Mci, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. 2016 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp.
2016. URL https: www.nnicccccccclla..oooo
CorpusID:2065944
Hongwei Xue, Tiankai Hang. Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang. Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recog-
nition (CVPR), pp. 5026-5035, 2021. URL https::/app. senanticsccoollr.orrg
Hu Ye, Jun Zhang, Siyi Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. ArXiv, has/2308.066211, 2023. URL/htpps://apl semant i cscho ar. org/ Corpus ID: 260886968
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion
models. ArXiv, abs/2302.05543, 2023. URL
CorpusID:25682711
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-frre controllable text-to-video generation. 258832670
URL
https: /apps://api.. www.ntincchooolaaaeeeeeecccc org/Corpus
:
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K. Wong. Uni-controlnet: All-in-one control to text-to-image diffusson mod-
cls. ArXiv, 16322205. 2023. URL
org
CorpusID:225888112
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211. 2022.
Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang. Yinpeng Chen, Lu Yuan, and Gang Hua. Designing a better asymmetric vggan for stablediffusion. ArXiv, abs/2306.0400 04633,, 2023. URLbttps. //api. semant waschoarr.... otaeccool....ooo/ooooorsoo333333
14

---

Published as a conference paper at ICLR 2024
Table 4: VersVi deo network details.
Versvidec-L
VersVideo-F
Architecture
LDM
LDM
video resolution
16 256 256 3
24 576 320
VAE factor f
z-shape
16 x 32 x 32 x
24 x 72 x 40 x
channels
128
320
Depth
2
Channel multiplier
1, 4 32, 16, 8 8
1, 1,2,4, 72, 18
Attention resolutions
Number of heads
8
Cross-Attention
Embedding dimension CA resolutions
768 32, 16, 8 77
768
72, 36, 18 77
CA sequence length
MEC kernel size
2D spatial 1D temporal
(3.3) + (3.1.1)
(3.3) + (3,1.1)
Temporal shift
(3.)
(3.)
Spatial-temporal excitation
(3,3,3)
(3,3,3)
Channel excitation squeeze
(1, 1)
(1, 1)
Channel excitation temporal
(3,)
(3.)
Channel excitation expand
(1. 1)
(L.
Motion excitation squeeze
(1. 1)
(1.
Motion excitation temporal
(3,
(3,3)
Motion excitation expand
(1.1)
(1.1)
MoE-STA
Number of Spatial FFNs
32
32
Number of Temporal FFNs
4
Training
Parameterization
v
v
# train steps
100 k
100
Learning rate Batch size per GPU
5 x 10
5 x 10
16
# GPUs
16
32
GPU-type
A100 - 40 GB
A100 40 GB 0.1
Pdrop
0.1
Diffusion Setups
Diffusion steps
1000
1000
Noise schedule
Lincar
Lincar
80
0.00085
0.00085
BT
0.0120
0.0120
Sampling
Sampler
DDIM
DDIM
Steps
30
30
CFG scale
7.5
7.5
Total Parameters during Inference
500 M
2
A VERSVIDEO NETWORK DETAILS
The paper introduces two distinct variants of video diffusion models, referred to as VersVideo-1 and VersVideo-H! The primary distinction between these two models lies in their dimensions: VersVideo- L has a dimension of 128, while VersVideo-Ba has dimension of 320, as detailed in Table In the ResBlock of the LDM, we have embedded Multi-Excitation Convolutions (MECs), which replace the original 2D convolution designs used for image diffusion. These MECs operate in parallel, providing a more efficient approach. Other parameters such as upsampling or downsam- pling, well as input output channels, remain consistent with the original design. Regarding Attention (MoE-STA), we have established 32 Feed-Foree -orwaadd Networks (FFNs), or 'experts', for spatial attention, and additional 4 FFNs for temporal attention during the training phase. It's important to note that during the inference stage, all experts are averaged to produce single FFN.
the MoE Ppatial-Temporr
15

---

Published as a conference paper at ICLR 2024
(BT/N,HW,
Training phase
(BHW (8.,/4
Vomm
reshape
Nams
(BHW, .7.0)
(RT, ())
W (ni0og
butch size
number of Enimes
(BT/N.WW.C)
(BHW/M.T.d)
#: widhh
feature dimension
spotial expents
Inference phase
M. oumber of temporal experts
reshape
Norm
ANaieNy pesodua
mmm
(OT, MWW.W.CO
(BHW, T,
average experts" weights
avenige experis weights
Figure 7: The proposed MoF. for the spatial-temporal transformer. In both spatial and temporal transformers, employ multiple FFN experts during the training phase. These experts trained using data that is randomly allocated to them. After each training iteration, the weights of the experts shared amongst each other at an exchange rate denoted by Eq. 7 After training, all the experts are averaged to ensemble into FFN, without increasing the inference costs.
B MOE-STA
Section 2.2 outlines the proposed MoE for spatial-temporal trans- formers. In this section, we provide additional details to enhance understanding.
The role of the FFN in transformers is to process the information aggregated by the attention mechanism. can learn to recognize and generate more intricate patterns based on the information it re- ceives from the attention block. The most recent prevalent training approach for transformers involves replacing the FFN layer with sparse MoE (Lepikhin et al., 2020 Du et al., 2021 Fedus et al. 2021). The sparse MoE includes multiple expert FFNs, each with unique weights, and trainable routing network, as shown in Fig. During both the training and inference phases, this routing work selects sparse set of experts for each input, enabling efficient scalability of transformer models through sparse computation.
nusie
Selt Attration
Norrn
As shown in Fig. 7 we introduce new MoE scheme that trains ex- perts without a router network, reducing the number parameters introduced by the router and simplifying training. Our MoE assigns tokens to experts by randomly partitioning them into groups. the each
Figure 8: Sparse MoE.
end of each training iteration, we perform weight averaging MoE's experts. After training, we can average the experts of each MoE into single FFN, without increasing inference cost.
Why does this MoE work? Merging weights of models for better performance is widely acknowl- edged in large language models (Jin et al., 2022) and image classification (Wortsman et al.) 2022 Ainsworth et al. 2022). This simple yet effective idea has been proven to improve accuracy and robustness. Similarly, proposed MoE merges experts' weights efficiently enhance the perfor- transformer. Unlike conventional model ensembles, we average the
mance of the model weights without incurring any additional inference memory costs.
16

---

Published as a conference paper at ICLR 2024
image
VQ-GAN loss
A
A2
Axr
Latoo!
Imuge
Image
KQ-GAN Joss
Our propased
Consistency loes
Latent
SM
$2
S1
Image Encoder
Temporal Compensated Decoder
temporal
Campensate temporal
A
encoder bleck
Sy
decoder bleck
intermediate fratures
module X
input video 8 reconstructed video
Figure 9: The comparison between the image VAE and the proposed TCM- Unlike the image VAF, which encodes and reconstructs image frames individually along the temporal dimension, we introduce TCMs and refinement UNet to enhance inter-frame consistency and video quality.
Sputtal Toddlz
Siganel
(87 Si
37. C R,80
S.AUU
D: betch das
Rippptilll resolation
cOurdd
Ma..8
Figure 10: Refinement UNet of TCM-VA VAE.
C TCM-VAE
C.I NETWORK DETAILS
There are two critical components in TCM-VAE: the Cempooal-Coopensatee Module (TCM) and the refinement UNet. We choose VQ-GAN (Esser et al.) 2020) as the image VAE backbone for TCM-VAE and train the decoder along with TCMs and a refinement UNet.
The encoding process of TCM is identical to the image encoding. Given an input video X of shape (B, C, T, H, W), we reshape the time axis the batch axis (BT, C, H. W), and then process the video as batch of images:
E(X) Am (... (A2 (A1(X)))).
(11)
The decoder with TCMs embedded can be represented as:
= R (S) (... (SM (SM(
(12)
where the refinement UNet. JM is the temporal compensated features using TCMs.
We denote T; as the i-th TCM block which consists of sequences of convolution layers and tivations. We implement all TCMs with the MEC block as detailed in Section [.2] because it is computationally efficient and also has strong temporal ability. The intermediate features are:
FM- SM (TM (ZM) 2M)
(13)
F S (Ti (=1) + 21),
given the input of decoder 2M
17

---

Published as a conference paper at ICLR 2024
It's important to note that in TCM- VAE, all tensors outside the TCM are 4D, specifically (N x T, C, H. W). Initially, we transform the input 4D tensor into a 5D tensor (N, C, H, W) before it's introduced to the TCM, which allows for operations
specific dimension within the TCM.
After this, the SD output tensor is converted back to a 4D tensor before it's passed to the subsequent TCM-VAE decoder blocks.
The refinement UNet is constructed using residual blocks, and notably, does not include any atten- tion blocks. A comprehensive depiction of the network can be found in Figure T0
C.2 VISUAL QUALITY
The VAE from pre-trained image models is often reused for most existing video latent diffusion models. However, the image VAE significant contributor video flickering and inconsistency. To illustrate this issue, we present several visual examples. We compare the widely-used image VAE StableDiffusion with our proposed TCM-VAE.
As for image VAE, we encode the video frames independently and then reconstruct them back into a video using an image decoder. For TCM- VAF, the encoding process remains unchanged, but we employ the TCM-VAE decoder with temporal awareness.
C.3 BOIG-AND-PLAY FOR OTHER VIDEO GENERATION MODELS
TCM-VAE offers another advantage in terms of compatibility. Since we freeze the image encoder during training, it is seamless to integrate TCM-VAE with various video diffusion models. This plug-and-play solution allows for easy integration with most existing models. For example, we apply TCM-VA, to VideoComposer (Wang al. (203d) and Videofusion (Luo al. 2023), and the flicker artifacts can be largely alleviated.
D ABLATION STUDY OF MEC
MEC a novel approach for apatial-temporal convolutions with dimension pooling across differ- axes, which avoids the over-simplification of standard 3D operations that often limits spatial- temporal performance in existing video diffusion models. It involves using multi-excitation paths, which include factorized 2D spatial ID temporal convolution (2D+1D), spatial-temporal excita- tion (STE), channel excitation (CE), and motion excitation (ME).
we provide a detailed description of MEC. Additionally, an ablation study con- ducted in Table to validate each excitation path of MEC, which includes the MoE-STA component. Various versions of the VersVideo model are trained with and without these paths, followed by an evaluation of their class-conding class-conditional generation using the UCF-101 dataset. The results suggest that the excitation modules improve the performance of MEC. When integrating all four excitation paths, the MEC demonstrates superior performance in both IS and FVD.
In Section 2.2
E
USER STUDY
To conduct the user study, we selected four widely acknowledged and open-sourced models, namely Text2video-zerra (Khachatryan al.] 2023). VideoFusion (Luo et al. (023). Zeroscope (Hugging Face 2023). and VideoComposer (Wang et al., 2023d). for visual comparison. The evaluation set consisted of 30 video prompts, and 16 evaluators compared the inter-frame consistency and overall
Table Ablation of MEC designs on UCFI01.
2D+1D
STE
CE
ME
IST
FVD
V V
69.5
249
V
68.0
264
67.7
266
68.3
260
V
V
r
72.9
207
18

---

Published as a conference paper at ICLR 2024
Text2vido0-acro 97.9%
Video
Text2video-zern
95.3%
Vers Video
Videof'usion
77.1%
Vedee
VideoFuusion
83.3%
VereVideo
ZeroScope
7..0%
VedvV
ZeroScope
Video
CompoCo
62.3%
VervVidoo
VideoComposer
88.5%
VersVideo
50%
50%
consistency
Overall quality
Figure 11: VersVideo v.s. prior work in text-to-video in terms of inter-frame consistency and overall quality win-rates evaluated by majority score of human evaluator preferences.
video quality (including inter-frame consistency, alignment of text-video, and perceptual factors) between videos from competing methods and one from our method, shown in random sequence. Figure shows that our method, VersVideo, was preferred over VideoFusion 83.3% of the time in terms of video quality, and VideoComposer 88.5% of the time. Our method also performed significantly better than the baseline methods in terms of inter-frame consistency in the user study.
19


Reviews:
[{"review_id": "B80LXGDOhW", "reviewer": "Reviewer_1xNe", "summary": "This paper proposed a versatile video generation model named VersVideo, which enhances spatial-temporal capability using textual, visual, and stylistic conditions. The model incorporates multi-excitation paths for spatial-temporal convolutions and multi-expert spatial-temporal attention blocks to improve temporal consistency and generation performance without escalating costs. It also uses temporal modules in the decoder to maintain inter-frame consistency and mitigate information loss. Besides, they design a unified ControlNet model suitable for various visual conditions.", "questions": "Can the current framework handle the situation with multiple condition inputs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well organized. The authors have well categorized and summarized previous methods. The authors explain the shortcomings of previous methods and the motivation for proposing the method very clearly.\n- The authors provide web demos for visualization. It is helpful for the readers to examine the effectiveness of the proposed method.\n- The authors provide a specially designed module for perceiving temporal information for video generation tasks. The design of the module is very innovative, and it shows improvement in quantitative indicators.\n- The decoder design and consistency loss are helpful for maintaining temporal consistency.", "weaknesses": "- The authors did not provide a visual comparison of the results. Providing a visual comparison would better demonstrate the effectiveness of the method. Besides, a user study is also needed to verify the effectiveness of the method.\n- Adding the impact of each module on visual effects in the ablation experiments would be more helpful in verifying what role each module plays.\n- The quality of the video-to-video editing results currently shown in the paper is not much better compared to other methods such as Render-A-Video and TokenFlow. The results generated in the current demo have severe color changes. Therefore, I doubt the effectiveness of this method in video fidelity.\n- Since many modules in the framework designed in the paper are related to temporal consistency, please provide indicators to measure temporal consistency. Currently, the paper only lists indicators of generation quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GhkYjE0iHo", "reviewer": "Reviewer_eX2J", "summary": "The paper proposes VersVideo for video generation. To capture the temporal information, multi-excitation paths for spatial-temporal convolutions and multi-expert spatial-temporal attention blocks. Also, the decoder of VAE is also finetuned to maintain inter-frame consistency. The authors demonstrate the effectiveness of VersVideo in multiple video generation tasks, e.g., text-to-video generation and conditional video generation.", "questions": "Please see my concerns in the weakness part.", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The paper studies video generation and proposes several blocks to enhance the temporal consistency of generated videos.\n\n2) The generated videos have plausible results.\n\n3) The authors demonstrate the capability of the proposed network in the controllable generation setting.", "weaknesses": "1) The details of the MoE attention are not explained very well. In section 2.2, the authors mentioned that they use MoE design to enhance attention. If my understanding is correct, the attention is performed on all channels, but the feedforward network after attention is applied to different groups. If the MoE attention is applied like this, how can the feature or attention be enhanced?\n\n2) Also, several details are missing for the temporal compensate decoder. In the paper, the authors do not mention what S1 stands for.\n\n3) For the Multi-excitation conv, the motivation for using four branches is not addressed clearly. The effectiveness of each branch is also not thoroughly ablated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bpd7g2Nkth", "reviewer": "Reviewer_CWze", "summary": "This paper addresses the challenge of creating stable and controllable videos by proposing a versatile video generation model. In contrast to existing video diffusion models that often limit spatial-temporal performance due to oversimplification of standard 3D operations, VersVideo introduces multi-excitation paths for spatial-temporal convolutions with dimension pooling across different axes and multi-expert spatial-temporal attention blocks. This approach significantly improves the model's spatial-temporal performance without increasing training and inference costs. To address information loss during the transformation from pixel space to latent features and back, temporal modules are incorporated into the decoder to maintain inter-frame consistency. The paper also presents a unified ControlNet model suitable for various conditions, such as image, Canny, HED, depth, and style.", "questions": "1. It might be beneficial to include a section discussing the limitations of your current approach and possible future work to address these limitations.\n2. If possible, consider releasing the code or at least providing more detailed implementation notes. This could help others replicate your results and build upon your work.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of multi-excitation paths and multi-expert spatial-temporal attention blocks enhances the model's spatial-temporal capabilities without significantly increasing computational cost. This is particularly impressive given the complexity of 3D operations in video generation. The solution proposed for minimizing the issue of information loss, mainly through the incorporation of the temporal module, helps achieve better inter-frame consistency, which is crucial for maintaining the temporal coherence of generated videos. The development of a unified ControlNet provides a more versatile approach to handle various conditions, leading to broad applicability.\n2. The quantitative improvements of the proposed method are notable. The related ablations offer reliable evidence about the effectiveness of the whole system.", "weaknesses": "1. Leveraging pre-trained models could limit the model's versatility and the ability to generalize across different datasets. Is it possible to validate the proposed designs to other SD models?\n2. The notable quantitative and qualitative improvements are appreciated. However, some common issues are not discussed in the paper, e.g., hand generation, multiple object generation, instruction following, and how to respond to relative location descriptions, etc. If these cases cannot be well-addressed, then they should be included in a limitation discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ubnGI7qXwE", "reviewer": "Reviewer_Cov9", "summary": "This work introduces a diffusion-based method for video generation. The paper proposes the VersVideo model that leverages multi-excitation spatial-temporal convolution, a Mixture of Experts attention blocks, and a temporal compensated decoder to address the challenges associated with spatial-temporal dynamics. Additionally, it integrates the ControlNet model for versatile video generation, allowing for a wider range of visual conditions.", "questions": "see weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-motivated and the proposal method is technically sound.\n2. The proposed method combines several existing powerful techniques to learn better spatiotemporal features.\n3. The proposed method achieves comparable performance on the MSR-VTT dataset.", "weaknesses": "1. The proposed method directly combines existing techniques, such as excitation network, MoE to enhance spatiotemporal feature representation, which makes the novelty a bit weak. The proposed Multi-Excitation Convolution applies excitation convolution multiple times. Original excitation convolution should be cited.\n\n2. From Table 1, the proposed fails to achieve better performance than Make-A-Video. Even compared with the methods trained on WebVid-10M, the performance improvement is marginal.\n\n3. From Table 2, the performance of the proposed method is a lot worse than the previous STOA method, e.g., VideoFusion. The author explains this by \"the VersVideo-L model has a significantly smaller parameter of 500M\". Why not use a larger model to verify the performance? Moreover, 1) Make-A-Video also reports performance on the UCF dataset, the results should be included; 2) the author should also provide performance under the same resolution as previous methods for better comparison.\n\n4. Results in Table 2 and Table 3 are from different training datasets. Better to keep it consistent for better comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)