CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Authors: Zhuoyi Yang
Jiayan Teng
Wendi Zheng
Ming Ding
Shiyu Huang
Jiazheng Xu
Yuanming Yang
Wenyi Hong
Xiaohan Zhang
Guanyu Feng
Da Yin
Yuxuan.Zhang
Weihan Wang
Yean Cheng
Bin Xu
Xiaotao Gu
Yuxiao Dong
Jie Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Video Generation
Diffusion model
Pretraining
Primary area: generative models

Abstract: We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos that align seamlessly with text prompts, with a frame rate of 16 fps and resolution of 768 x 1360 pixels. 
Previous video generation models often struggled with limited motion and short durations.
It is especially difficult to generate videos with coherent narratives based on text. 
We propose several designs to address these issues. 
First, we introduce a 3D Variational Autoencoder (VAE) to compress videos across spatial and temporal dimensions, enhancing both the compression rate and video fidelity. 
Second, to improve text-video alignment, we propose an expert transformer with expert adaptive LayerNorm to facilitate the deep fusion between the two modalities.
Third, by employing progressive training and multi-resolution frame packing, CogVideoX excels at generating coherent, long-duration videos with diverse shapes and dynamic movements. 
In addition, we develop an effective pipeline that includes various pre-processing strategies for text and video data.
Our innovative video captioning model significantly improves generation quality and semantic alignment. 
Results show that  CogVideoX achieves state-of-the-art performance in both automated benchmarks and human evaluation.
We publish the code and model checkpoints of CogVideoX along with our VAE model and video captioning model at https://github.com/THUDM/CogVideo.

Published as a conference paper at ICLR 2025
COGVIDEOX: TENTTTO-VDEGO DIFFUSION MOD- ELS WITH AN EXPERT TRANSFORMER
Zhuoyi Yang*!
Jiayan Teng*
Wendi Zheng
Ming Ding',
Shiyu Huang Guanyu Feng' Bin Xu
Jiazheng Xu
Yuanming Yang!
Wenyi Hongt
Xiaohan Zhang
Da Yin
Yuxuan Zhang
Weihan Wang"
Yean Cheng
Xiaotao Gu
Yuxiao Dong
Jie Tang
Zhhip AI
TTinghua University
Text Prompt: lightning bolt shatters mountaintop stone
out leaps the Monkey King in battle robes. Energy erupls, winds howl.
Text Prompt: A man put on a colorful wig.
Figure CogVideol can generate long-duration, high-resolution videos with coherent actions and rich semantics.
ABSTRACT
We present CogVideoX, a large-scale text-to-video generation model based diffusion transformer, which can generate 10-second continuous videos that align seamlessly with text prompts, with a frame rate of 16 fps and resolution of 768x 1360 pixels. Previous video generation models often struggled with limited motion and short durations. It is especially difficult generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we introduce a 3D Varia- tional Autocncoder (VAE) to compress vidcos across spatial and temporal dimensions, enhancing both the compression rate and video fidelity. Second, improve text-video alignment, we propose an expert transformer with expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing progressive training and multi-resolution frame packing, CogVideoX excels at generating coherent, long- -ung-uuriion videos with diverse shapes and dynamic movements. In addition, we de- velop effective pipeline that includes various pre-processing strategies for text and video data. Our innovative video captioning model significantly improves generation quality and semantic alignment. Results show that CogVideor VideoX achieves state-of-the-art performance in both automated bench- marks and human evaluation. We publish the code and model checkpoints
*Equal contributions. Core contributors: Zhuoyi, Jiayan, Wendi, Ming, Shiyu and Xiaotao. (ennggg222,teniyy2. @mails tsinghua.edu.cn. corresponding author: Visiting our demo website https: www.tiiiibbbbioiiiiieeeee-eeeeee to watch videos!
wwwiinghuaeeu.cor edu.cn

---

Published as a conference paper at ICLR 2025
of CogVideoX along with our VAE model and video captioning model at https:/ github.com THH.tiD.MCTTTTMMM,TTMT THUDM/CogVideo
INTRODUCTION
The rapid development of text-to-video models has been phenomenal, driven by both the Transformer architecture (Vaswani et al. 2017p and diffusion model (Ho et al. 2020). Early attempts pretrain and scale Transformers to generate videos from text have shown great promise, such as CogVideo (Hong et al. 2022) and Phenaki (Villegas et al. 2022 Mcanwhile, diffusion models have rocently made exciting advancements in vidco generation et al., 2022 Ho et al. 2022). By using Transformers as the backbone of diffuusion models, i.e., Diflusion Translormers (DiT) (Peebles & Xie 2023), lexi-to-video generation has reached a. new milestone, evidenced by the impressive Sora showcases (OpenAl 2024).
Despite these rapid advancements in DiTs, it remains technically unclear how to achieve long-term consistent video generation with dynamic plots. For example, previous models had difficulty generating a video based on a prompt like "a bolt of lightning splits a rock, and person jumps out from inside the rock".
In this work, we train and introduce CogVideoX, a set of large-scale diffusion transformer models designed for generating long-term, temporally consistent videos with rich motion semantics. We address the challenges mentioned above by developing a 3D Variational Autoencoder, expert. Transformer, progressive training pipeline, and video data. filtering and captioning pipeline, respectively.
First, to efficiently consume high-dimension video data, we design and train 3D causal VAE that compresses the video along both spatial and temporal dimensions. Compared to previous method(Blattman significantly reduce the sequence length and associated training compute and also helps prevent flicker in the generated videos, that is, ensuring continuity among frames.
et al., 2023) of fine-tuning 2D VAE, this strategy helps
Second, to improve the alignment between videos and texts, we propose an expert Transformer with expert adaptive Layer- Norm to facilitate the fusion between the two modalities. To ensure the temporal consis- tency in video generation and capture large- scale motions, we propose to use 3D full at- tention to comprehensively model the video along both temporal and spatial dimensions.
Homan Action
GP MoP
Sceap
251
N4.
Dynmi Quality
Dyanmic Degree
Third, as most video data available online lacks accurate textual descriptions, we de- velop video captioning pipeline capable of accurately describing video content. This pipeline is used to generate new textual de- scriptions for all video training data, which significantly enhances Cog CogVideoXX s ability to grasp precise semantic understanding.
Ogendern
Gaad
Appearazee style
Maliiple
Cltocts
OsgVilaoX-2B
Figure 2: The performance of apenly-accessibee Lext-to-video models in different aspects.
In addition, we adopt and design progressive training techniques, including multi-resolution frame pack and resolution progressive training, to further euhance the generation performance and stability of CogVideo) Furthermore, we propose Explicit Uniform Sampling, which stablizes the training loss curve and accelerates convergence by setting different timestep sampling intervals each data parallel rank.
To date, we have completed the CogVideox training with two sizes: billion and 2 billion, respectively. Both machine and human evaluations suggest that VogVidooX-5B outperforms well-known vidco models and CogVideoX-23B is very competitive across most dimensions.
Figure 2 shows the performance of CogVideoX-BB and CogVideoX-2B in different aspects. It shows that CogVideoX has the property of being scalable. As the size of model parameters, data volume, and training volume increase, the performance will get better in the future.
2

---

Published as a conference paper at ICLR 2025
Our contributions can be summarized as follows:
We propose CogVideoX, simple and scalable structure with a 3D causal VAE and an expert transformer, designed for generaling coherent, long-duration, high-action videos. It can generate long videos with multiple aspect ratios, up to 768x 1360 resolution, seconds in length, at 16fps.
We evaluate CogVideoX through automated metric evaluation and human assessment, compared with openly-accessible top-performing text-to-video models. CogVidook achieves state-of-the-art performance.
We publicly release our 5B and 2B models, including text-to-video and image-to- video versions, the first commercial-grade open-souree vidco generation modcls. We hope it can advance the filed of video generation.
2 THE COGVIDEOX ARCHITECTURE
In the section, we present the CogVideoX model. Figure 3 illustrates the overall architecture, Given a pair of video and text input, we design a 3D causal VAE to compress the video into the latent space, and the latents then patchified and unfolded into long sequence denoted as Evision- Simultancously, we encode the textual input into text embeddings Stext using T5 (Raffel al. 2020). Subsequently, Ztext and Evision are concatenated along the sequence dimension. The concatenated embeddings are then fed into a stack of expert transformer blocks. Finally, the model output are unpatchified shape, which is then decoded using a 3D causal VAE decoder to reconstruct the video. We illustrate the technical design of the 3D causal VAE and expert transfomer detail.
restore the original latent
X Expert Transformer blocks
Gite
Gate
Feed Forward
Gate
Gate
3D Full Attention
Scale Shift
Scale & Shift
Text Expert Expert AdalN
Vision Export AdalN
Text Encoder
Zocat
Zyiaton
Causal VAE
station.
Figure 3: The overall architecture of Cog VideoX.
2.1 3D CAUSAL VAE
Videos contain both spatial and temporal information, typically resulting in much larger data volumes than images. To tackle the computational challenge of modeling video data,
3

---

Published as a conference paper at ICLR 2025
Table 1: Ablation with different variants of 3D VAE. The baseline is SDXLl et al, 2023) 2D VAE. Flickering calculates the L1 difference between cach pair of adjacent frames to evaluate the degree of flickering in the vidco. We use variant B for pretraining.
Variants
Baseline
A
B
D
Compression
8x8x1
8x8x4
8x8x4
8x8x4
8x8x8
16x 16x8
Latent channel
8
16
32
32
128
Flickering PSNR1
93.2
87.6
86.3
87.7
87.8
87.3
28.4
27.2
28.7
30.5
29
27.9
we propose to implement video compression module based 3D Variational Autoencoders Yu et al. 2023b). The idea to incorporate three-dimentional convolutions to compress Videos both spatially and temporally. This can help achieve higher compression ratio with largely improved quality and continuity of video reconstruction.
Video: Fx
Rank 6: First frams padding
(4f 8
Stage
Downormpl
Upeample
Spatial & Temporal
comY outrut
Esc Stage
Dos Stage
Rank
Send
2x Downsmmpl
Uppamppe
Stage
Dac Stags
Rew frim Beak (-I)
Scadis Rask
Downeenple
Spatial
Upearpple
Rank Rank
Rank (kI)'s
tall padding
KL Regulanzer
Dac
Enceder
Decoder
Latent:
Jumnll (aiik)
(f-1)xAXx
(galding)
(a)
(b)
Figure 4: (a) The structure of the 3D VAE in CogVideoX. It comprises an encoder, a decoder and latent space regularizer, achieving 8x8x4 compression from pixels to the latents. (b) The context parallel implementation on the temporally causal convolution.
Figure 4 (a) shows the structure of the proposed 3D VAF. It comprises an encoder, a. decoder and Kullback-Leibler (KT.) regularizer. The encoder and decoder consist of symmetrically arranged stages, respectively performing 2x downsampling and upsampling by the interleaving of ResNet block stacked stages. Some blocks perform 3D) downsampling (upsampling), while others only perform 2D downsampling (upsampling).
We adopt the lemporally causal convolution (Yu et al. 2023b), which places all the paddings at the beginning of the convolution space, shown in Figure (b). This ensures that future information does not influence the present or past predictions.
We also conducted ablation studies comparing different compression ratios and latent channels in table 1 After using 3D structures, the reconstructed vidco shows almost no more jitter, and as the latent channels increase, the restoration quality improves. However, when spatial- temporal compression too aggressive (16x16 (66x16x8), even if the channel dimensions are correspondingly increased, the convergence of the model also becomes extremely difficult. Exploring VAE with larger compression ratios our future work.
Given that processing ong-duuaaionn videos introduces excessive GPU memory usage, apply context parallel at the temporal dimension for 3D) convolution to distribute computation among multiple devices. As illustrated by Figure 4 (b), due to the causal nature of the convolution, each rank simply sends a segment of length indicates the temporal kernel size. This results in relatively low communccation overhead.
to the next rank, where
During training, we first train a 3D VAE at 256 x 256 resolution and 17 frames to save computation. Randomly select. 8 or 16 fps to enhance the model's robustness. We observe
4

---

Published as a conference paper at ICLR 2025
that the model encode larger resolution videos well without additional training as it has no attention modules, but this isn't effective when encoding videos with more frames.
Therefore, conduct two-stage training by first training 17-frame videos and finetuning by context parallel on 161-frame videos. Both stages utilize a weighted combination of the L1 reconstruction loss, LPIPS (Zhang et al. 2018) perceptual loss, and KL loss. After few thousand training stops, we additionally introduce the GAN loss from 3D discriminator.
2.2 EXPERT TRANSFORMER
We introduce the design choices in Transformer for CogVideoX, including the patching, positional embedding, and attention strategies.
Patchify. The 3D causal VAE encodes video latent of shape T H C, where T represents the number of frames, H and W represent the height and width of each frame, C represents the channel number, respectively. The video latents are then patchified, generating sequence Zvision of length I H W. images at the beginning of the sequence to enable joint training of images and videos.
p. When 9 > 1, we repeat the first frame of videos and
2024 is a relative positional encoding that has been demonstrated to capture inter-token relationships effectively in LLMs, particularly excelling in modeling long sequences. To adapt to video data, we extend the original RoPE to 3D-RoPE. Each latent in the video tensor can be represented by 3D coordinate (I, y, t). We independently apply ID-RoPE to each dimension the coordinates, each occupying 3/8, 3/8, and 2/8 of the hidden states' channel. The resulting encoding is then concatenated along the channel dimension to obtain the final 3D-RoPE encoding.
3D-RoPE. Rotary Position Embedding (RoPE) (Su
Expert Adaptive Layernorm. We con- catenate the embeddings of both text and video the imput stage to better align vi- sual and semantic information. However, the feature spaces of these two modalities dif- fer significantly, and their embeddings may even have different numerical scales. To bet- ter process them within the same sequence, we employ the Expert Adaptive Layernorm to handle each modality independently. As shown in Figure 3 following DiT (Peebles & Xiel 2023). we use the timestep of the dilfu- sion process as the input to the modulation module. Then, the Vision Expert Adaptive Layernorm (Vison Expert AdaLN) and Text Expert Adaptive Layernorm (Text. Expert AdaLNN apply this modulation to the vision hidden states and text. hidden states, respec- tively. This strategy promotes the aligument of feature spaces across two modalities while minimizing additional parameters.
speial
#Resto
Inpia
Figure 5: The separated spatial and temporal attention makes it challenging to handle the large motion between adjacent frames. In the figure, the head of the person in frame + 1 cannot directly attend to the head in frame Instead, visual information can only be im- plicitly transmitted through other background patches. This can lead to inconsistency issues in the generated videos.
3D Full Attention. Previous works
(Singer et al. 2022, Guo et al. 2023) often employ separated spatial and temporal at- tention to reduce computational complexity and facilitate fine-tuning from text-to-image models. However, as illustrated in Figure 5] this separated attention approach requires extensive implicit transmission of visual information, significantly increasing the learning complexity and making it challenging to maintain the consistency of large-movement objects. Considering the great success of long- congecontext training in LLMs (AI@Meta 2024) and the efficiency of FlashAttention (Dao et al.) 2022), propose 3D text-video hybrid attention mechanism. This mechanism not only achieves better results but can also be easily adapted to various parallel acceleration methods.
5

---

Published as a conference paper at ICLR 2025
Previous Image-Video Joint Training
Our Training Method: Multi-Resolution Frame Packing
Image
Food length video
Example:
tackle video length: len: dropped
len: clipped
144 frama video (768 7681
72-frame video 788)
48 frame video
Large gap between training tasks.
Mult-FResolution Frame Pack with 3D-RoPE
Image1
mage??
image3
image4
Batch
Image1
video1
video2
video8
pad
video
image
video
pad
Figure 6: The diagram mixed-duration training and Frame Pack. To fully utilize the data and enhance the model's generalization capability, train videos of different duration within the same batch.
3 TRAINING COGVIDEOX
We mix images and videos during training. treating each image as a single-frame video. Additionally, we employ progressive training from the resolution perspective. For the diffusion setting, adopt v-prediction (Salimans & Ho 2022 and zero SNR (Lin et al. 2024), following the noise schedule used in LDM (Rombach et al. 2022
3.1 MULTI-RESOLUTON FRAME PACK
Previous video training methods often involve joint training of images and videos with a fixed number of frames (Singer al. 2022, Blattmann al., 2023). However, this approach usually leads to two issues: First, there is a significant gap between the two input types using bidirectional attention, with images having one frame while videos having dozens of frames. We observe that models trained this way tend to diverge into two generative modes based on the token count and not have good generalizations. Socond, train with fixed duration, we have to discard short vidcos and truncate long vidcos, which prevents full utilization of the videos varying number of frames. For different resolutions, SDXL(Podell et al. 2023) uses bucketed approach to address the issue of generating cropped images, but it makes the data and training pipeline more complex.
training, which means training videos of different lengths together. However, inconsistent data shapes within the batch make training difficult. Inspired by Patch'n 'n Pack (Dehghani et al.) 2024), place videos of different duration (also different resolutions) into the same batch to ensure consistent shapes within each batch, method refer
To address these issues, chose
Multi-Resolutee Resolution Frame Pack, illustrated in Figure
We use 3D RoPE to model the position relationship of various video shape. There are two ways to adapt RoPE to different. resolutions and durations. One approach is to expand the position encoding table and, for each video, select the front portion of the table according to the resolution (extrapolation). The other is scale fixed-length position encoding table to match the resolution of the video (interpolation). Considering that RoPE is a relative position encoding, we chose the first approach to keep the clarity of model details.
3.2 PROGRESSIVE TRAINING
Videos from the Internet usually include a significant amount low-resolution ones. And directly training on high-resolution videos is extremely expensive. To fully utilize data and save costs, the model is first trained on 256px vidcos to learn semantic and low-frequency knowledge. Then it is trained gradually increased resolutions, from 256px to 512px, 768px, to learn high-frequency knowledge. To maintain the ability of generating videos with different aspect ratios, we keep the aspect ratio unchanged and resize the short side to above

---

Published as a conference paper at ICLR 2025
resolutions. Finally, we do high-quality fine-tuning, See Appendix A Moreover, trained
an image-to-video model based on above model. See Appendix for details.
3.3
EXPLICIT UNIFORM SAMPLING
Ho et al. 2020) defines the training objective of diffusion as
Lsimpio (0) == E ||e. e (V v1 VI-avee (1e,
(1)
where t is uniformly distributed between 1 and T. The common practice is for each rank in the data parallel group to uniformly sample a value between and T, which is in theory equivalent to Equation [ However, in practice, the results obtained from such random sampling are often not sufficiently uniform, and since the magnitude of the diffusion loss is related to the timesteps, this can lead to significant fluctuations in the loss. Thus, we propose to use Explicit Uniform Sampling to divide the range from 1 to T into n intervals, where the number of ranks. Each rank then uniformly samples within its respective interval. This method ensures more uniform distribution of timesteps. As shown in Figure 10 (d), the loss curve from training with Explicit Uniform Sampling is noticeably more stable.
3.4 DATA
We construct collection of relatively high-quality video clips with text descriptions with video filters and recaption models. After filtering, approximately 35M single-shot clips remain, with each clip averaging about 6 seconds. We additionally use 2B images filtered with aesthetics score from LAIŌN-5B (Schuhmann et al., 2022) and COYO-700M (Byeon al. 2022) datasets assist training.
Video Filtering. Video generation models should capture the dynamic nature of the world. However, raw video data often contains significant noise for two intrinsic reasons: First, the artificial editing during video creation can distort the true dynamic information; Second, video quality may suffer due to filming issues such as camera shakes or using subpar equipment. In addition to the intrinsic quality of the vidcos, we also consider how well the vidco data supports model training. Videos with minimal dynamic information or lacking connectivity in dynamie aspects are considered detrimental. Consequently, we have developed a set of negative labels, which include:
Editing: Videos that have undergone noticeable artificial processing, such as re- editing and special effects, which compromise the visual integrity.
Lack of Motion Connectivity: Video segments with transitions that lack coherent motion, often found in artificially spliced vidcos or those edited from static images.
Low Quality: Poorly shot videos with unclear visuals or excessive camera shake.
Lecture Type: Videos focusing primarily on a person continuously talking with minimal effective motion, such lectures, and live-streamed discussions.
Text Dominated: Videos containing large amount of visible text primarily focusing on textual content.
Noisy Screenshots: Videos captured directly from phone or computer screens, often characterized by poor quality.
We first sample 20,000 videos and label each video as positive or negative by their quality. Using these annotations, we train 6 filters based on Video-LLaMA (Zhang et al. 2023b) to screen out low-quality video data. Examples of negative labels and the classifier's pertormance on the test set can be found in appendix K In addition, we calculate the optical flow scores and image aesthetic scores of all training videos, and dynamically adjust their threshold during training to ensure the dynamic and aesthetic quality of generated videos.
Video Captioning. Video-text pairs are essential for the training of text-to-video genera- tion models. However, most video data does not come with corresponding descriptive text.
7

---

Published as a conference paper at ICLR 2025
Therefore, it is necessary to label the video data with comprehensive textual descriptions. There are some video caption datasets available now, such as Panda70M (Chen et al. 2024b), COCO Caption (Lin et al. 2014), and WebVid Bain et al. (2021b). However, the captions in these datasets are usually very short and fail to describe the video comprehensively.
Input Video
Separated Frames
GPT-4
Image Recaption model
Image Captions
Summary
CogVLM
Distilation
Output: Long Video Caption
Short Video Caption Model
Finetuned Liama2
Short Caption
Finetune
CogVLM2-Video
Version1 Data Path
Version2 Data Path
Figure 7: The pipeline for dense video caption data generation. In this pipeline, we generate short video captions with the Panda70M model, extract frames to create dense image captions, and use GPT-4 to summarize these into final video captions. To accelerate this process, we fine-tuned Llama 2 model with the GPT-4 summaries.
To generate high-quality video caption data, we establish a Dense Video Caption Data Generation pipcline, as detailed in Figure The main idea to generate video captions with the help of image captions.
(2024b) to generate short captions for the videos. Then, employ the image recaptioning model CogVLMM (Wang et al. 2023ab used in CogView3 (Zheng et al. (2224a) to create dense image captions for each frame. Subsequently, we use GPT-4 to summarize all the image captions to produce the final video caption. To accelerate the generation from image captions to video captions, we fine-tune LLaMA2 (Touvron et al. 2023 using the summary data generated by GPT-4 (Achiam cet al. 2023 enabling largo-scale vidco caption data generation. Additional details regarding the video caption data generation process can be found in Appendix G
First, we use the video caption model from Chen
To further accelerate video recaptioning, we also fine- the-tune an end-to-end video understand- based the C22VIMM-Videe (Hong et al. 2024) and Llama3 (ATeMeta, 2024), by using the dense caption data generated from the aforemen- tioned pipeline. Examples of video captions generated by this end-to-end CogVL.M2-Caption model are shown in fig. 15 and Appendix H CogV LM2-Caption can provide detailed descrip- tions of video content and changes. Interestingly, we find that can perform video-to-video generation by connecting CogVideoX and CogVLM2-Cappe Caption, as detailed in appendix
ing model CogVLM2-Capple
4 EXPERIMENTS
4.1 ABLATION STUDY
We conducted ablation studies some of the designs mentioned in Section 2 to verify their effectiveness.
Position Embedding. We compared 3D RoPE with sinusoidal absolute position embed- ding. As shown in Figure 10a indicates the loss curve of RoPE converges significantly faster than absolute onc. This is consistent with the common choice in LLMss.
Expert Adaptive Layernorm. We compare three architectures in Figure Sa. g+ and Figure 10c MMDIT Easer et. al. (2024), Expert AdaL (CoVieCC)))))oo)) without Expert AdaLN.
8

---

Published as a conference paper at ICLR 2025
Figure& 8: Ablation studies on WebVid test dataset with 500 videos. MMDiTI has the same
number of parameters with the expert AdaLN. MMDIT2 has the same mumber of layers but twice number of paramcters. a, b, c mcasure FVD, measures CLIP4Clip score.
Cross-attention DiT has been shown to be inferior to MMDiT in (Esser et al. 2024), so we don't repeat. According to FVD, CLiP al.) (022) Score and loss, expert AdaLN significantly outperforms the model without expert A and MMDiT with the same number of parameters. We infer that expert adaptive layernorm is enough alleviate the difference in feature space between the two modalitics. So two independent transformers in MMDiT are not necessary, which greatly increases number of parameters. Moreover, the design of Expert AdaLN is more simplified than MMDiT and is closer to current LLMs, making it easier to scale up further.
Figure 8D and Figure 10b when we replace 3D full attention with 2D + 1D attention, the FVD will become much higher than 3D attention in early steps. We also observe that. 2D+1D is unstable and prone to collapse. We suppose that as the model size increases, such as 5B, training becomes more prone instability, placing higher demands on the structural design. The 2D+1DD structure, as discussed section 2.2 is not suitable for video generation tasks, which could lead instability during training.
3D Full Attention.
Explicit Uniform Sampling. From Figure Sp and Figure 10d we find that using Explicit Uniform Sampling can make more stable decrease in loss and get a better performance. In addition, in Table g we compare the loss each diffusion timestep alone between two choices for more precise comparison. We find that the loss at all timesteps is lower with explicit wniform sampling, indicating that this method can also accelerate loss convergence. We suppose that this is because the loss of different limesteps varies greatly. When the timesteps sampled for training are not uniform enough, the loss fluctuates greatly due to the above randomness. Explicit uniformity can reduce randomness, thereby bringing common decrease in all timesteps.
4.2 EVALUATION
4.2.1 AUTOMATED METRIC EVALUATION
VAE Reconstruction Effect We compared 3DVAE with other open-source 3DVAE on 256 x 256 resolution 17-frame videos, using the validation set of the WebVid (Bain et al. 2021ah. On table 2 our VAE achieved the best PSNR and exhibited the least jitter. Notably, other VAE methods use fewer latent channels than ours.
Evaluation Metrics. To evaluate the text-to-viden generation, we employ several metrics in Vbench (Huang et al. 2024) that are consistent with human perception: Hu- man Action, Scene, Dynamic Degree, Mul- tiple Objects, and Appearance Style. Other metrics, such as color, tend to give higher scores to simple, static videos, so we do not them.
Table 2: Comparison with the performance of other spatiotemporal compression VAEs.
Flickering
PSNR
Open-Sora Open-Sone Open-Sora-Plan
92.4
28.5
90.2
27.6
Ours
85.5
29.1

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation results of Cog CogVideoX-5B and Cog CogVideoX-2B.
Human Action
Dynamic Multiple Appear.
Dynamic 4PT4 Quality
Models
Scene
Degree
Objects
Style
Score
T2V- ct al. 2024
95.2
55.58
49.17
54.65
24.42
AnimateDif Guo ct al 2023
92.6
50.19
40.83
36.88
22.42
2.62
Video Ch et al. 2024a
95.0
55.29
42.50
40.66
25.13
43.6
2.68
OpenSora V1.2 Zheng et al 2024b
85.8
42.47
47.22
58.41
23.89
63.7
2.52
Show-1 (Zhang el al. 2023a
95.6
47.08
44.44
45.47
23.06
57.7
2en-2 2023
89.2
48.91
18.89
55.47
19.34
43.6
2.62
Pikajpık 2023
88.0
44.80
37.22
46.69
21.89
52.1
2.48
La Vie- 2annn et al.
2023b
96.4
49.59
31.11
64.88
25.09
2.46
Cog Vi VideoX-22
96.6
55.35
66.39
57.68
24.37
57.7
3.09
CogVideon V-d
96.8
55.44
62.22
70.95
24.44
69.5
3.36
For longer-generated videos, some models might produce videos with minimal changes between frames to get higher scores, but these videos lack rich content. Therefore, metrics for evaluating the dynamism become important. To address this, use two video evaluation tools: Dynamic Quality (Liao al. 2024) and GPT40-
(Yuan et al. 2024).
Dynamic Quality is defined by the integration of various quality metrics with dynamic scores, mitigating biases arising from negative correlations between video dynamics and video quality. OPPTo-MTScoR is metric designed to measure the metamorphic amplitude of time-lapse videos using GPT-4o, such as those depicting physical, biological, and meteorological changes.
Results. Table provides the performance comparison of CogVideoX and other models. CogVideoX-5B achieves the best performance in five out of the seven metrics and shows competitive results in the remaining two metrics. These results demonstrate that the model not only excels in video generation quality but also outperforms previous models in handling various complex dynamic scenes. In addition, Figure presents a radar chart that visually illustrates the performance advantages of CogVideoX. We present the time and space consumption during inference at different resolutions in appendix
4.2.2 HUMAN EVALUATION
In addition to automated scoring mechanisms, also establish comprchensive human evaluation framework to assess the general capabilities vidco generation models. Evaluators will score the generated videos four aspects: Sensory Quality, Instruction Following, Physies Simulation, and Cover Quality, using three levels: 0, 0.5, or 1. Each level is defined by detailed guidelines. The specific details are provided in the Appendix [
We compare Kling (2024.7), one of the best closed-source models, with CogVideoX-5B under this framework. The results shown in Table indicate that Cog VideoX-5B wins the human preference over Kling across all aspects.
Table 4: Human evaluation between Cog VideoX and Kling.
Sensory Quality
Instruction Following
Physics Simulation
Cover Quality
Total Score
Model
Kling
0.638
0.367
0.561
0.668
2.17
Cog CogVideoX-5B
0.722
0.495
0.667
0.712
2.74
5 CONCLUSION
In this paper, we present CogVideoX, a state-of-the-art text-to-video diffusion model. It leverages 3D VAE and Expert Transformer architecture to generate coherent long duration videos with significant motion. We are also exploring the scaling laws of video generation models and aim to train larger and more powerful models to generate longer and higher-quality videos, pushing the boundaries of what is achievable in text-to-video generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTZ
This work is supported by NSFC 62425601 and 62495063, Tsinghua University Initiative Scientific Rescarch Program 20233080067, New Cornerstone Science Foundation through the XPLORER PRIZE. We would like thank all the data annotators, infrastructure operators, collaborators, and partners. We also exlend our gratitude to everyone at Zhipu Al and Tsinghua Universily who have provided support, leedback, or contributed the CogVideoX, even if not. explicitly mentioned in this report. We would also like to greatly thank BiliBili for technical discussions.
REFERENCES
Pika beta. 2023. URL httpss://pkka: art art.h//
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Nlge Akkaya, Florencia. Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 0070:333300777 2023.
Al@Meta. Llama 3 model card. 2024. URL blob/maahn/KKDELLCCRRR
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021a.
Max Bain, Arsha Nagrani, Gil Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CC CCF international conference on computer vision, pp. 1728 1738, 2021b.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjic Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. htlps://odhs openai. com/popers/ dall-e-3. /apers//all-e--,
pdf, 2(3):8, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint or xixi::2311.1 11111177, 2023.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungiun Lee, Woonhyuk Back, and Sac- hoon Kim. 1oyo-700m: Image-text, pair dataset. https://github.ccom wwt..//oaakaaaaii//// coyo-dataset 2022.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVe /CVF Conference on Computer Vision and Pattern Recognition, pp. 7310 7320, 2024a.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wer Chao, Byung Eun Jeon, Yuwci Fang, Hsin-Ying 70m: Captioning 70m vidcos with multiple cross-modality teachers. In Proceedings of the IEEE/ (CVF Conference on Computer Vision and Pattern Recognition, pp. 13320 13331, 2024b.
Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344- 16359, 2022.
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim M Alabdul- mohsin, et al. Patch n aapack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Newral Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müllrr, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized lext-to-image diffusion models without specific tuning. arXiv preprint or orXi::2307.00 04755, 2023.
Jonathan Ho, Ajay Jain, and Picter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arviv preprinl ar arXiv:220000230 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for texl-to-video generation via transformers. arxiv preprinl arXiv:2205.11 15868, 2022.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, et al. Cogylm2: Visual language models for image and video understanding, orXie preprint
Stt:2408.16500 2024.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVE EVF Conference on Computer Vision and Pattern Recognition, 2024.
PKU- Lab and Tuzhan AI etc. Open-sora-plan, April 2024. URL https://doi org/ 10. www.52222eenoo.oo 10948109
Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2v-turbo: Breaking the quality bottleneck of video consistency model with mixod reward feedback. arXiv preprint arXiv:2405. 18750, 2024.
Mingxiang Liao, Hannan Lu, Xinyu Zhang, Fang Wan, Tianyu Wang, Yuzhong Zhao, Wangmeng Zuo, Qixiang Ye, and Jingdong Wang. Evaluation of tcxt-to-video generation models: A dynamics perspective, 2024. URL attts:://axxiv wrg/abs////a/oaaoooooooooee htggabbs2407.000944
Shanchuan Lin, Bingehen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 5404-5411, 2024.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end vidco clip retricval and captioning. Neurocomputing, 508:293 304, 2022.
OpenAI. Sora. 2024. URL https::/ wttp://aw....m//e///////rr/o
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFF/ CVF International Conference on Compuler Vision, pp. 4195-4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. ar Xix preprint Xrv:23307.00 01 01952, 2023.
12

---

Published as a conference paper at ICLR 2025
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the CEEE/CVF conference on computer vision and pattern recognition, pp- 10684-10695, 2022.
runway. Gen-2. 2023. URL http://s//muuaapp....ooooooo/oo/
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint orXiv:22220005 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katla, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 252224 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make- aivieo: Text-to-vider generation without text-video data. arXiv preprint arxiv:2209. 1479,, 2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09255 09807, 2023.
Sergey Tulyakov, Ming-Ye Yu Liu, Xiaodong Yang, and Jan Kantz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1526 1535, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference Learning Representations, 2022.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvim: Visual expert for pretrained language models. arXw preprint Xiw:2311.00070 2023a.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan Hc, Jiashuo Yu, Peiqing Yang, et al. Lavic: High-quality video generation with cascaded latent diffusion models. arXiu preprint arXtx:2300 15103, 2023b.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using va-vae and transformers. arXiv preprint arXhm:2104.10151 2021.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Hing-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CV CCVF Conference on Computer Vision and Pattern Recognition, pp. 10459, 2023a.
Lijun Yu, José Lezama, Nitesh Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats -iffusion-tokenizeer tokenizer key to visual generation. Xrxiv preprint arx0:23310005797 2023b.
13

---

Published as a conference paper at ICLR 2025
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Wor Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awan aware implicit generative adversarial networks. ar preprint wrXi:2200.10577 2022.
Shenghai Yuan, Jinfa Huang, Yongqi Xu, Yaoyang Liu, Shaofeng Zhang, Yujun Shi, Ruijie Zhu, Xinhua Cheng, Jiebo Luo, and Li Yuan. Chronomagic-benche A benchmark for meta- morphic evaluation of to-t----imeee video generation. arXiv preprint arXix:246.. 18522, 2024.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint XrX:::2309.158 2023a.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuncl audio-visual language model for video understanding, arXiv preprint arXiv:2306.0255 02858, 2023b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Wendi Zheng, Jiayan Teng, Zhuoyi Yang. Weihan Wang, Jidong Chen, Xiaotao Gu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogview3: Finer and faster text-to-image generation via relay diffusion. arXiv preprint arx: 03.05121,, 2024a.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024b. URL https: // thub wttphh/..wwa/ppppaatcc/aaaaaaaaa
14

---

Published as a conference paper at ICLR 2025
APPENDIX CONTENTS
Appendix A: Training Details
Appendix B: Loss Curve
Appendix C: More Examples
Appendix D: Image To Video Model
Appendix E: Related Works
Appendix F: Caption Upsampler
Appendix G: Dense Video Caption Data Generation
Appendix H: Video Caption Example
Appendix I: Video to Vidco via CogVidcoX and CogVLM2-Caption
Appendix J: Human Evaluation Details
Appendix K: Data Filtering Details
A TRAINING DETAILS
High-Quality Fine-Tuning, Since the filtered pre-training data still contains certain proportion of dirly data, such as subtitles, watermarks, and low-bitrate videos, we selected a subset of higher quality video data, accounting for 20% of the tofal dataset, for fine-tuning in the final stage. This step effectively removed generated subtitles and watermarks and slightly improved the visual quality, However, we also observed slight degradation in the model's semantic ability.
Visualizing different rope interpolation methods When adapting low-resolution position encoding to high-resolution, we consider two different methods: interpolation and extrapolation. We show the effects of two methods in Figure Interpolation tends to preserve global information more effectively, whereas the extraaooation better retains local details. Given that RoPF. is relative position encoding, We chose the extrapolation to maintain the relative position between pixels.
RoPE Exteapolation
RoPl Interpolstion
Figure 9: The comparison between the initial generation states of extrapolation and interpo- lation when increasing the resolution with RoPE. Extrapolation tends to generate multiple small, clear, and repetitive images, while interpolation generates a blurry large image.
Model & Training Hyperparameters eters in table 5 and table 0
We present the model and training hyperparam-
15

---

Published as a conference paper at ICLR 2025
Training Stage
stagel
stage2
stage3
stage4 (FTT)
Max Resolution
256x384
480x 720
768 1360
768 1360
Max duration
10s
10s
Batch Size
2000
1000
250
100
Sequence Length
25k
75k
700k
700k
Training Steps
400k
220k
120k
10k
Table Hyperparameters of CogvideoX-2h and Cog Video-5b.
Hyperparameter
CogvideoX-2b
Cog Video-5b
Number of Layers
30
Attention heads
32
48
Hidden Size
1920
3072
Position Encoding
sinusoidal
RoPE
Time Embedding Size
256
Weight Decay
le-1
Adam
1e-8
Adam $1
0.9
Adam
0.95
Learning Rate Decay
cosine
Gradient Clipping
1.0
Text Length
226
Max Sequence Length
82k
Lowest acsthetic-value
4.5
Training Precision
BF16
Table 6: Hyperparameters of CogvideoX-2b and Cog VogVi00--5b.
5b-480x720-6s
5b-768x1360-5s
2b-480x720-6s
2b-768x1360-5s
Time Memory 26GB
113s
500s
49s
220s
76GB
18GB
53GB
Table 7: Inference time and memory consumption of CogVideoX. We evaluate the model on bf, H800 with 50 inference steps.
256*384*6s
480*720*6s
768*1360*5s
2D+1D
0.38s
1.26s
4.17s
3D
0.41s
2.11s
9.60s
Table Inference time comparison between Full allention and 2D+1D 1D altention. We evaluate the model on bf. H800 with one dit forward step. Thanks to the optimization by Flash Attention, the increase in sequence length does not make the inference time unacceptable.
16

---

Published as a conference paper at ICLR 2025
B Loss
Eaft
(a) RoPE Sinusoidal
(b) 3D vs. 2D+1D 1D Attention
(c) Different Architecture
(d) w/ vs. w/o Explicit Uniform
Figure 10: Training loss curve of different ablations.
Table Validation loss different diffusion limesteps when the training steps 40k.
Timestep
100
300
500
700
900
w/o explicit uniform sampling
0.222
0.130
0.119
0.133
0.161
w/ explicit uniform sampling
0.216
0.126
0.116
0.129
0.157
C MORE EXAMPLES
More text-to-video examples are shown in Figure 11 and Figure
12
D IMAGE To VIDEO MODEL
We finetune an image-to-video model from the text-to-video model. Drawing from the Blattmann et al. 2023), we add an image as an additional condition alongside the text. The image is passed through 3D VAE and concatenated with the noised input in the channel dimension. Similar to super-resolution tasks, there is significant distribution gap between training and inference (the first frame of videos vs. real-world images). To enhance the model's 's robustness, we add large noise to the image condit ion during training. Some examples are shown in Figure 13 Figure 14 Cog VideoX can handle different styles of image input.
E RELATED WORKS
Video diffusion models Generating videos has been explored through various types of generative models, such as Generative Adversarial Networks (GANs) Yu et al. [022 Tulyakov et al. 2018), autoregressive methods (Hong et al.] 2022 Yan et al. 2021], and non- autoregressive methods (Villegas et al. 2022 Yu et al. 2023a). Diffusion models have recently gained significant attention, achieving remarkable results in both image generation (Rombach et al. 2022 Esser et al. 2024) and video generation (Sinerr et al.) 2022: Blattmann et al. 2023 Guo et al. (023). However, the limited compression ratio and simple training strategy often restrict the generation to low-resolution short-duration videos (2-3 seconds), requiring multiple super-resolution and frame interpolation models to be cascaded
rascaded(Singer (Singer et al. 2022 Ho et al., (0022) for a generation. This leads to generated videos with limited semantic information and minimal motion.
17

---

Published as a conference paper at ICLR 2025
To increase the compression ratio of videos and reduce computation costs, a common approach is to encode the video into a latent space using a Variational Autoen- (VAe), which is also widely used in image generation. Early video models usually directly use image VAF for generation. However, modeling only the space dimension can result. in jittery videos. SVDD(Blattmann et. al. 2023) tries to finetune the image VAE decoder to solve the jittering issue. However, this approach cannot take advantage of the tempo- ral redundancy in videos and still cannot achieve an optimal compression rate, Recently, some vidco models(Zheng et al. 2024b Lab & etc., 20224) try to use 3D VAE for temporal compression, but small latent channels still result blurry and jittery videos.
Video VAEs
18

---

Published as a conference paper at ICLR 2025
Text Preeepti A fow goldco retriveers plaing in the slow
Text Prosept: geldes sunglasaes hair, many roollop 'ackground, cuns centers
Procugt: delphins leop splash
Text Prompt: The cataAza sutaaice aooudd stack vintage televisions that show a variety of programt 1850 sci-3 movics, boene zuvio, sowa, aill, 1970s aituuns, dic, act in laage gallery the New York Mascum.
Text M Musheoom tirms
A
Figure 11: Text to video showcases. The displayed prompt will be upsampled before being fed into the model. The generated videos contain large motion and various styles.
19

---

Published as a conference paper at ICLR 2025
tronge tpward angle. slowly up, dragon suddenty appears toeberg, dragon spors you towards Hollywood movie style
Pronpp:: old fd-fssikoned autorooblee
Republic. While driving banths suddently fail belownn
up, the people blown up, movie winds
Teat Aaar
Prosepe: vintage apezda up pine stopp votuxies The spesis nubwods, pacches of earm it thive The itelf unmorded hillesne muntain, clear blue day showe claude
pantl following the with
Prungt:: by window, mar and slowty coffee slp eyes lonicing
doop Bhought.
Figure 12: Text to video showcases.
20

---

Published as a conference paper at ICLR 2025
Pronuet
Teat Prompe: Flowers grow
Text Prompte A raging turaaii flooded the village
Thet Phonept: A draggn' munch chaste out flames and hurras clomm a enall village
Text The undle and nophow looking st ruth other und then umiling and cebbnacing to cochath other
Figure 13: Image video showcases. The displayed prompt will be upsampled before being fed into the model.
21

---

Published as a conference paper at ICLR 2025
Text Peoergt: Aa clephant soonly walks out cloud of log, tae log shations and Dows
Primpi: through
breaking surfoce, splasting water, light partieles rundaity fly, leaves sway
Procupi: git
rubs againut puppy, pappy up
Teet
sharter her hair
Text Pronpti pappy cleaes syes, opena ik mouth, and
Figure 14: Image to video showcases.
22

---

Published as a conference paper at ICLR 2025
F CAPTION UPSAMPLER
To ensure that text imput distribution during inference is as close as possible to the distribution during training, similar to (Betker et al. 2023), use large language model to upsample the user's input during inference, making it more detailed and precise. Finctuned LLM can generate better prompts than zcro few-shot.
For image-to-video, we use the vision language model to upsample the prompt, such as GPT4V, PreVLM[Wange et al. 2023a
Zero-shot prompt for Text Upsampler
You are part of a team of bots that create videos. You work with assistant bot that will draw anything you say in square brackets. For example, outputting \" a beautiful morning in the woods with the peaking through the trees \" will trigger your partner bot to output a video described. You will prompted
of forest morning, by people looking to create detailed, amazing videos. The way accomplish this to take their short prompts and make them extremely detailed and descriptive.
There are a few rules to follow
You will only ever output single video description per user request.
When modifications are requested, you should not simply make the description longer. You should refactor the entire description to integrate the suggestions.
G DENSE VIDRO CAPTION DATA GENERATION
In the pipeline for generating video captions, we extract one frame every two seconds for image captioning. Ultimately, we collected 50,000 data points to fine-tune the summary model. Below is the prompt we used for summarization with GPT-4:
Prompt for GPT-4 Summary
We extracted several frames from this video and described each frame using an image understanding model, stored Dict(str: s
in the dictionary variable In Image_captions appears in the video, and the value is detailed description of the image at that moment. Please describe the content of this video in much detail possible, based the information provided by simage_captions', including the objects, scenery, animals, characters, and camera movements within the video. \n You should output your summary directly, and not mention variables like
the key is the second at which the image
in your response.
Do not include '\n' and the word 'video' in your response. Do not use introductory phrases such as: \"The video presents\", "TThe video depicts\", '"Tis video showcases\", 'TThe video captures)" and on. An Please start the description with the video content directly, such as \"A man first sits in chair, then stands up and walks to the kitchen.... \"An Do not use phrases like: \aas the video progressed)" and \ \TThroughout the video\". An Please describe
23

---

Published as a conference paper at ICLR 2025
the content of the video and the changes that occur, in chronological order. An Please keep the description of this video within 100 English words.
H VIDEO CAPTION EXAMPLE
Below we present more examples to compare the performance the Panda-70M video captioning model and our CogVLM2-Caption model:
A brown ceramic pot with a green sprouting plant is placed on white windowsill, illuminated by natural light from open window showing brick wall and lush outdoor scenery, The scene transitions to the flower full bloom, itss pink petals contrasting against nst the dark blue curtain backdrop. Als moments pass, the flower's appearance evolves, showcasing vibrant yellow centers and delicate ruffled edges, enhancing the serene atmosphere of the room. The setting remains consistent, emphasizing tranquility and domesticity through the gentle play of sunlight and shadow across the indoor space.
Figure 15: An example from CogVLLM2CCaption provides a detailed description of all specific objects and movements.
Caption Generated by Panda-70M
close up a white dragon with blue eyes.
Caption Generated by CogVLM2-Caption
serene expression and piercing blue eyes is showcased in various close-ups, its scales detailed with golden highlights. Initially, the dragon's horns are curved back, but time passes, they change to spiraling gold, adding elegance. The creature's demeanor shifts from tranquil majesty to more formidable presence, with its mouth agape revealing sharp teeth at times. Its expressions alternate between calmness and fierce determination, maintaining an air of mythical serenity throughout. The consistent neutral background focuses attention on the dragon's
meticulously crafted white dragon
24

---

Published as a conference paper at ICLR 2025
textured features and expressive eyes.
Caption Generated by Panda-70M
crab is walking on beach with a light bulb on back.
Caption Generated by CogVLM22Cappionn
hermit crab with dark, glossy shell reddish-broo legs is seen carrying illuminated light bulb its back across the sandy terrain of a beach at night. The scene transitions the crab
from soft glow to more pronounced illumination moves, highlighting its delicate 1imbs against the smooth sand and tranquil sea backdrop. This surreal tableau blends natural beauty with human ingenuity, creating serene yet whimsical atmosphere that emphasizes the crab's unique adaptation and the contrast between nature and technology in this quiet nocturnal setting.
Caption Generated by Panda-70M
A young black man is sitting on a cloud and reading a book with blue sky in the background.
Caption Generated by CogVLM2-Caption
young Black man with an afro hairstyle and neatly trimmed beard is seen sitting cross-legged on fluffy white clouds, deeply engrossed in reading book with red cover. wears plain white T-shirt and dark pants against a vivid blue sky dotted with cumulus clouds. Throughout the scenes, his expression remains one of deep concentration and peaceful contemplation, highlighting amidst nature's grandeur. The imagery suggests serene atmosphere that emphasizes solitude and introspection, with no other people or objects around him.
moment intellectual pursuit
25

---

Published as a conference paper at ICLR 2025
I VIDEO TO VIDEO VIA COGVIDEOX AND COGVLM2-CAPTION
In this section, we present several examples video-to-vides generation using VogVideoX and CogVLM2-Caption. Specifically, first input the original video into CogWLMM-Cappton to obtain the vidco's caption, and then feod this caption into the CogVidcoX model to generate new video. From the examples below, can be seen that our pipeline achieves high degree of fidelity to the original video, showing that CogVLM2-Caplion can capture almost all the details in the video.
Source Video
Caption Generated by CogVLM2-Caption
A man dressed wizard, with blue robe adorned with white stars and matching pointed hat, stands in dark cave. He is engaged in casting spells from open book, surrounded by mystical tlames that Illuminate the scene. Throughout the sequence, his right hand is raised to channel bright lightning bolts towards unseen targets, while his left hand holds the book firmly. His expression remains focused and serious, suggesting deep concentration on his magical endeavors. The atmosphere of mystery and ancient magic is enhanced by the surrounding darkness and the vivid display light and shadow.
Video Generated by CogVideoX
26

---

Published as a conference paper at ICLR 2025
Source Video
Caption Generated by CogVLM2-Caption
A picturesque evening descends on cliffside village, showcasing whitewashed buildings with blue domes that glow against the darkening ng sky. The Aegean Sea mirrors this celestial hue, creating a serene tableau devoid of people and vehicles. As time passes, the scene remains tranquil, illuminated by golden lights from within homes and lit pathways weaving between structures. A solitary windmill stands out, symbolizing local culture amidst peaceful setting. The absence visible human activity emphasizes stillness beauty of the coastal hamlet, inviting contemplation embrace.
Video Generated by CogVideoX
Source Video
Caption Generated by CogVLMM2-Caption
woman's 's eye, sharp focus and detailed with bold black eyeliner, reflects the Earth. vivid colors of blue oceans green continents stand out against clear iris, symbolizing deep connection between humanity our planet. Her expression remains introspection awareness. time passes, the reflection subtly shitts to include parts of Africa and Europe, suggesting global perspective. The contrast between her dark eyelashes and light skin accentuates the visual metaphor for unity and intercannectednnss, while her gaze suggests contemplation on environmental issues or profound sense of responsibility towards the world.
throughout, emphasizing
Video Generated by CogVideoX
27

---

Published as a conference paper at ICLR 2025
J. HUMAN EVALUATION DETAILS
One hundred meticulously crafted prompts are used for human evaluators, characterized by their broad distribution, clear articulation, and well-defined conceptual scope.
panel of evaluators instructed to assign scores for each detail on a scale from zero to one, with the overall total score rated on scale from 0 to 5, where higher scores reflect better video quality.
To better complement automated evaluation, human evaluation emphasizes the instruction- following capability: the total score cannot exceed the generated video fails to follow the instructions.
This part focuses mainly on the perceptual quality of videos, including subject consistency, frame continuity, and stabilily.
Sensory Quality:
Table 10: Sensory Quality Evaluation Criteria.
Score
Evaluation Criteria
1
High sensory quality: 1. The appearance and morphological features of objects in the video completely consistent 2. High picture stability, maintaining high resolution consistently Overall composition collo// cwwor/boooooooee boundaries match reality The picture is visually appealing
0.5 Average sensory quality:
The appearance and morphological features of objects the video are at least 80% consistent 2. Moderate picture stability, with only 50% of the frames maintaining high resolution 3. Overall composi- tion/color/ aalorr/bounee boundaries match reality by at least 70% 4. The picture has some visual appeal
0
Poor sensory quality: large inconsistencies in appearance and morphology, low video resolution, and composition/laye /layout not matching reality
Instruction Following:
This part focuses on whether the generated video aligns with the prompt, including the accuracy of the subject, quantity, elements, and details.
Table 11: Instruction Following Exaluation Criteria.
Score
Evaluation Criteria
1
100% follow the text instruction requirements, including but not limited to: elements completely correct, quantity requirements consistent, elements com- plete, features accurate, etc.
0.5
100% follow the text instruction requirements, but the implementation has minor flaws such as distorted main subjects or inaccurate features.
0 Does not 100% follow the text instruction requirements, with any of the following issues: 1. Generated elements are inaccurate Quantity incorrect 3. Elements are incomplete Features are inaccurate
This part focuses whether the model can adhere to the objective law the physical world, such as the lighting effect, interactions between different objects, and the realism of fluid dynamics.
Physics Simulation:
28

---

Published as a conference paper at ICLR 2025
Table 12: Physics Simulation Evaluation Criteria.
Score
Evaluation Criteria
1
Good physical realism simulation capability, can achieve: 1. Real-time tracking
2. Good action underslanding, ensuring dynamic realism of entities Realistic lighting and shadow effects, high interaction fidelity 4. Accurate simulation of fluid motion
0.5 Average physical realism simulation capability, with some degradation in real- time tracking, dynamic realism, lighting and shadow effects, and fluid motion simulation. Issues include: 1. Slightly unnatural transitions in dynamic effects, with some discontinuities 2. Lighting and shadow effects not matching reality 3. Distorted interactions between objects 4. Floating fluid motion, not matching reality
Poor physical realism simulation capability, results do not match reality, obvi- ously fake
Cover Quality: This part mainly focuses on metrics that can be assessed from single-frame images, including aesthetic quality, clarity, and fidelity.
Table 13: Cover Quality Evaluation Criteria.
Score
Evaluation Criteria
Image is clear, subject is obvious, display is complete, color tone is normal.
0.5 Image quality is average. The subject is relatively complete, color tone is normal.
0 Cover image resolution is low, image is blurry.
K DATA FILTERING DETAILS
In order to obtain high-quality training data, we designed set of negative labels to filter out low-quality data. Figure [6 presents our negative labels along with sample videos for cach label.In table 14 present the accuracy and recall of our classifier, trained based video-Ilama, on the test set (10% randomly labeled data).
Table 14: Summary of Classifies Performance on the Test Set. TP: True Positive, FP: False Positive, TN: True Negative, FN: False Negative.
Classifier
TP
FP
TN
FN
Test Aec
Classifier Editing
0.81
0.02
0.09
0.08
0.91
Classifier Static
0.48
0.04
0.44
0.04
0.92
Classifier Lecture
0.52
0.00
0.47
0.01
0.99
Classifier Text
0.60
0.03
0.36
0.02
0.96
Classifier Screenshot
0.61
0.01
0.37
0.01
0.98
Classifier Low Quality
0.80
0.02
0.09
0.09
0.89
29

---

Published as a conference paper at ICLR 2025
Japanese Eumy Rior
Editiig
frst
CANA
Lack of Motion Connectivity
Low Quality
FAN PUTN
Type
The Now Thi Thinking Allowed Foundation
Thi Howed Foundation
NEW THINKING ALLOWED
NEW THINKING ALLOWED
Text Dominated
Noisy Screenshots
Figure 16: Examples of negative labels for video filtering.
30


Reviews:
[{"review_id": "7INBUbMFtR", "reviewer": "Reviewer_DUFK", "summary": "The paper introduces CogVideoX, an advanced text-to-video generation model built on a diffusion transformer. It produces 10-second, high-resolution videos (768×1360) at 16 fps. To overcome challenges in video generation, the authors implement:\nA 3D Variational Autoencoder (VAE) for better compression and video quality.\nAn Expert Transformer with adaptive LayerNorm to enhance text-video alignment.\nProgressive Training and Multi-Resolution Frame Packing to create coherent, long-duration videos with significant motion.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. 3D-RoPE for Video Data: The adaptation of Rotary Position Embedding (RoPE) to 3D (3D-RoPE) is novel, effectively capturing spatiotemporal relationships and adding originality to positional encoding.\n2. The qualitative visualizations showcase various video domains, including scenes, single-object videos, and multi-object videos.", "weaknesses": "1. The paper highlights CogVideoX’s performance but lacks a detailed analysis of computational efficiency, including memory usage and training/inference time.\n2. The paper does not discuss the scalability of CogVideoX to longer video durations beyond 10 seconds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U9Jwpi4raq", "reviewer": "Reviewer_QWeS", "summary": "This paper is a text-to-video generation model using a diffusion transformer that generates 10-second, high-resolution videos aligned with text prompts without super-resolution or frame-interpolation. It improves upon previous models through a 3D causal VAE for text-video alignment, coherent, long-duration, and motion-rich videos. It achieves state-of-the-art performance and is open-source", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I believe this paper is good, an open-source and open-hyperparameters, will have a impact on community research.\n\n1. The paper is well written.\n2. This paper creates a well-defined dataset for text-to-video generation.\n3. This paper publicly release 5B and 2B models, including text-to-video and image-to-video versions.\n4. This paper achieves state-of-the-art performance compared with other text-to-video models.\n5. The generated high-resolution videos have very good quality.", "weaknesses": "1. The training process and model structure are somewhat unintuitive and slightly complex, raising some concerns about performance improvement in the future.\n2. There is a lack of detailed analysis on the ablation study. If there were a more detailed analysis on the ablation study, I would raise the score.\n3. Why is 2D + 1D attention unstable? In Figure 8, is the X-axis FVD and the Y-axis Training Steps?\n4. In 459 line, () is typo ?\n5. In 475 line, 17 frame is right ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TM8qyrAj8X", "reviewer": "Reviewer_Y5bt", "summary": "This paper proposes a text-to-video generation model focused on generating video with temporal consistency and rich motion in longer sequence. It proposes a 3D VAE for video compression, achieving high-quality reconstruction, and introduces a new Transformer architecture to enhance semantic alignment between text and video. Experimental results show that CogVideoX outperforms existing models, especially in complex dynamic scenes. With open-source release, CogVideoX has potential to advance research in video generation.", "questions": "* In the paper, ablation studies have been evaluated with only FVD scores. However, for Expert AdaLN, which focuses on alignment between text and video data, it would be reasonable to include other metrics, such as the CLIP score, to provide more robust validation of the ablation study results.\n\n* What causes the 2B model to underperform compared to other baselines and the 5B model except dynamic degree? \n\n* Can the model generate similar-quality videos without using the caption upsampler? The caption upsampler may hinder robust generalization performance.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* Easy to follow \n* Effective apporach to text-to-video generation based on diffusion transformer \n* Demonstration of high-quality video", "weaknesses": "* Overall, the performance improvement does not appear to be significant. For example, the CogVideoX-2B model outperforms only in the Dynamic Degree (compared to CogVideoX-5B). Additionally, CogVideoX-5B does not achieve the best performance across all models and metrics.\n* The computational cost (in both time and memory) and the complexity of data filtering and training seem high. Authors should specify these.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FH592Xjpnd", "reviewer": "Reviewer_xLCm", "summary": "The authors introduced CogVideoX, a large-scale text-to-video generation model based on diffusion transformers, addressing common issues in prior video generation models, such as limited motion and short duration. It introduces several designs like 3D VAE, which I think is novel.\n\nExperimental results demonstrate that CogVideoX achieves state-of-the-art performance across multiple machine metrics and human evaluations.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.There are few diffusion transformer-based models. This paper provides a comprehensive approach, from training a 3D VAE to constructing a text-video data processing pipeline, along with a robust model architecture and training design. The model and code are well-implemented.\n\n2.The paper is well-structured, and very easy to follow.\n\n3.The ablation study is thorough, verifying the effects of 2D+1D attention and 3D attention design in video generation, as well as different positional encodings. The 2B and 5B model training partially validates the scaling law.", "weaknesses": "1. The novelty at the model level is relatively weak, with two expert adaptive layer norms being a fairly simple design.\n2. The authors use T5 as the encoder, but comparisons with mixed text encoders are missing.\n3. The 3D VAE design is primarily compared with its own variants, without comparison to other VAE designs, such as the spatial compression capability of SD’s 2D VAE.\n4.Although the data processing pipeline is clearly outlined, the dataset is not publicly available.  It would be nice to open-source the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LD0vFEA0go", "reviewer": "Reviewer_YY2H", "summary": "CogVideoX is a large-scale text-to-video generation model using a diffusion transformer, capable of generating 10-second videos at 16 fps with a resolution of 768×1360 pixels. It addresses previous models' limitations in movement and duration by introducing three components: a 3D Variational Autoencoder for improved video compression and fidelity, an expert transformer with adaptive LayerNorm for enhanced text-video alignment, and progressive training with multi-resolution frame techniques for coherent, long-duration videos with significant motions. The paper introduces a text-video data processing pipeline to enhance generation quality and semantic alignment. CogVideoX achieves state-of-the-art performance in text-to-video generation, and its components. The authors also make the model weights of the 3D Causal VAE, the video caption model, and CogVideoX publicly available.", "questions": "1. There is a typo in line 310: \"Patch’n Pack.\"\n2. Line 459 seems to be missing a citation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The open-sourcing of the 3D Causal VAE, the video caption model, and the CogVideoX model significantly promotes future developments in video generation research.\n2. The method can generate videos with larger frames, better temporal consistency, and higher resolution.\n3. The proposed Multi-Resolution Frame Pack and Progressive Training techniques are interesting and meaningful.\n4. This paper introduces sufficient technical improvements to enhance the model's performance.\n5. Experimental results demonstrate that CogVideoX achieves superior performance compared to current text-to-video generation models.", "weaknesses": "1. Providing additional details in the methods section would enhance the paper’s completeness. For example, further explanation on implementing videos of different durations (and resolutions) in the same batch would be helpful. Although it is mentioned that the method is inspired by Patch n’Pack, a brief clarification would be beneficial.\n2. It would be better to discuss the additional computational costs associated with using 3D attention compared to the commonly used 2D+1D attention.\n3. The axis labels in Figure 8 appear to be mislabeled.\n4. It would be better to add a related work section to discuss video generation works and the differences from previous methods and architectures. Additionally, references for some methods are missing in Table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)