Latte: Latent Diffusion Transformer for Video Generation
Authors: Xin Ma
Yaohui Wang
Xinyuan Chen
Gengyun Jia
Ziwei Liu
Yuan-Fang Li
Cunjian Chen
Yu Qiao
Venue: Accepted by TMLR ()

Abstract: We propose Latte, a novel Latent Diffusion Transformer for video generation. Latte first extracts spatio-temporal tokens from input videos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial number of tokens extracted from videos, four efficient variants are introduced from the perspective of decomposing the spatial and temporal dimensions of input videos. To improve the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-art performance across four standard video generation datasets, \textit{i.e.}, FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, we extend Latte to the text-to-video generation (T2V) task, where Latte achieves results that are competitive with recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporating Transformers into diffusion models for video generation.

Published in Transactions on Machine Learning Research (03/2025)
Latte: Latent Diffusion Transformer for Video Generation
Xin Malt, Yaohui Wang"!, Xinyuan Chen?, Gengyu Jia Ziwei Liu", Yuan-Fang Li', Cunjian Chen' Yu Qiao²
Department of Data Science & AI, Faculty Information Technology, Monash University "Shanghai AI Laboratory *Nanjing University of Posts and Telecommnications *S-Lab, Nanyang Technological University
Reviewed on OpenReview; http:://pppnrrreeeeee/////////tr
Abstract
We propose Latte, a novel Latent Diffusion Transformer for video generation. Latte first extracts spatio-temporal tokens from imput vidcos and then adopts a series of Transformer blocks to model video distribution in the latent space. In order to model a substantial mumber of tokens extracted from videos, four efficient variants perspective of decomposing the spatial and temporal dimensions of input videos. To im- prove the quality of generated videos, we determine the best practices of Latte through rigorous experimental analysis, including video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strate- gies. Our comprehensive evaluation demonstrates that Latte achieves state-of-the-arl per- formance across four standard video generation datasets, é.e., FaceForensics, SkyTimelapse, UCF101, and Taichi-HD. In addition, extend Latte to the text-to-video generation (T2V) task, where Latte achieves results that are competitive with recent T2V models. We strongly believe that Latte provides valuable insights for future research on incorporat- ing Transformers into diffusion models for video generation. The project page is available at https: / h//////xinnn///// en, github. pellaattt_pojccc..
introduced from the
1 Introduction
Diffusion models Ho et al. 2020); Song et al. 2022bb a) powerful deep generative models for many tasks
in content creation, including image-to-inage generation Meng et al. (2022): Zhao al. (2022); Saharia
ct al. [2022a]; ; Parmar et al. (2023). text-to-image generation Zhou et al. (2023) Rombach et al. (2022)
Zhou et al. (2022); Ruiz et al. (0023 Zhang et al. (2023). and 3D object generation Wang et al. (2023)
Chen ct al. (2023); Zhou et al. (2021); Shuc et al. (2023). Compared to these succossful applications in
images, generating high-quality videos still faces significant challenges, which can be primarily attributed
the intricate and high-dimensional nature of videos that encompass complex spatio-temporal information
within high-resolution frames.
The significant role backbone models play in the success of diffusion models has also been investigated Nichol & Dhariwal (2021) Peebles & Xie (023): ; Bao et al. (2023) The U-Net Ronneberger et al. (2015)
which relies on Convolutional Neural Networks (CNNs), has held a prominent position in image and video
generation works Ho et al. (2022); ; Dhariwal & Nichol (0021). In comparison, DiT Peebles & Xie (2023)
and U-VIT Bao et al. (2023) adopt the architecture VTT Dosovitskiy et al. (2021) in diffusion models for image generation and achieve great performance. Moreover, DiT has demonstrated that the inductive
bias of U-Net is not crucial for the performance of latent diffusion models. On the other hand, attention-
based architectures Vaswani et al. (2017) present intuitive option for capturing long-range contextual relationships in videos. Therefore, a very natural question arises: Can Transformer- based latent diffussio models enhance the generation of realistic videos?
Corresponding author. Intern Shanghai AT Lab.

---

Published in Transactions on Machine Learning Research (03/2025)
Figure 1: Sample videos with a resolution of 512 X 512. Latle can generate photorealistic videos with temporal coherent content. Please click the image to play the video clip via Acrobat Reader.
In this paper, we propose Latte, the novel latent diffusion Transformers for video generation, which adopts video Transformer as the backbone. Latte employs a pre-trained variational autoencoder encode input videos into features in the latent space, where tokens are extracted from encoded features. Thus, a series of transformer blocks is applied to encode these tokens. Considering the inherent disparities between spatial and temporal information and a large number of tokens extradted from input videos, shown in Fig. 2 we design four efficient Transformer-based model variants from the perspective of disentangling the spatial and lemporal dimensions of input videos. We also give more insights about the impact of the degree of decoupling between the temporal and spatial modules within the different models, which are lacking in previous works Lu et al. (2024): Gupta et al. (2024).
Many best practices for convolutional models have been proposed, including text representation for question classification Pota et al. 2200)), and network architecture design for image classification He et al. (2016), etc. Nevertheless, Traasfoomer-bbsee latent diffusion models for vidco generation might demonstrate different characteristics, necessitating the identification of optimal design choices for this architecture. Therefore, we conduct a comprehensive ablation analysis, encompassing video clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Our analysis identifies best practices that enable Latte to generate photorcalistic videos with temporal coherent content (see Fig. [ and achieve slate-of-the-arl, performance across four standard video generation benchmarks, including FaceForensics Rössler et al. (2018), Sky'limelapse Xiong et al. (011)), UCF101 Soomro et al. (2012) and Taichi-HD Siarohin et al. (2019) in terms of Fréchet Video Distance (FVD) Unterthiner et al. (2018) Fréchet Inception Distance (FID) Parmar et al. (2021), and Inception Score (IS). In addition, we extend Latte to text-to-video generation tasks, which also achieve competitive results compared to current T2V models.
Our main contributions can be summarized as follows:
We present Latte, a novel latent diffusion Transformer, which adopts a video Transformer as the backbone. In addition, four model variants are introduced to efficiently capture spatio-temporal distribution in vidcos. The design insights of four model variants are also analyzed Sec. 4.2
To improve the quality of generated videos, we comprehensively explore design alternatives, including video clip patch embedding, model variants, timestep-class information injection, temporal positional
2

---

Published in Transactions on Machine Learning Research (03/2025)
embedding, and learning strategies, to determine the best practices of Transformer-basee diffusion models for video generation.
Experimental results on four standard video generation benchmarks show that Latte can gener- ate photorealistic videos with temporal coherent content outperforming state-of-the-art methods. Moreover, Latte shows competitive results when applied to the text-to-vidco generation task.
2 Related Work
Video generation aims to produce realistic videos that exhibit high-quality visual appearance and con- sistent motion sinultaneously. Previous research in this field can be categorized into three main categories. Firstly, several studies have sought to extend the capabilities of powerful GAN-based image generators to create vidcos Vondrick et al. (2016); Saito et al. (2017); Wang et al. 2020b al: Kahembwe & Ramamoorthy 22200 However, these methods often encounter challenges related to mode collapse, limiting their effec- tiveness. Secondly, some methods propose learning the data distribution using autoregressive models Ge iet al. (2022;; |akhimov et al. (2021); Weissenborn al. (2020); Yan et al. (2021); |Kondratyuk al.) (2024) While these approaches generally offer good video quality and exhibit more stable convergence, they come with the drawback of requiring significant computational resources. Finally, recent advances in video gener-
ation have focused on building systems based on diffusion models Ho et al. (2020); Harvey et al. (2022); Ho
let al.
2022]; Singer et al. )202)); Mei & Patel (2023): Blattmann et al.
(2023b); Wang et al. (2024a); Chen
et al.
(002cc) Wang et al. (202bb : Gupta et al. 20024 Ma et al. [2024a] resulting in promising outcomes.
However, Transformer-based diffusion models have not been explored well. A similar idea has been explored
in recent concurrent work VDT Lu et al. (2024), Gen'Tron Chen et al. (2024b). Open-Sora Plan Lin et al.
(2024), HunyuanVideo Kong al. (2024), Pyramidal Flow Jin et al.
2025), Cosmos Agarwal el. al. (2025)
and on. VDT, GenTron, and W.A.L.T use architecture akin to our Variant 3. The primary difference from the previous works is that we conduct systematic analysis of different Transformer backbones and
the relative best practices discussed in Sec. 3.2 and Sec. 3.3 on video generation. Please see Sec. A.3 for a more detailed discussion of the differences.
Transformers have become the mainstream model architccture and got remarkable success in many fields, such as image inpainting Ma al. (2022; 2021 2023), ; image super-resolution Luo et al. (2022): y Huang
et al. (2017), image cropping Jía et al. (2022), forgery detection Jía et al. 20021), face recognition Luo et al.
(2021a lalla b), natural language processing Devlin et al. (2019): Ma et al 2024b). Transformers initially emerged
within the language domain Vaswani et al. (0117;: Kaplan et al. (2020), where they quickly established a reputation for their outstanding capabilities. Over time, these models have been adeptly adapted for the task of predicting images, performing this function autoregrossively within both image spaces and discroto codebooks Chen et al. (2020); Parmar et al. (2018). In the latest developments, Transformers have been integrated into diffusion models, expanding their purview to the generation of non-spatial data and images. This includes tasks like text encoding and decoding Rombach et al. [2022]; Saharia et al. (2022b), generating
CLIP embedding Ramesh et al. 20222, and realistic image generation Bao et al. 2023
Peebles & Xie (2023
3
Methodology
We begin with a brief introduction of latent diffusion models in Sec. 3.1 Following that, we present the model variants of Latte in Sec. 3.2 Finally, the architectural design best practices are discussed in Sec. 3.3
3.1 Preliminary of Latent Diffusion Models
Latent diffusion models (LDMs) Rombach et al. (2022). LDMs are elficient diffusion models Ho el al. (2000g: Song et al. (2021b) by conducting the diffusion process in the latent space instead of the pixel space. LDMs first employ an encoder from pre-trained variational autocncoder to compress the input data sample x (2)ta()) into lover-dinensional latent code = E(x). Subsequently, it learns the data distribution through two key processes: diffusion and denoising.
3

---

Published in Transactions on Machine Learning Research (03/2025)
Variant 1
Variant 2
Variant 3
Variant 4
Noise
Variance
Temporal
Temporal
+
MLP
Linear and Reshape
MLP
Spatial
Temporal
Layer Norm
Layer Norm
Layer Norm
Temporal
Temporal
MHA
Transfomer Blocks
Ze t
Fusion
Spatial
Spatial
Layer Norm
Zt
MHA
MHA
Embed to Tokens
Temporal
Spatial
MHA
7/
Layer Norm
Spatial
Spatial
Layer Norm
2g
25
Embedding
Embedding
Embedding
Embedding
Video Frame Sequences
(a)
(b)
(c)
(d)
The Latte architecture for video generation. Four model variants of Latte are proposed to capture spatio-temporal information in videos efficiently. Each block depicted in light orange represents a Transformer block. The standard Transformer blocks (described in Fig. I0 the Appendix) are employed in (a) and (b). Mcanwhile, (c) and (d) employ our custom Transformer block variants. For the sake of simplicity, encoding and decoding processes for VAE are not shown in the diagram.
Figure 2:
The diffusion process gradually introduces Gaussian noise into the latent code 2, generating a perturbed sample 2 = Vaz
where e ~ (O, 1), following a Markov chain spanning T stages. In this context, a₃ serves as noise scheduler, with representing the diffusion timestep.
The denoising process trained to understand the inverse diffusion process to predict less noisy (HHo(2), Eo(z)) with the variational lower bound of log-likelihood reducing to Here, po is implemented using denoising model
4--1: p0(2)112))
Co = - log (1) 1 + E, DRz ((1(=11/=,, eg and is trained with the simple objective,
Ssimpe Expp(s), e> [|ee co(2., 1)1
(1)
In accordance with Nichol & Dhariwal (201), to train diffusion models with a learned reverse process covariance Eg, it is necessary to optimize the full DEL term and thus train with the full loss function, denoted as Comb. Additionally, Eg is implemented using eg.
3.2 The Latte architecture and its model variants
In this work, we propose Latte, Transformer-basel architecture, shown Fig. 2 (left), for video generation. Latte extends LDMs in the following ways. (1) The encoder & is used compress each vidco frame into latent space. (2) The diffusion process operates in the latent space of videos to model the latent spatial and temporal information. Eg is implemented with Transformer. We train all our models by employing both Eximple and Lelb. We propose four model variants of Latte to efficiently capture spatio-temporal information in videos. The variants are depicted in Fig. 2 (right).
Variant 1. As depicted in Fig, (a), the Transformer backbone this variant comprises two distinct types of Transformer blocks: spatial Transformer blocks and temporal Transformer blocks. The former focuses on capturing spatial information exclusively among tokens sharing the same temporal index, while the latter captures temporal information across temporal dimensions in "interleaved fusion" manner.

---

Published in Transactions on Machine Learning Research (03/2025)
Given video clip in the latent space VL RFXHXWxC, we first translate VL into sequence of tokens, denoted as 2 E R X X Here, F, H, W, and C' represent the number of video frames, the height, width, and channel of video frames in the latent space, respectively. The total number of tokens within a video clip in the latent space is n, X Th x nw: and d represents the dimension of cach token, respectively. Spatio-temporal positional embedding p is incorporated into s. Finally, we get the = 2 + p as the input for the Transformer backbone.
We reshape into 2, E Raxxaxd as the input of the spatial Transformer block to capture spatial information. Here, n4 X fw denotes the token count of each temporal index. Subsequently, 2 containing spatial information is reshaped into 2ς E R to serve as the input for the temporal Transformer block, which is used for capluring temporal information.
Variant 2. In contrast to the temporal "interleaved fusion" design in Variant 1, this variant utilizes the "late fusion" approach to combine spatio-temporal information Neimark ct al. (2021); Simonyan & Zisserman (2014). As depicted in Fig. 2 (b), this variant consists of an equal number of Translormer blocks as in Variant 1. Similar to Variant 1, the imput shapes for the spatial Transformer block and temporal Transformer block
are 2 E R7y and bt E Rxx xd respectively.
Variant 3. Variant 1 and Variant 2 primarily focus on the factorization of the Transformer blocks. Variant 3 focuses on decomposing the multi-head attention in the Transformer block. Illustrated in Fig. 2 (c), this variant initially computes self-attention only on the spatial dimension, followed by the temporal dimension. As result, each Transformer block captures both spatial and temporal information. Similar to Variant 1 and Variant 2, the inputs for spatial multi-head self-attention and temporal multi-head self-attention are 2, E X and 2z Rsxnyxd, respectively.
Variant 4. We decompose the multi-head attention (MHA) into two components in this variant, with each component utilizing half of the attention heads as shown in Fig. (d). We use different components to handle tokens separately in spatial and temporal dimensions. The input shapes for these diffcrent components are 2x E RR Xexd and 2t E x xd respectively. Once two different attention operations are calculated, we reshape ze E Raxu,xd into z E R=xx Then 2n is added to za. which is used as the input for the next module in the Transformer block.
After the Transformer backbone, critical procedure involves decoding the video token sequence to derive both predicted noise and predicted covariance. The shape of the two outputs is the same as that of the input Vz E Fx Following previous work Peebles & Xie (2023); Bao et al. (2023), we accomplish this by employing a standard linear decoder as well as a reshaping operation. We also provide more design signing about the four different model variants and show why variant achieves the best performance in Sec. 4.2
3.3 The Architectural Design Best Practices of Latte
We perform comprehensive empirical analysis of crucial components in Latte, aiming to discover the best practices for integrating the Transformer as the backbone within latent diffusion models for video generation.
3.3.1 Latent video clip patch embedding
To embed a video clip, we explore two methods as follows to analyze the necessity of integrating temporal information in tokens, i.e. 1) uniform frame patch embedding and 2) compression frame patch embedding.
Uniform frame patch embedding. As illustrated in Fig. 3 (a), we apply the patch embodding tochnique outlined in ViT Dosovitskiy et al.] (2021) to each video frame individually. Specifically, ny, nhy and nw are equivalent to F, A. ; and W. when non-overlapping image patches extracted from every video frame. Here, h and w denote the height and width of the image patch, respectively.
Compression frame patch embedding. The second approach is to model the temporal information in a latent video clip by extending the ViT patch embedding to the temporal dimension, as shown in Fig. 3 (b). We extract tubes along the temporal dimension with a stride of s and then map them to tokens. Here, ny is equivalent to in contrast non-overlapping uniform frame patch embedding. Compared to the former, this hot inherently incorporates spatio-temporal information during the patch embedding stage.
5

---

Published in Transactions on Machine Learning Research (03/2025)
W
X2
X1 22
X
H
FF
F
(a) uniform frame patch embedding
(b) compression frame patch embedding
Figure 3: The video clip patch embedding. (a) We sample F frames and embed each individual video frame into tokens using the method described in VIT. (b) We consider capturing temporal information and then extending the ViT patch embedding method from 2D to 3D and subsequently extracting tubes along the temporal dimension. For ease of understanding, we use the original video clip here to demonstrate the patch embedding method. The patch embedding in the latent space of videos follows the same processing approach.
Note that the context of using the compression frame patch embedding method, an additional step entails integrating a 3D transposed convolution for temporal upsampling of the output latent videos, following the standard linear decoder and reshaping operation.
3.3.2 Timestep-class information injection
From simple and direct integration to complex and nuanced integration perspective, we explore two methods for integrating timestep or class information into our model. The first. approach involves treating it as tokens, and we refer to this approach as all tokens. The second mothod is akin to adaptive layer normalization (AdaLN) following DiT Perez et al. (2018); Peebles & Xie (2023). We employ linear regression compute "e and Be based on the imput c, resulting in the equation AdaLN
7cLayeNNorm(h) + Be, where h represents the hidden embeddings within the Transformer blocks. Furthermore,
also perform regression acc which is applied directly before any residual connections (RCs) within the Transformer block, resulting in RCs(h, c) = och AdaT.N(h,,) c). We refer to this as scalable adaptive layer normalization (S- (S-AdaT.N). The architecture of S-AdaLN is shown in Fig, 10 of the Appendix.
3.3.3 Temporal positional embedding
Temporal positional embedding enables a model comprehend the temporal signal. We explore two methods as follows for injecting temporal positional embedding into the model: 1) the absolute positional encoding method incorporates sine and cosine functions with varying frequencies Vaswani al. (2017) to enable the model to recognize the precise position of each frame within the video sequence; 2) the relative posit
encoding method employs rotary positional embedding (RoPE) Su et al. (2024) to enable the model to grasp the temporal relationships between successive frames.
3.3.4 Enhancing video generation with learning strategies
Our goal is ensure that the generated vidcos exhibit the best visual quality while preserving temporal consistency. We explore whether incorporating two additional learning strategies, i.e., learning with pre- trained models and learning with image-video joint training, can enhance the quality of the generated videos.
Learning with pre-trained models. The pre-trained image generation models have learned what the world looks like. Thus, there are many video generation works that ground their models on pre-trained image gencration models to learn how the world moves Wang al. (2024a)) Blattmann et al. (2023a). However, these works mainly build on U-Net within latent diffusion models. The necessity of Transformer-basee latent diffusion models is worth exploring.

---

Published in Transactions on Machine Learning Research (03/2025)
We initialize Latte from pre-trained DiT model ImageNet Peebles & Xie (0023 Deng et al. (2009 Directly initializing from the pre-trained DiT model will encounter the problem of missing or incompatible parameters. To address these, we implement the following strategies. In pre-trained DřT, a positional embedding p E RR x xd is applied to each token. However, in our video generation model, we have a token count that is ny times greater than that of the pre-trained DiT model. We thus temporally replicate the positional embedding ny times from p E xaxxn to p E R Xny X2w xd, Furthermore, the pre-trained DiT includes a label embedding layer, and the number of categories is 1000. Nevertheless, the used video dataset either lacks label information or encompasses significantly smaller number of categories in comparison to ImageNet. Since we target both unconditional and class-conditional video generation, the original label embedding layer in DîT' is inappropriate for our tasks, we opt to directly discard the label embedding in DiT and apply xero-initialization.
Learning with image-video joint training. The prior work on the CNN-basod video diffusion model proposes joint imago-vidoo training strategy that greatly improves the quality of the generated videos Ho et al. (2022). We explore whether this training strategy can also improve the performance of the Transformer-based video diffusion model. To implement simultancous training for video and image generation, We append randomly selected video frames from the same dataset to the end the chosen videos, and each frame is independently sampled. In order to ensure our model can generate continuous videos, we propose two image-video joint training strate- gies: 1) tokens related to vidco content are used in the temporal module for modeling temporal information, while video frames tokens excluded; 2) all tokens are used to optimize the temporal module with new masking strategy. The first mask training strategy is that tokens related to video content are used in the temporal module to establish temporal relationships between video frames, referred to as Only Video. However, the performance is not optimal since images are only used to optimize the spatial modules. We thus propose a new training strategy that can make images also optimize the temporal modules while keeping our model to generate consistent videos, referred Suppose have frames in vidco clip and additional images at the end of the video, thus have mask matrix M E R(++)x(++1) The visualization of our temporal mask M can be seen in Fig. 4 We evaluate the zero-shot capability on UCF101 using two different mask strategies. The FVD and FID scores for Only Video are 705.72 and 75.98, respectively, while for Temporal Mask, they are 589.86 86 and 63.36. These results demonstrate the superiority of Temporal Mask.
P1 12
P
S/ 1ith frame in video
Ap the 1-th image mask value
mask value
Figure Temporal mask strategy.
as Temporal Mask.
4 Experiments
This section initially outlines the experimental setup, encompassing datasets, evaluation metrics, baselines, Latte configurations, and implementation details. Subsequently, we present ablation experiments for the best practice choices and model size of Latte. Finally, we compare experimental results with tha-art and present text-to-video generation results.
4.1 Experimental setup
Datasets. We primarily conduct comprehensive experiments on four public datasets: FaceForensics Rössler
et al. (2018). Sky Tkyclaale Xiong et al. (2018). UCF101 (Sooro ct al. (2012), and Taichi-HD Siarohin
et al. (019)). Following the experimental setup in Skorokhodov al. (2022). except for UCF101, we use the training split for all datasets if they are available. For UCF101, we use both training and testing splits. We extract 16-frame video clips from these datasets using a specific sampling interval, with each frame resized to 256x256 resolution for training.
Evaluation metrics. In the assessment of quantitative comparisons, we employ three evaluation metrics: Fréchet Vidco Distance (FVD) Unterthiner et al. 2018), Fréchet Inception Distance (FID) Parmar et al. (2021), and Inception Score (IS) Saito et al. (2017). Our primary focus rests on FVD, as its imago-basod counterpart FID aligns more closely with human subjective judgment. Adhering to the evaluation guidelines
7

---

Published in Transactions on Machine Learning Research (03/2025)
Model
N
Param
Variant 1
Variant 2
Variant 3
Variant 4
Latte-S
384
32.48M
Params (M)
673.68
673.68
676.33
676.44
Latte-B
768
12
129.54M
FLOPs G
5572.69
5572.69
6153.17 15
1545.15
Latte-L
24
1024
16
456.81M
XItte-XL
28
1152
16
673.68M
Table The number of parameters and FLOPs for different model variants.
Table 2: Details of Latte models. We
follow ViT and DiT configurations.
Method
FaceForensics
Sky Timelapse
UCF101
Taichi-HD HD
MoCoGAN Tulyakov al. 2018
124.7
206.6
2886.9
VideoGIT Yan [202T
185.9
222.7
2880.6
MoCoGAN-HD lian et
2021
111.8
164.1
1729.6
128.1
DIGAN Yu et al. 2022
62.5
83.11
1630.2
156.7
StyleGAN-V Skorokhodov et a.
2002
47.41
79.52
1431.
PVDM et al. (2023)
355.92
55.41
343.6
540.2
MoStGAN Shen et al. [2023)
39.70
65.30
1380.3
LVDM He
2023
95.20
372.0
99.0
VDT
Lu al. 2024
225.7
W.A.L.T
Gupla el al.
(0044)
258.1
Latte (ours)
34.00
59.82
477.97
159.60
Latte+IMG (ours)
27.08
42.67
202.23
97.09
Table 3: FVD values of video generation models different datasets. FVD values for other baseline models reported and sourced from the reference StyleGAN-V the original paper. Here, "IMG" means video-image joint training.
Subject Consistency
Background Consistency
Temporal Flickering
Motion
Dynamic
Aesthetie
Imaging Quality
Object
Methods
Smoothness
Degree
Quality
Class
ModelSeope Lao
87%
95.29%
98.28%
95.79%
66.39%
52.06%
58.57%
82.25%
VideuCrafter He el
2123
86.24%
92.88%
97.60%
91.79%
89.72%
44.41%
57.22 22%
87.34%
CogVideo Hong al
2022
92. 19%
95.42%
97.6-1%
98.17%
42 22%
38.18%
41.03%
73.40%
HiGen <nng et aL
90.07%
93.99%
93.24%
96 98.69%
90.17%
57 57.30%
63 9.%
86.06%
OpenSoral lan el. al.
2024
97.27%
96.24%
99.12%
99.17%
35.28%
59.10%
65.73%
61.53%
Latte
95.40%
98.80%
94.635
68.89%
61.59%
86.33%
Multiple
Human
Spatial
Appesarance
Tempural Style
Overall
Metbods
Color
Scene
Object
Action
Relationship
Style
Consistency
MadelScope Lmo et al.
2024
38.98%
92.40%
81.72%
33.68%
39.26%
23.39%
25.37%
25.67%
VideoCrafter et al.
2023
25.93%
93.00%
78.84%
36.74%
43.36%
21.57%
25.42%
25.21%
Cug Video
Hong
20222
18.11%
78.20%
79.37%
18.24%
28, 24%
22.01%
7.80%
7.70%
HiGen Qung
A724
22.39%
86.20%
86.22%
22.43%
44.88%
24.54%
25.14%
27.14%
OpenSoral lan Lin el
2024
21.91%
58.20%
90.90%
49.64%
17. 28%
20.01%
19.14%
20.39%
Latte
34.53%
90.00%
85.31%
41.53%
36.26%
23.74%
24.70%
27.33%
Table 4: Vbench evaluation results per dimension for different methods. A higher score demon-
strates better model performance for a certain dimension. We use bold and underline to mark the best and second model performances, respectively.
introduced by StyleGAN-V, we compute the FVD scores by analyzing 2,048 video clips, each comprising 16 frames. We only employ IS for assessing the generation quality on UCF101, as it leverages the UCF101-fine. tuned C3D model Saito et al. (017).
Baselines. We compare with recent methods to quantitatively evaluate the outcomes, including MoCoGAN
Tulyakov et al.
(2018), VideoGPPT Yan et al. (2021), MoCoGAN-HI Tian et al. (0211) DIGAN Yu et al. (2022), StyleGAN AN-V Skorokhodov al. (2022), PVDM Yu al. (2023). MoStGAND Shen et al. (2023). LVDM He al. (2023), VDT Lu et al. (2024) and W.A.L.T Gupta et al. (2024). Furthermore, we conduct
extra comparison of IS between our proposed method and previous approaches the UCF101 dataset.
8

---

Published in Transactions on Machine Learning Research (03/2025)
Latte configurations. A scries of N Transformer blocks are used to construct our Latte model, and the hidden dimension of each Transformer block is D with N multi-head attention. Following ViT, we identify four configurations of Latte with different numbers of parameters as shown in Tab. 2
Implementation details. We use the AdamW optimizer with a constant learning rate x 10 4 to train all models. Horizontal flipping the only employed data augmentation. Following common practices within generative modeling works Peebles & Xie] (2023); [aao et al. (002)), an exponential moving average (EMA) of Latte weights is upheld throughout training, employing decay rate of 0.9999. All the reported results are directly obtained from the EMA model. We borrow the pre-trained variational autoencoder from Stable Diffusion 1.4. All experiments are conducted on 8 NVIDIA A100 (80G) GPUs.
4.2 Ablation study
In this section, we conduct experiments on the FaceForensics dataset examine the effects of different designs described in Sec. 3.3 model variants described in Sec. 3.2 video sampling interval, and model size on model performance.
Video clip patch embedding. We examine the impact of two video clip patch embedding methods detailed in Sec 3.3.1 In Fig. 5a the performance of the compression frame patch embedding method notably falls behind that of the uniform frame patch embodding method. This finding contradicts the results obtained by the video understanding method ViVIT Arnab al. 2021). We speculate that using the compression frame patch embedding method results in the loss of spatio-temporal signal, which makes it difficult. for the Transformer backbone to learn the distribution of videos.
480
300
350
Unifocm frime patch rmboeking
takers
Abeolure position embeckinng
Compeesson frame putch embedding
S
Relalive positios enbedking
200
330
200
130
130
100
080
130
20
100
1.0k
120
06
00
11%
12%
120k
100
08
00c
198
120
1256
Thitup Pmm
Thai. finkk
(a) Video clip patch embedding
(b) Timestep-class information injection
(c) Temporal positional embedding
With Im
Frame interval 2
Variant 1
203
Without let prommiing
090
interval
300
Varlant 2
Frame interval
200
Varlant 3
200
280
Prame interval
Variant 4
Frame interval 16
123
150
100
180
(d) ImageNet pretraining
(c) Video sampling interval
(f) Model variants
Figure 5: Ablation of design choices. We design several ablation studies to explore best practices in Transformer-basee video diffusson models terms of FVD on FaceForensics. Please zoom in for better view.
Timestep-class information injection. As depicted in Fig. 5b the performance of S-AdaLN is signif- icantly better than that of all tokens. We believe this discrepancy may stem from the fact that all tokens only introduces timesteps or label information to the input layer of the model, which could face challenges in propagating effectively throughout the model. In contrast, S-AdaLN encodes timestep label informa-
9

---

Published in Transactions on Machine Learning Research (03/2025)
tion into the model in more adaptive manner for each Transformer block. This information transmission approach appears more efficient, likely contributing to superior performance and faster model convergence.
Temporal positional embedding. Fig. 5c illustrates the impact of two different temporal position embedding methods on the performance of the model. Employing the absolute position embedding approach tends to yield slightly better results than the alternative method. Since our model does not require dynamic temporal-spatial adaptability for generating variable-duration and multi-resolution videos (i.e., key strength of RoPE), ultimately choose absolute positional embedding the temporal positional embedding method.
Enhancing video generation with learning strategies. As illustrated in Fig. 5d we observe that the initial stages of training benefit greatly from the model pre-training on ImageNet, enabling rapid achievement of high-quality performance on the video dataset. However, as the number of iteralions increases, the performance of the model initialized with a pre-trained model tends to stabilize around a certain level, which is far worse than that of the model initializod with random.
This phenomenon can be explained by two factors: 1) the pre-trained model on ImageNet provides good representation, which may help the model converge quickly at an early stage; 2) there is a significant difference in data distribution between ImageNet and FaceForensics, which makes it difficult for the model to adapt the knowledge learned on ImageNet to FaceForensics. It is important to emphasize that the second point is particularly relevant for small-scale datasets (e.g. UCF101), where the domain discrepancy may outweigh the benefits of knowledge transfer. In contrast, large-scale datasets inherently exhibit greater diversity, which helps mitigate domain gaps compared to specialized small datasets. As a result, the first factor may play a more significant. role in general tasks, such as text-to-video generation.
As demonstrated in Tab. and Tab. 5 we find that image-video joint training ("Latte+IMG") leads to a significant improvement of FID and FVD. Concatenating additional randomly sampled frames with videos along the temporal axis enables the model to accommodate more examples within each batch, which can increase the diversity of trained models. Video sampling interval. We explore various sampling rates to construct a 16-frame clip from each training video. As illustrated Fig. [e] during training, there is a significant performance gap among models using different sampling rates in the early stages. However, the number of training iterations increases, the performance gradually becomes consistent, which indicates that different sampling rates have little effect model performance. We choose video sampling interval of 3 to ensure reasonable level of continuity in the generated videos to conduct the experiments of comparison to state-of-the-ar
Cosine Similrity Hobli
3.0
Cosine Simiarity M800
1.
Cosine Simibrity MaP
1.
dindert Eireibutee
6.
33
03
Lee
0.6
2.
03
6.6
3.4
0.4
62
32
-22
Parre brtc
Phate BrEx
6.0
38
(a) Block 0
(b) Block 7
(c) Block 13
(d) Cosine Similarity Distribu- tion
Figure 6: Frame cosine similarity and its distribution.
Model variants. We evaluate the model variants of Latte as detailed in Sec. 3.2 We strive to equate the parameter counts across all different models to ensure a fair comparison. We commence training all the models from scratch. As shown in Fig. 5f Variant 1 performs the best with increasing iterations. Notably, Variant exhibits roughly a quarter of the floating-point operations (FLOPs) compared to the other three model variants, as detailed in Tab. Therefore, it is unsurprising that Variant 4 performs the least favorably among the four variants.
10

---

Published in Transactions on Machine Learning Research (03/2025)
In Variant 2, half of the Transformer blocks initially employed for spatial modeling, followed by the remaining half for temporal modeling. Such division may lead to the loss of spatial modeling capabilities during subsequent temporal modeling, ultimately impacting performance. Hence, we think employing a complete Transformer block (inchuding multi-head attention, layer norm, and multi-linear projection) might be more effective modeling temporal information compared to only using multi-head atlention (Variant 3).
Model size. We train four Latte models of different sizes according to Tab. 2 (XL, L, B, and S) on the FaceForensics dataset. Fig. clearly illustrates the progression of corresponding FVDs as the number of training iterations increases. It can be clearly observed that increasing the model size generally correlates with a notable performance improvement, which has also been pointed out image gencration work Peebles & Xie (2023).
Latte-Xi
Latte-B
200
Latte
50
Design insights for four model variants. The design of the four variants stems from key question: to what extent should the temporal and spatial modules be decoupled in a video generation model? While similar model structures have been employed in some previous works Gupta et al. 22024); Lu et al. (2024), they have not analyzed how these different model structures affect the video gener- ation process. We have supplemented this with detailed experiments and analyses to help understand the impact of the degree of decou- pling between the temporal and spatial modules and to explain why Variant 1 achieves the best performance.
Figure The model performance of different Latte model sizes. In general, increasing the size of the model can significantly improve its performance.
Excessive consecutive spatial attention modules can impair temporal coherence. As shown in Fig. G we conduct a detailed analysis of Variant 2, calculating the cosine similarity matrix between frame features each block. The results demonstrate that adding spa- tial attention modules leads to continuous decrease in the mcan of inter-frame cosine similarity and an increase in variance. This uneven decrease in inter-franee cosine similarity ultimately results in numerous inverted relationships in the cosine similarity matrix, where, for instance, the cosine similarity between the first and sec- ond frames lower than that between the first and fifth frames. These inverted relationships are strong evidence of the disruption of temporal coherence. We believe the root cause of this phenomenon lies in the coupled temporal-spatial information failing to achieve effective decoupling through the separation of spatiotemporal mod- ules.
Cosine Smillrrty Distrbution
100
Spotial
Thrrpers
+ Spabal7arrpon pora
30-
10-
Percnaage
02
0.4
0.6
0.8
Semilarity Rango
Figure 8: The distribution of frame cosine similarity.
The features of the output frames from spatial attention do not match those from temporal attention. The above analysis disproves the design of stacking massive spatial attention modules, leading us to explore increasing the coupling degree between spatial and temporal modules. To this end, we design Variant 4, which modifies the alternating structure of spatiotemporal attention parallel structure within the Transformer block. However, experiments show that this model structure performs poorly. Analysis reveals that the performance degradation is due to the direct fusion of mismatched frame features. Specifically, as shown in Fig. g we visualize the distributions of inter-frame cosine similarity for the outputs of temporal attention, spatial attention, and their mutual inter-frame cosine similarity. The results show significant differences in the distributions of inter-frame cosine similarity for each type, with extremely low cosine similarity between the two. Clearly, directly adding these two types of features severely disrupts temporal information.
The design of Variant 3 is derived from Former Bertasius et al. 2021). Uulike Variant 1, which also employs an alternating spatiotemporal attention structure, Variant 3 has a higher degree of coupling. The
11

---

Published in Transactions on Machine Learning Research (03/2025)
spatial attention features imput into the temporal attention module, and only layer normalization is applied. Based on the above analysis, we believe this structure is not optimal. The simple normalization of spatial attention frame features is insufficient to make them suitable for the temporal attention layer. Experiments show that while Variant performs better than Variants and 4, it weaker than Variant 1. This result. further confirms the validity of our analysis.
4.3 Summary of best practices
Based on the ablation studies Section 4.2 we identify the best practices for Transformer-based latent video diffusion models. These include model variant 1, uniform frame patch embedding, S-AdaLNV absolute position embedding, and imago-video joint training. In the next section, compare our proposed Latte, incorporaling these best practices, with the current state-of-the-ard.
Method MoCoGAN
IS 1
FID
4.4 Comparison to state-of-the-art
10.09
23.97
VideoGPT
12.61
22.7
Qualitative results. Fig. g of Appendix illustrates the video synthesis results from Latte on UCF101, Taichi-HD, FaceForensics, and SkyTimelapse. Our method consistently delivers realistic, high- resolution video generation results (256x256 pixels) all scenarios. This encompasses capturing the motion of human faces and handling the significant transitions of athletes. Notably, our approach excels at synthesizing high-quality videos within the challenging UCF101 dataset, task where other comparative methods often falter. More visual results are available on the project page.
MoCoGAN-HD
23.39
7.12
DIGAN
23.16
19.1
StyleGAN-V
23.94
9.445
PVDM
60.55
29.76
Latte (ours)
68.53
5.02
Latte+IMG (ours)
73.31
3.87
Table 5: Inception Score and FID comparisons of Latte against other state-of-the-art on the UCF101 and FaceForensics datasets, respectively.
Quantitative results. In Tab. 3 we provide the quantitative results of Tatte and other comparative methods, respectively. Our method significantly outperforms the previous works on all datasets, which shows the superiority of our method on video generation. In Tab. 5 we report the FID FaceForensics and the IS UCF101 to evaluate video frame quality. Here, "IMG" means video-image joint training, Our method demonstrates outstanding performance with an FID value of 3.87 and an IS value of 73.31, significantly surpassing the capabilities of other methods.
4.5 Extension to text-to-vided generation
To explore the potential capability of our proposed method, we extend Latte to text-to-video generation. We
adopt the method shown in Fig. 2 (a) to construct our Latte T2V model. Sec. 4.2 mentions that leveraging pre-trained models can facilitate model training. Consequently, we utilize the weights of pre-trained PixArt-
a (512 512 resolution) Chen et al. (2024a) initialize the parameters of the spatial Transformer block in the Latte T2V model. Since the resolution of the commonly used video datasct WebVid-10M Bain high-resolution video dataset Vimeo25M
et al. 2021 is lower than 512 512, train model proposed in Wang et al. (2024a). We compare with the recent T2V models, including VidcoFusion Luo et al. (2023), VideoLDM Blattmann et al. (2023b). VideoCrafter fter He et al. (2023), CigVideo Hong et al. (0022). HiGen Qing et al. (0244). and OpenSoraPlat Lin et al. (2024) using the quantitative metrics established by VBench Huang et al. 22024, as shown Table It demonstrates that Latte can generate comparable T2V results.
5 Conclusion
This work presents Latte, a simple and general vidco diffusion method, which employs vidco Transformer as the backbone to generate videos. To improve the generaled video quality, we determine the best prac- tices of the proposed models, including clip patch embedding, model variants, timestep-class information injection, temporal positional embedding, and learning strategies. Comprchensive experiments show that Latte achieves state-of-the-art results across four standard video generation benchmarks. In addition, text- to- video results comparable to those of current T2V approaches. We strongly believe that Latte
12

---

Published in Transactions on Machine Learning Research (03/2025)
provide valuable insights for future rescarch concerning the integration of transformer-base backbones into diffusion models for video generation as well as other modalities.
Acknowledgments
This work is partially supported by the National Key R&D Program of China under Grant No. 2022ZD0160102, and the Science and Technology Commission of Shanghai Municipality under Grant No. 23QD1400800.
References
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXit:2501.0557 2025.
Amurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In International Conference on Computer Vision, pp- 6836-68444 2021.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In International Conference on Computer Vision, 2021.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Computer Vision and Pattern Recognition, pp. 22669-22617 2023.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In International Conference on Machine Learning, volume 2, pp. 4, 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint or arXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. Tn Computer Vision and Pattern Recognétion, pp. 22563-22575, 2023b.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-a: Fast training of dilfusion transformer for photorealistic text-to- image synthesis. In International Conference on Learning Representations, 2024a.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and 1lya Sutskever. Gener- ative pretraining from pixels. In International Conference on Machine Learning, pp. 1691-1703. PMLR, 2020.
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In International Conference on Computer Vision, October 2023.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen Hc, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into diffusion transformers for image and video generation. In Computer Vision and Pattern Recognition, 2024b.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In International Conference on Learning Representations, 2024c.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, pp. 248 -2-5. IEEF, 2009.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapler of the Associalion, for Compulational Linguistics: : Human Language Technologies, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthosis. Neural Information Processing Systems, 34:8780 8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Rep- resenlations, 2021.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In European Conference on Computer Vision, pp. 102 118. Springer, 2022.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Moera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. In European Conference on Compuler Vision, pp. 393-48 188-411. Springer, 2024.
William Harvey, Sacid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. Neural Information Processing Systems, 35:27953 27965, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognilion, pp. 770-777777777 1700778,500 2016.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint orXiv:2211.1322 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Syslems, volume 33, pp. 6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusson models. In Newral Information Processing Systems, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In International Conference on Learning Representations, 2022.
Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. Wavelet-srnet: A wavelet-based enn for multi-scale face super resolution. In International Conference Computer Vision, pp. 1689- 1689-1697, 2017.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Computer Vision and Pattern Recognition, 2024.
Gengyun Jia, Meisong Zheng, Chuanrui Hu, Xin Ma, Yuting Xu, Luoqi Liu, Yaleng Deng, and Ran He. Inconsistency-aware wavelet dual-branch network for face forgery detection. IEEE Transactions on Bio- metrics, Behavior, and Identity Science, 3(3):308-3)) -109 2021.
Gengyun Jia, Huaibo Huang, Chaoyou Fu, and Ran He. Rethinking image cropping: Exploring diverse compositions from global views. In Computer Vision and Pallern Recognilion, pp. 2446 2446-2455, 2022.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient video generative modeling. In International Conference on Learning Representations, 2025.
Emmanuel Kahembwe and Subramanian Ramamoorthy. Lower dimensional kernels for video discriminators. Neural Networks, 132:506 -20-006-20 2020.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arxie preprinl arXiv:2001.0883 2020.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hormung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, al. Vidcopoct: A large language model for zero-shot video generation. In International Conference Machine Learning, 2024.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXio:2412.00 03603, 2024.
Bin Lin, Yunyang Ge, Xinhua Cheng, Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, et al. Open-sora plan: Open-source large video generation model. arXiv preprint 0rXiv:2412.000 001111,001110100001 2024.
Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General- purpose video diffusion transformers via mask modeling. In International Conference on Learning Repre- sentations, 2024.
Mandi Luo, Jie Cao, Xin Ma, Xiaoyu Zhang, and Ran He. Fa-gan: Face augmentation gan for deformation- invariant face recognition. IEEE Transactions on Informalion Forensics and Securily, 16:2341 2355, 2021a.
Mandi Luo, Xin Ma, Zhihang Li, Jie Cao, and Ran He. Partial nir-vis heterogeneous face recognition with automatic saliency search. IEFE Transactions on Information Forensics and Security, 14:5003-501,, 2021b.
Mandi Luo, Xin Ma, Huaibo Huang, and Ran Hc. Style-based attentive network for real-world face halluci- nation. In Pollern Recognilion and Compuler Vision, pp. 262 273. Springer, 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Computer Vision and Pattern Recognation, June 2023.
Xin Ma, Xiaoqiang Zhou, Huaibo Huang, Zhenhua Chai, Xiaolin Wei, and Ran He, Free-form image inpaint- ing via contrastive attention network. In International Conference on Pattern Recognition, pp. 9242-9249. IEEE, 2021.
Xin Ma, Xiaoqiang Zhou, Huaibo Huang, Gengyun Jia, Zhenhua Chai, and Xiaolin Wei. Contrastive al att- tion network with dense field estimation for face completion. Pattern Recognátion, 124: 104:1084,5, 2022.
Xin Ma, Xiaoqiang Zhou, Huaibo Huang, Gengyun Jia, Yaohui Wang, Xinyuan Chen, and Cunjian Chen. Uncertainty-awaat image inpainting with adaptive feedback network. Expert Systems with Applications, pp. 121148, 2023.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Cin- emo: Consistent and controllable image animation with motion diffusion models. arXiv preprinl arXim:2407.1000 15642, 2024a.
Ziyu Ma, Shutao Li, Bin Sun, Jianfei Cai, Zuxiang Long, and Fuyan Ma. Gerea: Question-aware prompt captions for knowledge-ban
visual question answering. arXiv preprint arXiv:2402.02500 2024b.
Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In AAAI Conference on Artificial Intelligence, pp. 9117-9125, 2023.
Chenlin Meng. Yutong He, Yang Song, Jiaming Song, Jiajun Wu, fun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. In International Conference Computer Vision, pp. 3163-3172, 3172, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On buggy resizing libraries and surprising subtleties in fid calculation. arXw preprint arXiv:2104 11222, 5:14, 2021.
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero- shot image-to-image translation. In ACM Special Interest Group on Graphics and Interactive Techniques Conference, pp. 1-11, 2023.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pp. 4055-4005 PMLR, 2018.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In International Conference Computer Vision, pp. 4195-4205, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Antelligence, 2018.
Marco Pola, Massimo Esposito, Giuseppe De Pietro, and Hamido Fujita. Best practices of convolutional neural networks for question classification. Applied Sciences, 10(14):4710, 2020.
Zhiwu Qing, Shiwei Zhang, Jiayn Wang, Xiang Wang, Yujie Wei, Yingya Zhang, Changxin Gao, and Nong Sang. Hierarchical spatio-temporal decoupling for text-to-video generation. In Computer Vision and Pattern Recognition, 2024.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. In Computer Vision, Imaging and Computer Graphics Theory and Applications, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:224.006125, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffiusion models. In Computer Vision and Pattern Recognition, pp. 10684 10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Tn Medical Image Computing and Computer-Assitted Intervention, pp. 234-241. Springer, 2015.
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Ricss, Justus Thics, and Matthias NicBnor. Faceforensics: A large-scale video dataset for forgery detection in human faces. arXio preprinl arXiv:1803.00 097199, 2018.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM Special Interest Group Graphics and Interactive Techniques Conference, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text- to-image diffusion models with deep language understanding. Neural Information Processing Systems, 35: 36479-3644 2022b.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value 2017.
clipping. In International Conference Computer Vision, pp.
Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Mostgan-v: Video generation with temporal motion styles. In Computer Vision and Pattern Recognition, pp. 5661, 2023.
Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Computer Vision and Pattern Recognition, pp. 20875 20886, 2023.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Newral Information Processing Systems, 32, 2019.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Neural Information Processing Systems, volume 27, 2014.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An; Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. In International Conference on Learning Representations, 2023.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhosciny. Stylegan-v: A contimuous video generator with the price, image quality and perks of stylegan2. In Computer Vision and Pattern Recognition, pp. 3626-3636, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human action classes from videos in the wild. Center for Research in Computer Vision, 2(11), 2012.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhancod transformer with rotary position embedding. Nearocompuling, 568:127063, 2024.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In International Conference on Learning Representations, 2021.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Computer Vision and Pattern Recognition, pp. 1526-1535, 2018.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Syl- vain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprinl A.Xiv:1112.01111 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Neural Information Processing Systems, 30, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba, Generating videos with scene dynamics. In Neural Information Processing Systems, volume 29, 2016.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Computer Vision and Pattern Recognition, pp. 12619-12629, 2023.
Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. G3an: Disentangling appearance and motion for video generation. In Computer Vision and Pattern Recognition, pp. 5264 3264-5273, 2020a.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. Imaginator: Conditional spatio- temporal gan for video generation. In Winter Conference on Applications of Compuler Vision, 2020b.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, pp. 1-20, 2024a.
Yaohui Wang, Xin Ma, Xinyuan Chen, Cunjian Chen, Antitza Dantcheva, Bo Dai, and Yu Qiao. Loo: Generative latent image animator for human video synthesis. International Journal of Computer Vision, pp. 1-13, 2024b.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In Interna- tional Conference on Learning Representations, 2020.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Computer Vision and Pattern Recognition, pp. 2364 2373, 2018.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformors. arXiv preprint 0rXic:2104.10155 2021.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generaling videos with dynamics-aware implicit generative adversarial networks. In International Conference Learning Represenlations, 2022.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Computer Vision and Pattern Recognition, pp. 18456-18466, 2023.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala, Adding conditional control to text-to-image diffusion models. In International Conference Computer Vision, pp. 3-36-388444 2023.
Min Zhao, Fan Bao, Chongxuan Li, and Jun Zhu, Egsde: Unpaired seaag--o--mmage translation via energy- guided stochastic differential equations. In Neural Information Processing Systems, volume 35, pp. 3609 3623, 2022.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion throngh point-voxel diffusion. International Conference on Computer Vision, pp. 5826 5835, 2021.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chumyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-free training for text-to-image generation. In Computer Vision and Pattern Recognition, pp. 17907-17917, 2022.
Yufan Zhou, Bingchen Liu, Yizh Zhu, Xiao Yang, Changyou Chen, and Jinhui Xu. Shifted diffusion for text-to-image generation. In Computer Vision and Pattern Recognation, pp. 10157-10166, June 2023.
A
Appendix
A.1
The sampled video frames
We provide the sampled video frames of different methods shown in Fig, of Appendix.
A.2
The structure of S-AdaLN
In Fig. 10 of Appendix, we show the structure of S-AdaLN.
18

---

Published in Transactions on Machine Learning Research (03/2025)
A.3 Discussion about the difference from concurrent works
A similar idea has been explored in recent concurrent work VDT Lu et al. 2004 GenTron Chen et al. (2024b), W. Gupta et al. (2024). Open-Sora Plan Lin et al. (2024). Hunyuan iiaaVides Video Kong et al. (2024), Pyramidal Flow Jin et al. (2025), and so on. VDT, GenTron, and W.A.L.T use an architecture akin to our Variant 3. VDT primarily focuses on generating various video tasks, inchuding image-to-video generation and unconditional video generation, utilizing mask learning strategy. GenTron and W.A.I.T mainly fo- cus on general purposes, i.e, text-to-video generation and text-to-image generation. Open-Sora Plan and HunyuanVideo focus on large-scale, open-source video generation models system frameworks, respectively, Pyramidal Flow adopts the flow matching strategy, whose core idea and training paradigm are fundamentally different from diffusion-basct generative methods. Its primary goal is to improve generation efficiency. Cos- mos focuses on building a world foundation model for physical Al systems, such as robotics and autonomous driving. Its goal is to simulate the dynamic behaviors of the real physical world, emphasizing physical consis- tency. The primary difference from the previous works is that we conduct a systematic analysis of different Transformer backbones and the relative best practices discussed in Sec. 3.2 and Sec. 3.3 on video generation.
INDM SIAGAN
Ouosk
DIGAN
BOOM
SUNGANYY
IVTM
Oue
SHNGANV
14DM
Figure 9: Sample videos from the different methods on UCF101, Taichi-HD, FaceForensics and SkyTimelapse. respectively.
19

---

Published in Transactions on Machine Learning Research (03/2025)
@+
Scale
MLP
MLP
12.02
Scale, Shift
Layer Norm
L Layer Norm
$
a2
Scale
MHA
MHA
Ve.
Scale, Shift
Layer Norm
MLP
Layer Norm
Timosslep class
Embedding
Embedding
(b)
Figure 10: (a) The architecture of S-AdaLN described in Sec. 3.3.2 (b) The architecture of vanilla trans-
former block used in Fig. 2 (a) and (b). MLP and MHA mean the multi-layer perception layer and the melti-head attention, respectively.
20


Reviews:
[{"review_id": "Oh1bY8n2JS", "reviewer": "Reviewer_CAc1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. Careful ablation study over design choices – In particular, I can see the effort of the authors in claiming that it is more than just VLDM + Timesformer.\n2. Find the proper spatio-temporal transformer design – Interleaved fusion intuitively performs the best.\n3. Strong empirical results – Benchmarking is conducted on various setups, showing consistent improvements.\n\n---\n### Weaknesses\n\n1. Limited technical novelty – The main ideas of most components are from prior work, e.g., AdaLN from DiT, though the paper does a great job at engineering to carve out performance.\n2. Benchmark tables could be more up-to-date – The paper mainly compares against existing models but does not include some of the latest works. For example, WARP (Gupta et al., ECCV 2024) reports an FVD of 258.1 on UCF-101, and Latte (202.23) is actually better. Yet, adding more recent models would help give a more extensive and proper sense of comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the design choices for spatio-temporal transformer blocks in latent video diffusion models. Specifically, it reveals that the interleaved fusion design, which alternates spatial and temporal attention blocks, performs the best among other designs. In addition, the paper ablates the choices for patch embedding, positional embedding, timestep injection, and video-image joint training. Combining all together, the proposed Latte achieves SOTA for unconditional and class-conditional video generation benchmarks, while also being effective for text-to-video generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper is solid and carefully conducted, making it a strong contribution to TMLR. My only suggestion is to add the latest works in the benchmark tables to ensure a more comprehensive comparison.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lRkXei5GpW", "reviewer": "Reviewer_Ctig", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\nThe author proposes four variants of self attention layers (i) alternately applying spatial and temporal self attention layers (ii) first applies spatial attention blocks and then applies temporal attention blocks (iii) alternatively applying spatial and temporal multi head attention blocks (iv) decomposition of the multi head attention blocks into spatial heads and temporal heads. The author experimentally evaluates and discusses that the variant (i) is the best.\n\nFurther, the author proposes several best practices for the generative model including temporal positional embedding and image-video joint training.\n\nThe author conducts several experimental evaluations including ablation study. The proposed method demonstrates better generation performance than the existing generation methods on several benchmark datasets.\n\n\nWeakness\n\nAs the author says, the transformer-based diffusion video generation has already been proposed as VDT and GenTron. It would be better to add more detailed discussion about the difference from these works and experimental comparison.\n\nSince the author conducts various experimental evaluation, it may be clearer to add the section that summarizes the experimental findings e.g. in conclusion or before conclusion section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts detailed experimental analysis about the transformer-based video diffusion model. To incorporate spatial and temporal information of video generation, the author proposes four variants of self-attention layer. The proposed model demonstrates better generation performance than existing video generation methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discussion and Comparison to existing transformer-based video diffusion methods.\n\nSummarization of the experimental findings.", "broader_impact_concerns": "Though broader impact is not discussed, since the paper is about experimental evaluation of the model architecture and training method, I think the ethical risk of the paper is small.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QPC3iTwsQX", "reviewer": "Reviewer_LiRL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The analysis of architecture comparison is new and essential to explore video generation using transformers.\n- The generated videos in supplementary are realistic and good.\n\nMajor weaknesses:\n- The rationale behind some design choices is not clearly explained and may be redundant. \n  - Why do only these four model variants need to be compared? Especially, Variant 3 and 4.\n  - In Sec 3.3.2, the need to compare \"all-tokens\" timestep-class injection should be better explained, maybe with some reference to previous work. Also, DiT (and several follow-up works [1,2]) have shown the benefit of Ada-LN. \n  \n- The conclusion about initializing with pre-trained weights is not fully explained. The authors say that due to differences in Imagenet and Faceforensics datasets, initializing with scratch is better. But what about other datasets? \n  Contrary to this conclusion, their text-to-video model is intialised with Pix-art.\n\n- Contrary to the outcome that absolute position embedding performs better than RoPE. A lot of recent video generation models (For eg. [3,4,5]) have been using RoPE instead of absolute encoding. More insights into preferring one over another should be experimented and discussed by the authors.\n\nMissing related works:\nA lot of recent works [3, 5, 6] around transformer model-based video generation have been proposed and the paper should mention it and compare with them.\n\nMinor weakness:\n- Table 6. Why is IS computed on UC101 and FID on Faceforensics?\n- In 4.1, what does \"UCF101, we use both training and testing splits\" mean? Does this mean both splits are used for training? If yes, then FVD scores in Table 2 would be unfair.\n\n- I guess, Page 5, Variant 1, zt ∈ Rt×nf ×d should be zt ∈ Rs×nf ×d\n- The paper should rearrange the tables and figures to better follow the text.\n\t- Table 5 is referred before Table 4.\n  \t- In Sec. 3.3.2, Fig. 10 is referred before Fig. 4\n  \t- Sequence of figures in Figure 5, For eg. Fig 5e is referred before 5a,b,c,d\n\n[1] Ma et al, \"SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers\", ECCV 2024\n\n[2] Chen et al, \"GenTron: Diffusion Transformers for Image and Video Generation\", CVPR 2024\n\n[3] Jin et al, \"PYRAMIDAL FLOW MATCHING FOR EFFICIENT VIDEO GENERATIVE MODELING\", ArXiv, 2024\n\n[4] Yang et. al \"CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer\", arXiv 2024\n\n[5] Open-Sora Plan Team \"Open-Sora Plan: Open-Source Large Video Generation Model\", arXiv, 2024\n\n[6] \"HunyuanVideo: A Systematic Framework For Large Video Generative Models\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper extends the transformer-based diffusion model (DiT), originally proposed for class-conditioned image generation, to (unconditional, class-conditional and text-conditioned) video generation.\n\nThe paper analyses several architectural designs and training approaches to handle the spatial-temporal tokens of video.\n\nIn summary, their exploration and findings are:\n\n  \t- Four variants of model architecture. \n  \t    - Variant 1 with alternating spatial and temporal attention performs best\n  \t- Two video patch embedding.\n  \t\t- Uniform frame patch performs better than compression frame patch embedding\n  \t- Timestep-class injection.\n  \t\t- Adaptive layer norm perfoms better than all-tokens\n  \t- Temporal positional embedding. \n  \t\t- absolute performs slightly better RoPE embedding\n  \t- Learning strategy. \n  \t\t- Randomly initialized model performs better than imagenet initialized \n  \t\t- Image-Video joint training performs better than video-only training", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses", "broader_impact_concerns": "Given the video generation model, the authors may discuss the possible misuse of the model.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers consider the novelty of this work to be limited, but the majority appreciate the experimental evaluations/findings and proposed techniques are useful to the community. The editor agrees with most reviewers and recommends it be accepted with the following revisions:
* The authors should revise this paper based on the comments and responses.
* The authors should revise the claims and emphasize empirical contributions rather than algorithmic ones. 
* The authors are strongly encouraged to make the source codes and trained models available to the public as those are the main contributions of this work, and would make this work more impactful.
Decision: Accept (poster)