Survey of Video Diffusion Models: Foundations, Implementations, and Applications
Authors: Yimu Wang
Xuye Liu
Wei Pang
Li Ma
Shuai Yuan
Paul Debevec
Ning Yu
Venue: Accepted by TMLR ()

Abstract: Recent advances in diffusion models have revolutionized video generation, offering superior temporal consistency and visual quality compared to traditional generative adversarial networks-based approaches. While this emerging field shows tremendous promise in applications, it faces significant challenges in motion consistency, computational efficiency, and ethical considerations. This survey provides a comprehensive review of diffusion-based video generation, examining its evolution, technical foundations, and practical applications. We present a systematic taxonomy of current methodologies, analyze architectural innovations and optimization strategies, and investigate applications across low-level vision tasks such as denoising and super-resolution. Additionally, we explore the synergies between diffusion-based video generation and related domains, including video representation learning, question answering, and retrieval. Compared to the existing surveys (Lei et al., 2024a;b; Melniket al., 2024; Cao et al., 2023; Xing et al., 2024c) which focus on specific aspects of video generation, such as human video synthesis (Lei et al., 2024a) or long-form content generation (Lei et al., 2024b), our work provides a broader, more updated, and more fine-grained perspective on diffusion-based approaches with a special section for evaluation metrics, industry solutions, and training engineering techniques in video generation. This survey serves as a foundational resource for researchers and practitioners working at the intersection of diffusion models and video generation, providing insights into both the theoretical frameworks and practical implementations that drive this rapidly evolving field.

Published in Transactions on Machine Learning Research (MM/YYYY)
Survey of Video Diffusion Models: Foundations, Implementations, and Applications
Yimu Wang"
gömu.wang@ruuaterlo..om
University of Waterloo
Xuye Liu University of Waterloo
may..ina@nnnateeroo..o
Wei Pang" University of Waterloo
w:paang@uwaterloo.co
Li Ma¹
Hima@scaniineuff.ccm
Netfliz Eyeline Shudios
Shuai Yuan" x Duke University
shuai@tss.c Menaai@cs..uuke.de edu
Paul Debevec Netflir Eyeline Shudios
debevee@@sconlieeff..om
Ning Yu! Netflix Eyeline Studios
ning. Sinng.y@@accnniieeuf.ccmmm
Reviewed on OpenReview: https: // openreview. net/ forum? id= 20DDBObK
Abstract
Recent advances diffusion models have revolutionized video generation, offering supe- rior temporal consistency and visual quality compared traditional generative adversarial betworks-bascee approaches. While this emerging field shows tremendous promise in appli- cations, it faces significant challenges in motion consistency, computational efficiency, and ethical considerations. This survey provides a comprehensive review of diffusion-based video generation, examining its evolution, technical foundations, and practical applications. We present a systematic taxonomy current methodologies, analyze architectural innovations and optimization strategies, and investigate applications across low-level vision tasks such as denoising and super-resolution. Additionally, we explore the synergies between diffusion- based video generation and related domains, including video representation learning, ques- tion answering, and retrieval. Compared to the existing surveys (Lei et al.) 2024a b Melnik et al. 2024 Cao et al. 2023 Xing et al. (024c) which focus on specific aspects of video generation, such as human video synthesis Lei et al.) 2024a) or long-form content genera- tion (Lei et al. 2024b), our work provides a broader, more updated, and more fine fine-graine grainedind perspective on diffusion-baase dustry solutions, and training engincering techniques in video generation. This survey serves foundational resource for researchers and practitioners working at the intersection of dif- fusion models and video generation, providing insights into both the theoretical frameworks and practical implementations that drive this rapidly evolving field. A structured list of re-
approaches with a special section for evaluation metrics, in-
"These authors contributed equally to this work i "Correpponding author.

---

Published in Transactions on Machine Learning Research (MM/YYYY)
lated works involved in this survey is also available on GitHub: https://github.coo EEyeeliii-- Babs/Survey-Videe-DDfiss
Contents
Introduction
4
Foundations
5
2.1
Video generative paradigms
5
2.1.
GAN video models
5
2.1.2
Aulo-regressive video models
6
2.13
Video diffusion models
2.1.4
Auto-regressive video diflusion models
2.2
Learning foundations
2.2.1
Denoising diffusion probabilistic models (DDPM)
7
2.2.2
Denoising diffusion implicit models (DDIM)
8
2.2.3
Elucidated diffusion models (EDM)
8
2.2.4
Flow matching and rectified flow
9
2.2.5
Learning from foodback and reward models
2.2.6
One-shot and lew-shot learning
10
2.2.7
Training- free methods
11
2.2.8
Token learning
11
2.3
Guidances
12
2.3.1
Classilier guidance
2.3.2
Classifier-free guidance
13
4
Diffusion model frameworks
13
2.4.1
Pixel diffusion and latent diffusion
14
2.4.2
Optical-flow based diffusion models
15
2.4.3
Noise scheduling
2.4.4
Agent-based diffusion models
16
2.5
Architectures
16
2.5.1
UNet
16
2.5.2
Diffusion transformers
19
2.5.3
VAE for latent space compression
2.5.4
Text encoders
21
Implementations
23
3.1
Datasets
23
2

---

Published in Transactions on Machine Learning Research (MM/YYY)
3.2
Training engineering
23
3.3
Evaluation metrics and benchmarking findings
27
3.4
Industry solutions
28
Applications
30
Conditions
4.1.1
Image condition
30
4.1.2
Spatial condition
33
4.1.3
Camera parameter condition
34
Audio condition
35
4.1.5
High-level video condition
37
4.1.6
Other conditions
39
2
Enhancement
39
4.2.1
Video denoising and deblurring
40
4.2.2
Video inpainting
40
4.2.3
Video interpolation and extrapolation/ ppriiictiooniiiiiinnn
42
4.2.4
Video super-resolution
43
4.2.5
Combining multiple video enhancement tasks
................................
44
4.2.6
Summary
45
3
Personalization
45
4
Consistency
47
4.5
Long video
48
6
3-awweee video diffusion
50
4.6.1
Training on 3D datasel
50
4.6.2
Architecture for 3D difiisoo models
51
4.6.3
Camera conditioning
52
4.6.4
Inference- tricks
53
Benefits to other domains
53
5.1
Video representation learning
53
Video retrieval
54
5.3
Video QA and captioning
54
5.4
3D and 4D generation
54
5.4.1
Video dillusion for 3D generation
55
5.4.2
Video diffusion for 4D generation
55
6
Ethical Considerations
56
3

---

Published in Transactions on Machine Learning Research (MM/YYY)
Conclusion and outlook
58
1 Introduction
Benefits to other domains
Representation learning
Retrieval/Classificatto
QA/Understanding
3D and 4D generation
Applications
Conditions
Personalization
3D aware generation
Noise
Videos
Audios
Video generation models
Consistency
Video inpainting
Images
3D
Text
Long video
Denoising/Debluringg
Spatial
Camera Information
Super-resolution
Inter- and extra-polation
Foundations
Paradigms
Architectures
GAN
EDMs
UNet
VAE
DDPM
Flow matching
Training data
Implementations
Transformers
DDIM
Rectified flow
Real-world data
Training engineering
Evaluation and benchmarking
Text-encoders
Auto-regressive models
Synthetic data
Figure Overview of video generation methods. Generally, the input conditions can be noises, images, videos, audios, texis, and 3D point clouds. The architectures (UNet, VAF, and/or Transformers) are trained using GAN or diffusion models with training data of real-world data or synthetic data with different paradigms. The applications are in several folds, e.g., video personalization, consistency-aweat generation, and long video generalization. On the other side, video generation models can also benefit other video tasks, such as, video retrieval, understanding, and representation learning.
Vidco gencration (Ren et al. 2024 Zheng et al. 2024c Davtyan & Favaro 2022 has emerged as a critical and transformative technology in recent years. The ability to generate high-quality, realistic videos has numerous applications, from entertainment and advertising (Wang Shi 2023) to virtual reality (Hu al. 2021) and autonomous systems (Zhou al.) 2024c),, further enabling enhanced user experiences, cost-effective content creation, and new avenues for creative expression.
In the past few years, video generation (Clark et al. 2019 Aldausari et al. 2022 Hong et al. 2022 has
experienced remarkable progress, particularly with the adoption of Generative Adversarial Networks (GANs, [Goodfellow et al. 2014)). Researchers have implemented various strategics to improve the temporal coher- ence (Chai et al. (202),, realism, and diversity of generated videos. Despite these advancements, G-N-based mothods often face challenges in training stability and achicving high-quality outputs consistently. The in- troduction of diffuusion models (Ho et al. 2020; Nichol & Dhariwal 2021 (Sohl-Dickstein et al. 2015) has revolutionized this area by offering probabilistic framework that overcomes many limitations of GANs. Diffusion-based models (Kwak et al.] 2024 Chai et al. 2023 Wang & Yang 2024 Ho et al. 2022c) have demonstrated superior performance in generating temporally consistent and visually compelling videos, m1- tivating further research in the domain.
However, diffusion-based vidco generation presents several fundamental challenges. primary concern lies ensuring motion consistency across frames, critical factor for producing temporally consistent and realistic videos. Moreover, generated videos must adhere to physical rules, such accurate object dynamics and
4

---

Published in Transactions on Machine Learning Research (MM/YYYY)
environmental interactions, to maintain realism. Long video generation poses another challenge, requiring models to handle complex temporal dependencies over extended sequences. Furthermore, the computational demands of training diffusion- based models are substantial, often leading to inefficiencies that limit scalability. Similarly, inference acceleration remains a severe issuc, as generating high-quality vidcos in real-time is critical for many applications. Beyond these technical challenges, ethical considerations also play a pivotal role, including mitigating biases generated content and preventing the creation of harmful or misleading visual outputs.
In response to the rapid advancements and emerging challenges of diffusion-basse video generation, this survey provides a systematic analysis of current methodologies, recent advances, and future directions in the field.
Our contributions. This survey presents a comprehensive analysis of diffusion- video generation, focusing on both technical foundations and practical applications. While cxisting surveys (Lei et al. 2024a Melnik al., 2024 Cao al. 2023 Xing et al. 2024c] have addressed specific aspects of video generation, such human video synthesis Lei et al. 2024a or long-ford fong-form content generation (Lei et al.) 2024b),
work provides broader, more updated, and more fine-graines perspective diffusion-based approaches. Compared to related surveys (Xing et al.) 2024cc Melnik et al. (024)), we offer more extensive coverage of diffusion models and their applications with a detailed overview of datasets, evaluation metrics, industry solutions, and training engineering techniques for video generation. The main contributions of this paper are as follows:
To the best of our knowledge, this is the most comprehensive survey diffusion-based video gen- eration, including model paradigms, learning foundation, implementation details, applications, and relations to other domains.
Compared to related surveys, our survey covers broader scope of diffusion models and their ap- plications with a detailed overview of datasets, evaluation metrics, industry solutions, and training engineering techniques for video generation.
The rest of the survey is organized follows: concepts, including video generation paradigms such GAN-based models, auto-regressive models, and dif- fusion models. Section engineering techniques, and evaluation metrics, along with benchmarking findings to highlight model perfor- mance. Section such as video denoising, inpainting, interpolation, extrapolation, and super-resolution, as well as personaliza- tion, consistency, long-video generation, and emerging 3D-aware diffussio models. Finally, Section Dectionddiscusseee the benefits of cratusioon-based video generation other domains, such as video representation learning, re- trieval, QA, and 3D) and 4D generation, emphasizing its broader impact in advancing related fields. This structure provides comprehensive and cohcsive understanding of diffusion-based vidco generation, from foundational principles to advanced applications.
Section lays the groundwork by exploring foundational
g focuses on implementation, addressing practical aspects such as datasets, training
| covers diverse applications, including conditional generation tasks, enbancement methods
2 Foundations
2.1
Video generative paradigms
2.1.1
GAN video models
Generative Adversarial Networks (GANs) employ two-player minimax game between generator and discriminator, where the generator learns to produce realistic samples while the discriminator distinguishes between generated and real data. The objective function (Goodfellow et al. 2014 is formulated as:
min max V(D, G) E (mt) llgg D(x)) + Ez-p. (=) [log(1 D(G(z)))
(1)
where and D are the generator and discriminator networks, a real sample, and 2 is a random noise vector. (xx) indicates the probability that is real, while D(G(=)) reflects how real the generated sample
5

---

Published in Transactions on Machine Learning Research (MM/YYYY)
appears. For video generation, GANs have been extended to model temporal consistency through various architectural innovations and training strategies.
Early video GANs focused on temporal modeling through specialized architectures. TGAN (Saito et al. 2017] employs dual generators: a temporal generator creates motion features while an image generator pro- duces video frames. TGAN-v2 (Saito et. al., 2020) improves efficiency through multi-resolution generation using cascaded generator modules. MoCoGAN (Tulyakov et al., 2018) disentangles latent space into motion and content components, predicting motion vectors auto-regressively, while DVD-GAN (Clark et al., 2019) generates all frames in parallel with class conditioning. MoCoGAN-HD (Tian et al.
2020 advances motion-
content discntanglement through latent motion trajectory prodiction. DIGAN (Yu et al. 2021 leverages
implicit neural representations for efficient long-sequence generation with frame-pair focused motion discrim- ination.
StyleGAN2 Karras et al.] 2020) introduced expressive latent spaces for photorcalistic synthesis, estab- lishing foundations for video generation through sparse motion cues. StyleGAN-V (Skorokhodov et al., 2022 models videos as continuous-tinng time signals, enhancing temporal coherence for high-resolution, long- duration generation. Stylelnv (Wang et al. (023i) proposes non-autoregressive GAN inversion for smoother motion transitions and style transfer. AniFaceGAN (Wu et al. 2022c) adapts StyleGAN2 into 3D-aware GANs for multiview-coosiiseent face animation. Recent advances include SIDGAN (Muaz et al. 2023) for high-resolution dubbed video generation through shift-invariant learning and pyramid- based decoding, and StyleLipSyne (Ki & Min 2023) for temporally consistent lip-synced videos with pose-aware masking and latent smoothing.
Large-scale video generative models have emerged as significant advancement. Open and Advanced Large- Scale Video Generative Models (Wan et al. 2025) demonstrate the potential of scaling GAN archilect for complex video synthesis tasks. Hunyuan- Large (Sun et al., 2024b) introduces a 52-billion parameter mixture-of-oxperts model, demonstrating the ability of large-scale architectures to handle complex video generation tasks.Ther These advances go beyond traditional GANs and bring models that are more powerful and efficient for large-scale video generation.
2.1.2 Auto-regressive video models
Auto-Regressee models generate data by modelling the conditional distribution of each data point given its procoding data points. In the context of vidco gencration, Progresseeeeiv models generate realistic and coherent videos by predicting each frame based on previously generated frames. Generally, three strategies have been employed. The first is pixel-level auto-regression. Traditional representative methods, such Video Pixel Networks (VPN) (Kalchbrenner et al. 2017), used LSTMs to capture temporal dependencies and PixelCNN (Reed et al.) 2017) to model spatial and color dependencies within each frame. The second is frame-level autooregression. (Huang et al. 2022a) proposed auto-regressive GAN to predict frames based on a single still frame. The third is latent-level regresegr,,s,,,, which significantly saves processing time due to reduced data redundancy and achieves a good time quality trade-off (Rakhimov el al. 2020 Seo et al.
2022 Yan et al. 2021 Ge et al., 2022 Hong et al., 2023a
2.1.3 Video diffusion models
GAN-based and Auto-Regressive models often face challenges in maintaining temporal consistency and com- putational efficiency when generating videos. While GANs struggle with maintaining temporal consistency and often produce unstable or inconsistent frames, diffusion models excel at generating smooth, coherent sequences, particularly for long videos, because their iterative denoising process ensures frame-to-frane con- sistency. Also, despite Regressvee model can handle both continuous and discrete data, it normally requires high computational cost. In terms of frame quality, diffusion models generale more detailed and outputs compared to GANs, which are prone to artifacts from adversarial training, and autoregressive models, which face resolution limitations due to the compounding errors in soquential pre- diction (Moser al. 2024). Diffusion models also offer superior controllability and local editing capabilities because they can refine specific parts of the video during the denoising process without affecting other regions

---

Published in Transactions on Machine Learning Research (MM/YYYY)
while GANs and autoregressive methods generate frames in more holistic sequential manner Huung et al. 2024a).
Diffusion models offer a more efficient solution by framing video generation as a denoising process. Early models, such as the Video Diffusion Model (VDM) (Ho et al. 2022c adapted the U-Net architecture to 3D for spatio-temporal video generation but faced high computational costs and limited resolution.
Later, Imagen-Vided (Ho et al., 2022a) uses cascaded diffusion process (Ho al., 2022b) to generate high-resolution videos. It introduces temporal attention layers between spatial layers to capture motion information effectively. Make-a-Vides (Singer et al., 2022) is another powerful competitor in text-video synthesis by conditioning on the CLIP semantic space. It generates keyframes based on text input and uses a cascade of interpolation and upsampling diffusion models to ensure high consistency and fidelity. However, both of these pioneering models come with high computational costs. To address this, Magic Video (Zhou ct al. 2022) introduced low-dimensional latent embodding space in their diffusion process by pretrained (VAE), significantly reducing compulational demands. Video LDM (Blatlmann) et al. 2023b) also extended the adaptation of the Latent Diffusion Models (Rombach et al.) 2022 to text-to- video generation tasks by adding temporal attention layers to pre-trained toxt-to-image diffusion model and fine-tune them on labeled video data. Their model is capable of generating long-driving car video sequences auto-regressively and producing videos of personalized characters. Recently, CCEdit. (Feng et al.) 2024a) leverages diffusion models with a trident network architecture that decouples structure and appearance control, ensuring precise and creative editing capabilities.
periational-autt-ennee
2.1.4 Auto-regressive video diffusion models
Some recent works further exploit regr-sege framework in the context of video generation with diffusion models (Weng et al. 2024 Jin et al. 2024 Xie al.) (0244).. ART-V (Weng et al.] 2024) first utilized autoregressive diffusion to generate frames soquentially, employing masked diffusion to enhance coherence and reduce inconsistencies. Subsequently, Pyramidal Flow Matching (Jin et al. 2024) introduced spatial and temporal pyramids to optimize autoregressive generation, reducing computational costs and improving scalability. Progressive Autoregrossive Video Diffusion Models (Xie et al., 2024a) refined this by using a progressive noise schedule, enabling smoother transitions in long videos. Caus Vid (Yin et al. 2024) advanced the field by converting bidirectional diffussin models into causal autoregressive generators through asymmetric distillation, achieving real-time frame- frame-by-frand by-frame generation with low latency and reduced error accumulation.
2.2 Learning foundations
Video generation relies on different methods to produce realistic and coherent results. As the section discussed above, common paradigms in Figure I incluude GANs, autoregressive models, and diffussio models, each with trade-offs in quality and efficiency. This section introduces the core learning principles behind these methods, starting with diffusion processes, which have become the leading approach for high-quality and temporally consistent video generation. These foundational concepts serve as the basis for the specific diffusion model frameworks and implementations discussed in subsequent sections.
2.2.1 Denoising diffusion probabilistic models (DDPM)
Denoising diffusion probabilistic models (DDPM) Ho et al. )0200;; Nichol & Dhariwal (0021); Dickstin et al. 2015) consist of two interconnected Markov chains: forward diffusion process that gradually adds noise to the data, and a reverse process that removes noise to recover the data. The forward process transforms complex data distributions inlo a simpler prior, typically Gaussian noise, while the reverse process learns how to reconstruct the data from this noisy representation by modeling the reverse transitions. Data gencration involves first sampling a random noise vector from the prior, and then applying the reverse chain, step-by-step, to iteratively refine the noise into meaningful data. The key challenge lies in training the reverse Markov chain that it effectively mimics the true reversal of the forward process.
7

---

Published in Transactions on Machine Learning Research (MM/YYYY
Po px x
XT
Xt
X
X
g(xe Xt 1)
Figure The directed graphical model for DDPM al. 2020
Mathematically, let the data be sampled from a distribution Zo g( The forward Markov process produces a sequence of progressively noisier variables σı, 22, -2,,.-, or through a transition kernel (( The joint distribution of these variables, conditioned on the original data zo, can be expressed as:
T
(T( = II q<<+(==-1).
(2)
Typically, the transition kernel defined as Gaussian distribution:
q(=))=t-1)= = Nax 11 8,
(3)
where 3e E (0, 1) is a pre-determined noise schedule controlling the amount of noise added at each step. In the reverse process, the goal is to recover the data by sampling from reverse Markov chain. The chain starts with noise sample 2r p(ær) (o, I) from simple Gaussian prior, and transitions through learnable kernel po (x parameterized by neural networks. This kernel takes the form:
po( = (i (), Eg(x,, ()),
(4)
where uo( t) and Cx( represent the learned mean and variance, respectively. By iteratively applying this reverse kernel, we generate the data zo through sequence of updates starting from the noise sample Figure> illustrates the graphical model for DDPM.
ur and progressing through #T-1,
This iterative sampling process captures the reverse dynamics the original forward diffusion, gradually refining random noise into meaningful data point, such as an image or text.
2.2.2 Denoising diffusion implicit models (DDIM)
This original DDPM formulation of the generation process in the form of a reverse Markov chain has more recently been complemented by a non-Markovian alternative denoted as denoising diffusion implicit models (DDIM), which offers deterministic and more efficient generation process. In DDPM, the sampling process typically requires a large number of iterations achieve satisfactory results. Tn contrast, the DDMM is designed to accelerate the sampling process by reducing the number of iterations needed. Here, a backward denoising step can be computed with
2t-1 VaL-121- VI (I1
/1 ax-160(x1,
Voa
One distinct advantage of this formulation of the denoising process is that it allows for accurate reconstruction of the original input vidco øy from the noise at time step This technique, called DDIM inversion, can be utilized for applications such as image and video editing.
2.2.3 Elucidated diffusion models (EDM)
EDM Karras et al. 2022) introduce a unified and optimized framework for diffusion-based generative models by reinterpreting sampling and training processes. Unlike DDPMs, which rely on stochastic Markovian sampling, or DDIMs, which employ deterministic non-Markovian updates, EDMs aim to optimize both training and sampling efficiency by introducing tailored sampling schemes and reparamcterization strategics. EDMs build upon the common framework of diffusion models by considering the data distribution q perturbed by Gaussian noise, leading to a sequence of noisy variables {x+) over time.
8

---

Published in Transactions on Machine Learning Research (MM/YYYY)
While DDPMs use a stochastic reverse process, EDMs leverage ordinary differential equation (ODE) for deterministic sampling, as inspired by DDIMs. The probability flow ODE Song et al. (2020a) is expressed as:
dæ dt = (t) log pog at)),
(6)
where (t) is the noise schedule at time t. Unlike DDIMs, EDMs employ second-order solvers, such as Heun's method, to balance accuracy and computational efficiency.
The To ensure stable training, EDMs redefine the loss function with noise-dependent preconditioning. denoising target Do(æ; 0) is expressed as:
De(x;o) Cokip(o)xz + Cout (noc))))))
(7)
where Fg is a neural network, and Cskips Cout: and Gin are scale-dependent coefficients. The overall training objective becomes:
== Ea, (o)) + oe;0) - =ull?]
where A(o) is weighting function that emphasizes intermediate noise levels during training.
Compared to DDPMs, EDMs achieve faster sampling by reducing the number of neural function evaluations through efficient solvers and noise schedules. Unlike DDIMs, which rely on deterministic updates, EDMs integrate adaptive techniques to minimize truncation errors further, further improving fidelity and flexibil- ity. This unified and modular design allows EDMs to achieve better results while significantly reducing computational BeerheadKarrea et al. 2022
2.2.4 Flow matching and rectified flow
Despite denoising diffusion being powerful for video generation tasks, another branch in the family of differentiallqquatoon--aseed generative models began to arisc recently, namely the flow matching genera- tive models [ipmman models(Lipman et al. 2023 Liu et al. 2022c Tong et al. 2023). While diffusion models learn the score function of specific SDE, flow matching aims to model the vector field implied by an arbitrary ODE directly. A neural network is used for approximating the vector field, and the ODE can also be numerically solved to obtain data samples. The design of such ODE and vector field often considers lincarizing the sampling trajectory and minimizing the transport cost postiTon TTog et al., 2023 As a result, flow matching models have simpler formulations and fewer constraints but better quality. VoiceBox(Le al. 2023) shows the potential of flow matching in fitting large-scale speech data, and LinDiff(Liu et al. 2023b) shares similar concept in the study of vocoders. More importantly, the rectified flow (Liu et al. 2022c) technique in flow matching models further straightens the ODE trajectory in a concise way. By training a flow matching model again but with its own generated samples, the sampling trajectory of rectified flow theoretically approaches a straight.orward line, which improves the efficiency of sampling. Reclified FlowTLiu el al. 2022c 2023d) is a generative model that efficiently transitions between two distributions To and πı by solving ordinary differen- tial equations (ODEs). Building on flow matching matching(Lipman et al., (203)) and stochastic interpolants/ interpolantsAAlbbrgg (Alborgo et al. 2023), Rectified Flow introduces rectification mechanism to optimize the ODE path, improving the efficiency and stability of the sampling process. This approach establishes a strong theoretical baseline for diffusion acceleration and provides unified perspective generative models.
The core dynamics of Rectified Flow are governed by the following ODE:
dz, = t)d.
where Z, represents the data point at time t E [0, 1), which evolves contimuously along the flow. The vector field o( o(Z4.) I) determines the direction and velocity of Z, 's movement. This vector field is parameterized using a neural network, which is optimized to align the trajectories with the target distributions. The term represents an infinitesimal time step, enabling smooth transition betwoen To and NT.
To learn the vector field v, Rectified Flow minimizes the following objective:
min [ E [|[(x] Xo) x(Xx,))113 dt.
9

---

Published in Transactions on Machine Learning Research (MM/YYYY)
where Xo and X1 are sampled from the initial distribution To and target distribution Xi, respectively. The term X, = (X1 + (1 - ()Xx represents a linear interpolation between Xo and X1. The loss function measures the squared difference between the actual direction of interpolation (X) and the vector field u(X,,t). This ensures that v effectively captures the necessary dynamics to transition samples between the two distrributions while preserving their structure.
A key theoretical advantage of Rectified Flow is that the ODE formulation guarantees that paths do not cross. The vector field t(z,,t ensures unique and deterministic trajectories for all starting points, avoiding ambiguities in sampling and maintaining stability during inference. With its focus on optimizing vector fields and straightening trajectories, Rectified Flow offers a powerful and efficient alternative to traditional diffusion models. By combining a strong theoretical foundation with practical elficiency, it sets a new benchmark for generative modeling tasks.
2.2.5 Learning from feedback and reward models
The integration of human feedback into vidco diffusion models has evolved significantly, beginning with foundational ideas in reinforcement learning and progressing to advanced mechanisms leveraging multimodal and AT-driven annotations. Farly work C2M((rddin et al.] (021) user interaction by selecting objects in the scene and specifying their final location through mouse clicks to generate video in complex scenes. InstructVides (Yuan et al.) (2023) further emphasized aligning video generation with human preferences by incorporating temporal coherence loss:
F
Leoberence = EE0
[15511 - 0z) (0 Σ
- oj)
(9)
where vy and 0, denote predicted and ground truth frames. This method enhancod temporal consistency but primarily focused supervised fine-tuning with explicit human feedback signals. Subsequent, research intro- duced more scalable methods for integrating feedback. (Furuta et al. 2024b) demonstrated the use of binary AI feedback from vision-language models (VLMs) to optimize dynamic object interactions in vidcos. Their unified reinforcement learning framework leveraged AT annotation to align outputs with human-like quality assessments, overcoming the limitations of manual data annotation. This concept was further extended in T2V-Turbo Li et al. 2024d) and its subsequent version T2V-Turbo- V2 et al. 2024e)), which significantly improved the quality and speed of text-to-vides generation through the incorporation of AI feedback, enhanc- 2024aa) In parallel, efforts to construct comprehensive preference ing dynamic object interaction. Farurttt.nn.uuuuaaa et al.
datasets began to take shape. (Wu et al. 2024g) presented VIDEOPREFEL a large-scale dataset created using multinodal large language models (MLLMs) such GPT-4 Vision. This dalasel provided 135,000 preference annotations, allowing the training of VIDEORM, the first general-purpose reward model tailored for video preferences. VIDEORM incorporatod temporal dynamics, significantly enhancing the alignment of video outputs with human expectations. (Furuta et al.) 2024a) further proposed a robust. framework for improving dynamic interactions in video generation, showcasing Al feedback as a critical factor for realistic object behavior in generated content. Other works, such as FreeScale Qiu et al. 2024c indireelyy addressed alignment by improving high-resolution visual generation in video diffusion models. Their approach, while not explicitly focused on feedback, laid the groundwork for integrating multi-scale visual consistency with human-preferred preferred attributes in video content. Some advanced methods, like VINE et al.) 2024b) also highlighted the robustness of feedback-aligned generative priors for specific tasks like watermarking and video editing. These innovations showed how feedback mechanisms could extend beyond video generation to broader applications, ensuring fidelity and alignment with user intent.
2.2.6 One-shot and few-shot learning
One-shot and few-shot fine-tuning techniques enable generative models to adapt to new tasks or domains with minimal training data. These approaches are particularly valuable in scenarios where acquiring large datascts is infeasible, yet high-quality outputs are required. By leveraging either a single example (one shot) or a small set of examples (few shots), these methods refine pretrained generative models to specialize in specific tasks while maintaining generalization capabilities.
10

---

Published in Transactions on Machine Learning Research (MM/YYYY)
One-shot learning refers to model's ability to learn from just single example. In generative modelling, this is particularly useful when only one instance of the target data is available. For example, Tume-A-Vided (Wul et al. 2023b) demonstrates how a pre-trained text-to-image diffusion model can be fine-tuned on a single text-video pair, enabling it to generate new videos that maintain the temporal consistency of the original while adhering to text prompls. However, Tune-A-Video based on the template video and edits with different content prompts which will restrict the freedom of generative video.
Few-shot fine-tuning extends this capability by adapting models to new domains tasks using small number of training samples. Several techniques have emerged to enhance the generative capacity of models under few-shot scenarios. LAMP (Learn A Motion Pattern) (Wu et al. 2023c) proposes a motion learning model to capture the motion pattern from the training data and utilizes about 816 videos to tune the pretrained '2I model. However, it is constrained by their incomplete capture of image features. MAIM (Huang al., 2025) addresses this by integrating image features via the CLIP encoder, enabling videos with intricate scenes and multiple objects.
2.2.7 Training-free free methods
Training-free approaches enable direct video generation without additional training or fine-tuning, making them valuable for black-box scenarios.
Text2Video-Zert Khachatryan et al.
2023
introduced cross-frame attention based on pretrained text-to-
image diffusson model. FaleZero [Qi et al. 2023) enhanced this through intermediate attention maps, and Pree-Bloom (Huang et al. 2023a) combined GPT-3 with image diffuusinn models for improved semantic coherence. PEEKABOO (Jain et al. 2024b) enabled spatio-temporal control through masked diffusion using a latent diffusion model-based video generation model, while ControlVideo (Zhang et al. 2023e) extended ControlNet with an interleaved-frame smoother to generate consistent controllable and structurally smooth video. DreamVideo-2 (Wei al., (024bb) proposed zero-shot subject-driven video customization with precise motion control, enabling beajject-preeerving and motion-specifie video editing. Following this, VideoElevator (Zhang et al. 2024g) encapsulates T2V to enhance temporal consistency and harnesses T21 to provide more faithful details. Recent innovations include GPT4Motion (Lv et al., 2024)'s 's combination of GPT-4, Blender, and text-to-image diffusion models for physics-aware generation, Any V2V et al. 2024)'s 's universal editing framework built on an image editing model combined with an existing amage-to-video generation model to generate the edited vidco through temporal fcature injection. Video Custom Diffusion (Magic-Me) (Ma) et al.] 2024cb built a training-free free 3D Gaussian Noise Prior for video frame initialization, reconstructing inter-frame correlation to achieve identity-preserved generation.
2.2.8 Token learning
Token learning, such as Textual Inversion Gal et al. (2022). is another paradigm within generative modeling. Instead of fine-tuning the entire model, token lcarning focuses on training specific embeddings tokens that represent new concepts ideas. By using textual inversion, the model learns new token that can be combined with existing tokens to guide generation without requiring large datasets or significant changes to the model's architecture. However, optimizing the single token embedding vector has limited expressive capacity because of its limited optimized parameter size. In addition, using one word to describe concepts with rich visual features and details is very hard and insufficient. Animate-A-Story (He et al. (023a) ex- tended traditional textual inversion to timestep-varisn variable textual inversion (TimcInv) and adapted it to vidco generation lask to ensure flexible controls over structure and characters. DreamVideof (Wei al.) 2024c) further incorporated textual inversion to capture the fine appearance of the subject from provided images to generate videos with customized subjects.
The learning foundations discussed in this section provide the core principles behind modern video generation. These fundamental concepts provide the mathematical and conceptual paradigm, as shown in FFiure 1| which are necessary for undorstanding the specific diffusion model implementations and architectural designs explored in the following sections. Building upon these foundations, |subsection 2.4 examines the practical frameworks and implementations to these theoretical principles into real-world video generation systems.
11

---

Published in Transactions on Machine Learning Research (MM/YYYY
2.3 Guidances
2.3.1
Classifier guidance
In generative modeling, diffusion models such as DDPM and DDIM can be guided using classifiers to improve the quality of conditional generation. This approach, called classifier guidance (Dhariwal & Nichol 2021), integrates class information into the diffusion process by leveraging the gradients of a classifier trained on noisy data. The classifier's gradients help the model generate data conditioned on a given label, improving the overall sample quality for specific categories.
Mathematically, the dassifier-guide diffusion process modifies the reverse noising process by incorporating the classifier's log-probability gradients. This guidance is applied each timestep of the diffusion process, allowing the model to generate more accurate conditional samples.
The reverse noising process is represented as follows:
= Z (241)) po ( xx))
(10)
where pe is the reverse noising process from the original diffusion model, po (y(() the classifier's conditional probability, and Z is a normalizing constant.
We now describe how this approach be applied to both DDPM and DDIM frameworks. In particular, they(Dhariwal & Nichol, 2021) train classifier polylæ,,t) t) noisy images Ze, and then use gradients Vx, 21,, to guide the diffusion sampling process towards an arbitrary class label y.
Algorithm for classifier-guided DDPM sampling. In the DDPM framework, classifier guidance can incorporated by modifying the reverse diffusion process. The update at each timestep involves sampling from Gaussian distribution with a mean adjusted by the classifier gradients. Detailed workflow can be referred to Algorithm algorithm
Algorithm 1 Classifierguuide DDPM sampling, given diffusion model (wa(z)), Me (xx)) classifier po (y(y)Z)) x+). and gradient scale s. (Dhariwal & Nichol 2021)
1: Input: class label gradient scale s
2: Fr + sample from I0,
3: for = T to do
μ1 E (xxe), (x:)
ft-1 + sample from N(p + NVV, log logpo(ylar), E)
end for
7: return 20
Algorithm for classifier-guided DDIM sampling. For the DDIM framework, the deterministic nature of the sampling process requires a slightly different approach. The classifier gradient is used to modify the epsilon prediction, which guides the reverse sampling process DDIM. Deatailed workflow can be referred Algorithm algorithm
Algorithm 2 Classifier-guided DDIM sampling, given a diffusion model cœ(xz), classifier (z(()))) and gradient scale s. (Dhariwal & Nichol 2021)
1: Input: class label gradient scale s
2: fr +- sample from (00
3: for = to do
6 c co(x)) 11-avV log P (3)
5: Ft-1 -oi- laase + 1
Va:
end for return 20
12

---

Published in Transactions on Machine Learning Research (MM/YYYY)
In the DDIM case, it defines new epsilon prediction ((2)), which incorporates the classifier's guidance into the joint distribution of the diffusion model and the classifier:
{(x²) == (x)) / 1 av log log
(11)
This modified epsilon prediction is then used to guide the reverse sampling process, replacing the standard noise predictions from the original model.
The development of classifier guidance for diffusion models has sparked significant advances in video genera- tion applications. (Shi et al., 2023a) explored compositionality visual generation by using latent classifiers to guide diffusion models in semantic space, demonstrating effective control over multiple attributes and their relationships. (Chung et al. 2024) further reformulated classifier guidance to better respect manifold constraints crucial for temporal consistency in videos. (Lian et al. 2024) then incorporated large language models to generate dynamic scene layouts as intermediate representations, effectively guiding the diffusion process for coherent motion and interactions. These developments demonstrate the field's progression from basic guidance mechanisms to more sophisticated approaches specifically tailored for video generation chal- lenges.
2.3.2 Classifier-free guidance
While classifier-based guidance successfully allows for the conditional generation of data by leveraging the gradients from a pre-trained classifier, has several limitations. For instance, classifier guidance requires the maintenance of a separate classifier, and its performance is closcly tied to the quality of this classifier. To address these concerns, classisier-free guidonce(Ho & Salimans, 2022) offers alternative that achieves similar results without the need for dedicated classifier.
Classifier-free guidance modifies the model's predicted noise directly by using conditioning information, allowing for conditional generation without relying on external classifier gradients. This approach avoids the complexities training and maintaining an additional classifier.
Rather than training a separate classifier model, they(Ho, Salimans 2022 adopt an approach that jointly trains an unconditional denoising diffusion model, po(x), and a conditional model, po(x)y). Both models are parameterized through shared score estimator: co(x)) for the unconditional case and co(2), y) for the conditional case. They use a single neural network to parameterize both models and then perform sampling using a linear combination of conditional and unconditional scores, as shown in Equation 12
(0( v) = (1 + w)) -wee(()))
(12)
Here, w is a guidance scale that controls the strength of the conditional information. This formulation allows the model to perform conditional sampling without relying on classifier gradients.
Algorithm for Classifier- Guidance describes the sampling procedure for classifier-free guidance, which adjusts the score estimates directly within the diffusion model without requiring any classifier gradi- ents. Algorithms algorithm and algorithm describe training and sampling with classifier-free guidance in detail.
This algorithm leverages the classifier-free guidance to adjust the sampling procedure by computing the guided score at each timestep. Instead of using gradients from a classifier, the model directly modifies its own score estimates to account for the conditional information.
2.4 Diffusion model frameworks
This section explores several key frameworks within diffusion models, highlighting their distinctive approaches and applications.
This section builds on the theoretical foundations from subsscction 2.2 by introducing practical diffusion model frameworks for video generation. While the previous section focused principles, this section
13

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Algorithm 3 Joint training a diflusion model with classifier-free guidance
Require: Puncond: probability of unconditional training
1: repeat
2: (z, y) ~ p(z,y)
D Sample data with conditioning information from the dataset Randomly discard conditioning to train unconditionally Sample log SNR value
3: y with probability Puncond
A~p(N)
1)
2A axz oxe
Corrupt data to the sampled log SNR value > Oplimize the denoising model
7: Take gradient step on Volleo(<x.1)) e|l²
8: until converged
Algorithm 4 Conditional sampling with classifier free guidance
Require: w: guidance strength
Require: e conditioning information for conditional sampling
Require: A1,...,Ary increasing log SNR sequence with Å Amin AT Amax
2y 1))
2: for t= 1..... T do
3: é- + (1 w) (2) -wco(x-1))
Form the classificr-free guided score at log SNR A-11 A Sampling step (could be replaced by another sampler, e.g., DDIM)
(a) 1-
4: gtt +
01--1
2 ~ 1 (11-1, Kz), 11), (.
6: end for
7: return FT
focus illustrating how diffusion-based video generation builds theoretical foundations through range of practical strategies. The following sections examine four key implementation approaches: pixel-based and latent-based modeling, optical flow integration, adaptive noise scheduling, and agent-driven frameworks, each addressing specific challenges in video synthesis such as temporal consistency, efficiency, and controllability.
2.4.1 Pixel diffusion and latent diffusion
The theoretical diffusion processes introduced in isubsection 2.2 can be implemented through different ar- chitectural approaches, each offering distinct advantages for video generation. Pixel-based and latent-bace based diffusion models represent two fundamental implementation strategies that directly translate the mathe- matical paradigm of DDPM, DDIM, and other diffusion variants into practical video generation systems as shown in Figure
Remarkable progress has been made in developing large-scale pre-trained Bext-to-Vides Diffusson Models (VDMs), including proprictary models (e.g.. Mako-A-Video (Singer et al. 2023 Imagen Video (Ho et all 2022et. Video LDM (Blattmann et. al. 2023b), Gen-2 (Esser et al. 2023 and open-sourced ones (e.g., VideoCrafter (He et al. 2023b), ModeIScope12V (Wang et al. 2023ch These VDMs can be categorized into two primary architectures: (1) Pixel-based VDMs and (2) Lattent-base VDMs. The former directly operates on pixel values for denoising, as exemplified by Make-A-Vin Video Singer et al. (2023). Imagen Video Ho et al. (2022c), and PYoCo al. (2023). The latter manipulates the compressod latent space within variational autoencoder (VAE), demonstrated by Video LDM Blattmann et al. (2023b) and Magic Zhou et al. 22022) However, both of them have pros and cons. Pixel-based VDMs can generate motion accurately aligned with the textual prompt but typically demand expensive computational costs in terms of time and GPU memory, especially when generating high-resolution videos. Latent-based VDMs are more resource- efficient because they work in a reduced-dimension latent space. However, il is challenging for such a small latent space (e.g., 8 x for 64 x 40 videos) to cover rich yet necessary visual semantic details as described by the textual prompt. Thus, if the generated vidcos are often not well-aligned with the textual prompts or the generated videos are of relatively high resolution (e.g., 256 160 videos), the latent model will focus more spatial appearance but may also ignore the text-video alignment. Recently, Show-1 (Zhang et al.
14

---

Published in Transactions on Machine Learning Research (MM/YYYY)
2023a] integrates the strengths of both pixel and latent VDMs, resulting in a novel video generation model that can produce high-resolution videos of precise text-video alignment at low computational cost (15G GPU memory during inference). LATTE Ma et al. (024c) further proposes a novel latent diffusion transformer for vidco generation, which extracts spatio- temporal tokens from input vidcos and models vidco distribution in the latent space using Transformer blocks, achieving stale-of-the-arl performance across multiple video generation datasets by exploring optimal design choices such as video clip patch embedding, model variants, timestep-class information injection, and temporal positional embedding.
2.4.2 Optical-flow flow- based diffusion models
Another distinct approach within diffusson models leverages optical flow to maintain temporal coherence between frames. Optical flow is a common approach represent motion by estimating the displacement field between consecutive frames. Early approaches treated it as an optimization problem using handerafted
features to maximize visual similarity(Hoor & Schunck 1981 Black & Anandan 1993 Bruhn et al. 2005
Sun et al. 2014). Incorporating deep bearning-bssd methods revolutionized this field.
(et al. 2015) pioneered end-to-end optical flow estimation, demonstrating the potential of deep learning in this domain. Subsequent advancements on improved architectures and synthetic datasets further promoted
the progress of optical flow estimation Olg et al. 2017 Ranjan Black 2017 Sun et al.) 2018
2021 Hui
ct al. 2018 2020 Yang Ramanan 2019). RAFT & Deng 2020) included itcrative refinement with
correlation volumes, significantly boosting performance, while Flow FrloFoormerHuaaee et. al. 2022b Shi et al.,
2023cb applied attention mechanisms within optical flow usage. VideoFlow (Swi et al. 2023b) extended optical flow to vidco generation sccnarios and utilized temporal information across multiple frames, achieving high accuracy. (Hu et al. 2023 propose a Dynamic Multi-scale Voxel Flow Network (DMVFN) to explicitly model the complex motion cues of diverse scales between adjacent video frames by dynamic optical flow estimation to generate videos at lower computational costs. Also, Motion-12V (Shi et al. 2024) further refined the motion modelling by a diffusion- based motion field predictor and motion- augglnned temporal attention to generate more consistent videos even in the presence of large motion and viewpoint variation. By conditioning the optical flow between a previous and subsequent frame, this framework achieves realistic motion continuity, which is critical for applications in video generation and motion synthesis. Additionally, recent advancements explore upgrading pretrained image diffusion models to video generation by temporally warping imput noise. How I Warped Your Noise (Chang et al. 2024a) proposed a novel noise representation called integral noise (f-noise), preserving temporal correlations through an optical-flow-bssed noise warping algorithm. This foundational approach miligated issues like high-frequency flickering and texture-sticking artifacts, laying the groundwork for temporally coherent video diffusion. Building upon this, Resolution Integral Noise Warping (Deng ct al., 2025) method introduces training-free free approach where temporal correlations in noise are preserved by warping input noise using integral representations, enabling consistent frame generation while reducing computational overhead. These diffusion model frameworks demonstrate the versatility and adaptability of diffusion processes across different domains, from high-resolution static images to complex, temporally coherent videos. As shown in Figure 3 Go-with-the- Flow (Burgert et al. 2025) combined optical flow extraction from training videos with real-time noise warping and fine-tuning video diffusion models on paired warped noise and video data to enable robust and contolllo generation. Together, these methods demonstrate the progression from temporal correlation preservation to advanced motion control, showcasing the versatility and adaptability of diffusion models in video generation.
2.4.3 Noise scheduling
Diffusson models (Song et al., 2020b Ho et al. 2020 perturb data with Gaussian noise through a diffusson process for training, and the reverse process is learned to transform the Gaussian distribution back to the data distribution. Perturbing data points with noise populates low data density regions to improve the accuracy of estimated scores, resulting in stable training and image sampling. The forward process is controlled by the handcrafted noise schedule. However, current noise scheduling strategies remain handcrafted for each dataset, such as VP (Ho et al. 2020). VE (Song et al.] (020b), Cosine (Nichol & Dhariwal, 2021 and EDM [Karas et al.) 2022). These noise schodules perform well low-resolution RGB spaces but yield poorer results in higher resolutions (Dhariwal & Nichol 2022 Hoogeboom et al., (023). Recent studies [Chen. 2023 Hoogeboom et al. 2023) propose carefully designed noise schedules that outperform VP, VE, and
15

---

Published in Transactions on Machine Learning Research (MM/ (MM/YYYY)
New prompt or initial frame
Optiaal
Noise
Danoiser
Hlw
warping
Figure 3: An example for optical flow usage in video scenario, Go-with-the-Plow (Burgert et al. 2025 novel framework that predicts 3D scene dynamics across sequential frames, outperforming conventional single- frame approaches. By integrating multi-frame spatial-temporal relationships, it improves depth accuracy and visual fidclity in scene reconstruction.
Cosine schedules in RGB space. Besides, (Guo et al. 2023a analyzed the power spectrum, introduced a numorical quantification of noise levels, and proposed the weighted signal-to-noise ratio (WSNR) as a unified metric for noise levels both RGB and latent spaces. WSNR-equivalent training noise schedules significantly enhance high-resolution model performance across both latent and RGB spaces.
2.4.4 Agent-based diffusion models
Agent-based approaches have revolutionized diffusion models for multimedia generation, where agents are im- plemented as specialized neural modules with distinct functionality and decisionn-making capabilities. Drive- GAN (Kim et al. 2021) pioneered this direction by utilizing multi-agent frameworks for video generation through differentiable simulation. UniSim (Yang al.) 2024g) contributed by developing neural closed- loop sensor simulator using coordinated agents for scene reconstruction and sensor data generalion. MORA Yuaa et al. 2024ch established comprehensive framework with five specialized agents for different genera- tion stages, introducing self-modulated fine-tuning for dynamic agent contribution adjustment. VideoAgent Soni et al. 2024b) advanced the framework by integrating multimodal LLM feedback and real-world exe- cution for iterative refinement. AutoGen (Wu et al. 2024d) and MetaGPT (Hong et al. 2024a) extended these principles to broader generative tasks, implementing standardized communication protocols to enable scalable multi-agent collaboration. This progression shows how agent-based diffusion models have evolved from simple architectures sophisticated multi-agent systems capable of handling complex generation tasks while maintaining consistency through structured collaboration and feedback mechanisms.
2.5 Architectures
As shown in Figure 4 most text-conditional visual generation models comprise three key modules: a vari- ational autoencoder (VAE) compresses images and videos into latent space. Second, neural network, typically incorporating a U-Net or Transformer-bases backbone, performs the denoising within this latent space. The denoising process can be enhanced using optical flow human feedback to improve generation quality. A text encoder generates embeddings to guide the image or video generation process. These are the main architectures in video generation tasks as shown in Figure
2.5.1 UNet
The UNeu (Ronneeeberrr et al. 2015) architecture has become foundational in designing denoising models for visual diffusion applications, originally developed for medical image segmentation and now adapted widely for generative tasks across images, video, and audio. In image generation tasks, a UNet typically encodes an input into progressively lower-resolution latent representations while increasing feature channels through a series of encoding layers. This encoded latent is then upsampled back to the original resolution by corresponding decoding layers.
In diffusion models, UNet can operate in either the pixel space or the latent space. For example, Latent Diffusion Models (L.Mss) (Rombach et al. 2022 utilize UNet within lower-dimensional latent space, which
16

---

Published in Transactions on Machine Learning Research (MM/YYY)
Modele
Backbone
VAR
Tect Encoder
Pamma
Traising dodaect and slze
Resolutloa
Duration
Framnex
GIU slze
GPU thme
Acsdamis upesoourc motilee
Valaar
VQUiN3
UCF 80L BAIR
61:01
20 VAE
GPT-3
16.28
190 -20
RTX 9000
MageVil
UNe
WeB9e IIM HD.IG 131M
1251 1401
A303
M6in3.15
V.
70 VAE
6.78
WAb 199 HDAVO 100M
750 x -208
73
1900 Ral
USe
Jn VAR
140
RDF Teac
1128
1231
VV833
daps
VAR
CLI VITT
1.30
WAIVE VMM
155 -15
A383
MoghiT
UN4
70 VAE
UCF M01
130 - 1150
3.80
15
VV03
Mighi
JD VAR
300M
BEF
14)-300
SkyTTradaape
UNet
WAIVE: 13M
156 156
SytraeVile 010
USe
CHP
WARSEE 13M
A3034
VQGAN
230 224
A.00
50 US6
20 YAB
CLI
UCF MSRTTT
259 -24
Al33
VakaGai
UNs
CLIP
299 x -29
Asisnio-
290 2.41
UNa
YAE
Assrer DE24
UNS
CLI
1924
UNS
T--XL
JAB
290 -241
A383
-112
DIT
TWXXG
MISTTT
UN6
2D VAE
CUP
UCE 20L MSR.
A6300
S/T Me NINLII
DIT
20 VAE
CLI VIT.L
LARON-US
612 312
Web IIM, PMM
Open Siee Txvg it d. (ott
STDT
3D VAE
TUNNG
3.1
HD. NG. 13154 MinDista Vige. T.eeeKK
1300 730
2.10k
19
Sx 11333
2.51 km
Dd7 UNB
WF VVE
XXL
Pridss DUM
156 750
NPTV
20 VAL
CLP
122 312
2-4r
1 V202
1324-
3-13
02 31:3
Indaetry commeecial modils
RR31
2290 (2)
1345 1980
2000
Victulia
UNa
TI-XXL
TO XXL
2,0m
UCB 901, MSSIVTT
213 N. 886
3:5
A1303
30 VAE
XXL
0.50
UNe
20 VAE
CHP
904
VIGAN
CHP
MAITE 13M4
330 515
L12
ONE
3:30
WAYE: 13M
312
200
730 691
A3854
1231 1480
3-7x
2001
1231- 1480
6L241 1000
100Ck
30 VAL
3331 1860
Zeata
3231 1480
7.31
1B544
2100
10 2rma Alo-Bled
2391 1 752
721
NL
TO0M valco
UNe
175 NT.
3011 video
270
OiteOrtter ot
U.T
Vil VATI
630M Whisor 10A2
1354 570
Vile Coteeeeee C100740 part
Vilw V/7
Welld 1354
519 X 335
1933 1080
10
30 VAE
1001
1130%
Kaggit
1391-
6.711
522 512
LIFT
3B3 VAL
Ahatipu 18M
-310
1331% 1400
5-36
LIT
VAL
XILLM
1291-
Scrleee-11
VAE
MILMM
540 901
Vidw VAL
CLI LAMI
Table 1: Comparison of modules and parameters in different diffusion generative models and their industry applications. New models can be found in vvdeo generation arena leaderboard and ybench leaderboard (TAE refers to temporal autoencoder, and MLLM refers to multimodal large language model). Papers about related component and dataset: VideoVAI F.EEE(Zhe (Zhou et al., 2022), m mT5XXXL (uee 2020), WebVid et al.,
2021h,
UCF-101 UCF-101(Soomro et al. 2012). MSR- AXTTV et al.) 2016 MiraData (Ju et al. 2024]. Laion-coco
600M
LATON 2023) Taichi-HD (Siarohin et al. 2019), BkyTimelapso(Xnnn
et al. 2018), (Chhe- (Sterios & Poppe 2023). Vineo25M (Hunn al.
et al.
2024d), FaceForensics Rössler et al. 20188) Inter4K (Stergiou
2023b
MiraData (Zho et al.; 2023d), Vript. Vripl.YYng et al. 2024a).
allows for more efficient, high-resolution generation. In pixel space, UNet implementations are used for direct denoising, maintaining the input's 's spatial resolution throughout the process.
Modern implementations of UNet in diffusson models build upon the original by replacing ResNet blocks with Vision Transformer (ViT) bayors(Dsoovitkiiy et al. 2020). ResNet blocks apply 2D-Convoltionss, focusing on extracting spatial features, while the ViT blocks introduce spatial self-attention and cross-attention mechanisms. This configuration allows the model to condition generation based on text prompts or specific
17

---

Published in Transactions on Machine Learning Research (MM/YYY)
Downsampling
Pixel space
Lower-dineensional Intent space
Chaesiter guidance Chastier-frre guidance
Condition
Diftusion
EDM
Triss taper video
20 VAE
30 VAE
Denoising
ancoder
Ingust
xtr- !
K(T-
Z0
Tokenizer
feedback
Optical flow
Upllick
Traatheemr
Reward model (e.g. VIDEORM)
(eg. Flownet, Raft, etc.)
TEKLIPGPPT
UNI/U-VIT
DET
Text encoder
Figure A pipeline for diffusion-based visual content generation leverages a pre-trained variational au- toencoder (VAE), such 2D VAE, 3D VAF, VQGAN, TAF, to encode input images videos into lower-dimensional latent representation. Within this latent space, diffusion models (e.g., DDPM, DDIM, EDM) iteratively introduce noise and employ neural architectures, such as U-Net or Transformer-bass models, to learn a denoising process that reconstructs high-fidelity outputs. User-provided textual prompts undergo refinement through large language models (e.g., T5, CLIP, GPT) before being mapped into an embedding space via a trained text encoder. This embedding space serves as a conditioning mechanism, guiding the diffusion process ensure semantic coherence with the input prompt. Furthermore, the frame- work integrates optical flow estimation methods (e.g., FlowNet, Raft) to enhance motion consistency in generated video sequences and incorporates human feedback mechanisms (e.g., VIDEORM) to iteratively improve generation quality.
time steps. Matching resolution layers in the encoder and decoder are connected by residual connections, ensuring efficient information flow.
UNet has been further adapted handle temporal information for video-based diffusion tasks. Extensions like AnimateDiff incorporate temporal attention layers within the UNet to capture dependencies across frames. (An al.) 2023) keep using 2D UNet and enable motion learning by shifting the feature channels along the temporal dimension. Other modifications include transitioning from classic 2D-UNet to spatial- temporal factorized 3D UNet architecture. VDM (Ho et al. 2022c) propose a spatial-temporal factorized 3D UNet by adding temporal blocks for video generation, as a natural extension of the standard image diffusion model. Similarly, (Ho al. 2022a) builds on the similar video U-Net architecture to the cascaded image diffusion model Imagen SSharia et al. (2022), while (He et al., (202))
et al.) 2022 and
ModelScope Wang al.; 2023c Japlyy it. to the latent space (Rombach al.] 2022), as shown in Table et al.) 2025aa further generates the depth and optical flow of the whole video by utilizing 3D UNet architecture by adding extra-temporal modules, including temporal convolution and temporal attention layers after spatial convolution and spacial attention layers, improving frame consistency and visual quality for video generation.
To address the challenge maintaining temporal consistency across vidco frames, modern vidco diffusion models incorporate temporal attention mechanisms within their UNet architectures. Temporal attention enables the model to capture dependencies between frames by computing attention weights across the tem-
18

---

Published in Transactions on Machine Learning Research (MM/YYYY)
poral dimension. This mechanism is particularly crucial for video generation tasks where maintaining object consistency and smooth motion transitions is essential. The temporal attention operation can be formulated as:
QQKT
Attention(Q, K, V) = softmax
(13)
Vdz.
where Q, K, and V represent query, key, and value tensors derived from different temporal positions, and dy is the dimension of the key vectors. This formulation allows the model to selectively attend to relevant temporal information, euhancing the quality of generated videos by ensuring coherent motion and consistent object appearances across frames.
One notable implementation is Align Your Latents (Blatmman et al. 2023b which introduces sophisti- cated temporal atlention mechanism that operates in the latent space, significantly improving the temporal coherence of generated videos while maintaining high spatial resolution. As illustrated in Figure 5) this ap- proach demonstrates how temporal attention can be effectively integrated into cxisting diffusion architectures to enhance video generation capabilitics.
CS
spatial layer
z E BBT
Learned Downsammling
Resize
G2 m (m
0
spatial layer
(b t) h
the
Conv3D layer
Z
temporal layer
b
(b
0
spatial layer
2 De
spatia Iove
1++1
temporal layer
0
spatial layer
Ib tl @
temporal attention layer L)
Ib
XB
B- spatial west
Figure 5: Classic temporal attention design from (Blattmann et al. 2023b showing how temporal attention mechanisms improve video generation quality by maintaining temporal coherence across frames.
2.5.2 Diffusion transformers
The Diffusion Transformer (DiT) replaces UNet's traditional convolutional design with Vision Transformer (ViT) framework, offering improved spatial attention capabilities suited for generative tasks across images and videos (Peebles & Xie. 2023). DiT achieves this through 2D patchification process, where individ- ual image frames are divided into patches that are then treated as tokens, allowing the model to capture detailed spatial dependencies within each frame. GenTroneeChen et al. 2024ch adapted DiT from class to text conditioning and scaled GenTrone from approximately 900M to over 3B parameters demonstrating Transformer- diffusion models can be used in the visual generative domain. From Tabll 1 models including Caus Vidg Yin al.) 2024)] SiT(Ma et al.] 2024a),, and [Sora] also use DiT in video generation settings. However, for video generation, 2D tokenization alone is insufficien to capture the sequential de- pendencies critical for temporal coherence across frames. Approaches such ViViT(Arnal et al. 2021 and (rDrrTZhhnng et al. 2024c) address this limitation by extending DiT's patchification from 2D to 3D, introducing a 3D lokenization approach that sinultaneously captures both spatial and temporal relation- ships. One representative work is CogVideoX (Hong et al., (2023a), which uses STDIT to model spatial- temporal evolution in video generation, capturing long-range dependencies for improved motion consistency and semantic fidelity. In this structure, consocutive frames are processed as spatiotemporal patches, en- abling the model to understand dynamic changes across frames. Building on these approaches, frameworks
19

---

Published in Transactions on Machine Learning Research (MM/YYYY)
like VDT et al. 2024a] incorporate both causal attention, which restricts attention to previous frames, and sparse causal attention, which further limits the focus to subset of recent frames. VersVideo Xiang et al. 2023) incorporated multi-excitation paths for spatial-temporal convolutions with dimension pooling across different axes and multi-expert spatial-temporal attention blocks to further boost the model's spatial- temporal performance instead of simply extending 2D operations with temporal operations to significantly escalate training and inference costs. Text2Performer (Jiang et al. p023b/psoposes ae continuous VQ-diffuser to directly output the continuous pose embeddings for better motion modeling. These mechanisms enhance model's effectiveness for video-based diffusion tasks.
The transition from 2D to 3D patchification represents a fundamental architectural innovation for video generation. ViVIT (Arnab et al. 2021) introduces factorized encoder architecture that separates spatial and temporal modeling. As shown in Figure [ this model first extracts patch tokens from each frame using spatiotemporal tokenization. A spatial transformer encoder then processes the tokens within each frame independently to model local structure and appearance.
The model passes the outputs of each frame to temporal transformer that captures how frames relate over time. This two-step design learns both spatial and temporal patterns clearly and separately. For video generation, this setup helps control motion and appearance more preciscly and keeps the frames consistent over time. Separating spatial and temporal steps also makes the model easier to train and understand.
Temporal Transformer Fnooder
MLP
Class
Temporal Taken Embedaing
Head
...
Spatial Transformer Encoder
Spstial Transformer Encoder
Spatial Transforme- Encoder
Posilcetl Token Bmbediing
Emmbed to lokens
Figure 6: Factorized encoder (Model 2) in VIVIT (Arnab et al. 2021 ). The first transformer encoder models spatial interactions within each frame. The second transformer encoder captures temporal dependencies between frames. This factorized structure enables a late fusion of spatial and temporal information, which supports temporally consistent video generation.
2.5.3 VAE for latent space compression
Diffusion and denoising in RGB pixel space, as demonstrated in (Ho et al., 2020 Dhariwal & Nichol 2021
Saharia al. 2022 Ramesh et al. 2022 Ho al. 2022b). require high computational resources, significang increasing both training cost and inference latency. To mitigate its resource consumption, Latent Diffusion Models (LDM) Rombachh et al.) 2022 leverage variational autoencoders (VAEs) to compress images from pixel space into a more compact latent space enabling the diffusion process to occur in this optimized space. This approach enhances both training and inference efficiency by reducing the complexity of the data representation.
Classical VAEs for image and video compression include standard VAEs (ingmaa 2013) quantized versions like VQ al. 2021), VQGAN((Vn Den Oord et al. 2017 and their GAN-enhancod variants, which improve reconstruction quality for higher compression. For example, VideoGPT ( et al. 2021) use 3D-VQVAE in the latent space for video generation. Seine et al. 2024e) and MAGVIT Yu et al.
20

---

Published in Transactions on Machine Learning Research (MM/YYYY)
2023al) integrate 3D-VQGAN with discriminators for latent space encoding to achieve better visual quality. The VAE framework not only compresses effectively but also enable the training of multiple generative tasks across downstream tasks.
By integrating VAE in model building, some image generation models et al. 2022a Liet ct al.
2024b Podell
et al. 2023 Bao et al. 2023b Peebles & Xie 2023: Lu et al., 2024c Ma et al. 20244| Chen et al. 2023c
Li et al. 20241| Team 2024b; Esser al.
2024) also leverage latent space encoding/ decoding, freezing VAB
parameters during diffusion training and inference. Some diffusion models, including Mako-A-Video(Singg
et al. 2022 Imagen (Ho et al.
2022a Show-1(Zhang et al., 2023a), directly learn pixel distributions gener- ating videos, as shown in Table 1 However, video generation involves both spatial and temporal complexity, leading to higher computational costs. Besides, some represenlative diffusion video generation models, including Sora(Liu et al. 2024e). Phenaki[[Vllegaal et al.) 2022), VideoCrafter(Chnn et al., 2023a), Animate- et al. 2023). start using VAE compression in video settings before training and inference in latent space. Many video generation models, including Latte[Ma et al. 2024ch, Lavie Lavie(Huang al.) 2023b), MagieVideo(Zhoo et al. 2022). Align-your-latent (Blattmann al., 2023b), and AnimateDiffiGuuo et al. (023c), are devrived from Stable Diffusion's 's image 2D VAEs, training full 3D latent space from scratch is challenging. However, temporal compression in 2D VAE-based vidco models relies on uniform frame sampling, which ignores motion information, leading to low FPS and less smooth video generation.
Guo et al.] 2023cp, and VideoPoct((Koodratyy
Some approaches also start to utilize hybrid 2D-3D or fully 3D VAEs. For example, MagViT (Yu et al. 2023a), EasyAnimate (Xu et al., 2024c], Open-Sora Sora (Zheng et al. 2024c Lab & etc. 2024), and Cog Video X (Yang et al.] 2024jj employ hybrid 2D-3D VAFs, leveraging volumetric encoding for video generation. MAGVIT et al. 2023a) adopting 3D VQGAN structures integrating both 3D and 2D downsampling, 2024b) utilizing a fully 3D VAE with 3D convolutional encoder and overlapping downsampling for enhanced spatial-tempont fidelity. The MAGVIT architecture improves video compression by combining spatial and temporal information in one model. This helps reduce data size while keeping good video quality.
and MAGVIT- (2 Yu et al.
HunyuanVideo (Kong et al., 2025 Kling Mochi 1 TTea 2024a), and SkyReels- V (onelssvV(SSyyyaaa (SkyRReelss Al 2025) continue leveraging 3D VAEs for compressed training in latent space in industry scenarios. Step- Video- T2V el al. 2025) and 1ar el al. 2025) introduce novel Video VAE structures for efficient generation. However, training video VAE independently without ensuring compatibility often leads to a latent space gap, preventing accurate projection into pixel space (Zhao et al., 2024b). CV-VAE (Zhao et al., 2024b) introduces a novel approach using latent space regularization to train a video VAE that extracts continuous latent for generative video models while maintaining compatibility with existing pretrained image and video models. Also, to trade off lower memory and computational cost with slightly lower reconstruction quality, the latest video generation model, Moive Gen(teamMett 2024) further uses interleaved 2D-1D convolutional encoders in its VAE. This approach balances efficiency and quality by reducing the redundancy of temporal information while maintaining spatial coherence.
As video generation increasingly demands temporal consistency and causal modeling, recent approaches have introduced causal 3D VAE structures that process frames in a strictly sequential, past-dependent manner. As shown in Figure 7. CogVidcoX (Yang et al.) (0244) uses a 3D VAE with temporal causality to compress spatial and temporal information, ensuring that each output frame is based only on current. and earlier frames.
In the encoder-decoder pipeline, the model applics downsampling across both space and time, then recon- structs frames with mirrored upsampling layers. The VAE operates in a causal, frame- frame-by-frand fashion: it never uses future frames when encoding or decoding the present one, which helps ensure temporal coherence in longer video outputs.
2.5.4 Text encoders
In text-conditional visual generation models, the text encoder plays a crucial role in capturing semantic information from input text prompts, directly affecting the generated content. Early text-to-image models used text. encoders trained on paired text-image datasets, either trained from scratch Nichol et al. 2021
21

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Rank O: First frame padding
Video: F x H W
(4f+1) 1) 8h 8w
Enc Stage
Dec Stage
2x Downsample
2x Upsample
Spatial & Temporal
conv output
Enc Stuge
Dec Stage
Send
Rank
2x Downsample
2x Upsample
Enc Stage
Stage
Reev Rank (i-1)
Soad Rank
2x Downsample
Spatial only
2x Upsumple
Rank (--)): Rank (i-1)'s tail padding
Enc Stage
KL Regularizer
Stage
Encoder
Decoder
Latent:
time kemel-1
time kemel-1
xf+1)xhxw
(paddings)
(tails)
(a)
(b)
Figure 7: 3D causal VAE structure in CogVideoX Yang et al. 2024j). (a) The encoder compresses spatial and temporal features, followed by spatial-only layers and a KL regularizer to generate latent representations. The decoder mirrors this structure in reverse. (b) Temporally causal 3D convolutions are applied so each output frame depends only on the current and earlier frames, maintaining motion consistency.
Ramesh et al. 2021 or fine-tuned from pre-trained models such as CLIP et al. 2021 CLIP, which employs contrastive learning to align text and image embedding spaces, enables the text encoder to effectively represent both visual and linguistic semantics after being trained on large-scale multimodal datascts. Once an input text prompt tokenized and embedded, serves as a conditioning mechanism for the diffusion model's generative backbone. CLIP-based text encoders are widely used in text-to-image diffusion models, as shown in Table including DALLE-2 Ramesh et al. 2022), Stable Diffusion InffusioonRombade (Rombach
et al.] 2022), [Bo et al. 2023a Peebles & Xie 2023), Fit(Lu et al. 2024c), SiT et al. 2024a),
HunyuanDiT Li et al. 20241 and Scaling Diffusion EEser et al. 2024 Labs 2024 In these models, the
text encoder's parameters are often frozen to reduce computational and memory overhead during training, ensuring that most training resources are allocated to the diffusion process itself.
However, CLIP often struggle with understanding detailed text descriptions. To address this, large language models (LLMs) trained on extensive text corpora provide stronger text comprehension and generation capa- bilitics. Imagen (o et al. 2022al) compared CLIP with pre-trained LLMs such as BERT ( et al. 2019) and T5 et al. 2020 as text encoders. Their findings showed that scaling the sixe of the text encoder improves the quality of text-to-image generation. Notably, (Ho et al. 2022a) found that the T5-XXL encoder significantly enhances image-text fidelity, leading to its adoption in several models such as Latte et al. et al. 2024c), and SimDA(Xing et al.) 2024b). Some approaches further combine CLIP and T5 to enhance text comprehension. Step-Videon Video-TVVV (a et al., 2025) integrates CLIP, Step-LLM, and Video-VAE to refine alignment between textual descriptions and generated video content. To improve text encoding, ByT5[Xue et al. 2022b introduced byte-level tokenization-fre frree approach, effectively han- dling diverse text inputs. This has been leveraged in Movie-Gen, where By combined with MetaCLIP and UL2 jointly enhances text understanding for improved video generation. Multilingual contexts also benefit from LLM-based encoders. Open-Sora-Plan (Lab & elc. 2024) incorporates multilingual T5 (mT55XXL) alongside WF-VAN VFE, optimizing text representations for video synthesis across diverse linguistic structures. Recent image (Tan al. 2024c; Team 2024b) and vidcofYang et al.) 2024j Yi et al., 2025), generation mod- els employ large language models such BaichuanYYang al. (222a)). Llama (Tovvron al. 2023a b). and et al. 2021 to enhance the semantic understanding of complex text. SkyReels- V1 Sky Reels-
2024ch, Apen-SooraZhhee
22

---

Published in Transactions on Machine Learning Research (MM/YYY)
A1 2025) and HUnyuanVideo (Kong et al. 2025 further integrate pre-trained Multimodal Large Language Model (MLLM) as their text encoders to enhance visual-text. alignment and improve instruction adherence in diffusion models.
3 Implementations
In this part, we present a detailed analysis of the datasets, training engineering techniques, and evaluations of video gencration.
3.1 Datasets
In this part, we review the most popular datasets used for video generation, as summarized in table Specifically, we split the datasets into academic and commercial datasets.
Academic benchmark datasets. As long- standing problem, there exist numerous traditional text- to-video benchmark datasets, such as MSR-VTT (Xu et al. 2016), DiDeMo (Hendricks et al.] [2017]. LSMDC [Rohrbach et al. 2015), and VATEX [Wang et al. 2019), which consist of videos and corresponding captions collected from the real world or annotators. However, those datasets are always small, low quality, and cover only a small number of different areas, such as sports, cooking, and movies.
Building upon those traditional benchmark datasets, researchers have recently introduced datasets (Sanabria al. 2018 Zellers et al. 2021 Lee al. 2021 Chen al. 2024d Ju et al., 2024) with large number of
videos. For example, YT Temporal-180N Zellers et al. 2021 contains 180M videos collected from YouTube. InternVid (Wang et al. 2023h takes over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of a total of 4.1B words. Also, researchers noticed that video quality, such as resolution, is also important. Inter4k-14 1k (Stergiou & Poppe 2023 as a pioneer, introduces videos with a resolution of 4k, enabling the understanding and generation high-resolution videos.
On the other side, as manually collecting datasets is time-consuming and expensive, researchers started generate datasets using off-the-shelf models. VidProM (Wang & Yang 2024 contains 1.67 million unique text-to-video prompts generated by Pika VideoCraft2 (Chen et al. 2024b), and Texi2Video-Zero (Khacha- tryan et al. 2023
On the other side, other rescarchers focus on providing comprehensive vidco generation benchmark datasets (Huang et al. 2024b Wang & Yang 2024 from different perspectives, e.g., temporal consistency, style consistency, semantic consistency, and video quality. VBench takes 100 prompts to evaluate video generation models from different perspectives. FETV (Liu al., 2023e) reuses the texts from the MSR- VTT test set and WebVid for open-domain text-video pairs, guaranteeing the diversity of prompts. It also contains manually written prompts describing scenarios that are unusual in the real world, and cannot be found in existing text-video datasets.
Except for the benchmark dataset for video generation, there are also some datasets targeting face video generation, video inpainting, and video restoration. For example, VFHQ (Xie et al., 2022) is proposed for the face super-resolution, which contains over 16, 000 high-fidelity clips of diverse interview scenarios.
Commercial benchmark datasets. There are also some companies providing high-quality image/vides text paired data such as Pond5. Adobe Stock Shutterstockk, Getty [ooverr) Videvo, [Depssttphooos Sto-
ryblocks Dissolve. Freepik Vimco and Envato which are widely used in training vidco generation meth-
ods ZZheng el al. 2024c).
3.2 Training engineering
A comprehensive pipeline for data curation, preprocessing, and training nceded train large foundation model for video generation. In this part, we present an overview of data preprocessing, training techniques, and acceleration methods.
23

---

Published in Transactions on Machine Learning Research
Dutiests
Medaiities
Vialon
Ttect
Duralion
Renalutien
Domalns
sampllo
Source
Liceus
Acaoemma dataadi
DCF-10
Ferl
AVGL
13kc30
Yorlube
OC-aV
Kastigg
211T
V.A
AVGL 30
V (CR6L247)
Yorlube
VicCieet
Srallatic
10 Baakend
V [Ga1k)
Eahas
Srailatic
10 Staskned
5 [2401 k1
Seatlatic
1D Baxakas
Flick
Pacl
V (1..0
388-972
7anl
Dcal
1/9 (1B 10
(27 k1
Flickz
2-Chone
L5KD0
VATEX
Baal
1/5 (41. 50
Yourddo
OC-BV-40
Roal
AVG. 1100
T/5 (14.3)
92-
Yasisder
Orentve Comman BV-84
Vi0eCCSM
Vearet
T.A
Boal
AVG.
T/V (10.5 93
V.T
Real
AVG.
baty
T/5 (10 34
AGPL-8
10
(29
Yandee
Tognne Se Ioces
ED SD-VVLL--IIC.
V.T
Aato Capticning
AVG. 13
730
T/V (100 30)
2011
V.T.A
Real
T/5 (199 3-1
Yando
V.T.A
Lieese
Tool
HVG
7M0
(E39
V.lDoll
V.T
Systleti
Roal
AVCL 2
T T1062 50. V (0.45 M)
TV 300 4.6
FFTV
V.T
Bel
Real
T. 1019) (5-1)
MSL.VTT and WANV
DC BY ad
Captiraing
1.
Tyul
Taal
1/T T//V (1.00)
Tiveno
AVE Augn
V.1
Tyol
Real
AVE 16
V. (1066)
V.-7
Bed
Arpoo Capticaing
A003 6
230
1/5 (2) M0
Trul
Eoptiraing
AV0L 75
2M0
Carana Teflos (C0001)
EAPIX 3EN DOT
1.7
Systheth
Real
1/2 (005 k)
1.7
Syathedi
Reaa
1/1 (IC)
Aquche Eioeso
PutiPwoupe
Real
()
Apuciee Llneno
Commeniial Motasets
Niteeran da 1.2M
1.7
Systhati
(1.7 M)
Apocle 1ioemo
1.2
bysthett
T/1 0250 k)
oot 1.1
1.2
Syathedo
Ieaal
10
See Tream
L.V.V V
[tral
AVLL
230
1/V (11.333) T/V (99147))
Overandaa concoerral
V.7
Beal
737
Table 2: The overview of most popular datasets used in training video generation models. We also include image datasets they are usually used in training. "I", "V", "T", and "A" represent image, video, text. and audio. Other commercial datasets include those released by Pond5 Adobe Stock, (Shutterstcck, Getty
Coverr Videvo Depositphotos
Storyblocks
Dissolve Freepik
Vimeo. and Envato
Open-Sora Data Processing Pipeline v1.1
soene out
aesthelic soore
optical How Score
OCR
Caption
OCR Resuing
Thhe cenene beauty of
tce.
optical Fow
waterial cascadio imo scol.ck sooluded lush groen pool.
Top much laxts
X
Nigh eptical foo score
065 6.6
Match Stooe
oeW parquet
(pdl-torvidcot.ng
path
test
num, frames height
128
256
Cleaning Video Mota Info Labeling Additional Proceso
right pan left
Camera Motion
score thresholdd
/pothhh//udess mpp4
Conditioned Fllteiing
zoom in, z00m ouT
Ihreshold
toth-to/mmaggl png gin
206
Append to Capptoon
(pothto/mmmgge ping
deagon
750
The oo mera ls zooming In.
Training Analyze
Figure 8: The data preprocessing pipeline of Open-Sora Zheng et al. 2024c
Data preprocessing. Instead of directly using the original data from the dataset, more and more meth- ods [Zheng et al.) 2024c| Li et al. 2024 Yang et al. 2024j Blattmann et al., 2023a) have started to select, filter, and enhance vidco-text paired data before training the model as shown in fig. g OpenSora (Zheng et al.) 2024c), as a representative open-sourced method, first splits videos into shorter clips using scene detection (Ravi et al.) 2025). Later, clips are evaluated using the aesthetic and optical flow scores, and the clips with low scores are removed from training. After that, the qualified clips are captioned using machine captioners (Hong et al., 2024b] Li et al.] [2025a] Xu et al.) (2024)) and a matching score will be employed to evaluate the alignment between clips and their captions. Finally, samples with adequate scores (high
24

---

Published in Transactions on Machine Learning Research (MM/ (YYY
[1] SFT Truiniu
(RSSP) Traning
Vinal Filtr
Triiing
(1 (1:221) Torning
Image Data
4000) Reviltionn Filtrr
1200)
Thinnnn
File:
II (92P) Deduplication
IV 720P] Visul Piher
[V] Matiin Fileer
IV 480e1 Trainin
7301) Resolation Filker
(1 (92)) Filker
[T SFT) Tnining
II 392P) Fundunerral Filte
IV 4908) Visual Filter
Misiion Filke
7200) Resolation Hiler
Reselezim
LV
[V 152P3 Traccing
4000
LT 4300) Roclution Filer
Video Data
[V 182] Dedupticaion
(V 1922) Viuea Fihor
[V 152P] Pandanerral Filte
(V 1822) Mosion Filter
Todual Dat.
1927 Traning
Figure The data preprocessing pipeline of WAN (Wa et al. 2025). WAN first splits data into image, video, and text data. Then, different stages of filters applied to the data for further proprocessing, Fianlly, the data is splited into pretraining data and SFT data for imrpoving the quality of generated video data.
aesthetic quality, large video motion, and strong semantic consistency) are used for training the model. The overall pipeline is presented in fig. 8 Similarly, SVD (Blattmann et al.) (023a) filters out samples with low text-video alignment evaluated by CoCa Yu al. 2022a static videos (evaluated by optical flow), and high text overlays.
A recent framework, WAN Wan et al. 2025), as shown in fig. employs novel data pipeline which includes four fundamental dimensions for data collection and selection. For pretraining data, it evaluates data from fundamental dimensions (text detection, blur detection, synthetic data detection, and etc.), visual quality (including data clustering and scoring), motion quality (which splits data with optimal motion, medium- quality motion, camera-driven motion, low-quality motion, and shaky camera footage), and visual text data. For post-training data preprocessing, WAN improves the quality of image and video data separately, Similarly, to meet the needs of different stages of pre-training, (teamMeta (2024) curates 3 subsets of pre- training data with progressively stricter visual, motion, and content thresholds, which leads to more stable training process. While MovieGen and WAN apply post-training to improve the motion and aesthetic quality of the generated vidcos by finetuning the pre-trained model on a small finctuning set selected videos, Cosmos WFM (NVIDIA et al., 20255 fine-tuned in post. training to support diverse Physical AT applications, such as autonomous driving, with dedicated post-training datascts.
Training techniques. Video generation models (Blattmann et al. 2023a Zheng al. 2024ch are usually initialized with image models and pretrained with image data. Then, the models pretrained with video data for lcarning efficient video representations. After that, models are trained using high-quality data to form the final video generation models. However, due to the difficulty of the video generation problem, different training techniques (Blattmann et al. 2023a Xing et al. 2023 Hong et al. 2023a Yang et al, 2024j) are proposed. For example, CogVideo: (Yang et al. 2024jj employs a multi-resolution frame pack strategy to enable the generation of videos with varied lengths. And a progressive training strategy employed generate videos of different resolutions. This progressive training strategy is also employed by other works,
25

---

Published in Transactions on Machine Learning Research
Popk suNd
Darsan allodd Text to skleo
oottdd
Prondee
Tox: image Tech-nauldde
Text- to -i---t--dddee Fieeluiing
bakest
235 pa image training
250 785 videea
Custor
warl
vidleca
(a) Training pipeline of Cosmos NVIDIA et al. 2025
(b) Training pipeline of Movie Gen
teamMeta 2024
Figure 10: Training pipelines of Cosmos NVIDIA et al. 2025 and Movie Gen (teamMeta, 2024]. Cosmos first pretrains models and then finetunes on different downstream tasks for better performance while Movie Gen finetunes to achieve better quality of generated videos.
such as Hunyuan-Video Kong et al. 2025 the other side, to scale up the training video generation models, multi-node parallelization and VRAM-efficient scheduling are utilized (Yang al.
2024j Zheng
et al. 2024ch. Morcover, post training has shined the lights on vision-language models Cheng et al.
2025 NVIDIA et al. 2025 Yamaguchi et al. 2025), more and more studies (Wan et al. 2025 teamMeta
2024 include post-training to further improve the quality of generated video, as shown in fig. 10 A practical
scenario for post-training is to finetune the pretrained to suit the downstream tasks (NVIDIA et al. 2025 such as autonomous driving and robot manipulation.
Video generation acceleration techniques. Due to the high computational requirements of training and inferencing video generation methods, researchers have developed several acceleration techniques. For example, Open-Sora (Zheng et al.) 2024c) employs flash attention (Dao al. 2022), ZeRO (Rajbhandari et al. 2020h, and gradient checkpoint for kernel optimization, hybrid parallelism, and larger batch size training. Morcover, Open-Sora employs efficient STDiT as the vision encoder while preprocessing text and video data for acceleration. On the other side, parfaeeter-fffciien fine-tuning (PEFT) (Geng et al. 2024a Jia
let al.
2022), which widely applied in training large foundation (language and vision-language) models (Li
ett
2025a Touvron et al.) (023a b), could be used in vidco generation for training accelcration (Xing
et al.
2024b
Pan et al. 2022). We also include detailed discussion on long video generation acceleration
techniques in section 4.5
26

---

Published in Transactions on Machine Learning Research
3.3 Evaluation metrics and benchmarking findings
Static Quality
Temporal Quality
Consistency
Human Evaluation
Overall Aligument
Fine- Fine-grained Alignment (Semantic Consistency)
Static Quality
Metrics
Video Quality
Temporal Quality
Semantic Consistency
Video- Cideo-Condition Consistency
Temporal (Motion) Consistency
Style Consistency
Existing evaluation metrics include human evaluation and auto evaluation. Human evaluation (Liu et al. 2023c Huang et al., 2024b) always includes several aspects such as static quality, temporal quality (consis- tency), ), overall alignment, and fine-grained alignment (semantic consistency). And human evaluation is al- ways employed to test the alignment between auto-evaluation and human evaluators, such as VBench (Huang ct al. 2024b), which uses 16 human evaluations on 16 different dimensions mainly focusing on temporal and seraattic consistency. A human-verrfied metric is STREAM (Kim et al. 2023c testified under the same dimensions using auto-evaluation and human evaluation.
On the other side, auto-evaluation (Huang et al. 2024b c Fan et al. 2024 is cheaper and faster, it is widely applied to evaluate video quality and conditioned quality (semantic consistency and temporal consistency). Video quality usually only contains video as the input of metrics while conditioned quality also uses other modal inputs such as image, text, or audio to evaluate the generated video.
Video quality consists of static quality and temporal/motion quality. 1) For static quality, AIGCBench Fan et al. 2024) employs a pre-trained optical flow estimation model, i.e., RAFT TTedd & Deng 2020). and caleulates the mean square of the average flow score between adjacent frames from the generated video for estimating the motion effects Plow-Squrre-Man. Dover (Wu et al.) 2023a] is also employed for evaluating the static quality from aesthetic and technical perspectives. Moreover, the number of video frames is used for accessing the quality. On the other side, FETV (Liu et al.) 2023c) employs FID (Heusel et al. 2017) and FVD (Unterthiner et al., 2019) metrics. 2) For temporal/ motion quality, AIGCBench assesses it by calculating the average similarity between adjacent frames using CLIP (Radford et al. 2021) noted as GenVideo Clip (Adjacent frames). VBench (Huang et al.) (2024b) evaluates by subject, background, temporal flickering, motion, dynamic degree, aesthetic quality, and imaging quality using multiple pre- trained models including DINO, CL.PP, Amt, RAFT, and LATON. Bonneel et al. (2015) proposes a warping
27

---

Published in Transactions on Machine Learning Research (MM/YYY)
error for improving the consistency between frames. Except for those methods, STREAM (Kim et al, 2023cp is proposed to avoid the problem of FVD, which has a stronger emphasis on the spatial aspect than the temporal naturalness of video and demonstrates considerable instability and diverges from human evaluations. It evaluates temporal and semantic quality at the same time.
Besides those metrics, more and more researchers start to improving motion consistency and aesthetics of generated videos. 1) For accessing motion smoothness, WAN (Wan et al. 2025) employs Qwen2-V VL (Bai et al. 2023) with crafted prompts. Similarly, Movic Gen (teamMetal 2024 accesses the motion completeness and motion naturalness for better evaluating the quality of generated videos. 2) While the motion quality of video evaluates how smooth the vidco is, the realnoss and acsthetics evaluate the model's ability to generate photorealistic videos with aesthetically pleasing content, lighting, color, style, elc. To access them, Movie Gen recruits a group of human evaluator to measure which of the videos being compared most closely resembles a real video and which of the generated videos has more interesting and compelling content, lighting, color, and camera effects.
The metries for evaluating conditioned quality can be categorized into semantic consistency, temporal con- sistency, and style consistency. 1) In semantic consistency, for image/video-vided metrics, AIGCBench (Fan ct al. 2024] employs Mean Squared Error (MSE First) and Structural Similarity Index Mcasure (SSIM First) for evaluating the first generated frames. To further understand how the semantics are preserved across frames, AIGCBench calculates the cosine similarity between the imput image and each frame of gen- erated video using CLIP
CLIP). Moreover, MenVidee-Re/Viin measure the spatial structural similarity of the generated videos to the reference videos by calculating the SSTM (Structural Similarity Index Measure) between the corresponding frames of the generated and reference videos as the referenced video is at hand. For Text-to-Videe metrics, AIGCBench employs GenVideo-Tee Text Clip and GenVideo-Reffidee CLIP (Keyframes), calculating the similarity between the imput text refer- enced video and keyframes from generated videas using CLIP. Similar to AlGCBench, FETV (Liu et al. 2023cb propose to use stronger models, such as CLIP finctuned on MSR-VTT, BLIP, and UMP, and VQA models for evaluating the semantic consistency. Moreover, FETV also introduces a novel QA-based metric (otter-VQA). This metric generates yes-no questions on key elements from the text prompt via the Vicuna model and calculates the average number of questions that receive a positive answer by feeding the gen- erated (or ground-truth) videos and questions into the Video LLM the score. VBench (Huang et al. 2024b) evaluates the semantic consistency by objects, humans, color, relationships, scene, appearance, and temporal style using ViCLIP, CLIP, Tag2Text, UMT, and GRiT. 2) For temporal consistency, researchers usually directly employ the temporal quality metrics. Recently, TC-bench (Feng et al. 2024c) proposed to focus on the transition of objects, relations, and background for better temporal consistency. 3) For style consistency, VBench employs CLIP and VICLIP to assess the similarity between generated vidcos and the style description.
SSIM is also employed to
Except for those metrics, VBench (fig. 11 and Video Generation Arena also set up online evaluation platform, providing an accossible leaderboard for rescarchers.
3.4 Industry solutions
As shown Backbone column in Table 1 video generation industry has evolved significantly with the adoption of diffusion models that leverage advanced architectures like DiT, UNet, and U-VIT DiT (Diffusion Trans- former) widely used in models like W.A.T.TTGuupl et al. 2023). MovieGen Kling and Mochi Team 2024a), excelling capturing long-range dependencics for sharp, coherent visuals video generation tasks. NNet, utilized in models like Imagen video Lavie[
al. 2023g), /einee al., 2024e| and VLog- ger Zhuang et al. 2024), is known for its ability to handle both local and global features, making it ideal for high-resolution video generation. U-VIT (U-Transformer for video) is another backbone seen in models like VideoCrafter (Chen et al. 2023a) and VideoCrafter2 (Chen et al. 2024b), effective at processing both spatial and temporal dependencies for more accurate video generation. These architectures continue to shape the
industry's ' approach to advanced video generation.
Variational Autoencoders (VAE) are widely used in industry to learn latent video representations. As shown VAE column in Table 1. models such as HunyuanVider (Kong et al.] 2025), Mochi 1 [Tea 2024a). and
28

---

Published in Transactions on Machine Learning Research
Spaces lehtort Wanch Leaderboard
Flae
Communiy
VBench Leaderboard
Netoona th bederee do Wirsh Bl Grspereenee anscheed Sale donddto Medoo 2O11 Spulighd
moiee
Alginen sturd
and wastor dara
wiight
wight
Plane undiu tou pepttiige the
geoeuuie mmadt: Jn law chkingg Bistl limel wike Netees
Hoded informndio thtt the deaato of thee iide Ge Genertii Modea Ses
Grediic This snciedic and aptand and main ao by the men of
ASTRUCTION
Qualing
Reeel (cliotebbe)
City Soore
Setected Score
sefject mentstorcy
backgronnd constatency
seootheess
settwttc qualiyy
Smaging mal
Joca
S
70 32%
125
07. 020
77.055
70. 505
505
03 0.000
9.303 22.
51.705
136
125
100
27
12.428
Comfitnoo
00.328
02.138
50.23
Figure 11: The VBench leader board at Feb. 3rd, 2025. SORA still scores the highest some of the metrics, e.g. quality score and selected score.
Kling utilize 3D VAE for temporal consistency. On the other hand, models like Lavic Wang et al. 2023g) and Phenaki Villegss et al. 2022) utilize 2D VAEs, which offer greater computational efficiency but are less effective at capturing the temporal dynamics of video content. Mcanwhile, VQ-VAE, used in models like GLogger[Zhuaan et al. 2024) and Vchitect-2.0, introduces vector quantization to improve efficiency by discretizing the latent space. Additionally, Video VAE architectures, such as in VideoCrafter CraenerCherreen (Chen et all, 2023a) and I.TX-Video. (HaCohen et al. 2024), are specifically designed for video generation, optimizing the ability to handle both spatial and temporal complexities. The choice of VAE architecture thus directly influences the qualily, elficiency, and temporal coherence of video generation.
Text encoders, such as TX-XXL, are essential for aligning video with textual descriptions. Based on the column text encoder in the Table 1 models like Imagon vidco A.A.T../Guuii. et al., 2023), and MovicGen use T5-XXL for precise video-text alignment, enhancing generative capabilities. Meanwhile, models like Lavie[ et al., 2023g) and [Seine(Chen et al.] (202e)) leverage CLIP-based encoders, which offer flexibility for generating both short clips and full-length videos. T5-XXL excels in largo-scale generation, while CLIP provides versatility for diverse visual styles and content.
The scale of models varies, as shown in the parameter column of Table 1. with larger models like Sora (with ~ 33B parameters) and MovieGen (with ~ 30B parameters) offering scalability for large-scale produc- tion. Smaller models including Vchitect-2.0 and VideoPoet maintain strong performance while using fewer parameters (1B-2B) to balance performance and computational cost.
Training datasets such as WebVid-10M et al. 2021 and (imeo25M(Huan)) et al. 2023b) are commonly used for large-scale models like Seinee(Chen et al. 2024e] Lavie Wang et al. (023g] and Show- I from Table 1. ensuring diverse outputs. Models including Lumiereand VideoPoct choose to gather video and image data directly from public websites, rather than relying on pre-existing datasets.
Resolution plays a key role in video output quality. As shown in the resolution coulmn in the Table 1 GEN-3 Alpha and [Sora] support high resolutions like 4090 x 2160 and 1920 x 1080, ideal for cinematic production. In contrast, models like Seine et al.J 2024e] and Phenaki (Villegas et al. (022) offer lower resolutions like 320 512 or 64 64, balancing quality and computational efficiency.
29

---

Published in Transactions on Machine Learning Research (MM/YYY)
Video duration is also key consideration. Based on the duration column in the Table 1
W.A.L.T (.ptaa
Zhuang et al. 2024 excel in generating short-duration videos for platforms such as social media, while models such as Kling and Mira are capable of creating longer videos, suited for film production or extended content.
et al. 2023 and VLogger VhoggerrZZuuan
GPU requirements for training and inference vary. As shown in the GPU size and GPU hours column in the Table 1] Large models such as [how-1 and MovieGen need substantial GPU resources, with [Show1 requiring around 48 A100 GPUs. Smaller models like LTX- -ixxoo (HaCohen et al. 2024 can achieve faster processing, offering practical real-time video generation for commercial use.
The video generation industry continues to grow with models such as MovieGen and Sora catering to high- end production, while consumer-foeused models like "Logger(Zhuanp et. al. 2024 and EasyAnimate (Xul et al.) 2024c) offer fast solutions for short-form video content. The challenge remains to optimize video quality while managing computational resources for broader accessibility.
4 Applications
4.1
Conditions
In this section, we examine a variely of "conditions" that researchers use to guide the video generation process. Unlike purely unconditional or text-based generation, which can lack specific control or produce unsatisfactory temporal consistency, leveraging additional forms of conditioning provides more precise and robust ways to steer the synthesis. These conditions can stem from many cources-imager spatial con- straints, camera parameters, audio signals, high-level editing instructions, or even physical simulations -annss---ndddd and can serve different roles, from specifying global traits like style and identity to prescribing detailed frame- by-frame motion.
We organize these conditions into subtopics based the type of guidance they offer. First, we discuss image-based methods (Image condition 4.1.1. where single reference image sequence of images conveys both coarse semantic cues and fine-grained details for video generation. We then explore spatial conditions (Spatial condition 4.1.2), such as trajectories, bounding boxes, or scene layouts, which grant users exact control over object movement and scene composition. Next, we address camera parameter inputs (Camera parameter condition 4.13)) for 3D-consistent view syuthesis and cinematic manipulation. We also cover audio- based approaches (Audio condition [41.4) for syuchronizing speech, music, or other sound cues with visual content. Finally, we examine methods that focus on high-level editing (High-level video condition 4.1.5) and specialized application constraints (Other conditions 4.1.6), including incorporating physics-based rcalism, working with physiological signals, and utilizing timestamp information conditions. Through these diverse forms of conditioning, researchers aim to create video generation systems that balance creative freedom with strong user control, opening the door to a wide range of potential applications.
4.1.1 Image condition
Controlling video generation with images is crucial capability, as images offer visually intuitive and fine- grained control over content that text alone cannot achieve. This functionality serves as a foundational building block for various video applications, such as identity-pressrving video generation and image anima- tion.
Depending on how the conditioning image integrated into the diffusion model, the methods can be broadly calegorized into two types: semantic-level image conditioning and fine-grained image conditioning.
Semantic-level conditioning. Similar to text-conditioned video diffusion, semantic-level image condition- ing provides high-level coarse guidance for vidco generation. This is typically achieved by encoding the input image into a low-dimensional feature that retains rich semantic information, such as identily. These features are then fed into the video model through cross-attes attention layers in a manner similar to how text embeddings are incorporated into the network. For instance, DreamVideo (Wei et al. (024c) employs an identity adapter to encode the identity of the conditioning image, while ID-Animttor (He al. (024d) uses face adapter to capture the facial identity of human face. Additionally, CLIP features commonly utilized to extract
30

---

Published in Transactions on Machine Learning Research (M/YYYY)
CLIP encoder
Image
Imagee takenizer
CLIP feature
Cross
Duffision U.Net
Cross attentions
Diffusion U-Net
Difiusion U-Net CLIP feature conditioning
DIT
Control- Net condition
Input concatemation
Cross-frame attention
ID-Adapter
Input addition
High-level (semantic) condition
Low-level (fine grained) condition
Methods for controlling video generation using image-based conditions. We categorize these methods along a spectrum ranging from high-level semantic conditions to low-level conditions. Additionally, we illustrate four classic approaches commonly used to condition input images.
Figure 12:
global information from the image, which is then used to condition video generation Zhang et al. 2023d
Xing et al. 2023 Wang et al. 2023e Xu et al. 2024b Jiang et al. 2024b Zhao et al. 2024a).
Fine-grained-level conditioning. High-level control is not always desirable. In some cases, we aim fully control every detail of the generated vidco. At the extreme, fine-grained conditioning means the conditioning image directly corresponds to one of the frames the generated video. This is achieved by exposing the network to the fine-grained features of the conditioning image.
The most straightforward approach is to concatenate the image (or its latent representation) with the noise input. This concatenation typically occurs along the channel axis, with the conditioning image often serving as the first frame of the training video. This approach is widely adopted in many video generation models, including SVD Blattmann et al. 2023a VDT (Lu et al. 2024ap, Motion 12V (Shi et al.) 2024 AioooVideo (Gong et al., (024), and CogVidoon (Yang et al.) 2024j). Since the model predicts multiple frames simultaneously and the conditioning image usually a single frame, the image can either be repeated across all frames temporally padded with black pixcls, accompanied by additional mask channel to indicate the validity of the condition for each frame. Notably, this concatenation-base approach is also
widely used in image generation Brooks et al. (2023); Ke et al. (2024): Zeng et al. (202cc); He et al. 2024c)
Concatenation can also be done along the temporal axis (Shi ot al. 2024 Ma et al. 2024b Zhang et al. 2024j Gu et al. 2024a Dai et al. 2023), where the conditioning image is explicitly set as the first frame of the predictions. This method avoids the need for modifications to the model architecture.
While concatenation is often used for conditioning the first frame, Make Pixel Dance ęZeng et al. 2024a) demonstrates that conditioning both the first and last frames enables the model to implicitly learn large- scale motion generation, shown in Fig. 13
MagDiff Zhao et al. 2024a) designs pyramid structure to encode conditioning images, effectively capturing contextual information at multiple scales and enhancing the robustness of the input.
PIA (Zhang al. 2023f) introduces a plug-and-play image conditioning approach that enables scamless conditioning across various personalized lext-to-image models without the need for fine-tuning. This is achieved by non-invasively injecting conditioning features into the first convolutional layer of the U-Net.
Image conditioning can also be implemented using a ControlNet-like approach (Zhang et al. 2023b). For instance, DreamVideo (Wang el al., 2023a) incorporates image conditioning by forwarding the conditioning image through the downsampling blocks of the diffusion U-Net and injecting the resulting intermediate feature maps into the vidco model. Similarly, SparseCtrl (Guo et al. (022bb) employs a ControlNet-liko structure but extends it to support temporally sparse conditions, enabling the use of one or more frames conditioning inputs.
31

---

Published in Transactions on Machine Learning Research (MM/YYY)
"The ball in blue fire and turns into a dragon in fire."
"A man rides huge fish, flying from the water into the sky."
Figure 13: Make Pixel Dance Zeng et al. 2024a shows that by conditioning both on the first and last frame, the model implicitly learns to generate large-scale motion.
12V-Adapter Guo et al. 2023b) highlights a potential challenge of concatenating: when fine-tuning a video model initialized from a texieoo-video model, directly concatenating the input image can disrupt the fun- damental weights. To address this, they propose cross-frame attention mechanisms to enhance the image condition. Specifically, additional cross-frame attention layers introduced after each self-attention layer, where keys and values derived from the first frame, and learned queries target cach output frame. This architecture is later adopted in EMO (Tian el al., 2024). VideoCrafter (Chen el. al.] 2023a also use cross attentions for image conditioning, but uses an additional projection network to align the image embeddings with text embeddings.
Moreover, with the incorporation of optical flow, image conditions can be utilized more explicitly. By warping the image features using optical flow and decoding the warped image, the generated frames effectively "copy" information from specific locations in the conditioning image. This approach is employed in several flow-
based video generation methods Nii et al. 2023 Li al. 2024k)
Multi-level condition. Semantic-level and fine- -ne-ggaiied conditions are often combined to achieve more robust and precise control. For instance, both concatenation and CLIP feature cross attention can be integrated into the U-Net together to improve conditioning consistency (Xu et al. 2024b Xing et al. 2023 Zhang et al. 2023d), VideoBooth (Jiang al. 2024b) combines CL.IP feature cross-atiention with image latent cross-Irame attention to provide a balance of coarse guidance and fine detail control.
VideoComposer /Wang et al. 2023e] controls the details of generated videos by employing a Spatio-Tempont Condition Encoder, which encodes various modalities, such images, sketches, depth maps, and edge maps. Simultaneously, it leverages the CLIP embedding of an image to control the style of the generated video.
Consist12V (Ren et al. 2024 enhances image conditioning by integrating multiple branches that support both coarse and fine control. These include noise concatenation, image cross-attention, and temporal attention. Additionally, it introduces layout control by using the low-frequency components of the input image noise initializatin for the video generation process.
32

---

Published in Transactions on Machine Learning Research (MM/YYY)
4.1.2 Spatial condition
Tora, Motion-I2V, FreeeTraj 0bjCurl2555 Boximator, SG-I2V, MotionBooth
Object- Level Conditions
Scene-Level Conditions
Stroetscapes, SparsoCtrl, CeneMaster
Spatial Condition
Multi-Modal Controls
MVideo, DreamVideo-2, Motion Prompting
User-Driven Spatial Interactions
Poekaboo, DragNUWA, DragAnything
Spatial-conditioned video generation leverages explicit spatial information and constraints to guide the syn- thesis process, granting users greater control over the layout, movement, and dynamics of vidco elements. These spatial conditions may appear at multiple levels of gramularity from direct object trajectories and manipulations to structural guidance via depth maps scene layouts and can be combined with other modalities to ensure coherence and consistency. Ultimately, spatial conditions form the backbone for con- trolling motion, be it object-level movement or global camera transitions, resulting in videos that more laithfully align with user intent.
Object-level conditions. This category emphasizes controlling the motion and placement of individual objects, often through explicit trajectory specification. Such approaches give users ine-grained command over how objects appear and move throughout the generated scene.
Early examples include Tora (Zhang et al. 2024h), which enables users to input detailed trajectories for multiple objects, resulting in high-fidelity, multi-object videos that precisely follow the prescribed paths. Similarly, Motion-I2V (Shi et al. 2024) accepts sparse trajectory annotations and motion brushes directly on a reference image, ensuring temporal consistency and accurate object-level motion control.
Building upon these ideas, FreeTraj (Qiu al. 2024a supports free-form, user-defined trajectories to guide object motion, allowing flexible and intuitive spatial control. By integrating trajectory information into latent space features and employing temporal consistency modules, FreeTraj ensures scamless motion across frames while maintaining coherent scene appearances. Taking this concept further, ObjCtrl-2.5D (Wang et al., 2024i) introduces object-centin centric trajectory control in a 2.5D latent space. Users can define paths including rotations, translations, and interactions-actt across multiple objects. With techniques like keyframe-basoe in- terpolation and motion-aware attention, ObjCtrl-2.5D guarantees spatial fidelity and smooth transitions.
Beyond trajectory drawing, object localization methods also facilitate spatial conditioning. Boximator (Wang ct al. 2024d) allows users to control object positions through bounding boxcs, incorporating a self-tracking mechanism to maintain motion coherence without retraining. Similarly, SG-I2V (Namekata et al. 2024 introduces a training-free approach that leverages bounding boxes and user-defined trajectories in an image- to-video diffusson pipeline. By aligning self-attention feature maps across frames and applying frequency- based post-processing SG-I2V enforces temporal consistency, preserves high-frequency details, and enables precise zero-shot motion control without additional fine-tuning. Such bounding-boo bisiing--bxbbseeed approaches com- plement trajectory-driven mothods, offering additional avenues for achieving granular and visually consistent object movements. Additionally, MotiouBooth (Wu et al., 2024c) fine-tunes text-to-video (T2V) models for subject-level control, introducing tailored losses (e.g., subject region and subject token cross-attention losses) that preserve object identity while enabling flexible subject movement. Its training- free "motion injection" allows users to define unique object trajectories on the fly with minimal overhead, maintaining fidelity to the customized subject's 's appearance even under significant motion changes.
Scenc-level conditions. While object-level conditions dictate the movement of individual elements, scene- level constraints shape the overall layout, geometry, and camera perspective. Methods like Streetscapes (Deng et al.] 2024) rely street maps, height maps, and camera trajectories to ensure spatial coherence and
33

---

Published in Transactions on Machine Learning Research
Pase Sequence
frame
Entity flamingo with pink feathers, long slender lege, eurved neek, and tuft of feathers on its back
Location city
Entity
a gazelle with body covered in sleek tan fur, long legs, and elegant ourved horns a man with medium-lenge length straight brown hair, tall and slender, a gray orew-neok t-shirt, beige trousers
Location glacier
Figure 14: An example of integrating camera poses and object motion control for video generation 2024a
Fu al.
temporal stability in city-scale videos. SparseCtrl (Guo et al. 2024b) uses sparse sketches depth maps at keyframes to propagate structural guidance through the sequence, reducing the need for dense inputs while maintaining spatial fidelity. CineMaster (Wang et al. 2025a further emphasizes 3D-aware scene manipulation by integrating precise 3D bounding box placement, flexible object and camera motion, and interactive layout control into a two-stage workflow.
Multi-modal controls. A natural extension of spatial conditioning involves integrating other modalities, such text, images, or optical flow, to provide more holistic guidance. MVideo (Zhou et. al.) 2024a) fuses semantic (textual) and spatial (image and trajectory) imputs, employing motion-aware attention mechanisms to ensure that these heterogeneons signals inform both appearance and temporal dynamics, thus allowing for richer and more uallextually grounded video generation.
By combining spatial and trajectory conditions, methods can align object movements and appearances across time. DreamVideo-2 (Wei et al. 2024c) exemplifies this synergy by using a dual-stream diffusion architecture. Its Appearance Stream preserves spatial features while a Motion Stream enforces trajectory- based temporal dynamics. Optical flow maps and framo-wise image conditioning further ensure spatial consistency. Similarly, MotionPrompting (Geng al. 2024b) maps explicit motion prompts representing object or camera trajectories into latent space and refines them through two-stage generation process, achieving both stable global motion patterns and precise spatial fidelity.
User-driven spatial interactions. Beyond automated integration spatial cues, several works empower users to interactively manipulate scene content. Peekaboo (Wu et al., 2020) enables direct control over object placement and movement via spatial and temporal masks, offering flexible adjustments without retraining the model. DragNUWA (Yin al. 2023a allows users to guide arbitrary object movements and camera shifts by combining trajectories with textual and image inputs, unlocking complex video editing through multi- modal manipulation. Extending this idea further, DragAnything (Wu et al.] 2024f) leverages draggable points on objects so users can intuitively reposition and pose them across frames, enhancing interactivily and customization within video diffusion frameworks.
In essence, spatial conditions and motion controls are tightly interconnected. Specifying spatial guidance through trajectories, bounding boxes, scene layouts, or depth maps sets the stage for orchestrating how objects and viewpoints evolve over time. By blending object-level trajectories, scene-scale structures, and interactive user cues, these methods collectively push toward more controllable and user-aligned video gen- eration.
4.1.3 Camera parameter condition
Camera parameter conditioning, which precisely manipulates camera poses and trajectories, is crucial for achieving realistic 3D consistency in video generation. Equally important is local object motion con- trol crol-nefining how specific entities move within the scene that the overall dynamic is both coherent
34

---

Published in Transactions on Machine Learning Research (MM/YYYY)
and visually compelling. Recent works interweave these two aspects, providing new ways to blend cinemato- graphic flexibility with targeted motion editing.
Methods like CameraCtrl (He et al. 2024a) rely on camera pose and trajectory imputs to guide video generation with geometric representations and temporal attention, allowing creators to simulate dynamic viewpoints for virtual reality and film production. Extending these ideas, CVD (Kuang et al., (024) enforces epipolar constraints to maintain geometric and motion consistency across multiple perspectives. Further exploring 3D constraints, CamCo (Xu et al. 2024b) adopts Plücker coordinates to ensure coherent epipolar geometry, while Cavia (Xu et al., 2024 employs vies and cross- foas-frame attention layers to align spatial and temporal cues in multi-view vidcos. VD3D (Bahmani et al., 2024d) uses scene reprosentations like depth maps or NeRF,, merging appearance and geometry streams for more realistic 3D effects, and AC3D (Bahmani et al. 2024b) integrates Plücker coordinates with hierarchical training and cross-frams attention to manage complex motions across multiple viewpoints. Augmenting these efforts, MotionBooth (Wu et al.] 2024c) introduces a lightweight latent shift module for training- free camera motion control, synchronizing smooth camera movements with subject fidelity via concise loss functions and allention alignment. For more in-depth discussion camera conditioning, please refer to Sec. 1.6.3
On the object motion side, various approaches link bounding-box bex-level or trajectory- editing to camera control for deeper storytelling. D-rect-A-Video (Yang et. al. 20244) allows users to specify camera movements such as panning or zooming alongside simple bounding-boo box manipulation for objects, removing the need for explicit motion annotations. MotionCtrl (Wang et al.) 2024j) goes further by letting users provide detailed trajectory paths for both objects and the camera, integrating these into a diffusion-based model to achieve precise movements and strong frame-to-frame consistency. Likewise, 3DTrajMaster (Fu et al. 2024a) incorporates 6DoF' pose sequences for multi-entity 3D motion control, using 3D-motion grounded injector architecture that fuses text and pose embeddings and a gated attention mechanism within diffusion pipeline. CineMaster (Wang et al., 2025a) further couples bounding- box bos-aaeed object manipulation with camera parameter specification in a two-stage pipeline. A Blender- based interface collects 3D bounding boxes and camera trajectories, which are automatically interpolated before guiding a specialized diffusion model that disentangles camera from object motion for enhanced 3D alignment. By merging camera parameters with object trajectories, these methods deliver more comprehensive control over how scenes unfold, enabling cohesive, lifelike video generation.
4.1.4 Audio condition
Audio conditions enable synchronization between sound and visuals, crucial for applications like talking head generation and co-speech gesture animation. By leveraging audio inputs, models generate videos that reflect the nuances of speech, emotion, and rhythm inherent in the audio signal.
Several works have focused on generating expressive talking head videos synchronized with speech. EMO (Tian et al.] (024)) conditions video generation on audio input and a single reference image, using facial bounding box masks and head movement speed as weak conditions. By integrating audio features from pretrained models like wav2vec into crafussio-bbseed framework, EMO synchronizes facial expressions and lip movements with the input audio, capturing emotional tones.
Similarly, VASA-1 [u et al. 2024e) and DreamTalk (Ma et al. 2023) utilize audio features extracted with wav2vec models to generate talking face videos. VASA-1 incorporates audio features from previous frames to maintain temporal consistency, employing classifier-fre, guidance for motion generation. DreamTalk, uses a mansformer-based audio encoder to process audio windows, producing synchronized lip movements and expressions with the help of a lip synchronization expert and style predictor. Speech2Lip (Wu et al. 2023e) builds on these ideas by adopting implicit model focuses on lip movement in a canonical space while a geometry-aware mapping (GAMEM) handles pose variability for speech-insensilive elements like head movements. A contrastive sync loss further boosts synchronization quality, even with limited training data, yielding sharper visuals and more accurate lip alignment than previous methods. Expanding beyond facial animations, ANGIE (Liu et al., 2022b) addresses co-speech gesture generation by learning motion representations through unsupervised learning. It constructs codebooks of reusable gesture patterns and predicts future motions based on quantized motion codes and
strategy, where a speech-driven
35

---

Published in Transactions on Machine Learning Research (MM/ (YYY)
"Angry"
or
"Fear"
Source Image
Driving Audio
Diverse Synchronous Facial Dynamic
Emotion-Aware HD Texture
Fine-Grained Clear Teeth
Figure 15: In audio-conditioned video generation, in conjunction with traditional conditions (e.g., image, text), audio information is usually adopted to augment synchronization with talking iotion (facial ex- pressions, head movements, lip and teeth dynamics) to create a more realistic and emotionally expressive animation (Tan et al. 2024a).
audio inputs, resulting in natural gesture animations synchronizod with speech. Similarly, OmniHuman Lin et al. 2025) extends audio-driven generation for more comprehensive human animation tasks such as full- body movements and human-object interactions, leveraging a multi-condition training strategy (text, audio, and pose) in progrossive manner, and dclivers realistic motion synthesis and robust lip-syne accuracy.
For emotional expressiveness, EMOPortraits (Drobyshev et al. 2024 integrates speech-driven mode into one-shot head avatar synthesis, using multi-view dataset of extreme facial expressions (FEED) to generate highly expressive facial animations driven by audio inputs. FlowVQTalker (Tan et al. 2024a) combines normalizing flow and vector quantization to model and generate emotional talking face videos, predicting facial dynamics using audio inputs, a source image, and an emotion label.
To capture subtle expressions and head movements, AniTalker (Liu al. 2024d) and AniPortrait (Wei et al. 2024a) utilize motion representations learned through self-supervised learning. AniTalker introduces variance adapter to synthesize speech with controlled attributes, serving input to diffusion-based motion generator. AniPortrait employs a two-stage method, obtaining 3D representations from audio inputs and converting them into high-quality portrait animations using a diffusion model.
Additionally, CCVS (Le Moing et al. 2021) (Context-aware Controllable Vidco Synthesis) incorporates audio tracks as ancillary information influence the vidco's rhythm and mood, maintaining temporal coherence by utilizing context. frames and object trajectories.
36

---

Published in Transactions on Machine Learning Research (MM/YVV (YYY)
4.1.5 High-level video condition
T-NeeA-Video, Vid2Vid-Zero, Edit-A- Edit-A-Video, EI?, SimDA, UniEdit, AnyV2V, VidToMe, TokenFlow
Attention-based Editing
Gen-1, MagicEdit, VideoComposer, MoCA, Flow Vid, VideoControlNNet et al.), MotionClone, MotionDirector, SAVE, VideoConurolNet(HHu et al.)
Structural & Motion Guidance
Consistent Editing
Text2Video-Zero, Rerender A Video, Pix2Video, 12VEdit, RAVE, ControlVideo, Diffusion Video Autoencoder, ColorDiffusor
Latent Space Manipulation
LNA, VidEdit, Diffusion CoDeF, StableVideo, Lee al.
Canonical Representation
High-level Video Condition
DragVidco, VideoSwap
ReVideo,
Interactive Editing
Follow Holow-Your-Pose, MagicPose, AnimateAnyone, DisCo, Dream AniTalker, DPE
Specialized Control
Human-centric Editing
Instruction-basee Editing
InstructVid2Vidd InsV2V
SimDA, VidToMe, STEM Inversion, Tune-A- A-Vido
Erficiency-Oriented Methods
Efficiency & Adaptation
In recent years, diffusion-based video editing has become powerful paradigm for synthesizing and manipu- lating videos while maintaining consistency in appearance, motion, and structure. Research in this domain has flourished across different dimensions, including strategies for preserving spatio-temporal coherence, specialized user controls, and approaches that boost efficiency. Below, we group these methods under three broad categorics and detail the core idcas within cach.
Consistent editing. A large body of work addresses how to preserve uniformity across frames, focusing refined attention mechanisms, structural and motion guidance, latent space manipulation, and canon- ical representations. In terms of attention-basel approaches, some of the carliest breakthroughs emerged when lext-to-image diffusson models were exlended to video. Tune-A-Video et al., 2023b) introduced sparse-causal aratio-temports attention to adapt Stable Diffusion for one-shot video editing, while Vid2Vid- Zero Wang et al. (2033d) developed dense spatial-temporal attention module and cross- fross-frame attention capture longer-range temporal dependencies. Edit-A- A-Video (Shi al.) (2024) likewise addressed temporal flicker by injecting attention maps from previous frames and applying novel temporal-consistent blending
37

---

Published in Transactions on Machine Learning Research (MM/YYY)
method. Further enhancements came from methods like EI? Zhang et al. 2024k and SimDA SimDA(Xing et al, 2024b), which minimized feature shifts via Shift-restricted Temporal Attention Modules, instance centering, spectral normalization, and Latent-Shift Attention to achieve smoother updates with fewer parameters. Some tochniques also explicitly combine spatial and temporal features: UniEdit [Bai et al.) 2024) and AnyV2V jtt al. 2024) fuse sell-attention features across frames or integrate DDIM inversions to preserve both content and motion. Token-level operations further enhance frame alignment, with VidToMe[[i et al. 2024h) merg-
ing redundant tokens locally and globally, and TokenFlow GGeyer al. 2023 identifying nearest-neighhor tokens to propagate consistent structure.
Beyond direct attention, some methods leverage explicit structural or motion signals to guide the diffusion process. Gen-1 ( et al., 2023) demonstrated that providing depth maps alongside noisy latents ensures coherent scene geometry, while MagicEdit (Liew et al. 2023) and VideoComposer (Jian [Jiang et al. 2023a) used ControlNet or spatio-temporal encoders for depth, pose, or sketch guidance. Optical flow is also popular for ensuring consistent movement: MoCA((an et al.) 2023a) projects edits from the initial frame onto sub- sequent frames through flow warping, Flow Vid[Liang al., 2024a conditions the diffusion on flow-aligned features, and MideoControoNeet(CHi et al. 2023) uses similar flow-based constraints to reduce flicker. Motion- Clone(Ling et al. 2024cb instead relies on internal temporal attention signals to extract dominant motion cues, making it training-fres flexibility, as in Motion DoctionDirecto(// Director(Zhao et al. 2023c) (with dual-path LoRA) and SAVE (Son et al. 2023 (with a motion word and motion-aware cross-attention loss). A specialized example is VideoControlNet (Hu et al.) (Hu & Xu 2023). which segments the video into I-, P-, and B-frames for independent generation, motion-guided updates, and interpolation, respectively.
Splitting appearance and motion into separate pathways further enhances
Latent space manipulation is another powerful direction for preserving consistency. Text2 CerooKhachatryan et al. 2023) and Rerender A Video(Yang et al., 2023b/ warp the latent codes of initial
or subsequent frames to guide motion while keeping appearance stable. Pix2Video(Ceylan et al., 2023) ) and 12VEdit et al., 2024b inject self-attention features from prior frames to ensure visual continuity.
Other methods, such as RAVE el al. 2024) (noise shuflling and the grid trick) or Control Video (Zhegg et al.) 2023eb (an anterleaved-frame smoothing pipeline), unify frame latents during denoising. Diffusion 2023a) explicitly disentangles identity, motion, and background within la- tent representations to simplify partial edits, and ColorDiffusor (Liu et al. 2023a addresses color drift with dedicated Color Propagation Attention and an Alternated Sampling Strategy.
Video Autoencoder (Kim et al.
Finally, canonical representations use a consistent 2D atlas or parameterization to propagate edits across frames. Methods like Layered Neural Malases(Kasten et al.) 2021), VidEdit/Couairon (Couairon et al., 2023). and AiffusioonAlasChhaaaa et al., 2023b) map videos into an atlas space, apply edits there, and then map them (oDyanOuyaag
back to the original frames. CoDeF (Ouyang al. 2024a) extends this concept using a based design
for canonical and deformation fields, while StableVided Vivioiiohhhaa CChai et al. 2023) and Lee et al. (Lee et al., 2023) propagate keyframe edits via atlas-based correspondences, with the latter estimating dense semantic links to handle shape changes.
Specialized control. Several methods emphasize more direct or domain-specific control, whether through user interaction, human-centria capabilities, or language-basr based instructions. Interactive editing systems like DragVideo (Den et al. 2023) and
al. 2023) adapt point-basod image manipulation to the video domain by tracking user-selected points across time and optimizing latent codes for smooth motion. MeVideo[Mou et al. 2024) lets users edit. the first frame and then specify new motion trajectories, supported
by a three-stage training strategy that decouples content and motion. VideoSwap(Cu et al. 2024b) targets subject swapping via semantic point correspondences and layered atlases, maintaining consistent appearances in dynamic scenes.
Human-centric editing often centers on preserving identity, pose, and expressions across multiple frames. Approaches like Moalow-Your-Pse///ee et al., 2024d) and MagicPoseeChang et al., 2023a) typically train for appearance control first and then refine temporal or pose-specific features using modules like Control Specializod appearance encoders, in AnimatcAnyone Hu 2024) MotionEditor Tuu ct al. 2024), capture high-fidelity details that are integrated into the denoising process for better motion generation. Broader subject or background manipulations appear in DisCo Wang et al. 2024eb, CreamPose(Kaane et al. 2023
38

---

Published in Transactions on Machine Learning Research (MM/YYYY)
and AniTalker(Liiv (Lu et al. 2024d], leveraging ControlNect-asse architectures to separate recombine different video elements. DPF( Pang et al. 2023) focuses specifically on decoupling pose and expression through a bidirectional eyclic training scheme, skipping 3D Morphable Models for simpler pipelines.
iiaructtoo-baset editing harnesses natural language to guide transformations in video, making them ac- cessible to non-experts. Instruct Vid2Vid al.) 2024) and InsV2V (Cheng et al. (023) adapt Prompt- to-Prompt editing for multi-frame scenarios by generating training data from captioning systems. They preserve coherence via specialized objectives like an Inter-Frames Consistency Loss and strategies such Long Video Sampling Correction.
Efficiency-oriented methods. Since diffusion-based pipelines can be computationally intensive, numer- ous works locus on streamlining editing without degrading consistency. SimDA [ing el al.) 2024b) em- plays Latent-Shift Attention and lightweight adapters, while VidToMe(Li et al., 2024h) uses Token Merging (ToMe) to compress attention tokens both locally and globally, saving compute cycles across frames. STEM Inversion(Li al.) 2024g) applies an EM algorithm to find compact bases for each frame, reducing the dimensionality of video representations and speeding up inference. On similar note, Tune-A- Tume-A-Video/f et al. 2023b) demonstrates that carefully designed sparse spatio-temporal attention modules can maintain temporal stability while significantly cutting resource usage.
4.1.6 Other conditions
Recent advancements in video gencration have incorporated varions special conditions to enhance control, realism, and applicability. These conditions include audio inputs, physical simulations, trajectories, scene layouts, and physiological signals. Below, we survey these developments, categorizing the works based on the special conditions they employ.
Physics-basede video generation. Incorporating physics-based conditions ensures that generated molions adhere to real-world dynamics. PhysGen (Liu et al.] 2024c) conditions video generation on external forces and physical parameters like gravity and friction to simulate realistic object motion. Starting from a single image, it extracts physical properties, simulates dynamics using a physics engine, and renders the motion using diffusion models, allowing for physically grounded and controllable video sequences.
Similarly, MotionCraft AAiaa al., 2024) uses physics-based optical flow derived from simulations to guide video generation. By warping the latent space of pretrained diffusion model according to the optical flow, it animates the starting frame with the desired motion, enabling the synthesis of complex movements without retraining the model.
Extending physics-based conditioning to interactive scenarios, VideoAgent (oni et al. 2024a) introduces multi-agent interactions guided by instructional text prompts and environmental contexts. By simulating agent behaviors that adhere physical laws, it generates purposeful vidcos involving multiple interacting agents.
Physiological signal conditioning. Incorporating physiological signals opens avenues for health-related applications Synthetic Generation of Face Videos with Plethysmograph Physiology (Wang et al. 2022cl uses a given PPG signal to guide video generation, reflecting physiological signals like heart rate. Starting from a single image, it introduces variations to enhance the training of remote photoplethysmography models.
Timestamp conditioning. Timestamp conditions pinpoint precisely when events occur, ensuring proper sequencing and duration. Min'T (Wu et al., 2024h exemplifies this by binding each event a designated time range, using Rescaled Rotary Positional Encoding for coherence, and leveraging scene cut conditioning for seamless transitions. With LLM-based prompt enhancer to expand event captions, MinT maintains smooth multi-event progression, improves text alignment, and preserves subject integrity, highlighting the importance of tine-based control for narrative-rich video generation.
4.2 Enhancement
The enhancement of image and video qualities has been a long-lasting and well-studied topic in computer vision. Early literature focused on low-level (i.e., pixel-level) video enhancement, where we process videos
39

---

Published in Transactions on Machine Learning Research (MM/YYYY)
based on low-level photometric geometric features to resolve specific issues such video noise, motion blur, elc. However, in recent years, the success of generative models including GANs and diffusion models has offered new options to tackle these traditional low-level tasks. In this chapter, we review and discuss the potential of these new generative models on traditional video enhancement tasks including video denoising, video inpainting, video super-resolution, video interpolation and prediction.
4.2.1 Video denoising and deblurring
Vidco denoising aims to enhance vidco quality by reducing noise and restoring degradation issues like out- of-focus blur, motion blur, and compression artifacts such as moiré patterns (Rola et al. 2023). The noise or degradation defined in these tasks low-level meaning that it generally does not require much high-level (semantic/object-levee) understanding of the vidco.
Interestingly, the latest, diffusion models are inspired and structured with a demoising framework, so they seem to be a direct fit for video denoising. However, they have completely different focuses. The latest diffusion models are mostly designed for highly creative generation tasks, where use the model to generate from an image of complete noise. The whole process is more about "imagining" what might be hidden beneath the noise, rather than simply "removing" the noise. With large datasets, diffusion models learn the overall distribution of images rather than focusing on refining individual local patches based on appearance. In contrast, video denoising assumes that the input videos, although noisy, should start with enough quality, where least the main content should be be recognizable. The goal is to reduce noise while keeping the main content unchanged in the video, leaving little room for crcativity compared to generative diffusion models.
More broadly, there have been few successful generative models, not just diffusion models, specifically de- veloped for vidco denoising tasks. The reasons are thrco-fold.
Video denoising requires the model strictly maintain the original content of the video and avoid introducing extra new information. The task itself limits creativity, so generative modesls generally do not fit very well in this area.
Different focus:
Existence of good traditional methods. Since the task primarily only involves minor local refinement, video denoising generally not very challenging task. Also, due to its long history in research, many effective traditional methods already exist and can address the problem reliably. Using generative models for this relatively simple task may seem overkilling.
Efficiency. Compared to traditional methods, generalive models are much more complex, making them harder train and less efficient in practical applications.
4.2.2 Video inpainting
Video inpainting a process used to fill in the missing or corrupted parts of a video Quan et al. 2024 A masked area is usually specified as part of the input, and the task is aimed filling in that area using spatial and temporal cues to make the video appear seamless and coherent. Note that the masked area can be specified in various ways depending on the application scenario. Here are some examples.
In applications where the goal is to restore a corrupted area in the video, the masked area is defined as the region of coruppiin, and the task mainly focuses on filling in the area based on the local continuity of the frame.
In applications where the goal is to remove unwanted objects, the mask area is defined as the object masks, and the task focuses more on finding temporal correspondences to fill in the background behind the object.
While traditional methods typically adhere to a predefined problem setting with a specified masked area, recent diffusion methods aim to be more creative by incorporating language instructions and addressing
40

---

Published in Transactions on Machine Learning Research (MM/YYYY)
"AMINI Cooper driving down a road." (5.3
"A yellow maple leaf." " (2.7 s)
vain traveling over bridge mountains: (8.0 s)
Example of video inpainting with text instructions for vidco generation. The contents in the masked regions are inpainted under text instrubtions. Original figure from AVID (Zhang al. 2024i
Figure 16:
more challenging cases through complex generation. One such example is shown in Figure 16 where masks and instructions are provided to enable video editing applications as a video inpainting task Zhang et al. 2024i).
GAN (Generative Adversarial Network) models were first adopted to solve the inpainting problem from generative perspective. Starting from image impainting, Pathak al. (2016) uses inpainting self- supervised task to learn video representations, where adversarial loss is introduced to produce sharper results. Yu et al. (2018) also introduced local and global WGANs for enhanced results. Recently, [Chang ct al. (2019) extended this idca to free-form vidco inpainting, where the masks can have arbitrary shapes such as text captions, and proposed a temporal PatchGAN loss to enhance temporal consistency.
Diffusion moodels have also been explored for inpainting tasks, and such explorations mainly start from image inpainting. RePaint (Lugmayr et al.) 2022) applicd pretrained unconditional DDPM and modified the standard denoising strategy by sampling the masked regions from the diffusion model and sampling the unmasked areas from the given image. SmartBrush (Xie et al. 2023 explored image inpaiting with both text and shape guidance with precision control. They try to generate & new object to be filled in the shape specified by the masks. To preserve the background and improve consistency, they add an extra mask prediction module the diffusion process, which used guide the sampling process in the inference stage. For video inpainting, Green et al. (2024) reframe the task as a conditional generative modcling problem lackled with conditional video diffusion models. They use sampling schemes specific to inpainting problems to capture long- rangeange dependencies and show that their model can inpaint diverse novel objects that are semantically consistent with the context.
To better utilize the multi-modal capability of diffusion models, some research work has focused on adding inputs of other modalities, such as text instructions, into video inpainting, paving the way for real applications like text-guided vidco editing. AVID (Zhang et al. (0244i) builds on existing text-guided imagc-inpainting models and improves temporal consistency by integrating motion modules. They also proposed structure guidance module tailored for different types of masks, and middle-frame attention guidance method for
41

---

Published in Transactions on Machine Learning Research (MM/YYY)
put plate on counter
Fred is tolkiing to Barney and turning head.
Jumps over gear and then collects cošn.
Prediction
cLose drower
wiCmo ridiing the and talking.
Runs from Left right and jumps
Video
plate off
dancing across noom
Jumps
up gen.
Video
Infilling
Runs right. jumps down the ground.
and moves to Left.
Keeps woLking and drops off the plotform.
Control
by Texts
right. jumps Landinng
face.
Jueps from
Left.
Jumps gear drops off platfors.
cotppoaatiooppeeictttoon models, is possible add text instructions for better control. Original figure from Fu ct al. (2023)
Figure 17:
Examples video extrapolation/ prediction (Row 1&:2) and interpolation (Row 3). For diffusion
longer videos. Subsoquently, CoCoCo (Zi et al. 2024) proposes a more efficient motion capture module and an instance-aware region selection method for better "consistency, controllability, and compatibility" (and hence the name 'CoCoCo").
Furthermore, UniPaint (Wan et al. 2024 presents interesting argument that video inpainting and in- terpolation can be unified and mutually enhanced as both tasks involve filling missing information in the spatial and temproal domains, respectively, for which they propose spatial-tempora masking strategy dur- ing training. Wu et al. (2024b) introduces pure language-driven video inpainting task, where the specified inpainting masks are no longer nceded. Instead, they use natural language instructions, such as instructions remove objects, to guide and condition the inpainting process.
4.2.3 Video interpolation and extrapolation/prediiioo
Video interpolation aims lo generale intermediate frames between two conseculive frames to increase the video's frame rate. In contrast, video extrapolation or prediction focuses on forecasting the next frame following the input frames. Both tasks involve generating new frames along the temporal dimension and thus depend heavily understanding the motion between frames.
Traditionally, video interpolation and extrapolation/ppreiie pprediction are distinct tasks approached with different methodologies. Video interpolation focuses on finding or synthesizing intermediate frames to reflect the motion between existing frames, while video extrapolation requires more creativity and may rely on external domain knowledge or common sense to help predict future frames. Morcover, video interpolation has clear and direct real-life applications and generally produces more reliable results because it uses temporal cues from both preceding and succeeding frames. As result, video interpolation has traditionally been of greater interest compared to video extrapolation/ prrriionnnpeeiiiion..
However, from the perspective of diffusion models, video interpolation and extrapolation can be tackled in a more unified framework. One example is illustrated in Figure 17 Therefore, we discuss both tasks together in this chapter.
With the rise of large pretrained diffusion models, video interpolation and extrapolation are increasingly viewed as conditional generation tasks, rather than relying heavily on motion analysis and image warping.
42

---

Published in Transactions on Machine Learning Research (MM/YYYY)
In both cases, generative methods follow the same approach generating new frames based on the surround- ing context frames. This allows unified diffusion model to be trained to handle both interpolation and extrapolation.
One early example is MCVD (Masked Conditional Video Diffusion) (Voleti et al. 2022 1, where a diffusion model trained to generate current frames conditioned on past and future frames. By randomly masking these frames, the model adapts different tasks. If both past and future frames are masked, the model performs unconditional video frame generation. If only either past future frames masked, it handles past frame or future frame prediction, respectively. When neither are masked, the model performs video interpolation. This random masking enables MCVD to learn video interpolation, prediction, and generation in unified framework. Similar ideas also appeared in RaMViD (Random-Mask Video Diffussio) (Höppe et al., 2022), but instead of masking in the temporal dimension, RaMViD apply masks in the spatial domain and use unified framework to tackle video interpolation, infillin, and upsampling.
Moreover, some later methods also specifically explored the power of diffusion models on video interpola- tion. VIDIM uses cascaded diffusion models, where low-resolution target frames first generated before generating high-resolution versions conditioned these generated low-resolution frames et al. 2024a LDMVFI explored latent diffusion models to achieve better perceptual generation quality for video interpo- lation Danier et al. 2024).
Some other recent work has also adapted video tntterolation/exttaapaattt to specific application scenarios. Fu et al. (2023) proposes MMVG (Multi-modal Masked Vidco Generation) tackle a new adapted task called text-guided video completion (TVC) motivated by the uncertainty of frame prediction. Specifically, a vidco can have multiple potential future outcomes, it is better to add text guidance describe specific future that we want to generate. Apart from that, some work explored interpolation for specific styles such as cartoon (Xing et al. 2024a).
Overall, we have seen great differences between traditional and generative methods.
Creativity: Traditional methods adhere to the original content, motion, and physics in the video, while generative methods can generate in more creative way.
Traditional methods mostly rely on strong motion assumptions such as motion smoothness and linearity, so they may fail if the underlying motion is complex, nonlinear, or ambiguous (Jain et al. 2024a). Also, these smooth motion assumptions often result in overly smoothed predictions by traditional methods. However, diffusion models are less vulnerable to these problems due to their low dependence on motion understanding.
Dependence on motion:
Difficulty: Traditional methods are typically designed for simple settings, where only one interme- diate or future frame is predicted from context frames, with short time intervals between them to maintain smooth motion assumptions. In contrast, generative methods can handle more complex tasks, such as interpolating or predicting 5-10 frames at a time, and the context frames can be far apart time. An extreme example would be having only the first and last frames of a 5-second video. In this case, generative models are well-suited to generate the entire video since most of the information is completely made up. Traditional methods, however, would struggle to handle this scenario due to lack of context. We will discuss more about long video generation in Section 4.5
4.2.4 Video super-resolution
Video super-resolution is another well-established task in video restoration, focusing on enhancing low- resolution videos by upsampling the video frames to produce clearer and sharper high-resolution outputs. Early rescarch focused more on image super-resolution algorithms, followed by video super-resolution meth- ods, which introduced temporal consistency as an additional optimization objective. Some visual examples are shown in Figure 18
In recent years, diffusion models have also been applied to super-resolution tasks. StableSR WWng et al. 2024cb explored leveraging the prior knowledge from pre-trained diffusion models to improve super-resolutio quality. They employed frozen Stable Diffusion model to generate high-resolution images conditioned on the
43

---

Published in Transactions on Machine Learning Research (MM/YYYY)
LO
(6) Zoomed LR
(b) DSRGAN
(c Real ESRGAS
(d) FeMuSR
(o)
(
Figure 18: Video super-resolution examples of latest methods Original figure from StableSR Wang et al. 2024c
low-resolution features computed from pretrained super-resolution model. A trainable time-aware encoder module was added to combine the super-resolution features with the diffusion features. Similarly, Yuan et al. (0024b) proposed to utilize a frozen pre-trained text-to-image diffusion model as the base to accomplish text-to-vidoo super-resolution. They first use the image diffusion model to generate high-rcsolution images one-by-one and then refine the results through a temporal adapter.
Some latest methods have also investigated different ways to combine super-resolution networks (mostaly a VAE) and diffusion networks (U-Net). SATeCo (Chen et al. 2024f combined the super-rosolution (VAE) and diffusion model (U-Net) by inserting the U-Net structure in between the VAE encoder and decoder, together with some adapation or alignment modules. The latest VEnhancer (He et al. 2024b) also shares similar structure with additional space-time data augmentation and video-aware conditioning, which are trained to enhance Al-generated video on both space and time resolutions. In comparison, Upscale- A-Video (Zhou et al., 2024bb more designed for long videos. They first utilize the U-Net generate each video segment given optical flow inputs and perform global recurrent latent propagation to ensure consistency across a longer time range. The VAE decoder is then applied to decode high-resolution images. Both structures have shown pros and cons flexibility, fidelity and quality.
In addition to using diffusion models to enhance video resolutions, some research work has also studies how to enhance/aad /adapt low-resolution image/viden ion models so that it can also work well on high- resolution tasks. Guo et al. (2025a) propose a buning-free method through a pivot replacement strategy that leverages reliable semantic guidance from the low-resolution model, as well as a tuning method that integrate learnable multi-scale upsampler modules to learn structural details at the new higher resolution. To follow up, FreeScale (Qiu et al. 2024c) introduces tuning-free inference paradigm that fuses features from different perceptive scales and extracts the desired frequency for each part of the frame.
4.2.5 Combining multiple video enhancement tasks
As we have discussed in the previous chapters, many diffusion methods have unified and tackled multiple enhancement tasks the same time. To recall a few, UniPaint Wan et al. (2024) combines video inpainting and video interpolation, and MCVD (Voleti et al. 2022 unifies video interpolation and video prediction.
44

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Notably, VEnhancer (He al. 2024b) is plug-and-play method used to enhance low-quality video generated by other generation models by removing artifacts and increasing both of its spatial and temporal resolutions.
This trend mainly stems from the fiexibility of diffusion models which can be trained to generate outputs in a task-agnostic manner. As a result, the differences among enhancement models are more based on different training data and strategy instead of model structures.
4.2.6 Summary
In summary, recent diffusion-based methods have significantly advanced traditional video enhancement tech- niques, improving tasks such video denoising, inpainting, interpolation, extrapolation prriiiiiiireeiioin, and super-resolution. From this review, we see a clear trend follows.
Diffusion-based methods introduce greater creativity into traditional low-level video enhancement tasks. Leveraging the strength of pretrained diffusion models, these approaches can tackle more complex and challenging scenarios where traditional methods often fall short.
Diffusion models have also enabled the use multi-modal inputs, allowing enhancement tasks to be performed with additional inputs like text instructions. This advancement is revolutionary, as text-based input provides an effective means of making the enhancement process more controllable and customizable, opening new possibilities for video editing.
From the perspective of diffusion models, the distinction between image and video is becoming less significant. Traditional methods often develop image and video enhancement algorithms separately, as video enhancement requires careful attention to temporal correspondences and alignment (the "physics" of motion). In contrast, diffusion models, which are less dependent on explicit motion understanding, allow for more scamless and scalable adaptation between image and video models.
Nevertheless, diffusion models can introduce more fidelity issues due to the inherent randomness of the diffusion process. In video enhancement, the output is expected to preserve the original content or subject matter, but diffusion models may introduce new elements that were not originally present. Finding ways to balance fidelity and quality can be interesting topic for future rescarch.
4.3 Personalization
Personalization in video generation is essential for creating content that accurately represents specific sub- jects, capturing their unique identities, expressions, and motions Two primary approaches enable this personalization: subject-specific fine-tuning and subject-speelii-driiene inference. While fine-tuning ap- proaches carefully adapt model parameters to particular subject, baseed methods rely minimal references often single image to guide identity preservation and motion synthesis without retraining.
Subject-specific fine-tuning. Subject-specific fine-tuning involves adapting pre-trained model to a particular subject by retraining or modifying its parameters using data specific to that subject. This approach allows the model to embed detailed subject characteristics directly into its learned representation.
DreamVideo Wei et al. 2024c) finc-tunes an identity adapter on a few static images the subject, capturing subject-specific identity traits, while separately training motion adapter on target motion patterns. This decoupling allows the system to integrate both identity and motion seamlessly into the final output.
Similarly, PersonalVidco (Li et al. 2024c) adopts finc-tuning strategy to preserve high identity fidelity without degrading the model's motion and semantic capacities. By integrating an isolated identity adapter into the spatial self-attention layers of a pre-trained text-to-video diffusion model, PersonalVideo fine-tunes identity features without disrupting motion dynamics. This method enhances robustness through simulated prompt augmentation, ensuring that even with limited reference images, the model achieves reliable identity preservation alongside customizable style and motion.
45

---

Published in Transactions on Machine Learning Research (MM//YYY
DreamVideo
Training
Inference
Sidpets-ppece ded
+
+
*
deg" playing guttor
Su specific Fnetuning
trakiing
General data
DreamVideo-2
Sutjec-spedtic data
Figure 19: A high-level comparison of two personalization paradigms subject-specific finetuning and subject-spoeffc-drivee inferenco-illusrratee by DreamVideo (Wei et al.) 2024c) and DreamVideo-2 (Wei et al. 2024bb. In DreamVideo, each new subject (its images and desired motion) is used to finetune the model. In contrast, DreamVideo-2 leverages pretrained diffusion model trained on general videos (decom- posed into subjects and bounding boxes); it then generates videos for new subject based on a single image and bounding boxes without requiring additional finetuning.
By extending these techniques to scenarios without specialized video data, Still-Moving (Chefer et al. 2024 leverages text-to-image (T2I) customization and introduces lightweight spatial and temporal adapters for rext-to-videe pipelines. By training on duplicated still images (or "frozen videos"), it retains motion priors while aligning T2T and T2V features, enabling diverse personalization with only handful of static images. Tn parallel, Dynamic Concepts (Abdal et al. 2025) addresses more inherently dynamic concepts-subjee whose characteristic motion is as integral to their identity as appearance by fine-tuning Diffusion Transformer in two stages. First, it learns an appearance "basis" from an unordered set of frames, and then refines temporal dynamics by freezing the learned parameters and focusing on motion consistency. Finally, Vidco Alchemist (Chen et al. 2025a expands personalization to multi-subject, open-set scenarios by fine-tuning a Diffusion Transformer, and personalizes both foreground objects and backgrounds while mitigating overfitting through targeted data augmentation.
inference personalizes video outputs at run- time without retraining the model for each new subject. These methods typically rely on single reference image (and sometimes auxiliary inputs like audio or text prompts) to guide identity preservation and ex- pression synthesis.
Subject-speiffcddriven inference.
Many audio-driven talking head frameworks fall into this category. EMO (Tian et al. 2024h. VASA-1 Xu
et al. 2024e), AniTalker (Liu al., 2024d), and AniPortrait (Wei et al. 2024a] generate expressive talking head videos from single reference image and audio input. They rely diffusion-based models and carefully designed motion encoding strategies to preserve subject identity while animating expressions and head movements. Approaches like Dream Talk (Ma et al.] 2023). EMOPortraits (Drobyshev et al. 2024 and Flow VowwVQrlleer (Tan el al. (2224a) extend this concept by incorporating emotional expressiveness, pre- dicting personalized emotion intensity from audio without additional style references, and ensuring identity consistency even under intense expressive variations. Beyond facial animation, ANGIE (Liu al. 2022b) expands this paradigm co-speech gesture generation, personalizing gesture patterns from minimal inputs while retaining the speaker's characteristic style.
46

---

Published in Transactions on Machine Learning Research (MM/YYY)
Recent works further refine and expand subject-specfffe-drien inference into broader video customization scenarios. For instance, DreamVideo-2 (Wei et al. 2024b) introduces zero-shot subject-driven video gen- eration with precise motion control. Using only a single reference image and bounding box sequences to define the subject's trajectory, it employs reference attention mechanisms and mask-guided motion modules integrate subject appearance with flexible, fine-grained motion paths-achieving
achieving personalization without
additional fine-tuning steps.
Building this idea of inference-time personalization, ConsisID Yuan et al. 2024a demonstrates how decomposing identity features into low-frequency (global facial contours) and high-frequest (fine details) components can ensure identity preservation in text-driven synthosis. By integrating these components into a pre-trained Diffusion Transformer, ConsisID consistently captures the subject's 's essential features across varying poses and expressions, all without retraining for each subject.
Lastly, MCNet (Hong Xu, 2023) addressos motion transfer scenarios, animating still image to follow the dynamics of driving video while preserving identity through an implicit identity representation conditioned memory module. By extracting and reusing structural and appearance details from a single reference image, MCNet maintains subject fidelity throughout complex motions again, without retraining the target subject. Together, these approaches illustrate the growing versatility of inference- baaedd frameworks that balance flexible control, identity preservation, and visual fidelity without the burden of subject-specific fine- tuning.
4.4
Consistency
Simple Conditioning
Inconsistent Appearance & Abrupt Motion
Prior Methods
SEINE
"a dog wearing a Superhern oufil with cupe flying through the sky."
Toddy bear walk- ing down 5th Avenue.
A fast-paced ma- toreyele race windingg track.
Ours
Ours
Sputial
Temporul Conditioning. Layout Guidance
Consistent Appearance & Motion
Videns:
click to play in Adobe Acrobat
Figure 20: Comparison of image-to-video generation results obtained from SEINE and Consistl2V, SEINE shows degenerated appearance and motion as the video progresses, while ConsistI2V maintains visual con- sistency utilizing spatial and temporal conditions.
Improving the consistency of generated videos has been hot topic recently. Researchers have been working improving consistency in two different perspectives, i.c., semantic consistency, and motion consistency. Early attempts (Yu et al., 2021 Skorokhodov et al. 2022 Shen et al. 2023 Zhang et al. 2023ch for GAN- based vidco generation mothods mainly add additional discriminators or learned representations for control- ling consistency. For example, DI-GAN (Yu et al., (021)) introduces an implicit neural representations- video generator that improves motion dynamics by manipulating space and time coordinates along with a motion discriminator for temporal consistency. StyleGAN-V (Skorokhodov et al., 2022) employs continuous motion representations positional embeddings.
Diffusion models. Semantic consistency Ren et al. 2024 Zhang et al. 2024j Jiang et al.) 2024b
et al.) 2024h Zhou et al., 2024d Sun et al. 2023b) ensures the semantic meaning including objects and relationships between objects are smooth between different frames. The biggest problem semantic drifting, which refers to the gradual shift or change in the meaning or context of visual content as vidco progressos. StableVideo (Chai et al. 2023), as a pioneer work, proposes propagate the representation to the next frame ensure consistency. Similarly, TokenFlow (Geyer et al. 2023) preserves consistency by explicitly
47

---

Published in Transactions on Machine Learning Research (MM/YYYY)
propagating diffusion features based on inter-frame correspondences using sampled tokens for video editing. GLOBER (Sun et al. 2023a) employs a. novel ad advessarial loss along with conditioning the generation on the global features extracted by video encoder for global consistency. Specifically, for face video generation, EMO Tian et al.) 2024) preserves disual/identity consistency by encoding a reference image/video/ /audio and cross-allention for video generation.
Motion consistency consists of multiple aspects, such as video smoothness, reasonable object motion, and stable camera movement, while two different frameworks, including residual-based framework (Deng et al,, 2024 Zhou et al. 2024d) and optical flow-based framework, are proposed to ensure the motion consistency of generated videos. For the residual-based methods, they usually add additional motion representation extracted from previous frames or text prompts. As a representative work, Cinemo (Ma et al., 2024b) learns to generate residual images between a previous frame and a later frame for motion control. Moreover, it also incorporates a new noise refinement with discrete cosine transformation for controlling the input noise. VidcoComposer (Wang et al.] 2023f) uses motion vectors, depth sequences, mask sequences, and sketch sequences to assist in the generation of video frames. On the other side, in addition to the image-based residual, StoryDiffusiion (Zhou et al. 2024d proposes to split the long text prompts into multiple short text prompts to improve consistency.
For the optical flow-based models, they usually learn to generate optical flow between a previous frame and a later frame to maintain consistency. LFDM (Ni et al., 2023) introduces a flow predictor during training and injects flows into the denoising procedure for motion consistency. Similarly, Motion-12V (Shi et al. 2024) consists of two stages. It first generates optical flows based on the input image and then videos are generated conditioned on the concatenation of these optical flows. For video editing, CAMEL (Zhang al. 2024b) improves consistency through a frequency-based perspective, where the video is decomposed using low- and high-pass filters. Moreover, it learns additional motion prompts for further stabilize the video.
Apart from these methods, another popular paradigm for consistency is controlling the noise used in diffusion models. PYoCo (Ge et al.)
2023) is first proposed to employ a novel correlated noise model, which consists of noise shared among all frames and an individual noise conditioned on the previous frames. Following this, a f-noise (Chang et al.) 2024a) is proposed to warp noise following the flow and preserve the spatial Gaussian property, which is able to preserve the motion and semantic consistency at the same time.
One application of consistency is long video generation, which poses high requirements for motion and semantic consistency. long video generation (Ouyang et al.) 2024c) is always achieved by ensuring a smooth transition and semantic consistency among different frames. StoryDiffusion (Zhou et al. 2024d) proposes to use a novel consistent self-attention and semnnntic space temporal prediction for object consistency. Similarly, StreamingT2V (Henschel et al., (024) employs a long-term memory block, a short-term memory block, and also a new blending approach for infinitely long video generation. The short-term block focuses consistency between adjacent frames, while the long-term block forces the model to depend on the first several frames to avoid forgetting. Later, SEINE (Chen et al.) 2024ch proposes to employ a random-mask model for generating transitions based on textual descriptions. Multiple randomly masked images can be seen as images from different scenes and they can provide an overall comprehensive angle to consistency and long video generation. On the other side, 3D scene mesh (Fridman et al.) 2023) is also used for generating videos, which can be seen as another application of consistency. Specifically, the vidoo frames and moshes are updated through several pre-trained models and the consistency between different frames is preserved through the iterated refinement of the mesh.
4.5
Longg video
Long video understanding, including retrieval, classification, and generation, has been challenging problem for a long time. Videos are always of different lengths and might be infinitely long, such that models find it hard to keep content consistent among different frames. In this part, we will introduce representative methods in long video generation.
Almost all the traditional long video generation methods are divide-and-conquer-beee methods. They usually first generate keyframes and then generate other frames conditioned on these generated keyframes. TATS Ge et al. 2022) uses multiple models (interpolation and autoregressive transformers) to ensure the
48

---

Published in Transactions on Machine Learning Research (MM/YYY)
Multimodal Scripts
Auto regr-seee Prediction
Characters
A man in black suit with
slicked slicked-back back hoir, <Face ID>
headdress
Multimodal Scripts
Multimodal Scripts
and matching necklace, Face Ibe
Elements:
Crowded event
Multimodal Modeling with VLM
Elegant Lighting
Plot:
A mon woman are doncing closely crowded event. subtle palpable sense of connection or confrontation between the the woman, suggested by their direct gaze and proximity each other amidst crowded setting
Image
Render
Render
Render
Image-te toe-tt-vviiee video Gen
Image mmage-to-wae toaee-tt-vine viveideo Gen
KeyFrame 3224
KeyFrame 3225
KeyFrame 3226
Image Renderer Training
Inference
Face Embedding
Compressor
CLIP
SDXL
Compressed Tokens
Encoder
Decoder
Description Embedding
Figure 21: The framework of MovieDreamer. The autoregressive model takes multimodal scripts as input and predicts the tokens for keyframes. These tokens are then rendered into images, forming anchor frames for extended video generation. MovieDreamer ensures long-term coherence and short term fidelity in visual storytelling with the character's identity well preserved.
transition between frames is consistent. Later, to further improve the flexibility of long video generation, Brooks (2022) generates low-resolution sequences, which are then enhanced high resolution. Besides, Video (Hong al. 2023a) further interpolates non-key frames with the help of keyframes and text
conditions with two-stago-basee framework. Another work by Zhang et al. (2033c) is built upon StyleGAN- (Skorokhodov et al. 2022) which suffers from undesired content jittering for long video generation. They further proposed a novel B-Spline-basse motion representation to ensure temporal smoothness.
Diffusion models. Different from traditional models, the solutions in diffusion models are more diverse and can be split inlo two main categories, i.e., divide and conquer (sliding windows and key keyframes), autoregressive models (including sliding windows), and consistency. For divide-and-conquer based methods (EMO), the generation of long videos is split into several stages and for cach stage, the video is generated based the previous one the input condition for refinement. NUWA-XL YYin et al.) 2023b employs hierarchical method for generating long videos. Specifically, global diffusion model is first applied to generate the keyframes across the entire time range, and then local diffusson models recursively fill in the content between nearby frames. An earlier work (Harvey et al., 2022) proposes a R.A--ttlle generation pipeline for generating adapted long video frames continued on any other videos along with novel generation scheme (attention map) for not forgetting the carly frames. Gen-L-Vidco (Wang et al. 2023bb also employs hicrarchical strategy construct long videos through set of short videos. FreeNoise (Qiu al., 2024b), as an example, employs sequence of noises for long-range correlation and performs temporal attention over them by sliding windows. Moreover, it also uses a motion injection method to condition the generation of long videos on multiple text prompts. Besides, instead of considering directly generating long videos, MAVIN (Zhang et al. 2024a) chooses to first. generate several small clips and then generate the transition between them to maintain consistency with a boundary frame guidance module and a noise sampling strategy.
On the other side, autoregressive methods are drawing more and more research attention. For example, a sliding window, which be seen as application of autoregressive methods, is used to enable short-video
49

---

Published in Transactions on Machine Learning Research MM/YYYY)
F-m Houd)
MAW Ting Naa
Placker Cardltate
Objaverse-XL
Danchen
Nonamds
Repeoveniitcn
Camera Control with Depth
araran Scale
Scale
Scene Dataset for Deep Learning- based 3D Vision
maruse pracent
Training 3D dataset
Camera conditioning
3D 3Dawwrr architecture
Figure 22: For a video diffusion model to be 3D aware, the general paradigm includes training on 3D dataset, adding camera control the model, and introducing 3D-aware model architectures.
generator to generate long videos while the hidden representation of the video is interpolated when condition- ing on different prompts. Moreover, many methods (Gao et al. 2024 Zhao et al. 2025 Xie et al. 2024a) are proposed with multimodal autoregressive models. MovieDreamer usee a multimodal autoregressive model with diffusion models to generate long videos conditioned on multimodal scripts. The multi-modal autore- gressive model is designed capture the key semantics in the multimodal conditions. (Xie al., 2024a) further shows that existing models can be easily extended to autoregressive video diffusion models by varying the noise level instead of using stable noise level.
Apart from these methods, more models are proposed to address the high computational requirements and the efficiency using diverse strategies and solutions (Diffusion Forcing). Specifically, to speed up the generation procedure, Video-Infinity (Tan et al. 2024b) proposed to distribute the generation for one long video to multiple GPUs with the proposed clip parallelism and dual-scope attention modules. is able generale videos up 2,300 frames in approximately 5 minutes with x Nvidia 6000 Ada GPUs (48G), which is 100 times faster than the current methods.
4.6 3D-aware video diffusion
The world exists in three spatial dimensions plus time, while a video represents 2D projection of this 3D world. Consequently, video diffusion models should adhere to certain 3D constraints. These models generate multiple frames simultaneously and capture inter-frame relationships through attention layers spanning dif- ferent frames. Existing rescarch, such as SORA (Li al. 2024i), demonstrates that by training on largo-scale datasets, networks can implicitly learn 3D priors embedded within the data.
While this approach highlights promising direction for learning 3D priors, there remains significant value in explicitly incorporating 3D awareness into video diffusion models. Doing could improve 3D controllability and facilitate easier and more effective learning for the model.
4.6.1 Training on 3D dataset
Video diffusion models are typically trained on in-the-wild videos that feature arbitrary camera motions and dynamic scenes, making challenging for these models to learn robust 3D priors. To enhance a model's ability for 3D generalization and multi-view consistency, one straightforward approach is to fine-tune the video diffusion model using datasets that explicitly capture 3D characteristics.
50

---

Published in Transactions on Machine Learning Research (MM/YYYY)
RealEstete1OK 10,000
DifUissooeD 19,000
WILLRGBD 20,000
DL3DV-1OK 7,000
COLD 34,000
Objeverse 400,000
OpenVid 38,000
MVimgNeet 219,000
Intern 355 $55,000
Figure 23: A visualization of the 3D datasets used by Cavia (Xu et al. 2024a).
SV3D (Blattmann et al. 2023a fine-tunes a pre-trained stable video diffusion model on orbiting videos of 3D synthotic objects from the GSO (Downs et al. 2022) and OmniObject3D (Wu et al.] (023d) datasets. These orbiting videos include both static orbits, where the camera circles an object at regularly spaced azimuths with a fixed elevation, and dynamic orbits, where the camera trajectory irregular. Similarly, V3D (Chen al. 2024g) fine-tunes on the Objaverse dataset, leveraging 360-degree orbiting videos, along with proprictary datasets featuring similar setups (Junlin Han 2024 Melas-Kyriazi et al. 2024).
Generative Camera Dolly (Van Hoorick et al. 2024) introduces two synthetic datasets, Kubric-4D and ParallelDomain-41) which provide multi-view videos of dynamic scenes. By fine-tuning a Stable Video Diffusion (SVD) model solely on these synthetic datasets, they achieve zero-shot generalization to real-world novel view synthesis lasks.
CamCo (Xu et al., 2024b) fine-tunes video diffusion model on real-world camera fly-through data from the WebVid101 dataset. (Bain et al.) 2021), using estimated camera poses. This enables the model to generate scene-level videos with precise camera controls. Similarly, CVD (Kuang et al., 2024h combines RealEstaaee0KK [Zhou et al. 2018bb and WebVid10M (Bain et al. 2021 to train camera-controllable vidco diffusion model.
Diffusion4D LIANG et al. 2024) exlends video diffusion into the realm of 4D generation by fine-tuning on orbiting videos where both the camera and object are in motion, enabling models to capture spatiotemporal dynamics more offectively.
generation. They train their model large-scale dataset of approximately 0.4 million videos annotated with camera poses. These
Cavia (u et al. 2024a) focuses camera-condiitooned image-to-multi-viewww-e
videos are sourced from diverse collection of datasets, including Wild-RGBD (Xia et al. 2024). MVImgNet
Yuu et al. 2023c RealEstate10K (Zhou et al. 2018b), CO3D (Reizenstein et al. 2021 DL3DV-10K (Ling
ct al. 2024b), Objaverse (Deitke et al. 2022 2023), Diffusion4D (LIANG et al. 2024), OpenVid Nan et al.
2024), and InternVid (Chen et al. 2023b). A visualization of the dataset collections used in Cavia can
seen in Fig. 23
HumanVid Wang al. 2024h introduces human-centric dataset enriched with camera annotations,
enabling precise modeling of human-centric video generation.
KFC-W (Chou et al. 2024) introduces an auxiliary task of multi-view inpainting for training video models. This approach enables the model to learn 3D priors from large-scale, unposed internet photos.
CineMaster (Wang al.) 2025a proposes a customized data labeling pipeline that leverages existing depth prediction YYagg et al.] 2024e f)), instance segmentation (Ravi et al.) (024) and scene reconstruction (Zhang
et al. 2024d) models generate labeled data. This enables a 3D-aware model to train on large-scale, in-the-wil wild video data.
4.6.2 Architecture for 3D diffusion models
In addition to data-driven approaches for learning 3D priors, another promising strategy involves designing 3D-specific model architectures that incorporate inductive biases for generating view consistent images. Traditional U-Net based video diffusion models typically capture inter-frame relationships using temporal
51

---

Published in Transactions on Machine Learning Research (MM/YYY)
attention layers, which are implemented through per-pixel attention along the temporal dimension. While this enables information flow between frames at the same spatial positions, it is suboptimal for achieving 3D awareness.
Recently, multi-view diffusion models [Shi et al. 2023d
Tang et al. 2023 Yang et al. 2024b Liu et al.
2024b Hu et al.] (0244a] Kant al.
2024 Yang al.
2024d Oh et al. 2023 Tang et al. 2024) have
gained significant attention for static 3D scene generation. By employing cross- cross-frame attention, these models effectively enhance the spatial relationships necessary for 3D consistency. For static 3D scene generation,
epipolar-constraaned attention (Xu et al. 2024b; Huang al. 2023c Zheng et al. 2024a further enforces
strict adherence to 3D constraints, ensuring greater structural accuracy.
This concept has also been extended to multi-view video diffusion models (Li el al.
2024a Xie el al. 2024b;
Xu et al. 2024a). By integrating both cross-frame and cross-view attention mechanisms, video models are fine-tuned to support 4D gencration, enabling them to produce spatiotemporally consistent vidcos that adhere to 3D principles.
In Dike structures (Gao=et al. 2024 Chou et al. 2024 Liang et al. 2024b Gu et al. 2025 Wang et al, 2025a), cross-frame attention is inherently achieved as tokens for different temporal and spatial locations are jointly mixed through transformer, allowing the model to learn 3D priors directly from data.
4.6.3 Camera conditioning
Cameras serve a bridge between the 3D world and 2D images. Given 3D scene, intrinsic and extrinsic camera parameters determine how the 3D scene is projected into a 2D image. Adapting video diffusion model to be 3D-aware involves conditioning the model with camera parameters, allowing users to control the generated views. Typically, fine-tuning is required after introducing this additional conditioning, as discussod in Soc. 4.6.1
Several commonly used approaches exist. for conditioning video diffusion models with camera. parameters:
MLP-based embedding. A straightforward method is to use a fully connected multi-layer pereep- Iron (MLP) to embed camera parameters. These parameters are often represented as elevat and azimuth angles, as in SV3D (Voleti et al. 2024 Yang et al. 2024ch, panning and zooming movement (Yang et al. 2024hh, or as a relative camera extrinsic matrix (Van Hoorick et al., 2024 Wang et al., 2024k 2025a). This embedding serves as a global condition, typically introduced into the diffusion model via cross-attention layers.
Ray map. A camera defines a mapping from each pixel location in image space a ray in 3D space. This mapping can be represented as "ray map," which stores the ray origin and direction for each pixel location instead of RGB color. The ray map forms spatially aligned representation with six channels, matching the resolution of an image. This method is particularly suited for scene-level video diffusion models requiring pixel-aligned conditioning (Gaoset al. 2024 Watson et al., 2024
Wu et al. 2024el.
Plücker coordinates. Plücker coordinates are an alternative to a ray map, parameterizing a ray as (o x d, o), where o is the ray origin, d the normalized ray direction, and denotes the cross product. This representation provides a uniform way to encode rays, making it a suitable positional embedding for video diffusson models (Xu et al. 2024b
Bahmani et. al. 2024d He al. 2024a
Kuang et al. 2024 Yao et al. 2024 Wang et al.
2024h Zheng et al. 202 Liang et al, 2024b).
Point-based. Camera conditioning can also be achieved implicitly by motions of 2D or 3D points.
For instance, -2VControl-Camera (Feng et al. 2024b) employs point trajectories in the camera coordinate system as control signals, enabling precisc control over the generated views. Similarly, Motion Prompling (Geng el al. 2024a) converts camera poses into image-space point trajectories. This is achieved by projecting a 3D point cloud, estimated using a monocular depth estimator, onto 2D points image space. Diffusion as Shader (Gu et al.] (2025) leverages a tracking vidco of dense 3D points as control signals train a video DiT model, enabling detailed control over both camera and motion controls.
52

---

Published in Transactions on Machine Learning Research (M//YYYY)
4.6.4 Inference-tine time tricks
Interestingly, there are a few efforts being made to explore whether it is possible to directly use a pre-trained video diffusion model for 3D-aware video generation without any fine-tuning. ViVid-1-to-3 (Kwak et al. 2024h combines a video diffusion model with Zero-1-to-3 XL to jointly denoise multi-view videos, effectively leveraging the 3D priors from Zero-I-to-3 XL alongside the temporal consistency inherent in video diffusion models. NVS-Solver (You et al. 2024 guides the sampling process of pre-trained Stable Video Diffusion (SVD) model by iteratively modulating the score function with the given scene priors represented derived from warped imput views, effectively achieving novel view synthesis from sparse input views. Cam'rool (Hou ct al. 2024) directs video generation by employing DDIM inversion noise derived from point cloud renderings, enabling 3D-aware video synthesis without the need for model fine-tuning.
5 Benefits to other domains
5.1
Video representation learning
Video representation learning (Ravanbakhsh et al. 2024), foundational task in video understanding, it provides downstream tasks with powerful backbones and useful representations. Early methods (Sco ett al. 2020; Wu et al. 2022a) primarily relied on 2D Convolutional Neural Networks (CNNs) (He et al. Krizhevsky et al. 2012) to extract spatial features from individual frames. These methods treated video data as sequences of static images, which limited their ability to capture temporal dynamics. To address this, researchers began incorporating Rocurrent Neural Networks (RNNs) (Hopfield] 1982) and Long Short-Term Memory (LSTM) (Hochreiter & Schmidluber 1997) networks to model temporal dependencies across frames. However, these approaches (Liu et al.) 2016 2018) often struggled with capturing complex, long-range temporal relationships and integrating spatial-tempon information efficiently.
2016
The development of 3D CNNs marked a significant advancement, enabling simultaneous modeling of spatial and temporal features by applying convolutions across both space and time dimensions. This shift allowed for more holistic video representation (Tran et al. 2015), leading to better performance in tasks such as action recognition (Yang et al.] 2019 Kong et al.] (017)) and video captioning (Yan et al.) 2022). More recently, attention mechanisms and Transformers (Vaswani et al., 2017 have been introduced to further enhance video
reprosentation learning (Zhao et al. 2022 Gabeur al. 2020 Cao et al. 2022 Botach et al., 2022 Wu et al. 2022b). These models, especially those utilizing sel sell-attention, can capture long-range dependencies and complex interactions between frames, leading to more robust and generalizable video representations. The evolution of video representation learning has been characterized by a gradual shift from static image processing to dynamic, sequence-based modeling, reflecting the unique challenges of video data.
On the other side, video-language learning (Radford et al. 2021 Li et al. 2022) has been boosted by the integration of advanced models (Vaswani et al.J 2017; He al. 2016) and the increasing complexity of tasks that involve understanding both visual and textual information. Early attempts at video representation learning (Lei et al. 2021 ) relied heavily on separate processing of video and language inputs, where modality- dependent models were used to extract features from video frames and textual data independently. However, this approach often fails to capture the intricate relationships between the two modalities. The introduction of Transformer-bast models (Ye et al., 2023) marked a significant advancement, enabling more effective modeling of cross-modal interactions through self-attention mochanisms. These models can jointly process video and language inputs (Zhao et al.] 2023b), leading to improved performance on tasks, e.g., video captioning Li et al. 2023), video retrieval (Wang et al. 2024b), and video question answering (Pan et al. 2023).
As shown by previous works in image generation (Chi et al. 2024). a pretrained video generative model might be able to serve as prior, loss, or regularize in learning video representations. Recently, a pioneer work (Yhh et al. 20244g employs image and vidco generative models for hateful visual content recognition. It queries well-trained SOTA image and video generative models to automatically scale up training data pairs for hateful visual content. recognition, achieving superior performance.
53

---

Published in Transactions on Machine Learning Research (MM/YYY)
5.2 Video retrieval
Video retrieval, mainly video-text retrieval (VTR), which involves cross-modal alignment and abstract un- derstanding of temporal images (videos), has been popular and fundamental task of languago-grounding
problems (Wang et al.
2020a by 2021
Yu et al. 2023b)
Most existing conventional video-text retrieval frame-
works Yu et al. 2017 Dong et, al.
2019 Zhu & Yang 2020 Miech et. al. 2020 Gabeur et al. 2020 Dzabraev
et al. 2021 Croitoru et al.) 2021 Liu et al.) 2025b
Wang et al. 2025ch focus on learning powerful repre-
scntations for vidco and text and extracting modality-dependdar reprosentations with modality-dependee
encoders. Inspired by the success of self-supervised pretraining methods (Devlin el al. 2019 Radlord el al.
2019
Brown et al. (020) and vision-laguuage pretraining (Li et al.) 2020 Gan et al. 2020 [Singh et al.
2022
on large-scale unlabeled cross-modal data, recent works (Lei et al. 2021 Cheng et al. 2021 Gao
et al.
2021
Ma et al.]
2022 Park et al.) 2022 Wang et al.) 2022a Zhao et al.) 2022] Gorti et al. 2022 Luo
et al.
2022
Liu et al.
2022d
Cao et al.] 2022) have attempted to pretrain or fine-tune video-text retrieval
models
[Gorti et al.
2022: Lei et al. )021).
With the rapid progress in video generation, leveraging the advancement in video generation (Chen et al,
2024c
Muaz et al. 2023) to advance video retrieval by reranking (Zhou et al. 2024e Dutta et al., 2021] or other methods becomes more and more promising. On the other side, retrieval augmented generation could
also help faithful video generation (Arefeen el al. 2024
Rosa 2024
Tevissen al. 2024 He el al. 2023a
5.3 Video QA and captioning
Video question answering (VideoQA) (Zhong et al. 2022; Wu et al. 2024a) is a task to predict the correct answer given a question and a video. Early work (Zeng et al. 2017 Xu el al. 2017 Zhao et al.] 2017 Jang et al.) 2017)) focused on simpler factoid questions with the introduction of datasets like MSVD-QA and TGIF-QA. -A. Techniques during this period primarily relied attention mechanisms and memory networks to handle the spatial and temporal complexities of video data. Later, more challenging dalasets emerged, such as NEXT-QA, which emphasized inference and reasoning over merely recognizing visual facts. This shift necessitated the development of more advanced models, including Graph Neural Networks (GNNs) for relational reasoning and Transformers for their powerful sequence modeling capabilities, particularly when enhanced with cross-modal pre-training strategies.
Recently, the focus has increasingly shifted towards models that can handle multi-modal inputs and perform more complex reasoning tasks. Knowledge-bard based VideoQA datasets and techniques have also attracted more and more attention, pushing the boundaries of what VideoQA systems can achieve. Modular networks and neural-symbolic approaches are being explored to improve flexibilityy and interprotability, addressing the need for models that can reason about causal and temporal relationships in video content. On the other side, understanding long videos could be a big challenge to current models as long videos require a more comprehensive understanding among different frames and a long term memory.
In the future, pre-trained models originally designed for video generation can be used to enhance VideoQA by predicting future frames generating hypothetical scenarios to answer complex questions. Also, text- to-video generation methods can be used for generating diverse VideoQA benchmarks
5.4 3D and 4D generation
One emerging area of rescarch focuses on gencrating 3D and 4D scene representations, such as textured meshes, Neural Radiance Fields (NeRF) (Mildenhall et al.] 2021), and Gaussian Splatting (GS) (Kerbl et al. 2023). These representations hold significant potential for applications in 3D content creation and immersive virtual reality. However, due to the heterogeneous nature of 3D and 4D representations, and the scarcity of large-scale 3D and 4D datasets, it is difficult to design feed-forward neural networks capable of directly generating such representations.
To address these challenges, a common approach involves distilling knowledge from vidco generative models to reconstruct multiple viewpoints of a 3D scene. This strategy leverages advancements in video generation to synthesize complex 3D structures effectively. Recent studies have explored the potential of video diffusion
54

---

Published in Transactions on Machine Learning Research (MM/ MM/YYYY)
Vidco diffusion model 3D prior
Direct supervision
Video diffusion model motion prior
SDS loss family
Feed forward reconstruction
Multi-view video diffiusion
3D/DD scene representations (NeRF, Gaussian Splatting, Mosh,
Figure 24: A common paradigm for 3D and 4D generation is by distilling 3D or motion knowledge from video generation models to 3D and 4D scene representation.
models in facilitating 3D and 4D generation, offering promising avenues for bridging the gap between video and high-dimensional scene representation.
5.4.1 Video diffusion for 3D generation
When generating 3D scenes from 2D models using multiple views, major challenge is to maintain view consistency, which requires capturing inter-frame correlations across the generated views. Video diffusion modols provide a general solution to inter-frame correlations. By treating the temporal dimension as different viewpoints, pre-trained video diffusson models can be fine-tuned into multi-view diffuusion models. Several methods have been proposed to use a 2D video generation model for 3D scene generation:
V3D CCenn et al. 2024g) leverages the predictions of fine-tuned video diffusion models direct supervision to generate 3D Gaussian splatting or meshes. SV3D (Voleti et al. 2024) employs the outputs of a fine-tuned Stable Video Diffusion model to supervise Neural Radiance Field (NeRF) in the coarse stage, then uses SDS loss to refine high-frequee figh-freeuency details of DMTet (Shen et al.) 2021) initialized from the coarse stage. IM-3D MyrasiKyrrazii et al. 2024) adopts an instructive NcRF-to-NeRE (Haque et al.] 2023) pipeline for dist ilstiolation This involves iteralively adding noise to rendered results and using a diffusion model denoise the images, which then serve as targets for updating the 3D representation. VFusion3D (Junlin Han 2024) demonstrates the potential of fine-tuned video diffusion models as data generators for creating large-scale 3D datasets. These datasets can be used to train feed-forward 3D generation models. Hi3D (Yang et al, 2024cb follows coarse-to- fine 3D generation pipeline. It first generates low-resolution orbiting videos using a text-to-video model, then refines these into high-resolution orbiting videos with video-to-video model. Both models are fine-tuned from pre-trained video diffusion models with camera condition extensions. The high-resolution orbiting videos are subsequently used to reconstruct 3D meshes. ReconX (Liu et al.) 2024a) fino-tunes vidco diffusion models for sparsc-vicw novel view synthesis, cnabling efficient reconstruction of 3D scenes from limited input views. With recent advancements in large reconstruction models (LRMs) (Hong et al. 2023b Zhang et al. 2024e), Wonderland {Liang et al. 2024b) demonstrates that by training feed-forward 3D reconstruction model from the latent space of video generative model, remarkable scene generation results can be achieved. Some examples are shown in Fig. 25
5.4.2 Video diffusion for 4D generation
The incorporation of the additional time dimension in 4D generation significantly increases the complexity of developing and training leed-forward networks. Video diffusion models, functioning motion generators, offer promising solution by enabling the extraction of scene dynamics and motion information for 4D generation.
Several approaches leverage video diffusion models Lo Irain dynamic Neural Radiance Fields (NeRFs) or other 4D representations: MAV3D (Sineer et al. 2023), 4D-fy (Bahmani et al. 2024c), Dream-in-44] ZZeng ct al. 2024b), Align-Your You---ooorGGaannnn (Ling et al. 2024ab. and Animate124 (Zhao al. 2023c) apply Score Distillation Sampling (SDS) loss along the temporal axis to train dynamic NeRFs using video diffusion models. These methods enable the models to learn motion dynamics directly from video diffusion. Vidu4D
55

---

Published in Transactions on Machine Learning Research (MM/YYX)
Space & Novel Views
Video latent space
Feed- Reed-forwand 3D Recconssnnn cocticttooo
Figure 25: By training feed-forward reconstruction model from video generative model, Wonderland
Liang et al. (2024b) achieves remarkable 3D scene generation results.
(Wang et al. 2024g and STAG4D Zeng et al. 2024b) propose robust reconstruction pipelines to reconstruct dynamic Gaussian Surfels or Gaussian splatting with monocular videos generated by video diffusion models. PLA4D (Miao et al., 2024a) animates static 3D Gaussian splatting generated by 3D diffusion models, using video diffusion predictions combined with a motion aligmment model. TC4D (Bahmani et al., 2024a) uses SDS loss to supervise dynamic NeRFs, incorporating srajectory-conditionn global and local motion components, facilitating the generation of large-scale motions.
video model that disentangles
Dynamic motion knowledge models: EG4D (Sun et al. 2024a) use combination of SVD and SV3D models to directly generate multi- view videos, which are then used to supervise 4D Gaussian splatting. 4Real (Yu et al.) 2024a) generates 4D videos by using video diffusion model to supervise fxed-viewpoint video and freeze-time video with SDS loss. The use of deformable 3D GS ensures the temporal and view consistency. DreamGaussiandD (Ren et al., 2023) employs a training-free video-to-video pipeline that uses pre-trained video diffusion models refine the dynamic textures of generated mesh sequences.
also be combined with spatial 3D knowledge from multi-view diffusion
Several approaches aim to fine-tune video diffusion models into 4D-aware diffusion models, enabling the unified learning of spatial and temporal information. Such a direct 4D generator could be further distilled into a 4D scene representation for better temporal and view consistency or real-time rendering.
Diffusion4D [LING al., 2024) fine-tunes video diffusion model conditioned on camera pose and times- tamps to generate orbiting views of dynamic scenes, which are then used to optimize dynamic Gaussian splatting. 4Diffusion (Zhang et. al.) 2024c) and Animate3D (Jiang al. (224aa] train multi-view video diffusion models capable of simultaneously generating videos from different viewpoints, and then train dy- namic NeRFs with SDS loss. VideoMV (Zuo et al.) 2024) trains multi-view video diffusion model and a foed-forward Gaussian splatting reconstruction model that directly predicts GS from sparse views generated by the multi-view video diffusion model. Vivid-ZOO [Li et al. 2024a) reuses pre-trained 2D video and multi-view diffusion models to bridge the gap toward 4D generation.
SV4D Xie et al. 2024b) and CAT4D (wu al.) 2024e) train multi-view video diffusion models conditioned monocular reference videos, enabling video-to-4D tasks, shown in Fig. 26
6 Ethical Considerations
While video generation has been largely advanced by high-quality datasets
(LAION
2023
Miech al.
2019
Yang al.] 2024a Chen et al. 2024d), advanced model architectures (Lee et al.
2022
Seo et al. 2022
56

---

Published in Transactions on Machine Learning Research (MM/ (M/YYYY)
"A cat kneading dough"
Sparse dynamic capture 4D
Real video 4D
Text-to-vidoo-to-4D
Figure 26: CAT4D (Wu et al. 2024c trains a multi-view video diffusion model conditioned on a monocular video, which enables 4D view synthesis from casually captured monocular videos.
and training techniques (Wan et al., 2025 teamMeta 2024 they (Wan et al.
2025 Zheng et al., 2024c
teamMeta 2024 simultancously raise critical ethical concerns that demand careful consideration.
Deepfakes and Misinformation. Creating realistic fake videos of real people without consent raises serious
concerns about non-consensual intimate imagery (Zhao et al. 2021 Wu et al. 2022c Rössler et al. 2018 Liu et al. 2025a political manipulation, and spreading false information. This technology can undermine trust in authentic media and enable harassment campaigns. While image-based deepfake detection (Dang
et al. 2020 has been successful, video deepfake video detection (Kundu et al.) remains a serious challenge
to this area.
Consent and Privacy. Using someone's likeness, voice, personal content to generate videos without explicit permission violates their privacy rights and autonomy (Rosenblat et al. 2025 Zhang al. 2024f Chen et al.] 2024a). This extends to using training data that may contain copyrighted or private material without proper authorization.
Bias and Reprosentation. AI models trained on biased datasets can perpetuate harmful stereotypes Gal- legos el al.j 2024 Chen et al.J 2024a), underrepresent certain groups, or generale content. that reinforces discriminatory patterns. The bias in generated videos (Nadeem al. 2025) could amplify existing social incqualities and marginalize vulnerable communiiees.
Legal and Regulatory Challenges. Current laws (Goanta et al. 2023 struggle to address A Al-generated content, creating uncertainty around liability, intellectual property rights (Zhang et al. 2024F Chen et al. 2024a), and regulatory compliance. This legal ambiguity can enable harmful uses while hindering beneficial applications.
Transparcncy and Disclosure. There's an ongoing debate about when and how Al-generated content should be labeled. Clear disclosure (Chang al.) 20244b Yoo al.] 2024 Chen al. 2024a) helps viewers make informed decisions about the content they consume, but enforcement remains challenging. While applying watermarks on generated video (Hu et al.) 2024b) could be potential solution, more and more approaches should be explored to emphasize these considerations.
Quality Control and Safety. Video generation systems may produce harmful, offensive, or dangerous
content if not properly constrained. Ensuring robust content filtering (Poppi et al. 2024 Yoon et al.) 2025 Liang et al. 2025b] Liu et al. (2025a) and safety measures is crucial to prevent the creation of videos that could cause harm violate policies.
57

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Computational Resources and Environmental Impact. Training and running video generation models requires significant computational power, leading to substantial energy consumption and environmental costs Nagg et al. 2024) that should be considered in the development and deployment of these technologies.
Despite that, researchers have also proposed benchmark datasets (Miao et al. 2024b Pang et al. 2024 Chen
et al. 2025b Yeh al. 2024b Wang et al. 2025b) to access the safety of text-to-video generative models.
7
Conclusion and outlook
This survey presented a comprchensive analysis of diffusion-based vidco generation, tracing its evolution from traditional GAN-based approaches to current slate-of-the-art methodologies. Through our system- atic examination, we have highlighted how diffusion models have addressed many historical challenges video generation while introducing new opportunities and considerations for future research. The field of diffusion-based video generation has demonstrated remarkable progress in generating temporally consistent and visually compelling videos.
Our analysis reveals several key insights: First, the transition from GAN-based methods to diffusson models has significantly improved training stability and output quality, though computational efficiency remains a crucial challenge. Second, the integration of diffusion models with various architectural innovations has enhanced temporal coherence and motion consistency, particularly in long video generation. Third, the application of video generation methods has broadened considerably, from low-level vision tasks to complex video understanding problems.
Despite these advances, several critical challenges remain:
Computational Efficiency: Future rescarch should focus on reducing the computational demands of both training and inference, particularly for real-time applications.
Long- Video Generation: While progress has been made in handling long video generation, main- taining consistency and coherence in long videos remains challenging.
Adhering Physical Rules: Ensuring generated videos adhere to physical laws and maintain realistic object dynamics requires continued attention.
Ethical Considerations: As these technologies become more powerful, developing robust frameworks for addressing bias and preventing misuse becomes increasingly important.
Corresponding to these challenges, anticipate several promising research directions:
Efficient Architectures and Training Strategies. The computational demands of video gener- ation models present significant barrier widespread adoption and real-time applications (Bolya)
& HHofiman 2023 Li al. 2024j Kim et al., 2023b] Sun al. 2025 Kahatapitiya et al., 2024)
Future research should focus on developing token-level optimizations such as adaptive token selec- lion and learned pruning mechanisms thtt can identify and eliminate redundant spatial-temporal information while preserving critical motion dynamics. Additionally, hierarchical generation frame- works that decompose video synthesis into coarse-to-fine stages can leverage temporal coherence to reduce computational overhead. Novel training paradigms including progressive training (Wan jt al. 2025), curriculum learning for temporal sequences, and specialized attention mechanisms for video data offer promising avenues for achieving better quality--fficiency trade-offs.
Integration of Physical Priors and Domain Knowledge. Current vidco generation models often produce visually plausible but physically implausible results (Kang et. al. 2025a), limiting their applicability domains requiring realistic motion and object interactions. Future research should systematically incorporate physical understanding through physics-informed architectures that explicitly model laws such as conservation of momentum, gravity, and collision dynamics (Bansal
et al. 2024
Kang al. 2025b Bansal et al. 2025 For specialized applications such as medical video
58

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Li et al. 2024f or autonomous driving simulations (Fu et al 2024b incorporating domain-specilie knowledge through structured priors can enhance both realism and utility, This requires developing modular frameworks that can easily integrate external knowledge sources and differentiable physics simulators into the generation pipeline.
generation (Cao al. 2024
Hybrid Generative Approaches.
The integration of diffusson models with other generative frameworks (Li al. 2025b; Xie al. 2025) presents opportunities to leverage the complemen- tary strengths of different approaches while mitigating their individual limitations. Future research should explore combinations of diffusion models with GANs (Wang ct al. 2023jj for improved train- ing stability and generation speed, auloregressive models (Lee el al.) 2022 for better temporal coherence, and variational autoencoders (Pandey et al. 2022) for enhanced controllability. Addi- tionally, investigating unified frameworks (Guo et al. 2025b Zhang et al., 2025) that can seamlessly switch between different generative paradigms based on content requirements and computational constraints offers promising directions for more versatile video generation systems.
Model Safety and Ethical Considerations. As video generation technology becomes more sophisticated and accessible, ensuring model safety and addressing othical concerns (Miao et al.
2024b Pang et al. 2024 Chen et al. 2025b Yeh et al., 2024b Wang et al., 2025b Hu et al. 2024b) becomes increasingly critical, such as copyrights, data consent, privacy, and biases.
As the field continues to evolve, the relations between diffussio-based video generation and related domains, such as video understanding and 3D generation, will likely drive further innovations, The community's focus on addressing both technical and ethical challenges will be crucial in realizing the full potential of this technology while ensuring its responsible development and deployment.
References
Rameen Abdal, Or Patashnik, Ivan Skorokhodov, Willi Menapace, Aliaksandr Siarohin, Sergey Tulyakov, Daniel Cohon-Or, and Kfir Aberman. Dynamic concopts personalization from single vidcos, 2025. URL
https://arxiv
wrgg/ba////....o.... https://arriv.orr//a///00...
14844
Luca Savant Aira, Antonio Montanaro, Emanuele Aiello, Diego Valsesia, and Enrico Magli. Motioncraft: Physics-based based zero-shot video generation. arXiv preprint arXim:2405 2024.
Michael S. Albergo, Nicholas M. Boffi, and Eric Vanden-Fijnden. Stochastic interpolants: unifying frame- work for flows and diffusions. arXiv: 2303.08791 2023.
Nuha. Aldausari, Arcot Sowmya, Nadine Marcus, and Gelarch Mohammadi. Video generative adversarial networks: review. ACM Comput. Sure., 55(2):1-25, 2022.
Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXie preprint or orXiv:2304.08 2023.
Pierfrancesco Ardino, Marco De Nadai, Bruno Lepri, Elisa Ricci, and Stephane Lathuiliere. Click to move: Controlling video generation with sparse motion. In ICCV, pp. 14729-14738, 2021.
Md. Adnan Arefeen, Biplob Debnath, Md. Yusuf Sarwar Uddin, and Srimat Chakradhar. irag: An incre- mental retrieval augmented generation system for videos. CoRR, abs/2404.12300, 2024.
Dawit Mureja Argaw, Fabian Caba Heilbron, foon-Yound Lee, Markus Woodson, and In So Kweon. The anatomy of video editing: A dataset and benchmark suite for ai-assisted video editing. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, volune 13668, pp. 201 218, 2022.
Amurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid, Vivit: A video vision transformer. In ICCV, pp. 6836-6846 2021.
59

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Sherwin Bahmani, Xian Liu, Wang Yifan, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, Andrea Tagliasacchi, and David B. Lindell. Te4d: Trajectory- conditioned text-to-4d generation. In ECCV, pp. 53 72, 2024a.
Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, Andrea Tagliasacchi, David B Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera con- trol in video diffusson transformers. arxic preprint (r11::2111.1867, 2024b.
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B. Lindell. 4d-fy; Text-to-4d genera- tion using hybrid score distillation sampling. In CVPR, pp. 1996-8006, 2024c.
Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, al. Vd3d: Taming large video diffusion transformers for 3d camera control. ar preprint 20X1:2407.12781, 2024d.
Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, and Jiang Bian. Uniedit: A unified tuming-free framework for video motion and appearance editing, arXiv preprint arXiv:2402. 13185, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-VT:: A Versatile Vision-Language Model for Understanding, Localization, Text Read- ing, and Beyond, October 2023. URL http://axxxv.orrg/bbb//3333 12966 arXiv:2308.122 12966 TLDR: The Qwen-VL series is introduced, a set of large-scale vision-language models designed to perceive and understand both text and images that outperforms existing Large Vision Language Models (LVLMs).
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Videophy: Evaluating physical commonsensee for video generation, 2024. URL /ttps://arxiv arxiv. http//aiiiv....aaaaa/aaa/a atrg/bbs/2406.. 03520
Hritik Bansal, Zongyu Lin, Tianyi Xie, Zeshun Zong, Michal Yarom, Yonatan Bitton, Chenfanfu Jiang, Yizhou Sun, Kai-Wei Chang, and Aditya Grover. Vidcophy: Evaluating physical commonsense for video generation. In The Thirleenth International Conference on Learning Representations, 2025. URL https:
//openreview wttp://trtooomm//d--/0000v
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, pp. 22669-226777 2023a.
Fan Bao, Shen Nic, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. In ICML, pp. 1692 1717, 2023b.
Michael J Black and Padmanabhan Anandan. A framework for the robust estimation of optical flow. In ICCV, pp. 231-236, 1993.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXie preprint arXiv:2311.15129 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusson models. In CVPR, pp. 22563-22572 2023b.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion, 2023. URL abs/ aba/2303. 17604
org/
60

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Nicolas Bonneel, James Tompkin, Kalyan Sunkavalli, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. Blind video temporal consistency. TOG, 34(6), 2015.
Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin. End-to-end referring video object segmentation with multimodal transformers. In CVPR, pp. 4973-4985, 2022.
Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Ya Yu Liu, Alexei A. Efros, and Tero Karras. Generating long videos of dynamic scenes, 2022.
Tim Brooks, Aleksander Holynski, and Alexci A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pp. 18392 18402, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateuss Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrIPS, 2020.
Andrés Bruhm, Joachim Weickert, and Christoph Schnörr. Lucas/kanade meets horn/schunck: Combining local and global optic flow methods. Int. J. Comput. Vis., 61(3):211-28) 221,, 2005.
Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, et al. Go-with-the-flow: Motion-controllable video diffusion models using real-time warped noise. In CVPR, 2025.
Shuqiang Cao, Bairui Wang, Wei Zhang, and Lin Ma. Visual consensus modeling for video-text retrieval. In AAAI, pp. 167-175, 2022.
Xu Cao, Kaizhao Liang, Kuei-Da Liao, Tianren Gao, Wenqian Ye, Jintai Chen, Zhiguang Ding, Jianguo Cao, James M. Rehg, and Jimeng Sun. Medical video generation for disease progression simulation, 2024. URL htttp://arxiv wrrg httgg/bb//411.... 1194
Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S. Yu, and Lichao Sun. A comprehensive survey of ai-generated content (aige): A history of generative ai from gan to chalgpt, 2023.
Joao Carreira, and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
Duygu Ceylan, Chun-Har P. Huang, and Niloy J. Mitra. Pix2video: Video editing using image diffusion. In ICCV, pp. 23149 2016-2316,020 2023.
Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. Stablevideo: Text-driven consistency-aware diffusion video editing. In ICCV, pp. 23040 23050, 2023.
Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang, and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity-awee aware diffusion. In ICML, 2023a.
Pascal Chang, Jingwei Tang, Markus Gross, and Vinicius C Azevedo. How i warpod your noise: a temporally- correlated noise prior for diffusion models. In ICLR, 2024a.
Shao-Yu Chang, Twann-Tzong Chen, and Tyng-Luh Liu. Diffusionatlas: High-fidelity consistent diffusion video editing, arXiv preprint arXiv:2312.03772 2023b.
Ya-Liang Chang, Zhe Yu Liu, Kuan- Ying Lee, and Winston Hsu. Free-form video inpainting with 3d gated convolution and temporal patchgan. In ICCV, pp. 9066-90660909 9075, 2019.
61

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Yapei Chang, Kalpesh Krishna, Amir Houmansadr, John Frederick Wieting, and Mohit Iyyer. PostMark: A robust blackbox watermark for large language models. In Yaser Al-Onaizan, Mohit. Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1969-8987, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: URL httpps///claambhooogyy wrwp:t/oooooooooo//222/2./// wet.2222...lmmmmaai...m. wwn.p-main..500 506/
10.1 8653/v55 http:/........2..mm...
Hila Chefer, Shiran Zada, Roni Paiss, Ariel Ephrat, Omer Tov, Michael Rubinstein, Lior Wolf, Tali Dekel, Tomer Michaeli, and Inbar Mosseri. Still-moving: Customized video generation without customized video data. ACM Transactions on Graphics (TOG), (3(6): 11, 2024.
Chen Chen, Enhuai Liu, Daochang Liu, Mubarak Shah, and Chang Xu. Investigating memorization in video diffusion models, 2024a. URL https:://penrevvee. wttp://wwww......eoooomrdddrr/oo
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In CVPR, pp. 7310 7320, 2024b.
Jiahui Chen et al. Internvid: Learning text-to-video generation from web-scale video-text data. arXiv preprint Xr:::22303. 11123, 2023b.
Junsong Chen, YU Jincheng, GE Chongjian, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-alpha: Fast training of diffusion transformer for photorcalistic text-to-image synthesis. In ICLR, 2023c.
Long Chen and Jing Sun. Videogen: Generalized text-to-video generation. arXiv preprint XXXi::3066000022 2023.
Shoufa Chen, Mengmong Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Diffusion transformers for image and video generation. In CVPR, pp. 6441-6451, 2024c.
Ting Chen. On the importance of noise scheduling for diffusion models. arXiv preprint arXi::2301.10977 2023.
Ssai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang- Wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Cap- tioning 70m videos with multiple cross-modality teachers. In CVPR, pp. 13320 13331, 2024d.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Yuwei Fang, Kwot Sin Lee, Ivan Skorokhodov, Kfir Aberman, Jun-Yan Zhu, Ming-Hsuan Yang, and Sergey Tulyakov. Multi-subject open-sct personalization in video generation. arXiv preprint rrXi::2501006188 2025a.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In ICLR, 2024e.
Zhaorun Chen, Francesco Pinto, Minzhou Pan, and Bo Li. Safewatch: An efficient safety-policy following video guardrail model with transparent. explanations. In The Thirteenth International Conference Learning Representations, 2025b. URL
Zhikai Chen, Fuchen Long, Zhaofan Qiu, Ting Yao, Wengang Zhou, Jiebo Luo, and Tao Mei. Learning spatial adaptation and temporal coherence in diffusion models for video super-resolution. In CVPR, pp. 2024f.
Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Vidco diffusion models are effective 3d generators, 2024g.
62

---

Published in Transactions on Machine Learning Research (MM/YYY)
Jiaxin Cheng, Tianjum Xiao, and Tong He. Consistent todeeo transfer using synthetic dataset. arXiv preprint ar Xrv:22331 Ftt:2311.00211 1.00233, 2023.
Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?, May 2025. URL.tttp://axxx http:p////wwvw...aaaabba owt./ogg/////// http//abb/222..2222 21374
arXiv:2505.2133 213744 [cs].
Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss. CoRR, abs/2109.04299, 2021.
Xiaowei Chi, Rongyu Zhang, Zhengkai Jiang, Yijiang Liu, Yatian Wang, Xingqun Qi, Wenhan Luo, Peng Gao, Shanghang Zhang, Qifeng Liu, and Yike Guo. Mcchat: Empowering vim for multimodal 1lm inter- leaved text-image generalion, 2024.
Gene Chou, Kai Zhang, Sai Bi, Hao Tan, Zexiang Xu, Fujun Luan, Bharath Hariharan, and Noah Snavely. Generating 3d-consistent videos from unposed internet photos, 2024.
Ernie Chu, Shuo- Lin, and Jun-Cheng Chen. Video controlnet: Towards temporally consistent synthetic- (o-real video translation using conditional image diffussio models. ar preprinl Xiv:23300 19193, 2023.
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. Cfg++: Manifold- constrained classifier free guidance for diffusion models. arXiv preprint wxXi:2406.080770 2024.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets, arXiv preprint #xXi::197.06577, 2019.
Paul Couairon, Clément Rambour, Jean-Emmannee Haugeard, and Nicolas Thome. Videdit: Zero-shot and spatially aware text-driven video editing. TMLR, 2023.
Toana Croitoru, Simion-Vlad Bogolin, Marius Leordeanu, Hailin Jin, Andrew Zisserman, Samuel Albanie, and Yang Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In ICCV, pp. 11563-11573, 2021.
Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Fine-grained open domain image animation with motion guidance, 2023.
Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K. Jain. On the Detection of Digital Face Manipulation. In 2020 IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, June 2020. IEEE. doi: 10. 20009//xp422000222000000 leee. atg//occument/915578//
URL httpp:://eeexpplrr.
Duolikun Danier, Fan Zhang, and David Bull. Ldmvfi: Video frame interpolation with latent diffusion models. In AAAI, pp. 1472 1480, 2024.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. In NewrIPS, 2022.
Aram Davtyan and Paolo Favaro. Controllable video generation through global and local motion dynamics. In ECCV, pp. 68 84, 2022.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprinl XrXi::221.. 08051, 2022.
Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Volcti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl Von- drick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverso-xl: A universe of 10m+ 3d objects. Xix preprint or 201::23077.56663 2023.
63

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Boyang Deng, Richard Tucker, Zhengqi Li, Leonidas Guibas, Noah Snavely, and Gordon Wetzstein. Streetscapes: Large-scale consistent street view generation using autoregressive video diffusion. In STG- GRAPH, pp. 1-11, 2024.
Yitong Deng, Winnie Lin, Lingxiao Li, Dmitriy Smirnov, Ryan D Burgert, Ning Yu, Vincent Dodun, and Mohammad H. Taghavi. Infoiiterrsolltioon integral noise warping for diffusion models. In ICLR, 2025.
Yufan Deng, Ruida Wang, Yuhao Zhang, Yu-Wing Tai, and Chi-Keung Tang. Dragvideo: Interactive drag- style video editing, arXiv preprint 1tt.2282..2220 2023.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACI, pp. 4171-4186, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. In NewrIPS, pp. 8780-8794, 2021.
Jianfeng Dong, Xirong Li, Chaoxi Xu, Shouling Ji, Yuan He, Gang Yang, and Xun Wang. Dual encoding for zero-example video retrieval. In CVPR, pp. 9346 9355, 2019.
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In ICCV, pp. 2758-2766 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2020.
Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B. McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items, 2022.
Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, and Maja Pantic. Emoportraits: Emotion-enhanced multimodal one-shot head avatars. In Proceedings of the IEEE/ LEEE/CVF /CVF Conference Computer Vision and Pattern Recognition, pp. 8498 8507, 2024.
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model protraining with autoregressive blank infilling. arXiv preprint 1rXi::2003 10360, 2021.
Titir Dutta, Anurag Singh, and Soma Biswas. Styleguide: Zero-shot sketch-based image retrieval using style-guided image generation. IEEE Thans. Multim., 23:2833 224222222242 2021.
Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, and Aleksandr Petiushko. Mdmmt: Multidomain multimodal transformer for video retrieval. In CVPR Workshops, pp. 0354-3363, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pp. 12873-12883, 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighchchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In ICCV, pp. 7346 7356, 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthosis. In ICML, 2024.
Fanda Fan, Chunjie Luo, Wanling Gao, and Jianfeng Zhan. Aigcbench: Comprehensive evaluation of image- to-video content generated by ai, 2024. URL https::/aaxii: wwwaariww.w.aaaa...eaaa 2abbs//40000051111
Ruoyu Feng, Wenming Weng, Yanhui Wang, Yuhui Yuan, Jianmin Bao, Chong Luo, Zhibo Chen, and Baining Guo. Ccedit: Creative and controllable video editing via diffusion models, 2024a.
64

---

Published in Transactions on Machine Learning Research (MM/YYYY
Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, Siyu Zhou, and Qian He. [2vcontrol-camera: Precise video camera control with adjustable motion strength, 2024b.
Weixi Feng, Jiachen Li, Michael Saxon, Tsu-jui Fu, Wenhu Chen, and William Yang Wang. Te-bench: Benchmarking temporal compositionality text-to-video and image-to-video generation, June 2024c.
Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: generation, 2023.
Text-driven consistent scene
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Cheng Yang Fu, Jong-Chyi Su, William Yang Wang, and Sean Bell. Tell me what happened: Unifying text-guided video completion via multimodal masked video generation. Tn CVPR, pp. 10681-10692, 2023.
Xiao Fu, Xian Liu, Xintao Wang, Sida Peng, Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, and Dahua Lin. 3dtrajmaster: Mastering 3d trajectory for multi-entity motion in video generation. arXiv preprint Xiv: 12.4422.00 12.07779,, 07759, 2024a.
Yongjie Fu, Anmol Jain, Xuan Di, Xu Chen, and Zhaobin Mo. Drivegenvim: Real-world video generation for vision language model based autonomous driving, 2024b. URL /ttt://pp.///...ooooo//////200.0 16647
org/abs/2408/
Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, and Sherry Yang. Improving dynamic object interactions in text-to-video generation with feedback. arXie preprint arXiv:2412. 026177 2024a.
Hiroki Furuta et al. Learning dynamic object interactions in video generation through ai feedback. arXiv preprint, 2024b.
Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In ECCV, pp. 214 214-229, 2020.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint Xrx::22008.0 01618, 2022.
Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. Bias and Fairness in Large Language Models: A Survey. Computational Linguistics, 50(3):1097-1179, September 2024. doi: 10.1162/co a 00024. URL https:
//aclanthology..o //Aclaatboloryyorg/2244.....44-2
Place: Cambridge, MA Publisher: MIT Press.
Zhe Gan, Yen-Chun Chen, Linjic Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-ann-languuage representation learning. In NewrIPS, 2020.
Kaifeng Gao, Jiaxin Shi, Hanwang Zhang, Chunping Wang, and Jun Xiao. Vid-gpt: Introducing gpt-style autoregressive generation in video diffusion models, 2024.
Zijian Gao, Jingyu Liu, Sheng Chen, Dedan Chang, Hao Zhang, and Jinwei Yuan. Clip2tv: An empirical study transformer-based methods for video-text retrieval. CoRR, abs/2111.05610, 2021.
Ruiqi Gaos, Aleksander Holynskis, Philipp Henzler, Arthur Brussee, Ricardo Martin-Bruale Pratul P. Srinivasan, Jonathan T. Barron, and Ben Pooles. Cat3d: Create anything in 3d with multi-viow diffusion models. ańv 2024.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vegan and time-sensitive transformer. In ECCV, pp. 102-118, 2022.
Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: noise prior for video diffusion models. In ICCV, pp. 22930 22941, 2023.
65

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez- Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, Chen Sun, Oliver Wang, Andrew Owens, and Deqing Sun. Motion prompting; Controlling video generation with motion trajectories. arXiv preprint 2tt.::2112.220000 2024a.
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez- Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Motion prompting: Controlling video generation with motion trajectories. arXiv preprint 2rXim:2412.000 22000000 2024b.
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffuusion features for consistent video editing, arXiv preprint wrXiv:2307.10877 2023.
Catalina Goanta, Nikolaos Aletras, Ilias Chalkidis, Sofia Ranchordás, and Gerasimos Spanakis. Regulation and NLP (RegNLP): Taming large language models. In Houda Bouamor, Juan Pino, and Kalika. Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7722, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 2023.emmlp-mane m39. URL
Litong Gong, Yiran Zhu, Weijie Li, Xiaoyang Kang, Biao Wang, Tiezheng Ge, and Bo Zheng. Atomovideo: High fidelity image-to-video generation, 2024.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NewrIPS, 2014.
Satya Krishna Gorti, Noël Vouitsis, Junwei Ma, Keyvan Golestan, Maksims Volkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text-video retrieval. In CVPR, pp. 4996 5005, 2022.
Dylan Green, William Harvey, Saeid Naderiparizi, Matthew Niedoba, Yunpeng Liu, Xiaoxuan Liang, Jonathan Lavington, Ke Zhang, Vasilcios Lioutas, Setarch Dabiri, et al. Semantically consistent video inpainting with conditional diffusion models. arXiv preprint arXiv:2405.008 00251, 2024.
Xianfan Gu, Chuan Wen, Weirui Ye, Jiaming Song, and Yang Gao. Seer: Language instructed video prediction with latent diffusion models. In ICLR, 2024a.
Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Jun- hao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive semantic point correspondence. In CVPR, pp. 7621-7630, 2024b.
Zekai Gu, Rui Yan, Jiahao Lu, Peng Ti, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, and Yuan Liu. Diffusion shader: 3d-aware video diffusion for versatile video generation control, 2025.
Lanqing Guo, Yingqing Hc, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang, Qifeng Chen, et al. Make cheap scaling: A self-cascade diffusion model for higher-resolution adaptation. In ECCV, pp. 39-55, 2025a.
Qiushan Guo, Sifei Liu, Yizhou Yu, and Ping Luo. Rethinking the noise schedule of diflusion-based generative models. N/A, 2023a.
Xiefan Guo, Jinlin Liu, Miaomiao Cui, and Di Huang. 14vgen: Image as stepping stone for text-to-video generation. arXiv preprint Xrxi::2406. 02230, 2024a.
Xueyang Guo, Hongwei Hu, Chengye Song, Jiale Chen, Zilin Zhao, Yu Fu, Bowen Guan, and Zhenze Liu. Unidiffgrasp: A unified framework integrating vim reasoning and vim-guided part diffusion for open- vocabulary constrained grasping with dual arms, 2025b. URL https://arxiv: http:///aaxvivoo/aa///////////// wttp://////0..0055e5
Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, Haibin Huang, and Chongyang Ma. 12v-adapeer: A general imago-to-video adapter for diffusion models, 2023b.
66

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Mancesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning, arXiv preprint arXiu:2307.047725 2023c.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls to text-to-vidco diffusion models. In ECCV, pp. 330-348, 2024b.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorcalistic video generation with diffusion models. arXiv preprint ar 2rXiv:231.. 066622 2023.
Yoar HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Fitan Richardson, Fran Levin, Guy Shiran, Nir Zabari, Ori Gordon, Poriya Panet, Sapir Weissbuch, Victor Kulikov, Yaki Bitter- man, Zeev Melumian, and Ofrr Bibi. Ltx-video: Realtime video latent diffusion, 2024.
Ayaan Haque, Matthow Tancik, Alexci A. Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct- nerl2nerf: Editing 3d scenes with instructions. In ICCV, pp. 19683 19693, 2023.
William Harvey, Sacid Naderiparizi, Vaden Masrani, Christian Dictrich Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. In NeurIPS, 2022.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.0P) 021011, 2024a.
Jingwen He, Tianfan Xue, Dongyang Liu, Xinqi Lin, Peng Gao, Dahua Lin, Yu Qiao, Wanli Ouyang, and Ziwei Liu. Venhancer: Generative space-time enhancement for video generation. Xiv preprinl wrXiv:240.007666 2024b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016.
CVPR, pp.
Mingming He, Pascal Clausen, Ahmet Levent Taşel, Li Ma, Oliver Pilarski, Wenqi Xian, Laszlo Rikker, Xueming Yu, Ryan Burgert, Ning Yu, and Paul Debevec. Diffrelight: Diffusion-based facial performance relighting. In SIGGRAPH Asia, pp. 1-12, 2024c.
Xuanhua He, Quande Liu, Shengju Qian, Xin Wang, Tao Hu, Ke Cao, Keyu Yan, and Jie Zhang. Id-animator: Zero-shot pressiiy human video generation, 2024d.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent vidco diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:221.133221 2022.
Yingqing He, Menghan Xia, Haoxin Chen, Xiaodong Cun, Yuan Gong, Jinbo Xing, Yong Zhang, Xintao Wang, Chao Weng, Ying Shan, and Qifeng Chen. Animate-a-story: Storytelling with retrieval-angmented video generation. CoRR, abs/2307.0600 009440, 2023a.
Yingqing Hc, Shaoshu Yang, Haoxin Chen, Xiaodong Cun, Menghan Xia, Yong Zhang, Xintao Wang, Ran He, Qifeng Chen, and Ying Shan. Scalecrafter: Tuming-free higher-resolution visual generation with diffusion models. In TCLR, 2023b.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. In CVPR, pp. 961-970, 2015.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, pp. 5804 5813, 2017.
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nesslor, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NewrIPS, pp. 6626-6637, 2017.
67

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint orXtt:2207.12500 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrIPS, pp. 6840-6851 6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.00 02.000000000..00.0000000000 2022a.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 23(4))11-33 2022b.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arX4v:2204.03458, 2022c.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8):1735 1780, 1997.
Fa-Ting Hong and Dan Xu. Implicit identity representation conditioned memory compensation network for talking head video generation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 23062 23072, 2023.
Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. Depth-aware generative adversarial network for talking head video generation. In CVPR, pp. 3397 3406, 2022.
Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2024a.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-vidoo generation via transformers. In ICLR, 2023a.
Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, Peng Zhang, Debing Liu, Bin Xu, Juanzi Li, Yuxiao Dong, and Jie Tang, Cogvlm2: Visual language models for image and video understanding. arris preprint orXiv:2408. 16500, 2024b.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large ion model for single image to 3d, 2023b.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICML, pp. 13213 13232, 2023.
J J Hopfield. Neural networks and physical systems with emergent collective computational abilitics. PNAS, 79(8):2554-2555 1982.
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXie preprint arXim:2206.07000 2022.
Berthold KP Horn and Brian Schunck. Determining optical flow. Artif. Intell, 17(1-3):185 203, 1981.
Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. Training-free camera control for video generation, 2024.
Hanzhe Hu, Zhizhuo Zhou, Varun Jampani, and Shubham Tulsiani. Mvd-fusion: Single-view 3d via depth- consistent multi-view generation. In CVPR, 2024a.
Li Hu. Animate anyone: Consistent and controllable imago-to-video synthesis for character animation. CVPR, pp. 8153 8163, 2024.
68

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Miao Hu, Xianzhuo Luo, Jiawen Chen, Young Choon Lee, Yipeng Zhou, and Di Wu. Virtual reality: A survey of enabling technologies and its applications in iot. CoRR, abs/2103.06442,, 2021.
Runyi Hu, Jie Zhang, Yiming Li, Jiwei Li, Qing Guo, Han Qiu, and Tianwei Zhang. VideoShield: Regulating
Diffusion-based Video Generation Models via. Watermarking. October 2024b. URL http::://ppenrreviis
net/foruun'dd-uzz3qAyyy
Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou, A dynamic multi-scale voxel flow network for video prediction. In CVPR, pp. 6121-6131, 2023.
Zhihao Hu and Dong Xu. Videocontrolnct: A motion-guided vidco-to-vidoo translation framework by using diffusion model with controlnet. arxi preprinl ar wxXiu2230.14077. 2023.
Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text- to-videp generator with Ilm director and ldm animator. In NewrIPS, 2023a.
Jiahui Huang, Yew Ken Chia, Samson Yu, Kevin Yee, Dennis Küster, Eva Krumhuber, Dorien Herremans, and Gemma Roig. Single image video prediction with auto-regressive gans. Sensors, 22(9):3533,, 2022a.
Xinyue Huang et al. Lavie: Latent video encoding for diffusion models. arXiv preprint ar arXiv:2309.04512 2023b.
Yi Huang, Jiancheng Huang, Yifan Liu, Mingfu Yan, Jiaxi Lv, Jianzhuang Liu, Wei Xiong, He Zhang, Liangliang Cao, and Shifeng Chen. Diffusion model-based image editing: A survey. arXiv preprint arXiv:240..17225 2024a.
Yonglong Huang, Cheng Yu, Nannan Li, Fuqin Deng, Ruiquan Ge, and Changmiao Wang. Make an image move: Few-shot based video generation guided by clip. In ICPR, pp. 146-161, 2025.
Zehuan Huang, Hao Wen, Junting Dong, Yaohui Wang, Yangguang Li, Xinyuan Chen, Yan-Pei Cao, Ding Liang, Yu Qiao, Bo Dai, et al. Epidiff: Fahancing multi-view synthesis via localized epipolar-constrained diffusion. arXiv preprint Xii::2312.00622 2023c.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongshong Li. Flowformer: A transformer architecture for optical flow. arXiv preprint orXiv:2203.166 16194, 2022b.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwci Liu. Vbench: Comprchensive benchmark suite for vidco generative models. In CVPR, 2024b.
Ziqi Huang, Fan Zhang, Xiaojie Xu, Yinan He, Jiashuo Yu, Ziyue Dong, Qianli Ma, Nattapol Chanpaisit, Chenyang Si, Yuming Jiang, Yaohui Wang, Xinyuan Chen, Ying-Cong Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vbench++: Comprehensive and versatile benchmark suite for video generative models, 2024c.
Tak- Wai Hui, Xiaoou Tang, and Chen Change Loy. Liteflownet: A lightweight convolutional neural network for optical flow estimation. In CVPR, pp. 8981-8989, 2018.
Tak- Wai Hui, Xiaoou Tang, and Chen Change Loy. lightweight optical flow cnn non-reviitinng data fidelity and regularization. IEEE Trans. Pottern Anal. Mach. Intell,, 43(8):2555 2569, 2020.
Eddy Tlg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, pp. 2462-2470, 2017.
Siddhant Jain, Daniel Watson, Eric Tabellion, Ben Poole, Janne Kontkanen, et al. Video interpolation with diffusion models. In CVPR, pp. 7341-7351, 2024a.
Yash Jain, Anshul Nascry, Vibhav Vineet, and Harkirat Behl. Pockaboo: Interactive video generation via masked-dliflusion. In CVPR, pp. 8079 1789-888, 2024b.
69

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Y. Jang, Y. Song, Y. Yu, Y. Kim, and G. Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In CVPR, pp. 1359- 1367, 2017.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
Yanqin Jiang, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, and Jin Gao. Animate3d: Animating any 3d model with multi-view video diffusion. In NeurlPS, 2024a.
Yifan Jiang, Kevin Yang, et al. Videocomposer: Compositional text-to-video generation with latent diffusion models. arXiv preprint Xr3::2230. 029, 2023a.
Yuming Jiang, Shuai Yang, Tong Liang Koh, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2performer: Text-driven human video generation. In 1CCV, pp. 22690 22700, 2023b.
Yuming Jiang, Tianxing Wu,, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu. Videobooth: Diffusion-based video generation with image prompts. In CVPR, pp. 6689 6700, 2024b.
Yang Jin, Zhicheng Sun, Ningyuan Li, Kun Xu, Hao Jiang, Nan Zhuang, Quzhe Huang, Yang Song, Yadong Mu, and Zhouchen Lin. Pyramidal flow matching for efficient vidco generative modcling. arXiv preprint arXiv:24 10.4.0..000 05954, 2024.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. In NewrIPS, 2024.
Philip Torr Junlin Han, Filippos Kokkinos. Vfusion3d: Learning scalable 3d generative models from video diffusion models. arXiv preprint or orXiv:2403.1200 12034, 2024.
Kumara Kahatapitiya, Adil Karjauv, Davide Abati, Fatih Porikli, Yuki M. Asano, and Amirhossein Habibian. Object-centric diffusion for efficient video editing, 2024. URL htttps://axxiv org/ http://wwww...rraa////0..00..11. abs////0.../003/
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Tvo Daníhelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In ICML, pp. 1771 1779, 2017.
Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective, 2025a. URL https ftttps://arxv..om/ org/ abs/ Abss2411.022 02385
Bingyi Kang, Yang Yuc, Rui Lu, Zhijic Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How lar is video generation from world model: A physical law perspective, 2025b. URL httts::/
net//forummdd=22yyLXWHZZZZ
Yash Kant, Aliaksandr Siarohin, Ziyi Wu, Michael Vasilkovsky, Guocheng Qian, Jian Ren, Riza Alp Guler, Bernard Ghanem, Sergey Tulyakov, and Igor Gilitschenski. Spad: Spatially aware multi-view diffusers. In CVPR, pp. 10026-10038, 2024.
Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltope, James M Rehg. and Pinar Yanardag. Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models. In CVPR, pp. 6507 6516, 2024.
Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Saaelmmache-Shhizermae Drcamposc: Fashion image-to-video synthesis via stable diffusion. In ICCV, pp. 22623 22633, 2023.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pp. 8110-8119, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generalive models. In NewrIPS, pp. 26565-200 26577, 2022.
Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. TOG, 10, 2021.
70

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In CVPR, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühher, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. TOG, 42(4), 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In ICCV, pp. 15954 15964, 2023.
Tackyung Ki and Dongchan Min. Stylelipsync: Style-based personalized lip-sync video generation. In ICCV, pp. 22784 22793, 2023.
Gycongman Kim, Hajin Shim, Hyunsu Kim, Yunjey Choi, Junho Kim, and Eunho Yang. Diffusion vidco auloencoders: Toward temporally consistent face video editing via disentangled video encoding, In CVPR, pp. 6091-6100, 2023a.
Minchul Kim, Shangqian Gao, Yen-Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridging the gap between token pruning and token merging, 2023b. URL /ttp://aiivv.rrr/////2/222...0 01026
Pum Jun Kim, Seojun Kim, and Jaejun Yoo. Stream: Spatio-temporal evaluation and analysis metric for video generative models. In ICLR, 2023c.
Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards controllable high-quality neural simulation. In CVPR, pp. 5820 5829, 2021.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:11226614, 2013.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprinl arXim:2312. 14125, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jian- wei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, Andong Wang, Changlin Li, Duojun Huang, Fang Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Pengyn Li, Shuai Li, Weiyan Wang, Wenqing Yu, Xinchi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhentao Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Zumman Xu, Yangyu Tao, Qinglin Lu, Songtao Liu, Dax Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Caesar Zhong. Hunyuanvideo: A systematic framework for large video generative models, 2025.
Yu Kong, Zhiqiang Tao, and Yun Fu. Deep sequential context networks for action prediction. In CVPR, pp. 2862-3670, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NewrIPS, 2012.
Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A tuning- free framework for any Aideo-to-vider editing tasks. TMLR, 2024.
Zhengfei Kuang, Shengqu Cai, Hao He, Yinghao Xu, Hongsheng Li, Leonidas Guibas, and Gordon Wetzstein. Collaborative video diffusion: Consistent multi-video generation with camera control. arXiv preprint wwXiv2245.11444, 2024.
Rohit Kundu, Hao Xiong, Vishal Mohanty, Athula Balachandran, and Amit K Roy-Chowdhury. Towards Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content.
Jeong-gi Kwak, Erqum Dong, Yuhe Jin, Hansook Ko, Shweta Mahajan, and Kwang Moo Yi. Vivid-1-to-3: Novel view synthesis with video diffusion models. In CVPR, pp. 6775 6785, 2024.
71

---

Published in Transactions on Machine Learning Research
PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024.
Black Forest Labs. Flux, 2024.
LAION. Laion-coco.
http:///aaaoaaalllo/aaaooooooo/ 2023.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Loda Sari, Rashel Moritz, Mary Williamson, Vi- mal Manohar, Yossi Adi, Jay Mahadeokar, al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint arXiv:2306. 15687, 2023.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis. Advances in Newral Information Processing Systems, 14:14042-140055 2021.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, pp. 11523-11532, 2022.
Sangho Lee, Jiwan Chung, Youngjac Yu, Gunhee Kim, Thomas Breucl, Gal Chechik, and Yale Song.
Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning. ICCV, pp. 10254-10264, 2021.
Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang, Shape-aware text- driven layered video editing. In CVPR, pp. 14317-14326, 2023.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L.. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for aadeo-and-languae learning via sparse sampling. In CVPR, pp.
2021.
Wentao Lei, Jinting Wang, Fengji Ma, Guanjie Huang, and Li Liu. A comprehensive survey on human video generation: Challenges, methods, and insights, 2024a.
Wentao Lei, Jinting Wang, Fengji Ma, Guanjie Huang, and Li Liu. A comprehensive survey human video generation: Challenges, methods, and insights, 2024b.
Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai, Biao Zhang, Peter Wonka, and Bernard Ghanem. Vivid- 700: Multi-view video generation with diffusion model. In NewrIPS, 2024a.
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-ine to-iiagggeeeee generation. arXie preprint orXiv:2402.17845 2024b.
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C. H. Hoi. Align and prompt: langgag pre-training with entity prompts. In CVPR, pp. 4943-4953, 2022.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejum MA, and Chunyuan Li. LLaVA- weXT-interleave: Tackling multi-image, video, and 3d in large multimodal models. In The Thirleenth International Conference on Learning Representations, 2025a.
Hengjia Li, Haonan Qiu, Shiwei Zhang, Xiang Wang, Yujie Wei, Zekun Li, Yingya Zhang, Boxi Wu, and Deng Cai. Personalvideo: High id-fidelity video customization without dynamic and semantic degradation. arXiv preprint 1rXx::241111044 2024c.
Jiachen Li, Weixi Feng, Tsu-Jui Fu, Xinyi Wang, Sugato Basu, Wenhu Chen, and William Yang Wang. T2v- turbo: Breaking the quality bottleneck of video consistency model with mixed reward feedback. arxiv preprint orXiv:2405.1877 2024d.
Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, and William Yang Wang. T2v-turbo-v2: Enhancing video generation model post-training through data, reward, and condi- tional guidance design. arXiv preprint arXiv:2410.0555 10.06677, 2024e.
Linyuan Li, Jianing Qiu, Amujit Saha, Lin Li, Poyuan Li, Mengxian He, Ziyu Guo, and Wu Yuan. Artificial intelligence for biomedical video generation, 2024f. URL https:///rriii httpa/////aaii.o//aaa/////////// httg/abs//411.076111
72

---

Published in Transactions on Machine Learning Research (MM/YYY)
Maomao Li, Yu Li, Tianyu Yang, Yunfci Liu, Dongxu Yue, Zhihui Lin, and Dong Xu. A video is worth 256 bases: Spatial-tempora expectation-maxiiiattoo inversion for zero-shot video editing. In CVPR, pp. 7528 7537, 2024g.
Tong Li, Quanfeng Lu, Lirui Zhao, Hao Li, Xizhou Zhu, Yu Qiao, Jun Zhang, and Wenqi Shao. Unifork: Exploring modality aligument for unified multimodal understanding and generation, 2025b. URL https:
//arxiv.org/ aat///bb////5// wttpivv.rrgabb//55666 17202
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. In ICLR, 2023.
Xirui Li, Chao Ma, Xiaokang Yang, and Ming Hauan Yang. Vidtome: Video token merging for zero-shot video editing. In CVPR, pp. 7486 7495, 2024h.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, pp. 121-137, 2020.
Xuanyi Li, Daquan Zhou, Chenxu Zhang, Shaodong Wei, Qibin Hou, and Ming-Minn Cheng, Sora generates videos with stunning geometrical consistency. arXiv preprint arXiv: 2402.17403, 2024i.
Yizhuo Li, Yuying Ge, Yixiao Ge, Ping Luo, and Ying Shan. Dicode: Diffusionn-compresssd deep tokens for autoregressive video generation with language models, 2024j. URL htw.bttps///rraa..r/r/a///////// 04446
Zhengqi Li, Richard Tucker, Noah Snavely, and Alcksander Holynski. Generative image dynamics. In CVPR, 2024k.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Lūu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding, arXiv preprint (rxxi:2405.08748 20241.
Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Lim-grounded video diffusion models. In ICLR, 2024.
Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kumpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis. In CVPR, pp. 8207-8216, 2024a.
Hanwen Liang, Junli Cao, Vidit Goel, Guocheng Qian, Sergei Korolev, Demetri Terzopoulos, Konstantinos Plataniotis, Sergey Tulyakov, and Jian Ren. Wonderland: Navigating 3d scenes from single image. anXiv preprint arXiv:24 12. 12091, 2024b.
HANWEN LIANG, Yuyang Yin, Dejia Xu, hanxue liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei. Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models. NewrIPS, 2024.
Jingyun Liang, Yuchen Fan, Kai Zhang, Radu Timofte, Lue Van Gool, and Rakesh Ranjan. Movideo: Motion-aware video generation with diffusion model. In ECCV, pp. 56-74, 2025a.
Siyuan Liang, Jiayang Liu, Jiecheng Zhai, Tianmeng Fang, Rongcheng Tu, Aishan Liu, Xiaochun Cao, and Dacheng Tao. T2vshield: Model-agnostic jailbreak defense for text-to-video models, 2025b. URL
https://aaxit http//arxivv...r//b// http:o//aaab//0..15555505
Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng. Magicedit: High-fidelity and temporally coherent video editing, arXiv preprint ar arXiv:2308.1474 14749, 2023.
Gaojie Lin, Jianwen Jiang, Jiaqi Yang. Zerong Zheng, and Chao Liang, Omnihuman-1 Rethinking the scaling-up of one-stage conditioned human animation models. arXiv preprint (0:::2502000066 2025.
73

---

Published in Transactions on Machine Learning Research (MM/YYY)
Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via Ilm-guiedd planning, 2024a.
Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, et al. Stiv: Scalable text and image conditioned video generation. arXiv preprint arXi::4422.07330 2024b.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: to-A with dynamic 3d gaussians and composed diffusion modcls, 2024a.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, Xuanmao Li, Xingpeng Sun, Rohan Ashok, Aniruddha Mukherjee, Hao Kang, Xiangrui Kong, Gang Hua, Tianyi Zhang, Bedrich Benes, and Aniket Bera. D13dv-10k 10k: A large-scale scene dataset for doep lcarning-basee 3d vision. In CVPR, pp. 22160-22106 2024b.
Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. Motionclone: Training-fres motion cloning for controllable video generation. arXiv preprint arXiv:2406.0553 2024c.
Yaron Lipman, Ricky TQ Chen, Heli Com-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In iclr, 2023.
Alexander Liu, SouYoung Jin, Cheng-I I Lai, Andrew Rouditchenko, Aude Oliva, and James Glass. Cross- modal discrete representation learning. In ACL, pp. 3013 3013-305, 2022a.
Fangfu Liu, Wenqiang Sun, Hanyang Wang, Yikai Wang, Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. Reconx: Reconstruct any scene from sparse views with video diffusion model, 2024a.
Fangfu Liu, Hanyang Wang, Weiliang Chen, Haowen Sun, and Yueqi Duan. Make-your-3d: Fast and consis- tent subject-driven 3d content generation. In ECCV, pp. 389-406, 2024b.
Hanyuan Liu, Minshan Xie, Jinbo Xing, Chengze Li, and Tien-Tsin Wong. Video colorization with pre- trained text-to-image diffusion models. arXiv preprint arXiv:2306.00 01000173332 732, 2023a.
Haogeng Liu, Tao Wang, Jie Cao, Ran He, and Jianhua Tao. Boosting fast and high-quality speech synthesis with linear diffusion. arXiv preprinl urXi:22306. 230.023600,788, 2023b.
Jiayang Liu, Siyuan Liang, Shiqian Zhao, Rongcheng Tu, Wenbo Zhou, Aishan Liu, Dacheng Tao, and Siew Kei Lam. T2v-optjail: Discrete prompt optimization for text-to-video jailbreak attacks, 2025a. URL www./ogg////baeoeeeeoooooee
Jun Liu, Amir Shahroudy, Dong Xu, and Gang Wang. Spatio-temporal Istm with trust gates for 3d human action recognition. In ECCV, pp. 816-833, 2016.
Jun Liu, Gang Wang, Ling- Yu Duan, Kamila Abdiyeva, and Alex C. Kot. Skelcton-based human action recognition with global context-aware atlention Istan networks. IEEE Trans. Image Process., 27(4):1586 1599, 2018.
Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics physics-grounde grounded imago-to-vided generation. In European Conference on Computer Vision, pp. 360 378. Springer, 2024c.
Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, and Kai Yu. Anitalker: Ani- mate vivid and diverse talking faces through identity-dccouple facial motion encoding. arXio preprint arXiv:2405.03122 2024d.
Tianyi Liu, Kejun Wu, Yi Wang, Wenyang Liu, Kim-Hui Yap, and Lap-Pui Chau. Bitstream-cooruppeed video recovery: A novel benchmark dataset and method. In NewrIPS, 2023c.
Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu. Audio-driven co-speech 19-21386-2139,
gesture video generation. Advances in Neural Information Processing Systems, 35:21386 21399,, 2022b.
74

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Xingchao Liu, Chengyue Gong, et al. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR, 2022c.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In ICLR, 2023d.
Xuye Liu, Yimu Wang, and Jian Zhao. ELIOT: Zero-shot vidco-text retricval through relevanco-booste captioning and structural information extraction. In SRW of NAACL, pp. 381-391, 2025b.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: A review on background, technology, limitations, and opportunities of large vision models. Xiv preprinl (rXiv2242..11177 2024e.
Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine-gras fine-grained evaluation of open-domain text-to-video generation. In NewrIPS, 2023e.
Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, and Qin Jin. Ts2-net: Token shift and selection transformer for text-video retrieval. In ECCV, pp. 319-335, 2022d.
Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, and Wanli Ouyang. Unidream: Unifying diffusion priors for relightable text-to-3d generation. In ECCV, pp. 74 91, 2024f.
Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General- purpose video diffusion transformers via mask modeling. In ICLR, 2024a.
Shilin Lu et al. Vine: Video editing with generative feedback priors arxiv preprint, 2024b.
Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.1223 12376, 2024c.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pp. 11461-11471, 2022.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end vidco clip retricval and captioning. Neurocomputing, 508:203-304, 2022.
Jiaxi Lv, Yi Huang, Mingfu Yan, Jiancheng Huang, Jianzhuang Liu, Yifan Liu, Yafoi Wen, Xiaoxin Chen, and Shifeng Chen. Gpt4motion: Scripting physical motions in text-to-video generation via blender-oriented gpt planning. In CVPR, pp. 1430- 1440, 2024.
Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, Yu Zhou, Deshan Sun, Deyu Zhou, Jian Zhou, Kaijun Tan, Kang An, Mei Chen, Wei Ji, Qiling Wu, Wen Sun, Xin Han, Yanan Wei, Zheng Ge, Aojie Li, Bin Wang, Bizhu Huang, Bo Wang, Brian Li, Changxing Miao, Chen Xu, Chenfei Wu, Chenguang Yu, Dapeng Shi, Dingyuan Hu, Enle Liu, Gang Yu, Ge Yang, Guanzhe Huang, Gulin Yan, Haiyang Feng, Hao Nie, Haonan Jia, Hanpeng Hu, Hanqi Chen, Haolong Yan, Heng Wang, Hongcheng Guo, Huilin Xiong, Huixin Xiong, Jiahao Gong, Jianchang Wu, Jiaoren Wu, Jie Wu, Jie Yang, Jiashuai Liu, Jiashuo Li, Jingyang Zhang, Junjing Guo, Junzhe Lin, Kaixiang Li, Lei Liu, Lei Xia, Liang Zhao, Liguo Tan, Liwen Huang, Liying Shi, Ming Li, Mingliang Li, Muhua Cheng, Na Wang, Qiaohui Chen, Qinglin He, Qiuyan Liang, Quan Sun, Ran Sun, Rui Wang, Shaoliang Pang, Shiliang Yang, Sitong Liu, Siqi Liu, Shuli Gao, Tiancheng Cao, Tianyu Wang, Weipeng Ming, Wenqing Hc, Xu Zhao, Xuclin Zhang, Xianfang Zeng, Xiaojia Liu, Xuan Yang, Yagi Dai, Yanbo Yu, Yang Li, Yineng Deng, Yingming Wang, Yilei Wang, Yuanwei Lu, Yu Chen, Yu Luo, Yuchu Luo, Yuhe Yin, Yuheng Feng, Yuxiang Yang, Zecheng Tang, Zekai Zhang, Zidong Yang, Binxing Jiao, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Heung- Young Shum, and Daxin Jiang. Step-video-t2r technical report: The practice, challenges, and future of video foundation model, 2025.
75

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden and Saining Xie. Sit: Exploring flow and diffusson--ased generative models with scalable interpolant transformers. arxiv preprint Xrx:22240 08740, 2024a.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Cinemo: Consistent and controllable image animation with motion diffusion models, 2024b.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwci Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latle: Latent diffusion transformer for video generation. arXiv preprint arXim:2401.0080 00000, 2024c.
Yifeng Ma, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yingya Zhang, and Zhidong Deng. Dreamtalk: When expressive talking head generation meets diffusion probabilistic models. arXiu preprint or arXhu:2312.09100 2023.
Yiwei Ma, Guohai Xu, Xiaoshnai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to-end multi- grained contrastive learning for video-text retrieval. In ACM MM, pp. 638 647, 2022.
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: Pose-guided lext-to-video generation using pose-free videos. In AAAT, pp. 4117-4125, 2024d.
Ze Ma, Daquan Zhou, Chun-Hsiao Yeh, Xue-She Wang, Xiuyu Li, Huanrui Yang, Zhen Dong, Kurt Keutzer, and Jiashi Feng. Magic-me: Identity-specific video customized diffusion. arXiu preprint arXiv:2402.0000 093668 2024e.
Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, Natalia Neverova, Andrea Vedaldi, Oran Gafni, and Filippos Kokkinos. Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation, 2024.
Andrew Melnik, Michal Ljubljanac, Cong Lu, Qi Yan, Weiming Ren, and Helge Ritter, Video diffusion models: A survey, 2024.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Alcksei Stoliar, Elisa Ricci, Jian Ren, et al. Snap video: Scaled spatiotemporal transformers for texi-lo-video synthesis. In CVPR, pp. 7038 7048, 2024.
Qiaowei Miao, JinSheng Quan, Kehan Li, and Yawei Luo. Pla4d: Pixel-level alignments for text-to-4d gaussian splatting, 2024a.
Yibo Miao, Yifan Zhu, Yinpeng Dong, Lijia Yu, Jun Zhu, and Xiao-Shan Gao. T2vsafetybench: Evaluating the safety of text-to-vidoo generative models, 2024b. URL ptctps://arxiv. wwwp/rgg////b/////// 05965
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pp.
2019.
Antoine Miech, Jean- Baptittt Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End- to-end learning of visual representations from uncurated instructional videos. In ECCV, pp. 9876-9886,
2020.
Ben Mildenhall, Pratul P Srinivasan, Matthow Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM., 65(1):99 106, 2021.
Brian B Moser, Arundhati S Shanbhag, Federico Raue, Stanislav Frolov, Sebastian Palacio, and Andreas Dengel. Diffusion models, image super-resolution, and everything: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. Revideo: Remake video with motion and content control. Xi preprint arXiv:2405. 19865, 2024.
76

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Urwa Muaz, Wondong Jang, Rohun Tripathi, Santhosh Mani, Wenbin Ouyang, Ravi Teja Gadde, Baris Gecer, Sergio Elizondo, Reza Madad, and Naveen Nair. Sidgan: High-resolution dubbed video generation via shift-invariant learning. In ICCV, pp. 7833 7842, 2023.
Mohammad Nadeem, Shahab Saquib Sohail, Erik Cambria, Björn W. Schuller, and Amir Hussain. Gender Bias in Text-to- Video Generation Models: A case study of Sora, January 2025. URL http://arxiv.org arXiv:2501.01957 [cs].
Arijit Nag, Animesh Mukherjee, Niloy Ganguly, and Soumen Chakrabarti. Cost- Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Findings of the Association for Compulational Linguisties: EMNLP 2024, pp. 15681 15701, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18633/v1 /20224 finddiiginnnnss---- findings-cmmlp.922)
URL https://aclanthologgg wwwp:/y....../22../..// wttg/2024.findiinne findings-neninn
920/
Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, volume 13674, pp. 407-426, 2022. ISBN 778-3-031-19780-2 978-3-031-19781-9,
Koichi Namekata, Sherwin Bahmani, Ziyi Wu, Yash Kant, Igor Gilitschenski, and David B Lindell. Sg-i2v: Self-guided trajectory control in image-to-vided generation. arviv preprini arXiv:2411.0498 2024.
Kepan Nan, Rui Xie, Penghao Zhou, Tichan Fan, Zhenheng Yang, Zhijic Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-1mg Im: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.0233 0233711,00711 2024.
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In CVPR, pp. 18444 18455, 2023.
Alex Nichol, Prafulla Dhariwal, Adilya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πlya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arviv preprint arXiv:2112 1211.122111 10741, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In TCML, pp. 8162 8171, 2021.
NVIDIA, :, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung- Yi Lin, Huan Ling, Ming- Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang, Heng Wang, Ving-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, and Artur Zolkowski. Cosmos world foundation model platform for physical ai, 2025.
Yeongtak Oh, Jooyoung Choi, Yongsung Kim, Minjun Park, Chaehun Shin, and Sungroh Yoon. Control- dreamer: Blending geometry and style in text-to-3d, 2023.
Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kocheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. In CVPR, pp.
2024a.
Wenqi Ouyang, Yi Dong, Lei Yang, Jianlou Si, and Xingang Pan. 12vedit: via image-to-video diffusion models. arXiv preprint ar Xin::22405. 16537, 2024b.
video editing
77

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Yichen Ouyang, jianhao Yuan, Hao Zhao, Gaoang Wang, and Bo zhao. Flexifilm: Long video generation with flexible conditions, 2024c.
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efficiet image-
to-video transfer learning for action recognition. arXiv preprint arXiv:2206.13539 2022.
Junting Pan, Ziyi Lin, Yuying Ge, Xiatian Zhu, Renrui Zhang, Yi Wang, Yu Qiao, and Hongsheng Li. Detrieving-to-answer Zero-shot video question answering with frozen large language models. In ICCV, pp. 272 283, 2023.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. Diffusevaa: Elficient, controllable and high-fidelity generation from low-dimensional latents, 2022. URL https://arxis: org/abs/2201 00308
Yan Pang, Aiping Xiong, Yang Zhang, and Tianhao Wang. Towards understanding unsafe video generation, 2024. URL https://arxiv org/ abtp/bbb////..////// 2408.1288111
Youxin Pang, Yong Zhang, Weize Quan, Yanbo Fan, Xiaodong Cun, Ying Shan, and Dong-ming Yan. Dpe: Disentanglement of pose and expression for general video portrait editing. In CVPR, pp. 427 436, 2023.
Jae Sung Park, Sheng Shen, Ali Farhadi, Trevor Darrell, Yejin Choi, and Anna Rohrbach. Exposing the limits of video-text models through contrast sets. In NAACL, pp. 3586, 2022.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In CVPR, pp. 2536 2544, 2016.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. In ICLR, 2023.
Samuele Poppi, Tobia Poppi, Federico Cocchi, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Safe- CLIP: Removing NSFW Concepts from Tisionaand-Lannuuge Models, July 2024. URL http://arxiv/ org/abs/2311. 16254 arXiv:2311.6255/ [cs).
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In ICCV, 2023.
Bosheng Qin, Juncheng Li, Siliang Tang, Tat-Seng Chua, and Yueting Zhuang. Instructvid2vid: Controllable video editing with natural language instructions. In ICME, pp. 1 6, 2024.
Zhiwu Qing, Shiwei Zhang, Jiayu Wang, Xiang Wang, Yujie Wei, Yingya Zhang, Changxin Gao, and Nong Sang. Hicrarchical spatio-temporal decoupling for text-to-vidco generation. In CVPR, pp. 6635-6645, 2024.
Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning-free free trajectory control in video diffusion models. arXiv preprint orxiv:2406. 1.6..666 16863, 2024a.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free free longer video diffusion via noise rescheduling. In JCLR, 2024b.
Y. Qiu et al. Freescale: High-resolution video generation with cascaded diffusion models. arXiv preprint, 2024c.
Weize Quan, Jiaxi Chen, Yanli Liu, Dong-Ming Yan, and Peter Wonka. Deep learning-based image and video inpainting: A survey. Int. J. Comput. Vis., pp. 1-34, 2024.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
78

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1-67, 2020.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage. and Analysis, pp. 1-16, 2020.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint Xrx:::2006.100 10704, 2020.
Rustam Rakhimov, Artur Sanakoyeu, et al. Lvdm: Latent video diffusion models. arXiv preprint arXiv:2301.0020: 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In ICML, pp. 8821-8831, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint (xXi::224.00120 2022.
Anurag Ranjan and Michael J Black. Optical flow estimation using a spatial pyramid network. In CVPR, pp. 4161-4170, 2017.
Elham Ravanbakhsh, Yongqing Liang, J. Ramanujam, and Xin Li. Deep video representation learning: a survey, 2024.
Nikhila Ravi, Valentin Gabeur, Yuan- Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam Segment anything in images and videos. Xix preprint ar 0rXiv22408.001 2024. URL https: org/ /bb/// 2408.00713
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nico- las Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos. In ICLR, 2025.
Scott Reed, Aäron Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In ICML, pp. 2912-2921, 2017.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Largo-scale learning and evaluation of real-life 3d category recon- struction. In ICCV, 2021.
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamganssian-d:: Generative 4d gaussian splatting, 2023.
Weiming Ren, Huan Yang, Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation, 2024.
Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movic description. In CVPR, pp.
2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Tn CVPR, pp. 10684-10695, 2022.
79

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAT, pp. 234-241, 2015.
Kevin Dela Rosa. Video enriched retrieval augmented generation using aligned video captions. abs/2405.17706, 2024.
CoRR,
Sruly Rosenblat, Tim O'Reilly, and Ilan Strauss. Beyond Public Access in LLM Pre-Training Data. Technical report, April 2025. URL
arXiv:2505.00020 [cs].
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias NieBner. Faceforensics: A large-scale video dataset for forgery detection human faces. In CVPR, 2018.
Claudio Rota, Marco Buzzelli, Simone Bianco, and Raimondo Schettini. Video restoration based on deep learning: a comprehensive survey. Artif. Intell. Rev., 56(6):5317 5364,, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. In NeurIPS, pp. (6479-364944 2022.
Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In ICCV, pp. 2830 2839, 2017.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-elficient unsupervised training of high-resolution temporal gan. Inl. J. Compul. Vis., 128(10): 2586-2000 2606, 2020.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. How2: A large-scale dataset for multimodal language understanding, 2018.
Seonguk Seo, Joon- Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with large-scale benchmark. In ECCV, pp. 208-223, 2020.
Younggyo Seo, Kimin Lee, Fangchen Läu, Stephen James, and Pieter Abbeel. Harp: Autoregressive latent video prediction with high-fidelity image generator. In ICIP, pp. 3943 3947, 2022.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis. In NewrIPS, 2021.
Xisoqian Shen, Xiang Li, and Mohamed Elhoseiny. Mostgan-v: Video generation with temporal motion styles. In CVPR, pp. 5652-5661, 2023.
Changhao Shi, Haomiao Ni, Kai Li, Shaobo Han, Mingfu Liang, and Martin Renqiang Min. Exploring compositional visual generation with latent classifier guidance. In CVPR, pp. 853 862, 2023a.
Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videollow: Exploiting temporal cues for multi-frame optical flow estimation. arXiv preprint arXiv:2903.08.0 08340, 2023b.
Xiaoyu Shi, Zhaoyang Huang, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer++ Masked cost volume autoencoding for pretraining optical flow estimation. arXiv preprint arXiv:2303.011 012277, 2023c.
Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable inage-to-video generation with explicit motion modeling. In SIGGRAPH, pp. 1-11, 2024.
Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In ICLR, 2023d.
Chachun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon. Edit-a-video: Single video editing with object-aware consistency. Tn ACML, pp. 1215-1230, 2024.
80

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Chenyang Si, Weichen Fan, Zhengyao Lv, Ziqi Huang, Yu Qiao, and Ziwei Liu. Repvideo: Rethinking cross-layer representation for video generation. ar preprint arXiv:2501.0880 0.88949 2025.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Newr/PS, 2019.
Uri Singer, Adam Polyak, et al. Make-a-videst Text-to-video generation without text-video data. arXiv preprint 2rx: 14792, 2022.
Uriel Singer, Shelly Sheymin, Adam Polyak, Oron Ashual, lurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene gencration. arXiv preprint arXiv:2301.1123 112800 2023.
Amanprect Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision aligmment model. In CVPR, pp. 15617-15629, 2022.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In CVPR, pp. 3616-3618 2022.
ATy Skyreels v1: Human-centric video foundation model. fttp:://ggthuub.co/Skkkyy SkyReels-V1 2025.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, pp. 2256 2265, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ar preprint arXiv:2010.0255 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. score-based generative modeling through stochastic differential equations. In ICLR, 2020b.
Yeji Song, Wonsik Shin, Junsoo Lee, Jeesoo Kim, and Nojum Kwak. Save: Protagonist diversification with structure agnostic video editing. arXiv preprint arXiv:2312.022 02508, 2023.
Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, and Sherry Yang. Videoagent: Self-improving video generation. arXiv preprint arXiv:2410.10071 2024a.
Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, and Sherry Yang. Videoagent: Self-improving video generation, 2024b.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
Alexandros Stergiou and Ronald Poppe. Adapool: Exponential adaptive pooling for information-retaining downsampling, IEEE Trons. Image Process., 32:251 266, 2023.
Jonathan C. Stroud, Zhichao Lu, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid, and David A. Ross. Learning video representations from textual web supervision, 2021.
Deqing Sun, Stefan Roth, and Michael J Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. Iut. J. Comput. Vis., 106(2):115 137, 2014.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, pp. 8934 8943, 2018.
Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou. Loftr: Detector-free local feature matching with transformers. In CVPR, pp. 8922 8931, 2021.
Mingzhen Sun, Weining Wang. Zihan Qin, Jiahui Sun, Sihan Chen, and Jing Liu. Glober: Coherent non- autoregressive video generation via global guided video decoder. Tn NeurIPS, 2023a.
81

---

Published in Transactions on Machine Learning Research (MM/YYY)
Mingzhen Sun, Weining Wang, Xinxin Zhu, and Jing Liu. Moso: Decomposing motion, scene and object for video prediction. In CVPR, pp. 18727-18737, 2023b.
Qi Sun, Zhiyang Guo, Ziyu Wan, Jing Nathan Yan, Shengming Yin, Wengang Zhou, Jing Liao, and Houqiang Li. Eg4d: Explicit generation of 4d object without score distillation, 2024a.
Wenhao Sun, Rong-Cheng Tu, Jingyi Liao, Zhao Jin, and Dacheng Tao. Asymrnr: Video diffusion trans-
formers acceleration with asymmetric reduction and restoration, 2025. URL bttps://arxiv.org.e org/abs/
Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, Jiahao Bu, Zhongzhi Chen, Xuemeng Huang, Fengzong Lian, Saiyong Yang, Jianfeng Yan, Yuyuan Zeng, Xiaoqin Ren, Chao Yu, Lulu Wu, Yue Mao, Jun Xia, Tao Yang, Suncong Zheng, Kan Wu, Dian Jiao, Jinbao Xue, Xipeng Zhang, Decheng Wu, Kai Liu, Dengpeng Wu, Guanghui Xu, Shaohua Chen, Shuang Chen, Xiao Feng. Yigeng Hong, Junqiang Zheng, Chengcheng Xu, Zongwei Ti, Xiong Kuang, Jianglu Hu, Yigi Chen, Yuchi Deng, Guiyang Li, Ao Liu, Chenchen Zhang, Shihui Hu, Zilong Zhao, Zifan Wu, Yao Ding, Weichao Wang, Han Liu, Roberts Wang, Hao Fei, Peijie Yu, Ze Zhao, Xun Cao, Hai Wang, Fusheng Xiang, Mengyuan Huang, Zhiyuan Xiong, Bin Hu, Xuebin Hou, Lei Jiang, Jianqiang Ma, Jiajia Wu, Yaping Deng, Yi Shen, Qian Wang, Weijie Liu, Jie Liu, Meng Chen, Liang Dong, Weiwen Jia, Hu Chen, Feifei Liu, Rui Yuan, Huilin Xu, Zhenxiang Yan, Tengfei Cao, Zhichao Hu, Xinhua Feng, Dong Du, Tinghao Yu, Yangyu Tao, Feng Zhang, Jianchen Zhu, Chengshong Xu, Xirui Li, Chong Zha, Wen Ouyang, Yinben Xia, Xiang Li, Zekun He, Rongpeng Chen, Jiawei Song, Ruibin Chen, Fan Jiang, Chongqing Zhao, Bo Wang, Hao Gong, Rong Gan, Winston Hu, Zhanhui Kang, Yong Yang, Yuhong Liu, Di Wang, and Jie Jiang. Hunyuan-large: An open-source moe model with 52 billion activated parameters by lencent, 2024b.
Yuqing Sun, Yifan Xu, et al. Latent-shift: Exploring latent shifts for video generation. arXiv preprint arXiv:2302.0001 2023c.
Ryan Szeto and Jason J. Corso. The devil is in the details: A diagnostic evaluation benchmark for video inpainting. In CVPR, pp. 21022 220011, 2022.
Shuai Tan, Bin Ji, and Ye Pan. Flowvqtalker: High-quality emotional talking face generation through normalizing flow and quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 26317 26327, 2024a.
Zhenxiong Tan, Xingyi Yang, Songhua Liu, and Xinchao Wang. Vidco-infinity: Distributed long video generation, 2024b.
Zhiyu Tan, Mengping Yang, Luozheng Qin, Hao Yang, Ye Qian, Qiang Zhou, Cheng Zhang, and Hao Li. An empirical study and analysis of pect-to-image generation using large language model powered textual representation. arXiv preprint arXiv:2405.122 12914, 2024c.
Shitao Tang, Fuyang Zhang, Jiacheng Chen, Peng Wang, and Yasutaka Furukawa. Mvdiffusion: Enabling holistic multi-view image generation with correspondence-waare diffusson.. arXiv preprint, 2023.
Shitao Tang, Jiacheng Chen, Dilin Wang, Chengzhou Tang, Fuyang Zhang, Yuchen Fan, Vikas Chandra, Yasutaka Furukawa, and Rakesh Ranjan. Mvdiffusion++: A dense high-resolution multi-view diffusion model for single or sparse-view 3d object reconstruction. arXiv preprint arXin:2402 12712, 2024.
Genmo Team. Mochi 1. httpa:///ggiaaub..oooommgoooooo/
2024a.
Kolors Team. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint, 2024b.
The Movie Gen team Movie gen: A cast of media foundation models, 2024.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, pp. 402-419. 2020.
82

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Yao Teng, Enze Xie, Yue Wu, Haoyu Han, Zhenguo Li, and Xihui Liu. Drag-a-video: Non-rigid video editing with point-based interaction. aixiv preprint or NNw:2312.02990 2023.
Yannis Tevissen, Khalil Guetari, and Frédéric Petitpont. Towards retrieval augmented generation over large
video libraries. CoRR, abs/ abss/4466.hh 14938, 2024.
Linrui Tian, Qi Wang, Bang Zhang, and Liefeng Bo. Emo: Emote portrait alive-generating expressive portrait videos with audio2video diffusion model under weak conditions. arXiv preprint arXiv:2402.1100 17485, 2024.
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N Metaxas, and Sergey Tulyakov. A good image generator what you ncod for high-resolution vidco synthesis. In ICLR, 2020.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian FATRAS, Guy Wolf, and Yoshua Bengio. Improving and generalizing fow-based generative models with minibatch optimal transport. In ICML Workshop, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anme Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint arii::2302.139 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint or orXiv:2307.09250 2023b.
Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotem- poral features with 3d convolutional networks. Tn ICCV, pp. 148994499) 2015.
Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motioneditor: Editing video motion via conlent-aware diffusion. In CVPR, pp. 7882 7891, 2024.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz, Mocogan: Decomposing motion and content for video generation. In CVPR, pp. 1526-1535, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly, Fvd: A new metric for video generation. In ICLR Workshop, 2019.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017.
Basile Van Hoorick, Rundi Wu, Ege Ouguroglu, Kyle Sargent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl Vondrick. Generative camera dolly: Extreme monocular dynamic novel view synthesis. In ECCV, pp. 313 331, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2022.
Vikram Voleti, Alexia Marttnnaa, and Christopher Pal. Mevd masked conditional video diffussin for prediction, generation, and interpolation. In NewrIPS, 2022.
Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. aixiv preprint arxi::2403. 12008, 2024.
83

---

Published in Transactions on Machine Learning Research (MM/YYY)
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wai Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng, Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang, Wenmeng Zhou, Wente Wang, Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng, Zeyinzi Jiang, Zhen Han, Zhi- Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models, 2025.
Zhen Wan, Yue Ma, Chenyang Qi, Zhiheng Liu, and Tao Gui. Unipaint: Unified space-time video inpainting via mixture-of-coxeets. arXiv preprint arxiv:2412 (.fi:2412.034/// 2024.
Cong Wang, Jiaxi Gu, Panwen Hu, Songeen Xu, Hang Xu, and Xiaodan Liang. Dreamvideo: High-fidelity image-to-video generation with image retention and text guidance, 2023a.
Yun Wang, Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-I-video: Multi- text to long video generation via temporal co-denoising, 2023b.
Fu-Yun Wang, Zhaoyang Huang, Weikang Bian, Xiaoyu Shi, Keqiang Sun, Guanglu Song, Yu Liu, and Hongsheng Li. Animatelcm: Computation-efficn personalized style video generation without personal- ized video data, 2024a.
Haoran Wang, Di Xu, Dongliang He, Fu Li, Zhong Ji, Jungong Han, and Errui Ding. Boosting video-text retrieval with explicit high-level semantics. In ACM MM, pp. 4887 4898, 2022a.
Jiamian Wang, Guohao Sun, Pichao Wang, Dongfang Liu, Sohail A. Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Text is mass: Modeling stochastic embedding for texl-video retrieval. CoRR, abs/2403. 17998, 2024b.
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK Chan, and Chen Change Loy. Exploiting diffusion prior for real-world image super-resolution. Int. J. Comput. Vis., pp. 1-21, 2024c.
Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. Boximator: Generating rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01560 2024d.
Jiuniu Wang, Hangjic Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modclscope text-to-video technical report. arXie preprinl -Xxx:2308.065111 2023c.
Qinghe Wang, Yawen Luo, Xiaoyu Shi, Xu Jia, Huchuan Lu, Tianfan Xue, Xintao Wang, Pengfei Wan, Di Zhang, and Kun Gai. Cinemaster: A 3d-aware and controllable framework for cinematic text-to-video generation, 2025a. URL https://arxiv, org/ hwggaab//2502... 2ttp//2222000099999
Ruotong Wang, Mingli Zhu, Jiarong Ou, Rui Chen, Xin Tao, Pengfei Wan, and Baoyuan Wu. Badvideo: Stealthy backdoor attack against text-to-video generation, 2025b. URL https: org/ 2504.
16907
Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. Tn CVPR, pp. 9326 9336, 2024e.
Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot vidco oditing using off-the-sholf image diffusion models. arXiv preprint (xXi:22303.175)) 2023d.
Wenhao Wang and Yi Yang. Vidprom: A million-scale real prompt-gallery datasct for text-to-video diffusion models. In NewrIPS, 2024.
Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Swap attention in spatiotemporal diffusions for text-to-video generation, 2024f.
84

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. arXiv preprint 0aee23060020188 2023e.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jimniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. In NewrIPS, 2023f.
Xiaohan Wang, Linchao Zhu, Zhedong Zheng, Mingliang Xu, and Yi Yang. Align and tell: Boosting text- video retrieval with local alignment and fine-grained supervision. IEEE Trans, Multimedia, pp. 1-11, 2022b.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large- scale, high-quality multilingual dataset for ande--and-languanl research. In ICCV, pp. 4580-4590, 2019.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arxiv preprint arXiv:2309.155 15103, 2023g.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR, 2023h.
Yikai Wang, Xinzhou Wang, Zilong Chen, Zhengyi Wang, Fuchun Sun, and Jun Zhu. Vidu4d: Single generated video to high-fidelity 4d reconstruction with dynamic gaussian surfels. In NewrIPS, 2024g.
Yimu Wang and Peng Shi. Video-text retrieval by supervised sparse multi-grained learning. In FMNLP, 2023.
Yimu Wang, Shiyin Lu, and Lijun Zhang. Searching privately by imperceptible lying: A novel private hashing method with differential privacy. In ACM MM, pp. 2700 2709, 2020a.
Yimu Wang, Xiu-Shen Wei, Bo Xue, and Lijun Zhang. Piecewise hashing: deep hashing method for large-scale fine-grained search. In PRCV, pp. 432 444, 2020b.
Yimu Wang, Bo Xue, Quan Cheng, Yuhui Chen, and Lijum Zhang, Deep unified cross-modality hashing by pairwise data alignment. In TJCAT, pp. 1129-1135, 2021.
Yimu Wang, Shuai Yuan, Bo Xue, Xiangru Jian, Wei Pang, Mushi Wang, and Ning Yu. DREAM: Improving vidco-text retrieval through relevanco-based augmentation using large foundation models. In NAACL, pp. 3037 3056, 2025c.
Yuhan Wang, Liming Jiang, and Chen Change Loy. Styleinv: A temporal style modulated inversion network for unconditional video generation. In ICCV, pp. 22851 -22661, 2023i.
Zhen Wang, Yunhao Ba, Pradyumna Chari, Oyku Deniz Bozkurt, Gianna Brown, Parth Patwa, Niranjan Vaddi, Lalch Jalilian, and Achuta Kadambi. Synthetic generation of face videos with plethysmograph physiology. In Proceedings of the IEEE/CVF conference computer vision and pottern recognition, pp. 2022c.
Zhendong Wang, Huangjie Zheng, Pengeheng He, Weizhu Chen, and Mingyuan Zhou. Diflusion-gan: Train- ing gans with diffusion, 2023j. URL https: org/ abs abs///0..22...... 02262
Zhenzhi Wang, Yixuan Li, Yanhong Zeng, Youqing Fang, Yuwei Guo, Wenran Liu, Jing Tan, Kai Chen, Tianfan Xue, Bo Dai, and Dahua Lin. Humanvid: Demystifying training data for camera-controllable human image animation, 2024h.
Zhouxia Wang, Yushi Lan, Shangchen Zhou, and Chen Change Loy. Objetrl-2.5 d: Training-free object control with camera poses. arXiv preprint #xXi:2211..0772 2024i.
85

---

Published in Transactions on Machine Learning Research (MM/YYY)
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-, 2024j.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In SIGGRAPH, 2024k.
Daniel Watson, Saurabh Saxena, Lala Li, Andrea Tagliasacchi, and David J. Fleet. Controlling space and time with diffusion models, 2024.
Huawei Wei, Zejun Yang, and Zhisheng Wang. Aniportrait: Audio-driven synthesis of photorcalistic portrait animation. orXiv preprint 20X3::2403.110 10003..7699) 17694, 2024a.
Yujic Wei, Shiwei Zhang, Hangjic Yuan, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng, Feng Liu, Zhizhong Huang, Jiaxin Ye, Yingya Zhang, and Hongming Shan. Dreanvideo-2: Zero-shol subject-driven video customization with precise motion control, 2024b.
Yujie Wei, Shiwei Zhang, Hangjee Yuan, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng, Feng Liu, Zhizhong Huang, Jiaxin Ye, et al. Dreamvideo-2: ZZero-hoot subject-driven video customization with precise motion control. arXiv preprint arXiv:24 11.022410..8888 13830, 2024c.
Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, et al. Art-v: Auto-regeessive text-to-video generation with diffusion models. In CVPR, pp.
2024.
Dongming Wu, Xingping Dong, Ling Shao, and Jianbing Shen. Multi-level representation learning with semantic alignment for referring video object segmentation. In CVPR, pp. 4986-4995, 2022a.
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment user generated contents from aesthetic and technical perspectives. In ICCV, pp. 20087 20097, 2023a.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding, In NewrIPS, 2024a.
Hongjia Wu, Özgü Alay, Anna Brunstrom, Simone Ferlin, and Giuseppe Caso. Peckaboo: Learning- multipath scheduling for dynamic helerogeneous environments. IEEE Journal on Selected Areas in Com- munications, 38(10):222) 2220-22310, 2020.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, pp. 7623 7633, 2023b.
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In CVPR, pp. 4964 4974, 2022b.
Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, et al. Towards aagguaee-drveen video inpainting via multimodal large language models. In CVPR, pp. 12501-12511, 2024b.
Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Motionbooth: Motion-aware eustomized text-to-video generation. ar preprint Xrxiv:2406. 17758, 2024c.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen 1lm applications via multi-agent conversa- tions. In COLM, 2024d.
Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. Lamp: Learn motion pattern for lew-shot-based video generation. Xi preprint 2.1:23300.10000 2023c.
86

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Rundi Wu, Ruiqi Gao, Ben Poole, Alex Trevithick, Changxi Zheng, Jonathan T. Barron, and Aleksander Holynski. Cat4d: Create anything in 4d with multi-view video diffusion models. arXim:2411.18611 2024e.
Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Liang Pan Jiawei Ren, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: Lacge-voaabulary 3d object dataset for realistic perception, reconstruction and generation. In CVPR, 2023d.
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefci He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. European Conference on Computer Vision, pp. 331-348. Springer, 2024f.
X. Wu et al. Videoprefer: A comprehensive video preference dataset and reward model. ar preprind, 2024g.
Xiuzhe Wu, Pengfei Hu, Yang Wu, Xiaoyang Lyu, Yan-Pei Cao, Ying Shan, Wenming Yang, Zhongqian Sun, and Xiaojuan Qi. Speech2lip: High-fidelity speech to lip generation by learning from short video. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 22168 22177, 2023e.
Yue Wu, Yu Deng, Jiaolong Yang, Fangyun Wei, Qifeng Chen, and Xin Tong. Anifacegan: Animatable 3d-aware face image generation for video avatars. In NewrIPS, 2022c.
Ziyi Wu, Aliaksandr Siarohin, Willi Menapace, Ivan Skorokhodov, Yuwei Fang, Varnith Chordia, Igor Gilitschenski, and Sergey Tulyakov. Mind the time: Temporell multi-event video generation. arXiv preprint (xXix:2412.00 05266, 2024h.
Hongehi Xia, Yang Fu, Sifei Liu, and Xiaolong Wang. Rgbd objects in the wild: Scaling real-world 3d object learning from rgb-d videos. In CVPR, pp. 22378 22389, 2024.
Jinxi Xiang, Ricong Huang, Jun Zhang, Guanbin Li, Xiao Han, and Yang Wei. Versvideo: Leveraging enhanced temporal diffusion models for versatile video generation. In ICLR, 2023.
Desai Xie, Zhan Xu, Yicong Hong, Hao Tan, Difan Liu, Feng Liu, Arie Kaufman, and Yang Zhou. Progressive autoregressive video diffusion models. arXie preprint arXiv:2410.0811 10.01151, 2024a.
Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models, 2025. URL https://arxiv org/ abs 22w./506..15... 15564
Liangbin Xie, Xintao Wang, Honglun Zhang, Chao Dong, and Ying Shan. Vfhq: A high-quality dataset and benchmark for video face super-resolution. In CVPR Workshops, pp. 656 665, 2022.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In CVPR, pp. 22428-22432 2023.
Yiming Xie, Chun-Han Yao, Vikram Voleti, Huaizu Jiang, and Varun Jampani. Sv4d: Dynamic 3d content generation with multi-frame and multi-view consistency, 2024b.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsh Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors, 2023.
Jinbo Xing, Hanyuan Liu, Menghan Xia, Yong Zhang, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Tooncrafter: Generative cartoon interpolation. arXiv preprint arXiv:2405. 17933, 2024a.
Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang, Simda: Simple diffusion adapter for efficient vidco generation. In CVPR, pp. 7827-7839, 2024b.
Zhen Xing, Qiuun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models, 2024c.
Yi Xiong, Ayan Chakrabarti, Ronen Basri, Steven Gortler, David W Jacobs, and Todd Zickler. Learning from synthetic data for sky replacement, in images. In ECCV, 2018.
87

---

Published in Transactions on Machine Learning Research (MM/YYY)
Dejia Xu, Yifan Jiang, Chen Huang, Liangchen Song, Thorseen Gernoth, Liangliang Cao, Zhangyang Wang, and Hao Tang. Cavia: Camera-controllbble multi-view video diffusion with view-integrated attention. arXiv preprint arXiv:2410.1077 2024a.
Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. Cameo: Camera- controllable 3d-consistent image-to-video generation. arXiv preprint arXiv:2406.0228 02509, 2024b.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM, 2017.
Jiaqi Xu, Xinyi Zou, Kunzhe Huang, Yunkno Chen, Bo Liu, MengLi Cheng, Xing Shi, and Jun Huang, Easyanimate: A high-perffrmance long video generation method based on transformer architecture. arXiv preprint Xrx: 18991, 2024c.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vit: A large video description dataset for bridging video and language. In CVPR, pp. 5288-52509 5296, 2016.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See-Kiong Ng, and Jiashi Feng. Pllava Parameter- free llava extension from images to videos for vidco dense captioning. arzic preprint or 2xX1:2404.16999 2024d.
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-l: Lifelike audio-driven talking faces generated in real time. are preprint arXiv:2404.1066 2024e.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. CVPR, pp. 5026 5035, 2022a.
L Xue. mt5: A massively multilingual pre-trained text-to-text transformer. or arXiv preprint 0rXim:2010.1199) 2020.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte models. TACL, 10:291-306, 2022b.
Shin'ya ya Yamaguchi, Dewei Feng, Sekitoshi Kanai, Kazuki Adachi, and Daiki Chijiwa. Post-pre-training for Modality Alignment in Vision-Language Foundation Models, April 2025. URL http://arxiv.org/ann 2504. 1271.
12717 arXiv:2504.12717 [cs).
Liqi Yan, Qifan Wang, Yiming Cui, Fuli Feng, Xiaojun Quan, Xiangyu Zhang, and Dongfang Lin. Gl-rg: Global-local representation granularity for video captioning. IJCAI, pp. 2769 2775, 2022.
Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Vidcogpt: Vidco generation using vq-vae and transformers. arXiv preprint arXit:2104.10157, 2021.
Wilson Yan, Andrew Brown, Pieter Abbeel, Rohit Girdhar, and Samaneh Azadi. Motion-conditioneel image animation for video editing. arXiv preprint arXiv:2311.1888 2023a.
Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbeel. Temporally consistent transformers for video generation. In ICML, pp. 19062-39098, 2023b.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiu preprint arXiv:2309.1030 2023a.
Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. In NewrIPS, 2024a.
Fan Yang, Jianfeng Zhang, Yichun Shi, Bowen Chen, Chenxu Zhang, Huichao Zhang, Xiaofeng Yang, Jiashi Feng, and Guosheng Lin. Magic-boost: Boost 3d generation with mutli-view conditioned diffusion, 2024b.
88

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Gengshan Yang and Deva Ramanan. Volumetric correspondence networks for optical flow. In NeurIPS, pp. 794-805, 2019.
Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Chong- Wah Ngo, and Tao Mei. Hi3d: Pursuing high-rcsolution image-to-3d generation with vidco diffusion models. In ACM MM, 2024c.
Hao Yang, Chunfeng Yuan, Bing Li, Yang Du, Junliang Xing, Weiming Hu, and Stephen J. Maybank. Asymmetric 3d convolutional neural networks for action recognition. Pattern Recognit, 85:1-12, 2019.
Jiayu Yang, Ziang Cheng, Yunfci Duan, Pan Ji, and Hongdong Li. Consistnet: Enforcing 3d consistency for multi-view images diffusion. In CVPR, pp. 7079 7088, 2024d.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024e.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.094 2024f.
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Picter Abbeel. Learning interactive real-world simulators. In ICLR, 2024g.
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object mo- tion. In Special Interest Group on Computer Graphies and Interactive Techniques Conference Conference Papers '24, pp. 1 12, 2024h.
Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. Direct-a-video: Customized video generation with user-directed camera movement and object motion. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-12, 2024i.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender video: Zero-shot text-guided video- to-video translation. In SIGGRAPH Asia, pp. 1-11, 2023b.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arxiv:2408. 06072, 2024j.
Yining Yao, Xi Guo, Chenjing Ding, and Wei Wu. Mygo: Consistent and controllable multi-view driving video generation with camera control, 2024.
Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. Hitca: Hicrarchical temporal-aware video-language pre-training. In ICCV, pp. 15359-153700 2023.
Chen Yeh, You-Ming Chang, Wei-Chen Chiu, and Ning Yu. T2vs meet vlms: A scalable multimodal dataset for visual harmfulness recognition, 2024a.
Chen Ych, You-Ming Chang, Wei-Chen Chiu, and Ning Yu. T2vs meet VLMs: for visual harmfulness recognition. In The Thirly-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024b. URL httppp///pppgaaeeeeereeee//rrrra
A scalable multimodal dataset
Hongwei Yi, Shitong Shao, Tian Ye, Jiantong Zhao, Qingyu Yin, Michael Lingelbach, Li Yuan, Yonghong Tian, Euze Xie, and Daquan Zhou. Magic 1-for-1: Generating one minute video clips within one minute, 2025.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089 2023a.
89

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Ming Gong, Lijuan Wang, Zicheng Liu, Houqiang Li, and Nan Duan. Nuwa-xl: Diffusion over diffusion for extremely long video generation. In ACL, pp. 1309 1320, 2023b.
Tianwci Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators. arXiv preprint arXiv:2422.0777,, 2024.
KiYoon Yoo, Wonhyuk Ahn, and Nojun Kwak. Advancing beyond identification: Multi-bit watermark for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4031-4055, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 012/6533///224...//// Vacclantholooy.orr//0000 1etee...aac-110 naacl-long. 224/
224. URL https:
Jachong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, and Mohit Bansal. SAFREE: Training-free and adaptive guard for safe text-to-image and video generation. In The Thirteenth International Conference on Learning Representations, 2025. URL http:://pppaooooooo...eeoooaaaaa
Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot novel view synthesizer, 2024.
Heng Yu, Chaoyang Wang, Peiye Zhuang, Willi Menapace, Aliaksandr Siarohin, Junli Cao, Laszlo Attila Jeni, Sergey Tulyakov, and Hsin-Ying Lee. 4real: Towards photorealistic 4d scene generation via video diffusion models. In NewrIPS, 2024a.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual allention. In CVPR, pp. 5505 5514, 2018.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexan- der Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to- image generation. arXiv preprint or arXix:200. 10789, 2022b.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer. In CVPR, pp. 10459 10499,,00990000000 2023a.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuse Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer key to visual generation. In ICLR, 2024b.
Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, and Jingjing Liu. Multimodal federated learning via rastivvvvve representation ensemble. In ICLR, 2023b.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awar implicit generative adversarial networks. In ICLR, 2021.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, Guanying Chen, Shuguang Cui, and Xiaoguang Han. Mvimgnet: A large-scale dataset of multi-view images. In CVPR, pp. 9150 91609116,14111 2023c.
Youngjae Yu, Hyungjin Ko, Jongwook Choi, and Gunhee Kim. End-to-en end concept word detection for video captioning, retrieval, and question answering. In CVPR, pp. 2261-3269, 2017.
La Yuan et al. Instructvideo: Instructing video diffusion models with human feedback. arxiv preprind, 2023.
90

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyuan Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. Identity-preserving text-to-video generation by frequency decomposition. arv preprint or orXiv:2411 10..2411.110 17440. 2024a.
Xin Yuan, Jinoo Back, Keyang Xu, Omer Tov, and Hongliang Fei. Inflation with diffusion: Efficient temporal adaptation for text-to-video super-resolution In WACV, pp. 489-496, 2024b.
Zhengqing Yuan, Ruoxi Chen, Zhaoxu Li, Haolong Jia, Lifang He, Chi Wang, and Lichao Sun. Mora: Enabling generalist video generation via a multi-agent framework. arXiv preprint arXiv:2408. 18248, 2024c.
Rowan Zellers, Ximing Lu, Jack Hessel, Younjjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. In NeurIPS, pp. 23634-23651, 2021.
Kuo-Hao Zeng, Tseng- Hung Chen, Ching Yao Chuang, Yuan-Hong Liao, Juan Carlos Niebles, and Min Sun. Leveraging video descriptions to learn video question answering. In AAAI, pp. 4334-4340, 2017.
Yan Zeng, Guoqiang Wei, Jiani Zhong, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. In CVPR, pp. 8850 8860, 2024a.
Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxum Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. In ECCV, pp. 163-179, 2024b.
Zheng Zeng, Valentin Deschaintre, Nlyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling-Qi Yan, and Miloš Hašan. Rgb-x: Image decomposition and synthesis using material- and lighting-aware diffusion models. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers '24, pp. 1-11, 2024c.
Bowen Zhang, Xiaofei Xie, Haotian Lu, Na Ma, Tianlin Li, and Qing Guo. Mavin: Multi-action video generation with diffusion models via transition video infilling, 2024a.
David Junhao Zhang, Jay Zhangjic Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprinl or 2iv: 15818, 2023a.
Guiwei Zhang, Tianyu Zhang, Guanglin Niu, Zichang Tan, Yalong Bai, and Qing Yang. Camel: Causal motion enhancement tailored for lifting text-driven video editing. In CVPR, pp. 9079-9088, 2024b.
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi-view video diffusion model for 4d generation, 2024c.
Jihai Zhang, Tianle Li, Linjic Li, Zhengyuan Yang, and Yu Cheng. Are unified vision-language models necessary: Generalization across understanding and generation, 2025. URL https: org/abs/////aaaaooeeooooobb 2505. 23043
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arriv:24 038255 2024d.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Im: Large reconstruction model for 3d gaussian splatting, 2024e.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, Adding conditional control to text-to-image diffusion models. In ICCV, pp. 3813-3824 2023b.
Qihang Zhang, Ceyuan Yang, Yujun Shen, Yinghao Xu, and Bolei Zhou. Towards smooth video composition. In ICLR, 2023c.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12gee-xll: High-quality image-to-video synthesis via cascaded diffusion models, 2023d.
91

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Xuanyu Zhang, Youmin Xu, Runyi Li, Jiwen Yu, Weiqi Li, Zhipei Xu, and Jian Zhang. V2a-mark: Ver- satile deep visual-audio watermarking for manipulation localization and copyright protection. In ACM Multimedia 2024, 2024f. URL https://openreviers /t/ wtt.://www.......oooooooo/ooDEDa
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. orXiv preprint ar xxiv:2305.13077, 2023e.
Yabo Zhang, Yuxiang Wei, Xianhui Lin, Zheng Hui, Peiran Ren, Xuansong Xie, Xiangyang Ji, and Wang- meng Zuo. Videoelevator: Elevating video generation quality with versatile text-to-image diffusion models. arXiv preprint ar 4rXit:2408.05455 053344 2024g.
Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via alug-and-play modules in text-to-image models, 2023f.
Zhenghao Zhang, Junchao Liao, Menghao Li, Zuozhuo Dai, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Tora: Trajectory-oriented diffusion transformer for video generation. arXiv preprint arXiv:2407.21700 2024h.
Zhixing Zhang, Bichen Wu, Xiaoyan Wang, Yaqiao Luo, Luxin Zhang, Yinan Zhao, Peter Vajda, Dimitris Metaxas, and Licheng Yu. Avid: Any-longth video inpainting with diffusion model. In CVPR, pp. 7162- 7172, 2024i.
Zhongwei Zhang, Fuchen Long, Yingwei Pan, Zhaofan Qiu, Ting Yao, Yang Cao, and Tao Mei. Trip: Temporal residual learning with image noise prior for image-to-video diffusion models. In CVPR, pp- 8671 8681, 2024j.
Zicheng Zhang, Bonan Li, Xuecheng Nie, Congying Han, Tiande Guo, and Luoqi Liu. Towards consistent video editing with text-to-image diffusion models. In NewrIPS, 2024k.
Canyu Zhao, Mingyu Liu, Wen Wang, Weihua Chen, Fan Wang, Hao Chen, Bo Zhang, and Chunhua Shen. Moviedreamer: Hierarchical generation for coherent long visual sequences. In ICLR, 2025.
Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi- attentional Deepfake Detection. arXiv:2103.024/ [es}, March 2021. URL http:://aaxtii org/ 22003
02406 Xiv: 2103.02406.
Haoyu Zhao, Tianyi Lu, Jiaxi Gu, Xing Zhang, Qingping Zheng, Zuxuan Wu, Hang Xu, and Yu-Gang Jiang. MagDiff: Multi- altiiitttlggmmentt Diffusion for High-Fidelity Video Generation and Editing, pp. 205 221. Springer Nature Switzerland, September 2024a. ISBN 978303172649.. doi: 1111077977---001-77200
12, URL
http://dx. doi. org/ (0rgg/10.00. 1007 00--001--742-- 73649-726
Kai Zhao, Xiaoli Lin, et al. Dysen-vdm: Dynamic style euhanced video diffusion models. arXiv preprint 2tti:22355000013 2023a.
Nanxuan Zhao, Jianbo Jiao, Weidi Xie, and Dahua Lin. Cali-nce: Boosting cross-modal video representation learning with calibrated alignment. In CVPR, pp. 6317-6327, 2023b.
Rui Zhao, Yuchao Gu, Jay Zhangjic Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint 2:Xiu:2310.08400 2023c.
Ruoyu Zhao et al. Videofactory: Efficient video generation with fine-grained control. arXiv:3305001883, 2023d.
ar preprint
Shuai Zhao, Linchao Zhu, Xiaohan Wang, and Yi Yang. Centerclip: Token clustering for efficient text-video retrieval. In SIGIR, pp. 970-981, 2022.
Sijic Zhao, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Muyao Niu, Xiaoyu Li, Wenbo Hu, and Ying Shan. Cr-vae: A compatible video vae for latent generative video models. arXiv preprint arXiv:2405.0022 20279, 2024b.
92

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate124: Animating one image to 4d dynamic scene, 2023e.
Zhou Zhao, Jinghao Lin, Xinghua Jiang, Deng Cai, Xiaofei He, and Yueting Zhuang. Video question answering via hierarchical dual-level attention network learning. In ACM MM, pp. 1050-1058, 2017.
Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-
to-video diffusion model, 2024a.
Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text-and image-guided 4d scene generation. In CVPR, pp. 7300 7309, 2024b.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, 2024c.
Yaoyao Zhong, Wei Ji, Junbin Xiao, Yicong Li, Weihong Deng, and Tat-Seng Chua. Video question answer- ing: Datasets, algorithms and challenges. In EMNLP, pp. 6439 6455, 2022.
Jia Zhou, Yang Dong, et al. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprinl Xiv: 1110018, 2022.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instruc- tional videos. In AAAI, 2018a.
Qiang Zhou, Shaofeng Zhang, Nianzu Yang, Ye Qian, and Hao Li. Motion control for enhanced complex action video generation. arviv preprint arXiv:2/11.08328, 2024a.
Shangchen Zhou, Peiqing Yang, Jianyi Wang, Yihang Luo, and Chen Change Loy. Upscale-a-video: Temporal-consistont diffusion model for real-world vidoo super-rosolution. In CVPR, pp. 2535-2545, 2024b.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint orxiv:8805 1805.09977, 2018b.
Xingcheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C. Knoll. Vision language models in autonomous driving: A survey and outlook. IEEE Trans. Intell. Veh., pp. 1-20, 2024c.
Yiwen Zhou, Xiaoyu Lin, et al. Videofusion: Diffusion-based texl-to-video generation. ar preprinl arXiv:2302.05700 2023.
Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou. Storydiffusion: Consistent self-attention for long-range image and vidco generation. In NewrIPS, 2024d.
Zhongliang Zhou, Jielu Zhang, Zihan Guan, Mengxuan Hu, Ni Lao, Lan Mu, Sheng Li, and Gengchen Mai. Img2loc: Revisiting image geolocalization using multi-modality foundation models and image- retrieval-angmentee generation. In SIGIR, pp. 2749-2754, 2024e.
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text. representations. In CVPR, pp. 8743
8752, 2020.
Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream a vlog. arXio preprint arXiv:2401.004 14, 2024.
Bojia Zi, Shihao Zhao, Xianbiao Qi, Jianan Wang, Yukai Shi, Qianyu Chen, Bin Liang, Kam-Fai Wong, and Lei Zhang. Cococo: Improving text-guided video inpainting for better consistency, controllability and compatibility. arXiv preprint arXiv:2403. 12035, 2024.
Qi Zuo, Xiaodong Gu, Lingteng Qiu, Yuan Dong, Zhengyi Zhao, Weihao Yuan, Rui Peng, Siyu Zhu, Zilong Dong, Liefeng Bo, and Qixing Huang. Videomv: Consistent multi-view generation based on large video generative model, 2024.
93


Reviews:
[{"review_id": "Vd2GUwqDAP", "reviewer": "Reviewer_TCMd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper offers an exhaustive analysis, spanning foundational theories (diffusion processes, architectures) to practical implementations (datasets, training techniques) and real-world applications.\n* The authors delve into intricate technical details, such as the mathematical formulations of DDPM/DDIM, architectural modifications for video generation (3D UNet, temporal attention layers), and evaluation metrics.\n* The survey not only reviews current progress but also identifies emerging trends, such as 3D-aware diffusion, long-video generation, and cross-domain applications (e.g., 3D/4D scene synthesis). This detailed overview from multiple perspectives differs itself from existing related surveys.\n\n\nWeaknesses:\n\n* Many of the technical details from Section 2 are also covered in the review of image diffusion models. It would be better to focus more on the video-specific techniques.\n\n* While this manuscript recognizes that computational efficiency is a critical challenge, the authors are suggested to include quantitative analysis on the computational resources used by the SOTA video diffusion models.\n\n* The discussion on ethical considerations is very brief, primarily mentioning biases and misuse without delving into concrete frameworks or case studies. A more thorough analysis of privacy risks (e.g., deepfake generation, watermarking the video diffusion models) or societal impacts would strengthen the paper’s comprehensiveness.\n\n* It would also be better to discuss more on the specific future directions that the authors advocate for the researchers to further explore.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey provides a comprehensive review of video diffusion models, spanning their technical foundations, practical implementations, and diverse applications. The authors first outline the evolution of video generation paradigms, contrasting GANs and auto-regressive models with diffusion-based approaches, highlighting the latter’s superiority in temporal consistency and visual quality. They delve into learning frameworks (e.g., DDPM, DDIM, EDM), architectural innovations (UNet, Transformer-based designs), and guidance mechanisms (classifier and classifier-free guidance). The implementation section discusses datasets (e.g., WebVid-10M, UCF-101), training engineering techniques (data preprocessing, acceleration methods), and evaluation metrics (static quality and temporal consistency). Applications range from conditional generation (image, spatial, and camera parameter guidance) to video enhancement (denoising, inpainting, super-resolution), personalization, long-video generation, and 3D-aware synthesis. The paper also explores how video diffusion models benefit other domains, such as video representation learning, retrieval, and 3D/4D generation. Finally, it concludes with challenges (computational efficiency, long-video coherence) and future directions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My concerns are mainly from the lack of discussion on computational efficiency, ethical considerations, and more specific future directions. Please kindly refer to Section Strengths and Weaknesses for detailed suggestions.", "broader_impact_concerns": "Please kindly refer to the suggestion from the Weaknesses.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6aVee1OBoo", "reviewer": "Reviewer_xH7A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The review is thorough but lacks coherence in certain areas. For example, Section 2.2 lacks a necessary introduction—why does the discussion suddenly jump from the generation paradigms in Section 2.1 to specific technical details? What paradigms do these techniques correspond to? Additionally, Section 2.4.1 (Pixel Diffusion and Latent Diffusion) and Section 2.2 cover several overlapping topics, and the authors should further clarify these points. Emphasizing the connections between each section and organizing the manuscript in a way that aligns more closely with Figure 1 would make the paper clearer.\nMoreover, although the Evaluation section includes many rich metrics, it omits discussions on motion-related aspects such as motion consistency and motion smoothness. In the Data Processing Pipeline section, the referenced literature is relatively old, and the authors could consider introducing more recent works on video generation processes, such as Wan [1] and Hunyuan [2].\n\nMinor questions:\n* 2.1.3 Video Diffusion Models: The citation format in the first paragraph is incorrect.\n* The technical introduction of GANs is missing.\n[1] Wan: Open and Advanced Large-Scale Video Generative Models\n[2] Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a comprehensive review of the video generation field. It discusses various video generation methods based on different approaches such as GANs, autoregressive models, diffusion models, etc., as well as different model components like U-Net, DiT, and tokenizers. It also covers video training processes, including data processing, training engineering, and downstream tasks. The authors provide a complete and rich overview of video generation, and also touch on other areas such as video understanding.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see Strengths And Weaknesses.", "broader_impact_concerns": "This article is a comprehensive review of video generation. Perhaps it is necessary to point out the risks associated with the generated videos in areas such as privacy and copyright.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gApPDrRgc1", "reviewer": "Reviewer_9W7M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is extremely comprehensive on the methodology, architectures and downstream applications. The literature coverage and analysis even for relatively niche applications such as camera control and 3D-aware methods is quite thorough. I think the survey is well researched and there is a significant audience for the paper.\n\nOne area where the literature review and discussion feels a bit thin is the section on data and training. A lot of the progress in video diffusion has been made from the data and training side, so it feels a bit brief to only discuss training for two paragraphs. I think this is acceptable given that these details are not well covered in the literature in general.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides an extremely comprehensive overview of video diffusion models. The survey forms a nice taxonomy over the subfield, provides context of other video generation approaches, and covers many downstream applications of video generation. Compared to previous surveys which cover a specific application of video diffusion, this work provides a broad, bird's-eye view of the domain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It could be useful for the audience to discuss training and large-scale data filtering techniques such as the pipelines described in the MovieGen or Cosmos paper. This is not critical for acceptance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)