A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches
Authors: Ruibo Ming
Zhewei Huang
Jingwei Wu
Zhuoxuan Ju
Daxin Jiang
Jianming HU
Lihui Peng
Shuchang Zhou
Venue: Accepted by TMLR ()

Abstract: Future Frame Synthesis (FFS), the task of generating subsequent video frames from context, represents a core challenge in machine intelligence and a cornerstone for developing predictive world models. This survey provides a comprehensive analysis of the FFS landscape, charting its critical evolution from deterministic algorithms focused on pixel-level accuracy to modern generative paradigms that prioritize semantic coherence and dynamic plausibility. We introduce a novel taxonomy organized by algorithmic stochasticity, which not only categorizes existing methods but also reveals the fundamental drivers—advances in architectures, datasets, and computational scale—behind this paradigm shift. Critically, our analysis identifies a bifurcation in the field's trajectory: one path toward efficient, real-time prediction, and another toward large-scale, generative world simulation. By pinpointing key challenges and proposing concrete research questions for both frontiers, this survey serves as an essential guide for researchers aiming to advance the frontiers of visual dynamic modeling.

Published in Transactions on Machine Learning Research (07/2025)
A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches
Ruibo Ming'
arrb22@mall.tsinguuaaddd
Tsinghua University, Fun
Zhewei Huang"
kzwer@pku.eduu cn
StepPun
Jingwei Wu
wwgingguei220mmllsssuuu
StepFon
Zhuoxuan Ju
nymphhea@stt.ppe pku. cn
Peking University, Fun
Daxin Jiang
djiang@stephuu.comm com
StepPon
Jianming Hu
Unjjmmmail.tainee edu.com cn
Tsinghua University
Lihui Pengl Tsinghua University
Mhuipeng@mailltsiinge cm
Shuchang Zhou
shuchangg www.:eewwwwwwwnowonoommmmmmmoace com
StepFun, Megoii Technology
Reviewed on OpenReview: https: // openreview. net/ forum? id= ZN4rzrHLNZ
Abstract
Future Frame Synthesis (FFS), the task generating subsequent video frames from con- text, represents core challenge in machine intelligence and a cornerstone for developing predictive world models. This survey provides comprchensive analysis of the FFS land- scape, charting its critical evolution from deterministic algorithms focused pixel-level accuracy to modern generative paradigns that prioritize semantic coherence and dynamic plausibility. We introduce a novel taxonomy organized by algorithmic stochasticity, which not only categorizes existing methods but also reveals the fundamental architectures, datasets, and computational analysis identifies a bifurcation in the field's trajectory: one path toward efficient, real-time prediction, and another toward largo-scale, generative world simulation. By pinpointing key challenges and proposing concrete research questions for both frontiers, this survey serves as essential guide for rescarchers aiming to advance the frontiers of visual dynamic modeling.
advances in this paradigm shift. Critically, our
1 Introduction
The goal of the Future Frame Synthesis (FFS) task is to generate future frames from a sequence of historical frames (Srivastava et al. 2015) or even a single context frame (Xue al. (2016), optionally guided by supplementary control signals. The learning objective of FFS is also regarded as central to building world
model (Ha & Schmidhuber 2018a Hafner et al.) 2023). FFS is closcly related to low-level computer vision
techniques, particularly when hesiziigging lemporally adjacent frames (Liu el. al. 2017 Wu et al. 2022b Hu
et al. 2023b). However, FFS differs from other low-level tasks by implicitly requiring more sophisticated
'ooth authors contributed equally. :Correspoddnng authors.

---

Published in Transactions on Machine Learning Research (07 (02))
Deterministic synthesis
Raw pixel space
Feature space
Recurrent networks
Future semantic segmentation
Convolutional networks
Future depth prediction
Transformers
Motion-based synthesis
Future frame synthesis
Motion-
Stochastic synthesis
Stochasticity
Disentangling components
controllable
modeling
synthesis
Stochastic distributions
Content and motion
Strokes
Probahilistic models
Foreground and backgroumd
Instructions
Generative
synthesis
Diffusion-based generation
Token-based generation
Structure of our taxonomy. We categorize future frame synthesis (FFS) approaches based on the degree of stochasticity in the modeling paradigm.
Figure 1:
understanding of scene dynamics and temporal coherence traits typically associated with high-level vision tasks. The key challenge is to design models that can achieve this balance efficiently, using moderate number of parameters to reduce inference latency and resource consumption, thereby making FFS practical for real-world deployment. This unique positioning underscores the integral role of FFS bridging the gap between low-level perception prediction, and high-level understanding & generation in computer vision.
Early FFS methods primarily followed two design approaches, and these algorithms commonly referred to video prediction methods. The first approach involves referencing pixels from existing framos- typically the last observed frame to synthesize future content. However, such methods inherently struggle to model the appearance and disappearance of objects. These methods tend to produce accurate short-term predictions but degrade over longer time horizons. The second approach focuses on gencrating future frames from scratch. While these methods hold the potential model object emergence and disappearance, they primarily operate at the pixel level. As a result, they often fail to capture high-level semantic context, which is essential for realistic and imaginative generation.
Prior to work, two surveys published in 2020 (Oprea et al. 2020 Rasouli 2020) provide comprehensive
overviews of early technical developments in video prediction. More recently, several surveys have emerged text-to-video generative models (Liu et al. 2024b) and long video generation (Li et al. 2024 Sun et al. 2024b). In contrast, our survey emphasizes recent advances and explores the interplay between predictive and generative methodologies. We argue that the future of long-term FFS lies in the synergistic integration of prediction and generation techniques. Such a unified approach intogrates contextual constraints with semantic understanding, enabling more robust. and coherent. synthesis.
The field of FFS is rapidly moving from traditional deterministic prediction to large-scale generative ap- proaches as the latest vidco generative models continue to emerge. These models have made important advances in parameter scales, generation lengths, control capabilities, and training strategies, injecting new vitality. As a foundation, we introduce the problem formulation and core challenges in Section 2 Our taxonomy is organized around the degree of stochasticity in modcling approaches. In Section 3 we present deterministic algorithms that aim perform pixel-level fitting based on fixed target frames. However, pixel-level metrics tend to drive models to average over multiple plausible futures, often resulting in blurry
2

---

Published in Transactions on Machine Learning Research (07 (07/2025)
predictions. In Section examine algorithms that enable stochastic motion prediction. These include approaches that inject stochastic variables into deterministic models, well as methods based on explicit probabilistic modeling. Such methods enable sampling from motion distributions, yielding diverse yet plau- sible predictions beyond the target frame. Given current FFS algorithms' limited generative capacity, espe- cially for high-resolution videos involving object appearance and disappearance, we discuss the generative FFS methods in Section [ These methods prioritize producing coherent long-term sequences over pixel-level accuracy. Figure visually illustrates the laxonomy and serves as a struetural guide for the subsequent sections.
In Section 6 wc explore the broad applicability of FFS in areas such as world model, autonomous driving, robotics, film production, meteorology, and anomaly detection. These use cases demonstrate the role of FFS in dynamic scene understanding and interaction. In Section 7 we review prior surveys on video prediction and diffusion-based video generation. We also clarify our distinct focus: a comprehensive analysis of FFS spanning from deterministic to generative paradigms, emphasizing the growing role of generative models in producing realistic and diverse future predictions.
2 Future frame synthesis
2.1 Problem definition
The FFS task involves predicting future frames conditioned on previously observed video content. The primary objective is to develop models capable of accurately capturing future visual dynamics. Formally, this task can be formulated conditional generative modeling problem: given observed frames Xt, the goal is to generate future frames Xtyt This relationship can be expressed as the conditional probability distribution:
Xtat1sta ~ IP( (taitt);
(1)
In Eq. ( & denotes the initial time step, to marks the end of the observed frame sequence, and t3 indicates the final time step for FFS. The key challenge is to learn mapping function that models the complex spatio-temporal dependencies across frames. Here, P(X (exs)) denotes the conditional probability distribution over future frames given the observed sequence.
Many FFS algorithms incorporate multi-modal data M, including depth maps, landmarks, bounding boxes, and segmentation maps, to enhance scene understanding. They may also include human control signals City such as text instructions or sketch-based strokes, which guide the model to generate future soquences following specific intended trajectories:
Xty P(X X Merta Otyt1st))
In multi-modal prediction methods, such the recent auto-regressive prediction Bai et al. 2024
Peng
et al. 2024 Ming et al. 2024), multi-modal information might serve as the learning target as well:
(Xestiit; Mcg+1ag) ~ P(Xe++1ds Meytiit, Xe Mester Cestltts)
(3)
2.2 Paradigms and architectures
In the domain of FFS, three paradigms have emerged deterministie, stochastic, and generative each rep- resenting distinct modeling approach. Early deterministic methods are designed to minimize pixel-wise error. This objective inevitably produces blurry averages, which in turn motivate stochastic formulations that explicitly model uncertainty. Furthermore, the need for long-horizon and high-level plausibility pushes the field toward fully generative paradigms.
3

---

Published in Transactions on Machine Learning Research (07/2025)
The deterministic paradigm emphasizes pixel-level fitting to fixed target frames, typically employing low- level computer vision urchitectures such as Convolutional Neural Networks (CNNs) (Krizhevsky et al. 2012). Recurrent Neural Networks (RNNs) (Rumelhart et al. 1985 Long Short-Term Memory (LSTM) (Hochre- iter & Schmidhuber 1997) and U-Net (Ronneberger ct al. 2013). Recently, the Transformer architecture has gradually challenged the dominance of traditional CNNs and RNNs. A landmark work, Vision Trans- former (ViT) (Dosovitskiy et al.) 2021), divides images into fixed-size patches and treats these patches as sequences input into the Transformer for processing. Swin Transformer (Liu et al., 2021) further introduces hierarchical Transformer structure that handles features different scales through local window self- attention mechanism and progressively expanding receptive fields. In low-level vision tasks, IPT (Chen et al. 2021) employs pre-trained Transformers to address various low-level visual tasks. And Models like TimeS- former [Bertasius el al., 2021) and ViViT (Arnab et al.) 2021) utilize Transformers to process spatio-lemporal information in videos by combining video frames and time steps. During this period, models generally pursue efficiency and smaller parameter scales. This is constrainod by the prevailing GPU computing power at the time and aligns with their primary goal of addressing short-term, low-resolution prediction tasks.
For deterministic models, optimizing pixel-level metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) (Wang ct al.; 2004) often leads to blurry outputs, as the models tend to average over multiple plausible futures. Regarding the evaluation metrics, we will delve into the discussion in the next subsection. Unlike deterministic models that usually optimize pixel-level metrics, the stochastic paradigm introduces randomness into the prediction process by incorporating stochastic variables or distributions to model the inherent uncertainty in video dynamics. This approach aims to capture the variability and unpredictabilityy of video sequences, often producing results that deviate significantly from the ground truth. Probabilistic models, such Variational Autoencoders (VAEs) (Kingma & Welling 2014) and Generative Adversarial Networks (GANs) (Goodfellow et al.) 2020),, are commonly used to achieve this. This approach does not necessarily aim to generate entirely new video content but rather to capture the variability in future outcomes. The generative paradigm, on the other hand, prioritizes the synthesis of coherent and plausible video sequences over pixel-level fidelity. It leverages advanced generative models, such as diffusson models and large language models (LLMs),, to produce diverse and imaginative future frames that capture complex scene dynamics, including object emergence and disappearance. Recently, diffusion models have emerged a powerful paradigm for generative tasks, including FFS. These modcls, such DDPM (Ho ee al. 2020 learn to generate data by progressively denoising random noise signal. DifTusion models like Video Diffusion (Ho et al. 2022) and SVD (Blattmann et al. 2023a have demonstrated remarkable ability to produce high-quality, diverse, and temporally coherent video sequences. By leveraging the gradual denoising process, these models can effectively capture complex scene dynamics and generate realistic future frames, even for long-term predictions. Furthermore, flow-matching models (Tipman et al. 2022 Dao et al.] 2023 and rectified flow (Liu et al. 2022) have gained attention as an alternative approach to generative modeling, offering improved efficiency and scalability compared to traditional diffusion models. The rise of diffusion models and large reto-regressive models is inextricably linked to the significant growth in computational resources in recent years and the widespread utilization of large-scale, high-resolution web video datascts. As research evolves, the boundaries between these paradigms continue to blur, with growing efforts to integrate their strengths and build more capable and versatile FFS systems.
2.3 Overall challenges and development trends
The field of FFS faces several longstanding challenges, including the need for algorithms that balance low- level pixel fidelity with high-level scene understanding, the lack of reliable perceptual and stochastic evalua- tion metrics, the difficulty of achieving long-term synthesis, and the scarcity of high-quality high-resolution datasots that capture stochastic motion and object emergence and disappearance. This section outlines these key challenges and sets the stage for further discussion.
2.3.1 Learning objectives and evaluation metrics
Low-level metrics such as PSNR and SSIM assess only the pixcl-wise accuracy of predictions. To optimize for these metrics, models are typically trained using pixcl-space losses such as l or lg. They often lead to blurry predictions that closely match the ground truth, rather than sharper and more realistic generations
4

---

Published in Transactions on Machine Learning Research (07/2025)
that deviate from a phenomenon known as the perception-distortion trade-off BBluu & Michaeli 2018 As result, researchers are increasingly exploring alternative evaluation metries, including perceptual met- rics (e.g,, DeeP'SM (Dosovitskiy & Brox 2016)) LPIPS (Zhang et al. 2018 and stochastic metrics (e.g.. IS (Salimans et al., 2016), FID (Housel et al. 2017)) ). These metrics are believed to better align with human perceptual judgments. However, even classiliers trained on human-annolated perceptual data show limited agrocment with human judgments of image quality (Kumar et al. 2022).
In visual domains, models are typically evaluated based on the perceptual quality of their generated outputs. However, for many practical applications, perceptual quality may not be the most critical factor. For instance, Dreamer-V3 (Hafner et al. 2023) and VPT (Baker et al., 2022) have successfully built effective world models using low-resolution frame sequences. Moreover, most visual representation learning met are developed using relatively low-resolution images (Radford et al. 2021: He et al., 2022). We are concerned that an excessive pursuit of visual quality may bias model selection toward architectures that overfit low-level features. Beyond aligning with human perceptual judgments, evaluation metries should also be designed to assess a model's 's capacily to capture scene dynamics and temporal variations. Moreover, we also need to focus on how to leverage the capabilities of the FFS model to assist us in accomplishing more tasks (Agarwal ct al. 2025
Regarding the quality assessment of video generative models, some more comprehensive evaluation systems have been recently established. VBench (Huang et al. 2024) benchmark suite has been proposed to address these concerns by providing a comprehensive evaluation framework for video generative models. VBench evaluates video generative models comprehensively, covering perceptual quality, dynamics, temporal consis- tency, content diversity, and prompt alignment for holistic performance assessment. FVMD (Liu et al., 2024a) designs explicit motion features based on key point tracking, focusing on evaluating motion consis- tency in video generation. VBench-2.0 (Zheng et al., 2025) further focuses on the intrinsic faithfulness of generative models, such as whether they adhere to real-world principles, including physical laws and com- mon sense reasoning. Recently, some of the work has increasingly utilized multi-modal LLMs to evaluate the generated results. For example, ViStoryBench (Zhuang el al.) 2025) leverages GPT-4o (Hurst et al.] (024)) to assess the inteructionfollowwie ability generative models from multiple aspects. As people's pursuit of generating realism gradually increases, the ovaluation of real physical interaction and real-scene simulation will be promising research topic. Moreover, for FFS, we need to consider whether the generated results are reasonable given the observed results.
Even with improved evaluation metrics, optimizing them during training remains significant challenge. During training, researchers often use pre-trained ImageNet classifiers as feature extractors (Johnson et al. 2016 Kumar et al. 2022) to compare generated outputs with ground truth, thereby optimizing for both low- level and high-level features. Additionally, various GAN-based loss functions have been proposed enhance the perceptual quality of generated outputs (Huang et al.) 2017 Zhang et al. 2020b). Diffusion models seem to bring significantly stronger realism, but new issues such as slow convergence and high resource consumption still require people to invest effort in exploring.
2.3.2 Long-tern terg-eemman synthesis
Despite significant progress in short-term video prediction, synthesizing events over extended time horizons remains challenging due to long-term dependencies and complex object interactions in dynamic scenes. Naive application of short-term models in an iterative manner often leads to rapid quality degradation (Wu et al., 2022b Hu et al. (023b). Low-level vision methods such as DMVFN (Hu et al. 2023b) might primarily focus on video prediction for limited number of future frames, typically within 0.3 seconds. Owing to limited model capacity for forming a comprehensive understanding of the real world, most existing video synthesis models primarily model pixel-level distributions. When dealing with natural videos over long durations, these models strugglc to predict object dynamics while preserving visual quality. One promising direction is to incorporate high-level structural information (Villegas et al.] 20177] Ming et al. 2024). Leveraging such higher-order representations helps models retain key details and maintain temporal consistency over extended time scales. Utilizing the prior of large diffusion models, such as SVD (Blattmann et al.) 2023a) enables the generation of high-definitin videos ranging from 2 4 seconds. However, we need appropriate techniques to preserve the capabilities of these pre-trained models while unlocking their conditional gen-
5

---

Published in Transactions on Machine Learning Research (07/202) /025)
Action
3 4
4
4
3
3 4
4
4
3
43
43
5g
5 8
5
5
8
S
Moving MINIS
Something Somnething
Vimeo-gok
WebVid-roM
Dideeeeiidee-oMM
BDDiook
Figure 2: Dataset samples. We adjust the absolute resolution of the image for visualization while main- taining the relative size relationships.
eration functionalities (Wag et al. 2025b Yang et al. 2025). When generating longer videos, properly handling physical interactions presents greater challenges
(Agarwal al. 2025)
2.3.3 Generalization
The interplay between data volume and model complexity jointly determines the upper bounds of an algo- rithm'' 's performance. Despite the vastness of video data available on the Internet, the scarcity of high-quality video datasets suitable for video synthesis remains a limiting factor. Existing datasets often suffer from sim- plistic data distribution, low resolution, and limited motion diversity. These limitations hinder the ability of current vidco synthesis models to handle high-resolution content and large motion scales, thus restricting their practical utility to diverse and unseen scenarios. High-resolution video synthesis inherently challeng- ing and requires substantial computational resources (Blattmann et al., 2023b). The computational burden further complicates the real-time deployment in practical applications.
2.4 Datasets
The advancement of video synthesis models is highly dependent on the diversity, quality, and characteristics of training datasets. A common observation is that the suitability of datasets varies with their dimension- ality and size: lower-dimensional datasets, often smaller in scale, tend to exhibit limited generalization. Conversely, higher-dimensional datascts offer greater variability, contributing to stronger generalization ca- pabilities. Generally, video complexity and resolution have increased in recent years, as shown in Figure 2 In Table ! we summarize the most widely used datasets in video synthesis, highlighting their scale and available supervisory signals to provide comprchensive overview of the current dataset landscape. For datasets lacking detailed reports in their original papers or project pages, we estimate missing values using mean median statistics to ensure consistency across the analysis.

---

Published in Transactions on Machine Learning Research (07/2025)
Datasct
Category
# Videos
# Clip Frames
Resolution
Extra Annotations
KTH Action (Schuldt
2001
Human
391
95'
160 120
Class
Caltech Pedestrian (Dollar
2011
Human
1:37
824
640 480
Bounding Box
IMDB51 [Kuchne ct
Human
766
93"
414 x 404
Cluss
SJTU 4K
[Sng ct.
2013
General
13
300
3840 x 2160
KITTI (Griger et al.
2013
Traffie
151
323*
1242 x 375
OF, BBox, Sem, Ins, Depth
J-HMDB (Juuann al 10
Human
928
34'
320 x 240
OF, lus, HJ, Class
Peun Action Zhang el al
20
Human
2, 326
70'
480 x 270
Human Joint, Class
Moving MNIST
IS
(Srivsstavs
2015
Simulation
10, 000
20
64 64
MSR-VTT
Xu ct al.
2016
General
10, 000
408"
320 240
Class, Text Semantic, Instance, Depth
Cityscapes Conits et al.
2016
Traffic
46
869*
2048 x 1024
DAVIS17 [
Pont- I laser of
2011
General
150
73*
3840 x 20261
Semantic
SM-MNIST T (Denton & Fergus
2018
Simulation
customize
customize
64 x 64
nuSoenes Carserr cl. al. 2020
Traffic
1, 000
40
1600 x 900
BBox, Semantic
QST (Zhang et al. 2022aa
Sky
1, 167
215
1024 x 1024
11KKUSSP Sim et sL.
2021
General
4, 108
65
4096 x 2160
Youtube Driving (Zhang et
2022
Traffic
134
variable
variable
Action
OVenDV-2K Yang ct nl. 2029
Traffic
2, 139
variable variahle
variable
Text
MagonoMagic Yuan el al 2025b
General
2, 263
1920 x 1080
Text
UCF101 SSomro et al. 2012
Human
13, 320
187*
320 x 240
Class
Sports-1M 1M [Karpaliy et al.
2014
Human
1, 133, 158
variable
variable 6410 x 512
Class
Robotic Pushing [ Fimn et au.
2010
Robot
59, 000
25
Class
YouTube-&M (Abi-El-Hajja of al.
20
General
8, 200, 000
variable
variable
Class
Something-Sometti GGoval of 201
Object
220, 847
45
427 x 240F
Text
DiDeMo (Anee Hlnddiics ot 2017
Human
10, 464
variable
variable
Text
Sky Time-lupse Xiong et 2018
Sky
38, 207
32
640 360
ShapeStacks (Groth ri 2018
Simulation
96, 000
16
224 x 224
Semantic
Vimeo-yUK (Xue et nl. 2019
General
91, 701
448 x 256
D2-City [Che al. 2019
Traffic
11, 211
750
1080p / 720p
BBox
How Tow0011011 (Miecli et ol.
2019
General
136, 600, 000
variable
variable
Text
Kinetics-700 (Carreira et BL.
2019
Human
650,
variable
Class
RoboNet Daseri al] 2015
Robot
161, 000
93*
BDD100K
al.
Traffie
100,
1175*
1280
720
BBox, Sementic, Depth
VGG-Sound
Chen et al.
2020
General
199, 467
variable
variable
Class, Audio
WebVid-10%
Bain el.
General
10,
449*
336
Text
Time-lapse-D
Xue al.
Sky
16,
variable
variahle
Class
HD-VILA-100S1
al.
21122
General
100, 000, 000
variable
1280 720
Class, Text
CelebV-HQ Zhu al.
2022
Facial
35, 666
variable
512 512
Action
Celet
Yu
2023
Facial
70, 000
variable
512 512
Action, Text
SportsSlomo
Chen & Jang
2024
Human
130, 000
1280 720
Pa
Chen nl.
20/44c
General
70, 000, 000
variahle
1280 720
Class, Text
InternVideo2 Wang
20243
General
000, 000
variable
variable
Action, Text
VidGen- 1M
Tan al.
21/2
General
000, 000
variable
1280 720
Text
ConsisID
Yuan et al.
20255
Human
54, 239
129*
1280
720
Text
Video- IUM [hhag sl.
2025
General
10, 000, 000
variable
variable
Text
denotes the mean value. denotes the median value. OF: Optical Flow, BBox: Bounding Box, Sem: Semantic,
Ins: Instance, HJ:: Human Joints)
Table 1: Summary of commonly used FFS datasets, including dataset category, total number of videos, frame count for each video clip, image resolution, and additional annotations.
Challenges. 1. Unifying the organization of image and video data. A large proportion of computer vision research has historically focused on the image modality. As a result, image datasets are often more carefully curated and contain richer annotations. Representative largo-scale image datascts include YFCC100M Thomee et al. 2016), WIT400M (Radford et al. 2021), and LAION400M (Schuhmann et al., 2021). Given the scale of available image data, it is important to effectively leverage knowledge from foun- dational image models. When incorporating video data into training pipelines, it is often necessary to filter out low-quality segments and select an appropriate sampling frame rate.
2. Determining the proportion data from different domains. Computer graphics composite data, 2D anime data, real-world videos, and vidcos with special effects exhibit vastly different visual characteristics. Morcover, standardizing data from diverse sources to fixed resolution is challenging due to varying aspect ratios and the presence of resolution-dependent details such subtitles and textures. Many frame synthesis
7

---

Published in Transactions on Machine Learning Research (07/)025)
methods are sensitive to resolution partly due to the correlation between resolution and object motion
intensity {Sim et al. 2021 Hu et al. 2023b Yoon et al. 2024
3 Deterministic synthesis
Method
Publication
Main Ideas
Formulate precipitation nowcasting as spatio-temporal sequence forecusting problem, propese the convolutional LSTM build end-to-eud model. Use recurrent CNN with both bottom-up and top-down connections, with each neural layer making local residual predictions. Introduce
11
Shi et al.
2015
NoottNtt
ICLR'17
Lotter
2017
PreiHUNN
spatio-temporal L.STM unit.
Neu:IPS'17
Wang et al.
2017
that extracts and memorizes spatial and temporal representations simultaneously. Integrate 3D convolutions into RNNs to enhance local motion modeling, and make the present memory state interact with its long term historical records. Focus on enbaucing long-term action planning ability, and use spatio-temporal downstmpling to forecast at different scales.
63d-LSTA
ICLR'19
Wang et al.
2019
MST pred
BMVC'2
Viillr- Colrales et al.
2022
Multi-Scale Ad
Propose three feature learning strategies to address blurriness in FFS:
ICLR16
Mathieu et al.
2016
a multi-scal scale architerture, adversarial training, and an image gradient difference loss.
PrecCNN
Design a cascade multiplicative unit that provides more operations for previous frames, and capture the temporal dependencies through stacked operations. Learn synthesize videa frames by warping pixel values from existing ones, and unify video interpolation and extrapolation within single CNN framework. Propese simple CNN video prediction model without complicated tricks, and provide insights for selecting different architectures.
LICAI'18
Xu et al.
2018
DVF
ICCV'17
Liu
2017
SPmVP
CVPR'22
Gao et al.
2022
SDD
Learn motion vector and kernel for each pixel to synthesize the predicted pixel, and inherit the merits of both vector-based and kernel-based approsches. Decouple the background scene and moving objects, construct future frames using nou-rigid deformation of the background and alline trausformation of objects. Solve FFS as an optimization problem, with a pre- praitrained VFI module to construct lunction. By eliminating the domain gap, OPT is robust in general scenarios. On the basis of DVF, multi-srale coarse-to- fine prediction is added, and the imput processed by a dynamic routing subuetwork at the inference stage. Present a new efficient building block of Transformer based models for FFS, along with three competitive variants for video prediction.
ECCV118
Reda et al.
2018
r FVS
CVPR'20
Wu et al.
2020
OPT
CVPR'22
Wu et al DMVNN
2022b
CVPR'23
et. al.
2023b
VPTR
IVC'23
Bilodeau
2023
$28
ICCV'17
Introduce the task of predicting semantic segmentations of future frames, and show that predicting future segmentations is substantially better then sogmenting predieted frames. Decompoee the scene layout (semantic map) and motion (optical flow) ) into layers, and hope to explicitly represent objects and learn their clase-specific motion. Predict the depth maps of future frames using two-branch structure.
Luc et al.
2017
SADM
CVPR'21
Bei al.
2021
MAL.
WACV'23
Liu et al.
2023
One branch bandles future depth estimation, and the other aids image reconstruction.
Table 2: Overview of deterministic synthesis methods.
3.1 Raw pixel space
In short-term FFS, approaches operating in the raw pixel space have achieved promising results. In this section, review representative methods and discuss the associated challenges. We will first introduce the evolution of the architecture of popular backbones, and then highlight one of the most popular methods in video prediction in pixel space, motion-based synthesis.
3.1.1 Recurrent networks
Due to the inherent temporal nature of video data, RNNs became a natural choice for early future frame synthesis tasks. The core idea of such methods is to use RNN's memory units to model the temporal depen- dencies between frames. However, the main challenge lies in how to effectively extend RNN's ability to pro- coss one-dimensional sequences to high-dimensional spatio-temporal data like vidcos. PredNet (Lotter et al. 2017) pioneered the exploration of recurrent neural networks in video synthesis, drawing inspiration from predictive coding in neuroscience and employing a recurrent convolutional network to process video features effectively. Building on this foundation, ProdRNN (Wang et al.] 2017) introduces significant improvements by modifying LSTM architecture with a dual- memoryy structure, aiming to enhance ppatio-tempora modeling. Despite these advances, the model still faces challenges such gradient vanishing in video synthesis tasks.
8

---

Published in Transactions on Machine Learning Research (07/2025)
To address these limitations, ConvLSTM (Shi et al, 2015) emerges a pivotal model by ingeniously inte- grating LSTM with CNNs to effectively capture motion and spatio-temporal dynamics development that has significantly influenced subsequent video synthesis models. E3d-LSTM (Wang et al. 2019 further ad- vances the field by incorporating 3D convolutions into RNNs and introducing a gate-controlled self-attention module, thereby significantly improving long-lerm synthesis capabilities. However, the increased computa- tional complexity introduced by 3D convolutions may offset the performance gains in certain applications. network that operates
MSPred (Villar-Corrales et al. 2022) proposes a hierarchical convolutional- at multiple temporal frequencies to predict future video frames, as well as other representations such as poses and semantics.
Challenges. Despite their effectiveness in capturing temporal dependencies, recurrent nelworks face sev- eral challenges in video prediction tasks. Their inherently sequential nature, which enables feame-by-frann modeling, can result in high computational campexity-particulann evident from the significantly higher FLOPs and lower FPS observed in recurrent-based models compared to their recurrent-free counterparts (Tan et al. 2023). Additionally, recurrent networks are prone to gradient vanishing and exploding problems, which can severely hinder their ability to learn long-term dependen- cies Gaoo et al. 20222. These challenges underscore the need for alternative approaches that balance effi- ciency and performance such as recurrent-free models, which have demonstrated promising results across various video prediction tasks.
in high-resolution sccnarios. This is
3.1.2 Convolutional networks
CNNs play an instrumental role in the evolution of video synthesis technology. The progress began with Multi-Scale AdvGDL (Mathieu et al.) 2016), initiating a series of significant advances in the field. Fol- lowing this, PredCNN (Xu et al. 2018) establishes a new benchmark by outperforming its predecessor PredRNN (Wang al. 2017) across various datasets. The introduction of SimVP (Gao et al.] 2002) marks
another milestone in convolutional approaches to video prediction. Inspired by the advances of ViT (Doso-
vitskly et al. 2021), this approach introduces simplified CNN architecture to extract continuous tokens, demonstratinn that such a configuration can achieve comparable performance in video synthesis.
Challenges. Although simple to implement and fast, pure 2D CNN-based models are not well-suited for spatially shifting the pixels of input frames. CAIN (Choi et al., 2020 and FLAVR (Kalluri et al. 2023) respectively introduce channel attention and 3D U-Net architectures for intermediate frame synthesis, but they do not fully replace explicit pixel motion approaches such as kernel-based and flow-based methods. Moreover, in pursuit of efficiency, most CNN-based models used for FFS maintain relatively small parameter count, typically under 60M (Tan el al. over 1.5B parameters (Blattmann et al. 2023a in order to fully leverage large-scale datasets. Performing effective scaling CNN-based models remains a significant challenge. It is speculated that short-term prediction models for high-resolution, real-time applications and those aiming to leverage large datasets for enhanced generation capabilities will follow diverging development paths.
2023 By contrast, video dilfusion models have been scaled up to
3.1.3 Transformers
Since the groundbreaking design of ViT (Dosovitskiy et al. 2021 which applied pure Transformer directly to sequences of image patches to extract continuous tokens, the use of Transformers in frame synthesis has attracted significant attention. Video frame interpolation a task closely related to FFS (Luu et al. 2017). Shi et al. (2022) and Lu al. (2022) propose Transformer-based video interpolation frameworks that leverage self-attention mechanisms to capture long-range dependencies and enhance content awareness. The primary difference from convolution is that the self-attention mechanism can dynamically compute the correlation between cach patch and all other patches, thereby capturing global information, whereas convolution extracts features within local regions using fixed kernels. These methods further introduce innovative strategies, such as local attention in the spatio-temporal domain and cross-scale hindow-based attention, to improve performance and effectively handle large motions. Ye & Bilodcau (2023) present an efficient Transformer model for video prediction, leveraging a novel local spatio-temporal separation attention mechanism, and compares three variants: fully auto-regressive, partially auto-regressive, and non-autooregreessve,, to balance
9

---

Published in Transactions on Machine Learning Research (07/2025)
performance and complexity. Numerous ongoing studies contimue to explore the use of Transformers for modeling inter-frame dynamics under varying motion amplitudes and for addressing challenges in high- resolution frame synthesis
(Park et al. 2023 Zhang et al. 2024).
Challenges. Many researchers believe that Transformers perform particularly well on large-scale datasets (Zhai et al.) 2022 Smith et al.) 2023). However, most of the existing studies focus primarily high-level vision tasks. Furthermore, the success of Transformers in many tasks depends heavily on fully leveraging the capabilities of foundation models. In the context of image synthesis, best practices remain unclear (Li et al. 2023). These challenges also explain why generative methods, especially token-based meth- ods that directly draw on LLM architectures and training paradigms, which we will explore in Section 5.2 have become a more attractive direction for exploration.
3.1.4 Motion-based synthesis
Optical flow is a technique used to describe the movement of pixels between consecutive frames in a video. By calculating the displacement each pixel from one frame to the next, optical flow enables the warping of pixels from the current. frame to generate synthetic frames that represent the near future (Liu et al. 2017). Flow-based methods can be viewed as complements of kernel-based approaches (Niklaus et al, 2017] since the latter typically constrain the motion of pixel to a relatively small neighborhood (Cheng & Chen 2021 SDC-Net (Reda et al.) 2018) proposes hybrid approach that inherits the strengths of both vector- vector-based based and kernel-based methods. FVS (Wu et al. 2020) enhances synthesis quality by incorporating supplementary information, such as semantic maps and instance maps from input frame sequences. While effective, this method introduces challenges due to increased data modalities and higher computationa demands. OPT (Wu et al.) 2022b) estimates optical flow through an optimization-bant based approach. By iteratively refining the current optical flow estimation, the quality of the next frame can be significantly improved. This approach effectively leverages knowledge from the off-the-shelf optical flow models [Tedd & Deng, 2020) and video frame interpolation methods (Jiang et al. 2018 Huang et al. 2022b Although training is not required, the iterative optimization process during inference incurs substantial computational cost. DMVFN (Hu et al.) 2023b) improves dense voxel flow (Liu et al. 2017) estimation by dynamically adapting the network architecture based on motion magnitude. DMVEN further confirms the effectiveness of a coarse-to- toe fine, multi-scale, end-to-end model in addressing short-term motion estimation.
Challenges. Optical flow estimation remains active and extensively studied rescarch topic (Teed & Deng 2020 Huuang et al. 2022a Sun et al.] 2022; Dong & Ful 2024).. However, mainstream optical flow models trained on synthetic data with heavy augmentations often diverge significantly from the scenarios targeted by FFS. Morcover, the lcarning objectives of these models are not aligned with the goal of generating optical flow that facilitates accurate pixel warping and high-quality image synthesis. Xue et al. (2019) point out that for different downstrream tasks, it often necessary to fine-tune or even train a flow estimation network from scratch. Interestingly,
pettherrprforminggg optical flow networks may lead to degraded image synthesis results, they often emphasize ambiguous regions such occlusions and may lack sufficient spatial resolution (Niklaus& & Liu 2020 Huang et al. 2022b). In real-world scenarios, obtaining ground-truth optical flow labels remains a major challenge.
We believe that near-future frames can be synthesized with increasing accuracy as optical flow methods continue to improve. However, integrating optical flow methods into long-term video generation remains a significant challenge (Liang et al.) 2024). Optical flow is typically limited to predicting pixel motion over very short time spans and cannot assist in generating novel video content.
There are still ample opportunities for exploration in the representation of motion. For example, Gaus- sianProdiction Zhao et al. (2024) explores the use of 3D Gaussian representations to model the appearance and geomelry of dynamic scenes, enabling image rendering for future scenarios. MoSca (Lei el al., 2024) proposes the Motion Scaffold representation, which lifts the predictions of 2D foundational models (such as depth estimation, pixel trajectories, and optical flow) to 3D and optimizes them using physics-inspired constraints. Ultimately, MoSca reconstructs the geometry and appearance of the scene with a set of dynamic
Gaussians (Wu et al. 2024b). Building on video diffusion models, V3D (Chen et al. 2024g) leverages their
10

---

Published in Transactions on Machine Learning Research (07/2025)
learned world simulation capabilities to perceive the 3D world, introduces a geometrical consistency prior, and finetunes the video diffusion model into multi-view consistent. 3D generator. Seamlessly integrating new representation methods with existing frameworks presents novel challenges.
3.2 Feature space
Synthesizing in the raw pixel space often overburdens models, as it requires reconstructing images from scratch-a task especially challenging for high-resolution video datasets. This challenge has prompted shift in focus among researchers. Rather than grappling with the complexities of pixel-level synthesis, many studics have shifted toward high-lovel feature synthesis in the feature space, focusing on representations such as segmentation maps and depth maps. These approaches provide a more efficient means of handling the complexities of video data (Oprea et al. 2020).
Future semantic segmentation. Future semantic segmentation represents a progressive approach to video synthesis, primarily focusing on gencrating semantic maps for future vidco frames. This methodology departs from traditional raw pixel forecasting by utilizing semantic maps to narrow the synthesis scope and enhance scene understanding. In this context, the S2S model (Luc et al.) 2017) emerges as pioneering end-to-end system. It processes RGB frames along with their corresponding semantic maps as both input and output. This integration not only advances future semantic segmentation but also enhances video frame prediction, demonstrating the advantages of semantic-level forecasting. Building on this foundation, SADM (Bei et al.) 2021] introduces further innovation by integrating optical flow with semantic maps. This fusion leverages optical flow for motion tracking and semantic maps for appearance refinement, using the former to warp input frames and the latter to impaint occluded regions.
Future depth prediction. Depth maps, as 2D data structures that encode 3D information, can provide models with enhanced perception of the 3D world at minimal computational cost. Predicting future depth maps can benefit FFS tasks. MAL (Liu al. 2023) introduces meta-learning framework with a two-branch architecture, comprising future depth prediction and an auxiliary image reconstruction task. This framework improves the quality of synthesized future frames, particularly in complex and dynamic scenes.
Challenges. Future prediction in the feature space presents significant challenges due to the complex interplay between temporal dynamics and spatial context. Models must capture intricate motion patterns and accurately predict depths or semantic regions, which requires deep understanding of 3D scene structures and object interactions. Ensuring temporal consistency and spatial precision while handling occlusions, perspective changes, and complex backgrounds is crucial. The use of high-resolution feature maps and large-scale annotated datasets further increases computational and data demands. Generalizing to unseen scenes and objects remains major challenge, requiring robust models capable of adapting to diverse visual appearances and contexts. These challenges underscore the need for innovative approaches, such as meta- auxiliary learning, to enhance future prediction capabilities.
4 Stochastic synthesis
In its early stages, video synthesis was primarily regarded as a low-level computer vision task, with an emphasis on using deterministic algorithms to optimize pixel-level metrics such as MSE, PSNR, and SSIM. However, this approach inherently limits the creative potential such modcls by constraining possible motion outcomes to a single, fixed trajectory (Oprea et al., 2020). In response to this limitation, the field of video synthesis has undergone paradigm shift from relying short-term deterministic prediction to embracing long-term stochastic generation. This transition acknowledges that, although stochastic synthesis may produce results that deviate significantly from the ground truth, it is essential for fostering a deeper understanding and enhancing creativity in the modeling of video evolution.
In this scction, we explore stochastic synthesis methods, including GANs and VAEs, which were originally categorized for their emphasis on modeling randomness and uncertainty in video motion. Although these models possess generative capabilities, they were originally more closely associated with stochasticity due to
11

---

Published in Transactions on Machine Learning Research (07/2025)
Method
Publication
Main Ideass
VPN
Model the temporal, spatial, and color structure of video teasors, and eucade it as a four-dimemsioall dependeacy chain. Propaee a stochastic variational video profiction (SV2P) method, providing effective stochastic multi-frame predietion for real-world videos. Introduce conditional variational spproach to model the stochasticity. Learu representation that can be decoded to future segmentation, depth and optical flow. Iutroduce stochustic temporal model using a residual update rule in a latent space, inspired by differential equation discretiaaion schemes.
ICML/17
Kalchbraueer et al.
2017
ICLR'18
[abarizzdee
2018
PFP
2CCV220
Hu et al. 2020
SRVP
ICML'20
Pranceschi et
2020
PhyDNNet
Disentangle PDE dynamics from unknown complementary information, and propose a recurrent physical cell to perform "Donstrntrr prediction in latent space. Decompise video predittion: Use a VAE to model future human pisss,
CVPR22)
Guen & Thome
2020
TPK
ICCV'17
Walker et al.
2017
and a GAN to graemate future frames conditioned on these poses.
SVG
Learn prior model ancertainty generate frames by drawing samples, and combine them with deterministic estimate generate varied and sharp results. Model mapping function from using parefully-desinnes GAN and spatio-tempore arversarial objective. Combine the latent varistional variable model and adversarialll-trrinn to produce predictions that look more realistic and better cover the raage possible futures. Employ two discriminators Prediet future and past. Inuues while cuforcing the cousistency of bi-diree di-directional prediction. Argue that the blurry predictions of VAEs may caused by underfisting ing. and suggest
ICML/18
Denton & Fergus
2018
Vil2Vid
video (e.g., segmentation) to photoreulistic video,
New1PS'18
Wing at al.
2018
SAVP
models,
Lee et al.
2018
preprint
Relrospect Vele GAN
identify fake frames and fake contained image sequences.
CVPR11
Kwon & Park
2019
TTNN
ICCV'19
Castrejon ot al.
2019
increasing expressiveness of the latent distribations and using higher capacity likelihood models. Attempt to address the underfitting issue on large and diverse datasets,
GHVAE
CVPR221
Wu et al.
2021
greedily training each level of a hierarchical autoencoder to learn high-quality video predictions. Propose video representaionn network that utilizes INRs and meta-netwott to generate diverse novel videos, demonnstraiing superior performance in various video based geueralive tasks Introduce INRs into GAN to culance motion dynamics, and utiline motion diseriminator that effectively detects unnatural motions.
INR-V
TMLR22
Sen et al.
2022
DIGAN
ICLR'22
Yu
2022
StyleGAI--
Extend the paradigm of neural representations to build contimous-time generator, and design holistic discriminator that aggregates temporal information by concatenating frames' features, Develop an action-conditinn conditioned vidmo prediction model that explicilly models pixel motion. The model generalizes to unseen objects by decoupling motion and appearance. By providing appearance and motion information as conditions, reduce the prediction uncertainty of equally probable future.
CVPR'22
Skorokhooor et al.
2022
CINN
16
Finn et al. 2016
AMCC-CAN
ICML'18
Jang
2018
MoLoGAN
Generate video by mapping sequence of random vectors to sequence of video feames, and introduce a novel udversarial learning scheme utilising both image and video discrimimatos.. Study how to store abundant long-tern term contexts, and recall suitable motion context, especially complex motions from limited inputs.
CVPR118
Tulyakov el al.
2018
LMC
CVPRU11
[Lee et al.
2021
SLAMP
ICCV
Focus on learning stochastic variahles for separate content and motion.
Alsan et al.
2021
XMMVP
ICCV'23
Coustruct appearance motion natrices to decouple motion and appearance..
Zhong et al.
2023
LEO
Represent motion as sequence flow maps and synthesize videos in the pixel space, and utilize the lateut motion diffusion model to model the motion distribution. Decompose fulure frames into spatial content and lemporal motions. Predict temporal notions based on Net diffusion model.
1JCV224
Wang
2024c
DVD
AAAF21
Shen al.
2024
DrNet
11urIPS17 Decompose video frames into stationary and time-varying components.
Deston et al.
2017
DVGFC
Tackle the severe ill- posedess of luman action video prediction with sto-stage framnwwrrk: generating human pose sequence from random noise, then creating the human action video. Predict the future states of independent entities while reasoning about their interactions, and then gynthesize future frames.
ECCV18
Cai et al.
2018
CVP
ICCV'19
Ye et al.
2019
OCVPVV
1CIP'23
Decouple the processing of temporal dynamics and object interactions.
Tillar-Corzlle
2023
SlotForner
ICLR'33
Model ggatio-tempent relationships by ressoning object features, and predict ancurate future states of objects.
Wi et a 2023
0K1D
L4DC'23
Decompose video into moving objects, their attributes and the dynamic modes..
Comas et al.
2023
MOSO
CVPR/33
Identify motion, scene, and object as the pivotal elements of a video.
Sun
2023
Table 3: Overview of stochastic synthesis methods.
their primary focus on capturing the inherent variability and unpredictability of video sequences. Current research prefers to simulate the complex dynamics of the future using end-to- aad-to-end generative modeling strate- gics rather than by introducing stochastic distributions or probabilistic models into deterministic algorithms to obtain diverse outcomes. In the following Section 5 we will discuss generative synthesis approaches, such diffusion models and auto-regressive models, which are explicitly designed to prioritize the generation of
12

---

Published in Transactions on Machine Learning Research (07/)025)
diverse, high-quality video content. This distinction reflects the evolving focus of video synthesis research, shifting from an emphasis on stochasticity to a broader emphasis on generative capability.
4.1 Stochasticity modeling
Uncertain object motion can be modeled either by incorporating stochastic distributions into deterministic frameworks or by directly employing probabilistic models.
Stochastic distributions. In the early stages, VPN KKachhrenner et al. 2017 employs CNNs to perform
multiple predictions in videos based on pixel distributions, while SV2P (Babaeizadeh et al. 2018 enhances
an aedion-conddiiooed model (Finn et al., 2016) by introducing stochastic distribution estimation. Shifting the focus to a more holistic representation of video elements, the PFP model (Hu et al., 2020 proposes a
probabilistic approach for simultaneously synthesizing semantic segmentation, depth maps, and optical flow. Additionally, SRVP (Franceschi et al. 2020) leverages Ordinary Differential Equations (ODEs), while PhyD-
Net (uunn & Thome 2020)) employs Partial Differential Equations (PDEs) to model stochastic dynamics. A potential drawback lies in their assumption that physical laws can be lincarly disentangled from other factors of variation in the latent space an assumption that may not hold for all types of videos.
Probabilistic models. Building on the pioneering work Multi-Scale AdvGDL (Mathieu et al. 2016 adversarial training has substantially advancod FFS tasks by improving the prodiction of uncertain object motions. Similarly; vRNN (Castrejon et. al. 2019 and GHVAE (Wu et al., 2021 enhance VAΣs through the incorporation of likelihood networks and hierarchical structures, respectively, thereby contributing new dimension to the ongoing evolution of stochastic synthesis methods.
To address the challenges of pixel-level synthesis, several studies introduce intermediate representations. S2S (Luc et al. 2017) and Vid2Vid (Wang et al. 2018) incorporate adversarial training into future semantic segmentation frameworks. Additionally, the TPK model (Walker et al.) 2017) leverages a VAE to extract human pose information, followed by a GAN to predict future poses and frames. is worth noting that directly modeling stochastic distributions tends to yield broader predictive coverage but often results in poor visual quality. In contrast, probabilistic models can produce sharper results, but they often face challenges such mode collapse, Iraining instability, and high compulational cost. Bridging these two meChodologies, SAVP Lee et al. (2188) integrates stochastic modeling with adversarial training, achieving a balance between broad predictive diversity and improved visual quality.
cases of unforeseen events such as colli- sions, SVG (Denton & Fergus] 2018) models trajectory uncertainty using both fixed and learnable priors, effectively blending deterministic and probabilistic approaches. In similar vein, but with an emphasis temporal coherence, Retrospective Cycle GAN (Kwon & Park 2019) introduces sequence discriminator to detect fake frames. Building on the paradigm of implicit neural representations (INRs) for video (Sen) et al.) 2022), ; this concept of scrutinizing frame authenticity is further extended in DIGAN (Yu et al. 2022), where the focus shifts to a motion discriminator aimed at identifying unnatural motions. StyleGAN- V (Skorokhodov
Recognizing that object motion is largely deterninistic-exxeeeee
al. 2022 highlights motion consistency from a different perspective by incorporating continuous motion representations into StylcGAN2 (Karras al. 2020), enabling consistent generation in high-resolution settings.
Challenges. Although stochastic models are capable of capturing broad spectrum of plausible futures, they often struggle with poor visual quality and increased computational demands. Direct modeling of stochastic distributions often leads to blurred outputs, whereas probabilistic models may encounter issues such as mode collapse and training instability. Striking a balance between diversity, visual fidelity, and computational efficicncy remains a significant challenge. Moreover, the assumption that physical laws can be lincarly disentangled from other factors of variation may not hold across all types of videos, underscoring the need for more adaptable and generalizable models.
13

---

Published in Transactions on Machine Learning Research (07/2025)
4.2 Disentangling components
Stochastic synthesis algorithms primarily focus on modeling the randomness inherent in motion. However, this focus often overlooks the processes of object emergence and disappcarance in vidcos. As a result, many studies isolate motion from other video elements or artificially constrain its evolution, aiming to better understand motion dynamics while reducing the complexity of real-world scenarios.
4.2.1 Content and motion
Vidco synthesis algorithms address the inherent complexity of natural vidco sequences by emphasizing intri- cale visual details. To this end, they aim model appearances through fine-grained local information while simultaneously capturing the dynamic global content of videos. However, in applications such as robotic nav- igation and autonomous driving, understanding object motion patterns takes precedence over visual fidelity, This shift in priorities has driven the development of algorithms that emphasize object motion prediction and the disentanglement of motion from appearance. A notable early work, CDNA (Finn et al. 2016), sets a precedent by explicitly predicting object motion. It maintains appearance invariance, enabling general- ization to unscen objects beyond the training set. MoCoGAN (Tulyakov et al., 2018) learns to discntangle motion from content in an unsupervised manner, and the use of separate encoders for content and motion has since been widely adopted in video prediction models. This concept is further explored in LMC (Lee et al. 2021), where the motion encoder predicts motion based residual frames, and the content encoder extracts features from the input frame sequence. MMVP (Zhong et al.) 2023) takes a different approach by using a single image encoder, followed by two-stream network that separately handles motion prediction and appearance preservation before decoding. To address the stochastic nature of motion, AMC--AN (Jang iet al. 2018) models multiple plausible oulcomes through adversarial training. contrast, SLAMP (Akan et al. 2021 adopts non-adversarial approach that focuses on learning stochastic variables for disentangled content and motion representations. Further advancing this line of rescarch, LEO (Wang al. 2024ch and D- VDM (Shen et al., 2024) leverage diffusion models to achieve more realistic contentmmotinn disentanglement, demonstrating recent progress in this direction.
4.2.2 Foreground and background
In future frame prediction, the motion dynamics of foreground objects and background scenes often differ substantially. Foreground objects usually display more dynamic motion, while background scenes tend to remain relatively static. This distinction has motivated research lo predict the molion of these components separately, enabling a more nuanced understanding of video dynamics. A notable contribution in this area is DrNot (Denton al.] 2017), which specifically targets scenarios where the background remains largely static across video frames. The model decomposes images into object content and pose, and leverages adversarial training to develop a scene discriminator that determines whether two pose vectors originate from the same video sequence. Similarly, OCVP-VP Villar-Corrales et al. 2023) employs the siot scene parsing network SAVi (Kipf et al.) 2022) to segment scenes hicrarchically from the scene level down to individual objects. By focusing on such videos, prediction models can streamline their learning process by climinating the need to model complex scone dynamics. Both Human-centric tasks, such as predicting human movement and interaction with the environment, and object-centric tasks, such as tracking object motion and positioning, can benefit from this approach.
Human-centric. FFS often focuses foreground motion, especially in scenarios involving complex hu- man movements. A common assumption in these scenarios -refleces-eeleecteed across many specialized datasets is that the background remains relatively static, which is characteristic of datascts focusing on detailed human motion. This has led to a strong research focus on understanding and forecasting human poses to improve foreground motion prediction. A representative example is DVGPC (Cai et al. 2018), which predicts skele- ton motion sequences and transforms them into pixel space using skcleton-to-imas method effectively bridges abstract motion representations and the pixel-level demands of video prediction, demonstrating a nuanced understanding of the complexities inherent. in human-centric FFS tasks.
Transformer. This
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Object-centric. The concept of object-centric video prediction was first introduced by CVP Yee et al. 2019), laying the ioundation for this specialized subfield of video prediction. SlotFormer (Wu et al. 2023 introduces transformer-basee auto-regressive models to learn object-specific representations from video se- quences. This design enables consistent and accurate tracking of individual objects over time. A more recent advancement, OKID (Comas et al. 2023), uniquely decomposes videos inlo distinct components specifically, the attributes and trajectory dynamics of moving objects-by by employing a Koopman operator. This approach offers a more granular method for analyzing object motion in video sequences, setting apart from prior methods.
General. Methods focused on human poses objects have shown considerable promise specific video datasets, but their reliance predefined structures and limited adaptability to dynamic backgrounds hinder generalization. This limitation is reflected in their performance: while effective under controlled conditions, they often falter when confronted with background variation, revealing insufficient versatility for broader applications. To bridge this gap, MOSO (Sun et al., 2023) proposes a unified framework that identifies motion, scene, and object as the three pivotal elements of a video. It further refines content analysis by distinguishing between scene and object-where the scene denotes the background and the object denotes the foreground- as a finer decomposition of video content. MOSO's core contribution is a two-stage network architecture designed for general- purpose video analysis. In the first stage, the MOSO-VOL VQVAE model encodes video frames into token-level representations, trained via a vidco roconstruction task to learn informative embeddings. In the second stage, Transformers are employed to handle masked token prediction, enhancing the model's 's temporal reasoning capabilities. This design enables the model to perform a variety of token-level tasks, including video prediction, interpolation, and unconditional video generation.
Challenges. Disentangling content from motion, or foreground from background, in videos is complex due to the intricate interplay between temporal dynamics and spatial context. Models must accurately capture and predict motion patterns, depth, and semantic regions, while maintaining temporal consistency and spatial accuracy. The presence of occlusions, perspective changes, and complex backgrounds further increases the difficulty. The use of high-resolution feature maps and large-scale annotated datasets further exacerbates computational demands. Generalizing to unseen scenes and objects remains a major challenge, requiring robust models capable of adapting to diverse visual appearances and contexts. Applying the concept of separate processing to the generative mothods discussed later (in Section 5) also presents a significant challenge.
4.3 Motion-controllable synthesis
In the field of FFS, one specialized research direction has emerged that focuses on the explicit control of motion. This approach is distinct in its emphasis on forecasting future object positions based on user- defined instructions, in contrast to the conventional reliance on past motion trends. The central challenge in this domain lies in synthesizing videos that follow these direct instructions while preserving a natural and coherent flow-a task that demands a nuanced understanding of both user intent and motion dynamics within the video context. This challenge underscores the delicate balance between user control and automated imagination, signaling significant shift in how FFS models are conceptualized and implemented.
Strokes. Since there is no historical motion information available for generating videos from one single still image, several methods have emerged that allow for interactive user control. iPOKE (Blattmann et all 2021) introduces techniques in which local interactive strokes and pokes enable users to deform objects in a still image to generate a sequence of video frames. These strokes represent the user's intended motion model (Mahapatra & Kulkarni 2022) proposes a method for interactively controlling the animation of fluid elements. These advances underscore the growing importance of user-centric approaches in the domain of motion-controllabbee FFS.
for the objects. Building on this innovative direct the Controllable-Cinemaggee
Instructions.
The integration of instructions across various cndalitics-inchuding local strokes, sketches, and text- is becoming increasingly common in works aiming to capture user-specified motion trends. Video-
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Composer (Wang al. 2023a synthesizes videos by combining text descriptions, hand-drawn strokes, and sketches. This approach adheres to textual, spatial, and temporal constraints, leveraging latent video dif- fusion models and motion vectors to provide explicit dynamic guidance. Essentially, it can generate videos that align with user-defined motion strokes and shape sketches. In a similar vein, DragNUWA (Yin et al., 2023) primarily leverages lext for content description and strokes for controlling future motion, enabling the generation of customizable videos. These approaches advance the field of video generation by broadening the spectrum of user input modalities.
Challenges. Achieving natural and coherent video synthesis under explicit user control remains signifi- cant challenge. Models must accurately interpret user intent and generate videos that follow specified motion instructions while maintaining both temporal and spatial coherence. Striking a balance between user control and the model's autonomous imagination is essential. Ensuring that the generated videos are both visually compelling and contextually appropriate further increases the complexity, requiring a deep understanding of user intent and video dynamics.
5 Generative synthesis
The emergence and disappearance of objects introduce unpredictabiiity, and generative models require a profound understanding of the underlying physical principles that govern the real world. Rather than relying simplistic linear motion predictions extrapolated from historical frames, they address the challenge using sophisticated and imaginative modeling techniques. As result, tasks such as transforming one static image into one dynamic video often referred to as the image animation problem- hovlemm-avee emerged as promising candidates for applying generative vidco prediction techniques. In general, the two mainstream branches are aulo-regressive models and diffusion models. Although current auto-regressive models are inferior to diffusion models in terms of image generation quality, their structural consistency with LLMs is highly appealing for building unified multi-modal system.
5.1 Diffusion- based generation
Diffusion models (Ho et al. 2020 have emerged dominant approach for image generation. Early attempts at video prediction (Ho et al. 2022 Yang et al. 2023 Harvey el al. 2022 Voleti el. al.) 2022 Singer et al. 2023 which utilize pixel-space diffusion models by extending the conventional U-Net [Ronneberger et al, 2015 architecture to 3D U-Net structures, are constrained to generating low-resolution and short vidco clips due to high computational demands. The Latent Diffusion Model (LDM) (Rombach et al. 2022) extends this capability into the latent space of images, significantly enhancing computational efficiency and reducing resource consumption. This advancement has paved the way for applying diffusion models to video generation [Blatmaan et al. 2023b).
Latent diffusion model extensions. Extensions of LDM have demonstrated strong generative capabil- ities in video systhesis (Voleti et al. 2022 For instance, Video LDM (Blattman et al. 2023b)
leverages high resoluuun, and long-tern term video
pre-trained image models to generate videos, enabling synthesis. Similarly, SEINE (Chen et al. 2024e) introduces a versatile video diffusion model capable of gen- crating transition sequences, thereby extending short clips into longer videos. Burgert ct al. (2025) presents a scalable method for finetuning video diffusion models to control motion without modilying model ar- chitectures or training pipelines. introduces noise warping algorithm that replaces random temporal Gaussian noise with optical flow-guided warped noise, while maintaining spatial Gaussianity. Magic141 (Yi et al. 2025) decomposes the text-to-videe generation task into two simpler subtasks for diffusion step dis- tillation: text-to-image generation and image-to-video generation. Tt shows that image-to-vides generation converges more easily than direct text-to-video generation under the same optimization setup, and explores the trade-off between computational cost and video quality.
Text-guided generation and additional information. Recent rescarch efforts have focused on harness- ing additional modalities alongside RGB images to accomplish the task of text-guided video generation Hong
et al. 2022); 0 Singer et al. (2022): Chai et al. 2023 LFDM Ni et al. 2023) extends latent diffusion models
16

---

Published in Transactions on Machine Learning Research (07/2025)
Method
Publication
Main Ideas
MCVD
Build models from siuple wou-recuzreut 2D convolational architectures, and train them by randomly masking all past or future fromes. Introduce a temporal dimension into the latent space diffuston model, tzansforming au image generatoz into video generator. Propoes short- sl-ort-o-louu video diflusion model. ained at generating coherent long videss at the 'story-levell. Achieve user- meer-ffiendly motion control
22
Voleti et al.
2022
Video LDM
CVPR223
Blactmenn
2023b
SETNE
ICLR'24
Chen et al.
2024e
Go-With-the--
F 10w
CVPR225
Burgert 14 al.
2025
Ouuugh struciured latenl noise sampling derived from optical flow fields. Explore rapid video generation by decomppeeing the text-to video generation task into two subtasks for diffiusion step distillation.
Kagic
CVPR225
Y7 et al.
2025
LFDM
Syulhesize optical llww sequence in the latent space bused given conditions to warp the input inage. Inflate a pre-trained '21 modell along the temporal axis. and integrate globel sentence-level instructions into each generated frame. Investigate key design aspects of text to image- vidoo generative models, including noise scheduling for diffusson and multi-stagn training.
CVPR223
Ni et al.
2023
Soer
Gu et al.
2024
ICLR224
Video
ECCV'44
Girdhar et al.
2024
OynamnCratten
ECCV'24
Incorporate image guidance into the text-to-video diffusion model, by projecting images into a text-aligned context space. Keep the pre-trained T2V model unchauged and futroduce an additional condition encoder to process inpot like skotches, depth maps, and RGB keyframes.
Xing et al.
2024
StaiseCtal
ECCV244
Guo ot al.
2024
PEEK
Integrate nser-interactive control into the T2V model via masked actention module, without requiring extra training inference overhead.
CVPR'24
Jain et al.
2021
VACE
Organizing video task iaputs into unified interface to achieve a unifeed approach, involves injecting different task cmncpps inko the video syulhssis model. framewark for 3D aware and contmolable toxi-to-video generation, empowering users with precise control over the scene.
preprint
Jiaug ol al.
2025
Lin
SIGGRAPH25
Wang
2025a
SkyReeb-A2 A2
Introduce the novel elementss- to- videu tank, geverating diverse and high-quality videos with precise element control. Use the Appearance Injection Network to enbance appearance preservation Employ au Appenrance Noise Prior to relain the capabilities of pre-trained models. Enable users control the temporal mution of images with text, and devode motion-relatee instructions into videoe using generator equipped with motion module.
preprint
Pei ct ol. 2025
MicroCinenn Canena
CVPR'24
Wang et al.
2024bb
Liyel mnto
ECCV'24
Chen et al.
2024e
2VG
Propose cascaded approach to decouple the two factors of semantics and quality, enhmucing details through op spliuization with high quality data.
prepmint
Zhang et al
2023
Sky V2
Integrating multi-modal I.I.M, conducting progresive pnggrssiv-resoottion pretraining. achieving realistic long-form video synthesis and professional film-style generation.
preprint
Chen et al.
2025
Art.v
CVPRW224
Generate Imauees vonditioned ou the previous one auto rigessivel.
Weng al.
2024
Maintain the eapubilities of the pre- pre-trained model withon: modeling long-rang motions. Cast world modeling as an inspeevisedd sequence modeling problem by Lrandorming text, video, and velicle actious into discrete tokens. Study how pure Transformer-eest video classification models extract and encode loug spatio-temporal token sequences, outperforming previous CNNs. Reduce the computational requirements of video generalive models by modeling dynamics the latent spare.
GATA-
prepmint
Hu et al.
2023a
Video Transformet
ICLR'20
Weissebboru et al.
2020
LV
VISIGRAPP21
Rakhimov et al.
2021
Nuwa
ECCV-22 A unified multimodal model that handles langnage, images, and videos,
Wu et al. 2022
employing 3D Nearby Atleution mechanism to reduce computational complexity. For variable-lengli generation jon lasks targeling images of arbitrary simes or long-duration vidoos, introducr a global patch-basun and local visual tolen-based auto-regressive model.
Nuwa-Infinity
NearlP8
Liang et al.
2022
MMVG
Discrelize video Irames inlo visual lokens and propoee inodia masked video generation apprachh to tackle the text guided video completion task.
CVPR'23
Fu et al.
2023
VT
Introduce a 3D) tokenizer to quantize viders into spatio-temporal visual takens, and propase an embeddiig stralegy for masked video lokeu modeling. Represent diverse visual and denonstrate that visual ruto-regresive moreels seale effectively. Specifying both the input prompts and outputs of visual tasks as images, and performing standard masked image modeling on the stitch of input and output image paits. Explore the design of discrete token tokenizers to boost anto-vegressive models. and introduce lookup-free quantizer to enhance video tokenizers. Incorporate multimedal generative objectives including images, videos, text, and audio within auto-regressive Trausformer framework.
CVPR233
Yu et al.
20236
LVM
as sequences of discrete tokens,
CVPR224
Bai et al.
2024
Painter
CVPR223
Wang et al.
2023b
MAGVIT-2
ICLR'24
Yu al.
2024
VideoPoet
ICM1,24
Kondratyuk et al.
2021
Table 4: Overview of generative synthesis methods.
to synthesize optical flow sequences in the latent space based on textual guidance. Seer Gu et al. 2024
inflates Stable Diffusion (Rombach et al. 2022) along the temporal axis, enabling the model to utilize nat- ural language instructions and reference frames to envision multiple variations of future outcomes. Emm Video Girdhar et al. 2024) generates image conditioned on textual guidance and extrapolates it into
17

---

Published in Transactions on Machine Learning Research (07/2025)
video, making it adaptable to diverse textual inputs. DynamiCrafter (Xing et al. 2024 extends text-guided image animation to open-domain image scenarios. SparseCtrl (Guo et al. 2024 supports sketch-to-vide generation, depth-to-video generation, and video prediction with an expanded range of input modalities. Other methods, such as PEEKABOO (Jain et al., 2024). explore interactive synthesis, aiming to unlock unprecedented applications and creative potential. VACE (Jing el al., 2025) proposes a unified condition- ing format called the video condition unit, which encodes text instructions, reference images, and masks. It incorporates a context adapter module that injects task-specific context into the model using structured temporal and spatial representations. This design ensures coherence across frames and spatial regions dur- ing generation and supports diverse applications through flexible task composition. SkyReels-A2 (Fei et al. 2025) defines the elements-to-video task, which involves generating videos by integrating arbitrary visual elements such as characters, objects, and backgrounds through reference images under the guidance of text prompts. The model demonstrates a strong ability to preserve the distinct visual details of each individ- ual element throughout the generation process while simultancously producing results that are coherently blended. This capability represents a significant step forward in the field by shifting the focus of video syn- thesis from single-condition generation to multi-condition combinatorial control. CineMaster (Wang et al., 2025al) proposes 3D-aware, controllable text-to-video framework where users interactively define 3D control signals such as object bounding boxes and camera trajectories to guide a text-to-video diffusion model. To address the scarcity of annotated data, it introduces an automated pipeline that extracts 3D control signals from large-scale video datasets.
Preserving text guidance. Some works aim to achieve a more precise interpretation of textual guidance and preserve this information across the temporal dimension. MicroCinema (Wang et al. 2024b) adopts a divide-andd-conquer strategy to address challenges related to appearance and temporal coherence. It employs a two-stage generation pipeline, first creating an initial image using an existing text-to-image generator, and then introducing a dedicated text-guided video generation framework for motion modeling. LivePhoto (Chen et al. 2024d) proposes a framework that incorporates motion intensity as an auxiliary factor to enhance con- trol over motion dynamics. It further introduces a text re- weiieightng mechanism to emphasize motion-related descriptions, demonstrating strong performance in text-guided video synthesis tasks. 12VGen-XL (Zhang ct al. 2023 utilizes static images to provide semantic and quality-related guidance, highlighting the diversity of approaches in lext-guided video synthesis. SkyReels- V2 (Chen et al. (025)) introduces a multi-modal LLM to interpret narrative semantics by decomposing script-like prompts into shot-level descriptions and utilizes a specialized video captioning module to automatically annotate training data with rich cinematographic language. This methodology enables its 14B parameter model to generate vidcos exceeding 30 seconds in duration while exhibiting a professional cinematic style, representing the first open-source model to achieve form-fomm video synthesis that closely approximates the quality and coherence of film production.
Integrating auto-regressive models. While the dominance of diffusson models in generative tasks is rising, some researchers still attempt to preserve the architecture of auto-regressive models (Weng et all 2024), develop new diffusion architectures to integrate the strengths of both paradigms (Chen et al. 2024a). Early generative video synthesis algorithms are constrained by limited data availability and model scalability, yet they lay the groundwork for integrating LLMs and diffusion models in video generation. VDT (Lu et al., 2024) represents one of the earliest efforts to incorporate a Transformer- into diffusion models for video generation. Inspired by the success of DiT (Peebles & Xie 2023 in image synthesis, VDT replaces the conventional U-Net with a spatio-tempon
backbone
Transformer that explicitly models both spatial and temporal dependencies. W.A.L.T. (Gupta et al., 2025) provides the first successful empirical demonstration of a Transformer-based backbone for jointly training image and video latent diffusion models. Recently, GAIA-1 (Hu et al. 2023a) and Sora (Brooks et al., 2024) have also leveraged the strengths of DiT to enable more creative, generalizable, and scalable video synthesis. RIVER (Davtyan et al. 2023) leverages flow matching (Lipman et al. 2023) for efficient video prediction by conditioning on a small set of past frames in the latent space of a pre-trained VQGAN (Esser el. al.) 2021). Collectively, these ad advances underscore the potential of diffusion models to produce high-quality, controllable, and diverse video content, thereby pushing the boundarics of what is achievable in computer vision and artificial intelligence.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Method
Model Size
HunyuanVideo (Kng al. 2024
13B
T2V Ma et al. 2025
30B
Step-Video-T122 T12V [H nang el al. 2025
30B
Seaweed-7B (Seawead et al. 2025
7B
MAGI-1 Sand-A 2025
24B
Table 5; Recently open-sourced large models for video generation.
Recent open-sourced solution. HunyuanVideo [Kong et al. 2024 is very large open-source video generative model (13B parameters). Relying on key strategies such as data curation and image-video co- training, the model reaches a level of generation quality comparable to or even better than that of the leading closed-source models, which effectively narrows the performance gap between open-source and closed-source solutions. Ma et al. (2025) release an even larger video generative model, Step-Video-12 rameters, and a new benchmark for video generalion, Slep-Videos Viiie--TVV-Evall The model incorporates a deep compression video Variational Autoencoder for compact and efficient video representation, and leverages a flow matching approach to train DiT (Peebles & Xie 2022 architecture for denoising input noises into latent frames. To enhance the visual quality and miligate artilacts, video-based direct preference optimiza- tion method is applied. For the task of text-driven image-to-video generation, Huang et al. 2025 release Ee-Viide-T11y a model of equal scale with 30B parameters, along with its corresponding benchmark, Seop-Video-IIV-EER It is noteworthy that improving video generation performance does not necessarily require increasing the number of model parameters. Seaweed-7B (Seawead et. al.) 2025), with only 7B pa- rameters, demonstrates generation quality comparable that of models with tens of billions of parameters. This provides important insights into how the performance of medium-sized DiT models can be effectively enhanced without relying solely scaling up model size. Sand-AI (2025) has open-sourced 24B parameter video generative model, MAGI-1, that divides video frame sequences into fixed-length video chunks and adopts a chunk-wise auto-regressive diffusion framework, facilitating text-driven image-to-video generation through chunk-wise prompting. This design illustrates a promising direction in which high-fidelity video synthesis and fine-grained instruction control can be unified.
with 30B pa-
Challenges. While diffusion models have made significant strides in video generation, several critical challenges remain. Ensuring temporal coherence and consistency across frames is essential for achieving realism, yet remains major challenge (Chen et al. 20240 Xu et al. 2024). The computational efficiency and scalability of diffusion models are hindered by their resource-intensio nature, which limits their broader adoption (Poebles & Xie 2023). Issues of controllability and interpretability persist, as textual guidance does not always align with visual outcomes, and model behaviors often remain opaque. Data availability and diversity are critical for training robust models, but acquiring comprehensive and diverse datasots remains major bottleneck.
5.2 Token- based generation
Diffusion-based methods have garnered significant attention the realm of image and video generation. However, these models typically have smaller parameter scales compared to contemporary large-scale lan- guage models. Recent research has increasingly focused on exploring how LLMs can be employed for such tasks, leveraging their optimization techniques and accumulated insights to investigate the applicability of scaling laws in the visual domain.
Key components. Implementing token-based FFS requires two key components: a high-quality visual tokenizer and an efficiently scalable LLM framework. Innovations such as VQ-VAE (Van Den Oord et al, 2017) and VQGAN (Esser et al. 2021) intograte auto-rogressive models with adversarial training strategies tackle image quantization and tokenization. An effective visual tokenizer should minimize the number of tokens per image video clip segment while preserving near-lossless visual fidelity. However, the substantial
19

---

Published in Transactions on Machine Learning Research (07/2025)
token requirements for lossless r particularly for high-resohution images-present challenges in processing long video sequences during training, thereby limiting video generation capabilities.
Latent space modeling. Even before the advent of LLMs (Brown el al. 2020 Achiam et al. 2023
Transformers have already made a significant impact on time-series modeling. Video Transformer (Weis- senborn et al. 2020) pioneer the use of Transformer architectures in video synthesis by introducing an auto-regressive modeling approach. Despite its success, it inherits common limitations of Transformer-basee models, including high training resource demands and slow inference speed. The Latent Video Trans- former (LVT) (Rakhimov et al. 2021) introduces a novel latent-space approach that auto-regressively mod- els temporal dynamics and predicts future features, significantly reducing computational overhead. Other works, such VideoGen (Zhang et al. 2020c) and Video VQ-VAE (Walker et al., 2021), also leverage the VQ-VAE framework (Van Den Oord et al. 2017) to extract discrete tokens for video prediction. In con- trast, Phenaki (Villegas et al. (2022) improves upon the ViVIT (Arnab et al. 2021) architecture to extract continuous tokens. The NUWA framework (Wu et al.) 2022a) proposes a versatile 3D Transformer based encoder-decoder architecture that is adaptable to diverse data modalities and tasks, further demonstrating the potential of Transformers in vidco synthesis. NUWA-Infnitty (Liang et al. 2022) builds upon this with an innovalive generation mechanism designed to enable infinite high-resolution video synthesis, reflecting ongoing efforts to unify generative tasks across modalities.
In-context learning in visual domain. IVM (Bai et al. 2024 introduces sequential modeling to enhance the learning capacity of large-scale vision models, demonstrating the scalability and flexibility of sequence- based models in in-context learning. The concept of "visual sentence' is proposed, in which sequence of intrinsically related images is organized analogously to a linguistic sentence. This allows the model to leverage sequential information for sentence continuation and other visual tasks without relying on non- pixel-level knowledge. Painter (Wang et al.) 2023b) proposes a general framework for visual learning that enables images to 'speak' through in-context visual understanding, thereby enhancing both image generation and interpretation. SegGPT (Wang al.] 2023c] explores the use of GPT-based architecture for image segmentation, introducing the concept of "seginenting everything' and demonstrating the potential for unified segmentation under unsupervised learning seitings, thereby advancing generalization in visual segmentation tasks.
Advances in video generation. In the domain of video generation, MAGVIT Yuu et al. 2023b presents a masked generative video Transformer that efficiently processes videos by masking certain regions and predicting the missing segments. MAGVIT-v2 (Yu et al. 2024 suggests that Transformer- may surpass diffusion models in visual generation lasks, highlighting the pivotal role of visual tokenizers. VideoPoet (Kondratyuk et al.) (2024) introduces a large language model for zero-shot video generation, pushing the boundaries of unsupervised video synthesis. It enables users to generate or edit vidcos based on high-level textual prompts and excels al capturing temporal and contextual relationships within video data.
models
Text-guided generative video synthesis algorithms generate sequences of frames by integrating contextual visual information with textual guidance. The Text-guided Video Completion (TVC) task entails completing videos under various conditions, including the first frame (video prediction). the last frame (video rewind), or both (video transition), guided by textual instructions. MMVG (Fu et al. 2023) addresses the TVC task using an rugo-seeeessive encoder-decoder architecture that integrates textual and visual features, forming a unified framework capable of handling diverse video synthesis tasks.
Collectively, these studies integrate visual tokenizers and LLMs to establish unified and scalable frameworks for visual learning, thereby driving significant progress in FFS tasks. The evolving landscape of FFS research continues to highlight the potential of Transformers and LLMs in unifying generative tasks across modalities.
Challenges. Token-based generation for FFS faces siguificant challenges, partieularly in designing efficient visual tokenizers that balance a minimal number of tokens with near-lossless reconstruction, especially for high-resolution content. The high computational demands of Transformer-baser models also present barriers to adoption, particularly in resource-constrained environments. When computational resources are limited, the emergence phenomena reported in previous studies under this paradigm may fail to manifest. (Bai et al.
20

---

Published in Transactions on Machine Learning Research (07/2025)
2024h. Recent work (Sun et al. 2024a suggests that one reason token-based models struggle to match the visual quality of diffusion models is their limited integration with high-quality community assets, such robust training infrastructure and curated datasets.
6 Application Realms
The applications of FFS are extensive and diverse, permeating numerous domains and highlighting its increasing importance as a versatile tool. Across these domains, FFS serves as fundamental enabler for predictive modeling and decision-making, allowing systems to anticipate future states based on past obscrvations. This predictive capability is crucial for enhancing safcty, efficiency, and adaptability in dynamic environments.
World model. World models (Ha & Schmidhuber 2018b Zhu et al. 2024 provide a general-purpose framework for simulating and predicting the dynamics of complex systems. These models are widely employed in reinforcement learning and robotics, enabling agents to make informed decisions and perform actions that lead to desired outcomes. FFS serves as a key learning objective in the development of world models (Hafner let al.
2020 2021 2023 Wang al. 2024a Ge al. 2024] Agarwal et al.) (2025). Recent work (Escontrela al.
2024 demonstrates that video prediction also be incorporated into reward modeling to further
support reinforcement learning. GameNGen (Valevski et al.) 2025) has demonstrated exceptional world modeling performance and strong iestruction-folloiing ability in controllable FFS.
Autonomous driving. FFS is indispensable for autonomous vehicles, including self-driving cars and drones, as it enables them to anticipate the movement. of objects, pedestrians, and other vehicles. This pre- dictive capability critical for ensuring safe and efficient navigation. For instance, GAIA-1 (Hu et al. 2023a) employs a unified world model that integrates multimodal LLMs and diffusion processes to prodict control signals and future frames, thereby enhancing decision-makinn capabilities in autonomous systems. In most existing driver-assistance systems, visual input is first transformed into structured representations succe such objects, lane markings, and traffic lights followed by downstream predictions based on these modalities. Effectively leveraging raw visual information for trajectory prediction in real-world scenarios remains a sig- nificant challenge (Nayakanti et al.) 2023 Varadarajan et al.] 2022j. Most existing trajectory prediction approaches rely solely on historical detection records of surrounding vehicles and pedestrians. Recent stud- ies (Gu et al.] 2023 have demonstrated that fully exploiting semantic information from visual inputs can significantly improve behavior prediction in dynamic scenes.
Robotics. In the field of robotics, FFS is employed to guide robotic agents through dynamic environments. It enables them to effectively plan paths, manipulate objects, and avoid obstacles, demonstrated by (Finn) & Levine 2017). By predicting future states, robotic systems can make proactive decisions, thereby enhanc- ing adaptability and operational efficiency in complex environments. The GR-1 and GR-2 approaches (Wul
ct al. 2024a Cheang ct al. 2024) demonstrate that visual robotic manipulation can benefit significantly
from largesccale video generative pre-training. After being pre-trained on large-scale video datasets, GR-1/2 can be seamlessly fine-tuned on robot-specific data, exhibiting strong generalization unseen scenes and objects.
Film production. FFS has found valuable applications in the film industry, particularly in special effects, animation, and pre-visualization. It assists filmmakers in generating realistic scenes and enhancing the overall cinematic experience. For example, Mahapatra & Kulkarni (2022) utilize FFS to generate visually compelling sequences that enhance narralive coherence and support artistic expression in filmmaking.
Meteorological. FFS plays a vital role in weather forecasting by assisting meteorologists in simulating and predicting atmospheric dynamics. By accuratoly forccasting future spatio-temporal patterns, FFS enhances the precision of weather prediction models, as demonstrated by (Shi et al. (2017). This capability is essential for both operational weather forecasting and disaster preparedness.
21

---

Published in Transactions on Machine Learning Research (07/2025)
Anomaly detection. Liu et al. (018 propose video anomaly detection approach based on future frame prediction, under the assumption that normal events are predictable whereas abnormal events deviate from expected patterns. The method introduces a motion constraint alongside appearance constraints to ensure that the predicted future frames align with the ground truth in both spatial and temporal dimensions.
Overall, these diverse applications underscore the significance and broad potential of FFS as a powerful tool for understanding and interacting with the dynamic world. Its capacity to predict future states from past observations positions it valuable asset across wide spectrum of domains, including artificial intelligence, robotics, entertainment, and beyond.
Challenges. When applying FFS methods to real-world scenarios, several potential challenges may arise. For instance, domain-specific applications may require solutions for few-shot learning (Gui et al. 2018) or test-time adaptation (Choi et al. 2021). Another challenge lies in the effective integration of vectorized data with image- baso- inputs. Morcover, interpretability remains a concern in real-world applications, as end-Lo-end FFS methods often lack transparency and difficult. to interpret.
7 Related Work
In the related work section, delineate survey's unique focus FFS by comparing it with existing literature on video prediction and video diffusion models, highlighting both similarities and key distinctions.
Previous surveys on video prediction. The field of video prediction, along with related areas such as action recognition and spatio-temporal predictive learning, has witnessed significant progress in recent years, largely driven by doep lcarning techniques. Several comprchensive surveys have provided overviews of state-of-the-ar methods, benchmark datasels, and evaluation protocols in this domain. Zhou et al. (2020) review next-frame prediction models developed prior to 2020, categorizing them into sequence-to-one and sequence-to-see to--uuee------eenceeeeee architectures. The survey compares these approaches by analyzing their architectural designs and loss functions, and provides quantitative performance comparisons based on standard datasets and evaluation metrics. Oprea et al. (2020) present a comprehensive review of deep learning methods for video prediction, outlining fundamental concepts and analyzing existing models based on a proposed tax- onomy. The survey also includes experimental results to enable quantitative asscssment of the state-of-the. art Rasouli (2020) provide an overview of vision- based prediction algorithms with a focus on deep learning approaches. They categorize prediction tasks into vidco prediction, action prediction, trajectory prediction, body motion prediction, and other related applications, and discuss common network architectures, training strategies, data modalities, evaluation metrics, and benchmark datasets. While these surveys focus on early technical developments in video prediction, our survey specifically emphasizes the synthesis aspect of FFS and its evolution towards generative mothodologics, highlighting the rising significance of generative models.
Kong & Fu (2022) survey state-of-the-art techniques in action recognition and prediction, covering existing models, representative algorithms, technical challenges, action datasets, evaluation protocols, and future rescarch directions. Tan et al. (2023) introduce OpenSTL, unified benchmark for spatio-temporal predictive learning, which categorizes methods into recurrent-based and recurrent-free models. The paper provides standardized evaluations on multiple datasets and offers an in-depth analysis of how model architecture and dataset characteristics influence performance. For our survey, these reviews are excellent supplements regarding technical details such as architectures and benchmarks.
Surveys on video diffusion models. More recently, the emergence of video diffusson models has led to several specialized surveys. Xing et al. (2023) present comprehensive review of video diffusion models in the era of Al-generated content, categorizing existing works into video generation, video editing, and other video understanding tasks. The survey provides an in-depth analysis of the literature in these arcas and discusses current challenges and future research trends. Li et al. (2024) survey recent advances in long video generation, summarizing existing methods into two key paradigms: divide-and seride-and-conquees and temporal auto-regressive modeling- They also provide a comprehensive overview and categorization of datascts and evaluation metrics, and discuss emerging challenges and future directions in this rapidly evolving field. [Snn et al. (2024b) review Sora, OpenAT's text-to-video model, categorizing the related literature into three themes -volutionary
22

---

Published in Transactions on Machine Learning Research (07/2025)
generators, excellence in pursuit, and realistic panoramas wanoeamas-while while also discussing datasets, evaluation metrics, existing challenges, and future directions. Complementarily, Liu et al. (2024b) provide comprehensive analysis of Sora's underlying technologies, system design, current limitations, and its potential role in the broader landscape of large vision models. Melnik et al. (2024) offer an in-depth exploration of the critical components of video diffusion models, focusing on their applications, architectural design, and lemporal dynamics modeling. These surveys offer highly specific analyses of recent models. In contrast, our survey provides broader review of the FFS methodology, positioning models like Sora as examples within the evolving landscape of the generative approaches to FFS.
Our survey focus. Our survey provides a comprehensive review of both historical and recent works in FFS, with a particular focus on the transition from deterministic to generative synthesis methodologies. The survey highlights key advances and methodological shifts, emphasizing the growing role of generative models in producing realistic and diverse future frame predictions.
8 Conclusion
In this survey, we have systematically charted the evolution of Future Frame Synthesis (FFS), tracing its journey from deterministic prediction to the era of large-scale generative models. Our taxonomy, organized by the degree of algorithmic stochasticity, not only categorizes the field but also illuminates the fundamental methodological shifts that have defined its progress. Looking forward, we argue that the trajectory of FFS is not monolithic but is bifurcating into two distinct, yet potentially synergistic, research frontiers, each with its own set of grand challenges and open questions.
The First Frontier: Pragmatic Real-Time Synthesis. This direction prioritizes efficiency, precision, and applicability for high-definition, low-latency tasks. Rather than pursuing general world knowledge, its goal is to perfect low-level objectives like video compression, frame interpolation, and short-term forecasting, Key open questions in this domain include:
Hybrid Model Architectures: Can we design lightweight, deterministic modules for instance, advanced optical flow or motion field estimators that can be efficiently 'plugged into" large generative models? Such hybrids could leverage the generative prior for context while using the deterministic module to enforce short-term, pixel-perfect motion accuracy at a low computational cost.
Knowledge Distillation for Motion: How can the rich, implicit motion understanding learned by massive models like SVD (Blatlmann el. al. 2023a) or Sora (Brooks al., 2024) be elfectively distilled into compact, recurrent-free networks? Developing novel distillation techniques tailored for temporal dynamics, rather than just static image featuros, is critical for real-world deployment on edge devices.
The Second Frontier: Generative World Simulatioo. This path more ambitious, aiming to develop models with fundamental understanding of the physical world, capable of generating diverse, coherent, and long-duration vidcos by leveraging vast computational and data resources. This rescarch directly contributes to the long-term vision of creating general-purpose world models. The most pressing challenges here are not just about scaling, but about fundamental model capabilities:
Bridging the Paradigm Gap: How do we unify the strengths of disparate synthesis paradigms? Current methods excel at either precise motion warping (motion- based synthesis) or creative content generation (diffusson models), but rarely both. pivotal breakthrough would be a unified framework that generates novel objects and events while adhering strictly to the predicted trajectories of existing elements.
Solving the Tokenizer Bottleneck: Is the persistent quality gap between token- based generative models and their diffusion-based counterparts a fundamental limitation of processing visual data as discrete se- quences, or is it an engineering problem of tokenizer design and training infrastructure? Exploring hybrid auto-regressive/ /iiffusion backbones or developing novel, lookup-free quantization schemes could unlock the scalability of LLM-like architectures for high-fidelity video generation.
From Semantic to Physical Control: Current controllability is largely limited semantic instructions like text sparse inputs like strokes. The next leap is to enable fine-grained, quantitative control over the
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
underlying physics of a scene. Can we instruct model to generate video where 'the ball has a mass of 5kg and a friction coefficient of 0.3,' and see a physically plausible result? This requires moving beyond pattern recognition to building genuine, albeit simplified, simulation engines.
Progress in this second frontier is fundamentally gated by the evolution of our evaluation metrics. Metrics must evolve to reward not just perceptual quality, but also long-term temporal coherence, causal reasoning, and adherence to physical principles. The ultimate objective is transform FFS from a task of visual pattern continuation into one of dynamic world simulation. As have detailed, achieving this vision requires concerted effort across model architecture, data curation, and evaluation philosophy. We anticipate that the insights and categorization provided in this survey will serve as valuable roadmap for rescarchers navigating this exciting and rapidly changing domain.
Broader impact. The performance improvements of future prediction models that do not incorporate generative content do not bring significant negative social impacts. With the rise of Al-generation methods, the way we create and consume video content is rapidly evolving. While Al-generated technologies hold the potential to save substantial time for content marketers and video creators, they also raise numerous ethical concerns. On the positive side, AI video generation can save both time and money for content creators, expand their imagination, and democratize video creation, making it easier for small businesses and individuals to compete with companies that have large markcting budgets. However, it's inevitable that distinguishing between real and Al-generated content will become increasingly challenging, Without proper safeguards, there is a risk of misinformation, propaganda, and the manipulation public opinion. Generative videos might misrepresent people and events, reflect biases and discrimination from the training set, pose legal and copyright issues, and potentially lead to job losses for human video editors and animators. Apart from imposing constraints on content generation service providers, such preventing inappropriate generation and adding watermarks, the exploration of Al-Generated Media Detection methods is also very useful. To mitigate risks, content generation service providers can prevent inappropriate generation and add watermarks. Exploring methodologies for detecting Al-generated media (Zou el al. 2025 Chen et al. 2024b holds significant value as well.
References
Sami Abu-EE-Haijp Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-Sm: A large-scale video classification benchmark. ColR,, abs/1609.08675, 2016. URL attp:://axxv..rg//bbb/////
08675
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. wwXiv:233300871, 2023.
Niket Agarwal, Arslan Ali, Macicj Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. ar preprint arXim:2501.0557 2025.
Adil Kaan Akan, Erkut Erdem, Aykut Erdem, and Fatma Güney. Slamp: Stochastic latent appearance and motion prediction. In ICCV, 2021.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, 2017.
Amurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A vidco vision transformer. In ICCV, 2021.
Mohammad Babacizadch, Chelsca Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In JCLR, 2018. URL httptt:://ppeereiiieeeeee// wttp://wwwweeoooo//////////----.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In CVPR, 2024.
24

---

Published in Transactions on Machine Learning Research (07/2025)
Max Bain, Arsha Nagrani, Gil Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
Bowen Baker, lgge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampodro, and Jeff Clunc. Vidco pretraining (vpt): Learning to act by watching unlabeled online videos. NeurIPS, 2022.
Xinzhu Bei, Yanchao Yang, and Stefano Soatto. Learning semantic-aware dynamics for video prediction. CVPR, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In ICML, 2021.
Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Björn Ommer. ipoke: Poking a still image for controlled stochastic video synthesis. In ICCV, 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Domínik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusson models to large datasets. arXiv preprint arXiv:231.11512, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023b.
Yochai Blau and Tomer Michacli. The percoption-distortion tradcoff. In CVPR, 2018.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL https:// www.reseerr.c/////m htapes///ppeaai..ccrreeeerc//
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NewrIPS, 2020.
Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, Michael Ryoo, Paul Debevec, and Ning Yu. Go-with-the-flor Motion-conttrllablle video diffusion models using real-time warped noise. Tn CVPR, 2025.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Amush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. muscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Haoye Cai, Chunyan Bai, Yu-Wing Tai, and Chi-Keung Tang. Deep video generation, prediction and completion of human action sequences. Tn ECCV, 2018.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-706 human action datasct. arXiv preprint arXio:1907.0696 2019.
Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional vrmns for video prediction. In ICCV, 2019.
Wenhao Chai, Xun Guo, Gaoang Wang, and Yan La. Stablevideo: Text-driven consistency-awwre diffusion video editing. In ICCV, 2023.
Zhengping Che, Guangyu Li, Tracy Li, Bo Jiang, Xucfeng Shi, Xinsheng Zhang, Ying Lu, Guobin Wu. Yan Liu, and Jieping Ye. D2-city: largo-scale dashcam video dataset of diverse traffic scenarios. arXiv preprint ar xrXio:1904.0197 2019.
25

---

Published in Transactions on Machine Learning Research (07 (025))
Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-langung--action model with web-scale knowledge for robot manipulation. arXiv preprint ar arXiv:2410.0611
2024.
Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Todrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. NeurIPS, 2024a.
Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Juncheng Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengchen Ma, et al. Skyreels-v2: lennth film generative model. arXiv preprint arXiv:2504.1300 19074, 2025.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, 2021.
Haoxing Chen, Yan Hong, Zizheng Huang, Zhuoer Xu, Zhangxuan Gu, Yaohui Li, Jun Lan, Huijia Zhu, Jianfu Zhang, Weiqiang Wang, et al. Demamba: Ai-generated video detection on million-scale genvideo benchmark. arXiv preprint arxi:2205. 19707, 2024b.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual datasct. In ICASSP, 2020.
Jiaben Chen and Huaizu Jiang. Sportsslomo: A new benchmark and basclines for human-centrie video frame interpolation. In CVPR, 2024.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wel Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In CVPR, 2024c.
Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng, Yu Liu, Yujun Shen, and Hengshuang Zhao. Livephoto: Real image animation with text-guided motion control. In ECCV, 2024d.
Xinyuan Chen, Yaohui Wang, Lingiun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In ICLR, 2024e.
Zhikai Chen, Fuchen Long, Zhaofan Qiu, Ting Yao, Wengang Zhou, Jiebo Luo, and Tao Mei. Learning spatial adaptation and temporal coherence in diffusion models for vidco super-rcsolution In CVPR, 2024f.
Zilong Chen, Yikai Wang, Feng Wang, Zhengyi Wang, and Huaping Liu. V3d: Video diffussin models are effective 3d generators. arXiv preprint arXiv:2403.067 2024g.
Xianhang Cheng and Zhenzhong Chen. Multiple video frame interpolation via enhanced deformable separable convolution. TPAMT, 44(10))7029-7000 -045,, 2021.
Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee. Channel attention is all you need for video frame interpolation. In AAAI, 2020.
Myungsub Choi, Janghoon Choi, Sungyong Baik, Tac Hyun Kim, and Kyoung Mu Lec. Test-time adaptation for video frame interpolation via meta-learning. TPAMI, 44(12):9615 9628, 2021.
Armand Comas, Christian Fernandez Lopez, Sandesh Ghimire, Haolin Li, Mario Sznaier, and Octavia Camps. Learning object-centric dynamic modes from video and emerging properties. In L4DC, 2023.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In CVPR, 2016.
Quan Dao, Hao Phung, Binh Nguyen, and Anh Tran. Flow matching in latent space. ar preprint arXiv:2307.0660 2023.
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In CoRL, 2019.
Aram Davtyan, Sepehr Sameni, and Paolo Favaro. Efficient video prediction via sparsely conditioned flow matching. In 1CCV, 2023.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In ICML, 2018
Emily L Denton et al. Unsupervised learning of disentangled representations from video. In NewrIPS, 2017.
Piotr Dollar, Christian Wojck, Bernt Schicle, and Pictro Perona. Pedestrian dctection: An evaluation of the state of the art. TPAMI, 34(4):743-761, 2011.
Qiaole Dong and Yanwei Fu. Memflow: Optical flow estimation and prediction with memory. In CVPR, 2024.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In NewrlPS, 2016.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. In TCLR, 2021.
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. NewrIPS, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, et al. Skyreels-a2: Compose anything in vidco diffusion transformers. arXiv preprinl arXiv:2504.044p 2025.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In ICRA, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NewrIPS, 2016.
Jean- Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual vidco prodiction. In ICML, 2020.
Tsu-Jui Fu, Licheng Yu, Ning Zhang, Chnng-Yang Fu, Jong-Chyi Su, William Yang Wang, and Scan Bell. Tell what happened: Unilying text-guided video completion via multimodal masked video generation. In CVPR, 2023.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In CVPR, 2022.
Zhiqi Gc, Hongzhe Huang, Mingze Zhou, Juncheng Li, Guoming Wang, Siliang Tang, and Yucting Zhuang.
WorldGPT: Empowering LLM as multimodal world model. In ACM Mullimedia 2024, 2024. URL https: //openreview.netee wttperreviieeeee//uuuumuu. (iti:rrrmmmiintttaaaaaaaaaaa
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJR, 32(11):1221 1237, 2013.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samanch Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Factorizing text-to-video generation by explicit image conditioning. In ECCV, 2024.
27

---

Published in Transactions on Machine Learning Research (07/2025)
lan Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commuuicaatinss of the ACM, 63(11): 139 144, 2020.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, He- una Kim, Valentin Hacnel, Ingo Fruend, Peter Yianilss, Moritz Mueller-Freite -rritarr et al. The something something' video database for learning and evaluating visual common sense. In ICCV, 2017.
Oliver Groth, Fabian B Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision-based
physical intuition for generalised object stacking. In ECCV, 2018.
Junru Gu, Chenxu Hu, Tianyuan Zhang, Xuanyao Chen, Yilun Wang, Yue Wang, and Hang Zhao. Vip3d: End-to-end visual trajectory prediction via 3d agent queries. In CVPR, 2023.
Xianfan Gu, Chuan Wen, Weirui Ye, Jiaming Song, and Yang Gao. Seer: Language instructed video
prediction with latent diffusion models. In The Twelfth ICLR, 2024. URL http:://operreriew.ne//
forum?3d==gigglyy0331
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsuper- vised video prediction. In CVPR, 2020.
Liang-Yan Yan Gui, Yu-Xiong Wang, Deva Ramanan, and José MF Moura. Few-shot human motion prediction via meta-learning. In ECCV, 2018.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Sparsectrl: Adding sparse controls text-to-video diffusion models. In ECCV, 2024.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Fei-Fei Li, Irfan Essa, Lu Jiang, and José Lezama. Photorcalistic video generation with diffusion models. In ECCV, 2025.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. NewrIPS, 2018a.
David Ha and Jürgen Schnidhuber. World models. arXiv preprinl arXiv:1803, 10122, 2018b.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020. URL http://pppmeiiiie.neeeeooooreess
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021. URL https:// wttp:///oorroiddddoorrrr222.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrpp. Mastering diverse domains through world models. arXiv preprinl 2rX1::2301.04104, 2023.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusson modeling of long videos. NewrIPS, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NewrIPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NewrIPS, 2020.
Jonathan Ho, Tim Salimans, Alexcy Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. NewrIPS, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735 1780, 1997.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Largo-scale pretraining for text-to-video generation via transformers. Xiv preprint 2005 15868, 2022.
28

---

Published in Transactions on Machine Learning Research (07/2025)
Anthony Hu, Fergal Cotter, Nikhil Mohan, Corina Gurau, and Alex Kendall. Probabilistic future prediction for video scene understanding. In ECCV, 2020.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXic:2309. 7080, 2023a.
Xiaotao Hu, Zhewei Huang, Ailin Huang, Jun Xu, and Shuchang Zhou. A dynamic multi-scale voxel flow network for video prediction. In CVPR, 2023b.
Haoyang Huang, Guoqing Ma, Nan Duan, Xing Chen, Changyi Wan, Ranchen Ming, Tianyu Wang, Bo Wang. Zhiying Lu, Aojic Li, et al. Step-vidco-ti2y technical report: A stato-of-the-art text-driven image-to-video generation model. arXiv preprint urxi::2503. 11251, 2025.
Rui Huang, Shu Zhang, Tianyu Li, and Ran Hc. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In ICCV, 2017.
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In ECCV, 2022a.
Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV, 2022b.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, pp. 21807 21818, 2024.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint or 2rXiv:2410.2127 2024.
Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusson. In CVPR, 2024.
Yunseok Jang, Gunhee Kim, and Yale Song. Video prediction with appearance and motion conditions. In TCML, 2018.
Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael J Black. Towards understanding action recognition. In ICCV, 2013.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik slomo: High quality estimation of multiple intermediate frames for video interpolation. In CVPR, 2018.
and Jan Kautz. Super
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.0755 2025.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In ECCV, 2016.
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukeuoglu. Video pixel networks. In ICML, 2017.
Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran. Flavr: Flow-agnostic video represen- tations for fast frame interpolation. In WACV, 2023.
Andrej Karpathy, George Toderici, Sanketh Shelly, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
Tero Karras, Milaa Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. NewrIPS, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
29

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-Centric Learning from Video. In ICLR, 2022.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming Chang Chiu, et al. Vidcopoct: A large language model for zero-shot video generation. Proceedings of Machine Learning Research, 385:25105-251244 2024.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systemalic framework for large video generalive models. arXiv preprint arXie:2412, 03603, 2024.
Yu Kong and Yun Fu. Human action recognition and prediction: A survey. IJCV, 130(5):1366-1444 1401, 2022.
Alex Krizhevsky, llya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In NeurIPS, 2012.
Hildegard Kuchne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In ICCV, 2011.
Manoj Kumar, Neil Houlsby, Nal Kalchbrenner, and Ekin Dogus Cubuk. Do better imagenet classifiers assess perceptual similarity better? TMI.R, 2022. ISSN 2835-8856. URL. httpp:://pparrrreeeeneee//eeeeee qrGKGZZvHO Expert Certification.
Yong-Hoon Kwon and Min-Gyu Park. Predicting future frames using retrospective cycle gan. In CVPR, 2019.
Alex X Lee, Richard Zhang, Frederik Ebert, Picter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv:1804.0155 2018
Sangmin Lee, Hak Gu Kim, Dac Hwi Choi, Hyung-II Kim, and Yong Man Ro. Vidco prediction recalling long-term motion context via memory aligument learning. In CVPR, 2021.
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. arXiv preprint arXiv:2405.1742) 2024.
Chengxuan Li, Di Huang, Zeyu Lu, Yang Xiao, Qingqi Pei, and Lei Bai. survey on long video generation: Challenges, methods, and prospects. arXiv preprint arXiv:2403. 16407, 2024.
Wenbo Li, Xin Lu, Shengju Qian, and Jiangbo Lu. On efficient transformer- bassed image pre-training for low-level vision. In IJCAT, 2023.
Jian Liang, Chenfei Wu, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis. In NewrIPS, 2022.
Jingyun Liang, Yuchen Fan, Kai Zhang, Radu Timofte, Luc Van Gool, and Rakesh Ranjan. Movideo: Motion-aware video generation with diffusion model. In ECCV, 2024.
Yaron Tipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXim:2210.022) 2022.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023. URL http:s//wppppaneeeeeeeeeeeeeeoep wwt.://oorrur//dd////ooooor.
Huan Liu, Zhixiang Chi, Yuanhao Yu, Yang Wang, Jun Chen, and Jin Tang. Meta-auxiliary learning for future depth prediction in vidcos. In WACV, 2023.
Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjic Liao. Fréchet vidco motion distance: A metric for evaluating motion consistency in videos. In First Workshop on Controllable Video Generation @ICMT.24, 2024a. URL httee:///ppooonoeeeeeeeee//eeeer
30

---

Published in Transactions on Machine Learning Research (07/2025)
Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao. Future frame prediction for anomaly detection-a new baseline. In CVPR, 2018.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.000 03000, 2022.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujic Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402. 1222..111 17177, 2024b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Ziwei Liu, Raymond A Yeh, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In ICCV, 2017.
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. In ICLR, 2017. URL https://openreviien wttp://www.....ooooooooooeeeaaa.
Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General- purpose video diffusion transformers via mask modeling. In ICLR, 2024.
Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In CVPR, 2022.
Pauline Lue, Natalia Neverova, Camille Couprie, Jakob Verbeek, and Yann LeCun. Predicting deeper into the future of semantic segmentation. In ICCV, 2017.
Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model. Xixiv preprial or arxi:2200.. 10248, 2025.
Aniruddha Mahapatra and Kuldeep Kulkarni. Controllable animation of fluid elements in still images. In CVPR, 2022.
Michael Mathicu, Camille Coupric, and Yann LeCun. Deep multi-scale vidco prediction beyond mean square error. In ICLR, 2016.
Andrew Melnik, Michal Ljubljanac, Cong Lu, Qi Yan, Weiming Ren, and Helge Ritter. Video diffusion models: A survey. arXiv preprint arXiv:2405.031 2024.
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
Ruibo Ming, Jingwei Wu, Zhewei Huang, Zhuoxuan Ju, Jianming Hu, Lihui Peng, and Shuchang Zhou. Advancing auto-regressive continuation for video frames. arXiv preprint ar arXic:241.00375: 2024.
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In ICRA, 2023.
Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min. Conditional image-to-video generation with latent flow diffusion models. In CVPR, 2023.
Simon Niklaus and Feng Liu. Softmax splatting for vidco frame interpolation. In CVPR, 2020.
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution. ICCV, 2017.
31

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Sergin Oprea, Pablo Martinez- Martinez-Gonzalez, Alberto Garcia- John Alejandro Castro-Vargas, Vargas, Sergio Orts- Escolano, Jose Garcia-Roddrigezz and Antonis Argyros. A review deep learning techniques for video prediction. TPAMI, 44(6):2806 2826, 2020.
Junheum Park, Jintae Kim, and Chang Su Kim. Biformer: Learning bilateral motion estimation via bilateral transformer for 4k video frame interpolation. In CVPR, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.0971 09744 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, and Bo Du. Multi-modal auto-regressive modeling via visual tokens. In ACM MM, 2024.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Sorkine-Hornung, and Lue Van Gool. 2017.
The 2017 davis challenge on video object segmentation.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
R Rakhimov, D Volkhonskiy, A Artemov, D Zorin, and E Burnaev. Latent video transformer. In VISI- GRAPP, 2021.
Amir Rasouli. Deep learning for vision-based prediction: A survey. arXiv preprint arXim:2007.0009: 2020.
Fitsum A Reda, Guilin Liu, Kevin Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, and Bryan Catanzaro. Sdc-net: Video prediction using spatially-displaced convolution. ECCV, 2018.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning internal representations by error propagation, 1985.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NewrIPS, 2016.
Sand-AI. Magi-1: Autorcgressive vidco generation scale, 2025. URL https://static.mag...o static/files/MAGII pdf
Christoph Schuhmann, Robert Kaczmarczyk, Aran Komatsuzaki, Aarush Kalla, Richard Vencu, Romain Beaumont, Jenia Jitsev, Theo Coombes, and Clayton Mullis. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. In NewrIPS Workshop Datacentric AI, number FZJ-2022-00923,, Jülich Supercomputing Center, 2021.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In TCPR, 2004.
Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, et al. Seaweed-7b: Cost-effective training of video generation foundation model. arXiv preprint arXiv:2504.0663 2025.
Bipasha Sen, Aditya Agarwal, Vinay Namboodiri, and C. V. Jawahar. INR-V A continuous representation space for video-based generative tasks. Trans. Mach. Learn. Res., 2022, 2022.
32

---

Published in Transactions on Machine Learning Research (07 2025))
Cuifeng Shen, Yulu Gan, Chen Chen, Xiongwei Zhu, Lele Cheng, Tingting Gao, and Jinzhi Wang. Decouple content and motion for conditional image-to-video generation. In AAAT, 2024.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolu- tional lstm network: A machine learning approach for precipitation nowcasting. In NeurIPS, 2015.
Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. Deep learning for precipitation nowcasting: A benchmark and a new model. In NewrIPS, 2017.
Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming- Hsuan Yang. Video frame interpolation transformer. In CVPR, 2022.
Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. Xvfi: extreme video frame interpolation. In ICCV, 2021.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint ar Xrv:22200. 14792, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songvang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video gener- ation without text-video data. In TCLR, 2023. URL http:p////pppppeeeeeeaer/r/rrret wtt.:///.oorrrrrddddd//lglggggg.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A contimuous video generator with the price, image quality and perks of stylegan2. In CVPR, 2022.
Samuel L Smith, Andrew Brock, Leonard Berrada, and Soham De. Convnets match vision transformers at scale. arXiv preprint arXiv:2310. 16764, 2023.
Li Song, Xun Tang, Wei Zhang, Xiaokang Yang, and Pingjian Xia. The sjtu 4k video sequence dataset. In QoMEX, 2013.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv:1212.00 0400, 2012.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representa- tions using lstms. In ICML, 2015.
Deqing Sun, Charles Herrmann, Fitsum Reda, Michael Rubinstein, David J Fleet, and William T Freeman. Disentangling architecture and training for optical flow. In ECCV, 2022.
Mingzhen Sun, Weining Wang, Xinxin Zhu, and Jing Liu. Moso: Decomposing motion, scene and object for video prediction. In CVPR, 2023.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXio preprint arxiv:2406 206.0556, 2024a.
Rui Sun, Yumin Zhang, Tejal Shah, Jiahao Sun, Shuoying Zhang, Wenqi Li, Haoran Duan, Bo Wei, and Rajiv Ranjan. From sora what we can see: survey of text-to-video generation. arXiv preprint ar Xiv:2405.1100 2024b.
Cheng Tan, Siyuan Li, Zhangyang Gao, Wenfei Guan, Zedong Wang, Zicheng Lin, Lirong Wu, and Stan Z Li. Openstl: A comprehensive benchmark spatio-temporal predictive learning. NeurIPS, 2023.
Zhiyu Tan, Xiaomeng Yang, Luozheng Qin, and Hao Li. Vidgen-Im: A largo-scale datasct for text-to-video generation. arXiv preprinl arXio:2408. 02669, 2024.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
Bart Thomoe, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59 73, 2016.
33

---

Published in Transactions on Machine Learning Research (07/2025)
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In CVPR, 2018.
Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines. In The Thirteenth ICLR, 2025. URL
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017.
Balakrishnan Varadarajan, Ahmed Hefuy, Avikalp Srivastava, Khaled Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguclov, et al. Multipath+++: Efficient information fusion and trajectory aggregation for behavior prediction. In ICRA, 2022.
Angel Villar- Corrales,,,,, Ani Karapelyan, Andreas Boltres, and Sven Behnke. Mspred: Video prediction multiple tempora scales with hierarchical recurrent networks. In British Machine Vision Conference (BMVC), 2022.
Angel
object dynamics and interactions. In Internation Conference on Image Processing (ICIP), 2023. Ismail Wahdan, and Sven Behnke. Object-centric video prediction via docoupling of
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In TCML, 2017.
Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In ICLR, 2022.
Vikram Voleti, Alexia Dolicoer-MMartnean and Chris Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. In NewrIPS, 2022.
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video forecasting by gencrating pose futures. In ICCV, 2017.
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with VQVAE. CoRR, abs/2103.01950, 2021. URL htt.s:tpppp. //arxi: .rg. htgg/abs/2103. 01950
Qinghe Wang, Yawen Luo, Xiaoyu Shi, Xu Jia, Huchuan Lu, Tianfan Xuc, Xintao Wang, Pengfei Wan, Di Zhang, and Kun Gai. Cinemaster: 3d-aware and controllable framework for cinematic text-to-video generation. In SIGGRAPH, 2025a.
Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Vidco-to-vidoo synthesis. In NewrIPS (NeurIPS), 2018.
Xiang Wang, Hangjic Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. NewrIPS, 2023a.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, and Jiwen Lu. Worlddreamer: Towards general world models for video generation via predicting masked tokens. arXiv preprint arXiv:2401.09985, 2024a.
Xiaojuan Wang, Boyang Zhou, Brian Curless, Ira Kemelmaacher-Shizzernaa Aleksander Holynski, and Steve Seitz. Generative inbetweening: Adapting image-to- models for keyframe interpolation. In The Thirteenth ICLR, 2025b. URL
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In CVPR, 2023b.
Xinlong Wang. Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Towards segmenting everything in context. In ICCV, 2023c.
34

---

Published in Transactions on Machine Learning Research (07 (07/)025)
Yanhui Wang, Jianmin Bao, Wenming Weng, Ruoyu Feng, Dacheng Yin, Tao Yang, Jingxu Zhang, Qi Dai, Zhiyuan Zhao, Chunyu Wang, et al. Microcinema: A divide-and-comquurr approach for text-to-video generation. In CVPR, 2024b.
Yaohui Wang, Xin Ma, Xinyuan Chen, Antitza Dantcheva, Bo Dai, and Yu Qiao. Leo: Generative latent image animator for human video synthesis. IJCV, 2024c.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvide02: Scaling foundation models for multimodal video understanding. In ECCV, 2024d.
Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. In NeurIPS, 2017.
Yunbo Wang, Lu Jiang, Ming-Hsuan Yang, Li-Jia Li, Mingsheng Long, and Li Fei-Fei. Eidetic 3d LSTM:
A model for video prediction and beyond. In ICLR, 2019. URL http:////ppeeeeeeeeeeeeeeeeeeeee B11KS2AqtX
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 13(4))600-6)) 612, 2004.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR, 2020. URL
Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jianmin Bao, Yuhui Yuan, et al. Art-v: Auto-regressive text-to-video generation with diffusion models. In CVPR, 2024.
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In CVPR, 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual
synthesis pre-training for neural visual world creation. In ECCV, 2022a.
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pro-training for visual robot manipulation. In The Twelfth ICLR, 2024a. URL httt::::/popeerreeeeeeeetttrrrrp http:///ww.....eeo.oooooooooooaa
Tong Wu, Yu-Jie Yuan, Ling-Xing Miio Zhang, Jie Yang, Yan-Pei Cao, Ling-QU Qi Yan, and Lin Gao. Recent advances in 3d gaussian splatting. Computational Visual Media, 10(4):613 642, 2024b.
Yue Wu, Rongrong Gao, Jacsik Park, and Qifeng Chen. Future video synthesis with object motion prediction. In CVPR, 2020.
Yue Wu, Qiang Wen, and Qifeng Chen. Optimizing video prediction via vidco frame interpolation. In CVPR, 2022b.
Ziyi Wu, Nikita Dvorník, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. In ICLR, 2023. URL http:t/ss//oppeeeriiieeeetfooorr
1d=TFbwV6I0VL
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Läu, Gongye Liu, Xintao Wang, Ying Shan, and Tien-Tsin Wong. Dynamicrafter: Animating open-domain images with video diffusion priors. In ECCV, 2024.
Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu-Gang Jiang. A survey on video diffusion models. ACM Computing Surveys, 2023.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jicbo Luo. Learning to generate time-lapse vidoos using multi-stage dynamic generative adversarial networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
35

---

Published in Transactions on Machine Learning Research (07/2025)
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024.
Ziru Xu, Yunbo Wang, Mingsheng Long, Jianmin Wang, and M KLiss. Predenn: Predictive learning with cascade convolutions. In IJCAI, 2018.
Hongwei Xue, Bei Liu, Huan Yang, Jianlong Fu, Houqiang Li, and Jiebo Luo. Learning fine- the-grained motion embedding for landscape animation. In ACM MM, 2021.
Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution vidco-language reprosentation with large-scale video transcriptions. In CVPR, 2022.
Tianfan Xue, Jiajun Wu, Katherine Boumna,, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. NeurIPS, 29, 2016.
Tianfan Xue, Baian Chen, Jiajun Wi, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. IJV, 127:1106-1225, 2019.
Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et al. Generalized predictive model for autonomous driving. In CVPR, 2024.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation.
Entropy, 25(10):1469) 2023.
Serin Yang, Taesung Kwon, and Jong Chul Ye, VibiDSampler: Enhancing video interpolation using bidi- rectional diffusion sampler. In The Thirteenth ICLR, 2025. URL /tttps://openreviex http://wwww.ee/oumimimimeme
nNYA7tcJSE
Xi Ye and Balllumme-Alexndeee Bilodeau. Video prediction by efficient transformers. Image and Vision Computing, pp. 104612, 2023.
Yufei Ye, Maneesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. ICCV, 2019.
Hongwei Yi, Shitong Shao, Tian Ye, Jiantong Zhao, Qingyu Yin, Michael Lingelbach, Li Yuan, Yonghong Tian, Enze Xie, and Daquan Zhou. Magic 1-for-1: Generating one minute vidco clips within one minute. arXiv preprint arXi::2502. 07701, 2025.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv:2308.000 08089, 2023.
Kihwan Yoon, Yong Han Kim, Sungjei Kim, and Jinwoo Jeong. Iam-vfi: Interpolate any motion for vidco frame interpolation with motion complexity map. In ECCV, 2024.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingving Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning, In CVPR, 2020.
Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, and Wayne Wu. Celebv-text: A large-scale facial text-video dataset. In CVPR, 2023a.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming Hsuan Yang, Yuan Hao, Irfan Essa, al. Magvit: Masked generative video transformer. In CVPR, 2023b.
36

---

Published in Transactions on Machine Learning Research (07/2025)
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauplmann, Boqing Gong, Ming-Hsuan Yang, Trfan Fssa, David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In The Twelfth ICLR, 2024. URL htttps://pperrevieett wttp:/twwweeeeeeeeeeeeeeeeeeeeee ntt/ferumnrrne htt.//ormmmmdddggiee giggrggC
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awarr implicit generative adversarial networks. In ICLR, 2022. URL https://
openreview wtt.:aaneeettooouodddddccdeeeeee
Shenghai Yuan, Jinfa Huang, Xianyi He, Yunyang Ge, Yujun Shi, Liuhan Chen, Jiebo Luo, and Li Yuan. Identity-preserving text-lo-video generation by frequency decomposition. In CVPR, 2025a.
Shenghai Yuan, Jinfa Huang, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, and Jiebo Luo. Magictime: Time-lapse video generation models as metamorphic simulators. TPAMI, 2025b.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
Guozhen Zhang, Chunxu Liu, Yutao Cui, Xiaotong Zhao, Kai Ma, and Limin Wang. VFIMamba: Video frame interpolation with state space models. In NeurIPS, 2024. URL https:://pperreiiie.neeet/// Id=4s5UsBUsUS!
Jiangning Zhang, Chao Xu, Yong Liu, and Yunliang Jiang. Dtvnet+: A high-resolution scenic dataset for dynamic time-lapse video generation. arXiv preprint arXiv:2008. 04776, 2020a.
Kai Zhang, Shuhang Gu, and Radu Timofte. Ntire 2020 challenge on perceptual extreme super-resolution: Methods and results. In CVPR workshops, 2020b.
Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action- conditioned contrastive policy pretraining. European Conference on Computer Vision (ECCV), 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as a perceptual metrie. In CVPR, 2018.
Runze Zhang, Guoguang Du, Xiaochuan Li, Qi Jia, Liang Jin, Lu Liu, Jingjing Wang, Cong Xu, Zhenhua Guo, Yaqian Zhao, et al. Dropletvideo: A dataset and approach to explore integral spatio-temporal consistent video gencration. arXiv preprint arXiv:2503.0600
2025.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv:2311.0414 2023.
Weiyu Zhang, Menglong Zhu, and Konstantinos G Derpanis. From actemes to action: A strongly-supervised representation for detailed action understanding. In ICCV, 2013.
Yunzhi Zhang, Wilson Yan, Pieter Abbeel, and Aravind Srinivas. Videogen: generative modeling of videos using vae and transformers. 2020c.
Boming Zhao, Yuan Li, Ziyu Sun, Lin Zeng, Yujun Shen, Rui Ma, Yinda. Zhang, Hujun Bao, and Zhaopeng Cui. Causianppedictiin:: Dynamic 3d gaussian prediction for motion extrapolation and free view synthesis. In SIGGRAPH, 2024.
Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, et al. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness. arXiv preprint ar 25w:2503.217553 2025.
Yiqi Zhong, Luming Liang, Ilya Zharkov, and Ulrich Neumann. Mmvp: tion. In ICCV, 2023.
based video predic-
37

---

Published in Transactions on Machine Learning Research (07/2025)
Yufan Zhou, Haiwei Dong, and Abdulmotaleb El Saddik. Deep learning in next-frame prediction: A bench- mark review. IEEE Access, 8:69273-6005 -62283, 2020.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022.
Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, et al. is sora a world simulator? a comprehensive survey on general world models and beyond. arXiv preprint arXiv:2405.00 03520, 2024.
Cailin Zhuang, Ailin Huang, Wei Cheng, Jingwei Wu, Yaoqi Hu, Jiaqi Liao, Zhewei Huang, Hongyuan Wang, Xinyao Liao, Weiwei Cai, et al. Vistorybench: Comprehensive benchmark suite for story visualization. arXiv preprint (rXxi:2505.248800 2025.
Yueying Zou, Peipei Li, Zekun Li, Huaibo Huang, Xing Cui, Xuannan Liu, Chenghanyu Zhang, and Ran He. Survey on ai-generated media detection: From non-mllm to mllm. arXiv preprint arXiv:250..05240, 2025.
38


Reviews:
[{"review_id": "FBrBONexNn", "reviewer": "Reviewer_6qW7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nOverall, the survey is valuable to the community. It provides extensive explanations on the latest future frame synthesis methods since this topic has been one of the most attractive and critical venue in content creation and generation field.\n\nThe authors have conducted extensive surveys on methods, from deterministic, stochastic to generative models, to illustrate current mainstream models.\n\nWeakness:\n\n1. In section 2.2, since authors mentioned broad methods in three categories, I think it would be great to briefly mention all types of models, transformers, diffusions, flow matching, etc.\n\n2. in 2.3.2, it is vague to simply mention short-term or long-term video synthesis. Since with the capabilities of the diffusion models, the video length can be extremely extended and major large foundational video models have shown impressive results in predicting/generating long videos. It is better to clarify and provide more context, how many seconds do short-term videos have?\n\n3. Since this survey focuses on methodology, while I think it is worth discussing beyond architecture details, for example, how multimodal information is handled? Whether cross-attention layers have been used?\n\n4. For equation (1), (2), I think it is a bit vague. I would recommend also consider modalities (not just controls) and refine equations in a general format.\n\n5. Overall, since authors mentioned that this survey focuses on method, but I think this survey misses too many details of method/models. For instance, it would be great to explain details of self-attention and how successful it is, similarly to convolutions.\n\n6. Minor comment: given the facts that we have seen a number of large foundational video generation models from industries. It would be great to summarize technologies used by industries as well.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a survey on future frame synthesis by introducing the dataset, methods and applications related to FFS tasks. The survey divide the methods of the topic into three aspects, deterministic, stochastic and generative models. The survey focuses on exploring different methods for FFS.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have listed the requested changes in the weakness section. Please reference it.", "broader_impact_concerns": "I do think it is worth mentioning the impact concerns since content creation/generation can easily cause/raise safety/fairness issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eBTo9Fg65k", "reviewer": "Reviewer_HFqA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written and easy to read.\n- The break down of the algorithms by stochasticity is intuitive and easy to follow.\n- The challenges sections associated with each method discussed highlights and makes clear where the field is.\n\nWeaknesses\n- I am not sure how novel the stochasticity-based taxonomy is", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is a survey that covers future frame synthesis (FFS). Several surveys exist in the space of future frame synthesis, however, this survey focuses on framing algorithms and approaches based on how well they cover stochasticity. After defining the FFS problem and using a broad definition that incorporates conditioning on human guidance and non-human created auxiliary data. The survey breaks the problem of future frame synthesis by deterministic, stochastic, and generative approaches. Each of the three approaches is then further broken down by more refined methods types, and the challenges of each are discussed. Beyond the summary of the field provided, the contribution is provided in the conclusion where the recommendation to split FFS research into two subfields is provided. One to focus on high-definition video applications, and the second developing algorithms that learn to appropriately model physics and the world.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None", "broader_impact_concerns": "I have no concerns about broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zZZS6jgGhZ", "reviewer": "Reviewer_H6Vr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The technical language and descriptions are generally clear and easy to follow.\n\n* The discussion on the methodological shift from deterministic to generative approaches is particularly interesting and provides valuable insights.\n\nWeaknesses: please see the comments below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a survey on the Future Frame Synthesis (FFS) problem, which focuses on generating future frame sequences conditioned on existing content. It reviews relevant datasets and methods—both deterministic and generative—to provide a comprehensive overview of the field.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be helpful to add the recent survey paper on video diffusion models [1] to provide a more comprehensive overview.\n\n2. I suggest including more recent works on video generation evaluation [2–3] in Section 2.3.1. Although they are not specifically designed for FFS, comparisons and discussions with them could enrich the survey and offer valuable perspectives.\n\n3. Recent works have attempted to establish connections between neural video synthesis and 3D representation learning (such as 3DGS) [4–5]. It may be interesting to discuss whether these developments can offer any insights for the FFS task.\n\n4. It would strengthen the paper to include a dedicated section discussing future trends and highlighting some of the most promising directions based on the current progress.\n\n5. For consistency, please clarify sub-subsection titles by removing the periods at the end (e.g., changing \"3.1.1 Recurrent networks.\" to \"3.1.1 Recurrent networks\") to maintain a uniform style throughout the paper.\n\n6. Consider adding more figures to illustrate the problem formulation, representative datasets, and task pipelines, which could help improve the paper’s readability. There is not any figure in the paper yet.\n\n7. While involving stochasticity in modeling approaches is an important shift, it does not necessarily imply a method is truly \"generative,\" where the goal is typically to maximize the likelihood of the observed data samples. It might be helpful to clarify this distinction more explicitly in the paper.\n\n\nRef: \\\n[1] Melnik A, Ljubljanac M, Lu C, Yan Q, Ren W, Ritter H. Video diffusion models: A survey. arXiv preprint arXiv:2405.03150. 2024 May 6. \\\n[2] Huang Z, He Y, Yu J, Zhang F, Si C, Jiang Y, Zhang Y, Wu T, Jin Q, Chanpaisit N, Wang Y. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024 (pp. 21807-21818). \\\n[3] Liu J, Qu Y, Yan Q, Zeng X, Wang L, Liao R. Fr\\'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos. arXiv preprint arXiv:2407.16124. 2024 Jul 23. \\\n[4] Lei J, Weng Y, Harley A, Guibas L, Daniilidis K. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. arXiv preprint arXiv:2405.17421. 2024 May 27. \\\n[5] Chen Z, Wang Y, Wang F, Wang Z, Liu H. V3d: Video diffusion models are effective 3d generators. arXiv preprint arXiv:2403.06738. 2024 Mar 11. \\", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)