Video-GPT via Next Clip Diffusion
Authors: Shaobin Zhuang
Zhipeng Huang
Ying Zhang
Fangyikang Wang
Canmiao Fu
Binxin Yang
Chong Sun
Chen Li
Yali Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Video; Diffusion; LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: Video-GPT treats video as new language for visual world modeling.

Abstract: GPT has shown its remarkable success in natural language processing. However, the language sequence is not sufficient to describe spatial-temporal details in the visual world. Alternatively, the video sequence is good at capturing such details. Motivated by this fact, we propose a concise Video-GPT in this paper by treating video as new language for visual world modeling. By analogy to next token prediction in GPT, we introduce a novel next clip diffusion paradigm for pretraining Video-GPT. Different from the previous works, this distinct paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, by autoregressively denoising the noisy clip according to the clean clips in the history. Extensive experiments show our Video-GPT achieves the state-of-the-art performance on video prediction, which is the key factor towards world modeling (Physics-IQ Benchmark: Video-GPT 34.97 vs. Kling 23.64 vs. Wan 20.89). Moreover, it can be well adapted on 6 mainstream video tasks in both video generation and understanding, showing its great generalization capacity in downstream.

VIDEO-GPT VIA NEXT CLIP DIFFUSION
Shaobin Zhuang!.* Zhipeng Huang? 2
Ying Zhang
Fangyikang Wang Yali Wang
Canmiao Fu
Binxin Yang* Chong Sun Chen Li
Shanghai Jiao Tong University University of Science and Technology of China *Zhejiang University 'WeChat Vision, Tencent Inc.
"Shenzhen Key Lab of Computer Vision and Pattern Recognition, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences SShanghai AI Laboratory
(hahahahaha) mrrjju.edd... (kenelmfu, binxinyang, waynecsun, chasellij jtasellij@tencett @tencent. com wangfangyikangeea ju.edu.ueue..m en yl.wang@siat.ac.
(ethanphuang, yinggzhang) ) @tencent.com
ABSTRACT
GPT has shown its remarkable success natural language processing. However, the language sequence is not sufficient to describe spatial-temporal details in the visual world. Alternatively, the video sequence is good at capturing such de- tails. Motivated by this fact, we propose concise Video-GPT in this paper by treating video as new language for visual world modeling. By analogy to next token prediction in GPT, we introduce novel next clip diffusion paradigm for pretraining Video-GPT Different from the previous works, this distinct paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, by autoregressively denoising the noisy clip according to the clean clips in the history. Extensive experiments show our Video-GPT achieves the state-of-the-ar performance on video prediction, which is the key factor towards world model- ing (Physics-IQ Benchmark: Video-GPT 34.97 vs. Kling 23.64 vs. Wan 20.89). Moreover, it can be well adapted on 6 mainstream video tasks in both video gener- ation and understanding, showing its great generalization capacity in downstream.
INTRODUCTION
Over the past few years, natural language processing has been mainly driven by training large lan- guage models (LLMs) from web-scale text data. In particular, GPT series Radford & Narasimhan (2018); Radford et al.] (2019); [Bronn et al. (2020); Achiam et al. (2023) have demonstrated re- markable generalization capabilities on various downstream language tasks, which further opens the possibility towards general artificial intelligence. However, the language sequence is often good at expressing high-level abstractions, while it may not be sufficient to capture rich spatial-temporal details in the visual world Liu et al. (2024); Yang et al. (2024b). For example, it is difficult to de- scribe how to tie a knot by language. Alternatively, the video sequence has been considered as a preferred candidate describe such details, since world at different spatial and temporal resolutions Bai et al. (2023); Liu et al. (2024). Hence, there language for visual world modeling?
record visual knowledge of our dynamical
natural question: Can we treat video
The recent studies have shown that, video generation is a promising direction to achieve this goal OpenAl| (2024); [Bruce al. (2024; [Kondratykk et al. (2023). The typical design is video diffu- sion by adding noise and denoising gradually Song et al. (202T): Ho et al. (2020). Although such (2024); Kong et al. (2024); Kuaishoul 2004); Wang et al. (2025). it often suffers from difficulty in long-term future prediction that is critical factor of world modeling Ouyang et al. (2024); Li al. (2024). To address this problem, the autoregressive attempts have been made for long-context video modeling, on analogy of next
paradigm has achieved significant progress Bar-Tal
token prediction in L.MS Kondratyuk al. 2023) Liu et (004); Bai et 2023). However,
Work done as interns at WeChat Vision, Tencent Inc.
"orresponning author.

---

Songling Words Tixt
Sumpling Cltys Video
Sanpling Clips Adding
ribbow Arounsl box (ürst annd then pust
Clip
Tic
ribbon
around
the
GPT
ribbon
around
the
box
(a) Language -+ GPT
(b) Video - Video-GPT
Next clip diffusion. We draw an analogy with GPT's next token prediction and model each video clip visual word by denoising the next noisy clip, conditioning on the previous video.
Figure I:
compared with its diffusion counterpart, the generation performance of this paradigm still needs to be further improved. In order to leverage ad advantages from both paradigms, several approaches combine diffusion and autogregressive modeling
unified transformer style Xiao al. (2024);
Deng et al. (2024); Zhou et al. (2024); Huang et al. (2025b); Hu et al. (2024); Deng et al. 2025
But these design mainly works on the image domain, without insightful analogy between language
and video.
To this gap, we propose concise Video-GPT, by analogy with generative pretraining design in GPT. Inspired by next token prediction in GPT, we introduce novel nexl clip diffusion paradigm for our Video-GPT. Specifically, we creatively treat a clip in the video as the role of a word in the language, since both of them describe local temporal information, respectively, in the video and language sequence. However, different from a discrete word token, it is often challenging to predict continuous video clip in the next step. To address this difficulty, design within an interleaved sequence of noisy and clean clips in the temporal order. As shown in Fig. [] one can randomly sample number of clips from training video. For each clip, we generate its noisy clip by adding noise on it, based on the forward process of diffusion. Then, we re-arrange noisy and clean clips temporal order, constructing an interleaved clip sequence. Consequently, for each noisy clip, we leverage the clean clips in the previous as context, and reconstruct the corre- sponding clean clip by the diffusion loss. In this case, our Video-GPT inherits advantages from both GPT and diffusion, but within the basic unit of a video clip. This allows it to capture long-term video prediction as well short-term video generation for effective generative pretraining on videos.
introduce flexible diffusion
Our contributions can be summarized as follows. First, we introduce concise and self self-supervised pretrained Video-GPT, by analogy to GPT. Different from GPT, our Video-GPT aims at modeling rich spatial-temporal information in the visual world. Second, we design novel next clip diffu- sion paradigm for pretraining Video-GPT. Diffcrent from the previous works mentioned above, this hybrid paradigm allows Video-GPT to tackle both short-term generation and long-term prediction, via integrating diffusion and autoregressive design in the video clip level. Finally, our pretrained Video-GIP' achieves state-of-the-art performance on Physics-IQ Benchmark and Kinetics-600, fully demonstrating its potential for visual world modeling. After fine-tuning on 6 downstream video gen- eration and understanding tasks, it also achieved preferable performance for generalization.
2 RELATED WORK
Video Diffusion Models. With diffusion models demonstrating remarkable performance in im- age generation Saharia al. (2022); Rombach et al. (002)) Dhariwal Nichol (202)) y; Ramesh et al. (2022), video diffusion models Ho et (202)) also surpasses traditional methods Clark et al.
(2019); Saito
Saito et al. (2018) 8): Kahembwe & Ramamoorthy (2019) in video generation. With the emer-
gence of large-scale text-video datasets Chen et al. (2024b); Wang et al. (202cc) 2023c). text-to- video diffusion models Blattmann et al. (2023) Zhuang et al. (2024): Yang et al. (202cc); OpenAl (2024); [ang et al. (2255; ; Kong et al. (2024); Wang et al. (202b)) exhibit realistic generation capa- bilities and strong transferability to downstream tasks Yue et al. (2025): Hu & Xu (2023) However, the supervised paradigm based text annotations faces two challenges scales up. 1) Quantity
2

---

00
00
0000
Video-UPP
Video-EPP Pra-Tnining
Ourgee NS Noiey Cliys Cl Clean
IITT- (SOV MO. EAEVE MICA
MILLO
Abcrate Mark Thanse Parch Lord
MGGA)
Abw
Forine
ASME Modk Porch
Staai
Figure 2: Video-GPT pretraining framework. The full attention mask is shown in Fig. 10
Table 1: Progressive training strategy.
Stage
Resolution
Frame Num.
Frame Interval
Clip Num.
Train Steps
Flexible
16
4
16
300000
2
Flexible
48
4
Uniform (2,3, ....... 48
25000
3 Flexible
Uniform (4,5, 4,5.---,
Uniform [2,3,
48
40000
Flexible
Uniform 4, 5, 12]
Uniform 3, .80)
20000
of Annotated Data: Current methods require meticulous data filtering and annotation processes Menapace et al. (2024) Polyak al. (2024), resulting substantially smaller data scale compared to LLM et al. (2024): Yang et al. (2024a); Kaplan et al. (2020). 2) Quality of Annotated video. While increasing the granularity of descrip- Data: Text cannot fully capture the meaning
tions Chen et al. (2023a): Betker et al.) (Segalis et al. (2023) alleviates this issue, the difficulty of annotating such descriptions significantly increases as video length scales up.
Autoregressive Video Models. Image' TrageeTranfommer PaansfforeerPPrmmarr Parmar et al. (2018) migrated next token predic- tion image generation pixel space. Chameleon Team (2024) and LlamaGen Sun al. (2024) perform pretraining images in latent space. LVM Bai et al. (023) and Video World Ren et al. (2025) reveal that models can acquire world knowledge from videos. LWM Liu et al. (2024) com- presses videos into latent space for pretraining, but focuses video understanding capabilities. In general, next token prediction based visual generation Wang et al. (2024d) still underperform com- pared with the most advanced diffusion modelssLabs (2024) y Liao al. (2025); Wang et al. (2025); Kong et al. (2024). contrast, our Video-GPT adopts next clip diffusion paradigm, which diffu- sion generation within video clips and applies autoregressive generation between clips. It enhances video generative modeling while utilizing historical information as a self-supervised condition.
Video Generation Models with Hybrid Modeling. Recently, video generation models that hy- bridize autoregressive and diffusion modeling have begun to emerge. Self-forcing Huang et al. (2025a) based on the paradigm of next frame prediction, using diffusion for generation within each frame, and employing autoregression between frames. APT2 Lin et al.] (2025) introduces the autoregressive adversarial post-training to achieve real-time video generation. Video-GPT differs from them in two fundamental ways: 1) Self- through self-supervised pre-training similar to LLM, does not require any annotations other than videos in this process. Through pre-training (2024b). we achieve and surpass the physical world modeling capabilities models like Gen3 Runway (2024) and Sora OpenAl (024). 2) Next Clip Diffusion: Video-GPP is autoregressive at the clip level. This is more efficient and flexible as it allows parallel, bidirectional processing (diffusion) within multi-frame clip, but maintains temporal order between clips.
Pre-Training Video-GPT is pre-trained
the open source Panda-70M dataset Chen et al.
the top closed-souree

---

3 VIDEO-GPT
In this section, we first introduce our Video-GPT via next clip diffusion. Subsequently, we describe how adapt it for typically downstream tasks in both video generation and understanding. For better understanding, we also present the pseudocode of Video-GPT in Sup. A
3.1 INPUT: CLIP SEQUENCE CONSTRUCTION
As mentioned before, we notice that a clip in the video plays a similar role as word the language. Hence, we first divide training video into number of clips to construct the basic processing unit in our Video-GPT. Specifically, we uniformly sample N frames from each training video. Then, we randomly divide these frames into K clips, where K is sampled by K Uniform 3. N...
Forward Diffusion Process on Each Clip. Since diffusion has shown the powerful capacity of video generation, propose to leverage it to predict the next video clip based the previous videos. According to this reason, we perform the forward diffusion process on latent of each clip by adding Gaussian noise. Specifically, for the k-th clip, first apply continues VAE Rombach eet al. (2021) to compress each frame in this clip, and patchify it to obtain the latent feature of each frame (i.e., the feature matrix of all the patch tokens in this frame). Next, choose flow matching Lipman et al. (2022) for diffusion, due to its efficiency. According to flow matching, we add the
noise on the latent feature in the following way.
(k, i, 04) a (k,, (1 oR)Ek.i
(1)
where ((k, i) is the latent feature of the i-th frame in the k-th clip, and (k,,) is the noisy feature of this frame. The weight og is sampled by ag Uniform[0, 1], and the noise εx, is sampled by
Ek,s ~ N (0, I). Note that, in order to inference the content in the video clip in parallel, we apply the same og. for all frames in the video clip during training.
Noise- Clean Interleaved Clip Sequence. After getting the noisy clips, we next arrange the input sequence for our Video-GPT. By analogy of GPT, we leverage the historical clips as temporal context to denoise the noisy clip in the next step. But instead of using the noisy clips in the history CCen ct al. (0024)): Yi et 2025), we leverage the original clean clips in the history, in order denoise the next clip conditioned on the clean and correct context. Hence, we feed both noisy and clean clips as input to our Video-Gie GP.. To distinguish different frames in these clips, we add extra tokens for specializing them. 1) Clip-Level Token Form of Clean Clip: For each clean clip, we add the boundary tokens with the latent feature of each frame in this clip,
CL(k, i) [<img>, ((k, i), <limg>].
Hence, the clip-level token form of the k-th clean clip is CL(k,) :) = [CL(k, (k, 1), CL(k, No)),, where N& is the number of frames in this clip. 2) Clip-Level Token Form of Noisy Clip: For each noisy clip, we add two extra tokens with the noisy latent feature of each frame in this clip,
NS(k,) i) [<diff> ag, (k, i, ax)].
(3)
where <diff> is a denoising hint token indicate that (kk i) is noisy. Moreover, we also add the weight on for the k-th noisy clip, in order to provide the timestep information the flow matching. As result, the clip-level token form of the k-th noisy clip is NS(k; = [NS(K, 1), .... NS(k, N&)]]- 3) Noise-Clean Interleaved Input: As shown in Fig. 2 (a), after formulating both clean and noisy clips, arrange them together the input sequence to Video-GPT. Since we aim at sequential modeling, we take a pair of noisy and clean clips as group, and arrange these groups in the temporal order. As a result, we obtain an interleaved sequence of noisy and clean clips input, i.e.,
Input = :)),
..... NL( (
NS(K,)))
(4)
3.2 PRETRAINING: NEXT CLIP DIFFUSION
After obtaining the input sequence, we feed it into our Video-GPT for generative pretraining. By analogy to GPT, use the vanilla transformer architecture for conciseness. The next question is how to build up relations between noisy and clean clips for next clip diffusion. As shown in Fig. 2 (b), we introduce hierarchical masking method to indicate such relations in the attention operation.
4

---

$90.0
Video-GPT
Inférence Ateution Mask.
Figure 3: ViTeo-GPT inference framework. We iteratively denoise the 2nd noisy clip NS(2,:) :) to C2, and use it along with the 1st clean clip CL(1,;) :) to condition the prediction of the 3rd noisy clip NS(3.:). :). The number of frames in each clip also vary during inference.
Clip-Level Mask. As our input is a clip sequence, we first need to define the dependence among clips. By analogy to the dependence of words in GPT, we basically apply causal relation as clip dependence. 1) Clean Clip Mask. As shown in Fig. 2 (b), the k-th clean clip depends on itself and the previous (k 1) clean clips. 2) Noisy Clip Mask. Alternatively, the k-th noisy clip depends on itself for sure. However, instead of depending
the previous (k 1) noisy clips, it depends on the previous (k - 1) clean clips. The reason is that, these clean clips in the history provide the correct temporal context for denoising the k-th noisy clip. It is the core idea in next clip diffusion.
Frame-Level Mask. Since each clip consists of several frames, we need to further specify the dependence among frames in the defined clip-level mask. 1) Clean Frame Mask. As shown in Fig. (bg, for the i-th frame in the k-th clean clip, it depends on itself, the (i-1) 1) frames in this clean clip, and all the frames in the previous (k: 1) clean clips. 2) Noisy Frame Mask. Alternatively, for the i-th frame in the k-th noisy clip, it depends on itself, all the other frames in this noisy clip, and all the frames in the previous (k 1) clean clips. Note that, different from the causal design for the clean frame, the i-th noisy frame depends on all the other frames in the same noisy clip. The reason is that iterative denoising turns noisy clips into clean clip as history condition, the mask of noisy frames will not affect later inference and bidirectional attention can better enhance generation quality Yin et al. 2202);; Kim et al. (202I).
Patch-Level Mask. As shown in Eq. and each frame (clean or noisy) is formulated by extra hint tokens, and the latent feature of this frame which actually refers to the feature matrix of all patch tokens in this frame. Hence, we need to further specify the dependence among these hint tokens and patch tokens in the defined frame-level mask. 1) Clean Patch Mask. As shown in Fig. 2 (b), we follow the similar design of GPT, the dependence is causal among tokens, i... <img>, ((A, i), and <limg>. But differently, the clean frame feature (k, i) is actually the feature matrix of all the patch tokens in this frame. Since the patch tokens describe spatial relation rather than temporal one, define the full dependence among these patch tokens. 2) Noisy Patch Mask. Similar to clean patch mask, we define the causal dependence among tokens, i.e., <diff>, Ok, and (i i). We also define the full dependence among patch tokens in noisy frame as the same as the one in clean frame.
Training Target. Based on this hierarchical masking, our Video-GPT can effectively leverage the masked attention to denoise the k-th noisy clip in the next step, according to the previous (A: 11) clean clips of input sequence. Subsequently, we compute the Le loss between each denoised clip feature and its corresponding clean clip feature to pretrain our Video- GPT. It is worth mention- ing that, different from previous diffusion works, we employ Video-GPP to predict the video clip directly instead of noise Chen et al.] (2023a); Wang et al. (2023b) velocity Esser et al. (2024); Yang et al. (2024c), to keep the training setting as simple as possible, allowing to easily adapt the pretrained Video-GPT for various downstream tasks.
Progressive Training. As the length of the video frame increases, the computational cost of the attention operation will increase quadratically. To alleviate such computation, we leverage simple but effective progressive training strategy. Specifically, we start training from short videos to long videos. As shown in Tab. the reason is that in the initial pretraining stage use 16 frames- with each clip containing only one frame for next- frame prediction then gradually increase both the number of clips and frames per clip for Video-GPT pretraining.
5

---

Vider
Vilce
Video
Anding
Lewe
Lais
Valeo-IPT
VAG Doolis
VSl Enaddr
VAU Cronlet
MMc
Seffron Taree
(1) Video Classiiiationn
(3) Midto Retrivaal
(D Video Ofjecnt segnemrttio
Figure 4: Fine-tuning Video-GPT on downstream tasks.
1VM Conditi
Vidao GPT
Salie
Conditin
Scatc VicooGPE
LYDd
Figure 5: Qualitative results Physics-IQ Benchmark. The videos predicted by Video-GPT
based on condition frames are more consistent with physical laws than other methods.
3.3 INFERENCE: AUTOREGRESSIVE VIDEO PREDICTION
After pretraining Video-GPT, one can naturally leverage for video prediction, by analogy GPT. As shown in Fig. (a). vidco prediction can be autoregressively performed in the inference phase. Specifically, we treat the previous denoised clips (outputs of Video-GPT in the history) as the clean clips, and leverage them as temporal context to denoise the (A: 1) noisy clip,
DNS(K + 1,:) Video-GPT DNS(L.:) DNS(K,)). NS(k + +.:))
(5)
where DNS(k+1,3 is the (k++11 1)-hh denoised clip. Additionally, DNS(1,:) is clean clip which initially given as input for video prediction. Note that, the input Video-GPT this inference phase is a causal formulation. Hence, the clip-level mask is defined as the causal one for each iteration autoregressive prediction, as shown Fig. (b). We use all previously generated clips as context, up to the maximum sequence length supported by model pre-training. To generate longer videos than fit within this context window, a standard sliding window approach is adopted.
3.4 GENERALIZATION ON DOWNSTREAM VIDEO TASKS
As we noticed, GPT has shown its powerful generalization capacity various downstream NLP tasks. Hence, we expect that Video-GPT can show the similar capacity on various downstream video tasks. For this reason, we adapt Video-GPT address typical downstream video tasks in Fig. covering both video generation and understanding. For video generation, we focus on class-to- video generation, text-to-video generation, and image animation. For video understanding, we focus on video classification, video retrieval, and video object segmentation.
Class-to-Videe Generation and Text-to- Pext-to-Video Generation. As shown in Fig. we use a similar processing mode for these two types of tasks, since both generation tasks of Video-GPT tioned on extra tokens (either from a category tag or a video description). Hence, we introduce a causal attention between extra tokens and clip tokens, i.e., extra token only depends other extra tokens, while a clip token depends on both extra and clip tokens.
condi-
Image Animation and Video Object Segmentation. Since both tasks are to predict video content conditioned on the given sequence, we discuss them together here. For image animation, we are given the first frame. The goal is to predict the rest video conditioned on the first frame. For video

---

Table 2: Quantitative comparison of models evaluated on the Physics-IQ Benchmark.
Model
w/o. Text
Spatial IoU
Spatio Temporalt
Weighted Spatial IoUr
MSE]
Phys.
Scoret
Sora (12V) (12V) OpenA1
2024
X
0.138
0.047
0,063
0.030
10,00
Pika 1.0 (12V Prkal ahs 2124
X
0.140
0.041
0.078
0.014
13.00
SVD (12) Blatmaan
2023
0.132
0.076
0.073
0.021
14.80
1VM (V2V Bar el mi.
20221
0.100
0.147
0.077
0.021
18.02
Lumiere (12V) Bar-
2024
0.173
0.016
19.00
Open-Sora-Plan 1
Ct al.
2024
2024
X
0.142
0.139
0,074
0.021
19.42
Wam2.1 (12V) Wane et al.
2025
0.153
0,100
0.112
0.023
20.89
HinyuanVidea Kong
21/24
22.36
Gen 3 (12V Rumway 2202
0.201
0.115
0.116
0.015
22.80
Lumiere (V2V]Bar et 2024
0.170
0.155
0.093
0.013
23.00
Kling1 .6 (12V) Kliaishou 2124
X
0.197
0.086
0.144
0.025
23.64
LTX-Video-I2WV (F et
2024
26.80
Cogvideo 12V-5B
27.1
Scine (V2V) Chen al (V2V))Chen
50
0.163
0.208
0.131
0.012
29.13
VideoPoet (V2V) Kondral byy
2023
0.204
0.164
0.137
0.010
29.50
Video-GPT (V2V)
0.198
0,240
0.144
0.007
34.97
Table 3: Quantitative comparison of video generation models evaluated on the Kinetics-600.
Model
Parameters
Architecture
Text Encoder
364 FVD
FVD(5000)
Opes, Sora L
2024
m115-XXL Xne al. 2022
95.36
(Baile Seine
Vanilla Transformer
94.61
Rudford
2022
3.8B
Table 4: Inference setting ablation.
Number Frames Each Video Clip
History Conditioned Classifier- Free Guidance
Hyperparams.
32
2.0
2.5
3.5
Phy. IQ Score
0.00
6.21
18.44
29.90
32.86
34.94
32.18
25.09
31.96
32.22
32.65
32.72
31.88
Table 5: Training setting ablation.
Pretraining Parudigm
Pretruining Frame Number
Add Noise Clean Clip
Datuset Scule
Hyperpurams.
Next Token Prediction
Next Clip Diffiusion
16
NO
YBS
1M
70M
Pby. 10 Score
21.59
34,94
22.06
33.09
34,94
32.54
33,09
23,16
33,09
object segmentation, we are given a video and the first frame with the masked objects. The goal is to track the masked objects throughout the video. To address these tasks by fine-tuning, we leverage the input frame or video as the clean clip in our Video-GPP and the video to be generated as the noisy clip. The mask strategy follows the settings in Sec. 3.2
Video Classification and Video Retrieval. For video classification, we pool clean clip features the clean clip feature for contrastive
from Video-GPT for linear probing. For video retrieval, learning with the text feature from a text encoder, c.g., CLIP- LLP--ViT-L-Patchl- Radford et al. (2021).
4 EXPERIMENTS
4.1 IMPLEMENTATION
Dataset. In pretraining, we employ Panda-70M Chen et al. (2024b) as the training dataset. In order vertify the impact of data scale on model performance, we also employ Open Vid- Nan et al. (2244) for ablation. For downstream tasks, we fine-tune the model with supervised datasets. Specifi- cally, for class-to-video and video classification, we conduct experiments on UCF-101 [Somro al. (012)). For text-to-video generation, we fine-tune the model with OpenVid-1M Vid-1M, and for the video retrieval task, we used videos and annotated texts from the Panda-70M. For video object segmen- tation, randomly select 1000 cases from G Zhuang al. (2025). Notably, for image ciimation-whieee is high demand within the industry, there is no commonly recognized bench- mark. Therefore, we colloct three datasets from the internet to evaluate the generalization capability of pretrained Video-GPP on image animation. These datasets correspond to the categories of lie down, transform, and fly, with each dataset containing fewer than hundred videos.
Settings. In this paper, we use VAE from SDXL Podell et al. (2023), with Video-GPT inher- iting the model architecture from Phi-3-mini Abdin al. (2024). We additionally incorporated
7

---

Figure 6: Qualitative results of Video-GPT on class-to-video generation on UCF-101.
with white bears! head, wearing a plaid shürt, He is seated in voow. with booksshllf in the baskground. The naw appears to be in step thought or contemplation, as gazess off to side. The room has warn. and oozy abwoosperee, with the bookshelf
video features
bald
"A view of beach nestled cove. beach, with its light brown sand, weets the deep bluce water the edgee. on the right side the cove, there's a hill. Above all this, thee siey a bright bluce and oloudllsss. The mountains in the distance are staoked up- Away from the kustle and bustle city life.
fillest with various bookes papers."
HELLY
Figure 7: Qualitative results of Video-GPT on text-to video-generation.
clean_input input and no no loised__r lesed_inpee input layers Xiao et al. (2024) to transform VAE-encoded image latents into tokens, well no moiseed_sot sed_output layer convert output tokens back into image latents, with these components having significantly fewer parameters than the main transformer. During the pretraining phase, we employ 320 H20 GPUs and implement a progressive training strat- egy as shown in Tab. More pretraining and fine-tuning settings can be found in Sup.
4.2 PRETRAINED MODEL. EVALUATION
We evaluate pretrained Video-GPT performance on Physics-IQ Benchmark Motamed et al. (2025) and Kineticx-600(Carree Carreira et al. (2018), which are used to evaluate the ability of models in predicting deterministic and uncertain videos respectively. We set the historical conditioned classifier-free guidance scale c=3. Unless specified all subsequent experiments keeps the same setting.
Physies-IQ Benchmark. Physics-IQ Benchmark presents short video clip (3 seconds) depicting real-world physical motion and asks the model to predict future frames (5 seconds). Since macro- scopic physical laws typically exhibit deterministic behavior, we used the Physics-IQ Benchmark to assess the model's ' ability to predict deterministic future events. As shown in Tab. g Video-GPT has superior performance in modeling the physical world (more than 5 points higher than the 2nd). In addition to Video-GPT, the results of LVM [ai et al. (2023). Open-Sora-Plan v1.3 Lin et al. (2024), and Seine Chen al. (2023b) are also obtained by our own testing. This result shows that the model based on next clip diffusion pretraining paradigm can better learn world knowledge from video data.
Kinetics-600. Kinetics-600 contains 0.3M human motion videos, and human motion often known for being unpredictable. Therefore, we measure the distance Unterthiner et al.) (2018) between model-generated videos and videos from Kinetics-600 to evaluate the model's 's capability to predict highly uncertain future events, requiring the model to predict 13 future frames based on the 3 given frames. As shown in Tab. [3] Video-GPT achieves the best FVD among all models with a vanilla transformer architecture, rather than the more popular U-Net Ronneberger et al. (2015) DiT Pee- bles & Xie (2022). which fully demonstrates the effectiveness of our pretraining.
Visualization. As shown in Fig. 5 Video-GPT accurately predicts the water filling and machine lifting physics, while other methods Bai et al. (2023): Chen et al. (2023b) show obvious errors.
4.3 ABLATION STUDY
We ablate the properties of Video-GPT pretraining and inference in the Physics-IQ Benchmark. Unless specified, we perform ablation experiments based the Video-GPT Stage 3 in Tab. |
8

---

Table 6: Class to video quantitative comparison on the UCF-101. Table 7: Video classification
linear probe on UCF-101.
Model
Resolution
Architecture
YAE
FVDL
Cog Vider Hong al.
2022
160
160
Dual-channel Transformer
20
626
Method
Top-1
MalieMMaee al
256 256
DrT
2D
478
MemDPC Uha 2020
56.1
TATS srai.
128 128
Teme-Sensitvee Transformer
332
VideoMAEY AAAEI/IEE 20232
56.4
Omni Tokenaze Whils 2024b
256 256
Vanilla Transformer
3D
191
Video-GPT
VideoFusion
U-Net
Table 8: Video retrieval
MAGVITy2-7WES X et 21124
Vanilla Transformer
3D
109
ACDIT(Hu
DET
zero-shot on the MSR-VTY
Make-A-Vinn Vio.o Singg Vt 2022
256 256
U-Net
2D
81
MAGVITv2 Yi A074
Vanilla Transformer
3D
58
Method
Re
LARP Wang al 20243
128 128
Vanilla Transformer
3D
57
VideoCLIP Xu et
2021
10.4
FAR
128
128
DrT
2D
57
Supporteel Parick 2020
12.7
Video-GPT (from scratch)
240
320
Vanilla Transformer
2D
489
FIT Bain ot a. 2021
18.8
Video-GPT (finetune)
240
320
Vanilla Transformer
2D
53
Video-GPT
22.8
<lie down?
Strunsform
S/y>
Figure 8: Qualitative results of Video-GPT on lie down, transform and fly image animation. The frame in the blue box the prefix frame we are given as history condition.
Figure 9: Qualitative results of Video-GPT video object segmentation. The frames in blue box are condition frames. The frame in red box is the 1-st frame of the condition with object mask, and we circle the segmented object red. Frames in green box are the segmentation result.
Number of Frames in Video Clip for Inference. shown Tab. within a certain range, as the number of frames N₁ processed in parallel for inference within each video clip increases, the quality of the videos generated by the model improves significantly. This thoroughly validates the superiority of our proposed next clip generation paradigm for vidco generation tasks.
History Conditioned Classifier-Fres Guidance Scale for Inference. We conduct experiments based on N,=16. As shown in Tab. 4. the Stage 4 model shows the strongest performance when
adjust the scale of history conditioned classifier-free guidance Ho (2022) c=3.0.
Number of Frames for PreTraining. As shown in Tab. the results show that as the number of pretraining frames increases, the videos generated by Video-GPT are more and more line with physical laws. This reveals that the time window of model pretraining increases, the model's modeling of world knowledge is getting better and better.
Add Noise to Clean Clip for Pretraining. During inference, there is deviation between DNS and CL. To fill this deviation, we add slight noise to clean frames during training as
& (k. i) = (3 (k.
(k. i) + 3- J
(6)
where 33 and Yk.s Uniform(0,1 refer the basic retention degree and random retention degree of the clean frame respectively and cx, is sampled by εk,i / I). We set 3=0.9. As shown in Tab. 5 the result shows that adding noise to the clean clip during training improves model performance.
Dataset Scale for Pretraining. We conduct ablation study on the impact of dataset scale based on Open Vid- 1M and Panda-70M. As shown in Tab. 5 the results demonstrate that after expand- ing the scale of videos for pretraining, Video-GPT's modeling capability for the world significantly improved. This further indicates that Video-GPP Mede has substantial room for improvement, as its pre- training utilize almost all video data the internet without annotations, similar GPT.

---

4.4 DOWNSTREAM TASK EVALUATION
We fine-tuned Video-GPT on downstream tasks, it demonstrates that our model acquired excel- lent prior knowledge and modal representations through large-scale self-supervised pretraining with videos. Unless specified, we perform experiments based on the Stage 3 pretraining model in Tab. D
Class-to-Vides Generation. As shown in Tab. Video-GPT achieves the state-of-the-art per- formance on UCF-101 at high resolution. without the better 3D VAE. In addition, we test the Video-GPT trained from scratch on UCF-101, and the results show that our pretraining plays very important role in the downstream task of tasss-oo-vido.. Qualitative results are presented in Fig.
Sexttto-Video Generation. We employ the same text encoder as SDv2 Rombach et al. (2021) We filter Open Vid-IM based on motion and aesthetic scores Lin et al. (2024) and obtain 0.3M pairs for training, which relatively small dataset. As shown Fig. demonstrates that our Video-GPT can achieve effective cross- modal generation with only limited amount of text-video pairs.
Image Animation. Fig. we fine-tune Video-GPT 2K steps on three training sets with no more than 100 videos each, it shows amazing generalization ability on test cases outside the training set.
Video Classificatton. As shown in Tab. 7 Video-GPT linear probe on UCF-101 surpasses Video- MABv2 et al. 2023a), pretrained model commonly used in the field of video understanding.
Video Retrieval, In Tab. g Video-GPT achieves impressive zero-shot performance on the MSR- VTT et al. (2016) and surpasses VideoClip Xu et al. (2021) which was trained from scratch.
Video Object Segmentation. We finetune Video-GPT 5K steps on GetIn-IM IK subset. Fig. 9 it alse present good generalization performance when transferred to low level understanding task.
5 CONCLUSION
We introduce Video-GPT, concise yet powerful large video foundation model that unites autore- gressive modeling with diffusion through next clip diffusion. Treating each video clip as a word token allows the model to inherit the self- suff-supervieed property of GPT while retaining the sharp syn- thesis quality of diffusion. Pretraining on 70M unlabeled videos yields sfatc-of-the-art accuracy on deterministic Physics-IQ Benchmark and uncertain Kinetics-600 forecasting, and fine-tuning trans- fers seamlessly to six challenging and diverse downstream tasks including generation and under- standing. Future work will explore multi-modal pretraining, reinforcement--drivenn world interaction.
10

---

ACKNOWLEDGMENTS
This work was supported by Guangdong Science and Technology 2024TQ08X365).
Program (Grant No.
ETHICS STATEMENT
Our Video-GPT is a concise yet powerful large video foundation model that unites autore- gressive modeling with diffusion through next clip diffusion. To ensure ethical compliance, our training data is carefully selected from public sources and take deliberate measures to minimize potential biases, fully aligning with universal ethical guidelines. We explicitly emphasize that this framework not intended for misuse in achieving harmful purposes; downstream users are encouraged to adhere to ethical principles when applying the technology. Additionally, all authors declare no conflicts of interest related to this work.
REPEATABILITY STATEMENT
To ensure full reproducibility, we will publicly release all code and data necessary to replicate our experiments. Comprehensive implementation details, including model architecture, hyperparame- ters, and training methodology, are provided in this paper and its appendix. We are committed to open-soureing all essential resources to ensure that our findings can be fully verified and built upon by the research community.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Has- san Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Singh Behl, Alon Ben- haim, Misha Bilenko. Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio C' Csearr Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allison Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Jun- heng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Young Jin Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Liliang Ren, Corby Rosset, Sam- budha Roy, Olli Saarikivi. Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xianmin Song, Olatunji Ruwase, Praneetha Vaddamanu, Xin Wang. Rachel Ward, Guanhua Wang. Philipp Andre Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang. Ziyi Yang, Donghan Yu, Cheng-Yuan Yuan Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yunan Zhang, Xiren Zhou, and Yifan Yang. Phi-3 technical report: A highly capable language model locally your phone, ArXiv, 14212404.14219, 2024.
OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji. Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bermadeet-Shapiree Christopher Berner, Lenny Bog- donoff, Oleg Boiko, Made laine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson. Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings. Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim on Posada Fish- man, Juston Forte, Is abella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Goginemi, Gabriel Goh, Raphael Gontijo-Lppes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy. Jesse Han. Jeff Har- ris, Yuchen He, Mike Heaton, Jo hannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey,
11

---

Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shan- tanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitschei- der, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lec, Jan Leike, Jade Leung, Daniel Levy, Chak Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Mar- tin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, An- drey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M Mly Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan. Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O' Keeff, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr. John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on on Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss. Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. 2023.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L. Yuille, Trevor Darrell, Jiten- dra Malik, and Alexei A. Efros. Sequential modeling enables scalable learning for large vision models. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22861-22877 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1708-1718, 2021.
Omer Bar-Tal Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, Oliver Wang, Deqing Sun, Tali Dekel, and Inbar Mosseri. Lumiere: A space-time diffusion model for video generation. ACM SIGGRAPH Conference and Exhibition Computer Graphics and Interactive Techniques in Asia, 2024.
James Betker, Gabriel Goh, Li Jing, TimBrooks, Jianfeng Wang, Linjie Li, LongOuyang, Jun- tangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDhariwal, CaseyChu, YunxinJiao, and Aditya Ramesh. Improving image generation with better captions. URL https://app. semanticscholass wtw.://oo.....oeoooooooooootooo.
A. Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, and Dominik Lorenz. Stable video diffusion: Scaling latent video diffusion models large datasets. ArXiv, abs/2311.15111 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin,
12

---

Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.14165, 2020.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal M. P. Behbahani, Stephanie Chan, Nicolas Manfred Otto Heess, Lucy Gonzalez, Simon Osindero. Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zoina, Jeff Clune, Nando de Fre- itas, Satinder Singh, and Tim Rocktaschel. Genie: Generative interactive environments. ArXiv, 1oss22022.13911, 2024.
João Carreira, Eric Noland, Andras Hanki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600.. ArXiv, abs/1808 0121180.001. (000.00344 01340, 2018.
Boyuan Chen, Diego Marti Monso, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion, ArXiv, abs/2407.01392. 2024a.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang. James T. Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. ArXiv, abs/2310.00426, 2023a.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming-Hsuai Hsuan Yang. and S. Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13320-13331, 2024b.
Eisin-Ying
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. ArXiv, abs/2310.20700) 2023b.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation complex datasets. arXiv: Computer Vision and Pattern Recognition, 2019.
Google DeepMind. Gemini 2.5 flash, 2025. URL
Chaorui Deng, Deyao Zhu, Kunchang Li, Shi Guang, and Haoqi Fan. Causal diffusion transformers for generative modeling. ArXiv, htbb///412.1299..... 2024.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14668 2025.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, abs/2105.05233, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur' elien Rodriguez, Austen Gregerson, Ava Spataru, Bap tiste Roz- jère, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cris tian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan. Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab A. AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Geor- gia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju-Qing Jia, Kalyan Vasuden
13

---

Alwala, K. Upasani, Kate Plawiak. Keqian Li, Ken-591 neth Heafield, Kevin R. Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Rantala- Yeary. Lau- rens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pa- supuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Has- san, Naman Goyal, Narjes Torabi, Niko lay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji, Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong. Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Ro main Sauvestre, Ron nie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sa hana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov. Shao- liang Nie. Sharan Narang, Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhos- ale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Vir ginie Do Vish Vogeti, Vladan Petrovic, Weiwei Chu. Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yiqian Wen, Yiwen Song, Yuchen Zhang. Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aaditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Ben- jamin Leonhardi, Yo- (Bernie) Huang, Beth Loyd, Beto de Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichten- hofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang- Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm ann Frank J. Kanayet, Frank Seide, Gabriela Medina Florez, Gabriclla Schwarz, Gada Badeer, Geor- gia Swee, Gil Halpern. Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan. Hamid Shojanazeri, Han Zou, Hannah Wang. Han Zha, Haroun Habeeb, Har- rison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina- Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Japn- Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard. Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kaixing(Kai) Wu, U KamHou, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian I.,, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko. Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini San- thanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Fgebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong. Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pe dro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollár, Polina Zvyagina, Prashant Ratanchan- dani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Ro-
14

---

han Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon. Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sung-Bac Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang. Wes Bouaziz, Will Con- stable, Xia Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi Hc, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilama 3 herd of models. ArXiv, 0bs/2400.211883 2024.
Patrick Esser, Sumith Kulal, A. Blattmann, Rahim Entezari, Jonas Muller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Laccy, Alex Goodwin, Yannik Marck, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. ArXiv, abs/2403.03206, 2024.
songwei Ge, Thomas Hayes, Harry Yang. Xiaoyue Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. In European Conference Computer Vision, 2022.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training ima- genet in hour. ArXiv, abs/1706.02677, 2017.
Yuchao Gu, Weijia Mao, and Mike Zheng Shou. Long-context autoregressive video modeling with next-frame prediction. ArXiv, abs/2503.19322. 2025.
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran. Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion. arXiv preprint wxXi::2501.00/00 2024.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-augmenteed dense predictive coding for video representation learning. In European Conference on Computer Vision, 2020.
Jonathan Ho. Classifier-free diffusion guidance. ArXiv, abs/2207.1259g 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusson probabilistic models. In Conference and Workshop Neural Information Processing Systems, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. ArXiv, 0bb//2004003455, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. ArXiv, abs/2205. 1la/220551568,, 2022.
Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei- Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive conditional modeling and diffusion transformer. ArXiv, abs/2412.077200, 2024.
Zhihao Hu and Dong Xu. Videocontrolnett: A motion-guided video-to-video translation framework by using diffusion model with controlnet. ArXiv, abs/2307. 14073, 2023.
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. ArXiv, abs/2506.08009, 2025a. htt.htss::
7/api. www.nteatttt..m www.aaaiisscchoaarroooo/ooom htwpaatttsss..tt.a...ooooopaaaaa
Zhipeng Huang, Shaobin Zhuang, Canmiao Fu, Binxin Yang, Ying Zhang, Chong Sun, Zhizheng Zhang, Yali Wang. Chen Li, and Zheng-Jun Zha. Wegen: A unified model for interactive multi- modal generation chat. ArXiv, abs/2503.01 115, 2025h.
15

---

Emmanuel Kahembwe and Subramanian Ramamoorthy. Lower dimensional kernels for video dis- criminators. Neural networks: the official journal of the International Neural Network Society, 132:506-520, 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361 2020.
Taehoon Kim, Gwangmo Song, Sihaeng Lee, Sangyun Kim, Yewon Seo, Soonyoung Lee, Se- ung Hwan Kim, Honglak Lee, and Kyunghoon Bae. L-verse: Bidirectional generation be- tween image and text. 2022 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. (6505-16515, 2021.
D. Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, Yong Cheng. Sing-Chang Chiu, Josh Dillon, Irfan Essa, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David C. Minnen, David Ross, Grant Schindler, Mikhail Sirotenko, Kihyuk Sohn, Krishna So- mandepalli, Huisheng Wang, Jimmy Yan, Ming Yang. Xuan Yang. Bryan Seybold, and Lu Jiang. Videopoet: A large language model for zero-shot video generation. ArXiv, 2000/2312.14125. 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jia-Liang Xiong, Xin Li, Bo Wu, Jianwei Zhang, Kathrina Wu, Qin Lin, Junkun Yuan, Yanxin Long, Aladdin Wang, An- dong Wang, Changlin Li, Duojun Huang, Fan Yang, Hao Tan, Hongmei Wang, Jacob Song, Jiawang Bai, Jianbing Wu, Jinbao Xue, Joey Wang, Kai Wang, Mengyang Liu, Peng-Yu Li, Shuai Li, Weiyan Wang. Wenqing Yu, Xi Deng, Yang Li, Yi Chen, Yutao Cui, Yuanbo Peng, Zhen Yu, Zhiyu He, Zhiyong Xu, Zixiang Zhou, Xavier Xu, Yang-Dan Tao, Qinglin Lu, Song- tao Liu, Daquan Zhou, Hongfa Wang, Yong Yang, Di Wang, Yuhong Liu, Jie Jiang, and Cae- sar Zhong. Hunyuanvideo: A systematic framework for large video generative models. ArXiv, abs/2412.036033, 2024.
Kuaishou, Kling ai, 2024. URL htt.httts///kkingee www.gggwi...iii....ooo
Black Forest Labs. Flux, 2024. URL htps:://blacfforttslaass
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, et al. Flux. kontext: Flow matching for in-context image generation and editing in latent space. arXiv preprint arXiv:2506. 15742, 2025.
Chengxuan Li. Di Huang, Zeyu Lu, Yang Xiao, Qingqi Pei, and Lei Bai. A survey on long video generation: Challenges, methods, and prospects. ArXiv, abs/2403.164077 2024.
Chao Liao, Liyang Liu, Xun Wang. Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation. ArXiv, abs/2505.05477 2025.
Bin Lin, Yunyang Ge, Xinhua Cheng. Zongjian Li, Bin Zhu, Shaodong Wang, Xianyi He, Yang Ye, Shenghai Yuan, Liuhan Chen, Tanghui Jia, Junwu Zhang, Zhenyu Tang. Yatian Pang, Bin She, Cen Yan, Zhiheng Hu, Xiao wen Dong, Lin Chen, Zhang Pan, Xing Zhou, Shaoling Dong, Yonghong Tian, and Li Yuan. Open-sora plan: Open-source large video generation model. ArXiv, abs/2412.00131,, 2024.
Shanchuan Lin, Ceyuan Yang, Hao He, Jianwen Jiang, Yuxi Ren, Xin Xia, Yang Zhao, Xuefeng Xiao, and Lu Jiang. Autoregressive adversarial post-training for real-time interactive video gen- eration. ArXiv, 00352 2025. URL https: /tl:://app /api semant cscholar o CorpusID:2793054
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. ArXiv. abs/2210.02747, 2022.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. ArXiv, abs/2402.082688 2024.
16

---

Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017.
Zhengxiong Luo, Dayou Chen, Yingya Zhang. Yan Huang, Liangsheng Wang, Yujun Shen, Deli Zhao, Jinren Zhou, and Tien-Ping Tan. Videofusion: Decomposed diffusion models for high- quality video generation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 10209-10218, 2023.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. ArXiv, abs/2401. 1.3048, 2024.
Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag. Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, and S. Tulyakov. Snap video: Scaled spatiotemporal transformers for text-to-video synthesis. 2024 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7038-7048, 2024.
Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles? ArXiv, 0bs/2501.09003. 2025.
Kepan Nan, Rui Xie, Penghao Zhou, Tichan Fan, Zhenheng Yang, Zhijic Chen, Xiang Li, Jian Yang, and Ying Tai. Openvid-Ing Im: A large-scale high-quality dataset for texteoo-video generation. ArXiv, abs/2407.02371, 2024.
OpenAl. Video generation models as world simulators, 2024. URL iindx//vidoo-geeeraiiin
openaaeena.... .com/
Yichen Ouyang, Jianhao Yuan, Hao Zhao, Gaoang Wang, and Bo Zhao. Flexifilm: Long video generation with flexible conditions. ArXiv, abs/2404.1 18620, 2024.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam M. Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, 2018.
Mandela Patrick, Po-Yao (Bernie) Huang, Yuki M. Asano, Florian Metze, Alexander Hauptmann, João F. Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. ArXiv, abs/2010.02824, 2020.
William S. Peebles and Saining Xie. Scalable diffusion models with transformers. 2023 IEEE/CVF
International Conference on Computer Vision (ICCV), pp. 4172-4182, 2022.
PikaLabs. Pika 1.5, 2024.
Dustin Podell, Zion English, Kyle Lacey, A. Blattmann, Tim Dockhorn, Jonas Muller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthe- sis. ArXiv, 0bs/2307.001522 2023.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang. Ki ran Ja- gadeesh, Kumpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sum- baly, Sai Saketh Rambhatla, Sam S. Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin. Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen- Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kinstain. Zecheng He, Zijian He, Albert Pumarola, Ali K. Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Car- leigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffmann Jonas Kohler, Kaolin Fire, Karthik Sivaku- mar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du. Movie gen: cast of media foundation models. ArXiv, abs/2410.13720.. 2024.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018.
17

---

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- lional Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. ArXiv, abs/2204.0611 06122, 2022.
Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. 2025.
Videoworld: Exploring knowledge learning from unlabeled videos. ArXiv,
Robin Rombach, Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10000 10685,, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. ArXiv, abs/1505.04553 2015.
Runway. Gen-3, 2024. URL htt.hhtttt///rwwaaetee com/
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. ArXiv, abs/2205. 2022.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128:2586 2606, 2018
Eyal Segalis, Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. A picture is worth thousand words: Principled recaptioning improves image generation. ArXiv, abs/2310.16656, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xiaoyue Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a- video: Text-to-video generation without text-video data. ArXiv, abs/2209. 14792, 2022.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Machine Learning, 2021.
Khurram Soomro, Amir Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. ArXiv, abs/1212.0400 2012.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng. Ping Luo, and Zehuan Yuan. Au- toregressive model beats diffusion: Llama for scalable image generation. ArXiv, abs/2406.06525, 2024.
Chameleon Team. abs/2405.09818 18, 2024.
Chameleon:
Mixed-modal carly-fusion foundation models.
ArXiv,
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric challenges. ArXiv, abs/1811.01777, 2018.
Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. Jianyuan Zeng, Jiayu Wang, Jingfeng Zhang, Jingren Zhou, Jinkai Wang. Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Xiaofeng Meng, Ningying Zhang, Pan- deng Li, Pingyu Wu, Ruihang Chu, Rui Feng. Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang, Wei Wang. Wen-Chao Zhou,
18

---

Wente Wang, Wen Shen, Wenyuan Yu, Xianzhong Shi, Xiaomin Huang, Xin Xu, Yan Kou, Yan- Mei Lv, Yifei Li, Yijing Liu, Yiming Wang, Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi. Yutong Feng, Zeyinzi Jiang. Zhengbin Han, Zhigang Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. ArXiv, abs/2503.20314, 2025.
Hanyu Wang. Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. Larp: Tokenizing videos with a learned autoregressive generative prior. ArXiv, abs/2410.21264, 2024a.
Junke Wang, Yi Jiang, Zehuan Yuan, Binyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnitokenizer: A joint imagevideo tokenizer for visual generation. ArXiv, abs/2406.09999, 2024b.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang. and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14549-14560, 2023a.
Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang. Mingwu Zheng, Xin Tao, Fei Yang, Pengfei Wan, and Di Zhang. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. ArXiv, abs/2410.08260, 2024c.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu. Yingli Zhao. Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Lian zi Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need. ArXiv, abs/2409. 18869, 2024d.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang. Yinan He, Jiashuo Yu, Pe der Yang. Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Y. Qiao, and Ziwei Liu. Lavie: High-quality video generation with cascaded latent diffusion models. ArXiv, abs/2309.15100 2023b.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Jian Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Y. Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. ArXiv, 0as/2307.06642. 2023c.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. ArXiv, abs/2409. 11340, 2024.
Hu Xu, Gargi Ghosh, Pa-Yao (Bernie) Huang, Dmytro Okhonko, Armen Aghajanyan, and Flo- rian Metze Luke Zettlemoyer Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In Conference on Empirical Methods Natural Language Processing, 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5288-5296, 2016
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In North American Chapter of the Association for Computational Linguistics, 2020.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin. Kai Dang. Keming Lu, Ke- Yang Chen, Kexin Yang. Mei Li, Min Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yunyang Wan, Yunfei Chu, Zeyu Cui, Zhenru Zhang, and Zhi- Wei Fan. Qwen2 technical report. ArXiv, abs/2407. 10671, 2024a.
19

---

Sherry Yang, Jacob Walker, Jack Parker-Hop Holder, Yilun Du, Jake Bruce, Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making, ArXiv, abs/2402.17139, 2024b.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Xiaotao Gu, Yuxuan Zhang, Weihan Wang, Yean Cheng, Ting Liu, Bin Xu, Yuxiao Dong, and Jie Tang. Cogvideox: Text-to-video diffusion models with am expert transformer. ArXiv, abs/2408.06077 2024c.
Hongwei Yi, Shitong Shao, Tian Ye, Jiantong Zhao, Qingyu Yin, Michael Lingelbach, Li Yuan, Yonghong Tian, Enze Xie, and Daquan Zhou. Magic 1-for-1: Generating one minute video clips within one minute. ArXiv, abs/2502.07701, 2025.
Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Frédo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast causal video generators. ArXiv, abs/2412.07772, 2024.
Lijun Yu, José Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuu Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In International Conference Leurning Representations, 2024.
Zhengrong Yue, Shaobin Zhuang, Kunchang Li, Yanbo Ding, and Yali Wang. V-stylist: Video stylization via collaboration and reflection of mllm agents. ArXiv, abs/2503. 12077, 2025.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma. Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. ArXiv, abs/2408. 1039, 2024.
Shaobin Zhuang, Kunchang Ii, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang- Vlogger: Make your dream vlog. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8806-8817, 2024.
Shaobin Zhuang, Zhipeng Huang, Binxin Yang. Ying Zhang, Fangyikang Wang, Canmiao Fu, Chong Sun, Zheng-Jun Zha, Chen Li, and Yali Wang. Get in video: Add anything you want to the video. ArXiv, abs/2503. 062668 2025.
20

---

Algorithm Video-GPT Pre-training
1: Input:
2: V: The set of training videos.
3: Mg: Video-GPT model with parameters 0.
4: VAEg: The pre-trained VAE Encoder.
5: T: Total number of training steps.
6: Main Training Loop:
7:
for step = + do
8: 1. Input Sequence Construction (Sec. 3.1
9:
f- sample_video( V)
> Sample a video from the training set
all frames +- sample n frames(1)
D Sample n frames from the video > K is randomly sampled
11:
clips + partition into k cllps(al frames)
12:
Simput <-
Initialize full input sequence for the model Initialize ground truth clean latents for loss
13:
Strget f []
14:
for
- (ongth(clips) do
15:
Cs t clips ke
16:
42 < VAE,;
b Encode clean clip to latent space 4
// (Optionaa) Add slight noise for training robustnesx (Sec. Kk.noisy add slighttnoise((f] ($x))
4.3
Eq.
// Create a noisy version via forward diffusion (flow matching) t sample_ from uniform(O,
18:
b Saaple single noise level for the clip Sample from a standard Gaussian > Noisy latent features
19:
Sk ~ N(, I) with shape of 4k
20:
1R + ag + (1 og)en. Ex
1/ Format clips into token sequences NS; t- format noisy sequence Ue ag) CL. formaa_cleannsequeeeeeeee sequnnceeeeeeeeeeeeeeeeee k,nis)
21:
b c.g. [< diff >; Olky MA...... D Deg. [< img >, 1x,1, /mgg >,...
22:
23:
Append NS; and CL₁ Sinput Append 4 to Surget
24:
> Target for loss is the original clean latent
25:
end for
// 2. Pre-training with Next Clip Diffusion (Sec. 3.2 // Hierarchical atlention mask is applied internally by Mg predicted_
MMMMannnn (Smpu))))
D. Model forward pass
28:
3/3. Loss Calculation and Optimization
// L. loss is calculated only outputs corresponding noisy inputs C 12, loss(predicted $ Sunga) C.backward()
29:
30:
31:
optimizer.stepO
32:
optimizer..eero_rradd)
33: end for
A PSEUDOCODE OF VIDEO-GPT
A. TRAINING PSEUDOCODE
As shown in Algorithm
21

---

Algorithm 2 Video-GPT Autoregressive Inference
1: Input:
2: Mg: Pre-trained Video-GPT model.
3: VAËg, VAED: VAE Encoder and Decoder.
4: C1: Initial conditioning video clip (the first clip).
5: N: Total number of clips to generate.
6: c: Classifier-Pae Guidance scale (Sec.
7: T: Total number of denoising steps.
8: 1. Initialization Generate GenerateVideo(C,, Vi
9: DNS1 & VAEg(C1) Generated Latents [DNS]]
P Encode the initial clip to a Denoised Sequence.
> Initialize list of generated latent clips.
2. Autoregressive Generation Loop (Sec. 3.3
12: for k = to N - do
13:
N + sample comple_from__ee www.o/aausiianaaaaaaaaaaaa (aussian((happ((NSS))
Prepare noisy input for clip kt + 1.
14:
3. T steps denoising
15:
for == 1 to T do Contert t- Generated_ Latents
16:
Clean latents as context.
17:
// 4. Denoise using History Conditioned Classifier- Free Guidance
imputccon create input input_sequeneee sequence(Conteaz, N Smex)
19:
predcond & Me (imputccndd)
Conditional prediction (with history).
imprst create sapp___npp____e Supputsequence(( (0, N Smere)
21:
preduncoms t Mo (iippuuttuncc
Unconditional prediction (null context). Predictions with CFG.
22:
cf Jeond f preduncond + c. (predlondd - pr preduncona)
23: N Snext <- Flow_ @loo__acchin((( ycond
(cconidd 1)
b Update noisy latent via flow matching.
24:
end for
25:
APPEND(Generated Latents, Suxx)
Append the newly generated clean latent clip.
26: end for
27: 5. Decode the final video
28: Final_ Video_ Clips +
29: for each latent_clij _lip in __ipp)__ppppppp
do (laeelatent_
30:
"video_cle
31: APPEND(Finen LaniFFinal__eeeee Video Video_Clips, video_clip
end for
33: Final_Video
Video <- (Finaannat(Fnnaa__ieded
return Final Pinal_Video
A.2 INFERENCE PSEUDOCODE
As shown in Algorithm
A.3 UNIFORM SAMPLING PSEUDOCODE
As shown in Algorithm
B MORE IMPLEMENTATION DETAILS
As shown in Tab.
15
16
and
18
C MORE VISUALIZATION RESULTS
The full mask of the input in Fig. 2 (a) is shown in Fig.
10
22

---

Table 10: Pretraining Stage setting.
Table 9: Pretraining Stage I setting.
coofiig
coedie
Pasde 2014
optimazer
Adam Lostschiloy & Hutter 2017
optinseer
Adank Loitchoor & Honerr 2013
optiminer monentumm
Pi:75 (1172--1000 (19,0000
optietizr momectun
Weight decay
0,1 contisiinnt Ie-4
weight
scheshake
censine Ie-4
learning rale
loaning rite
learning rate
wormup deps et al 2017
1000
warmup steps neeoGooyal et 2019
1000
sotal sxps
300000
total steps
25000
lorgest sids
mouiston
fexilie
320
input frame
resulution
flexible 320
48
frune sieval
longest
sideo dip
Frame interval
compasing
video clip
Uniform (2, 3 48) $20
computing unit
Table 11: Pretraining Stage 3 setting.
Table 12: Pretraining Stage setting.
coafig
Punda- 70M
config
Panda: 70M
optimizer
Adam Losbchilov & Hutter 2017
optinuzer
Adlam Lodichilov & Hutter
optimizer momentum
P1:71 11.712-0031 1.50
optimizer momentuum
weight decay
weight decay
0.1 comsistent
learning schedule
consintent 1o-4
learning rate schodole
learning nie
learning rate
2o-5
warnnup Repo/Goyal al. 2015
1000
warmup StepeGGoyal at 2017
1000
total steps
40000 48 flexible
Intal steps
20000
input Thame
input
80
resulution
resulution
flexible
longest side
longest
330
inrerval
Uniform ~ Uniform 22,3 483 320
Frame interval video clip computing unit
Unifoomm4 (4, 5. Uniform 22,3 500
video clip
computing unit
330
Table 13: Class to video fine-tuning on UCF- 101 setting.
Table 14: Text to video fine-tuning on Ope-Vidd0.3MM setting.
config
config
OpenVid-O.3M
optimizer
AdamW
Lostchilov & Homter 2017
optimizer
Adam W
Lostcchlley & Hluttr 2017
optimizer momentum
optimioor momentum
weight decay
0.1
decay
0.1 contistent le-4
learning schedule
learning rate
learning mie
learning role
warmup steps
Goyal G
al. 2017
1000
warmup StepeGryyll et 2017
1000
total steps
165000
total steps
70000
input frame
16
frame
24
resulution
240x 320
resulution
Hexiible 320
longest side
longest
interval
frame interval video clip computing unit
4
vidoo clip
1
computing unit
64
128
Table 15: Image animation fine-tuning on an- imation dataset setting.
Table 16: Video classification fine-tuning on UCF-101 setting.
confi
Anim. Datnser
confii
UCF-
uptimizer
Addmm Losichilov & Hotter 2017
optinuzer
Adamm Losbchilov Hutter 2017
aptimizer momentum
optimizer momentum
P1. U.S. TA
weight decay
comsisent 0.1
weight decay
0.1
learning schedule
learning schedule
consistent le-4
learning
learning rate
warmup steps Goyal al. 2017
warmup SlepsGoyaa al. 2017
1000
lotal steps
2000 48
total steps
10000
input frame
input frame
16
resulution
flexible 320 flxxible
resulution
x24x224
longest side
longest
234
interval
frame inserval
video clip
video clip computing unit
computing unit
Table 17: Video retrieval fine-tuning on Panda-70M.
Table 18: Video object segmenration fine- tuning GetIn-1K setting.
config
Punda-700 70M
sonfiig
Gelle-1K
optimzzr
W Lostchilov Hutter 2017
optinuzer
Adamm Losbchilov & Hutter 2017
optimizer momentum
31,32-0.9, 1:00
optimizer momentum
Pizls (.d DAD
weight decay
0.1 consistent 1e-4
weight decay
a.)
learning rate schedale
learning rale schedule
consistent 1e-4
learning
learning
warmup stepe
2017
warmop stepe Gooal et 2017
1000
total steps
200000
total steps
5000
input frame
input frame
4%
resulution
221
resulution
flexible
longest side
224
longest
224
frame inserval
flexible
Hextble
video
video clip
computing
computing unit
8
23

---

Figure 10: The full attention mask of Fig. 2 (a) training input.
24

---

Algorithm 3 GenerateRandomlLis
1: Input:
2: num wum_fromes: Total number of frames (integer, 2 1).
3: Output:
4: parts: list of positive integers that sum num__rrames
5: // Initialization
6: parts <-1
Initialize the output list. Track the last cut position (start at O).
7: previous t-
8: //2. Decide the number of segments internally (no extra imput)
9: k: random integer(1, num sum__famme) sampling logic.
b Number of parts; chosen uniformly to mimic
m k-
Number of cut points sample; yields no cuts.
33. Sample and sort cut positions from the valid range 12: cuts sorted(random_ haaarrrnddodnaaslllaeeeee (ange(1, num fram_e)) m))
{l...., TL....,Tum_franee (anple(range(1
Distinct cuts in
- 1];; sorted ascending.
13: // 44 Build parts by consecutive differences
14: for each cut in cuts do APPEND(parts, cul - previous) previous cul.
15:
D Length of the segment between previous and cul. Advance the cursor to the current cut.
16:
17: end for
APPEND(parts, num_frames previous)
Append the final segment from the last cut to
num num_frames
19: return parts
We show the results of Video-GPT for zero-shot long video prediction in Fig.
12
In order to demonstrate the capabilities of Video-GPT more comprehensively, we test Video-GPT's prediction ability for videos with partially static clips (minimal motion) and partially with abrupt motion. As can be seen from Fig. [3] Video-GPT is able to handle such extreme video cases. Although there is still much room for improvement, we believe that Video-GPT has indeed learned highly generalizable world knowledge from massive amounts of video data.
As can be seen from Fig. 14 we employ Video-GPT prediction to generate 1-minute long video. Experiments have shown that Video-GPT can generate videos up to 1-minute long while maintaining temporal coherence and content consistency. As shown in Fig. 15, we generate 1-minute videos using Open-Sora-Plan Lin et al. (2024). From the comparison between Fig. [4]and 15, we can find that the final images of the video generated by Open-Sora-Plan has been significantly collapsed, while the image generated by Video-GPT is still relatively clear. This comparison again verifies the strong temporal robustness of our Video-GPT.
D LIMITATIONS
Our model exhibits remarkable performance and generalization across diverse tasks, affirming the strength of the next clip diffusion paradigm. It important to note, however, that our experiments are conducted with a moderately scaled model due to available computational resources. This choice, while pragmatic, has not detracted from the quality of our good results. Looking forward, further scaling may reveal additional performance gains. Nonetheless, the current configuration already sets a high benchmark, underscoring both the efficacy and robustness of our approach in video generative modeling.
25

---

0-5s
5-10s
10-15s
0-5s
5-10s
10-15s
0-5s
5-10s
10-155
0-5s
5-10s
10-15s
Figure 11: The long video zero-shot prediction result generated by Video-GPT. The frames in blue box are condition frames. Frames in green box are the generated prediction result.
26

---

0-5s
5-10s
10-15s
0-5s
5-10s
10-15s
Figure 12: The long video zero-shot prediction result generated by Video-GPT. The frames in blue box are condition frames. Frames in green box are the generated prediction result.
27

---

Figure 13: The partly stationary clips (minimal motion) and partly abrupt motion video zero- shot prediction result generated by Video-GPT.
To test the model's ability to handle extreme cases with partly stationary clips (minimal motion) and partly abrupt motion, we copied an image times to simulate static video, and then edited the image using FLUX-KAS Kontext Labs Labs al. (2025). The edited image then used as the 12th frame of the video to simulate a abrupt, large motion, and Video-GPP (stage4) predicted the next 48 frames. The frames in the orange boxes given stationary video frames, the red box is abrupt motion frame, and the green boxes are video frames generated by Video-GPT prediction.
28

---

0-6s
6-12s
12-18s
18-24s
24-30s
30-36s
36-42s
42-48s
48-54s
54-60s
Figure 14: 1-minute video generated by Video-GPT.
The frame the red boxes conditional video frame, the green boxes are video frames generated by Video-GPT prediction.
29

---

0-668
6-122
12-18k
18-244
24-30s
30-36s
36.428
42-48
48-54s
54.60s
Figure 15: 1-minute video generated by Open-Sora-Plan Lin et al. 2024).
The frame in the red box is conditional video frame, the green boxes are video Trames predicted by Open-Sora-Plam
30

---

E DECLARATION OF USE OF LARGE LANGUAGE MODELS (LLM)
We affirm that this paper was primarily written by the authors. Large Language Models (LLMs) were utilized solely general-purpose assistive tools for language refinement, grammar correction. and stylistic improvements during the writing process. Specifically, Gemini 2.5 Flash (DeepMind) 2025 was employed for minor text polishing and rephrasing enhance clarity and readability. No LLM was used for conceptual ideation, experimental design, data analysis, or generating any substantive content of the research.
31


Reviews:
[{"review_id": "EBBnYjrMIt", "reviewer": "Reviewer_sVcH", "summary": "This Manuscript introduces `video-gpt`, a generative self-supervised solution to represent videos. The main idea is to train video embeddings in a GPT style, where each clip is acting as a token. The training idea is to have interleaved noisy and clean clips, and the training objective is to de-noise the noisy clips.", "questions": "1- Authors propose `Clips as tokens` but later they propose frame-level and patch-level masking. It reads to me as `partial-token` masking. IMO, it would be better not to name Clips as tokens.\n\n2- What is the intuition behind having interleaved noisy and clean clips during training? Why not going with a classic next frame prediction formulation and have `k clean clips` and diffuse the `k+1th noisy clip`? What is the advantage of interleaved modeling? \n\n3- Why `from scratch` model performs poorly in `Tab 6`?\n\n4- In Section 3.3, are all previous K clips (some of them being generated diffused clips) being used to predict the k+1? or there is a limit on K to keep the context window capped?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "State of the results on multiple tasks is the main strength of this paper.\n\nThe idea of interleaved clip level noise/de-noise, although intuitive, is novel IMO.\n\nVideo level self-supervision has been overlooked, IMO. Research like this can bring more attention and opens the road for future works in video domain.", "weaknesses": "Some design choices for training is not trivial to me and I need more clarification. (will ask in question section).\n\nI believe such a method works only on single camera and continious (or single scene) videos. If there is a POV change in a video, like Movies or TV shows, I believe that it will break the whole network.\n\nMotion is not modeled very well in this work. I am curious to know how this model can predict videos where there is partly stationary clips (minimal motion) and partly abrupt motion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8qGmDsDzw6", "reviewer": "Reviewer_vBbq", "summary": "This paper proposes Video-GPT, a concise large video foundation model that unifies autoregressive modeling and diffusion via a novel \"next clip diffusion\" paradigm. Inspired by GPT’s next token prediction, the model treats video clips as \"visual words\" to model spatial-temporal details in the visual world—addressing the limitation of language sequences in capturing such details. The key design involves constructing interleaved sequences of noisy and clean clips, using hierarchical attention masking to leverage historical clean clips for denoising future noisy clips.", "questions": "1. Could you clarify the core differences between Video-GPT’s \"next clip diffusion\" and prior hybrid diffusion-autoregressive models? Specifically, how does your clip-level autoregressive design and hierarchical masking outperform their frame-level or pixel-level combinations?\n2. The paper compares the performance of Video GPT with other models in video prediction, achieving SOTA in Physics-IQ. However, it doesn't examine its performance in other aspects that need to be considered in video generation, such as motion quality and subject consistency.\n3. How does Video-GPT perform on longer videos (e.g., 1 minute or more) in terms of temporal coherence and content consistency? Have you tested it against long-video generation models (e.g., Flexifilm, Open-Sora-Plan) and observed any performance degradation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The \"next clip diffusion\" paradigm is a creative combination of autoregressive modeling (from GPT) and diffusion (for high-quality generation). Treating clips as visual words and using historical clean clips as context for denoising is a novel adaptation of language modeling to video, filling the gap between discrete text tokens and continuous video data. This hybrid design effectively unifies short-term generation and long-term prediction.\n2. As a unified video foundation model, Video-GPT bridges video generation and understanding tasks, advancing the goal of visual world modeling. Its strong performance on physics-aware prediction (Physics-IQ) indicates progress in learning world knowledge from video.\n3. Its generalization to six downstream tasks highlights its potential as a backbone for diverse video applications.", "weaknesses": "1. Insufficient comparison with hybrid baselines: The paper mentions prior works that combine diffusion and autoregressive modeling but lacks a detailed comparison of their core differences. \n2. There is a lack of comparisons with some newer autoregressive + diffusion video generation models, such as self-forcing, apt2.\n3. Limited analysis of architectural choices: The model inherits Phi-3-mini’s architecture and SDXL’s VAE without justifying these choices. There is no comparison with other architectures (e.g., DiT, U-Net) or VAEs (e.g., 3D VAE vs. 2D VAE) to show whether these selections are critical to performance. Additionally, the progressive training strategy’s effectiveness is only validated via frame count ablation, without analyzing how frame interval or clip number affects convergence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "reKZMFXqzj", "reviewer": "Reviewer_8PYT", "summary": "This paper proposes Video-GPT, a foundation model for video pre-training based on a \"next clip diffusion\" paradigm. The core idea is to treat video clips as analogous to words in a sentence. The model is trained to denoise a future \"noisy\" clip conditioned on a history of preceding \"clean\" clips, effectively combining an autoregressive structure at the clip level with a diffusion process for content generation within each clip. The authors demonstrate the model's effectiveness on video prediction benchmarks and show its generalization capabilities by fine-tuning it on six diverse downstream video generation and understanding tasks.", "questions": "The central question is regarding the evaluation strategy. Why did the authors choose to focus on the Physics-IQ benchmark and omit direct, qualitative side-by-side comparisons with state-of-the-art open-domain video generation models that are the current de facto standard for assessing generation quality? Without this comparison, the claims of SOTA performance feel narrow and potentially misleading.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The primary strength of this work is the impressive engineering effort demonstrated in building and evaluating a complete system. The model achieves a state-of-the-art score on the Physics-IQ benchmark, suggesting its pre-training paradigm is effective at capturing physical dynamics and motion continuity. Furthermore, the extensive fine-tuning across a wide array of both generation and understanding tasks showcases the versatility and potential of the resulting pretrained model.", "weaknesses": "Despite the strong results on specific benchmarks, this paper has significant weaknesses that undermine its contribution as a top-tier research publication. Firstly, the technical novelty is limited; the \"next clip diffusion\" idea is a combination of existing autoregressive and diffusion frameworks rather than a fundamental new technique. Secondly, and more critically, the evaluation feels dated and deliberately avoids direct comparison with the true state-of-the-art in video generation quality. The paper heavily relies on the Physics-IQ benchmark while making no qualitative or quantitative comparisons against contemporary leading models known for their visual fidelity. This positions the work more as a technical report for an existing system than a paper pushing the research frontier, especially given the rapid progress in the field over the past year.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3nhiwPTpG", "reviewer": "Reviewer_wpuq", "summary": "This paper introduces Video-GPT, a novel foundation model for video generation and understanding, built upon an elegant analogy of treating video as a new language. The core contribution is a \"next clip diffusion\" pretraining paradigm, which ingeniously combines autoregressive modeling and diffusion. By treating video clips as \"visual words,\" the model autoregressively predicts the next clip by denoising a noisy version, conditioned on the history of previously generated clean clips. This self-supervised approach allows for effective pretraining on large-scale unlabeled video data. The pretrained Video-GPT achieves state-of-the-art performance on the Physics-IQ benchmark, demonstrating a strong capacity for world modeling, and shows excellent generalization across six diverse downstream video generation and understanding tasks.", "questions": "1. The inference process is autoregressive, where the model's own generated (denoised) clips are used as the clean history for subsequent steps. Have you investigated the model's robustness to error accumulation? For instance, does a minor artifact in a generated clip DNS(k)$ degrade the quality of all future clips?\n2. You mention that the model is trained to predict the clean clip directly (x-prediction) rather than the noise (ϵ-prediction) or velocity (v-prediction) to keep the training simple. This is a departure from many modern diffusion frameworks. Could you elaborate on this design choice? Did you experiment with other prediction targets, and did x-prediction yield superior performance?\n3. The progressive training strategy in Table 1, which starts with short clips (effectively next-frame prediction) and gradually increases clip length, is interesting. Could you provide more intuition on why this curriculum is effective? Does learning fine-grained temporal dynamics first provide a better foundation for the model before it tackles longer-range dependencies?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed \"next clip diffusion\" paradigm is a novel and insightful method for unifying autoregressive and diffusion models for video. The concept of conditioning the denoising of a future clip on the clean history of past clips is a clever and distinct approach that effectively leverages the strengths of both modeling families for long-term video prediction.\n2. The paper presents exceptionally strong empirical results. Achieving a state-of-the-art score of 34.97 on the Physics-IQ benchmark, significantly outperforming prior work, is a standout achievement that validates the model's ability to learn physical dynamics. Furthermore, the model's strong performance across a wide array of 6 downstream tasks (including generation and understanding) underscores its quality and versatility as a powerful video foundation model. The ablation studies are thorough and convincingly support the main design choices.", "weaknesses": "1. The proposed input formulation, an interleaved sequence of noisy and clean clips [NS(1), CL(1), ..., NS(K), CL(K)], effectively doubles the sequence length processed by the transformer compared to methods that only use historical context. Given the quadratic complexity of attention, this could be a significant computational bottleneck, potentially limiting scalability. A brief analysis of the computational trade-offs would be beneficial.\n2. The paper states that frames are divided into K clips, K∼Uniform{2,3,...,N}. This process is central to the method, but its details are sparse. It is unclear if clips are contiguous blocks or formed differently. The impact of this random clip partitioning strategy on training stability and performance is not ablated, yet it seems like a critical hyperparameter.\n3. While the paper provides a good overview of related work, the discussion on how \"next clip diffusion\" specifically differs from other recent hybrid autoregressive-diffusion models for video (e.g., VideoPoet, SEINE) could be more detailed. A deeper comparative analysis would help to better contextualize the novelty of the proposed conditioning and generation scheme", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)