VideoMAR: Autoregressive Video Generation with Continuous Tokens
Authors: Hu Yu
Biao Gong
Hangjie Yuan
DanDan Zheng
Weilong Chai
Jingdong Chen
Kecheng Zheng
Feng Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Video generation
Autoregressive model
Continuous tokens
Primary area: deep_learning

Abstract: Masked-based autoregressive models have demonstrated promising image generation capability in continuous space. However, their potential for video generation remains under-explored. 
Masked-based autoregressive models have demonstrated promising image generation capability in continuous space. However, their potential for video generation remains under-explored. 
In this paper, we propose \textbf{VideoMAR}, a concise and efficient decoder-only autoregressive image-to-video model with continuous tokens, composing temporal frame-by-frame and spatial masked generation.
We first identify temporal causality and spatial bi-directionality as the first principle of video AR models, and propose the next-frame diffusion loss for the integration of mask and video generation.  
Besides, the huge cost and difficulty of long sequence autoregressive modeling is a basic but crucial issue. To this end, we propose the temporal short-to-long curriculum learning and spatial progressive resolution training, and employ progressive temperature strategy at inference time to mitigate the accumulation error. 
Furthermore, VideoMAR replicates several unique capacities of language models to video generation. 
It inherently bears high efficiency due to simultaneous temporal-wise KV cache and spatial-wise parallel generation, and presents the capacity of spatial and temporal extrapolation via 3D rotary embeddings.
      On the VBench-I2V benchmark, VideoMAR surpasses the previous state-of-the-art (Cosmos I2V) while requiring significantly fewer parameters ($9.3\%$), training data ($0.5\%$), and GPU resources ($0.2\%$).

VideoMAR: Autoregressive Video Generation with
Continuous Tokens
Hu Yu'
Biao Gong?* Hangjie Yuan? DanDan Zheng² Weilong Chai²
Jingdong Chen
Kecheng Zheng² Feng Zhao' !
MoE Key Lab of BIPC, University of Science and Technology of China Independent researcher
https: /ttp//gttttb..tt/ com/ inclusionAt/Ming Haaa///in-ViieeMAR VideoMAR
Reference Image
Figure 1: Autoregressive image-to-video samples of VideoMAR with continuous tokens.
Abstract
Mask-based autoregressive models have demonstrated promising image generation capability in continuous space. However, their potential for video generation remains under-explored. In this paper, we propose VideoMAR, a concise and efficient decoder-only autoregressive image-to-video model with continuous tokens, composing temporal frame-by-frame and spatial masked generation. We first identify temporal causality and spatial bi-directionality video AR models, and propose the next-frame diffusion loss for the integration of mask and video generation. Besides, the huge cost and difficulty of long sequence autoregressive modeling a basic but crucial issue. To this end, we propose the temporal short-to-long curriculum learning and spatial progressive resolution training, and employ progressive temperature strategy at inference time mitigate the accumulation unique capacities of language models to video generation. It inherently bears high efficiency due simultancuus temporal-wiss KV cache and spatial-win parallel generation, and presents the capacity of spatial and temporal extrapolation via
the first principle of
Furthermore, VideoMAR replicates several
"Project lead "Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

3D rotary embeddings. On the yBench-12V benchmark, VideoMAR surpasses the previous state-of-the-art (Cosmos 12V) while requiring significandly fewer parameters (9.3%), training data (0.5%), and GPU resources (0.2%).
Introduction
have unified and dominated language tasks with promising intelligence in generality and versatility, demonstrating promising path toward artificial general intelligence (AGI). Recently, MAR series methods [19 [0] 351
Building upon autoregressive (AR) models, large language models (LLMs) P4] 22
have demonstrated great success of conducting autoregressive image generation in continuous space. However, its potential for autoregressive video generation remains under-explored. Compared to image, video data is temporally sequential, making it more suitable for autoregressive modeling.
naive way video autoregressive modeling directly adapts the paradigm of language models [22 which factorizes frames into discrete tokens and applies next-token prediction (denoted as NTP) in raster-scan order [1 22 I. However, this paradigm for video generation suffers from several limitations: 1) Discrete tokens deviate from the inherent continuous distribution of video data and inreparaby induce significant information loss. 2) The unidirectional modeling of visual tokens deviate from the inter-dependency nature of tokens within identical frame, and may be suboptimal in performance 19 10 3) NTP demands substantial inference steps for video generation.
Compared to NTP, mask-based autoregressive generation is a more promising direction [9]. However, it is nontrivial incorporate mask mechanism into autoregressive video generation. A desired way is to sequentially generate each frame depending on all the previous context frames. While, this poses challenges for the introduction of mask. Prevailing methods B 36) apply mask to each frame, but introduces rraining-inferene gap. NOVA 8 proposes to decompose the temporal and spatial generation via generating the coarse features frame-by-frame and refines each frame with a spatial layer, but complicates the framework and weakens the temporal smoothness. MAGI [39] mitigates this issue by appending a complete copy of video sequence during training, but doubles the sequence length and training cost. Therefore, mask-based video autoregressive generation, lying promising but challenging paradigm, still requires further exploration.
In this paper, we propose VideoMAR, decoder-only autoregressive video generation model with continuous tokens, integrating temporal frame-by-frame and spatial masked generation. To meet the requirement of sequentially generating each frame depending all the previous context frames, VideoMAR preserves the complete context and introduces a next-frame diffusion loss during training. Besides, the extremely long token sequences of video data poses significant challenges in both efficiency and difficulty. To this end, we propose tailored strategies for training and inference. During training, propose the short-to-long curriculum learning to reduce the training difficulty and cost, and establish the two-stage video generation. During inference, long token sequence generation is prone suffer from severe accumulation error in late frames, due to exposure bias issue 38 We identify that temperature plays crucial role eliminate this error and propose the progressive temperature strategy.
pnngressive-resoluttni
training to support higher resolution
Furthermore, VideoMAR replicates several unique capacities of language models video genera- tion, e.g. key-value cache and extrapolation, demonstrating the potential for multi-modal unification. For example, thanks to our design, VideoMAR inherently bears high efficiency due to simultaneous wese KV cache and spatial-wise parallel generation. VideoMAR, for the first time, also unlocks the capacity of simultancous spatial and temporal extrapolation for video generation via incorporating the 3D-RoPE. On the VBench-12V benchmark, VidcoMAR achieves better performance compared to the Cosmos bascline, with much smaller model size, data scale, and GPU resources.
Related Work
2.1 Autoregressive Video Generation
Raster-scan autoregressive models. Similar to the VQ quantization and NTP paradigm in au- toregressive image generation models 9
24
28 21, some methods also employ this paradigm
D.. For example, VideoPoet [17 employs a decoder-only transformer architecture to processes multi-modal inputs, incorporating mixture of
for autoregressive video generation 117 29 25

---

multi-modal generative objectives. Cosmos trains the autoregressive-based world foundation model via video generation.
Mask-based autoregressive models. Mask-based autoregressive models predict the masked tokens given the unmasked ones. They introduce bidirectional transformer and predict randomly masked tokens by attending to unmasked conditions 3 36 I 8 39]. This paradigm enhances vanilla AR by predicting multiple tokens at every step. For example, MAGVIT [36] tackles various video synthesis tasks with a single model, via randomly masking the video sequence. Genie [3] proposes interactive video game generation in an unsupervised manner and generates videos frame-by-framee
Inspired by the continuous tokens in MAR [19], some recent works also propose to combine continuous tokens and masked generation models for video generation [8 39]. For example. NOVA [8] generates the
coarse features frame-by by-fraame and refines each frame with a spatial layer. MAGI (39] proposes
complete teacher forcing by conditioning masked frames on complete observation frames.
2.2 Diffusion-based Long Video Generation
Recently, some diffusion based video generation models attempt to extend the inference video length and achieve autoregressive- video generation. These methods can be mainly divided into two types. The first type [31 12 34 basically follows video diffusion models to repetitively generate video clips, and sequentially cascade these video chunks to achieve longer video generation. For example, Caus Vid 34 transforms bidirectional models into fast autoregressive ones through distillation. The
second type S 2 [4 applies varying levels of noise to different video chunks, imitating the causality of autoregressive generation. For example, DiffusionForcing [5] applies higher noise level to the later frame in each chunk and shifts across frames for the generation of more frames. MAGI-1 [2] follows this paradigm by applying varying noise levels to different chunks while maintaining the noise level identical for frames in each chunk. These methods still fall in the range of diffusion models, employing diffusion model for the generation of each frame or video chunk. Therefore, these methods are basically different from the autoregressive video generation, which employs AR models for the token- wise generation.
3 Preliminary
Task definition and symbology setting. This paper focuses on the image-to-video autoregressive video generation, which sequentially generates the next frame forming the complete video with the given initial image and text prompt. For notation elarity, we depict the symbology settings in Table
Autoregressive video generative model. The limitations of NTP paradigm is clearly presented in the introduction part. In this section, we cast on the mask-based gen- eration paradigm. Mask-based generation common paradigm for autoregressive image generation 19], which randomly masks partial image tokens and predicts these masked tokens with remaining visible tokens. Compared to NTP, this paradigm has the ad- vantage of parallel token generation each step, significantly reducing the inference steps. When extended to autoregressive video generation, most existing methods [3] [6] treat the video sequence similarly as image, treating all tokens within each video frame equally and using bidirectional attention for the whole video.
Table 1: Symbology settings.
T H
Frames x height x width in latent space. Number of video tokens n-th token. Number of video frames / t-th frame. All visible masked tokens in frame Text prompt condition.
N/
T/S,
ST/ST"
p (C,, St. ST) = ST ..... ST C, ST
(3.1)
However, such operation bears several limitations: 1) These methods face the dilemma of either failing to inference framame (36), or facing rraiiing-inferecce inference bias when performing temporally sequential inference B]. For example, the previous context frames are partially masked during training. While, the frame-by-frame
generation requires all the tokens in previous frames are available for the next frame prediction. 2) It depends on fixed-length video frames, which can lead to poor scalability in context and issues with coherence over longer video durations. It sacrifices the unique context extension potential of the token-wisg wise modeling in AR models, which is verified possible and important
3

---

visible
masked
predicted
Next-frame diffusion loss
Stage 1
Stage 2
$2
S4
Si
Temporal
Spatial progressiv-rres
short-to-long
$2
S3
VideoMAR
(Frame-wase wase Intru frame bidirectional
S
S2
Sq
Continuous Video VAE
S1
S2
S3
S4
Attention mask
1: -
256x:25-
(a) Framework Overview
(b) Efficient Training
Figure 2: Framework of VideoMAR. (a): Training flowchart of VideoMAR. We employ frame-wise causal attention mask for temporal causality. Besides, we introduce the next-frame diffusion loss the spatial partial-masked frame, which has complete previous frames.
(b): Efficient training of
VideoMAR. We apply temporal short-to-long curriculum learning and spatial progressive-resoluiio for reducing the training difficulty and cost of VideoMAR.
language models 27 3) It is incompatible with the inference optimization methods, such as KV cache. Recently, some methods attempt to optimize the mask-based generation 39]. While, they either separate the temporal and spatial modeling, or double the sequence length. In this paper, starting from the desired way of sequentially generating each frame depending all the previous context frames, we propose VideoMAR, a simple and novel paradigm for mask-based autoregressive video generation. VideoMAR is free from all these three limitations.
4 VideoMAR
4.1
Framework Overview
Video autoregressive model generates each frame conditioning on all the previous frames. Mask-based generation models generate the masked tokens with the remaining visible ones. Therefore, mask- based video AR model should generate the masked tokens each frame with all the tokens in the preceding frames and the visible tokens in the current frame. To this end, we propose VideoMAR, the decoder-only mask-based autoregressive video generation model. As shown in Figure VideoMAR works with continuous tokens via first compressing the video into continuous tokens with video VAE. Then, VideoMAR adopts frame-wise causal attention mask, enabling temporal causal and spatial bidirectional modeling.
Next-frame loss. To fit the temporal causal inference and mask generation properties of VideoMAR, we devise the next-frame diffusion loss as depicted Figure 2] Specifically, we randomly mask partial tokens in certain frame [1, For the preceding frames before frame i, all the tokens are remain unchanged. For the frames after frame i, all the tokens are masked. This processed video sequence maintains temporal causality with frame-wise causal attention mask. The token-wise diffusion loss optimization only applied to the masked tokens in frame I. The tokens after frame are free from loss optimization not all tokens in the previous frames available. During training. the frame t is randomly chosen between [1, T), spanning all the frames.
p SC..S Sr) = I p (ST .....
(4.1)
4.2 Training
Temporal short-to-long curriculum learning. Due to the long sequence nature of video data, would be quite difficult and inefficient to directly model this long sequence. Different from the joint

---

bidirectional video frames modeling of video diffusion models, video autoregressive model generates the next frame based on previous frames. This temporally sequential property highlights the priority of early frames for both training and inference.
To this end, propose the temporal short-to-long curriculum training. Specifically, we first train on the short video clips with much shorter sequence length. In this stage, VideoMAR processes the capacity of clear visual quality and basic temporal motion modeling, with substantially reduced training cost and difficulty. When the early frames in the short clips are converged, we extend the video frame length to progressively capture larger temporal motion modeling capacity. This progressive short-to-long curriculum learning strategy successfully decompose the long sequence modeling difficulty to various phases in a very efficient way.
Spatial two-stage progressiv--resoltion training. In the first stage, VideoMAR trains on low resolution of 256 256. During this stage, adopt the above mentioned short-to-long curriculum learning. With lower resolution and curriculum learning, VideoMAR possesses basic autoregressive video generation capacity with high efficiency. In the second stage, we finetune the model with higher resolution of 480 x 768 to support higher resolution video generation. To deal with the significantly increased token sequence, adopt VAE with higher compression ratio. Thanks the employed relative positional encoding, this finetuning stage is empirically verified efficient.
4.3 Inference
Accumulation error. AR models inherently suffer from exposure bias problem each token is predicted with preceding GT tokens during training. However, during inference, all preceding tokens are the corresponding predicted ones, which may be incorrect. Such exposure bias especially obvious for video generation which has long sequence length. To solve this, we first explore the importance and eгтor of each generated frame. We find that the carly-frame is vital for motion degree and suffer from relatively small accumulated error [8. while the late-frame focuses more on keeping visual quality and motion smoothness and suffers from more accumulated error, as shown in Figure M. Furthermore, we uncover that temperature plays key role error suppression, where low temperature reduces the accumulated error (Detailed analyses and ablation of temperature on generation result is available in the supplementary material). With this observation, we propose the progressive temperature strategy, applying smaller temperature to the later frames. Specifically, the temperature varies from 1 to 0.9 across frames. The lower temperature in the later frame effectively suppresses the accumulated error and thus achieves much better visual quality. Besides, the slighter lower temperature is enough to keep motion smoothness given the previous dynamic video frames.
38 Specifically,
Efficient inference. VidcoMAR generates video tokens via mask- parallelized generation, while maintaining the frame-by-frame generation. This paradigm combines the advantage of the spatial token parallel generation and temporal KV cache acceleration. As shown in Table 2 we present the inference steps comparison with NTP paradigm, where VideoMAR significantly reduces the steps 20x times (from 1440 to 64). Based on the spatial parallel generation, VideoMAR further reduces the inference time via enabling the KV cache (from 672s to 134s). Compared to NTP, our method achieves more than 10x acceleration (from 1941s to 134s).
Table 2: Efficient inference of VideOMAR. VideoMAR combines the advantage of the spatial token parallel generation and temporal KV cache acceleration. Note that the NTP" listed only for inference speed comparison, and we did not train such model. Methods
Steps (spatial x temporal)
Inference time (S)
NTP
1024x6
1310
VideoMAR(stagee)
w/o KV Cache
64x6
417
w/ KV Cache
64x6
107
NTP
1440x6
1941
VideoMAR(stage2)
w/o KV Cache
64x6
671
w/ KV Cache
64x6
134
Spatial and temporal extrapolation ability. Context token length extrapolation is a basic capacity of L.M,, which can generate millions of tokens significantly longer than the training sequence length. Position encoding (PE) method plays key role in this capacity. In this paper, we experiment with different PE methods, including the absolute cosine PE and RoPE (27 (extrapolation ability comparison between different PE methods are available in the supplementary material), In the
5

---

final implementation, we apply 3D- 3D-RoPE as the only position encoding method, and for the first time simultaneously boost the spatial and temporal extrapolation ability of video autoregressive model. VideoMAR can generate videos of varying resolutions and length, while only trained on fixed resolution and aspect ratio. Visual examples are available in Figure We also depict more arbitrary scaling examples in Sec. D. where our method can flexibly generate videos of varying aspect ratios.
Experiments
5.1 Implementation Details
Experimental setup. We employ the general decoder-only architecture as the backbone of Video- MAR. The VideoMAR backbone consists of 36 transformer layers with a dimension of 1536. We mostly follow MAR [19] for the implementation of token-wise diffusion loss. The denoising MLP consists of 3 blocks with dimension of 1280. We adopt the masking and diffusion schedulers from MAR [99 using a masking ratio between 0.7 and 1.0 during training, and progressively reducing it from 1.0 0 following a cosine schedule with 64 autoregressive steps during inference. In line with common practice [3]. we train with a 1000-step noise schedule but default to 100 steps for inference. For the text prompt, following the practice in FAR [5), we employ Qwen2-1.55 (32 as our text encoder and adopt cross attention for text condition injection. For the visual tokenizer, we adopt Josmos-Tokeniln
Tokenizer with 4 X 8 X 8 compression in the temporal and spatial dimensions. The temporal short-to-long, curriculum learning is arranged with frame length order of (5, 13, 25). For the second stage (480 x 768 resolution), we employ
[ For the first stage (256 256 resolution), we employ
with 8 x 16 x 16 compression. We utilize the AdamW optimizer [20] (31 = 0.9, 82 = 0.95) with weight decay of 0.02 and a base learning rate -4 in all experiments. All the weights are trained from scratch with 64 NVIDIA H20 GPUs.
Datasets. For image-to-video training, we employ 0.5M internal video-text pairs.
Evaluation. We use VBench-12V [16] to evaluate the capacity of image-to-video generation across all the 9 dimensions. For given text prompt, we randomly generate 5 samples, each with video size of 25 x 256 x 256 for the first stage and 49 x 480 x 768 for the second stage. We employ classifier-free guidance with value of 3.0 to enhance the quality of the generated videos in all evaluation experiments. Each latent frame is generated with 64 autoregressive steps
5.2 Quantitative Comparison
VideoMAR is comparable with diffusion image-to-video models and significantly suppresses the AR counterpart with much lower training costs. We perform quantitative comparison with mainstream and cutting-edge image-to-video models, which can be divided into two types: diffusion models and autoregressive models. For diffusion model type, the compared models include 12VGen-XI. (37, Consist12V [26). SEINE (7), V2de [6), Cog VideoX-122 (33, Step- Video-TI2V 15 and Magi-l 1 12]. For the autoregressive type. Cosmos [I] is the most powerful autoregressive image-to-vine video model. As shown in Table despite its significantly smaller size (1.4B vs. 13B) data scale (0.5M vs. 100M), training cost (64 H20 GPUs vs. 10000 H100 GPUs), VideoMAR remarkably outperforms Cosmos (84.51 a variety of sub-dimensions. VideoMAR also rivals some diffusion-basee including ConsistI2V and VideoCrafter-ID V1V with much lower training costs. For the latest SOTA diffusion models, like Seep-Vieo-TTn TI2V and Magi-1, our method still lags behind. While, our method focuses the more challenging and promising AR-paradigm video generation and it's already quite convincing and promising of VideoMAR to achieve such performance with quite limited resources.
84.22) on the VBench-12V benchmark across
image-to-videe models
5.3 Qualitative Results
High motion smoothness and visual quality. We present the qualitative comparison in Figure 3 VideoMAR demonstrates high visual quality in each frame and smooth motions across adjacent frames. The motion type of VideoMAR contains both object motion, camera motion, and stable scene transitions. contrast, Cosmos suffers from poor quality and details due to the employment of discrete tokens. Besides, the motions in Cosmos mostly camera motion, with few object motions.

---

Table 3: Image-to-video evaluation on VBench. We have classified existing video generation
methods into different categories for better clarity. The baseline data is sourced from VBench-12V
16.. The data of Cosmos tested with official code and recommended parameters.
Model
params
data
Score
Tora
Score
12V
Score
Qual.
Subj.
12V
Back.
12V
Came. Subj. Back.
Moti. Cons.
Smoo. Degr.
Mon.
Dyna.
Qual.
Aest.
Imag.
Qual.
Diffasion
Magi-1
24B
89.28
96.12
82.44
98.39
99.00
50.85
93.96
96.74
98.68
68.21
64.74
69.71
Step- Video-T12V
30B
5M
88.36
95.50
81.22
97.86
98.63
49.23
96.02
97.06
99.24
48.78
62.29
70.44
Cog V12
86.70
97.19
94.34
98.40
33.17
70.01
SEINE
10M+
85.52
92.67
78.37
97.15
96.94
20.97
95.28
97.12
97.12
27.07
64.55
71.39
12VGen-XI.
35M
85.28
92.11
78.44
96.48
96.83
18.48
94.18
97.09
98.34
26.10 10
64.82
69.14
Consist12V
10M
84.07
91.91
76.22
95.82
95.95
33.92
95.27
98.28
97.38
18.62
59.00
66.92
VideoCrafter-12V
10M
82.57
86.31
78.84
91.17
91.31
33.60
97.86
98.79
98.00
22.60
60.78
71.68
Antorggessive models
Cosmos
5B
100M
84.16
92.51
75.81
95.99
97.36
25.56
97.12
96.59
99.47
20.33
55.82
59.90
Cosmos
13B
100M
84,22
92.60
75.83
96.17
97.35
25.43
97.69
96.77
99.40
18.70
55.84
60.15
VideoMAR staeel
1.4B
0.2M
82.56
91.74
73.39
96.64
96.24
16.80
97.08
98.78
99.32
13.01
52.05
51.61
VideoMAR-stage
Refermnce Image
Cosmos
VideoMAR
Figure 3: Visual comparison between Cosmos 5B and VideoMAR on image-to-video generation. The motions in Cosmos are mostly camera motion, with few object motions. Cosmos is also prone to have failure cases, which induces abrupt object change and poor consistency. For example, the surfing man disappears in the third row, and the texture oranges changes in the fifth row.
Spatial and temporal extrapolation. Context token extrapolation capacity desired property for autoregressive visual generation. For spatial dimension extrapolation, few current methods show such ability. For temporal extrapolation, some methods claim longer video generation ability, via repeatedly generating video chunks. In this paper, VideoMAR demonstrates the capacity of simultancous spatial and temporal extrapolation to generate video of larger resolution and duration in This is achieved in training ffree manner without chunk- split.
Figure 4
Extreme long video generation. Besides the boosted resolution scaling ability, we further verify the extremely long video generation capacity of VideoMAR. As shown in Figure we autoregressively generate videos of 20 second (12 fps), instead of repeatedly generating multiple video clips. While the maximum training length is 4 second. For the generation of such long video, we discuss three kv cache methods. (1) Full kv-cache. We cache all the history frames for the generation of next frame. This strategy cache increasingly more features, inducing slower inference speed and larger memory consumption for the generation latter frames. (2) Window kv-cache. This strategy maintains constant length cache and is temporally smooth due to temporal locality. While, the
7

---

(1) Original x 450 768)
extrapolation
(2) Spenial (49 1024)
(3) Temporal (73 480 768)
Spanial Tempural 1024)
(4)
Figure Spatial and temporal extrapolation capacity of VideoMAR.
short memory also induces poor long-range consistency. (3) First frame & window kv-cache. The cached first frame provides the anchored spatial information and the window frames provide temporal smoothness. We empirically recommend the third strategy for its superior performance.
(2)
(3)
Iaput
generating longer (out training scope)
Figure 5: Extreme long video generation capacity of VideoMAR. VideoMAR can, for the first time, autoregressively generate longer vidcos far beyond the training length, instead of repeatedly generating multiple video clips. It supports x3 longer generation without severe degradation, and x5 longer generation with reasonable motions. Different rows represent various kv cache methods.
Video-to-video generation. Besides autoregressive image-to-video generation, VideoMAR also unlocks video-to-video generation. Given videos of various frame length, VideoMAR can generate more frames based on the given video. As shown in Figure 6 we present the visual results with two frames as condition. Video condition contains not only spatial prior but also temporal motion clues, which enables VideoMAR have desired motion type and dynamic degree.
5.4 Ablations
In this section, we verify the effectiveness of each design of VideOMAR, including 1) the causal attention mask; 2) the next-frame diffusion loss; and 3) the temperature strategy. As shown in Table we measure their effectiveness with the VBench-12V metric. Obviously, each design is beneficial to the performance. Besides, the additional advantage inference acceleration is analyzed in Table
Table 4: Ablation study of VideoMAR in stage I on VBench-I2V.
Frame I Loss
Causal Attn
Temperature
Total Score
I2V Score
Quality Score
X
78.81
86.57
71.05
X
X
79.76
87.66
71.86
X
80.72
88.90
72.54
V
82.56
91.74
73.39

---

Reference Images
Figure 6: Visual results of VideoMAR on video-to-videe generation, with two frames condition.
Imagres
Total mask
VideoMAR (temporal autoregressive modeling)
Figure 7: Visual comparison between the total mask baseline and our temporal autoregressive modeling VideoMAR in stage one. The baseline has poor local consistency with the reference image.
Effectiveness of temporal autoregressive modeling. To highlight the advantages of temporal autoregressive modeling, we denote baseline as Total mask (w/o Frame Loss, w/o Causal Attn, w/ Temperature strategy), which treats the video sequence similarly as images, shown in Equation 3.1 As shown in Figure 7 temporal autoregressive modeling obviously has higher visual quality and smoother motion than the baseline. The fully random masked generation of the baseline method is prone to have poor consistency with the reference image in local regions, c.g., the motorcycle the second row is broken, and the human face in the last row is distorted.
Temperature strategy & Accumulation error.
To validate the effectiveness temperature strategy for reducing accumulation error, we present the visual comparison in Figure 8 Without the tempera- ture strategy, the late frames of the generated video tend to collapse due to accumulated error the previous frames. For example, the buildings suffer from severe distortion and lose its structural shape. In contrast, VideoMAR adopts low temperature in the late frames, which maintains the dynamic degree across frames and high visual quality of each frame.

---

Figure 8: The effects of the temperature strategy on visual results. The late frames prone suffer from large accumulated errors and poor quality without the incorporation of the temperature strategy.
Discussion and Future Work
Despite the superior results achieved, admit that there still exist some limitations that are worth further exploration: 1) VideoMAR has the capacity of multiple tasks unification within this single paradigm, including text-to-image, text-to-video, image-to-video, video-to- iido--t-viddoo, and vidco editing. However, due to limited resources, we first explore image-to-video and video-to-video tasks, and leave other tasks in future work. 2) VideoMAR naturally function interactive world model via replacing the prompt with frame-level interactive action condition. We will verify this in future work.
7 Conclusion
In this paper, we propose VideoMAR, a concise and efficient decoder-only autoregressive image-to- video generation model with continuous tokens, composing temporal frame-by- by-fame and spatial masked generation. VideoMAR retains complete previous context frames and introduces the frame- wise next-frame loss for training. Besides, we solve the huge cost and difficulty of long sequence autoregressive modeling with several crucial designs both training and inference. On the VBench- 12V benchmark, VideoMAR achieves cutting-edge performance.
8 Acknowledgment
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC. The Al-driven experiments, simulations and model training performed the robotic Al-Scientist platform of Chinese Academy of Sciences.
10

---

References
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint 2XXi::5501.03775 2025.
[2] Sand AL. Magi-1: 1: Autoregressive video generation at scale. 2025.
[3] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
[4] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[5] Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion foreing: Next-token prediction meets full full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081-24111 2024.
[6] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310-7320, 2024.
[7] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. In The Twelfth International Conference on Learning Representations, 2023.
[8] Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui. Huchuan Lu, Shiguang Shan, Yonggang Qi, and Xinlong Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412. 14169, 2024.
[9] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[10] Lijie Fan, Tianhong Li. Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint 2w..xx::410.33300 2024.
[11] Michael Fuest, Vincent Tao Hu, and Björ Ommer. Maskflow: Discrete flows for flexible and efficient long video generation. arXiv preprint arXiv:2502.1125:: 2025.
[12] Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tade- vosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2403.14773 2024.
[13] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
[14] Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei- Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive conditional modeling and diffusion transformer. In ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling.
[15] Haoyang Huang, Guoqing Ma, Nan Duan, Xing Chen, Changyi Wan, Ranchen Ming, Tianyu Wang, Bo Wang, Zhiying Lu, Aojie Li, al. Step-video-ti2v technical report: state-of-the-art text-driven image-to-video generation model. arXiv preprint arXiv:2503.1125:. 2025.
11

---

[16] Ziqi Huang, Yinan He, Jiashuo Yu. Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 21807-21818, 2024.
[17] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung. Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXi::2312.14125: 2023.
[18] Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on Robot Learning, pages 143-156. PMLR, 2017.
[19] Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424-56445, 2024.
[20] Ilya Loshchilov, Frank Hutter, al. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 5:5, 2017.
[21] Xiaoxiao Ma, Mohan Zhou, Tao Liang, Yalong Bai, Tiejun Zhao, Huaian Chen, and Yi Jin. Star: Scale-wise text-to-image generation via auto-regressive representations. arXiv preprint arXiv:2406. 10797, 2024.
[22] OpenAl. Chatgpt, 2022. https///opeaaa.cc//bbooo/aaaatte
[23] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1677, 2020.
[24] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821 8821-8831. Pmlr, 2021.
[25] Shuhuai Ren, Shuming Ma, Xu Sun, and Furu Wei. Next block prediction: Video generation via semi-auttoreerressive modeling, arXiv preprint arXiv:2502.07737, 2025.
[26] Weiming Ren, Huan Yang. Ge Zhang, Cong Wei, Xinrun Du, Wenhao Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04322 2024.
[27] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, Neurocomputingg 127033, 2024.
[28] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 2rXiv:2406.06525, 2024.
[29] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint 4xXiv:2409.18869, 2024.
[30] Yuqing Wang, Tianeei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.0275: 2024.
[31] Wenming Weng, Ruoyu Feng, Yanhui Wang, Qi Dai, Chunyu Wang, Dacheng Yin, Zhiyuan Zhao, Kai Qiu, Jiammin Bao, Yuhui Yuan, et al. Art-v: Auto-regressive text-to-video generation with diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7395-7405, 2024.
[32] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint 10777 2024.
12

---

[33] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong. Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprini arXiv:2408.0660 2024.
[34] Tianwei Yin, Qiang Zhang, Richard Zhang, William T Freeman, Fredo Durand, Eli Shechtman, and Xun Huang. From slow bidirectional to fast autoregressive video diffusion models. arXiv preprint 21X1::2412.07772. 2, 2024.
[35] Hu Yu, Hao Luo, Hangjie Yuan, Yu Rong, and Feng Zhao. Frequency autoregressive image generation with continuous tokens. arXiv preprint arXiv:2503.05300 2025.
[36] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459- 10469, 2023.
[37] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang. Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.0412 2023.
[38] Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. arXiv preprint arXiv: /906.0248, 2019.
[39] Deyu Zhou, Quan Sun, Yuang Peng, Kun Yan, Runpei Dong, Duomin Wang, Zheng Ge, Nan Duan, Xiangyu Zhang, Lionel M Ni, et al. Taming teacher forcing for masked autoregressive video generation. arXiv preprint 1rxv:25011.2388. 2025.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims in the abstract and introduction parts accurately reflect this paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Discussion and Future Work Section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We verified the effectiveness of every design of this method in the ablation
part.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the details of our paper in both the method and experiment parts.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer:
[No]
Justification:
We will soon release the code and checkpoints of our method after submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ Caapaae//CoSSSSeeieeeeee
set.aiisssiooiiiooooooooooaammmc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify the experimental setting/details in the beginning of the experimental part.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The statistical values reported in this paper are achieved on the widely ac- knowledged benchmark.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report both the training GPU resources and the inference time of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://wwwr..ccoool/lll//cccccaa
Answer: [Yes]
Justification:
This paper is free from the ethics problem.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consider- ation due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: direction.
We discuss the significance of this paper to the community and its future
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our method is trained from scratch with internal data.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the generated videos of our method in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Position Encoding
In this paper, we mainly verify two types of position encoding methods: 1) absolute cosine position encoding, and 2) relative position encoding.
Absolute cosine position encoding. Absolute positional encoding assigns a unique encoding vector to each position in the sequence, which is usually applied fixed-length sequence. However, it neglects relative relationships and suffer from poor extrapolation of length. Unseen sequence lengths during training usually lead to performance degradation. As shown in upper part of Figure the extrapolated frames fail to maintain the consistency.
Relative position encoding, Relative position encoding (RoPE) is the dominant position encoding method in language models, demonstrating great success sequence length extrapolation. For video data, we adopt the 3D-RoPE, covering part Figure D RoPE endows our method with superior length extrapolation ability, with negligible visual quality degradation.
both the spatial and temporal dimensions. As shown in bottom
(1) Original (25 256 256)
Absolute position encoding
extrpolatioo
e
Temporal (37 256 256)
(3) Spatial (25 x 384 384)
(4) Spatiol Temporal (37 384 384)
Original 256)
extrapolation
Relative position encoding
Temporal (37 x 256 x 256)
Spotial 384
(4) Spatial& Temporal (37 384 384)
Figure 9: Visual comparison between absolute and relative position encoding methods.
B More Ablations on Temperature Strategy
In this section, we present more analyses of the temperature. As demonstrated in the main paper that temperature plays key role accumulation error suppression. In Table we demonstrate the VBench-12V scores under different temperatures. It is obvious that reducing the temperature helps consistently elevates the performance, e.g., the 12V Subject Score, the Subject Consistency Score, the Motion Smoothness Score, the Aesthetic Quality Score, etc.
Note that reducing the temperature may also negatively affect the dynamic degree score according to the table. While, in practical experiments, we find that the dynamic degree change is actually much small across various temperatures. The larger dynamic degrees in the bottom row are mainly attributed to the existence of failure cases, which can have unreasonable generation in late frames due accumulation error.
To this end, as demonstrated in the main paper, this paper adopts the progressive temperature strategy to allocate lower temperature for the later frames. Specifically, adopt the exponential function 0.9 + 10-+10000 10-11-0++ (t++1, with the frame t [0, Note that employing constant and relative low temperature
21

---

(e.g.. 0.9) can achieve similar performance, but with slightly lower Dynamic Degree score. In practical usage, these two setting are both recommended.
Table
Ablations on the temperature strategy.
Total
12V
12V
Subj. Cons.
Back.
Moti.
Dyna.
Aest.
Imag.
Temperature
Score
Subj.
Back.
Cons.
Smoo.
Degr.
Qual.
Qual.
1.00
82.19
95.22
97.71
89.70
95.59
98.95
24.80
50.94
55.18
0.98
83.13
96.06
98.08
91.77
96.14
99.21
23.58
52.38
56.71
0.96
83.56
96.49
98.21
92.92
96.36
99.34
20.73
53.17
58.11
0.94
84.02
96.94
98.26
93.92
96.80
99.43
19.31
53.72
59.00
0.92
84.31
97.41
98.33
95.06
96.94
99.49
16.67
54.34
60.18
0.90
84.82
97.92
98.39
97.17
97.27
99.58
9.87
55.95
62.48
Ours
84.82
97.85
98.38
97.13
97.20
99.57
10.98
55.81
62.34
C
Prompts for Videos in Main Manuscript
Due to page limit, we omit the corresponding prompts for each video in the main manuscript. In this section, we list the prompts of each Figure.
Prompts of Figure in the main manuscript:
tangerines in a metal bowl on table
Close up eye blink
Small fireworks explode and unfold in the air
Tall trees with thick trunks and green foliage dominate the scene, set in misty forest with dirt path winding through the undergrowth. The atmosphere foggy. creating a mystical and serene ambiance. The ground is covered with leaves and small plants, and the light filtering through the trees is soft and diffused. The overall style realistic, with a slight blur due to the mist.
Flower bloom
Prompts of Figure in the main manuscript:
A group of people in yellow raft is rowing through turbulent waters
Tall trees with thick trunks and green foliage dominate the scene, set in misty forest with dirt path winding through the undergrowth. The atmosphere foggy, creating a mystical and serene ambiance. The ground is covered with leaves and small plants, and the light filtering through the trees is soft and diffused. The overall style realistic, with a slight blur due to the mist.
a man on a surfboard riding a wave in the ocean
sailboat is drifting on the ocean
tangerines in a metal bowl on table
Aerial view a coastal landscape featuring large, rugged rock formations jutting out into the ocean. The rocks are light brown with rough texture, surrounded by blue-green waves crashing against them. The coastline sandy with patches of green vegetation. The background includes a hazy sky and distant landforms. The overall style is realistic with clear, detailed imagery.
book fire with flames coming out of it
A large cruise ship with red funnel is docked at port, surrounded by several smaller boats and industrial structures. The port area includes warehouses, parking lots, and various equipment. The scene is set against backdrop of calm blue water. with a long pier extending into the distance. The overall style is realistic, with clear visibility and no noticeable blur. The background includes distant landmasses and clear sky.
22

---

Prompts of Figure 4 in the main manuscript:
Waves gently wash over a sandy beach, creating intricate patterns in the wet sand. The water is clear and slightly foamy as it moves in and out. The horizon is visible in the distance with a clear blue sky above. The beach is empty, and the scene is calm and serene. The overall style is realistic with high clarity and no noticeable blur.
A lush, green mountainous landscape with dense vegetation overlooks a serene body of water. The central mountain has a rugged, steep peak, and the surrounding area features rolling hills and valleys. The sky is partly cloudy with patches of blue, casting soft shadows on the terrain. The scene is bright and clear, with no visible human presence. The overall style is realistic, capturing the natural beauty and tranquility of the environment.
Prompts of Figure in the main manuscript:
A small, red spaceship with transparent dome is floating in vast, mountainous landscape. The spaceship has sleek, rounded design with a glowing blue thruster at its rear. Inside the dome, humanoid figure with a green complexion and round head is seated, facing forward. The figure appears to be wearing a spacesuit, though the details of the suit are not clearly visible.The spaceship is the main object, and it consistently centered in the scene. The background features a series of dark. jagged mountains against a reddish-orange sky, suggesting desolate, alien environment. The landscape barren, with no vegetation or structures visible, emphasizing the isolation of the scene. The spaceship remains at relatively consistent distance from the mountains, maintaining mid-air position above the ground. Throughout the sequence, the spaceship moves steadily from left to right across the screen. The shot a medium shot, capturing the entire spaceship and part of the surrounding environment. The style of the scene is animated, with a somewhat cartoonish and colorful aesthetic, reminiscent of a Pixar-like animation. The humanoid figure inside the spaceship appears calm and focused, with no discernible facial expressions due to the distance and the dome's s transparency.
The video features a close-up up of a green apple with a gradient from green to red, showcasing its textured skin and small yellow spot. As time passes, the apple's 's glossy surface reflects light, emphasizing its freshness and natural beauty. The fruit's stem is brown, and it has a small yellow spot near the stem, which may indicate maturity or natural imperfection. The apple's 's vibrant colors and the subtle shadows beneath it highlight its three-dimensional form and the detailed texture of its skin.
The video begins with a wide shot of a large, ornate building with two prominent towers, situated vast, open landscape. The building is surrounded by dry, barren land with patches of greenery. A road runs parallel to the building, with a few cars visible on it. The sky is clear with a few clouds, and the overall color palette is dominated by earthy tones. As the video progresses, the camera moves closer to the building, providing clearer view of its architectural details. The building appears to made of stone or similar material, and its design suggests it may be of historical or cultural significance. The surrounding landscape is mostly flat, with a few hills in the distance. The video ends with a closer shot of the building, highlighting its grandeur and the serene environment in which it is located.
The video features a series of close-up shots of fresh vegetables on a wooden cutting board, with a focus on cherry tomatoes, carrots, bell peppers, lettuce, and garlic. The vegetables are arranged with coarse salt and rosemary, suggesting preparation for a flavorful dish. The lighting is soft, highlighting the textures and colors of the produce, creating warm, inviting atmosphere. As the frames progress, the arrangement of the vegetables slightly changes, with the addition a yellow bell pepper and a purple onion, and the garlic cloves are shown in various states, from whole to peeled. The consistent theme is the anticipation of cooking with fresh ingredients.
A female doctor, dressed in blue scrubs and a white hijab, stands confidently outdoors, her hands in her pockets. She wears a stethoscope around her neck, indicating her medical profession. The setting a modern healthcare facility with a blurred background, suggest- ing a professional environment. Over time, her expression remains serious and focused, reflecting her dedication to her role. The lighting suggests it could be early morning or late aftermoon. The background includes greenery and a building, hinting at tranquil yet
23

---

professional atmosphere. Her attire and demeanor convey a sense of professionalism and cultural modesty.
Aerial view of tropical resort featuring several overwater bungalows with thatched roofs arranged in a symmetrical pattern extending from a central pier. The bungalows are situated in clear, turquoise water with visible darker patches indicating underwater features. The scene transitions from a closer view of the bungalows to a wider view, revealing a sandy beach lined with lush green vegetation and additional resort structures. The overall style realistic, capturing the serene and picturesque nature of the tropical location.
Prompts of Figure in the main manuscript:
motorcycle driving down road
person riding atv dirt track
pink lotus flower in the middle of pond
young girl smelling a pink flower
Prompts of Figure in the main manuscript:
Waves gently wash over sandy beach, creating intricate patterns in the wet sand. The water is clear and slightly foamy as moves in and out. The horizon is visible in the distance with clear blue sky above. The beach is empty, and the scene is calm and serene. The overall style is realistic with high clarity and no noticeable blur.
A coastal cityscape with numerous tall, modern skyscrapers in light brown and beige hues lines the background, transitioning to sandy beach with turquoise sea in the foreground. The beach is populated with sunbathers and beachgoers, with several colorful umbrellas and small boats visible the water. The scene shifts from view of the buildings perspective of the coastline, revealing more of the beach and the sea. The water is clear and calm, reflecting the sunlight. The overall style is realistic, with high clarity and vibrant colors.
broader
D Arbitrary Scaling
In the main manuscript, we mainly demonstrate the spatial and temporal extrapolation capacity of our method. In this section, we show that our method can perform arbitrary resolution scaling, including temporal longer/shorter, spatial larger/smaller generation and their arbitrary combination. To show such capacity,
present four more resolution (Hl x W x T) examples, including 1) 240 x 384 25 in Figure 10 2) 240 384 x 73 in Figure [] 3) 240 x 768 x 49 in Figure 12, and 4) 480 x 768 37
in Figure 1 Note that the model is trained 480 x 768 49.
E
More Visual Comparisons with the Baseline Method
We present more visual comparisons between the Cosmos baseline and our method on image-to-video generation in Figure and 15
F More Visual Results
We present more visual results of our method image-to-vides generation in Figure 16
17
and 18
We present more visual results of our method on video-to-vides generation in Figure 19 20 and
G Inference Time Comparison with Baseline Method
We compare the inference time between the Cosmos baseline and our method in Table achieves comparable inference time than the Cosmos baseline. While, the efficiency of VideoMAR can be further improved with smaller autoregressive steps and diffusion steps. We currently employ autoregressive steps of 64 and diffusion steps of 100.
6 Our method
24

---

HX W 240033xX222
Figure 10: Arbitrary resolution scaling video samples with H x W X T = 240 x 384 x 25.
Table 6: Inference time comparison between Cosmos and VideoMAR.
Methods
Cosmos-5B
Cosmos-13B
VideoMAR-stagel
VideoMAR-stage2
Time/s
93
207
107
134
H
Analyses of Motion Degree
Our method processes basic motion type in very efficient way with limited resources. While, it still lacks in large and complex motions, due to small data scale and from-scratch training. this section, we further verify that method can generate complex motions, once given such data. Specifically, we collect two motion types, covering 1) blink and 2) blossom, each with several video samples. We finetune our model these collected data respectively. In Figure 22] Our method successfully replicate these complex and large motions. thus a future work to enlarge training data scale with more complex motions to improve the generalization ability of our method.
25

---

I 1
Figure 11: Arbitrary resolution scaling video samples with H x W x T 240 x 384 X 73.
X 763x49
Figure 12: Arbitrary resolution scaling video samples with H x W x T 240 X 768 x 49.
Figure 13: Arbitrary resolution scaling video samples with H W T 480 X 768 37.
26

---

Reference Imager Figure 14: Additional visual comparison between Cosmos and method tidge-to-video generation.
Cosmos
VAR
Reference Imnge:
Cosmos
VideoMAR
Figure 15: Additional visual comparison between Cosmos and method tigdeo generation.
27

---

Reference Image
Figure 16: Additional visual results of our method image-to-video generation.
Reference Image
Figure 17: Additional visual results of our method on image-to- video generation.
28

---

Reference Image
Figure 18: Additional visual results of our method on image-to-vides generation.
Reference Images
Figure 19: Additional visual results of our method on video-to-video generation.
29

---

Reference Images
Figure 20: Additional visual results of our method on video-to-video generation.
Reference Images
Figure 21: Additional visual results of our method on video-to-video generation.
30

---

Reference Image
Figure 22: Finetuning results the two additional collected motion types (blink and blossom) of our method on image-to-vice vidvo generation.
31


Reviews:
[{"review_id": "sMaHjSsDi6", "reviewer": "Reviewer_FRQN", "summary": "The paper introduces VideoMAR, a decoder-only autoregressive video generation framework using continuous tokens. Unlike prior work that relies on discrete tokens or diffusion methods, VideoMAR combines temporal frame-by-frame generation with spatial masked modeling, enabling efficient and high-quality video synthesis. VideoMAR outperforms prior AR models (e.g., Cosmos) on the VBench-I2V benchmark while being more efficient in terms of model size, data, and compute. It also demonstrates promising generation quality and extrapolation capabilities.", "questions": "How does VideoMAR perform on longer or more complex video scenarios (e.g., narrative structure, multiple interacting objects, real-world dynamics)?\n\nHow critical is 3D RoPE to the spatial-temporal extrapolation ability? Have you compared against other encoding strategies?\n\nHow is the inference speed compared to modern diffusion-based methods", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nQuality & Clarity: The paper is well-written and clearly structured, providing extensive experiments, detailed ablations, and visual comparisons to validate the effectiveness of each proposed component.\n\nSignificance: VideoMAR demonstrates superior performance compared to previous autoregressive models while significantly reducing model size, data requirements, and computational resources, making it practically impactful.\n\nOriginality: Introduces novel training techniques (temporal short-to-long curriculum, spatial progressive resolution, progressive temperature schedule) and extends continuous token-based masked autoregressive modeling to video generation.\n\nWeaknesses\nScope of Validation: Limited exploration on diverse datasets or complex video scenarios (e.g., long narratives) to robustly demonstrate generalization.\n\nNovelty: Components such as masked-based generation and curriculum learning individually have precedent in prior works, somewhat reducing overall conceptual novelty.\n\nComputational Bottlenecks: Although efficient, the method’s scalability to significantly longer sequences or higher resolutions without substantial computational overhead requires clearer exploration.", "was_revised": true, "final_justification": "I am satisfied with the rebuttal and will keep my rating", "rebuttal": "**Q1: Exploration on diverse datasets or complex video scenarios.**\n\nThank you for this insightful question regarding generalization. We agree that generalization is fundamentally linked to the scale and diversity of the training data. While our model excels on distributions seen during training, pushing the boundaries of generalization further is a broader challenge for the field, and we believe expanding datasets with richer motion is a critical direction for future work.\n\nIn this paper, we demonstrate our model's robust generalization capabilities through both state-of-the-art benchmark performance and targeted experiments on high-motion scenarios. **(1) Benchmark Performance:** VideoMAR achieves state-of-the-art results on the comprehensive VBench benchmark. This performance across diverse scenarios inherently validates our model's strong generalization on established tasks. **(2) Adaptation to Complex Motion:** We specifically tested the model's capacity to handle high-motion dynamics. As detailed in Supplement Section 8, after fine-tuning on specialized datasets (*e.g., eye blinks, flower blossoming*), our model successfully captures these complex movements (Figure 14). This proves the architecture's ability to adapt and generalize to new, challenging motion profiles when provided with targeted data.\n\n----\n\n**Q2: Scaling efficiency for larger spatial-temporal resolutions.**\n\nThanks for highlighting efficiency. Our model's efficiency is empirically demonstrated by its ability to outperform a powerful baseline like Cosmos with a fraction of the resources. Furthermore, we have validated its scalability by successfully training at high resolutions (49x512×768). The fact that this was achieved under standard hardware constraints is not a limitation but rather a powerful validation of our model's efficient design. We contend that our architecture is fully equipped for further scaling, which is a matter of resource allocation, not a fundamental challenge to our method.\n\n----\n\n**Q3: 3D RoPE to extrapolation ability**\n\nWe appreciate the opportunity to elaborate on our model's extrapolation capabilities. This ability is thoughtfully designed and stems from two key elements working in concert. **First**, the autoregressive design of VideoMAR provides a natural foundation for generating longer videos by modeling them sequentially. This sets the stage for extrapolation. **Second**, our experiments, detailed in Section 1 of the supplement, show why 3D-RoPE is so important. We found that models with absolute position encodings could not extrapolate to new lengths, whereas our use of 3D-RoPE, a relative encoding, successfully enabled this capability. This demonstrates that it is a critical part of our model's design for achieving robust generalization in sequence length.\n\n----\n\n**Q4: Inference speed compared to modern diffusion-based methods.**\n\nWe append the inference speed comparison with modern diffusion-based methods, including HunyuanVideo and Wan2.1, in the following table. All the values are achieved with single batch size on single H20 GPU. Our method demonstrates competitive inference speed. We will also append this result in the revised version. Note that the inference speed of VideoMAR can be further improved with smaller mask steps and diffusion steps.\n\n| Methods           |  Model size      | Resolutin       |  Inference time (s)    |\n| :---                    |    :----:              |        :----:        |              :----:           | \n| HunyuanVideo  |  13B                 | 49x608x800   | 428   |\n| Wan2.1             |  1.3B                | 49x480x832   | 103   |\n| Wan2.1             |  14B                 | 49x480x832   | 414   |\n| VideoMAR        |  1.4B                | 49x512x768   | 134   |"}, {"review_id": "9SSdu2xHdx", "reviewer": "Reviewer_QwuR", "summary": "This paper presents VideoMAR, a decoder‑only, mask‑based autoregressive (AR) framework for image‑to‑video synthesis using continuous tokens. Instead of the classic next‑token (NTP) paradigm with discrete vocabularies, VideoMAR compresses video into continuous latent tokens via a VideoVAE, then generates each frame sequentially (temporal causality) while predicting masked tokens within a frame bidirectionally. On the VBench‑I2V benchmark, VideoMAR (1.4 B params, 0.5 M videos) outperforms the prior AR state‑of‑the‑art (Cosmos) across total and sub‐dimension scores (84.82 vs. 84.22), while using an order of magnitude less data, compute, and model size. It also approaches some diffusion‑based I2V methods at far lower resource cost.", "questions": "1. Have you evaluated VideoMAR on any external I2V datasets (e.g. WebVid, YouCook2) or real‑world videos?\n2. An ablation comparing curriculum learning (short‑to‑long; progressive res) to a single‑stage training schedule would clarify its contribution to final quality vs. merely speeding convergence.\n3. Can you quantify per‑frame error accumulation (e.g., LPIPS drift) with and without temperature scheduling?\n4. Although NTP makes inference intractable, do discrete‑token AR baselines (e.g. raster‑scan MAR) achieve similar VBench scores if trained on the same data?", "limitations": "Yes, it's discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. VideoMAR sets a new AR I2V baseline, delivering strong VBench‑I2V performance with minimal resources. This demonstrates that continuous‑token, mask‑based AR can rival heavy diffusion pipelines.\n2. The integration of a next‑frame diffusion loss into mask‑based AR video, combined with curriculum strategies and temperature scheduling, is novel.\n3. The paper is well‑organized; Figures 2–7 clearly illustrate the framework, training curriculum, inference strategies, and qualitative gains.\n4. KV‑cache and spatial grouping cut inference from thousands to ~100 s, making AR I2V viable in practice. The 3D‑RoPE extrapolation results (longer/ higher‑res videos without retraining) are compelling.\n\nWeaknesses:\n1. Evaluation focuses exclusively on VBench‑I2V. It remains unclear how VideoMAR generalizes across other I2V datasets or domain shifts.\n2. Ablations cover frame loss, causal mask, and temperature, but omit: effect of curriculum vs. fixed schedules, and comparison to NTP in quality (beyond speed).\n3. Reported VBench scores lack variance estimates or multiple runs. Small differences (e.g., 84.22→84.82) may lie within noise.", "was_revised": true, "final_justification": "I thank the authors' reply and decide to maintain my score.", "rebuttal": "**Q1: Exploration on diverse datasets or complex video scenarios.**\n\nThank you for this insightful question regarding generalization. We agree that generalization is fundamentally linked to the scale and diversity of the training data. While our model excels on distributions seen during training, pushing the boundaries of generalization further is a broader challenge for the field, and we believe expanding datasets with richer motion is a critical direction for future work.\n\nIn this paper, we demonstrate our model's robust generalization capabilities through both state-of-the-art benchmark performance and targeted experiments on high-motion scenarios. **(1) Benchmark Performance:** VideoMAR achieves state-of-the-art results on the comprehensive VBench benchmark. This performance across diverse scenarios inherently validates our model's strong generalization on established tasks. **(2) Adaptation to Complex Motion:** We specifically tested the model's capacity to handle high-motion dynamics. As detailed in Supplement Section 8, after fine-tuning on specialized datasets (*e.g., eye blinks, flower blossoming*), our model successfully captures these complex movements (Figure 14). This proves the architecture's ability to adapt and generalize to new, challenging motion profiles when provided with targeted data.\n\n----\n\n**Q2: Comparison between curriculum and fixed schedule.**\n\nThank you for the question about our training schedule, we are happy to elaborate on the rationale.\nOur adoption of a short-to-long curriculum schedule is a deliberate design choice, grounded in both a clear rationale and decisive empirical validation from our early-stage experiments. \nWe selected this approach for **two primary reasons**. First, it significantly improves training efficiency, and its progressive nature aligns perfectly with the sequential modeling principles of autoregressive frameworks. \nSecond, crucially, this rationale was confirmed by direct empirical comparison. We trained two models from scratch for 20 epochs at a 25x256x256 resolution using 16 H20 GPUs at our early-stage experiments. The model with the curriculum schedule began to converge and generate roughly plausible videos. In stark contrast, the model trained with a fixed-length schedule failed to converge and produced no meaningful output under the identical setting. This result confirms that the curriculum is not merely a minor optimization but a fundamental component that enables the successful training of our model.\n\n----\n\n**Q3: Comparison between VideoMAR and NTP.**\n\nWe appreciate the opportunity to clarify our comparison strategy and the rationale for our model's design. Our work aligns with findings from prior researches (*e.g.*, MAR, Fluid), which have consistently shown that mask-based generation with bidirectional context outperforms causal NTP for visual tasks under fair comparisons.\n\nTo provide a robust empirical comparison, we benchmarked our model against Cosmos, a leading NTP-style baseline. We were pleased to find that our method achieves superior results on VBench while being significantly more resource-efficient. Given these results and the considerable resources required to train a new large-scale NTP model from scratch, we believe this comparison provides a clear and fair assessment of our method's advantages.\n\n----\n\n**Q4: Variance of the VBench score.**\n\nThank you for emphasizing the importance of statistical rigor. We provide a three-part analysis that confirms the robustness and significance of our results.\n\n**(1) Score Stability and Robustness**: VBench is already designed for reliability by averaging over five runs per sample. To provide further validation, we conducted three additional evaluation runs for both our model and the Cosmos baseline. As shown in the table below, the variance is negligible (less than 0.1), confirming that the reported scores are stable and the performance gap is consistent.\n| Methods                 |  Model size      | Total Score        |   I2V Score    |  Quality Score  |\n| :---                          |    :----:              |        :----:           |       :----:        |         :----:         | \n| Cosmos                  |  5B                   | 84.18 (±0.04)   |       92.39       |        75.97        |\n| VideoMAR-stage2  |  1.4B                | 84.82 (±0.02)   |       93.94       |        75.69        |\n\n**(2) Significance of the 0.6 Point Margin**: We wish to emphasize that a 0.6 point difference on VBench is significant in the context of state-of-the-art models. On the official VBench leaderboard, top-performing methods are often separated by margins of less than 0.1 points. Therefore, our lead of 0.6 points represents a substantial improvement.\n\n**(3) The Overarching Context of Efficiency**: Our model achieves this superior score while demanding only a fraction of the resources: 0.5\\% of the training data and 0.2\\% of the GPU hours compared to the baseline. This monumental leap in efficiency, combined with a statistically significant performance gain, validates the efficacy and novelty of our approach.\n\n----\n\n**Q5: Quantization of per‑frame error accumulation.**\n\nThanks for your valuable suggestions. We have performed the suggested LPIPS analysis, which decisively confirms the effectiveness of our temperature schedule in combating temporal drift.\n\n**(1)** We list the LPIPS drift (lower is better) comparison in the following table. **Experimental setting**: (a) we adopt VideoMAR to generate videos of resolution 49x512x768. (b) The experiments are conducted on the VBench-I2V with 246 samples. (c) We generate the corresponding videos with and without the temperature schedule. (d) We calculate the LPIP distance between the i-th frame and the first frame (0-th frame), and average across all the 246 videos. \n**Experimental conclusion**: the quantization results are consistent with our claim, that temperature schedule helps smooth the generated video.\n| LPIPS drift ($10^{-2}$)         |  0-th     | 8-th    |  16-th   |  24-th  |  32-th  |  40-th |  48-th | \n| :---                                        |   :----:   |  :----:  |  :----:    |  :----:   |   :----:  |  :----: |  :----:   | \n| w/o temperature schedule   |  0          |  2.51  |   3.03    |  3.36    |  3.55   |  3.66  |  3.80   |\n| w/ temperature schedule     |  0          |  1.96  |   2.38    |  2.63    |  2.86   |  2.99  |  3.08   |\n\n**(2)** We also quantize the performance gain of the temperature schedule via VBench scores in table 4 of the main manuscript, where temperature schedule elevates VBench score from 80.72 to 82.56."}, {"review_id": "sNuN6cp8Zj", "reviewer": "Reviewer_EZ8c", "summary": "The paper proposes an MAR-based continuous-token video generation framework and introduces a suite of efficient training and inference strategies that markedly accelerate inference speed. Notably, the method attains competitive performance while using substantially fewer parameters and training resources.", "questions": "1. For mitigating error accumulation, the authors simply tune hyper-parameters. Have Framepack[1] or similar training strategies been explored?\n\n    [1] Zhang, Lvmin, and Maneesh Agrawala. \"Packing input frame context in next-frame prediction models for video generation.\" arXiv preprint arXiv:2504.12626 (2025).\n2. Table 2 is unclear. Under the full VideoMAR method, what differentiates “NTP” from “w/o KV Cache”? Moreover, the reported 672 s for “P6-180” is unexplained—please clarify the source of this figure.\n3. In the training setup, different resolution stages employ visual tokenizers with varying compression ratios. Please justify this choice and add experiments using a consistent compression ratio for comparison.", "limitations": "The paper does not investigate MAR’s adaptation to video in sufficient depth. As the authors themselves note, images and videos differ; yet the masking strategy merely reveals preceding frames to the current frame while excluding future frames from mask prediction. This design discards MAR’s bidirectional capability along the temporal axis, effectively reducing it to an image-level bidirectional operation.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The authors are, to the best of my knowledge, the first to introduce an MAR-style generation strategy into the video domain and achieve promising results.\n2. With markedly fewer parameters and reduced training resources, the method outperforms Cosmos I2V on the VBench-I2V benchmark.\n\n**Weaknesses:**\n1. The degree of novelty and technical depth is insufficient for a top-tier conference. The work largely adapts an existing image-masking strategy to video, but does not discuss or experiment with more video-specific adaptations—for instance, the masking scheme still targets only the current frame.\n2. Presenting progressive training (increasing resolution and clip length) as a primary contribution is problematic; this is now a standard practice in image/video generation, so the added value here is limited.\n3. The paper lacks qualitative (visual) comparisons with competing methods, which are essential for assessing video generation quality.", "was_revised": true, "final_justification": "After reviewing the authors' rebuttal, I find that most of my concerns have been addressed through clear explanations and additional experiments. Therefore, I raise my rating to 4 (Borderline Accept).", "rebuttal": "**Q1: The degree of novelty and technical depth.**\n\nWe respectfully disagree. Our paper's contribution is a novel masking framework specifically engineered for video.\n\n**(1)** Contrary to the reviewer's claim, our work moves beyond simple adaptations. We conducted extensive experiments on various video-specific masking strategies to validate our final model design. For example, we experiment with the ''total mask'' strategy, which randomly masks each frames of the video (Line 96-116, 253-256, and Figure 6)\n\n**(2)** Our central innovation, the next-frame diffusion loss, is explicitly designed to solve a fundamental problem: the mismatch between training objectives and inference-time generation. By perfectly aligning the training loss with the next-frame prediction task, our method eliminates the common training-inference gap. This principled solution is the primary contribution of our paper and represents a non-trivial advancement in video modeling.\n\n----\n\n**Q2: Presenting progressive training as a primary contribution.**\n\nWe respectfully argue that this perspective misses the specific context of our contribution. The innovation is not progressive training in isolation, but its strategic and effective integration within a masked autoregressive system for video generation.\n\n**(1) Contextual Novelty**: While progressive training is indeed prevalent in diffusion models, its application to autoregressive video models is notably rare and poses unique challenges. To our knowledge, our work is the first to successfully implement and scale progressive training for masked video generation. Unlike prior methods, our system demonstrates its viability and effectiveness at a large scale, which we believe is a novel contribution.\n\n**(2) Systematic Impact and Efficiency**: The use of progressive training is not an isolated trick but a cornerstone of our system's design. This strategy is directly responsible for the remarkable efficiency gains that allow our model to surpass the strong Cosmos baseline using significantly fewer computational resources. The from-scratch training cost of our method is highly competitive and sets a new benchmark for efficiency in this domain. Furthermore, the short-to-long curriculum aligns perfectly with the sequential nature of autoregressive modeling, making it a principled rather than an ad-hoc choice.\n\n----\n\n**Q3: Qualitative comparisons with competing methods.**\n\nThanks for the comment. Our selection of baselines was deliberate and follows the established standards in autoregressive video generation. We specifically chose Cosmos as our primary comparison because it represents the state-of-the-art and is the most relevant baseline for our image-to-video task. We are confident that this comparison provides a clear and rigorous evaluation of our method's performance. If the reviewer has other specific methods in mind, we would be happy to discuss their relevance.\n\n----\n\n**Q4: The exploration of error accumulation.**\n\nThank you for the feedback. We will break down our response into three parts:\n\n**On Our Temperature Strategy**: We argue that the perceived simplicity of our temperature strategy is its primary strength, not a weakness. To our knowledge, it is the first training-free method proposed to mitigate error accumulation in this context. This is a significant advantage over the few existing alternatives, which typically rely on complex and costly training-based perturbations. Our approach is therefore highly novel, practical, and demonstrably effective, representing a valuable contribution to the field.\n\n**On the Mention of Framepack**: We must firmly state that citing Framepack is inappropriate and falls outside standard reviewing principles. Framepack was publicly released on April 17th, less than one month before the NeurIPS submission deadline of May 15th. It is a classic case of concurrent work. More importantly, from a technical standpoint, Framepack's approach is fundamentally incompatible with our model's principles. Its anti-drifting method sacrifices the strict causal prediction chain by incorporating bi-directional context, meaning it is no longer a purely autoregressive model. This design also limits its flexibility, as it demands specialized training for different sampling methods. Our method, in contrast, preserves the causal structure and requires no such modifications.\n\n**On ''Similar Training Strategies''**: We find the comment on ''similar training strategies'' too general to be actionable. For a constructive discussion, we would have appreciated specific references that the reviewer believes are relevant. Without concrete examples, we are unable to provide a direct comparison. We remain confident, based on our thorough literature review, that our system design is novel and state-of-the-art for this specific problem domain.\n\n----\n\n**Q5: NTP explanation in Table 2.**\n\nThanks for the suggestion. We will clarify in three points:\n\n**(1)** The ''NTP'' term in table 2 represents replacing the mask-based generation paradigm of VideoMAR with the vanilla next-token-prediction paradigm, while keep the other parts identical (*e.g.*, the same architecture and token-wise diffusion loss). \nThe ''w/o KV Cache'' term in table 2 represents following the mask-based generation paradigm design of VideoMAR, while without the incorporation of frame-wise KV Cache. \nThe comparison between the ''NTP'' and ''w/o KV Cache'' thus reflects the inference speed difference between the  next-token-prediction paradigm and mask-based generation paradigm. We will also append the above explanation in the revised manuscript.\n\n**(2)** This is a common operation in previous MAR-series methods (such as MAR and Fluid), where NTP and mask-based paradigms are extensively compared and discussed.\n\n**(3)** 672 is a typo. It should be 671, consistent with Table 2. Thanks for pointing out this. \n\n----\n\n**Q6: Different compression ratios.**\n\nThanks for the suggestion. This is a deliberate design choice to maintain computational feasibility for higher resolution training. By increasing the compression ratio, we effectively manage the token sequence length, ensuring that training remains efficient and fits within our available GPU memory budget. Our strong results at these resolutions validate the success of this practical strategy.\n\n----\n\n**Q7: The summarized limitation from reviewer.**\n\nThe basic concept of autoregressive generation is causality. It is also the core concept in our method that our generation is causal in temporal dimension and bidirectional in spatial dimension.\nTherefore, the reviewer's claim that our design '' *discards MAR’s bidirectional capability along the temporal axis* '' is incorrect. Our model, by design, never had temporal bidirectionality, as this is antithetical to causal generation. We believe the reviewer has confused our model's spatial bidirectionality with its strict temporal causality."}, {"review_id": "S0UlUXYTYR", "reviewer": "Reviewer_Xwue", "summary": "This paper presents VideoMAR, a masked autoregressive (MAR) framework for video generation. VideoMAR applies a causal masked autoregressive formulation for temporal modelling, while using bidirectional attention for spatial modelling. This separation enables KV caching, significantly improving inference efficiency compared to standard next-frame prediction models. In addition, the authors employ progressive training strategies (short-to-long and low-res-to-high-res) to enable 768p resolution video generation. Evaluated on VBench, VideoMAR outperforms Cosmos (another autoregressive baseline) while remaining less competitive than recent diffusion-based models that require much more compute.", "questions": "See weakenesses.", "limitations": "No. Would be great to add a small limitations section to discuss the model's current fallback as well as different areas to improve.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper proposes a clean, well-motivated design. The use of KV caching provides a substantial inference speed-up, and the generation quality over Cosmos demonstrates a solid performance improvement.\n\n[Weaknesses and Suggestions]\n- The core design of VideoMAR appears to be a direct extension of masked autoregressive (MAR) frameworks originally developed for image modeling. Similar video modelling designs using MAR, as used here, have already been explored in prior works such as NOVA, MAGI, and MarDini. The progressive training techniques for longer videos and higher resolution are widely adopted in the literature, especially in diffusion-based models, and also present in the aforementioned autoregressive works. The discussion in Lines 109–121 attempts to contrast VideoMAR with non-MAR video models, but fails to distinguish VideoMAR meaningfully from existing MAR-based models. Specifically, point 3 seems to critique the separation of temporal and spatial modeling, yet this is often considered a strength (e.g., in MarDini), as it enables more scalable architectures in a even more parameter-efficient way. More clarification is needed on why this is framed as a limitation in existing work and how VideoMAR differs.\n- In table 3, the categorisation of models into “Diffusion” and “Autoregressive” is somewhat inconsistent and confusing:\n    - The proposed model still relies on a diffusion framework, so calling it purely autoregressive may be misleading.\n    - Models such as MAGI-1, which appear in the “Diffusion” category, are actually autoregressive in design.\n    - Cosmos, the main baseline, is not a pure video generation model—it is more of a pretrained system for robotics and control tasks, which makes it an imperfect point of comparison. Including more relevant autoregressive video generation baselines would strengthen the empirical evaluation.\n-  From the appendix samples, the generated videos appear to exhibit low frame rates and minimal object motion. It would be helpful for the paper to 1. Explicitly report the frame rate (FPS) of generated videos. 2. Include training and generation configurations that impact motion dynamics (e.g., frame stride, temporal length, augmentation strategy).\n\n- The term “video-to-video” used in the paper may be misleading. The setup described, generating future frames conditioned on two reference frames, is better described as video extension. Typically, “video-to-video” refers to style transfer, semantic layout preservation, or conditioned transformation between videos.", "was_revised": true, "final_justification": "The rebuttal has resolved most of my concerns.", "rebuttal": "**Q1: VideoMAR appears to be a direct extension of masked autoregressive (MAR) frameworks.**\n\nWe respectfully disagree with this assessment. VideoMAR represents a significant advancement over standard MAR frameworks by specifically tailoring the masking mechanism to the unique properties of video.\n\n**First**, our core contribution lies in the exploration and integration of video-specific masking strategies. We systematically investigated multiple approaches, including a total mask strategy that randomly masks entire frames, to find the optimal design. This thorough investigation goes far beyond a direct extension.\n\n**Second**, the paper dedicates significant discussion to justifying our choice of the next-frame diffusion loss. This design is not arbitrary. It was chosen because it uniquely eliminates the training-inference discrepancy common in generative models. Our next-frame diffusion loss training fits the best with inference paradigm of next-frame generation, without training-inference gap.\n\n**Third**, we clearly distinguish VideoMAR from related works. As for the mentioned related works, NOVA and MAGI, we comprehensively discussed their differences in Line 37-42, and Line 73-77. NOVA proposes to decompose the temporal and spatial generation via generating the coarse features frame-by-frame and refines each frame with a spatial layer, but complicates the framework and weakens the temporal smoothness. MAGI mitigates this issue by appending a complete copy of video sequence during training, but doubles the the sequence length and training cost. MarDini is similar to NOVA in design, and proposes that MAR handles temporal planning, while DM focuses on spatial generation. We don't include MarDini in the paper since it is actually  different from MAR. MAR focuses on token-wise modeling with a small diffusion MLP (less than 10M), while MarDini focuses on image-wise modeling with a large transformer architecture (288M).\n\n----\n\n**Q2: Presenting progressive training as a primary contribution.**\n\nWe respectfully argue that this perspective misses the specific context of our contribution. The innovation is not progressive training in isolation, but its strategic and effective integration within a masked autoregressive system for video generation.\n\n**(1) Contextual Novelty**: While progressive training is indeed prevalent in diffusion models, its application to autoregressive video models is notably rare and poses unique challenges. To our knowledge, our work is the first to successfully implement and scale progressive training for masked video generation. Unlike prior methods, our system demonstrates its viability and effectiveness at a large scale, which we believe is a novel contribution.\n\n**(2) Systematic Impact and Efficiency**: The use of progressive training is not an isolated trick but a cornerstone of our system's design. This strategy is directly responsible for the remarkable efficiency gains that allow our model to surpass the strong Cosmos baseline using significantly fewer computational resources. The from-scratch training cost of our method is highly competitive and sets a new benchmark for efficiency in this domain. Furthermore, the short-to-long curriculum aligns perfectly with the sequential nature of autoregressive modeling, making it a principled rather than an ad-hoc choice.\n\n----\n\n**Q3: The discussion in Lines 109–121.**\n\nThanks for your attention in this discussion. We wish to clarify that the primary focus of this paragraph is to analyze different masking strategies within the MAR framework, not to contrast VideoMAR with non-MAR models.\nThe temporal and spatial separation we critique refers specifically to MAR-based models like NOVA. As we demonstrate, such designs can lead to severe temporal flickering and inconsistency. We did not include MarDini in this specific comparison because its image-wise modeling paradigm falls outside the scope of MAR's token-wise approach, and it remains closed-source without a public benchmark for evaluation.\n\n----\n\n**Q4: The categorization in table 3.**\n\nOur categorization in Table 3 follows established practices in generative modeling (*e.g.*, Fluid [1] and RandAR [2]), based on the fundamental modeling scope (*e.g.* token-wise modeling or not).\n\n**(1)** We classify models as ''diffusion-type'' if they operate holistically at the image or clip level, and ''autoregressive-type'' if they operate sequentially at the token level. This is a standard distinction.\n\n**(2)** Under this principle, VideoMAR, which models continuous tokens sequentially, is correctly categorized as autoregressive, consistent with prior work like Fluid [1] and RandAR [2].\n\n**(3)** Conversely, MAGI-1 performs clip-wise modeling and is thus considered a diffusion-type model. This also explains its strong performance, as it can leverage powerful pre-trained diffusion backbones, an advantage not available to token-wise models trained from scratch.\n\n**(4)** Cosmos is explicitly described as an autoregressive world model in its original paper. Its applications in robotics are downstream fine-tuning tasks, not a reflection of its core generative architecture. We also note that Cosmos was trained with massive resources (*e.g.*, 10,000 H100 GPUs), which underscores the efficiency and scalability of our approach.\n\n[1] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens\n\n[2] RandAR: Decoder-only Autoregressive Visual Generation in Random Orders\n\n----\n\n**Q5: Parameter about frame rates and others.**\n\nThanks for your interest in these details. We are happy to clarify. The motion dynamics in our generated videos are primarily determined by the training data distribution. To validate VideoMAR's capability on more complex scenarios, we direct the reviewer to our supplementary material (Section 8). As shown in Figure 14, by fine-tuning on specialized datasets, our model successfully replicates challenging motions like ''*blinking*'' and ''*blossoming*''.\nThe generated videos have a frame rate of 12 fps. During training, we subsampled the source videos by a factor of 2 (*i.e.*, sampling one frame for every two).\n\n----\n\n**Q6: The term of video-to-video.**\n\nThank you for the excellent suggestion. We have corrected this term in the revised manuscript."}]

Author rebuttal:

Meta-review: This paper presents VideoMAR, a decoder‑only, mask‑based autoregressive (AR) framework for image‑to‑video synthesis using continuous tokens. After rebuttal, it received scores of 4444. All the reviewers are positive about the paper, commenting that the proposed method is novel, which achieves superior performance with minimal resources, and is practically impactful. The paper is also well-written, contains comprehensive experiments and detailed ablations. Reviewers also commented that their concerns have been largely addressed during the rebuttal. Therefore, the AC would like to recommend acceptance of the paper.
Decision: Accept (poster)