Title: VideoMAR: Autoregressive Video Generation with Continuous Tokens
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video generation

Autoregressive model

Continuous tokens
Abstract: Masked-based autoregressive models have demonstrated promising image generation capability in continuous space. However, their potential for video generation remains under-explored. 
Masked-based autoregressive models have demonstrated promising image generation capability in continuous space. However, their potential for video generation remains under-explored. 
In this paper, we propose \textbf{VideoMAR}, a concise and efficient decoder-only autoregressive image-to-video model with continuous tokens, composing temporal frame-by-frame and spatial masked generation.
We first identify temporal causality and spatial bi-directionality as the first principle of video AR models, and propose the next-frame diffusion loss for the integration of mask and video generation.  
Besides, the huge cost and difficulty of long sequence autoregressive modeling is a basic but crucial issue. To this end, we propose the temporal short-to-long curriculum learning and spatial progressive resolution training, and employ progressive temperature strategy at inference time to mitigate the accumulation error. 
Furthermore, VideoMAR replicates several unique capacities of language models to video generation. 
It inherently bears high efficiency due to simultaneous temporal-wise KV cache and spatial-wise parallel generation, and presents the capacity of spatial and temporal extrapolation via 3D rotary embeddings.
      On the VBench-I2V benchmark, VideoMAR surpasses the previous state-of-the-art (Cosmos I2V) while requiring significantly fewer parameters ($9.3\%$), training data ($0.5\%$), and GPU resources ($0.2\%$).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents VideoMAR, a masked autoregressive (MAR) framework for video generation. VideoMAR applies a causal masked autoregressive formulation for temporal modelling, while using bidirectional attention for spatial modelling. This separation enables KV caching, significantly improving inference efficiency compared to standard next-frame prediction models. In addition, the authors employ progressive training strategies (short-to-long and low-res-to-high-res) to enable 768p resolution video generation. Evaluated on VBench, VideoMAR outperforms Cosmos (another autoregressive baseline) while remaining less competitive than recent diffusion-based models that require much more compute.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper proposes a clean, well-motivated design. The use of KV caching provides a substantial inference speed-up, and the generation quality over Cosmos demonstrates a solid performance improvement.\n\n[Weaknesses and Suggestions]\n- The core design of VideoMAR appears to be a direct extension of masked autoregressive (MAR) frameworks originally developed for image modeling. Similar video modelling designs using MAR, as used here, have already been explored in prior works such as NOVA, MAGI, and MarDini. The progressive training techniques for longer videos and higher resolution are widely adopted in the literature, especially in diffusion-based models, and also present in the aforementioned autoregressive works. The discussion in Lines 109–121 attempts to contrast VideoMAR with non-MAR video models, but fails to distinguish VideoMAR meaningfully from existing MAR-based models. Specifically, point 3 seems to critique the separation of temporal and spatial modeling, yet this is often considered a strength (e.g., in MarDini), as it enables more scalable architectures in a even more parameter-efficient way. More clarification is needed on why this is framed as a limitation in existing work and how VideoMAR differs.\n- In table 3, the categorisation of models into “Diffusion” and “Autoregressive” is somewhat inconsistent and confusing:\n    - The proposed model still relies on a diffusion framework, so calling it purely autoregressive may be misleading.\n    - Models such as MAGI-1, which appear in the “Diffusion” category, are actually autoregressive in design.\n    - Cosmos, the main baseline, is not a pure video generation model—it is more of a pretrained system for robotics and control tasks, which makes it an imperfect point of comparison. Including more relevant autoregressive video generation baselines would strengthen the empirical evaluation.\n-  From the appendix samples, the generated videos appear to exhibit low frame rates and minimal object motion. It would be helpful for the paper to 1. Explicitly report the frame rate (FPS) of generated videos. 2. Include training and generation configurations that impact motion dynamics (e.g., frame stride, temporal length, augmentation strategy).\n\n- The term “video-to-video” used in the paper may be misleading. The setup described, generating future frames conditioned on two reference frames, is better described as video extension. Typically, “video-to-video” refers to style transfer, semantic layout preservation, or conditioned transformation between videos.\n\nQUESTIONS:\nSee weakenesses.\n\nLIMITATIONS:\nNo. Would be great to add a small limitations section to discuss the model's current fallback as well as different areas to improve.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an MAR-based continuous-token video generation framework and introduces a suite of efficient training and inference strategies that markedly accelerate inference speed. Notably, the method attains competitive performance while using substantially fewer parameters and training resources.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The authors are, to the best of my knowledge, the first to introduce an MAR-style generation strategy into the video domain and achieve promising results.\n2. With markedly fewer parameters and reduced training resources, the method outperforms Cosmos I2V on the VBench-I2V benchmark.\n\n**Weaknesses:**\n1. The degree of novelty and technical depth is insufficient for a top-tier conference. The work largely adapts an existing image-masking strategy to video, but does not discuss or experiment with more video-specific adaptations—for instance, the masking scheme still targets only the current frame.\n2. Presenting progressive training (increasing resolution and clip length) as a primary contribution is problematic; this is now a standard practice in image/video generation, so the added value here is limited.\n3. The paper lacks qualitative (visual) comparisons with competing methods, which are essential for assessing video generation quality.\n\nQUESTIONS:\n1. For mitigating error accumulation, the authors simply tune hyper-parameters. Have Framepack[1] or similar training strategies been explored?\n\n    [1] Zhang, Lvmin, and Maneesh Agrawala. \"Packing input frame context in next-frame prediction models for video generation.\" arXiv preprint arXiv:2504.12626 (2025).\n2. Table 2 is unclear. Under the full VideoMAR method, what differentiates “NTP” from “w/o KV Cache”? Moreover, the reported 672 s for “P6-180” is unexplained—please clarify the source of this figure.\n3. In the training setup, different resolution stages employ visual tokenizers with varying compression ratios. Please justify this choice and add experiments using a consistent compression ratio for comparison.\n\nLIMITATIONS:\nThe paper does not investigate MAR’s adaptation to video in sufficient depth. As the authors themselves note, images and videos differ; yet the masking strategy merely reveals preceding frames to the current frame while excluding future frames from mask prediction. This design discards MAR’s bidirectional capability along the temporal axis, effectively reducing it to an image-level bidirectional operation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents VideoMAR, a decoder‑only, mask‑based autoregressive (AR) framework for image‑to‑video synthesis using continuous tokens. Instead of the classic next‑token (NTP) paradigm with discrete vocabularies, VideoMAR compresses video into continuous latent tokens via a VideoVAE, then generates each frame sequentially (temporal causality) while predicting masked tokens within a frame bidirectionally. On the VBench‑I2V benchmark, VideoMAR (1.4 B params, 0.5 M videos) outperforms the prior AR state‑of‑the‑art (Cosmos) across total and sub‐dimension scores (84.82 vs. 84.22), while using an order of magnitude less data, compute, and model size. It also approaches some diffusion‑based I2V methods at far lower resource cost.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. VideoMAR sets a new AR I2V baseline, delivering strong VBench‑I2V performance with minimal resources. This demonstrates that continuous‑token, mask‑based AR can rival heavy diffusion pipelines.\n2. The integration of a next‑frame diffusion loss into mask‑based AR video, combined with curriculum strategies and temperature scheduling, is novel.\n3. The paper is well‑organized; Figures 2–7 clearly illustrate the framework, training curriculum, inference strategies, and qualitative gains.\n4. KV‑cache and spatial grouping cut inference from thousands to ~100 s, making AR I2V viable in practice. The 3D‑RoPE extrapolation results (longer/ higher‑res videos without retraining) are compelling.\n\nWeaknesses:\n1. Evaluation focuses exclusively on VBench‑I2V. It remains unclear how VideoMAR generalizes across other I2V datasets or domain shifts.\n2. Ablations cover frame loss, causal mask, and temperature, but omit: effect of curriculum vs. fixed schedules, and comparison to NTP in quality (beyond speed).\n3. Reported VBench scores lack variance estimates or multiple runs. Small differences (e.g., 84.22→84.82) may lie within noise.\n\nQUESTIONS:\n1. Have you evaluated VideoMAR on any external I2V datasets (e.g. WebVid, YouCook2) or real‑world videos?\n2. An ablation comparing curriculum learning (short‑to‑long; progressive res) to a single‑stage training schedule would clarify its contribution to final quality vs. merely speeding convergence.\n3. Can you quantify per‑frame error accumulation (e.g., LPIPS drift) with and without temperature scheduling?\n4. Although NTP makes inference intractable, do discrete‑token AR baselines (e.g. raster‑scan MAR) achieve similar VBench scores if trained on the same data?\n\nLIMITATIONS:\nYes, it's discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VideoMAR, a decoder-only autoregressive video generation framework using continuous tokens. Unlike prior work that relies on discrete tokens or diffusion methods, VideoMAR combines temporal frame-by-frame generation with spatial masked modeling, enabling efficient and high-quality video synthesis. VideoMAR outperforms prior AR models (e.g., Cosmos) on the VBench-I2V benchmark while being more efficient in terms of model size, data, and compute. It also demonstrates promising generation quality and extrapolation capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nQuality & Clarity: The paper is well-written and clearly structured, providing extensive experiments, detailed ablations, and visual comparisons to validate the effectiveness of each proposed component.\n\nSignificance: VideoMAR demonstrates superior performance compared to previous autoregressive models while significantly reducing model size, data requirements, and computational resources, making it practically impactful.\n\nOriginality: Introduces novel training techniques (temporal short-to-long curriculum, spatial progressive resolution, progressive temperature schedule) and extends continuous token-based masked autoregressive modeling to video generation.\n\nWeaknesses\nScope of Validation: Limited exploration on diverse datasets or complex video scenarios (e.g., long narratives) to robustly demonstrate generalization.\n\nNovelty: Components such as masked-based generation and curriculum learning individually have precedent in prior works, somewhat reducing overall conceptual novelty.\n\nComputational Bottlenecks: Although efficient, the method’s scalability to significantly longer sequences or higher resolutions without substantial computational overhead requires clearer exploration.\n\nQUESTIONS:\nHow does VideoMAR perform on longer or more complex video scenarios (e.g., narrative structure, multiple interacting objects, real-world dynamics)?\n\nHow critical is 3D RoPE to the spatial-temporal extrapolation ability? Have you compared against other encoding strategies?\n\nHow is the inference speed compared to modern diffusion-based methods\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}