Title: SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: World Model

Video Prediction

Robotic Control
Abstract: World models allow agents to simulate the consequences of actions in imagined environments for planning, control, and long-horizon decision-making. However, existing autoregressive world models struggle with visually coherent predictions due to disrupted spatial structure, inefficient decoding, and inadequate motion modeling. In response, we propose Scale-wise Autoregression with Motion PrOmpt (SAMPO), a hybrid framework that combines visual autoregressive modeling for intra-frame generation with causal modeling for next-frame generation. Specifically, SAMPO integrates temporal causal decoding with bidirectional spatial attention, which preserves spatial locality and supports parallel decoding within each scale. This design significantly enhances both temporal consistency and rollout efficiency. To further improve dynamic scene understanding, we devise an asymmetric multi-scale tokenizer that preserves spatial details in observed frames and extracts compact dynamic representations for future frames, optimizing both memory usage and model performance. Additionally, we introduce a trajectory-aware motion prompt module that injects spatiotemporal cues about object and robot trajectories, focusing attention on dynamic regions and improving temporal consistency and physical realism. Extensive experiments show that SAMPO achieves competitive performance in action-conditioned video prediction and model-based control, improving generation quality with 4.4× faster inference. We also evaluate SAMPO's zero-shot generalization and scaling behavior, demonstrating its ability to generalize to unseen tasks and benefit from larger model sizes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSAMPO introduces a novel world model that combines temporal causal modeling with coarse-to-fine visual auto-regression. The framework addresses key limitations in existing autoregressive world models through three main innovations: a hybrid autoregressive architecture that maintains both temporal and spatial coherence, an asymmetric multi-scale tokenizer that enhances scale-aware attention, and a trajectory-aware motion prompt module that provides explicit spatiotemporal priors for robot and object trajectories. Experimental results demonstrate the SOTA performance in video prediction and model-based RL of SAMPO.\n\nSTRENGTHS AND WEAKNESSES:\nSAMPO demonstrates strengths in world model design: e.g. the trajectory-aware motion prompt module provides explicit spatiotemporal priors, improving the model's ability to capture dynamic interactions and physical dependencies. Experimental results validate the state-of-the-art performance in video prediction, motion modeling accuracy, and robot control tasks.\n\nHowever, I'm somewhat skeptical about how helpful the generated videos actually are, as the examples shown by the authors in Figures 4 and 6 are all very short clips where the robot's movements are quite limited in range.\n\nQUESTIONS:\nWhat are the frame rate and total duration of the generated videos?\n\nIn Table 3, why is there such a large variation in success rates between different tasks? And why is the average success rate for the \"simulator\" row 100%?\n\nIn the second row of the right part of Figure 4, doesn't the red small object appear to be occluded by the robotic arm rather than disappearing?\n\nLIMITATIONS:\nThe planning experiments in this paper were all conducted in simulation platforms. The paper would be more convincing if experiments were performed in real-world environments.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new world model framework named SAMPO, which incorporates next-scale prediction and causal autoregressive prediction. More specifically, the authors propose to use an asymmetric tokenizer to represent each frame in different resolutions, while predicting frames autoregressively using a causal masking approach. This allows SAMPO to generate frame-wise trajectories based on action conditions. At the beginning of the generation, a prefix is added that includes a motion prompt, summarizing the motion history of the past frames. Overall, the proposed model outperforms related methods across multiple datasets, showing strong quantitative and qualitative results.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- Straightforward and effective approach\nThe ideas introduced in the paper are straightforward and well-motivated. For example, it is reasonable to assume that different scales are needed for different parts of the prediction process. At the same time, the proposed method is based on the established autoregressive training strategy, showing strong generalization and scaling properties.\n- Strong Performance and convincing results\nThe provided quantitative and qualitative results underline the advantages that are offered by the proposed method over related works. Across multiple tasks, including action-conditioned and reward-based generation, the proposed model offers strong performance.\n- Thorough ablation study and analysis\nThe ablation study provides an important look into how each of the proposed components improves the performance. The authors cover most of the interesting ablation scenarios.\n\n### Weaknesses\n- Typographical errors in figures\nThere are multiple typographical errors in the paper, especially in Figure 2 (e.g., \"Enocder\", \"uantization\"). However, this should be easily fixed.\n- Clearer comparison with related work\nIt appears that the proposed approach is close to iVideoGPT, but adds the scale-variant component and some additional ideas. While the ablation study is thorough, it would be nice to make the relationship between those two components clearer.\n\nQUESTIONS:\n- How does an AR-based model perform compared to a Hybrid AR model with the same number of parameters? What is the overhead introduced by Hybrid AR?\n- How does the error change over the rollout trajectory, especially compared to other models?\n- Do the experiments on the BAIR dataset starting with 1 context frame use the motion prompt? For RoboNet, does the motion prompt only consider the two first frames?\n\nLIMITATIONS:\nYes (in Supplementary Material)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SAMPO, a video world model that predicts future videos in an autoregressive manner while generating multi-scale tokens in a coarse-to-fine fashion for each frame. It introduces an asymmetric tokenization process that allows future predictions to focus more on compact dynamics, thereby improving efficiency. Additionally, it incorporates motion prompts as extra cues for scene dynamics. Its high prediction capability results in superior performance in perception quality, visual planning, and model-based reinforcement learning.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nS1) Extensive experiments demonstrate that the proposed method surpasses previous world models, such as DreamerV3 and iVideoGPT, in perception quality, visual planning, and model-based reinforcement learning.\n\nS2) There are no major flaws in the writing, making it easy to follow.\n\nS3) The code is provided in the supplementary material for reproducibility.\n\n# Weaknesses\n\nW1) **Novelty concern.** The main benefits claimed by the authors are primarily derived from VAR (NeurIPS 2024), which proposes a next-scale coarse-to-fine prediction architecture. Additionally, the extraction and use of motion prompts are highly similar to those in TraceVLA (ICLR 2025).\n\nW2) **Confusing motivation.** It is unclear why predicting only coarser scales would enable the model to focus more on dynamic regions. Theoretically, the ratio of dynamic areas does not increase when the frame scale is reduced. If static content dominates the frame, it will still dominate at lower resolutions. Could you provide some references to support that? Also, as the model only predicts coarse frames, does this imply it cannot perform future rollouts by conditioning on the newly generated frames?\n\nW3) **Missing ablations.** It would be beneficial to see how changing the predicted granularity (full scales / coarser scales only) affects the performance in downstream planning tasks. Can predicting even coarser scales continue to improve efficiency and downstream performance? Is there a trade-off?\n\nW4) **Unclear details.** In the zero-shot generalization experiments described in Line 297-301, how are the action conditions transferred? Additionally, how is the reward predicted in the MBRL experiments? The paper only provides some implementation choices in Line 214-215 but does not specify what is ultimately used.\n\nW5) **Real-world experiment.** It would be great if the authors can do a real-world verification with some simple setups. For example, estimating the rewards of several robot policies with the proposed world model and comparing to the actual evaluation ranks.\n\nQUESTIONS:\nPlease see the Weaknesses part.\n\nLIMITATIONS:\nPlease see the Weaknesses part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SAMPO, an intuitive way to improve robotic world models with a hybrid multi-scale prediction framework. The hybrid decoding choices help with spatial consistency via VAR-like methods and ensure temporal consistency via autoregressive prediction. The paper also describe and propose solution to several practical difficulties, such as asymmetric tokenizer to handle different kinds of video frames efficiently.\n\nResults have shown that this method is efficient and effective compared to several strong baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nGenerally, this paper presents a method that naturally extend prior work and is executed well.\n\n- The method is intuitive and builds on the past work smoothly. SAMPO's integrated design, particularly its hybrid multi-scale approach appears to effectively balance the need for both spatial detail and smooth temporal transitions.\n- The paper is well-written and progressively describes a complex system, which involves several components and several method improvements.\n\n\nWeakness:\n\n- SAMPO involves several sophisticated modules and have quite some inherent complexity. This may make future improvements on the base more difficult.\n- SAMPO utilizes a sparse assumption. The asymmetric tokenizer's approach of sparsely tokenizing future frames is effective for efficiency, but it would be valuable to discuss how well (or limited) if this sparse representation scales to capture details in highly complex real-world dynamics.\n\nQUESTIONS:\n- We understand that scaling such system requires both cost of training and cost of obtaining data. But I would still be interested to see if the authors have ideas on how to further scale up the model. And besides these costs, are there other considerations for scaling such model?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}