SAMPO: Scale-wise Autoregression with Motion Prompt for Generative World Models
Authors: Sen Wang
Jingyi Tian
Le Wang
Zhimin Liao
lijiayi
Huaiyi Dong
Kun Xia
Sanping Zhou
Wei Tang
Gang Hua
Venue: NeurIPS 2025 poster ()
Keywords: World Model
Video Prediction
Robotic Control
Primary area: reinforcement_learning

Abstract: World models allow agents to simulate the consequences of actions in imagined environments for planning, control, and long-horizon decision-making. However, existing autoregressive world models struggle with visually coherent predictions due to disrupted spatial structure, inefficient decoding, and inadequate motion modeling. In response, we propose Scale-wise Autoregression with Motion PrOmpt (SAMPO), a hybrid framework that combines visual autoregressive modeling for intra-frame generation with causal modeling for next-frame generation. Specifically, SAMPO integrates temporal causal decoding with bidirectional spatial attention, which preserves spatial locality and supports parallel decoding within each scale. This design significantly enhances both temporal consistency and rollout efficiency. To further improve dynamic scene understanding, we devise an asymmetric multi-scale tokenizer that preserves spatial details in observed frames and extracts compact dynamic representations for future frames, optimizing both memory usage and model performance. Additionally, we introduce a trajectory-aware motion prompt module that injects spatiotemporal cues about object and robot trajectories, focusing attention on dynamic regions and improving temporal consistency and physical realism. Extensive experiments show that SAMPO achieves competitive performance in action-conditioned video prediction and model-based control, improving generation quality with 4.4× faster inference. We also evaluate SAMPO's zero-shot generalization and scaling behavior, demonstrating its ability to generalize to unseen tasks and benefit from larger model sizes.

SAMPO: Scale-wise Autoregression with Motion PrOmpt for Generative World Models
Sen Wang'
Jingyi Tian' Le Wang R Zhimin Liao' Jiay Li Huaiyi Dong'
Kun Xia' Sanping Zhou' Wei Tang" Hua Gang*
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi an Jiaotong University 2 University of Illinois at Chicago Ammzon..cmm Inc.
Abstract
World models allow agents to simulate the consequences of actions imagined environments for planning, control, and long-horizon decision-makinn However, existing autoregressive world models struggle with visually coherent predictions due to disrupted spatial structure, inefficient decoding, and inadequate motion mod- eling. In response, we propose Scale-wise Autoregression with Motion PrOmpt (SAMPO), a hybrid framework that combines visual autoregressive modeling for intra-frame generation with causal modeling for next-fraid frame generation. Specifi- cally, SAMPO integrates temporal causal decoding with bidirectional spatial atten- tion, which preserves spatial locality and supports parallel decoding within each scale. This design significantly enhances both temporal consistency and rollout efficiency. To further improve dynamic scene understanding. we devise an asym- metric multi-scale tokenizer that preserves spatial details in observed frames and extracts compact dynamic representations for future frames, optimizing both mem- ory usage and model performance. Additionally, we introduce a trajectory-awarr motion prompt module that injects spatiotemporal cues about object and robot trajectories, focusing attention on dynamic regions and improving temporal con- sistency and physical realism. Extensive experiments show that SAMPO achieves competitive performance in condiinnondi video prediction and model- based control, improving generation quality with 4.4x faster inference. We also evaluate SAMPO's zero-shot generalization and scaling behavior, demonstrating ability to generalize to unseen tasks and benefit from larger model sizes.
Introduction
Building a world model that can simulate the physical environment and respond to the actions agents
is central challenge the path to artificial general intelligence (AGI) [2. 11, 77, 33, 52, 72, 73]. Recently, video generation has been integrated into world models, enabling models to generate future frames based on agent actions, simulating dynamic environments and making it possible for agents to anticipate outcomes and make informed decisions [24, 50, 4]. Despite growing progress, designing world model that is simultaneously high-fidelity, temporally consistent, and efficiently scalable remains an open problem.
Prior works have advanced video-based world models by formulating future prediction as an action- conditioned generation problem. These approaches can be categorized into three major families based on their generative paradigms: masked modeling, diffusion-based models and autoregressive models.
X Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

I hove isamed
Coople stati suiid
Next Taken Prediction in Time
Actlot-codittand Video Prediction
Byans Interacthv Infammttion
Tone May Preteees
Amertian
A
[A3
Trucher did lirteet
Drfices Speac line
Large-Sane Robot Datasets
SAMPO
2016
Z+
Multi -ccll
Tokenkzer
Up-interpplation
Modal baxed Reinforoemet Leomingg Hem
Image MTt Mation Froupt
Future Froves
Peficiaan Envirmmon Shaps
Figure I: SAMPO Overview. SAMPO is a scale-wise autoregressive world model for video predic- tion and robotic control. It models temporal dynamics through frame-wise causal generation, while capturing spatial structure via multi-scale tokenization and coarse-to- caass-tto-fine to-fine prediction. A trajectory- aware motion prompt further enhances spatiotemporal grounding. SAMPO supports high-fidelity, action-conditioned rollouts for visual planning and model-based reinforcement learning.
Masked modeling [75, 8, 19, 68, 51] achieves efficient pretraining by reconstructing missing patches, yet often sacrifices temporal consistency and causality due to its localized objective. Diffusion-basee models [56, 7. 26, 65] produce high-fidelity video via iterative denoising, but suffer from slow inference and limited interactivity. In contrast, autoregressive models [38, 77, 60, 62] generate tokens sequentially, preserving causal structure and supporting in-context prediction [10]. making them better aligned with the requirements of world models, where accurate, interactive and temporally
consistent forecasting is essential.
Despite their success in building world models [60], current autoregressive approaches still face the following limitations: 1) Structural degradation due to raster-scan flattening [45, 54, 32j. Flattening disrupts the spatial locality of video frames, hindering the model's ability to capture long-range de- pendencies across space, leading to physically implausible generation, such object disappearances or blurred manipulators (Fig. 4). 2) Slow and error- arror-prone next-token prediction Ieads to inefficiency and the accumulation of errors during generation [5, 41]. 3) Insufficient modeling of salient motion and interactions, which diminishes the physical realism and smoothness of dynamic scenes [74].
To address these challenges, we propose SAMPO, a scale-wise autoregressive framework that combines bidirectional spatial attention within frames and causal temporal modeling across time. SAMPO introduces a new autoregressive formulation tailored for world models, combining next- scale spatial prediction [45] with temporal causal generation, thereby unifying spatial coherence and temporal consistency under scalable architecture. Specifically, the model autoregresses over time while generating each frame's token maps in a coarse-to-fine manner, progressively from low to high resolution with parallel token generation within each scale. Compared to raster-scan flattening, which disrupts spatial continuity and object boundaries [62, 32), our hicrarchical token generation effectively preserves spatial locality and structural coherence within each frame and supports scalable and efficient generation across resolutions.
To further balance spatial detail and dynamic modeling [58, 60), we devise an asymmetric multi- scale tokenizer based on vector quantization [14, 48]. Observed frames are densely tokenized to preserve static background and contextual information, while future frames use sparse tokenization to emphasize dynamic changes and reduce redundancy [18]. As shown Fig. 1, this design improves inference speed while maintaining visual fidelity. Notably, this formulation supports token-level integration of visual inputs and agent actions. With autoregressive scalability, SAMPO can be pretrained on large-scale robot datasets [37], enabling generalizable and control-centric world models across diverse tasks and settings.
While improving visual fidelity is desirable goal, the core objective of an interactive world model is to accurately predict future states in response to agent actions [46, 33]. Existing approaches often struggle to model meaningful dynamic interactions [60, 58, 621, particularly in environments domi- nated by static or quasi-static frames [13, 9], resulting in blurred or inconsistent object interactions. motion prompt module that provides
To address this limitation, we introduce a trajectory- spatiotemporal cues about object and robot trajectories within the observed frames [31, 30J. These motion prompts serve as dynamic priors, guiding the model's attention toward interaction-rellvant

---

regions, such as robotic arms and manipulated objects. By explicitly conditioning on motion tra- jectories, SAMPO improves its capability to model object-agent interactions, maintain temporal consistency and capture underlying physical dependencies.
In summary, the main contributions of this study can be summarized as follows:
1. We propose SAMPO, scale-wise autoregressive framework that combines temporal causal modeling with noarse-to-fine visual autoregression and an asymmetric multi-scale VQ tokenizer, preserving spatial locality while significantly improving generation efficiency.
2. We introduce trajectory-aware motion prompt module that provides explicit spatiotemporal priors over robot and object trajectories, enhancing the model's 's ability to capture dynamic interactions and physical dependencies in complex manipulation tasks.
3. Extensive experiments demonstrate that SAMPO outperforms existing state-of-the- methods in terms of video quality, motion modeling accuracy, and robot control performance, offering a new insight for scalable and structurally coherent world model design.
2 Related Work
2.1 Generative Models for World Modeling
World Modeling as an Embodied Simulator. World models have emerged as a fundamental paradigm for enabling agents to reason about and interact with complex environments. Broadly, world models serve two complementary purposes: constructing internal representations that abstract the external world and predicting its future evolution to guide decision-making [20, 33, 11]. Farly works emphasized building compact, latent models that capture essential environmental dynamics, supporting tasks such as planning and policy learning in model-based reinforcement learning [21, 28).
Recent advances in video generation and large multimodal models have shifted attention toward direct pixel-level predictions of future world states [77, 7, 8, 26], providing richer supervision and expanding the applicability of world models to diverse tasks, from robotic manipulation to embodied social simulation. Beyond pixel prediction, critical evolution in world models lies in supporting interactivity and control [60]. An effective world model should not only generate visually plausible futures but also simulate the consequences of agent actions and respond with feedback. This capability enables agents to interact with imagined environments in a closed-loop manner - testing actions, observing outcomes, and refining strategies accordingly [6, 11]. Such interactive modeling is essential for decision- makinng tasks that require dynamic adaptation, from robot manipulation to embodied reasoning. In this work, we focus on advancing interactive world models toward efficient, structurally coherent visual dynamics prediction, integrating spatial structure priors with scalable autoregressive architectures.
Visual Autoregressive Modeling. VAR introduces a new generation paradigm that redefines au- toregressive learning as next-scale prediction, enabling transformers to better capture visual distribu- tions [45]. By replacing raster-scan ordering with multi-scale token map prediction, VAR preserves spatial locality, reduces sequential dependency, and enables efficient parallel decoding within each scale. Inspired by these insights, coarse-to-fin multi-scale generation has begun to influence a broad range of fields, including high-resolution image synthesis(23]. 3D generation [70], multimodal LLM [78, 79] and robotic manipulation [17]. However, these methods have yet to explore integrating VAR into intra-frame generation and still rely on suboptimal raster-scan ordering in image generation.
2.2 Motion Prompt for Visual Dynamies Modeling
Visual prompt have emerged as lightweight alternative to architectural changes for guiding multi- modal models [43, 64, 63, 67, 61]. Early efforts introduce coarse overlays or fine-grained masks to input images, steering large -asionnlanguage models toward target regions [43, 64, 67, 53]. While effective for object localization [63, 61], these prompts encode only static spatial cues and fail to capture object motion, limiting their suitability for world model learning and control. To address this limitation, recent works have introduced motion-aware prompting techniques that explicitly encode spatio-temporal dynamics [16, 74, 34, 59]. Motion prompting [16] controls video diffusion models using sparse or dense motion tracks, enabling realistic and controllable object and camera dynamics. TraceVLA [74] overlays tracked trajectories [3]] visual prompts to inject temporal context into
3

---

vision languaeee action models without architectural changes. Complementarily MoVideo [34] integrates optical flow and depth features to enhance motion fidelity and temporal consistency in video generation. These methods demonstrate that motion-aware prompt can enhance dynamic fidelity without compromising efficiency. However, they often target generative tasks or depend on offline trajectories. In contrast, we propose an online motion prompting scheme that integrates with interactive world models for efficient and physically constrained visual control.
3
Method
In this section, we elaborate on the proposed SAMPO, scale-wise autoregressive world model that integrates temporally causal modeling with bidirectional spatial attention in each frame. We commence with brief background on visual autoregressive modeling and formulate the problem.
3.1 Preliminaries and Problem Statement
Next-scale prediction. VAR [45] introduces novel generation paradigm that predicts images hierarchically from coarse to fine token maps. Instead of autoregressively generating a flattened raster-scan sequence, VAR decomposes an image into multi-scale token maps and models generation at each spatial scale, conditioned on all previous scales. Formally, given hierarchical token maps {2(1), z 22)), 2(4),, }. where each token map 2(1) HH: from low to high resolution, the generation objective can be factorized as:
L ((2(1) (1)) II p(2(0 (2(1) (1-1))
(1)
where z11) denotes the token map at scale l. Each finer scale prediction is conditional on all previously generated coarser token maps, enabling coherent and efficient spatial modeling.
This coarse-to-fine framework, while originally developed for images, preserves spatial locality and enables parallel decoding. When combined with frame-wiss causal modeling, it naturally extends to spatiotemporal modeling and forms the basis of SAMPO for the structured world model.
World Model Formulation. We formulate world models as an interactive video prediction prob- lem [60], where the model simulates future observations and rewards conditioned past observations and actions, which can be formalized as a partially observable Markov decision process (POMDP), defined as: M (S, O, ø. A. J,p, r)).. At each timestep f, the agent receives partial observation E A, and transitions to new latent state 8(+1 ~ p( p($t+1 8t, az).
O, E O, takes an action receiving a reward r, r(s at). The objective
learn policy I (a; 01:)) that maximizes the expected discounted return. To support this, a world model approximates the environment's 's transition dynamics by learning the predictive distribution: p(0.11, Pt+1
O1:t; a1:t).
3.2 Scale-wise Visual Autoregressive for World Models
Hybrid Autoregressive Architecture. We propose SAMPO, a scale-wise visual autoregressive architecture for world modeling over multimodal inputs, which unifies temporal and spatial generation through a coarse-to fine decoding scheme. This design enables our world model to preserve both temporal causality across frames and spatial semantic coherence within each frame. Our experiments demonstrate that the hybrid architecture improves generation quality while also accelerating inference.
Specifically, given an input sequence of observation frames V = {fr T=** we first discretize each frame ft into a hierarchy of multi-scale token maps using vector-quantized tok- enizers [14, 48], yielding {z(") ",,W; | 1= 1,...,4}, where L denotes the total number of spatial scales. We then adopt a hybrid decoding scheme, which is autoregressive across frames (temporal) while generating tokens in a coarse-to-fine fo-fine manner within each frame (spatial). The hybrid autoregressive likelihood formulated as:
L}.
p((2( (1)
(<TT))
(1) IIII ==1==
(2T) 22 aaz),
where 2/ denotes all token maps from previous frames, 2 represents coarser-scale maps already decoded within the current frame, and a is the action sequence from the initial to the current time

---

Contect taken Asymmetric Tokenizer
becoder
Deceder
Enbeddin
Extropy
SAMPO
Proffi
Frope1
Frope2
Up Interpolation
Mulh-scol
Hed
Muhti-scole Enceder
Enooder
Observed with Nastion
Short taken
Prompt
Dynomic
Block-wise cousol mosk
Future Fromes
Figure 2: The overall framework of SAMPO. The observed and future frames are discretized by a multi-scale tokenizer to obtain dense and sparse token maps, which are then autoregressively predicted across time, while following coarse-to-fine decoding order within each frame. Motion prompts extracted from observed frames are injected alongside visual tokens to guide dynamic modeling.
step. Actions are integrated by linear projection and added to the start token embedding. At the k-th generation step of the (-th frame, the observed frames are encoded into compact features along with previously decoded tokens, forming the prefix for generating the next-scale tokens x All tokens for the current step are generated in parallel, using a block-wiss causal mask to ensure each token attends only to the prefix. During inference, we employ KV-caching [15] and no mask is needed.
Asymmetric Multi-scale Tokenizer. In robot-centric world modeling, observed frames (e.g., those acquired before taking actions) and future frames (e.g.. those imagined during planning) exhibit distinct characteristics [58, 60]. Observed frames typically contain complex static spatial layouts, sensor noise, and rich contextual cues. In contrast, future frames primarily reflect sparse motion involving the robotic arms and manipulated objects, while most background remains static, assuming a relatively stable camera viewpoint without significant egomotion.
To better align with this asymmetry, propose asymmetric multi-scale tokenizer. Observed frames (t - To) are tokenized across all spatial scales E ,I....,,yutti yielding fine- fine-grained token maps with dimensions 21)) E z W: at each scale. While for future frames (I > To). we select only sparse subset of coarser scales E (1,..., ... [eeud) with Luu < Liuut, reducing token redundancy and focusing modeling capacity on dynamic regions. The encoded token map is defined as:
2 (
ift < To. LE {1,... Luuit (T((f)) Cross Z1:T); ift To, 1 {1,... Laub)
Ti(f(),
(3)
where Ti(-) denotes the scale-specific tokenizer at scales l. For future frames (t > To). cross- attention incorporates information from observed frames. This asymmetry leads to efficient and structured representation by enhancing scale-aware attention during generation and disentangling static background priors from dynamic foreground variations.
3.3
Trajectory-aware Motion Prompt
While next scale autoregression improves spatial coherence, it remains insufficient for dynamic understanding, especially under static quasi-static training distributions [13, 9]. To mitigate this, we incorporate explicit trajectory-aware motion prompts that guide the model to focus on dynami- cally relevant regions. We extract motion prompts using Co CoTracker3 [30], a point-tracking model. Specifically, we adopt the scaled _nnlinee variant for efficient and robust trajectory extraction.
Following the definition of J. in Sec. 3.2, we uniformly sample a regular grid of query points on the first frame f1 with a predefined grid size G, resulting in N G2 initial points (xx Vi N
(1)
A lightweight variant fine- -unet on 15k world videos via pseudo-labeling, showing significant improve- ments in robotic tracking benchmarks such as RoboTAP [49].
5

---

These query points are tracked across frames by CoTracker3, generating raw trajectories:
P. P:= { 10 ) for i ,...,NN
where (x: denotes the predicted 2D location of point at frame t. For world modeling in robotic manipulation, we extract trajectories focus on the dynamics of the robot arms and manipulated objects, rather than static backgrounds or noisy artifacts. To this end, we filter raw CoTracker3 outputs based on two criteria. First, trajectories with low average tracking confidence < are discarded. Second, we compute the displacement over short temporal window Al 4 frames to identify static points: d [
(1+-0)
(0) :3
Vi
(S)
and discard trajectories where d/4t) remains below 2% of the image diagonal throughout the sequence. The resulting dynamic trajectories serve as motion prompts, enabling spatiotemporal reasoning about interactive agents and objects. The retained dy- namic trajectories are overlaid onto the original observation to form the motion prompt (see Fig. 3). Following [74], concatenate the motion prompt with the observation, separated by a special token, to construct dual-branch input. This design en- hances the spatiotemporal grounding of the model and improves its ability simulate physically plau- sible interactions.
Miction Prompt
Point Tracking
Observed Fromes
Uniforbly Ropicette
Figure 3: Motion Prompt generation.
3.4 Training Objectives
SAMPO is trained autoregressively predict future token maps over both temporal and spatial scales. Let 2 E iR xWVVV denote the predicted logits at scales 4, where V represents the size of the codebook, and 2 HHHiWii are the corresponding ground-truth token indices. The training objective is multi-scale cross-entropy loss over future frames:
(i) = -- A ETSTO 1=1
Σ 1.3
log 2000 (1,5) 2 (10) (1.5)
(6)
H.W:
where A is a scale-specific weight (detailed Appendix A.2), and To denotes the final observed frame. The loss is computed only on future frames, encouraging prediction conditioned on context. Action-conditioned prediction with reward. For control-centric applications (e.g., model-based RL), the objective can be optionally extended with auxiliary heads, including reward prediction or trajectory decoding. For example, a linear head can be applied to the last token't 's hidden state in each frame, using a MSE loss for reward prediction [60]. This modification enables more effective task-relevant learning, improving performance in control tasks [35].
3.5 Implementation Details
We design an asymmetric multi-scale tokenizer built on VQGAN [14], which independently encodes observed and future frames using separate codebooks of size 8192. Both share a CNN backbone but their parameters are independently updated. Our tokenizer is pretrained on Open X-Fmbodiment [37] with reconstruction and commitment loss.
Asymmetric Tokenizer.
Transformer. SAMPO follows the decoder-only architecture [40] implemented in VAR [45]. To maintain temporal consistency, we introduce a start token [S] at the beginning of each frame, which segments the frame and enables autoregressive prediction with teacher forcing. Additionally, we apply fixed ID sine-cosine embeddings to encode temporal positions, following standard practice in visual Transformers [39, 12, 25J. By default, our Transformer pretrained on Open X-Embodiment [37] has 16 blocks and width of 1024, which we refer to as the Base size or -B (~~353M parameters).
Motion Prompt. We apply dropout mechanism randomly omit motion prompts, avoiding the limitation of using a single observation frame and enhancing robustness across both prompted and unprompted conditions. During inference, when the number of observed frames is shorter than the required window size [30], frames are uniformly replicated to satisfy the temporal input constraint. Full training recipe including model architecture and data preprocessing is provided in Appendix A.

---

Table 1: Video prediction results on BAIR and RoboNet datasets. "." indicates values not reported.
We report Fréchet Video Distance (FVD) [47] as the primary metric, complemented by PSNR [27],
SSIM [55], and LPIPS [71] for perceptual quality assessment. LPIPS and SSIM are scaled by 100.
BAIR [13]
FVDJ PSNRT SSIM
LPIPS)
RoboNet [9]
FVD4
PSNRT
SSIMI LPIPS
action-free & 64x64 resolution
actton-condittoned & 64x64
resolation
Mis [19]
93.7
Mis [19]
133.5
23.2
80.5
4.2
FirVid [4]
93.6
FitVid [4]
62.5
28.2
89.3
2.4
MCVD [51]
89.5
16.9
78.0
SVG [50]
123.2
23.9
87.8
6.0
MAGVIT [68]
62.0
19.3
78.7
12.3
GHVAB [57]
95.2
24.7
89.1
3.6
iVideoGPT [60]
75.0
20.4
82.3
9.5
iVideoGPT [60]
63.2
27.8
90.6
4.9
SAMPO
65.7
22.3
86.7
8.4
SAMPO
57.1
29.3
94.1
3.3
action-conditioned & 64x64 resolution
action-condttioned & 256x2.56 resolution
MaskViT [19] 70.5
MaskViT [19] 211.7
20.4
67.1
17.0
VideoGPT [60]
60.8
24.5
90.2
5.0
iVideoGPT [60]
197.9
23.8
80.8
14.7
SAMPO
55.5
26.7
94.7
3.7
SAMPO-L
175.3
25.3
84.7
12.3
T+0
T=2
T-4
T-8
T-10
T=0
T-2
T-4
T-8
T-10
P
(Staerned)
(Ctsevvss
iidooPT
WeleoPP
IX World Medel (act aatToo-conditoone
(action-con) (action-condtionn Ciffon-condittonee
RoboNer
SAMPO
SAMPO
Figure 4: Qualitative comparison of video prediction. We compare SAMPO with iVideoGPT on the 1X World Model and RoboNet datasets under action-conditioned settings. SAMPO shows clear advantage modeling complex backgrounds and capturing dynamic object interactions over time.
4 Experiments
We conduct extensive experiments to validate the effectiveness of SAMPO across multiple settings, including action-free and hetion-conditioned video prediction, visual planning, and model-based reinforcement learning (MBRL). We follow standardized evaluation protocols [60] and report results on established benchmarks. Training details for each benchmark are provided Appendix B.
4.1 Video Prediction Performance
We evaluate on three categories benchmarks: BAIR Robot Pushing [13] for low-resolution video prediction, (ii) RoboNet [9] for large-scale action-conditioned prediction, and (iii) IX World Model [1] for real-world human and robotic interactions diverse indoor environments, designed for open-domain video prediction. We predict 15 future frames from context frame on BAIR, 10 future frames from context frames on RoboNet and IX World Model. Action-conditioned applied during rollouts.
Datasets and Setup.
In Tab. and Tab. SAMPO achieves the best FVD in video prediction. Qualitative comparisons, shown in Fig. 4, demonstrate significant improve- ments in perceptual quality and motion realism. These results highlight the effectiveness of scale- wise generation in modeling multi-scale dynamics and the role of araeectory-aware motion prompts in guiding future predictions.
Table 2: Video prediction results on 1X.
IX
FVD4 PSNR1 SSIMI LPIPS
action-conditioned 256x256 resolution
iVideoGPT
251.8
24.1
78.3
20.3
SAMPO
227.1
25.7
80.3
18.7

---

Table Visual planning performance in VP We report the success rates across tasks, and the average success rate excluding Flat Block. In addition, provide the mean and standard deviation of the success rates (in %) on average in 3 random seeds.
Robosuite Push
Flat
Open Drawer
Open Slide
Blue
Green Button
Red
Upright Block
Avg.
Method/ / Task
Block
Button
Button
Sucvesst
Simulator
93,5
13,3400
76,740
71,7#14
100,0
96,740.
90.0
90,0=00
100.0
FitVid [4]
67.7464
9.242.8
25.348.2
35,3455
94.0+4.2
84,045.5
58,745.
51,3=2.
65.6
SVG' [50]
79.8+3.3
2.0
16.748
$7,3-11.0
97,3427
81.340.5
76.0*9.
48.7
72.5
MCVD [51]
77,3
4.0=1.4
11,7214
18.3#144
95.0+441
83,340.0
73.3
56.7=2.7
64.3
MaskViT [19]
82,6*2.
4.0
4.0
8,726.5
94,7
64.0
24.0
62.2
52,1
Struct-VRNN, [36]
55.4
4,7
2,7
12.7:669
86,71442
68.0
30,714422
33.3
44.2
iVideoGPT [60]
78.3
3,3
37.5
16.1
95.6
82,5
92.2
44.7=1.7
70.1
SAMPO
80,72144
5.5
40.3
18.3253
97_32117
85,315.3
94.722.1
46.122.7
72.2
100
Tootner
Nenx
Homeer
0e.
13
Metolrerid Aggrepoted
43
Success Rote (%)
60
Ensironment Sheps (x110))
Ensinemment Stapp (X10f)
Enviromment Stape (x110")
100
Dee Lact
Hookk N1
-00
Drwopenns)
DrQq
MBPO (Friid))
MaPo (1tideoSP1)
60.
(SAMPO))
43
0.0
0.2
0.4
0.
0.8
100
Normalized Environment Steps
20
Emiirrment Steps (x100)
Ensiromentt Steps (M10f)
Envinonment Strppa (H11*)
Model-based RL with SAMPO. We report the success rate (in %) across tasks, along with the average success rate across all tasks and the 95% confidence interval [3] calculated from random seeds. All models are pre- pre-trained with world models [42].
Figure 5:
4.2 Planning and Reinforcement Learning
Visual Planning with Action-Conditioned Rollouts. As perceptual metrics are not strictly correlated with control performance, we further verify the applicability of SAMPO on vp² [46]. visual planning benchmark. Each environment provides noisy, scripted interaction trajectories. Following [46, 60], train SAMPO on 5k trajectories for Robosuite [76] and 35k for RoboDesk [29], and compare against established baselines.
As shown in Tab. 3, SAMPO achieves the best results in four tasks and the second-best results in the other two tasks. Compared to the baseline [60], the average performance is improved by 2.1%, indicating its capability to achieve high-fidelity perception while excelling in control performance.
Model-Based Reinforcement Learning. Beyond planning, an effective world model must facilitate efficient policy learning through interaction. In this work, we evaluate SAMPO by incorporating it into a model-based policy optimization (MBPO) framework [28], which extends the replay buffer with synthetic rollouts to train an actor-critic RL algorithm, implemented based on DrQ-v2 [66]. Experiments are conducted on six robotic manipulation tasks from the Meta- World benchmark [69].
Fig. 5 shows that SAMPO significantly accelerates policy convergence compared to VVidooGPT while also improving the policy's upper bound. This enhancement primarily driven by the temporal consistency and semantic coherence facilitated by our hybrid framework, in conjunction with the motion prompt. Moreover, SAMPO significantly outperforms the state-of-the-ar MBRL method, DreamerV3 [22], regarding both sample efficiency and success rate.
4.3 Ablation Studies & Model Analysis
We perform ablation studies to isolate the contributions of each component. Results are in Tab. 4.
8

---

Table Abalation of SAMPO on the action-conditioned RoboNet [9] a resolution of 256x256 The first two rows compare the baseline AR model with SAMPO model using a hybrid VAR architecture. Later rows add enhancements to SAMPO, including motion prompt. ID temporal position embedding, and model scaling. "A": improvement over the SAMPO-S model.
Description
Para.
Model
Motion
E.
FVD
SSIMT
A
AR
436M
AR [60]
X
x
197.9
80.8
Hybrid AR
207M
SAMPO-S
X
x
227.4
76.4
(0.00, 0.00)
Motion
232M
SAMPO-S
x
217.8
78.8
(.9.6,
Temp. Emhed.
232M
SAMPO-S
/
193.8
81.5
(433.6, +5.1)
Scale up
353M
SAMPO-B
/
184.1
83.2
(43.3, +6.8)
548M
SAMPO-L
175.3
84.7
(-52.1, +8.3)
Table 5: Performance and speed trade-off. We benchmark FVD, PSNR, average success rate VP and inference speed using one A800 GPU with a batch size of 16 on the BAIR.
Method
Spatial Scales
FVDJ
PSNR1
Avg. Successt
Inference Time.
AR13SM [60]
60.8
24.5
70.1
9.05 s vid.
SAMPO-S
2, 4, 8]
80.7
23.1
72.2
1.61 s / vid
SAMPO-S
[1, 4, 8, 10]
73.1
24.3
71.3
3.27 s vid
SAMPO-S
[1. 2. 3, 5, 6]
55.5
26.7
70.6
2.06 s/ vid.
Figure 6: Zero-shot performance. The results show that SAMPO can effectively generalize without special design and finetuning, underscoring its potential for world models. Zoom in for better view.
Effectiveness of Hybrid Framework. First, we evaluate the effectiveness of the hybrid framework, which combines scale-wise autoregressive generation with bidirectional spatial attention. Starting with the SAMPO-S model, we observe competitive performance in both FVD and SSIM, confirming its ability to generate higher-quality video. The parallel prediction of multiple tokens within each scale effectively reduces autoregressive steps, improving efficiency. In Tab. 5, SAMPO with various scales configurations significantly improve the inference speed. These demonstrate that our hybrid framework not only enhances generation quality but also accelerates inference.
Motion Prompt and Temporal Embedding. Both motion prompts and temporal position embedding improve temporal consistency and semantic coherence, as shown in Tab. 4. Particularly, temporal position embedding plays a erucial role in maintaining consistent temporal dynamics, resulting in significant improvements in FVD and SSIM.
Analysis of Spatial Scale Design. Fewer scales improve inference speed but reduce spatial fidelity. Notably, smaller inter-scale strides yield better performance under similar compute budgets, indicating that residual-bast scale-wise autoregression benefits from gradual resolution refinement due to its sensitivity to abrupt scale transitions.
Scaling Laws. As GPT-style world model, we explore the scaling behavior of SAMPO with different sizes (depth 12, 16, 20) in Tab. 4. The observed improvements in FVD and SSIM align with the scaling laws, as larger models capture more complex temporal and spatial dependencies, improving both generation quality and consistency.
Zero-shot Generalization. We further assess SAMPO's zero-shot generalization by evaluating a model pretrained on the Open X-Embodiment dataset [37] and tested on the IX World Model dataset [1] without fine-tuning. As shown in Fig. 6, SAMPO generates high-quality videos with temporally consistent dynamics, demonstrating its strong generalization ability. More zero-shot visual examples are in Appendix C.

---

Conclusion and Discussion
In this work, we present SAMPO, scale- wise autoregressive world model. It integrates a hybrid autoregressive framework with an asymmetric tokenizer to perform temporal and spatial token maps generated across multiple scales, and further leverages a lightweight motion prompt module to enhance dynamic scene understanding. Owing to its hybrid architecture, SAMPO maintains both temporal consistency and spatial coherence, achieving state-of-the-are performance in video prediction and model-based RL benchmarks. However, despite its strong accuracy and scalability in both simulated and reinforcement learning environments. SAMPO still struggles with long-term modeling, leading to eπror accumulation over time. Future work could explore techniques for improving long-term consistency and reducing error propagation.
Acknowledgement
This work was supported in part by the National Key Research and Development Project under Grant 2024YFB4708 100, National Natural Science Foundation of China under Grants 62088 102, U24A20325 and 12326608, and Key Research and Development Plan of Shaanxi Province under Grant 2024PT-2CK-80.
References
[1] 1X Technologies. 1X World Model Challenge, June 2024. URL https://www. 1x. tech/ discover/Ix-wowt wwwdiix-woooddddddoddeeeee
[2] N. Agarwal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint 00557, 2025.
[3] R. Agarwal, M. Schwarzer, S. Castro, A. C. Courville, and M. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 34:29304-29320, 2021.
[4] M. Babacizadeh, M. T. Saffar, S. Nair, S. Levine, C. Finn, and D. Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.13195, 2021.
[5] G. Bachmann and V. Nagarajan. The pitfalls of next-token prediction. ICML, 2024.
[6] W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1eX.iv:1866.. 261, 2018.
[7] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Tay- lor, T. Luhman, E. Luhman, C. Ng. R. Wang, and A. Ramesh. Video gener- ation models as world simulators, 2024. URL video- dedeo-generttion
wwwraatiiooondnleseeeeeeeeeeeeee wwdems as - simulators.
[8] Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: Generative interactive environments. In ICML, 2024.
[9] S. Dasari, Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, Singh, Levine, and
C. Finn. Robonet: Large-scale multi-robot learning. CoRL, 2019.
[10] H. Deng, T. Pan, H. Diao, 7. Luo, Y. Cui, H. Lu, S. Shan, Y. Qi, and X. Wang. Autoregressive video generation without vector quantization. ICLR, 2025.
[11] J. Ding, Y. Zhang, Y. Shang, Y. Zhang, Z. Zong, J. Feng, Y. Yuan, H. Su, N. Li, N. Sukiennik, al. Understanding world or predicting future? comprehensive survey of world models. arXiv preprint arXiv:241 1. 14499, 2024.
[12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
10

---

[13] F. Ebert, Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. CoRL, 12(16):23, 2017.
[14] P. Esser, R. Rombach, and B. Ommer. Taming transformers for high-resolution image synthesis. 2021.
In CVPR, pages
[15] S. Ge, Y. Zhang, Liu, M. Zhang. J. Han, and Gao. Model tells you what to discard: Adaptive kv cache compression for llms. ICLR, 2024.
[16] D. Geng, C. Herrmann, J. Hur, F. Cole, S. Zhang, T. Pfaff, T. Sopez-Guevard, C. Doersch, Y. Aytar, M. Rubinstein, et al. Motion prompting: Controlling video generation with motion trajectories. CVPR, 2025.
[17] Z. Gong, Ding, S. Lyu, Huang, M. Sun, W. Zhao. Z. Fan, and D. Wang. Carp: Visuomotor policy learning via coarse-to- to-nie autoregressive prediction. arXiv preprint arXiv:2412.06782, 2024.
[18] H. Guo, Y. Li, T. Zhang, Wang. T. Dai, S.-T. Xia, and L. Benini. Fastvar: Linear visual autoregressive modeling via cached token pruning. arXiv preprint arXiv:2503.2336) 2025.
[19] A. Gupta, S. Tian, Y. Zhang, J. Wu. R. Martin-Marthnn and L. Fei-Fei. Maskvit: Masked visual pre-training for video prediction. ICLR, 2023.
[20] D. Ha and J. Schmidhuber. World models. NeurIPS, 2018.
[21] D. Hafner, T. Lillicrap. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. ICLR, 2020.
[22] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models, 2023. URL https://arxiv. org/abs/2301.04/
2023.
[23] J. Han, J. Liu, Y. Jiang. B. Yan, Y. Zhang. Z. Yuan, B. Peng, and X. Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.04431, 2024.
[24] H. He. Y. Zhang. L. Lin, Z. Xu, and L. Pan. Pre-trained video generative models world simulators. arXiv preprint arXiv:2502.07825, 2025.
[25] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[26] J. Ho. T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. Fleet. Video diffusion models. NeurIPS, 35:8633-8644 2022.
[27] Q. Huynh-Thu and M. Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, 44(13):800-811, 2008.
[28] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. NeurIPS, 32, 2019.
[29] H. Kannan, D. Hafner, C. Finn, and D. Erhan. Robodesk: A multi-task reinforcement learning benchmark, 2021. URL https::/ggitiii thub. com/ gowgge resereerrreerrcccrroooeeeee.
[30] N. Karaev, I. Makarov, J. Wang, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker3: Sim- pler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.11881, 2024.
[31] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: It is better to track together. In ECCV, pages 18-35. Springer, 2024.
[32] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M.-C. Chiu, et al. Videopoet: large language model for zero-shot video generation. ICML, 2024.
11

---

[33] Y. LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-62, 2022.
[34] J. Liang, Y. Fan, K. Zhang, R. Timofte, L. Van Gool, and R. Ranjan. Movideo: Motion-aware video generation with diffusion model. In ECCV, pages 56-74. Springer. 2024.
[35] H. Ma, J. Wu, N. Feng, C. Xiao, D. Li, J. Hao, J. Wang, and M. Long. Harmonydream: Task harmonization inside world models. ICML, 2024.
[36] M. Minderer, C. Sun, R. Villegas, F. Cole, K. P. Murphy, and H. Lee. Unsupervised learning of object structure and dynamics from videos. NeurIPS, 32, 2019.
[37] A. O'Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, Gupta,
A. Mandlekar, A. Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration O. ICRA, pages 6892-6903. IEEE, 2024.
[38] J. Parker-Holder, P. Ball, J. Bruce, V. Dasagi, K. Holsheimer, C. Kaplanis, A. Moufarek,
G. Scully, J. Shar, J. Shi, Spencer, Yung, M. Dennis, S. Kenjeyev, S. Long, V. Mnih, H. Chan, M. Gazeau, B. Li, F. Pardo, L. Wang. L. Zhang, F. Besse. T. Harley, A. Mitenkova, Wang, Clune, D. Hassabis, R. Hadsell, Bolton, S. Singh, and T. Rocktäschel. Genie A large-scale foundation world model, 2024. URL https: http:///eeemdddooggggo////aaaooe google//iscooerr/////////ooo blog/ ter//gnie-225 2- a- wwr-llrrggsaalle foundattoooniiooon. -ow.daatioo---ooo--ooo-oooe
[39] W. Peebles and S. Xie. Scalable diffusion models with transformers. In ICCV, pages 2023.
[40] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[41] D. Rohatgi, A. Block, A. Huang, A. Krishnamurthy, and D. J. Foster. Computational-sttinn -taatissial tradeoffs at the next-token prediction barrier: Autoregressive and imitation learning under misspecification. arXiv preprint 22XXv:25002.14665, 2025.
[42] Y. Seo, K. Lee, S. L. James, and P. Abbeel. Reinforcement learning with action-free pre-training from videos. In ICML, pages 19561-19579. PMLR, 2022.
[43] A. Shtedritski, C. Rupprecht, and A. Vedaldi. What does clip know about red circle? visual prompt engineering for vims. ICCV, 2023.
[44] J. Tian, L. Wang, S. Zhou, S. Wang, J. Li, H. Sun, and W. Tang. Pdfactor: Learning tri- perspective view policy diffusion field for multi-task robotic manipulation. In CVPR, pages 15757-15767, 2025.
[45] K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NeurlPS, 37:84839-84865, 2024.
[46] S. Tian, C. Finn, and J. Wu. A control-centric benchmark for video prediction. ICLR, 2023.
[47] Unterthiner, S. Van Steenkiste, Kurach, R. Marinier, M. Michalski, and Gelly. To- wards accurate generative models of video: A new metric & challenges. arXiv preprint 1110v:1812.01777,, 2018
[48] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. NeurIPS, 30, 2017.
[49] M. Vecerik, C. Doersch, Y. Yang, T. Davchev, Y. Aytar, G. Zhou, R. Hadsell, L. Agapito, and J. Scholz. Robotap: Tracking arbitrary points for few-shot visual imitation. In ICRA, pages 5397-5403. IEEE, 2024.
[50] R. Villegas, A. Pathak, H. Kannan, D. Erhan, Q. V. Le, and H. Lee, High fidelity video prediction with large stochastic recurrent neural networks. NeurIPS, 32, 2019.
[51] V. Voleti, A. Jolicoeur-Martinnn and C. Pal. Mevd-masked conditional video diffusion for prediction, generation, and interpolation. NeurIPS, 15:23371-23385 2022.
12

---

[52] B. Wang, X. Meng, X. Wang. Z. Zhu, A. Ye, Y. Wang, Z. Yang. Ni, G. Huang, and X. Wang. Embodieedreamer Advancing real2sim2real transfer for policy training via embodied world modeling. arXiv preprint ar arXiv: .51980 2025.
[53] S. Wang, L. Wang, S. Zhou, J. Tian, J. Li, H. Sun, and W. Tang. Flowram: Grounding flow matching policy with region-aware mamba framework for robotic manipulation. In CVPR, pages 121 2217-12186, 2025.
[54] X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang. Z. Li, Q. Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024.
[55] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600-6) 61166000-126.600 2004.
[56] Z. Weng, X. Yang, Z. Xing, Z. Wu, and Y.-G. Jiang. Genrec: Unifying video generation and recognition with diffusion models. NeurlPS, 2024.
[57] B. Wu, S. Nair, R. Martin,Martin, L. Fei-Fei, and C. Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In CVPR, pages 2318-2328, 2021.
[58] J. Wu, H. Ma, C. Deng, and M. Long. Pre-training contextualized world models with in-the-wild videos for reinforcement learning. NeurlPS, 36:39719 19-39499,, 2023.
[59] J. Wu, X. Li, Y. Zeng. J. Zhang, Q. Zhou, Y. Li, Y. Tong, and K. Chen. Motionbooth: Motion- aware customized text-to-video generation. NeurIPS, 2024.
[60] Wu, Yin, N. Feng, X. He, D. Li, J. Hao, and M. Long. ivideogpt: Interactive videogpts are scalable world models. In NeurIPS, 2024.
[61] M. Wu, X. Cai, J. Ji, J. Li, O. Huang, G. Luo, H. Fei, G. Jiang, X. Sun, and R. Ji. ControlmlIm: Training free visual prompt learning for multimodal large language models. NeurIPS, 37: 45206 45234,, 2024.
[62] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157, 2021.
[63] Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441. 2023.
[64] L. Yang, Y. Wang, X. Li, X. Wang, and J. Yang. Fine-grained visual prompting. NeurlPS, 36: 2023.
[65] M. Yang, Y. Du, K. Ghasemipour, J. Tompson, D. Schuurmans, and P. Abbeel. Learning interactive real-world simulators. ICLR, 1(2):6, 2023.
[66] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved daga-nugeented reinforcement learning. ICLR, 2022.
[67] H. You, H. Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-F. Chang, and Y. Yang. Ferret: Refer and ground anything anywhere any granularity. arXiv preprint arXiv:2310.07704, 2023.
[68] L. Yu, Y. Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Hao, Essa, al. Magvit: Masked generative video transformer. In CVPR, pages 10459- 10469, 2023.
[69] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, pages 1094-1100 PMLR, 2020.
[70] J. Zhang, F. Xiong, and M. Xu. 3d representation in 512-byte: Variational tokenizer is the key for autoregressive 3d generation. arXiv preprint arXiv:2412.02200 2024.
[71] R. Zhang, P. Isola, A. A. Efros, F. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, pages 586-595, 2018.
13

---

[72] G. Zhao, Ni, X. Wang, Z. Zhu, X. Zhang, Y. Wang, G. Huang, X. Chen, B. Wang, Y. Zhang, al. Drivedreamer44: World models are effective data machines for 4d driving scene represen- tation. In CVPR, pages 12015-12026, 2025.
[73] G. Zhao, X. Wang, C. Ni, Z. Zhu, W. Qin, G. Huang, and X. Wang. Recondreamer++: Harmonizing generative and reconstructive models for driving scene representation. arXiv preprint arXiv:2503. 18438, 2025.
[74] R. Zheng, Y. Liang, S. Huang, J. Gao, H. Daumé III, A. Kolobov, F. Huang, and J. Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. ICLR, 2025.
[75] D. Zhou, Q. Sun, Y. Peng, Yan, R. Dong, D. Wang. Ge, N. Duan, X. Zhang, L. M. Ni, et al. Taming teacher forcing for masked autoregressive video generation. CVPR, 2025.
[76] Y. Zhu, J. Wong. A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, Y. Zhu, and K. Lin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[77] Z. Zhu, X. Wang, W. Zhao, C. Min, N. Deng. M. Dou, Y. Wang, B. Shi, K. Wang, C. Zhang, et al. Is sora a world simulator? a comprehensive survey on general world models and beyond. arXiv preprint wrXiv:2405.03522 2024.
[78] X. Zhuang, Y. Xie, Y. Deng, L. Liang, Ru, Y. Yin, and Y. Zou. Vargpt: Unified understanding and generation in a visual autoregressive multimodal large language model. arXiv preprint arXiv:2501.12327 2025.
[79] X. Zhuang, Y. Xie. Y. Deng. D. Yang. Liang. Ru. Y. Yin, and Zou. Vargpt-v1. 1: Improve visual autoregressive large unified model via iterative instruction tuning and reinforcement learning. arXiv preprint eXiv:2504.00944 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately reflect the contributions of SAMPO, including the novel framework and improvements in video prediction and model- RL, which are sulp-supported by the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses limitations, including the inability of the motion prompt method to handle single-fran frame inputs, as detailed in Appendix D. This limitation stems from the requirement of multiple frames to extract motion trajectories.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: While the paper does not include formal theoretical results, the experimental results provide strong empirical support for the model design. The effectiveness of SAMPO, especially in terms of video prediction and model-based reinforcement learning, validates the proposed framework through extensive experimentation.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides a comprchensive description of the experimental setup and results, including model architecture, training datasets, and hyperparameters, ensuring that the experiments reproducible. Detailed information is available in Appendix A and B for both the model and experimental configurations.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper provides open access the code necessary reproduce the experimental results, as described in the supplemental material. While the data is not included, the code ensures that the results can be replicated.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
serrides//oodSuumiisee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips. wttpcccppppppppeeeeeceeccccceeet /wwc///gg///////eee/eeeee httgg/ddd////dduubonnnn /et////ooduasssseeeeeeeeee wwwrbbiisssooooooiiioico
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The paper specifies all necessary training and test details, including data splits, hyperparameters, and optimizer settings, in Appendix A.I, ensuring the experiments can be reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
17

---

The paper reports error bars and statistical significance in Tab. 3, with success rates including mean and standard deviation values, and explains the method for calculating them in the experimental setup section.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides detailed information about the compute resources in Tab. C, including the type of compute workers (A800 GPUs), memory requirements, and execution times, ensuring transparency in resource usage.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https: ttttus:///nerips. cccpublic//eiiccccccccccec wtwuiieeieeeennnnnn
didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The research follows the NeurIPS Code of Ethics, as it does not involve human subjects or sensitive data, and no ethical concerns deviations from the guidelines identified.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeetm of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our models are trained solely on publicly available robot datasets, eliminating the risk of misuse, such as deepfakes involving significant events or figures.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
19

---

The paper credits the creators and original owners of the datasets and models, explicitly mentioning the relevant licenses and terms of use, and cites the original sources to ensure compliance.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets such as datasets, code, or models that require documentation. The focus is on using existing models and datasets, and no new assets are released as part of this research.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsoureing or research with human subjects. The experiments are conducted using existing datasets and models, with no participation from human subjects or crowdsourced data collection.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve any research with human subjects or crowdsourcing. so IRB approval or equivalent review is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
The research does not involve Large Language Models (LLMs) as a core component, focusing instead on enhancing world models and reinforcement learning, which do not depend on LLMs.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https httpa//://euuuuueeeeeeeee hatoge///earripssccoooffeereeeee httprriip....cooo..eeeeee/////// for what should or should not be described.
21

---

A Implementation Details
A.1 Details of Network
Table A: Hyperparameterss of Asymmetric Tokenizer.
Tokenizer
Low-resolution
High-resolution
Parameters
114M
310M
Resolution
256
Obs. scale
11,2, 5, 6, 10, 13, 16]
[1, 2, 3, 5,
5, 6, 8, 10, 13, 16]
Fut. scale
[1.2, 3,4.
[1. 2, 3, 4. 10]
Embedding dim
64
64
Codebook size
8192
8192
Norm
GroupNorm
GroupNorm
Norm group
32
32
Activation
SiLU
SiLU
Table B: Hyperparameters of Transformer.
Transformer
SAMPO-S
SAMPO-B
SAMPO-L
Parameters
207M
328M
523M
Layers
12
16
20
Heads
12
16
Hidden dim
768
1024
Feedforward dim
768
1024
1024
Dropout
0.05
0.067
0.083
Norm
LayerNorm GELU
LayerNorm GELU
LayerNorm
Activation
GELU
Table Hyperparameters for Training.
Low-resolution
High-resolution 2nw.utttiooonneeaaaee 22655226)
Conlig
Pre Preit [37]
BAIR [13]
RoboNet (9]
VP2 [46]
Pre-train [37]
RoboNet (9]
WM (1]
GPU Device
8 A800
GPU days
24/24
3/2
10/ 12
4/3
20/ 11
10/27
10 27
Training iteration
IM IM
0.2M/ 0.1M
0.6M / 0.6M
0.2M/ 0.2M
0.3M 1.3M/0.AC 0.44M
0.15M 0.5M
0.15M / 0,55M
Batch size
128 128
128 128
128 128
64 16
32/32
32/
32 32
Sequence length
16/16
16/ 16
12/12
12/12
16/ 16
12/12
12 12
Context frames
2/2
1/1
2/2
2/2
2/2
2/2
2/2
Future frames
6/-
7/-
6/-
6/-
6/-
6/-
6/-
Learning
5/1 1c
1/116-1
1/116-
1/116-f
5/11e-
1/11e-
1/116-1
LR Schedule
Cosine
Weight decay
Grad clip
Warmup steps
Optimizer
AdamW
Gradient
(0.9, 0.999)
Weight decay
Mixed precision
Motion prompt dropout ratio
0.5
Sampling top-k Sampling top-p
100
1.0
Tokenizer. In the proposed asymmetric multi-scale tokenizer, we adopt a hierarchical design to balance spatial detail preservation and computational efficiency [45]. As detailed in Tab. A, for observed frames (L To), dense tokenization applied across all spatial scales Lfuit = 10. For future frames (6 To). sparse subset of coarser scales L = 6 is used, focusing dynamic regions while minimizing redundancy [18]. Both observed and future frames use codebook size of 8192 with embedding dimension of 64.

---

Transformer: The SAMPO transformer utilizes scalable decoder-only architecture, inspired by GPT-2, efficiently model spatiotemporal dynamics. Unlike iVideoGPT [60], which relies on specialized tokens for frame segmentation, we initialize each frame with a single start token [S]. This token serves a dual function: marks the beginning of intra-frame autoregressive generation and naturally defines the temporal boundaries between consecutive frames. Fixed 1D sine-cosine positional embeddings are applied to encode temporal dynamics, following standard practices in vision Transformers [39, 12, 25, 44]. We design set of models with different sizes, as illustrated in Tab. B. Model scaling adheres to linear relationship between depth d, width w, head count h, and dropout rate dr:
w = 64d,
d,
dr = 0.1 d/24.
A.2 Details of Training
Training setup. Tab. C summarizes the hyperparameters used SAMPO across datasets. Training proceeds via uniform segment sampling with dataset-specific step sizes (see Tab. D), where step lengths are tuned to match each dataset's native temporal frequency. For tokenizer training, we use the initial observed frames context, while the transformer trained full-length sequences.
Scale-specific weight. The scale-specific weight À Eq. (6) balances the contribution of each spatial scale to the total loss. Given a multi-scale tokenizer with patch sizes Lpatch = [1, .... Lk]. where = Len(Lpatch) is the number of spatial scales, A is defined as:
L Ek=1 1
K.
(8)
where L, denotes the spatial dimension of patches at scale I, and L3 denotes the number of tokens per scale (e.g., La x Lı patch grid contains L tokens). The denominator >k=1 L normalizes the token counts across all scales, ensuring that finer-grained resolutions (larger Lạ) are not overshadowed by coarser ones (smaller L2). This ensures the model prioritizes dynamic regions over static ones, which are more likely to change across frames. Such weighted strategy is essential for capturing the underlying spatial patterns in robot-centric world modeling, leading to more effective learning.
Algorithm 1X Dataset Preprocessing Pipeline
Require: Dataset root paths Dusin D Target frame rate furget. Minimum segment length Tmin 51, Clip length Teip
30
Require: Joint index mapping
(....., ....,,24 - CHIP_YAW, Ynw....,Angula Velocity)
1: repeat
2:
M,S.R, RR Dimin
> Load metadata, segment indices, robot states, and video frames Temporal downsampling
3: S, R. Downsample(S, V,
30
Junn
4:
U UnniuuSeegmens(())
Extruct unique action segments
5:
for all l do
6:
Tstart; Tend FindBounds(S s)
Segment boundary detection
7:
Tend Tstan < Tiin then continue
8:
Skip short segments
9:
end
10:
V. Virsun Fendl R, +- R R[Tsun Tendl
Sequence cropping Create temporal windows
11:
W < SlidingWindow(12> Telp))
12:
for all E W do
13:
V. <- V. |wo + Tetipl Run + Rolw 1 + Taip]
b Windowed subsequences
14:
if Mr < 15 then
15:
continue
b Skip ultra-short clips
end if
17:
SaveNPZ(V u. Rws "train")
Compressed storage
end for
19:
end for Validation (Daa))
Symmetric validation processing
21:
until Dataset processed

---

Table D: Detailed Dataset Mixture. We include the detailed number of trajectories and the number of dataset sampling weight in the pretraining mixture. These include 41 dataset from Open X- Embodiment [37].
Dataset
Traj.
Step size
Sampling weight
Pretrain
Kuka
580392
3
8.33%
Language Table
442226
8.33%
Fractal (RT-1)
87212
1
8.33%
RoboNet
82649
1
8.33%
BC-Z
43264
8.33%
Bridge
28935
2
8.33%
Droid
29437
10
8.33%
Agent Aware Affordances
24000
66.6
8.33%
ManiSkill Dataset
21346
20
8.33%
Robo Set
15603
5
7,5%
Functional Manipulation Benchmark
15350
10
7.5%
Isaac Arnold Image
3214
0.3125% 15
Stanford MaskViT
9200
0.3125%
UIUC D3Field
768
0.3125% 5
Taco Play
3603
0.3125%
Jaco Play
1085
0.3125%
Roboturk
1995
0.3125% 7
Viola
150
0.3125%
Toto
1003
0.3125% 10
Columbia Cairlah Pusht Real
136
0.3125% 7
Stanford Kuka Multimodal Dataset
3000
0.3125%
Stanford Hydra Dataset
570
0.3125%
Austin Buds Dataset
50
0.3125% 1
NYU Franka Play Dutaset
456
0,3125%
Furniture Bench Dataset
5100
0.3125%
UCSD Kitchen Dataset
150
0.3125%
UCSD Pick and Place Dutaset
1355
0,3125%
Austin Sailor Datasct
240
0.3125% 3
UTookoo PR2 Tabletop Manipulation
240
0.3125%
UTokyo Xarm Pick and Place
102
0.3125% 3
UTokyo Xarm Bimanual
70
0,3125%
KAIST Nomprehensile
201
0.3125% 3
DLR SARA Pour
100
0.3125% 3
Grid
107
0,3125%
DLR EDAN Shared Control
104
0.3125% 3
ASU Table Top
110
0.3125% 4
UTAustin Mutex
1500
0.3125%
Berkeley Fanue Manipulation
415
0.3125% 3
CMU Playing with Food
174
0.3125%
CMU Play Fusion
576
0.3125%
CMU Stretch
135
0.3125% 10
USC Cloth Sim
684
0.3125
Mimic Play
323
10
0.3125%
Total
1,407,330
100.0%
A.3
Details of Data
Real Robot Dataset. total, we use a subset 41 datasets the Open X-Embodiment dataset [37], as shown in Tab. D. The datasets used in this work consist of both real-world data and simulator- generated data, providing a rich and diverse foundation for action-conditioned video prediction and model-based reinforcement learning.
Preporcess on 1X World Model Dataset. The World Model Dataset is a large-scale multimodal dataset for training and evaluating robotic world models in dynamic environments. It includes sensory data from IX Technologies' EVE humanoid robots performing tasks like door opening, cloth folding, and obstacle navigation. The dataset contains synchronized 512x512 RGB video frames,

---

25- 25-dimensional proprioceptive state vectors, and metadata, with state vectors capturing joint angles, gripper openness, and velocities. Video frames are stored MP4 format, with segmentation and configuration details provided in binary segment indices and JSON files.?
Preprocessing follows custom pipelines developed to align with iVideoGPT [60] evaluation protocols, as detailed in Algorithm 1, including parsing raw files, frame-state alignment, and task-specific normalization. This dataset facilitates standardized evaluation of temporal dependency learning, multi-modal integration, and real-world generalization for robotics."
B Benchmark Setup
Our experiments follow the same evaluation protocols as iVideoGPT [60]. For completeness, we provide a brief introduction to the following three experiments:
Video Prediction. We evaluate SAMPO using four metrics: SSIM [55], PSNR [27], LPIPS [71], and FVD [47]. In line with previous work [62, 60, 4, 51, 19), we address the stochasticity of video prediction by sampling 100 future trajectories for each test video and selecting the best one for PSNR, SSIM, and LPIPS. All 100 samples are used for FVD evaluation.
Visual Planning. We finetuning the pretrained SAMPO on VP2 datasets and integrating it into interface compatible with the official VP2 visual planning code. For Robosuite tasks, a trajectory is deemed successful if the reward, which reflects the distance the goal, falls below 0.05. In contrast, for Robodesk tasks, success is defined by a reward value below -0.5, with the environment returning either or -1, where -1 signifies success.
Visual Model-based RL. For Model Rollout, the initial rollout batch size set to 640, with an interval of 200 environment steps, batch size of 32, and horizon of 10. For Model Training, the initial training steps are 1000. The tokenizer training interval is 40 environment steps, while the transformer training interval is 10 environment steps. The batch size is 16, with a sequence length of 12, context frames set to 2, and 5 sampled future frames for tokenization. The learning rate le -4 with weight decay, and the optimizer used is Adam. The model-based RL real data ratio is 0.5.
C Additional Visualization
In this section, we present additional qualitative results of SAMPO across various datasets to complement the main text. We showcase video prediction results in Fig. 7, 8, 9, 10, 11, 12; zero-shot performance in Fig. 13; the illustration of motion prompt in Fig. 14; and the visual comparison with the iVideoGPT [60] in Fig. 15.
BAIR Robot Pushing
T:D
T:2
(Obderwed)
Predicted
Preficte
Figure 7: Additional visualization on the BAT RRobot Pushing dataset for action-conditioned video generation in low resolution (64 64).
2 Dataset and code are available at: soddelrrwwddate under cc-by-nc-sa-4.0 license and https://github. com/ www.teeeoooooooooogggeeeee un- der Apache-2.0 license
For further details, see:
1x. www.:..w..tccccc.teeeceeeeeeeeee http://w...te///eeooeee/eeo/aaaa *Code is available https://github. com/s-tian//pp2

---

Open X- Embodiment
T:0
T-l
T-2
T-3
T-4
T:5
T:6
T:7
T:8
T:9
T:30
T:0
T-12
T-13
T-14
T-19
(Observed)
Predieted
T:30
T:11
T:12
T:13
T:14
T:15
(Observed)
Predicted
(Dlserved)
Predicted
(Olsorved))
Predicted
Figure 8: Additional visualization on the Open X-Embodiment dataset for action free pretraining in low resolution (64 x 64).
RoboNet
(Observed)
Predicted
T:5
T:7
T:9
T:10
(Observed)
Predicted
Figure 9: Additional visualization on the RoboNet dataset for action action-condinn conditiood video generation high resolution (256 256).
1X World Model Trl
THD
T+2
T-d
7
T:0
THE
T57
T:a
TEP
T:10
T:ll
(Gbserved) Predicted
T:5
T:5
T:30
(Observed)
Predicted
Figure 10: Additional visualization on the 1X world model dataset for action-conditioned video generation in high resolution (256 x 256).
5

---

vp² RoboDesk
T:0
T=1
T:3
T:4
T
T:6
T=8
T:9
T-10
T-11
(bbervvdd)
Predicted
T=O
T=1
T:3
T:4
T
T:6
T=8
T:9
T-10
T-11
(Ooserved)
Predicted
Vp RoboSuite
T=O
T=l
T=2
T=3
T=4
T=5
T=6
T=7
T=8
T=9
T=10
T=11
(Observe4)
Predicted
T=O
T=1
T:3
T:4
T:6
T=8
T:9
T:10
T:11
(Observed)
Predicted
Figure 11: Additional visualization on the 2P2 benchmark for action-conditioned video generation in low resolution (64 x 64).
Meta World
T=O
T=1
T=2
T=3
T:4
T=5
T=6
T=8
T=9
T=10
T=11
8.08
0:11
0:16
0125
0,25
3
0,45
0,56
22:03
23.55
(Observed)
T=0
T=1
T=2
T:3
T=5
T=6
T=7
T=8
T:9
T=10
T=11
(Observed)
2221
3.47
10:00
4.01
1357
25.20
25.04
30,00
30,00
Predicted
T:O
T=1
T:2
T:3
T=4
T:5
T=6
T=7
T:8
T-9
T=10
T-11
0.04
0.26
0.04
0,08
0,07
0.20
10:11
10:15
(Ooserved)
0,05
D
0
10.12
16400
Predicted
Figure 12: Additional visualization on Meta- World tasks for natiionconditioned video generation low resolution. Each row corresponds different task: Coffee Push, Plate Slide, Handle Pull Side. Both actual reward and predicted reward are shown in the upper left corner.

---

Zero-Shot
T:9
(Observed)
Predicted
T:9
(Observed)
Tub
7-4
1-5
A
(Observed)
Predicted
6
6
64
60
(Observed)
Figure 13: Additional visualization of zero-shot performance for action-free video generation in high resolution. The first two rows show zero-shot results the RoboNet dataset, while the last two rows illustrate performance on the IX dataset. Pretraining was conducted on the Open X-E X-Embodiment dataset. The results in this figure supplement Fig. 6 in the main paper.
Motion prompts
Figure 14: Additional visualization of motion prompts at varying resolution. Motion prompt
examples are visualized on the RoboNet, 1X, BAIR, and Vp2 datasets. The image in the bottom right corner shown at resolution of 64 64. For high-resolution images (256 x 256), the grid size is set to 16, while for low-resolution images (64 x 64), the grid size is set to 12, supplementing the content of Sec. 3.3 the main paper. Zoom in for better view

---

RoboNet
T=0
T=1
T=2
T=3
T=5
T=6
T=7
T=8
T=9
T=10
T=11
(Observed)
T=0
T=1
T=22
T=33
T=4
T=5
T-6
T=7
T-8
T=9
T=10
T=11
(Observed)
Mideo6PT
SAMPO
vp2 RoboDesk
T=O
T=4
T=6
T=11
(Obsrvved)
Nideo6PT
SAMPO
vp² RoboSuite
T=0
T-1
T-2
T-3
T:4
T=5
T=6
T=7
T=8
T=9
T=10
T=11
(Observed)
Figure 15: Additional visual comparison with iVideoGPT for condin-ondi video generation on RoboNet (256 x 256) and VP2 (64 x 64) datasets. The first column shows the ground truth (GT), the second column displays the predictions made by iVideoGPT, and the final column presents our predictions. As shown, our results not only effectively maintain spatial coherence in visually cluttered environments, but also better align with the ground truth motion trajectories, indicating better performance in capturing scene dynamic behaviors. This figure supplements Fig. in the main paper, providing further evidence of our model's s enhanced accuracy in trajectory prediction.

---

D Limitations and Future Work
Single-frame Input Limitation. The current motion prompt method relies on multi-frame obser- vation sequences to extract motion trajectories (e.g., point tracking for t = to T frames using CoTracker3 as described in Sec. 3.3). However, when the input consists of only a single frame (To = 1), it is impossible to generate effective motion prompts, as trajectory extraction requires at least two frames to compute the displacement.
This limitation essentially stems from a fundamental issue in the setting: for a world model, a single frame only provides "starting point' for static scene, whereas the diversity of dynamic interactions (e.g., a robotic arm that may move in different directions) results in a multimodal distribution of future states. Consequently, the model needs to rely on random sampling or implicit prior generation of possible results.
To address this, potential improvements could involve the design of an implicit dynamic prior based on a single frame (e.g. generating -raje-ttajectoriis via geometric constraints or a physics engine), or introducing stochasticity control mechanism that balances generation diversity with physical plausibility.
Motion Prompt as Way of Incorporating Historical Information. In Sec. 3.1 of the paper, the world model is formalized as a partially observable Markov decision process (POMDP), where the integration of historical information plays crucial role in enhancing the model's ability to predict future states and develop more meaningful strategies.
In Sec.3.3, the trajectory-nne aaaare aware motion prompt is introduced as external module designed to inject historical information into the hybrid autoregressive framework. While this approach has been shown to improve performance through ablation studies (Tab.4). it currently serves as an intuitive, but not necessarily the most optimal, solution for incorporating historical data.
Future work could explore more refined alternatives, such as:
1. Implicit Dynamic Modeling: Directly learning spatiotemporal saliency via attention mechanisms without explicit trajectory extraction (e.g., combining neural differential equations to model continuous motion fields);
2.
End-to-End Motion Guidance: Integrating trajectory prediction heads into the backbone network for joint optimization of both motion prompts and frames generation;
E Broader Impact
Broader Impact of SAMPO Model. The research presented in this paper introduces SAMPO, a model that enhances both the quality of generated content and the speed of inference, offering significant advantages over traditional autoregressive models. By mitigating issues such as object disappearance and inaccurate predictions commonly encountered in previous models, SAMPO ensures higher-quality generation. Moreover, its fast inference capabilities enable real-time decision- making and dynamic environment interactions, making it promising solution for applications in robotic control, video prediction, and model-based reinforcement learning.
Potential Negative Societal Impacts. However, as with any powerful generative model, there are potential negative societal impacts. One concern is the misuse of the technology in creating deepfakes or generating realistic video content for disinformation, surveillance, or manipulation purposes. While SAMPO's primary application is in improving robotic systems, underlying techniques could be applied in harmful ways if left unchecked.
Mitigation Strategies for Responsible Use. To mitigate these risks, it is essential to consider safeguards such as restricted access to the model, robust detection systems for misuse, and ethical guidelines for deployment. Additionally, ensuring transparency in the development and application of such models and creating oversight mechanisms will help prevent unintended societal consequences. Future work should also explore the potential for bias in the model's predictions, ensuring fairness and ethical deployment across diverse groups and settings.


Reviews:
[{"review_id": "6W1f1xHdQs", "reviewer": "Reviewer_cg6d", "summary": "This paper introduces SAMPO, an intuitive way to improve robotic world models with a hybrid multi-scale prediction framework. The hybrid decoding choices help with spatial consistency via VAR-like methods and ensure temporal consistency via autoregressive prediction. The paper also describe and propose solution to several practical difficulties, such as asymmetric tokenizer to handle different kinds of video frames efficiently.\n\nResults have shown that this method is efficient and effective compared to several strong baselines.", "questions": "- We understand that scaling such system requires both cost of training and cost of obtaining data. But I would still be interested to see if the authors have ideas on how to further scale up the model. And besides these costs, are there other considerations for scaling such model?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nGenerally, this paper presents a method that naturally extend prior work and is executed well.\n\n- The method is intuitive and builds on the past work smoothly. SAMPO's integrated design, particularly its hybrid multi-scale approach appears to effectively balance the need for both spatial detail and smooth temporal transitions.\n- The paper is well-written and progressively describes a complex system, which involves several components and several method improvements.\n\n\nWeakness:\n\n- SAMPO involves several sophisticated modules and have quite some inherent complexity. This may make future improvements on the base more difficult.\n- SAMPO utilizes a sparse assumption. The asymmetric tokenizer's approach of sparsely tokenizing future frames is effective for efficiency, but it would be valuable to discuss how well (or limited) if this sparse representation scales to capture details in highly complex real-world dynamics.", "was_revised": true, "final_justification": "My main concern of the paper is initially the sparsity assumption, which is valid in this setting as the author argues. I still have a little doubt on whether this method can generalize on more complex scenarios, but I would like to keep my positive reviews since I would like to keep the discussion in the scope and settings proposed by the authors, especially when it is a valid/common setting.", "rebuttal": "We sincerely thank Reviewer cg6d for providing a detailed review, valuable suggestions, and a positive evaluation of our work.\n\n---\n>## Q1: Inherent Complexity\n\nWe would like to clarify that SAMPO is intentionally designed to be modular and extensible, serving as a **flexible framework** rather than a rigid system.\n\nThe **core of SAMPO lies in its hybrid autoregressive framework**, which unifies spatial and temporal modeling via scale-wise generation and causal decoding. This design directly mitigates the spatial degradation and temporal drift seen in earlier AR world models.\nIn fact, several components of SAMPO, such as the tokenizer and the prompt module, can be replaced or extended. For instance, the asymmetric tokenizer can be substituted with 3D VQ-VAE, MaskViT-style tokenization, or ViT-VQ, while the motion prompt can incorporate trajectory embeddings or CLIP-style visual goals. Additionally, the causal modeling can be upgraded with diffusion-style decoders to improve sample diversity without compromising consistency [1].\n\nAs evidenced in Tab. 4, these **loosely coupled, problem-driven modules** make SAMPO a flexible foundation that readily absorbs advances from diverse generative paradigms. Consequently, SAMPO remains compatible with emerging world-modeling approaches while providing a unified structure for systematic comparison and improvement.\n\n[1] *Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. ArXiv 2025.*\n\n>## Q2: Sparse Assumption\n\nOur asymmetric tokenization design is based on the **nature of our data**: large-scale robotic manipulation datasets (e.g., Open X-Embodiment), where most scenes are tabletop, with fixed camera views and limited motion from the robot arm and manipulated objects. In such settings, the assumption of largely static backgrounds is valid, and sparse tokenization allows the model to focus on dynamic regions without wasting computation on redundant, static areas.\n\nEven in highly dynamic scenes, **maintaining sparsity remains essential for ensuring computational tractability**. Full-resolution tokenization over multiple future frames quickly becomes infeasible in terms of memory and latency.\nOur method provides a flexible framework for dynamic environments by focusing tokens on motion-related regions. This approach is aligned with recent findings in video generation, such as [2], which demonstrate that spatial-temporal sparsity can retain high visual fidelity even in complex scenarios.\nMoreover, our design is compatible with adaptive refinement strategies. For instance, techniques like progressive token refinement [3] or adaptive spatial masking [4] could be incorporated into our framework to further improve the modeling of fine-grained spatial details in dynamic regions. We believe that integrating such mechanisms with our asymmetric tokenization would allow SAMPO to scale gracefully to more challenging domains.\n\n[2] *Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. ICML 2025.*\\\n[3] *PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models. CVPR 2025.* \\\n[4] *AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders. CVPR 2023.*\n\n\n>## Q3: Scalability Beyond Compute\n\nWe argue that building scalable world models, particularly in the context of robotics, requires more than simply increasing model size or dataset volume. Below, we outline several key factors beyond cost that we consider essential for building more robust and generalizable systems:\n\n1. **Heterogeneous embodiment and control abstraction.** Robotic datasets are often heterogeneous in embodiment (e.g., arm type, degrees of freedom, gripper morphology) and control interfaces. To scale beyond specific robot morphologies, SAMPO's architecture must accommodate variable control tokens and state embeddings. Future extensions can integrate low-rank control factorization or policy-conditioned dynamics modeling to generalize across robot bodies without explicit retraining.\n\n2. **Viewpoint and context invariance.** Scaling to real-world deployments requires robustness to diverse camera viewpoints, lighting, and scene compositions. We believe this demands view-invariant tokenization and context-decoupled motion representations. The SAMPO framework can naturally support such invariances by conditioning future frame tokenization on shared multi-view context (e.g., via cross-attention), allowing consistent dynamics modeling across cameras.\n\n\n3. **Curriculum pretraining and transfer.** We advocate a structured curriculum that begins with Internet-scale open-domain videos to acquire basic physical knowledge, progresses to first-person human-object interaction data (e.g., EPIC-Kitchens) to ensure operational diversity, and finally specializes on robotic datasets. This staged progression aligns with the *\"Data Pyramid\"* proposed in [5].\n\n\n4. **Imagination fidelity under open-world uncertainty.** As world models are used for planning and long-horizon reasoning, scaling must also address uncertainty estimation and semantic consistency. We envision extending SAMPO with retracing rollouts, behavioral priors, or diffusion-based latent refinement to enhance fidelity under distribution shift, enabling safer policy training and evaluation in the wild.\n\nIn summary, we view **scaling not as a brute-force endeavor, but as a systems-level challenge involving representation, generalization, and modularity**. SAMPO's design opens up promising avenues in all three directions, and we plan to investigate these dimensions in future iterations.\n\n[5] *GR00T N1: An Open Foundation Model for Generalist Humanoid Robots. ArXiv 2025.*"}, {"review_id": "CBR64ILJRL", "reviewer": "Reviewer_Auas", "summary": "The paper proposes SAMPO, a video world model that predicts future videos in an autoregressive manner while generating multi-scale tokens in a coarse-to-fine fashion for each frame. It introduces an asymmetric tokenization process that allows future predictions to focus more on compact dynamics, thereby improving efficiency. Additionally, it incorporates motion prompts as extra cues for scene dynamics. Its high prediction capability results in superior performance in perception quality, visual planning, and model-based reinforcement learning.", "questions": "Please see the Weaknesses part.", "limitations": "Please see the Weaknesses part.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\nS1) Extensive experiments demonstrate that the proposed method surpasses previous world models, such as DreamerV3 and iVideoGPT, in perception quality, visual planning, and model-based reinforcement learning.\n\nS2) There are no major flaws in the writing, making it easy to follow.\n\nS3) The code is provided in the supplementary material for reproducibility.\n\n# Weaknesses\n\nW1) **Novelty concern.** The main benefits claimed by the authors are primarily derived from VAR (NeurIPS 2024), which proposes a next-scale coarse-to-fine prediction architecture. Additionally, the extraction and use of motion prompts are highly similar to those in TraceVLA (ICLR 2025).\n\nW2) **Confusing motivation.** It is unclear why predicting only coarser scales would enable the model to focus more on dynamic regions. Theoretically, the ratio of dynamic areas does not increase when the frame scale is reduced. If static content dominates the frame, it will still dominate at lower resolutions. Could you provide some references to support that? Also, as the model only predicts coarse frames, does this imply it cannot perform future rollouts by conditioning on the newly generated frames?\n\nW3) **Missing ablations.** It would be beneficial to see how changing the predicted granularity (full scales / coarser scales only) affects the performance in downstream planning tasks. Can predicting even coarser scales continue to improve efficiency and downstream performance? Is there a trade-off?\n\nW4) **Unclear details.** In the zero-shot generalization experiments described in Line 297-301, how are the action conditions transferred? Additionally, how is the reward predicted in the MBRL experiments? The paper only provides some implementation choices in Line 214-215 but does not specify what is ultimately used.\n\nW5) **Real-world experiment.** It would be great if the authors can do a real-world verification with some simple setups. For example, estimating the rewards of several robot policies with the proposed world model and comparing to the actual evaluation ranks.", "was_revised": true, "final_justification": "The rebuttal addresses my concerns. The proposed method is effective and may provide valuable insights to the community. Thus, I update my rating to borderline accept.", "rebuttal": "We would like to sincerely appreciate Reviewer Auas for the comprehensive review and insightful questions.\n\n---\n>## Q1: Novelty Reformulation\n\nWe emphasize that SAMPO introduces a **novel hybrid autoregressive architecture** , which seamlessly unifies intra-frame scale-wise generation with inter-frame causal modeling, tailored for efficient and action-conditioned world modeling. \n\nAlthough inspired by the coarse-to-fine decoding paradigm in VAR, SAMPO is not a direct adaptation; rather, it is a task-specific redesign that extends the idea **from static image generation to temporally causal, action-driven generative rollouts in robot-centric settings**. Moreover, SAMPO introduces an **asymmetric multi-scale tokenizer** (Sec. 3.2), which encodes observed frames densely to preserve semantic context and future frames sparsely to focus on dynamic content. This design significantly improves inference efficiency while retaining visual fidelity (Tab. 5), and is **not addressed by VAR**, which applies uniform scale-wise encoding without considering temporal asymmetry or efficiency in rollout.\n\nWe empirically observed that naively extending VAR's residual decoding to sparse future frames yields poor reconstruction quality. To address this, we devised a **scale-specific weighting scheme** during training (Appendix A.2), which adaptively balances loss contributions across scales. This adjustment is crucial for stabilizing generation when tokenizing future frames sparsely and ensures that both coarse dynamics and fine details are faithfully captured across rollout steps.\n\n\nRegarding motion prompts, **SAMPO and TraceVLA differ in both objective and mechanism**. TraceVLA overlays offline trajectories as visual prompts to enhance policy learning but lacks a generative component. By contrast, SAMPO integrates motion trajectories directly into the generation pipeline via a trajectory-aware prompt module (Sec. 3.3), which provides spatiotemporal priors that guide the model toward dynamic regions during rollout. This integration improves temporal consistency and physical realism, as confirmed by both quantitative gains (Tab. 4) and qualitative results (Fig. 4).\n\nWe hope this clarifies that SAMPO is a task-specific, modular redesign that advances beyond prior works by unifying coarse-to-fine generation, autoregressive rollout, and dynamic prompting in a single, scalable framework for robot-centric world modeling.\n\n>## Q2: Clarified Motivation\n\nAs described in ***Lines 289–292***, our **residual quantization** scheme assigns each scale to model the residual between its coarser‑scale prediction and the ground truth. This decomposition naturally delegates large, structural appearance shifts in space and time to the coarse scales (e.g., scales [1–2]), which capture the dominant dynamics, such as an arm moving from left to right. The finer scales (e.g., scales 6 or 8) then refine the output by adding texture and spatial detail. This principle aligns with the perceptual observation that motion signals often reside in low-frequency components (as also supported by [1, *Sec. 3.1*]; [2, *Sec. 3.2*]) and that residual quantization naturally enforces a progression from global to local dynamics. In robotic tabletop settings, where backgrounds are mostly static, such separation is especially effective. As shown below, modeling coarser spatial scales (e.g., [1–4]) improves dynamic understanding, while removing coarse levels leads to worse performance.\n\n|Spatial Scales|Tokens / Img.|FVD ↓|PSNR ↑|\n|:-:|:-:|:-:|:-:|\n|[1,2,3,4,5,6]|91|55.5|26.7|\n|[1,2,3,4,5]|55|57.7|25.4|\n|[1,2,3,4]|30|59.3|24.8|\n|[1,2,4,8]|85|80.7|23.1|\n\n\n\nAlthough we predict only a subset of finer-scale tokens, each frame is still represented by **91 tokens** (e.g., scales=[1,2,3,4,5,6]), which is significantly richer than prior work (e.g., iVideoGPT's 16-token setup). **These predicted tokens can be directly reused as input for the next step in the rollout.** Moreover, Tab. 5 shows that this configuration achieves a strong balance between speed and fidelity, and supports visual planning and RL tasks where rollout is essential. We plan to clarify these design choices in the final version.\n\n[1] *Generating Diverse High-Fidelity Images with VQ-VAE-2. NeurIPS 2019.*\\\n[2] *Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample. NeurIPS 2020.*\n\n>## Q3: Additional Ablations\n\nWe conducted an additional ablation study to assess how the number of predicted spatial scales affects **video quality** and **downstream planning performance**, following the setting of Tab. 5 and Tab. 3.\nDue to time constraints, we conducted two additional configurations and did not re-train a full-scale encoder ([1–16]) as it is computationally expensive.\n\nWe find that using [1–5] scales achieves comparable planning success as [1–6], with slightly worse video quality. Reducing to [1–4] leads to a moderate drop in fidelity, but still captures the key dynamics. The primary difference lies in texture detail, not motion quality. Notably, [1–4] achieves 3.2× faster rollout. This highlights a clear trade-off: fewer scales improve efficiency, while more scales enhance visual quality. We believe this flexibility allows users to choose the best setting for their application. We will include this result in the revision.\n\n|Spatial Scales|FVD ↓|PSNR ↑|Avg. Success ↑|Inference Time ↓|\n|:-:|:-:|:-:|:-:|:-:|\n|AR (iVideoGPT)|60.8|24.5|70.1|9.05 s/vid|\n|[1,2,3,4,5,6,8,10,13,16]|-|-|-|-|\n|[1,2,3,4,5,6]|55.5|26.7|72.2|2.06 s/vid|\n|[1,2,3,4,5]|57.7|25.4|71.3|1.38 s/vid|\n|[1,2,3,4]|59.3|24.8|70.6|0.63 s/vid|\n\n>## Q4: Clarified Details\n\n**Zero-shot experiments setup.** \nFollowing the setting used in iVideoGPT, Fig. 6 reports results from an **action-free zero-shot** setting, as further detailed in **Appendix Fig. 13** for clarity. We will update the caption and main text to avoid confusion.\nWe adopt this design because cross-robot generalization under action conditioning is inherently difficult, primarily due to heterogeneous embodiment. Different robots often differ in action space dimensionality, control semantics, and actuation properties. As discussed in our response to ***Reviewer cg6d (Q3)***, transferring actions across such differences remains a significant challenge. By removing the need for action tokens, we avoid the action-space mismatch and instead evaluate the model's ability to generalize visual dynamics such as object trajectories and arm motion across varied robot embodiments and manipulation scenes.\nWe agree that exploring action-conditioned zero-shot generalization is a valuable future direction. In future work, we plan to investigate action-space alignment and conditioning strategies to support this goal. We will clarify this design decision in the revised manuscript.\n\n**Reward Prediction Module.** \nIn our implementation, reward prediction is integrated into the transformer's token-level modeling pipeline. After tokenizing video frames and inserting start tokens `[S]`, each action embedding is linearly projected and added to the corresponding `[S]` token.\nWe attach a lightweight linear head to the hidden state of the token following `[S]` and the compressed visual tokens, mapping it to a scalar reward via regression. This head is trained jointly with the next-token prediction loss using an MSE objective, enabling frame-aligned, token-synchronized reward prediction without auxiliary branches.\nIn MBRL tasks such as Meta-World, SAMPO can directly generate both future observations and rewards, enabling data augmentation in the replay buffer. This simplifies the coupling between world model and policy learning, and supports efficient, rollout-based policy optimization.\n\n>## Q5: Real-world Experiment\n\nWe apologize that the limited hardware access during the rebuttal period, we were unable to conduct real robot evaluations.\nHowever, we plan to include such evaluations in future work by estimating the expected rewards of real-world robot policies using SAMPO and comparing them with actual execution outcomes. Prior work such as DreamerV3 has demonstrated that learned world models can simulate imagined trajectories for offline policy learning and evaluation. Given SAMPO's strong performance in both video prediction and planning, we believe it is well-suited for this purpose, and we will report results in future revisions. We greatly appreciate Reviewer Auas for this valuable suggestion."}, {"review_id": "pylyfKGhXh", "reviewer": "Reviewer_2sYs", "summary": "The paper proposes a new world model framework named SAMPO, which incorporates next-scale prediction and causal autoregressive prediction. More specifically, the authors propose to use an asymmetric tokenizer to represent each frame in different resolutions, while predicting frames autoregressively using a causal masking approach. This allows SAMPO to generate frame-wise trajectories based on action conditions. At the beginning of the generation, a prefix is added that includes a motion prompt, summarizing the motion history of the past frames. Overall, the proposed model outperforms related methods across multiple datasets, showing strong quantitative and qualitative results.", "questions": "- How does an AR-based model perform compared to a Hybrid AR model with the same number of parameters? What is the overhead introduced by Hybrid AR?\n- How does the error change over the rollout trajectory, especially compared to other models?\n- Do the experiments on the BAIR dataset starting with 1 context frame use the motion prompt? For RoboNet, does the motion prompt only consider the two first frames?", "limitations": "Yes (in Supplementary Material)", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- Straightforward and effective approach\nThe ideas introduced in the paper are straightforward and well-motivated. For example, it is reasonable to assume that different scales are needed for different parts of the prediction process. At the same time, the proposed method is based on the established autoregressive training strategy, showing strong generalization and scaling properties.\n- Strong Performance and convincing results\nThe provided quantitative and qualitative results underline the advantages that are offered by the proposed method over related works. Across multiple tasks, including action-conditioned and reward-based generation, the proposed model offers strong performance.\n- Thorough ablation study and analysis\nThe ablation study provides an important look into how each of the proposed components improves the performance. The authors cover most of the interesting ablation scenarios.\n\n### Weaknesses\n- Typographical errors in figures\nThere are multiple typographical errors in the paper, especially in Figure 2 (e.g., \"Enocder\", \"uantization\"). However, this should be easily fixed.\n- Clearer comparison with related work\nIt appears that the proposed approach is close to iVideoGPT, but adds the scale-variant component and some additional ideas. While the ablation study is thorough, it would be nice to make the relationship between those two components clearer.", "was_revised": true, "final_justification": "This paper proposes a straightforward but effective new method that combines next-scale prediction and causal autoregressive prediction for world modelling. Overall, this paper presents a valid contribution to the field. Therefore, I recommend to accept this paper.", "rebuttal": "We greatly appreciate Reviewer 2sYs for the careful reading of our paper and constructive comments in detail. We also appreciate the pointed-out typos, which we will correct in a future revision.\n\n---\n>## Q1: Clearer Comparison\n\nWe clarify that SAMPO is **fundamentally different** from iVideoGPT, rather than a straightforward extension with incremental components. As detailed in our response to ***Reviewer cg6d (Q1)***, SAMPO introduces a **novel hybrid autoregressive framework** that performs **multi-scale bidirectional spatial modeling** within frames while maintaining causal temporal modeling across frames. This design addresses key limitations in iVideoGPT, including the **loss of spatial structure and inefficient inference** due to raster-scan decoding. In addition, the asymmetric tokenizer and motion prompt modules are introduced to tackle challenges specific to action-conditioned world models. These motivations are elaborated in ***L174–179 and L190–193*** of the main text. We will make the distinction between the two frameworks clearer in the revised version.\n\n\n>## Q2: AR vs Hybrid\n\nFollowing the decoder-only architecture implemented in VAR, we designed our **hybrid autoregressive framework to support scaling across model sizes**. Due to the time constraints of the rebuttal period, we were unable to retrain the asymmetric multi-scale tokenizer and hybrid AR model from scratch under strict parameter parity.\nHowever, we present the following results from existing experiments with motion prompt disabled, which provide partial insight into parameter efficiency.\n\n| Description | Model | # Para. | FVD ↓ | SSIM ↑ |\n|-|:-:|:-:|:-:|:-:|\n| Hybrid AR |  SAMPO-S  | 207M | 227.4 | 76.4 |\n| Hybrid AR |  SAMPO-B  | 328M | 193.7 | 81.3 |\n| AR        | iVideoGPT | 436M | 197.9 | 80.8 |\n| Hybrid AR |  SAMPO-L  | 523M | 180.1 | 83.1 |\n\nSAMPO-B uses **25% fewer parameters** than iVideoGPT while achieving slightly better performance in both FVD and SSIM. This suggests that the hybrid autoregressive framework improves parameter efficiency. Furthermore, SAMPO supports larger-scale deployment via its decoder-only architecture, as evidenced by the **even stronger performance of SAMPO-L**.\nBased on our empirical results, the hybrid AR design appears to reduce computational overhead rather than increase it. Unlike token-level autoregression, which suffers from sequential decoding bottlenecks, our intra-frame bidirectional attention mechanism is fully parallelizable. As demonstrated in the ablation study provided in response to ***Reviewer Auas (Q3)***, the hybrid AR framework not only improves modeling capacity but also enhances runtime efficiency, making it particularly suitable for scalable world modeling.\n\n\n>## Q3: Trajectory Rollout Error\n\nWe thank the reviewer for the thoughtful question. While our current submission does not explicitly include quantitative plots showing how the prediction error evolves over the rollout horizon, we provide **qualitative and indirect evidence of temporal stability in Fig. 4 and Fig. 15** (in Appendix). Compared to prior autoregressive baselines such as iVideoGPT, SAMPO demonstrates consistently better spatial-temporal fidelity across extended horizons, particularly in long rollout settings like RoboNet and 1X World Model (10 frames). This can be attributed to our hybrid spatiotemporal autoregression and the use of coarse-to-fine generation, which better **preserves high-level motion early on and gradually refines details at finer scales**. In contrast, we observed that standard raster-scan autoregressive models tend to accumulate more low-level noise over time.\n\nWhile qualitative visualizations indirectly reflect temporal degradation, we agree that directly plotting error versus timestep curves would provide more intuitive insight. We intend to include such visualizations in the revised version. If including the full curve in the main text is not feasible, we will highlight this analysis and treat it as an important direction for future work, aimed at better assessing and improving long-horizon prediction stability.\n\n>## Q4: Motion Prompt Usage\n\nWe apologize for insufficient clarity in our experimental setup. In BAIR experiments (1 context frame), we omitted motion prompts to ensure a fair comparison with baseline methods.\nIn RoboNet experiments (2 context frames), we utilized CoTracker to extract point trajectories from the first two frames. The extracted trajectories were filtered by motion magnitude and tracking confidence, and subsequently projected onto the image plane. We will clarify these details in the paper to avoid confusion."}, {"review_id": "mY4QNoKqv4", "reviewer": "Reviewer_gRv6", "summary": "SAMPO introduces a novel world model that combines temporal causal modeling with coarse-to-fine visual auto-regression. The framework addresses key limitations in existing autoregressive world models through three main innovations: a hybrid autoregressive architecture that maintains both temporal and spatial coherence, an asymmetric multi-scale tokenizer that enhances scale-aware attention, and a trajectory-aware motion prompt module that provides explicit spatiotemporal priors for robot and object trajectories. Experimental results demonstrate the SOTA performance in video prediction and model-based RL of SAMPO.", "questions": "What are the frame rate and total duration of the generated videos?\n\nIn Table 3, why is there such a large variation in success rates between different tasks? And why is the average success rate for the \"simulator\" row 100%?\n\nIn the second row of the right part of Figure 4, doesn't the red small object appear to be occluded by the robotic arm rather than disappearing?", "limitations": "The planning experiments in this paper were all conducted in simulation platforms. The paper would be more convincing if experiments were performed in real-world environments.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "SAMPO demonstrates strengths in world model design: e.g. the trajectory-aware motion prompt module provides explicit spatiotemporal priors, improving the model's ability to capture dynamic interactions and physical dependencies. Experimental results validate the state-of-the-art performance in video prediction, motion modeling accuracy, and robot control tasks.\n\nHowever, I'm somewhat skeptical about how helpful the generated videos actually are, as the examples shown by the authors in Figures 4 and 6 are all very short clips where the robot's movements are quite limited in range.", "was_revised": true, "final_justification": "I still find it difficult to believe that such short-horizon video predictions can have a substantial impact on robot control, especially given that the robot’s motion in the videos presented in the paper appears to be minimal, involving free movement in space or simple interactions with a single object. Nevertheless, I sincerely appreciate the detailed responses provided by the authors in this rebuttal.", "rebuttal": "We greatly appreciate Reviewer gRv6 for providing a detailed review, insightful questions and forward-looking feedback.\n\n---\n>## Q1: Utility of Generated Videos\n\nFirst, SAMPO is designed as a **control-oriented world model**, where the goal is to produce short but accurate rollouts to support planning and reinforcement learning, rather than generating long-horizon videos. This design choice aligns with the nature of robotic manipulation tasks in BAIR and RoboNet, which typically involve **short-horizon interactions** in constrained tabletop environments with fixed camera views.\n\nSecond, the brevity of predicted sequences does not diminish their utility. In practice, 10 to 15-frame predictions from 1–2 context frames have proven sufficient for downstream tasks like visual planning (Tab. 3) and MBRL (Fig. 5). SAMPO also supports action-conditioned generation, reward modeling, and exhibits strong zero-shot transfer to unseen manipulation tasks, without being restricted by sequence length. Thus, even short clips are effective for training and deploying intelligent agents.\n\nThird, generating longer sequences introduces greater **uncertainty due to error accumulation**, a challenge that remains an open problem in long-horizon prediction. Instead of relying on long open-loop rollouts, our method emphasizes producing short but accurate predictions that are recomputed at every step. This approach is consistent with many prior works in Model Predictive Control (MPC) [1, 2].\n\nNevertheless, we agree that extending prediction horizons and modeling more diverse motion patterns, such as mobile manipulation and outdoor scenarios, is a valuable future direction.\n\n[1] *Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv 2018.* \\\n[2] *Deep Visual Foresight for Planning Robot Motion. ICRA 2017.*\n\n>## Q2: Frame Rate & Duration\n\nThe temporal length of our generated videos follows the standard rollout settings used in prior work (iVideoGPT, MAGVIT, MaskViT, etc.). Specifically, the BAIR dataset operates at 20 FPS, so predicting 15 frames results in a 0.75s video. In contrast, RoboNet and 1X operate at 10 FPS, and our setting of 10 predicted frames yields a 1.0s video.\n\n>## Q3: Large Variation in Success Rates\n\nBased on our understanding and experiments, the performance gap across VP2 tasks is mainly due to:\n\n- **Task complexity.** Tasks like `Push blue Button` are visually explicit, while others like `Open Slide` require fine-grained contact modeling and involve occlusion, making them harder to model.\n\n- **Visual ambiguity.** At low resolution, it is often unclear whether key interactions (e.g., contact) occur, leading to underestimation of visually plausible predictions.\n\n- **Noisy reward evaluation.** VP2 relies on pretrained classifiers rather than ground-truth rewards, which can misjudge success and amplify inter-task variance.\n\n\nThe \"100%\" aggregate score for the simulator baseline results from **normalizing each model's average task success by the simulator's average**, making the simulator's score always 1.0 (100%). This follows the evaluation convention in [3]. We will clarify this normalization step in the revised version, both in Sec. 4.2 and the captions of Tab. 3. \n\n[3] *A Control-Centric Benchmark for Video Prediction. ICLR 2023.*\n\n\n>## Q4: Occluded Red Small Object \n\nThe key comparison lies in **motion consistency**. In iVideoGPT's result (Fig. 4, second row right), the robot arm's predicted trajectory deviates from the ground truth, causing occlusion of the red object and indicating a failure in maintaining accurate object-track continuity. In contrast, our method preserves spatial coherence between the robot and the object, avoiding unnatural occlusion and ensuring consistent interaction. This demonstrates that our model generates more realistic and task-consistent motion predictions.\n\n>## Q5: Plans in Real-world\n\nThe experiments reported in ***Sec. 4.2 (Visual Planning and Reinforcement Learning)***, though conducted in simulation, are performed under low-level control settings, bridging the gap to real-world deployment. Although new hardware experiments are not feasible within the rebuttal period, we are actively transitioning our system to real robots.\n \nInspired by recent real-world deployments of world models (e.g., CoTVLA, WorldVLA), we believe SAMPO is well-suited for deployment on physical robots. Concretely, we identify two integration paths:\n\n- **Future-State Priors for VLA Policies:** SAMPO can be integrated with Vision-Language-Action (VLA) models like RT-2 and OpenVLA by providing short-horizon predictions that help ground actions in anticipated dynamics. This improves motion consistency and decision reliability.\n\n- **Synthetic Rollouts for Data Augmentation:** SAMPO generates physically coherent future states, enabling efficient data augmentation for robotic policy learning.\n\nWe will highlight these real-world deployment strategies in the revision to better convey SAMPO’s practical impact. We sincerely appreciate your forward-looking feedback!"}]

Author rebuttal:

Meta-review: This paper introduces SAMPO, a hybrid autoregressive world model that combines scale-wise spatial generation with temporal causal modeling for robotic applications. The work addresses key limitations in existing autoregressive world models through an asymmetric multi-scale tokenizer and trajectory-aware motion prompts.

All four reviewers provided positive evaluations, with three recommending acceptance (ratings of 5, 4, and 5) and one initially borderline (rating 3 raised to 4 after rebuttal). The reviewers praised the paper's technical soundness, comprehensive experimental evaluation, and strong empirical results across multiple datasets and tasks including video prediction, visual planning, and model-based reinforcement learning.

Reviewer concerns were primarily focused on novelty questions regarding the relationship to VAR and iVideoGPT, the practical utility of short-horizon predictions, and missing ablation studies. The authors provided thorough responses in their rebuttal, clarifying that SAMPO introduces a novel hybrid architecture specifically designed for action-conditioned temporal modeling, rather than being a straightforward adaptation of existing methods. They effectively addressed concerns about the sparse tokenization assumption by explaining its appropriateness for robotic manipulation datasets, and provided additional ablation studies demonstrating the trade-offs between prediction scales and performance.

The rebuttal successfully convinced the initially skeptical reviewer to raise their score, while the other reviewers maintained their positive recommendations.
Decision: Accept (poster)