Astra: General Interactive World Model with Autoregressive Denoising
Authors: Yixuan Zhu
Feng Jiaqi
Wenzhao Zheng
Yuan Gao
Xin Tao
Pengfei Wan
Jiwen Lu
Jie Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: world model
video generation
Primary area: generative models

Abstract: Recent advances in diffusion transformers have empowered video generation models to generate high-quality video clips from texts or images. However, world models with the ability to predict long-horizon futures from past observations and actions remain underexplored, especially for general-purpose scenarios and various forms of actions. To bridge this gap, we introduce Astra, an interactive general world model that generates real-world futures for diverse scenarios (e.g., autonomous driving, robot grasping) with precise action interactions (e.g., camera motion, robot action). We propose an autoregressive denoising architecture and use temporal causal attention to aggregate past observations and support streaming outputs. We use a noise-augmented history memory to avoid over-reliance on past frames to balance responsiveness with temporal coherence. For precise action control, we introduce an action-aware adapter that directly injects action signals into the denoising process. We further develop a mixture of action experts that dynamically route heterogeneous action modalities, enhancing versatility across diverse real-world tasks such as exploration, manipulation, and camera control. Astra achieves interactive, consistent, and general long-term video prediction and supports various forms of interactions. Experiments across multiple datasets demonstrate the improvements of Astra in fidelity, long-range prediction, and action alignment over existing state-of-the-art world models.

Published as a conference paper at ICLR 2026
ASTRA GENERAL INTERACTIVE WORLD MODEL WITH AUTOREGRESSIVE DENOISING
Yixuan Zhu Jiaqi Fengl.", Wenzhao Zhengl.
Yuan Gao², Xin Tao?, Pengfei Wan?, Jie Zhou', Jiwen Lu
TTinggua University Kuaishou Technology
Project Page: https: http://ewwr..aaaa..a..aaao///eea
Code: https://gitai com
World exploration
(b) Manipulation video prediction
(c) Autonomous driving
Figure I: Our Astra enables interactive and versatile world modeling across exploration, robotics, and autonomous driving. With our enhanced design spanning framework architecture to training and inference, it delivers precise responsiveness to user instructions and strong long-term consistency, achieving coherent high-fidelity vidcos that faithfully follow instructions.
ABSTRACT
Recent advances in diffusion transformers have empowered video generation models to generate high-quality video clips from texts or images. However, world models with the ability to predict long-horizon futures from past observations and actions remain underexplored, especially for general-purpose scenarios and various forms of actions. To bridge this gap, we introduce Astra, an interactive general world model that generates real-world futures for diverse scenarios (e.g., autonomous driving, robot grasping) with precise action interactions (e.g., camera motion, robot action). We propose autoregressive denoising architecture and use temporal causal attention to aggregate past observations and support stream- ing outputs. We use noise-augmented history memory avoid over-reliance on past frames to balance responsiveness with temporal coherence. For precise action control, we introduce action-aware adapter that directly injects action signals into the demoising process. We further develop a mixture of action ex- perts that dynamically route heterogeneous action modalities, enhancing versatil- ity across diverse real-world tasks such as exploration, manipulation, and camera control. Astra achieves interactive. consistent, and general long-term video pre- diction and supports various forms of interactions. Experiments across multiple datasets demonstrate the improvements of Astra in fidelity, long-range prediction, and action alignment over existing state-of-the-art world models.
Equull contribution; Project leader; "Corresponding author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Building generative world models an emerging field where the ability to synthesize realistic and coherent video trajectories serves as proxy for understanding and simulating the underlying dy- namics of the world. With the rapid advances in visual generation (Rombach al., 2022; Blattmann et al., 2023; Yang et al., 2025; Brooks et al., 2024; Wan et al., 2025), numerous video genera- tion models have emerged and can perceive contextual cues and synthesize high-fidelity videos of open- woen-wordd scenarios. These advances serve the foundation for broader world simulation tasks, including game engines, autonomous driving, and spatial intelligence.
Standard text-to-video (T2V) image-to-video (I2V) models typically produce only short, self- contained video clips conditioned on prompts or reference images. They lack the ability to generate coherent long- mong-horzzon rollouts that respond adaptively to external stimuli such as agent movements, viewpoint changes, or control signals. Without this responsiveness, these models fall short of simu- lating the interactive and causal dynamics of the real world. Furthermore, existing video generators are constrained by the finite temporal window of diffusion models, which prevents them from pro- ducing extended video sequences, Although recent works (Mao al., 2025; He al., 2025; Song et al., 2025; Teng et al., 2025) have explored video continuation techniques or hybrid frameworks that combine autoregression with diffusion, they often struggle to strike balance between main- taining consistency with historical frames and remaining responsive to new inputs. In addition, the autoregressive generation process introduces error accumulation, leading to degraded quality and coherence in long-term predictions. As result, despite impressive progress in generative fidelity, existing approaches remain largely passive-capable of rendering visually compelling content yet lacking the interactivity, adaptability, and robustness required for truc world simulation.
To address these challenges, we present Astra, simple yet powerful framework for building highly interactive world models. At its core, our method adopts autoregressive denoising paradigm, where we augment a pre-trained video diffusion backbone with an action-aware adapter, as shown in Figure 2. This design preserves the high generative quality of diffusion models while enabling precise conditioning on agent actions, thereby allowing the model to produce coherent futures that respond instantly user inputs. A central difficulty in world modeling is balancing long-term tem- poral consistency with action responsiveness. To mitigate this, we propose noise-as-mask strategy that softly corrupts historical frames during training. This reduces the dominance of visual context, forcing the model to integrate both history and action cues when predicting the next video chunk. Moreover, real-world interactive environments involve heterogeneous action modalities from cam- era controls and body pose to robot manipulations. To enhance versatility across such settings, we design a Mixture of Action Experts (MoAB), where modality-specific experts specialize in different action types under learnable routing mechanism. This enables our model to unify diverse inter- action signals within a single framework, making it broadly applicable across scenarios spanning embodied robotics, immersive video simulation, and Long-horizon world exploration.
We conduct extensive experiments on diverse open-source benchmarks, including Sekai (Li et al., 2025a), SpatialVID (Wang et al., 2025), RT-1 (Brohan et al., 2022), nuScenes (Caesar et al., 2020) and Multi-Cam Video (Bai et al., 2025). As illustrated Figure 1, Astra achieves state-of-the- art performance in action-driven video prediction, generating sequences that are highly interactive while maintaining visual coherence and dynamic consistency. Furthermore, our framework also demonstrates strong generalization across tasks and environments, underscoring its potential foundation for next-generation visual world models.
2 RELATED WORK
Video generation models. In recent years, denoising diffusion models (Ho et al., 2020; Song et al., 2020) have become the dominant paradigm in generative modeling, celebrated for their high fidelity and controllability. Following their success in text-to-image (T2I) synthesis (Rombach et al., 2022; Dhariwal & Nichol, 2021), early video generation methods (Blattmann et al., 2023; Ho et al., 2022) extended diffusion models to the temporal domain, typically by inflating mmage-based UNets (Ron- neberger et al., 2015) with additional temporal layers. More recently, Sora (Brooks al., 2024) and following studies (Zheng et al., 2024; Yang et al., 2025; Ma et al., 2024) pioneer high-quality, high-resolution text-to-video (T2V) diffusion models. These models commonly use the diffusion
2

---

Published as a conference paper at ICLR 2026
Chunk
Chunks
Step
Step
Chunk-wise causal conditioning
Autoregressive Denoising World Framework
Astra
Image
Actions
Overview of the proposed Astra. Our autoregressive denoising world model generates future video chunk by chunk from an initial image, actions, and optional prompts. Chunk-wise causal conditioning enforces temporal coherence and faithful action response.
Figure 2:
transformer (DiT) architecture (Peebles & Xie, 2023) to capture complex spatial and temporal co- herence. More recent models, including (Song et al., 2025) and (Zhang & Agrawala, 2025). further improve long-horizon consistency. Beyond pure diffusion, hybrid frameworks have been proposed to reconcile long-range prediction with high-quality synthesis. By combining autoregression for temporal modeling and diffusion for local realism, approaches such as StreamingT2V (Henschel et al., 2025) and MAGI (Teng et al., 2025) achieve extended video rollouts. However, these meth- ods still struggle with error accumulation across long sequences and offer limited responsiveness to external actions, leaving gap between video generation and true interactive world simulation.
Visual world models. Beyond video generation, visual world models aim to capture the causal dynamics of the environment, enabling agents to simulate future trajectories and interact with the world. Unlike text-to-video or image-to-vides models that generate short clips conditioned on static inputs, visual world models explicitly incorporate history and actions, making them essential for tasks such as planning, control, and embodied intelligence. Several recent works demonstrate this trend. (Wu al., 2024) extends autoregressive video transformers to integrate actions and rewards, allowing agents to predict how future observations evolve in interactive environments. (Wang et al., 2024) formulates world modeling as masked-token prediction in discrete latent space, supporting multimodal conditioning and open-ended environment simulation. (Huang et al., 2025; He et al., 2025) adapts pretrained video diffusion models into an autoregressive framework with causal action guidance, making them capable of controllable video prediction. In navigation contexts, (Bar et al., 2025) employs conditional diffusion-transformoe facilitating planning in unfamiliar scenes. Meanwhile, (Cen al., 2025) introduces a joint vision- language-action (VLA) framework that learns to model both world states and agent behaviors in an autoregressive manner. At larger scales, frameworks such (Bruce et al., 2024; Agarwal et al., 2025; Liu et al., 2024) demonstrate that scaling up transformer architectures and extending context windows significantly improve rollout fidelity and generalization across domains. Recently, (Mao et al., 2025) uses Masked Video Diffusion Transformer (MVDT) that masks input features to improve video quality. Astra instead adopts noise-as-mask strategy that selectively degrades past visual context, prompting the model to better integrate action signals and follow user commands. While such approaches demonstrate the promise of world models for interactive environments, they still face error accumulation, temporal drift in long rollouts, and limited responsiveness to diverse actions. Overcoming these issues remains key to building reliable and scalable visual world models.
to generate plausible future agent observations,
PROPOSED APPROACH
In this section, we present Astra, an autoregressive denoising framework that achieves real-world video prediction and enjoys high interactivity, versatility, and consistency. Our key idea is to harness the visual generation power within a pre-trained text-to-video diffusion model and incorporate the chunk-wise autoregressive prediction by using previously generated clips conditions. We will start by reviewing the background of autoregressive denoising models, and then describe our designs of Astra, including action-aware adapter for precise conditioning and noise-augmented history memory for long-term consistency, and a mixture action experts (MoAE) for unifying diverse action inputs. The overall framework of our Astra illustrated in Figure 3.
3

---

Published as a conference paper at ICLR 2026
Action
Training
Input
Action
Cond.
Latent
Next chunk
Flow Transformer
M c
Flow Transformer block
Action Encoder
Linear *
noise
za C,)
Image
chunks
Inference
ACT-Adapter
Cross Attn.
W
Sample
FFN
Generated video
g
Dec.
Trainable
Feature
Update
Frozen
Output
(a) Autoregressive denoising world framework
(b) Action-aware flow transformer
Figure 3: The overall framework of Astra. The Action-Awar Flow Transformer (AFT) injects action signals into the latent space via ACT-Adapter (right), which aligns action features through an encoder and adds them to each transformer block. During training (left top), the model learns next-chunk prediction with flow matching. During inference (left bottom), it autoregressively gen- erates video chunks conditioned on history and action streams, producing interactive videos.
3.1 PRELIMINARY: AUTOREGRESSIVE DENOISING MODEL
We adopt autoregressive denoising framework. which integrates the hong-horizon modeling of autoregression with the high-fun high-fideitty synthesis of diffusion. Given video sequence discretized into chunks x1:A I:N the generation objective is:
p(ZLNN IN
II
p(z'
2 zt)).
(1)
For each step, the next chunk # is predicted through a denoising process, trained using flow matching. Specifically, we first sample a noisy interpolation of the target chunk:
x (1 2) ttc,
e~ C,,, t [0,
(2)
and train the flow model my to estimate the clean direction:
(()) = Eit.c [|00 1 =1) - o (2 )1)
(3)
where v" is the ground-trute -tuun velocity. In inference, autoregressive generation proceeds by denoising from noise to obtain z++,, which is then appended to history for predicting future chunks. This iterative AR AR-denois loop enables long-range, consistent, and high-quality video prediction.
3.2 INTERACTIVE WORLD MODELING VIA AUTOREGRESSIVE DENOISING
Modern video generation models (Brooks et al., 2024; Wan et al., 2025) have demonstrated re- markable progress simulating realistic visual dynamics. These models benefit from large-scale pretraining, which enables them to implicitly acquire partial knowledge of 3D spatial perception, temporal dependencies, and even simple physical patterns such motion and force. However, de- spite their impressive fidelity, these models still fall short in constructing real-world scenes that can be preserved, interacted and explored. This raises a key question: Are T2V models really world mod- els? In our view, the defining property of world model is interactivity tte ability to adapt genera- tion dynamically in response to arbitrary action inputs at arbitrary moments. While diffusion- based global prompts
models can be conditioned do not enable fine- fine-graines grained, online interaction. To address this limitation, we turn to the autoregres- sive framework, which naturally supports stepwise prediction conditioned both past observations
attributes, such conditioning mechanisms

---

Published as a conference paper at ICLR 2026
and current actions. Unlike diffusion models that generate video in single pass, autoregression allows for instantancous responses to action inputs, enabling controllable and adaptive rollouts. By integrating this property with the generative power of denoising models, we design autoregres- sive denoising framework that achieves both high-quality synthesis and interactive controllability. Although prior works have explored combining autoregression and denoising, reconstructing this hybrid paradigm for world modeling remains non-trivial. Beyond simply chaining autoregression and denoising, we must carefully define the observation-action interface and design mechanisms to balance the trade-off between long-term consistency and immediate responsiveness
Given the previous video chunks 2!xi=1 we aim model the conditional distribution of the next chunk p(=" | 1= In principle, this prediction can be realized by a variety of generative models. To ensure high visual fidelity, we choose to leverage a pre-trained video flow matching model vy as the predictor, leveraging its strong video synthesis capability. However, integrating such model into an interactive world framework introduces two central challenges: (1) How to represent actions and quantify their impact future visual dynamics; (2) How to effectively incorporate action signals into a pre-trained diffusion backbone while preserving its generative quality.
Since our goal is to enable instantaneous responses to action inputs a' (e.g. an instruction such as "turn right"). the effect of an action should manifest as a direct transformation on the predicted video chunk. Inspired by the formulation of optical flow, we interpret this transformation as a shift in video features, which, in diffusion models, correspond to latent representations within the de- noiser. Accordingly, we treat the action as additional conditioning signal for the diffusion model, applied directly to its latent feature space. This requirement poses a challenge for existing video diffusion architectures, which are typically composed of stacked Transformer blocks (DiT) and rely on tross-attention layers to align video latents with textual embeddings. Such mechanisms are not naturally suited for modeling finegrriieed petion-induced shifts. To overcome this, we introduce the aetion-awary flow transformer adapter (ACT-Adapter) that augments pre-trained video DiT into an autoregressive denoising model capable of integrating action signals as latent-space transformations, preserving the generative power of the backbone while modeling the action's 's influence.
As shown in Figure 3, we introduce an action encoder to project actions into a feature space aligned with the video latents. The resulting action features are injected into the denoising model through element-wise addition at every block, ensuring that action signals directly modulate the latent rep- resentation. To maximize reuse of pre-trained knowledge, we freeze all parameters of the flow transformer except for the self-attention layers. In addition, we insert a lightweight adapter mod- ule-a single linear layer initialized the identity matrix-after each self-attention block. This enables the model to gradually learn action-aware transformations while maintaining the stability of the pre-trained backbone. For the history condition z. z 21:0-1, we adopt the frame-dimension conditioning strategy that concatenates the previous chunks with the predicted chunk along the tem- poral dimension before being processed by the flow transformer. Together with the action a' and prompt c. the full condition set for the denoising model uy is C = fz !s- a alif.cc)
To enhance the effect of actions, we propose an action-free guidance mechanism (AFG), inspired by class-free guidance (CFG). During training, action conditions are randomly dropped, forcing the model to predict without action inputs. At inference, we compute a guided velocity field:
U guided = Uy (z,, t, 0) + s (vo(z,. f, a) Uy (zt, t, 0)),
(4)
where s is the guidance scale, 2, is the combined latent and denotes the null-action condition. This technique sharpens the action effect, yielding more precise responses user inputs.
3.3 HISTORY CONDITION WITH NOISY MEMORY
After addressing the challenge of action control, we turn to another open problem: balancing the long-term temporal consistency with responsiveness to actions. Prior works have shown that gen- erating coherent long videos requires conditioning on extended history. However, we observe a trade-off: increasing the length of history improves temporal consistency but weakens action re- sponsiveness. We refer to this phenomenon as visual inertia-the tendency of the model to rely heavily on past visual information while overlooking user actions. This arises because roal-world datasets contain predominantly smooth motions. leading the model to prioritize continuity over abrupt, action-driven changes. To mitigate this inherent contradiction, we avoid naively shorten- ing the conditioning horizon and instead seek a more elegant solution. Considering the asymmetry
5

---

Published as a conference paper at ICLR 2026
Actions
Input
Router
Top-k scores
Emb.
projectors
Expert
Expert
Expert
Flow Transformer
6'
o
0/1 mask
Actived experts
Figure 4: Mixture of Action Experts (oloAE).Action signals from diverse modalities are projected into a shared space, augmented with a history mask, and routed to aodallty-specializee experts. A dynamic router selects top-k experts, whose outputs aggregated into unified embeddings and fed into the Flow Transformer, enabling versatile and precise action-condin conion-coniiioneed generation.
between dense visual inputs and sparse action signals, we propose to reduce the dominance of vi- sual conditioning by introducing controlled corruption. Unlike (Mao et al., 2025), which randomly masks visual tokens, we adopt a noise-as-mask strategy: injecting random noise into condition- ing video to degrade and blur its information content (Figure 3). This design offers two advantages. First, it requires no architectural modifications or additional learnable parameters in the denoising model. Second, by corrupting the visual context, it prevents the model from directly copying clean frames and forces it to integrate action cues into generation. The corruption noise independent of diffusion noise, inference model learns to balance reliance on action and history, thereby overcoming visual inertia. To further extend the effective history horizon, we adopt the compression approach of (Zhang & Agrawala, 2025), which retains the first frame while compressing the intermediate history into compact visual tokens, preserving oong-range temporal information without overwhelming the action signal.
clean historical frames. Through this training strategy. the
3.4 MIXURE OF ACTION EXPERTS FOR DIVERSE SCENARIOS
Interactive world modeling often involves multi-modal inputs, including camera observations, body pose, and discrete action commands. These heterogeneous signals differ in structure and scale, making it challenging for single model to capture their characteristics. To address this, we propose the Mixture of Action Experts (MoAH), modular framework that routes different modalities to specialized experts, producing a unified action representation for the denoising model.
As shown in Figure 4, each action and discrete keyboard/mouse commands aemd-is first mapped into a shared action space via modality-specific projector Rm, as ā' = Rm, (a))), where m E {cam, rob, cmd] denotes the specific modality and i is the sequence index. A router network then computes gating scores g' = Router (ā') to select the top-K relevant experts. Each chosen expert Eg, implemented as independent MLPs, transforms the aligned features into task-relevant representations, The expert outputs are then aggregated according to the router's gating scores to produce the final action em- bedding e' = Σ 1= g Eg (ā'). The embedding sequence e 1: is then fed into the flow transformer. To account for both historical and current actions, we augment the action space of â' with an addi- tional binary indicator specifying whether the input corresponds to past current action.
continuous camera pose aeam robot pose arob>
Combining MoAE with history-conditioned latent inputs allows the model to generate video chunks that are temporally coherent and responsive across modalities. This design improves modality spe- cialization, scalability to new signals, efficiency by activating only relevant experts, and overall versatility, enabling high-fidelity predictions complex interactive scenarios.
4 EXPERIMENT
4.1 EXPERIMENT SETUPS
Datasets. To train model, leverage diverse set of datasets covering autonomous driving, egocentric exploration, multi-camera rendering, and robot control, as shown Table Specifically, nuScenes (Caesar et al., 2020) for vehicle pose prediction, Sekai (Li et al., 2025a) and

---

Published as a conference paper at ICLR 2026
Table 1: Datasets used in experiments, along with their actions and sample sizes. For each dataset,
we list the action type, followed by the dimensionality of its representation.
Dataset
Action
Scenario
Size
nuScenes (Caesur et al., 2020)
Camera (7)
Autonomous driving
850
Sckui (Li al., 2025a)
Camera (12)
Walking & drone view
50K
SpatialVid (Wang et al., 2025)
Camera & keyboard mouse
In-the-wild videos
200K
RT-1 (Brohan al., 2022)
Robotic pose (7)
Rohot manipulation
9978
Multi-Cam Video (Rai al., 2025)
Camera (12)
Human motion
136K
Total
397K (360 hours)
Table Quantitative comparison of diffferent models. Astra demonstrates superior visual quality and instruction-folloving performance across variety of real-world scenarios.
Instruction Following
Subject
Background Consistency
Motion Smoothness t
Aesthetic Quality 1
Imaging Quality
Method
Consisteney t
W (Wan al., 2025)
0.061
0.854
0.903
0,958
0.489
0.691
MatrixGame (He al., 2025)
0.268
0.916
0.928
0,981
0.441
0.748
Yume (Mao et al., 2025)
0.652
0.936
0.938
0.985
0.523
0.741
Astra (Ours)
0.669
0.939
0.945
0.989
0.531
0.747
VDD (Wang et al., 2025) for large-scale in-the-wild videos with rich camera annotations, Multi-Cam Video (Bai al., 2025) for synthetic multi-view sequences, and RT-1 (Brohan et al., 2022) (via Open X-Embodiment (O'Neill et al., 2024)) for robot action trajectories. All videos are resized and cropped to 480p, and action annotations are temporally aligned by interpolating every 4 frames, matching the temporal compression of the video VAE. Together, these datasets provide complementary action signals (vehicle, camera, and robot pose) that support unified action-aware world modeling. For evaluation, we construct Astra-Bench. a benchmark comprising 20 held-out samples from each dataset, designed to cover a diverse range of real-world scenarios.
Training details. We initialize our model from the pre-trained video diffusion model (Wan et al., 2025) and train on GPUs (80 GB each) with a per-GPU batch size of 1. Optimization is per- formed with AdamW (Loshchilov & Hutter, 2017) using learning rate of 1c 5 for 30 epochs, requiring about 24 hours to converge. The training is conducted in the latent space of a 3D VAE. In pixel space, the number of condition frames randomly sampled from [1, 128], while the number of target frames is fixed to 33. Additional implementation details are provided in Section A.
key aspects of world models: visual quality and instruction following (camera motion tracking). using six fine-grained metrics. For instruction following, we assess whether generated videos accurately reflect intended walking directions and camera move- ments. While pose estimation tools such as MegaSaM (Li et al., 2025b) can automate this process, inaccuracies in camera motion prediction and quantization errors limit their reliability. We therefore adopt human evaluation to ensure accurate assessment. We adopt metries from VBench (Huang et al., 2024) for the remaining dimensions (i.e., subject consistency, background consistency, mo- tion smoothness, aesthetic quality, and image fidelity). All test vidcos are generated at 480x832 resolution, 20 FPS, and 96 frames, using 50 inference steps for each model.
Metrics. Astra-Bench evaluates
4.2 MAIN RESULTS
We evaluate our method on Astra-Bench and compare against recent state-of-the-art video gen- eration and world modeling approaches, including Wan-2.1 (Wan et al., 2025), Matrix-Game (He al., 2025) and YUME (Mao al., 2025). As shown in Table 2, our method consistently out- performs baselines across all metrics, demonstrating strong advantages in both visual quality and cectoo-conditionee responsiveness. Astra achieves consistently superior results, surpassing existing video generation and world modeling approaches in both fidelity and controllability. Our model produces videos with sharper details, smoother motion, and stronger temporal coherence (Figure 5), leading to higher scores on visual quality metrics such as subject and background consistency, mo- tion smoothness, and overall aesthetic appeal. More importantly. it demonstrates a clear advantage in instruction following: human evaluations confirm that the generated trajectories more faithfully follow intended camera movements and action directions compared to prior methods. Importantly, while competing approaches often suffer from error accumulation and drift during long rollouts,

---

Published as a conference paper at ICLR 2026
Input image
Action-driven video generation
Figure 5: Qualitative results on action-driven real-world exploration. Starting from a single initial frame, our model generates long-ter ten-termm exploration videos with high visual fidelity, smooth and coherent dynamics, and precise responsiveness to action inputs.
Astra maintains stability across extended horizons (Figure 6), making particularly suitable for real-world interactive scenarios where both high fidelity and reliable action following are critical.
4.3 ANALYSIS
Action integration via ACT-Adapter. Our lightweight ACT Adapter provides an efficient way to inject action features into the pre-trained flow transformer. Freezing most parameters while tun- ing only the adapter and attention layers ensures maximum reuse of generative knowledge. Abla- tion results (Table 3, cross attn. adapter) demonstrate that ACT-Adapter achieves stronger action- conditioned performance than the cross-attention adapter used in He et al. (2025), confirming its effectiveness as a simple yet powerful extension for enhancing interactivity.
Action-free guidance enhances action responsiveness. We find that action-free guidance effec- tively amplifies the influence of action signals during inference. By learning to predict both with and without actions, the model gains stronger controllability, achieving sharper responses commands while preserving stability. As shown in Table (w/o AFG), this mechanism proves particularly useful for improving action responsiveness in long video rollouts.
Noisy memory mitigates visual inertia. The proposed voise-as-mask strategy alleviates the issue of visual inertia by weakening the dominance of historical context over action inputs. This encourages the model to rely on external controls rather than simply extrapolating from past frames. As shown in Table (w/o noise), the model achieves stronger responsiveness abrupt or unexpected actions, while still maintaining long-term temporal coherence and stability in video generation.
Adapting diverse scenarios with MoAE. MoAE allows Astra to process diverse action modali- ties in a unified way. By routing inputs to modality-specializz experts, it provides both versatility and precision across domains such as robotics and navigation. While joint training on heteroge- neous datasets may slightly reduce performance cross-domain generalization and enables broader data usage-critical for real-world applications. Our ablation in Table 3 (w/o MoAE), trained only camera-action data since it cannot process other modalities, further shows that MoAE markedly improves action-conditioned video generation.
any single scenario, MoAE greatly improves
4.4 EXTENDED APPLICATIONS
Our interactive world model not limited to standard video prediction benchmarks, and it naturally extends wide range of real-world applications. Thanks to its balanced design of action-aware
8

---

Published as a conference paper at ICLR 2026
Input image
Wan-2.1
Matrix-Gan
YUME
Ours
Figure 6: Qualitative comparisons on ection-drivn real-world exploration. Given the initial im- age and action sequence, Astra generates exploration sequences that maintain strong visual fidelity, coherent dynamics, and accurate responsiveness user-specified actions.
Table 3: Ablation studies. We assess the contribution of each component Astra, ensuring all experiments are conducted using the same random seed for fair comparison.
Instruction Following
Subject Consistency
Background Consistency f
Motion Smoothness t
Aesthetic Quality t
Imaging Quality
Method
w/o AFG w/o noise
0.545
0.841
0.892
0.957
0.492
0.703
0.359
0,903
0.927
0.979
0.523
0.739
cross attn, adapter
0.642
0.926
0.903
0.948
0.512
0.694
w/o MoAE
0.651
0.930
0.941
0.975
0.520
0.727
Astra (Ours)
0.669
0.939
0.945
0.989
0.531
0.747
conditioning, noise- augiened memory, and modular action encoding, the model can flexibly adapt to diverse tasks such as camera control, manipulation video prediction, long-horizon exploration, and autonomous driving, shown in Figure 7. This versatility arises from unifying temporal sistency with responsive action integration, making it a general-purpose framework for simulating, interacting with, and editing dynamic environments.
Autonomous driving, We extend Astra to autonomous driving using the nuScenes dataset (Caesar al., 2020), which provides multi-view videos and diverse traffic contexts. Given ego-vehicle ob- servations and discrete control actions (c.g., turn left, move forward), Astra generates realistic future driving videos that capture vehicle motion, road geometry, and agent interactions. Its combination of long-term coherence and precise action responsiveness enables interactive driving simulation.
Manipulation video prediction robotic settings, Astra predicts future video chunks conditioned on current observations and manipulation actions, simulating fine- mee-grained interactions such as grasp- ing or tool use with high fidelity and temporal coherence. These predictive rollouts support planning. policy learning, and safe exploration in robot learning.
Camera control. Astra supports interactive camera control through action signals specifying cam- trajectories such as panning and viewpoint shifts. Conditioned on these poses, it generates videos that follow instructed motions while maintaining spatial and temporal consistency. This enables con- trollable, cinematic camera movement for both creative and practical applications.
Multi-agent interaction. In Figure 8, we illustrate the ability to handle multi-agent scenarios through first-person driving example where the ego-vehicle overtakes two cars. Conditioned on a sequence of action commands, the model produces a coherent rollout that follows the specified trajectory while plausibly simulating the motions of surrounding vehicles. This highlights that the

---

Published as a conference paper at ICLR 2026
aa) Autonomous driving
(b) Manipulation video prediction
(c) Camera control
Figure 7: Extended applications of Astra. Our framework handles diverse scenarios: (a) tonomous driving, predicting long-horizon traffic dynamics from control inputs; (b) manipula- tion, conditioning robot actions object interactions; and (c) camera control, reflecting viewpoint changes in coherent videos. These demonstrate Astra's versatility for interactive world modeling.
Input image
Multi-agent driving interaction
Figure 8: Multi-agent interaction of Astra. Given specified action sequence, Astra generates smooth, realistic multi-agent interactions, such as an ego-vehicle overtaking other cars.
autoregressive denoising process, noise-augmented memory, and MoAE-based action conditioning together enable stable multi-agent interactions with accurate action following.
CONCLUSION
In this work, we present Astra, simple yet effective framework for building interactive world models that unify real-world video prediction with precise action conditioning. By equipping pre- trained video diffusion backbone with a lightweight action-aware adapter, noise-augmented mem- ory for balancing history and responsiveness, and mixture of action experts for versatile control, our model achieves interactive, consistent, and versatile video generation across diverse real-world scenarios. Extensive experiments show long-temn term consistency, visual fidelity, and instruction fol- lowing. We believe Astra offers a practical path toward more general, scalable simulators for world modeling in exploration, robotics, autonomous driving, and embodied intelligence.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62125603, Grant 62336004, Grant 62321005, and in part by the Beijing Natural Science Foundation under Grant No. L247009.
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501 (ttv:2501.0357.. 2025.
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang, Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controllee generative rendering from single video. arXiv preprint arXiv:2503. /647, 2025.
Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. In CVPR, pp. 15791-15801, 2025.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominík Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics 2022.
transformer for real-world control at scale. arXiv preprint arXiv:22
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, et al. Video generation models as world simulators. URL 3, 2024.
hups::/openai. com/research/vider -ewerriiiddde---iiooonnee generation geweration-models--eeeeeeeee warieel------wodd----iiiiee
Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative inter- active environments. In ICML, 2024.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR. pp. 11621-11631, 2020.
Jun Cen, Chaohui Yu, Hangjic Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, al. Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.21538 2025.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. NeurIPS, 34:8780-8794 2021.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. CoRR, 2024.
Xianglong He. Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Biao Jiang, Mengyin An, Yangyang Ren, et al. Matrix-game 2.0: An open-source, real-time, and streaming interactive world model. arXiv preprint a50iv:2508. 13009, 2025.
Roberto Henschel. Levon Khachatryan, Hayk Poghosyan, Daniil Hayrapetyan. Vahram Tadevosyan, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. In CVPR, pp. 2568 2568-2577, 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33: 6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. NewrIPS, 05:8633-8646 2022.

---

Published as a conference paper at ICLR 2026
Siqiao Huang, Jialong Wu, Qixing Zhou, Shangchen Miao, and Mingsheng Long. Vid2world: Craft- ing video diffusion models to interactive world models. arXiv preprint arXiv:2505. 2025.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, pp. 21807-21818, 2024.
Zhen Li, Chuanhao Li, Xiaofeng Mao, Shaoheng Lin, Ming Li, Shitian Zhao, Zhaopan Xu, Xinyue Li, Yukang Feng, Jianwen Sun, et al. Sekai: video dataset towards world exploration. arXiv preprint arXiv:2506. 150.. 2025a.
Zhengqi Li. Richard Tucker, Forrester Cole, Qianqian Wang. Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust struc- ture and motion from casual dynamic videos. In CVPR, pp. 10486-10496, 2025b.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention. arXiv preprint arXiv:2402.08266 2024.
Xinhao Liu, Jintong Li. Yicheng Jiang, Niranjan Sujay. Zhicheng Yang, Juexiao Zhang, John Abanes, Jing Zhang, and Chen Feng. Citywalker: Learning embodied urban navigation from 2025.
web-scale videos. In CVPR, pp.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 100001171,.05001, 2017.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03045 2024.
Xiaofeng Mao, Shaoheng Lin, Zhen Li, Chuanhao Li, Wenshuo Peng, Tong He, Jiangmiao Pang, Mingmin Chi, Yu Qiao, and Kaipeng Zhang. Yume: An interactive world generation model. arXiv preprint arXiv:2507. 177477444 2025.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In ICRA, pp. 6892-6903, IEEE, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pp. 4195- 4205, 2023.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- image segmentation. In MICCAI, pp. 234-241. Springer, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020.
Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, and Vincent Sitzmann. History-guided video diffusion. arXiv preprint arXiv:2502.06764 2025.
Hansi Teng, Hongyu Jia, Lei Sun, Lingzhi Li, Maolin Li, Mingqiu Tang, Shuai Han, Tianning Zhang, WQ Zhang, Weifeng Luo, al. Magi-1: Autoregressive video generation at scale, arXiv preprint arXiv:2505.13211 2025.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming 7hao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.203/1 2025.
Jiahao Wang. Yufeng Yuan, Rujie Zheng, Youtian Lin, Jian Gao. Lin-Zhuo Chen, Yajie Bao, Yi Zhang, Chang Zeng, Yanxi Zhou, et al. Spatialvid: A large-scale video dataset with spatial annotations. arXiv preprint 2500095009006666 2025.
12

---

Published as a conference paper at ICLR 2026
Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, and Jiwen Lu. World- dreamer: Towards general world models for video generation via predicting masked tokens. arXiv preprint 00XX::24000.0985,, 2024.
Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. NeurIPS, 37:68082-68 119,, 2024.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. 2025.
Lvmin Zhang and Maneesh Agrawala. Packing input frame context in next-frame prediction models for video generation, 2025. URL https: /atpiiv.//ariiveeaa org/ abs /504. 12626.
Zangwei Zheng, Xiangyu Peng. Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. URL https://github. com/tpcaitech/Open-SSor 2024.
13

---

Published as a conference paper at ICLR 2026
A MORE EXPERIMENTAL DETAILS
A.1 DATASETS
We use the training data sampled from the nuScenes (Caesar et al., 2020), Sekai (Li et al., 2025a), SpD (Wang et al., 2025), Multi-Cam Video (Bai et al., 2025) and RT-1 (Brohan et al., 2022). To ensure balanced exposure across these datasets, we apply a set of sampling weights that control the frequency with which data from each source is drawn during training. A summary of dataset statistics is provided in Table 1, with detailed descriptions reported as follows:
nuScenes:
nuScenes is autonomous vehicle dataset collected from AV approved for testing public roads and it contains the full 360 sensor suite (lidar, images, and radar). It comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. We use its 7 dimensions pose parts as our action.
Sekai is high-quality first-person view worldwide video dataset with rich annotations for world exploration. It consists of over 5,000 hours of walking or drone view (FPV and UVA) videos from over 100 countries and regions across 750 cities. The action of this dataset is the camera pose.
Sekai:
SpatialVID: SpatialVID is a dataset consisting of large corpus of in-the-wild videos with diverse scenes, camera movements and dense 3D annotations such per-frame camera poses, depth, and motion instructions. SpatialVID has a total of 7,089 hours of dynamic content. We use the 7- dimensional camera parts as our action.
RT-1: RT-1 is large, diverse dataset of robot trajectories that includes multiple tasks, objects and environments. It contains over 130k individual demonstrations constituting over 700 distinct task instructions using large variety of objects. We use its 7 dimensions ee-space pose parts as our action. We use Open X-Embodiment (O'Neill et al., 2024) version of the RT-1 dataset.
Multi-Cam Video: Multi-Cam Video is synthetic dataset rendered with multiple cameras cap- turing the same scene simultaneously. Animated characters are placed in diverse 3D environments, while cameras follow predefined trajectories to simulate synchronized shooting. We use 10K videos with detailed camera annotations as action signals.
A.2 TRAINING DETAILS
We initialize our model using the pre-trained weights of the Wan-2.1 base model (Wan al., 2025). Our training is run on 8 GPUs (80G) with a per-GPU batch size of 1. We train our model with AdamW (Loshchilov & Hutter, 2017) optimizer and a learning rate of 1e-5 for 30 epochs. Our training runs take approximately 24 hours to converge. All vidcos are resized and cropped to the training resolution (480 x 832). The model is trained on the latent space produced by a 3D VAE. In pixel space, the count of condition frames is randomly sampled from the range [1, 128), whereas the number of target frames is consistently set to 33.
A.3 MODEL ARCHITECTURE
Our framework builds on Wan-2.1 (Wan et al., 2025), a large-scale video diffusion model composed of 30 stacked flow transformer (DiT-style) blocks, each containing multi-head self-attention and feed-forward layers with residual connections. Wan-2.1 serves as a strong pre-trained backbone, providing rich generative priors for high-quality video synthesis. To enable interactive control, we introduce two lightweight yet effective extensions: ACT-Adapter and Mixture of Action Experts (MoAE). The ACT-Adapter is implemented as a single linear layer inserted after each self-attentio block. It is initialized as an identity mapping and fine-tuned jointly with the attention parameters, allowing the model to inject action features into the latent space while preserving stability of the pre-trained weights. In parallel, MoAE provides modular action encoding mechanism. It consists of linear router that projects heterogeneous action modalities into a shared space, followed by a set of MLP-based experts specialized for different action types. Specifically, we support camera control actions represented as 7 or 12-dimensional vectors, robotic actions represented as 7-dimensional vectors, and navigation commands expressed through discrete keyboard and mouse inputs. The routed outputs are combined into a unified representation that is fed into the ACT-Adapter, ensuring both precision and versatility. Together, these components augment Wan-2.1 1 into autoregressive
14

---

Published as a conference paper at ICLR 2026
Table A: Quantitative action-alignmeen comparison. We complement the human-rated instruction-followwe metric by reporting rotation and translation errors that directly measure how well generated camera motions align with the commanded actions.
Method
RotEr 4
TransErr
Instruction Following
Imaging Quality 1
Wan-2.1 (Wan al., 2025)
2.96
7.37
0.061
0.691
YUME (Mao et al., 2025)
2.20
5.80
0.268
0.741
MatrixGame (He et al., 2025)
2.25
5.63
0.652
0.748
NWM (Bar et al., 2025)
2.47
6.13
0.311
0.635
Astra (ours)
1.23
4.86
0.669
0.747
denoising model capable faithfully following actions while maintaining long-horizon temporal coherence. During inference, set the scale for action-free guidance (AFG) to be 3.0.
A.4 METRIC DETAILS
We follow YUME (Mao et al., 2025) to use human evaluation and VBench (Huang et al., 2024) met- rics assess the performance of our model. For instruction following, automated sumera-motion estimators (e.g.. MegaSaM (Li et al., 2025b)) can provide approximate motion labels, but their pre- dictions often suffer from inaccuracies and quantization errors, especially under complex scene ge- ometry or fast motion. To ensure reliable assessment, we therefore rely purely on human evaluation. Specifically, we recruit 20 users to inspect each generated sequence together with the corresponding action command. The instruction-follot following score is computed as the ratio of users who agree that the generated motion faithfully reflects the specified action direction and action type. For more ob- follow prior works (Bai et al., 2025; He et al., 2024) and measure how closely
jective evaluation, the camera motion in the generated video matches the ground-trutt trajectory. using MegaSaM to estimate camera poses. As shown in Table A, Astra achieves lower rotation and translation than existing methods, consistent with the human-rated instruction-followe results. Together, these metrics offer fine-graines quantitative view of action alignment, further demonstrating the precise interactive control enabled by our model.
A.5 PARAMETER ANALYSIS
Astra is designed to be lightweight, adding far fewer parameters than prior interactive world mod- els such YUME (Mao et al., 2025) and MatrixGame (He et al., 2025). As shown in Table B, MatrixGame introduces heavy cross-attention modules and large action encoders, resulting in sig- nificant training and inference overhead, while YUME depends In contrast, Astra adds only two small components: ACT-Adapter (a single linear layer after each self-attention block) and MoAE (a lightweight linear router plus small MLP experts, with only one expert active per step).
much larger 13B backbone.
These additions contribute only minor increase in parameters, making Astra the most parameter- efficient model among the compared methods. Because the backbone remains frozen and the added modules are shallow, the computation cost nearly unchanged, enabling fast, stable training and long-horizon horizon rollout without heavy architectural modifications. Overall, Astra achieves strong action- condinncondiiioned performance with the lowest parameter and compute overhead its class.
B COMPARISON WITH EXISTING WORLD MODELS
Table compares four representative world models-Wan-22. (Wan al., 2025), MatrixGame (He et al., 2025), YUME (Mao et al., 2025), and Astra Astra-across three key dimensions: domain, control modality, and interaction horizon. In terms of domain, Astra designed for general-purpose scenar- ios, showcasing their versatility in wide range of applications. MatrixGame,, however, is tailored specifically for game-related contexts, limiting its use to game-specific environments. YUME fo- cuses on walking-specifts domains, which is niche area compared to the general-purpose methods. For control modalities, Wan-2.1 relies solely on text input, reflecting a language-driven paradigm. MatrixGame and YUME adopt traditional keyboard and mouse controls. Astra, enabled by MoAE, supports multiple modalities ciaiiiiie-innnuunnng camera input, keyboard/mouse, and robot pose-provee pose-providing more flexible and intuitive interaction. Finally, regarding the interaction horizon, Wan-2,1 and Ma-
15

---

Published as a conference paper at ICLR 2026
Table B: Parameter comparison. Astra introduces the smallest parameter overhead among all methods, adding only lightweight adapters while preserving the efficiency of the frozen backbone.
Method
Base model
Trainable Params.
Note
NVM (Bar et al., 2025)
CDiT-XL. Wan2. 1-1BB
1B
Full tuning
YUME (Mao et al., 2025)
~ 14B
Full tuning
MatrixGame (He al., 2025)
Wan2.1-1.3B
1.8B
Full tuning. cross-attn adapters
Astra (ours)
W-n2.1-1.3B
366.8M
Tuning adapters & self-atm.
Table C: Comparative overview of various world model methods, detailing their respective domains of application, supported control modalities, and interaction horizons.
Method
Domain
Control
Interaction Horizon
Wan-2.1 (Wan et al., 2025)
General
Text
A few seconds
MatrixGame (He et al., 2025)
Game- Game-specific
Keyboard / Mouse
A few seconds
YUME (Mao et al., 2025)
Keyboard / Mouse
8- 10 seconds
Camera;
Astra (Ours)
General
Keyboard Mouse;
8-10 seconds
Robot pose
trixGame are limited to short spans of a few seconds. YUME extends this to 8-10 seconds, and As- tra matches this longer horizon through noisy memory and the input-packing technique of (Zhang & Agrawala, 2025). Combined with its multi-modal control and general-purpose domain, Astra emerges the most comprehensive solution among the compared methods.
C MORE RESULTS
C.I EXTENDED APPLICATIONS
As shown in Figure 7, Astra framework is designed to generalize across wide range of in- teractive video prediction tasks, demonstrating strong adaptability in domains such as autonomous driving, robotic manipulation, and camera control. In driving scenarios, Astra generates realistic long-horizon rollouts that capture complex road dynamics while responding accurately to control signals like steering or acceleration. For manipulation, it predicts object interactions conditioned on robot actions, enabling fine-grained and physically consistent outcomes. In camera control, Astra follows viewpoint instructions such as panning, zooming, or rotation, while maintaining temporal and visual coherence. Together, these applications highlight Astra's 's versatility and effectiveness as unified world modeling framework capable of handling heterogeneous action modalities in diverse real-world settings.
C.2 OUT-OF-DOMAIN GENERALIZATION
We further evaluate Astra on out-of-domain (OOD) scenes- including indoor environments, styl- ized anime videos, and even Minecraft gameplay- none of which are present in the training distri- bution. Across all cases in Figure A, Astra produces coherent, action condinnnnd rollouts: given camera or navigation commands, it generates futures that accurately follow the instructed motion while maintaining global structure and temporal consistency. In indoor scenes, the model handles complex layouts and viewpoint shifts; in anime clips, remains responsive despite fast and unpre- dictable dynamics; and in Minecraft, it adapts to drastically different textures and rendering styles while still executing the intended camera movement. These results show that Astra's autoregressive denoising framework, noise- augmented memory, and action-aware conditioning generalize effec- tively under substantial distribution shift. We also present two examples applying different complex action sequences to the same scene (Rows 3 and in Figure A), both of which are followed faithfully.
scenes-including
C.3 COMPARISONS WITH MORE METHODS
To more comprehensively situate Astra within the broader landscape of visual world modeling, we provide extended quantitative and qualitative comparisons with additional proaches. Specifically, we compare against methods such as Navigation World Models (NVM) (Bar
ap-
16

---

Published as a conference paper at ICLR 2026
Input image
Action-driven video generation
Figure A: Out-of-domain generation results of Astra. Astra generalizes to scenes not seen during training, including indoor environments, Minecraft worlds, and animation-style scenes, producing coherent futures that follow camera navigation commands. The last two rows show two distinct complex action sequences executed within the same scene.
WAN
Our
NVM
Ours
Input image
Action-driven video gemerution
Figure B: Qualitative comparison with NVM (Bar al., 2025). Astra consistently produces visu- ally coherent results while faithfully following action inputs.
et al., 2025) in Figure B and Table A, which are representative of recent approaches integrating multimodal inputs and action-conditioned predictions. Our results demonstrate that Astra achieves competitive or superior performance across key metrics, including visual fidelity, temporal consis- tency, and action responsiveness. While direct numerical comparisons are limited by differences in evaluation protocols and datasets, qualitative visualizations show that Astra generates long-horizon, coherent videos that faithfully follow user-specified actions, outperforming prior methods captur- ing the causal dynamics of the environment. These comparisons highlight Astra's ability to combine accurate action conditioning with long-term rollout stability, confirming its effectiveness as general interactive world model across diverse scenarios. It is worth noting that many of the most advanced world models, such as Genie-3 (Bruce et al., 2024), are currently closed-source or provide only limited evaluation interfaces, making direct numerical comparison infeasible.
C.4 QUANTITATIVE COMPARISONS ON LARGER DATASETS
To further address concerns about evaluation scale, conduct an expanded study a larger held- out set from the Walker (Liu et al., 2025) dataset. City Walker is an egocentric urban navigation corpus collected from t-thh-wildd YouTube walking videos with pose estimates. We sample 100
17

---

Published as a conference paper at ICLR 2026
Astra consistently achieves higher visual quality and more reliable action following when evaluated on fully unseen scenes,
Table D: Quantitative comparison on City Cityer dataset.
Instruction Following
Subject
Background Consistency
Motion Smoothness f
Aesthetic Quality
Imaging Quality
Method
Consisteney 1
Wan-2.1 (Wan al., 2025)
0.084
0.827
0.843
0.913
0.417
0.632
MatrixGame (He al., 2025)
0.247
0.923
0.939
0,946
0.426
0.653
Yume (Mao et al., 2025)
0.619
0.933
0.927
0.972
0.511
0.628
Astra (Ours)
0.641
0.948
0.944
0.983
0.554
0.695
Imaging quality Action following
1.0
1.0
0.8
0.8
Imaging quality
Action following
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
10 20
30
40
60
80
Context video length
Figure C: Effect of visual inertia. As the history length increases, video quality improves, but the action-following score drops sharply, illustrating the visual inertia phenomenon.
scene images, each paired with its future ground-truth action trajectory, producing 100 long-horizon rollouts that robustly test cross-scene generalization and action adherence. Across this larger evalua- tion, Astra consistently achieves the best action-ie action-following following accuracy and video quality, outperforming all baselines in metries such as motion consistency, temporal stability, and VBench perceptual scores (Table D). These results confirm that Astra's 'ss strong performance is not an artifact of small-set eval- uation, and that the model maintains robust generalization and reliable action responsiveness even under substantially expanded test conditions.
D DISCUSSION ON VISUAL INERTIA
Previous work (Zhang & Agrawala, 2025) has shown that longer video context can improve gen- eration quality. However, interactive world modeling, observe that increasing context length can reduce action responsiveness. We term this phenomenon visual inertia-the model's tendency over-rely on past visual frames while neglecting user actions. To examine this, we train Astra with different history lengths without our proposed noisy-memor mechanism. As shown in Fig- ure C, the followng score drops substantially as context length increases, indicating that overly clean and long visual histories can dominate the model's decision process. This motivates our noise-augmented memory design, which intentionally reduces visual dominance and encourages the model to integrate both historical context and action signals when generating future frames.
E LIMITATIONS
Despite the promising performance of Astra, our framework still faces limitations in inference effi- ciency. Since it builds on diffusion-based generation with autoregressive rollouts, producing long- horizon interactive videos requires multiple denoising steps per frame, making real-time deployment challenging. This constraint limits its applicability in latency-sensitive scenarios such as online con- trol or interactive robotics. To address this, future work could explore distillation student-teacher compression strategies that retain the fidelity and responsiveness of Astra while reducing inference cost, thereby paving the way for lightweight, real-time world modeling.
18


Reviews:
[{"review_id": "FS3A0rAUuF", "reviewer": "Reviewer_toMU", "summary": "In this paper, author proposes Astra, an interactive world model that extends pre-trained diffusion models for long-horizon, action-conditioned video prediction. \n\nThe core contribution lies in three components in an auto-regressive denoising framework: (1) an action-aware adapter that injects action signals into the latent space of a pre-trained diffusion model, (2) a noise-augmented history memory mechanism that balances temporal consistency and action responsiveness, and (3) a mixture of action experts that routes heterogeneous action modalities to specialized experts. \n\nAstra is evaluated on self-proposed benchmark consisting of diverse datasets and demonstrates some improvements in long-range prediction stability compared to state-of-the-art models.", "questions": "**Question (1)**: In the evaluation of Table 1, Wan-2.1 [6] is used as a baseline. How is Wan-2.1 adapted to accept continuous action inputs during evaluation?\n\n**Question (2)**: YUME [1] also extends Wan-2.1 [6] for interactive video prediction. Could the authors explain why Wan-2.1 is chosen as the base model instead of YUME?\n\n**Question (3)**: Astra-Bench uses both MegaSaM [7] and human evaluations for “instruction following.” Could the authors clarify how the numerical values of \"instruction following\" in Tables 1 and 2 are computed?\n\n**Question (4)**: All experiments train on a dataset mixture across five domains. There is no zero-shot evaluation on held-out domains. Does the model generalize to completely new environment unseen during training?\n\n**Question (5)**: Authors claim that increasing the length of history improves temporal consistency but weakens responsiveness (Line 257). However, no supporting quantitative data are provided. Could the authors supply such evidence?\n\n**Typographical Error**: “Eperimental” → “Experimental” (Line 594).\n\n[6] Wan, Team, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen et al. \"Wan: Open and advanced large-scale video generative models.\" arXiv preprint arXiv:2503.20314 (2025).\n\n[7] Li, Zhengqi, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. \"MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos.\" In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10486-10496. 2025.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Strength (1)**: The paper is well-organized. Authors explain the core ideas with clear diagrams and concrete algorithmic descriptions. \n\n**Strength (2)**: Astra is a single model across multi-modal action spaces, covering camera poses, keyboard/mouse inputs, and robot poses. \n\n**Strength (3)**: The proposed solutions exhibit several elegant and practical design choices:\n- The action-free guidance mechanism offers a simple, original mechanism to amplify action effects without heavy architectural changes.\n- The noise-augmented history memory is an elegant, parameter-free training strategy to reduce “visual inertia” and force the model to rely more on action signals, improving responsiveness without modifying the backbone.\n\n**Strength (4)**: The authors conduct evaluations across multiple domains, including autonomous driving, egocentric, and robotic settings, showcasing reasonable coverage.", "weaknesses": "**Weakness (1)**: The definition and formulation of action signals are insufficiently specified. The paper does not clearly describe how different types of actions (e.g., camera poses, keyboard/mouse inputs, robot poses, etc.) are represented, parsed, and projected to the action encoder.\n\n**Weakness (2)**: A comparable method, YUME [1], is not discussed in Section 2 (Related Work). The paper does not clearly articulate how Astra differs from or improves upon YUME, which weakens presentation.\n\n**Weakness (3)**: Experimental validation of design choices is limited. For example, no ablation study isolates the contribution of the Mixture of Action Experts (MoAE). A comparison against a simpler variant without a gating network would clarify whether MoAE provides meaningful gains.\n\n**Weakness (4)**: Quantitative comparisons with existing world modeling methods are lacking. Although authors cite several relevant works [2, 3, 4, 5], Astra is not evaluated against them, making it difficult to assess the model's relative performance and significance. \n\n**Weakness (5)**: Although Astra is positioned as a general interactive world model trained on a mixture of five datasets, all evaluations are conducted on held-out data drawn from these same domains. It remains unclear whether the model generalizes to unseen environments.\n\n**Weakness (6)**: The paper combines pose tracking with human evaluation to assess \"instruction following\" in Astra-Bench, but the metric definition, aggregation procedure, and scoring protocol are not clearly specified. It is difficult to compare against future work. \n\n[1] Mao, Xiaofeng, Shaoheng Lin, Zhen Li, Chuanhao Li, Wenshuo Peng, Tong He, Jiangmiao Pang, Mingmin Chi, Yu Qiao, and Kaipeng Zhang. \"Yume: An interactive world generation model.\" arXiv preprint arXiv:2507.17744 (2025).\n\n[2] Cen, Jun, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li et al. \"WorldVLA: Towards Autoregressive Action World Model.\" arXiv preprint arXiv:2506.21539 (2025).\n\n[3] Huang, Siqiao, Jialong Wu, Qixing Zhou, Shangchen Miao, and Mingsheng Long. \"Vid2World: Crafting Video Diffusion Models to Interactive World Models.\" arXiv preprint arXiv:2505.14357 (2025).\n\n[4] Bar, Amir, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. \"Navigation world models.\" In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15791-15801. 2025.\n\n[5] Bruce, Jake, Michael D. Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai et al. \"Genie: Generative interactive environments.\" In Forty-first International Conference on Machine Learning. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5hwiiiZd76", "reviewer": "Reviewer_AW9Q", "summary": "This paper introduces Astra, a framework for interactive world modeling that generates long and temporal coherence video sequences across diverse scenarios. Astra enhances a pre-trained video model with an light-weight action-aware adapter for precise action conditioning, a noise-augmented history memory during training to ensure long-term consistency, and a mixture of action experts to effectively handle diverse action inputs.", "questions": "1. In Section 3.3, the explanation of the types of random noise and blur used is unclear. Please provide a more detailed description.\n2. It was not properly analyzed the lightweight action-aware adapter model complexity. Please provide a more detailed description and comparison.\n3. In Figure 6, compared to YUME, the results from Astra appear to exhibit some color shift. Could the authors explain the cause of this phenomenon?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper uses a lightweight action-aware adapter for precise action conditioning.\n2. Astra achieves good responsiveness and is able to generate long, temporally coherent video sequences by employing a noise-as-mask strategy during training.\n3. Astra employs a mixture of action experts to effectively adapt to diverse scenarios and handle various types of action inputs.", "weaknesses": "1. Mixture of action experts idea is similar to [1, 2] and action-aware adapter is similar to [3, 4]. Please provide a conceptual comparison with these reference.\n2. The paper does not thoroughly analyze the underlying reasons why the noise-as-mask strategy enables the generation of long, temporally coherent video sequences.\n3. The paper does not explain why the router network performs so well across diverse scenarios and with various types of action inputs.\n\n[1] Mixture of Action Expert Embeddings: Multi-Task ACT\n\n[2] DriveMoE: Mixture-of-Experts for Vision-Language-Action Model in End-to-End Autonomous Driving\n\n[3] Long-Context Autoregressive Video Modeling with Next-Frame Prediction\n\n[4] Epona: Autoregressive Diffusion World Model for Autonomous Driving", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vz6piu3U8j", "reviewer": "Reviewer_ynyx", "summary": "The paper addresses a key limitation in current video generation and world modeling approaches which are lack of interactivity and long-horizon consistency. While diffusion-based models are able to generate high-fidelity videos from text or images, they often produce short, self-contained clips, fail to respond dynamically to user actions or control signals, and struggle with error accumulation in long rollouts. The paper addresses this by building a general-purpose, interactive world model that can simulate realistic futures across diverse domains (e.g., driving, robotics, exploration) while maintaining responsiveness to actions and temporal coherence. It proposes a lightweight module named as ACT-Adapter that injects action signals directly into the latent space of a pre-trained video diffusion backbone, a training strategy of noise-augmented history memory training which corrupts historical frames to reduce over-reliance on visual context and improve responsiveness., and mixture of action experts that handles multiple action modalities of camera, robot pose, and keyboard/mouse. Astra-bench is the benchmark suite used for evaluation which spans across multiple datasets to evaluate the visual quality and instruction-following performance.", "questions": "There seems to be some contradiction in training details. Section 4.1 mentions that the number of target frames is fixed to 33. Appendix A.2 mentions that the number of target frames is set to 32. Can the authors please clarify this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the limitation of passive video generation by showing interactive world modeling where video synthesis is conditioned on external actions .\n2. The framework proposes a single, general-purpose model by training on a diverse datasets of driving, robotics, exploration and handles heterogeneous action types via a Mixture of Action Experts.\n3. The paper proposes a noisy memory training strategy which forces the model to reply on action signals and not over-rely on past visual information.", "weaknesses": "1. The ACT-Adapter seems be to showing a minimal performance improvement in Table 2. The ablation study in Table 2 shows it provides a score of 0.669 on Instruction Following, while a cross attn. adapter achieves 0.642, suggesting the performance gain of the new adapter is relatively small.\n2. The comparison to baseline methods in Table 1 does not reflect a fair comparison.\na). Since Wan2.1 is the pre-trained backbone of Astra, it would be an ablation of the paper instead of baseline.\nb). MatrixGame and YUME are described as domain-specific (\"game-specific\" and \"walking-specific\"). Since Astra is trained on a general mixture of data from multiple domains (driving, robotics, walking), the comparison of domain-specific with a general model does not seem to show a fair comparison. It creates a question that is it the data that is giving a good performance or is it the architecture and training strategy helping with the performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XoVIR0wFho", "reviewer": "Reviewer_qwc8", "summary": "The paper introduces Astra, a new framework for building a general-purpose, interactive world model. The central problem it addresses is that existing video generation models (like diffusion transformers) can create high-fidelity clips but lack true interactivity—they cannot generate long, coherent videos that dynamically and precisely respond to external user actions (e.g., camera controls, robot actions, or vehicle movements).", "questions": "See in Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper's primary strength lies in its proposal of a Astra framework, which skillfully combines the powerful generative capabilities of pre-trained diffusion models with an autoregressive, action-conditioned paradigm, effectively bridging the gap between high-fidelity video generation and real-time interactivity. The authors' core contributions are embodied in three innovations: (1) A lightweight ACT-Adapter for efficiently injecting action signals into the pre-trained model while preserving its knowledge; (2) An innovative \"noise-augmented memory\" strategy to overcome the \"visual inertia\" problem, forcing the model to prioritize responding to actions rather than simply repeating historical frames; (3) The introduction of a Mixture of Action Experts (MoAE) module, which enables flexible handling of heterogeneous action inputs from different domains, enhancing the model's generality. Experimental results validate the effectiveness of this design, with the model performing exceptionally well on the key \"Instruction Following\" metric.", "weaknesses": "**W1** The authors constructed a new benchmark, Astra-Bench, for evaluation. According to the paper, this benchmark is \"comprising 20 held-out samples from each dataset\". This scale is extremely small and likely insufficient to robustly evaluate the model's generalization capabilities, which could lead to biased evaluation results. \n**W2** The paper's title claims it is a \"General Interactive World Model\". It is questionable whether this is sufficient to support such a broad \"general\" claim. Out-of-domain scenarios with different physics or interaction types (e.g., fluid dynamics, complex object stacking, multi-agent interaction) are needed.\n**W3** Can interaction modalities from different domains all be mapped to control signals of the same dimension? For example, the degrees of freedom (DoF) of an embodied agent are far greater than the degrees of freedom of autonomous driving.\n**W4** The first embodied agent interaction in the appendix not only has blurry artifacts, but also exhibits incorrect affordances and false interactions, which are challenges that remain to be addressed.\n\nTypos:\nThere is a clear spelling error in the title of Appendix A: \"A MORE EPERIMENTAL DETAILS\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)