Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
Authors: Moo Jin Kim
Yihuai Gao
Tsung-Yi Lin
Yen-Chen Lin
Yunhao Ge
Grace Lam
Percy Liang
Shuran Song
Ming-Yu Liu
Chelsea Finn
Jinwei Gu
Venue: ICLR 2026 Poster (2026)
Keywords: world models
robotics
manipulation
model-based planning
imitation learning
video generation
Primary area: applications to robotics, autonomy, planning
TL;DR: Cosmos Policy is a simple framework for adapting a large pretrained video model (Cosmos-Predict2) into a state-of-the-art robot policy that can generate actions, future states, and values and plan action trajectories with high success rates.

Abstract: Recent video generation models demonstrate remarkable ability to capture complex physical interactions and scene evolution over time. To leverage their spatiotemporal priors, robotics works have adapted video models for policy learning but introduce complexity by requiring multiple stages of post-training and new architectural components for action generation. In this work, we introduce Cosmos Policy, a simple approach for adapting a large pretrained video model (Cosmos-Predict2) into an effective robot policy through a single stage of post-training on the robot demonstration data collected on the target platform, with no architectural modifications. Cosmos Policy learns to directly generate robot actions encoded as latent frames within the video model's latent diffusion process, harnessing the model's pretrained priors and core learning algorithm to capture complex action distributions. Additionally, Cosmos Policy generates future state images and values (expected cumulative rewards), which are similarly encoded as latent frames, enabling test-time planning of action trajectories with higher likelihood of success. In our evaluations, Cosmos Policy achieves state-of-the-art performance on the LIBERO and RoboCasa simulation benchmarks (98.5\% and 67.1\% average success rates, respectively) and the highest average score in challenging real-world bimanual manipulation tasks, outperforming strong diffusion policies trained from scratch, video model-based policies, and state-of-the-art vision-language-action models fine-tuned on the same robot demonstrations. Furthermore, given policy rollout data, Cosmos Policy can learn from experience to refine its world model and value function and leverage model-based planning to achieve even higher success rates in challenging tasks. We release code, models, and training data at https://research.nvidia.com/labs/dir/cosmos-policy/.

Published as a conference paper at ICLR 2026
COSMOS POLICY: FINE-TUNING VIDEO MODELS FOR VISUOMOTOR CONTROL AND PLANNING
Moo Jin Kim 1.2
Yihuai Gao 1.
Tsung-Yi Lin'
Yen-Chen Lin'
Yunhao Ge
Grace Lam
Percy Liang*
Shuran Song
Ming-Yis Yu Liu!
Chelsea Finn²
Jinwei Gu²
NVIDIA
Stanford University
Cosmos Policy
Actions,
[a. 4/+1 9411-11 R
Current State,
Fine-tune video foundation model (NVIDIA Coemos)
Current propri: q. E R- and
Cument images:
Future State, s'
Difusion Transformer
Future propro: &.w & Re and Futur image:
Derobved lhhers
anbeddd
Foodiorward
Pirmay
Task Description
Cross altendoo
"put candies in bowl"
Sell-attention
Value, V(s)
Ep R 4 1 =
Figure 1: We present Cosmos Policy, a state-of-the-art robot policy fine-tuned from the NVIDIA Cosmos- Predict2-2B video foundation model. Cosmos Policy handles multimodal inputs and multi-view camera images and predicts (I) robot action chunk, (2) future state (represented by robot proprioception and image observa- tions), (3) value (expected rewards-to-p to-dss-to-ooooo at future state). No architectural changes are made to the base video model, and modalities are jointly modeled through video diffusion learning objective.
ABSTRACT
Recent video generation models demonstrate remarkable ability to capture com- plex physical interactions and scene evolution over time. To leverage their spa- tiotemporal priors, robotics works have adapted video models for policy learning but introduce complexity by requiring multiple stages of post-training and new ar- chitectural components for action generation. In this work, we introduce Cosmos Policy, simple approach for adapting a large pretrained video model (Cosmos- Predict2) into an effective robot policy through a single stage of post-training on the robot demonstration data collected on the target platform, with architectural modifications. Cosmos Policy learns to directly generate robot actions encoded as latent. frames within the video model's latent diffusion process, harnessing the model's pretrained priors and core learning algorithm to capture complex action distributions. Additionally, Cosmos Policy generates future state images and val- ues (expected cumulative rewards), which are similarly encoded as latent frames, enabling test-time planning of action trajectories with higher likelihood of suc- cess. In our evaluations, Cosmos Policy achieves state-of-the--rt performance on the LIBERO and RoboCasa simulation benchmarks (98.5% and 67.1% average success rates, respectively) and the highest average score in challenging real-world bimanual manipulation tasks, outperforming strong diffusion policies trained from scratch, vidco model-based policies, and state-of-the- -hr vision- lssion-anguago-ace models fine-tuned on the same robot demonstrations. Furthermore, given policy rollout data, Cosmos Policy learn from experience to refine its world model and value function and leverage model-based planning to achieve even higher suc- cess rates in challenging tasks. We release code, models, and training data htttps:/ nwidia com/ hoom/labe labs dir/ cttpc/cooospooiii...o.
INTRODUCTION
Correspondence Moo Jin Kim <mooj nkêcs. stan fdud.

---

Published as a conference paper at ICLR 2026
Large pretrained video generation models have shown innrinmmpesiii impressive ability to generate physically plau- sible and temporally coherent videos (NVIDIA et al., 2025: Wan et al., 2025; Yang et al., 2024; Bao et al., 2024; Kong et al.. 2024; Zheng et al., 2024). Unlike pretrained vision- vision-language models- which learn semantic concepts from static image-text pairs and have been popularized as robot policy backbones by recent vision-language-actino
(VLA) model research ((Brohan et al., 2023; Kim et al., 2024; Intelligence et al., 2025; Li et al., 2025b)) -pretrr-re video generation models learn temporal causality, implicit physies, and motion patterns from millions of videos. These spa- tiotemporal priors hold significant value for robotics applications. In this work, we explore how to effectively leverage video models for robotic control and how they can incorporate policy rollout data to refine their world models and enable more effective planning.
Prior works have made significant progress on adapting video models for robotic manipulation, leveraging both robot action data and "action-less" Internet video data to train generalizable policies and perform new tasks with small amounts of demonstrations (Liang et al., 2025; Zhong et al,, 2025; Hu et al., 2024; Liao et al., 2025; Unitree, 2025; Feng et al., 2025; Yang et al., 2025; Wang et al., 2025). However, these works often require multiple training stages (e.g., video fine-tuning followed by action module training) and introduce new architectural components, such as separate action diffusers or inverse dynamics models. Other works avoid these complexities by training unified video-action models (Li et al., 2025a; Zhu et al., 2025), but they do not leverage pretrained video models due to their custom design, limiting their ability to capitalize on the spatiotemporal priors.
In this work, we address these limitations with Cosmos Policy: effective robot policy that is adapted from a pretrained video model (Cosmos-Prediet2-25 (NVIDIA et al., 2025)) through a single stage of post-training on robot demonstrations. Unlike prior works which carefully design separate action modules and algorithms,, Cosmos Policy makes no architectural modifications and instead leverages the pretrained model's core learning mechanism to capture action distributions. Since video models are effective at modeling complex, high- nigh-dimensional, multimodal distributions and can generate temporally coherent videos with hundreds of frames, we hypothesize that their learning algorithms are well-suited for representing actions alongside other modalities. Following this reasoning, we directly fine-tune a video model to simultancously generate robot actions, future state images, and future state values (expected total cumulative rewards), all of which we encode as latent frames within the model's ' latent diffusion sequence. With future state and value predic- tions, Cosmos Policy use best-of-N sampling to plan by generating candidate actions, imagining their resulting future states, ranking these states by predicted value, and executing the highest- action. This search process produces trajectories that are more likely to succeed at the task
Our main contribution is the Cosmos Policy approach for fine-tuning pretrained video models to in- corporate different modalities that enable visuomotor control and planning. We evaluate our method in two modes: first as a direct policy (without planning) and then with model-based planning using the future state and value predictions. As a direct policy, Cosmos Policy achieves new state of the art in both the LIBERO and RoboCasa simulation benchmarks (98.5% and 67. 1% average success rates, respectively). outperforming diffusion- based policies trained from scratch, video-based poli- cies (e.g., UVA, Video Policy), and even fine-tuned VLAs (e.g.. To.s. Open UniVLA, DP-V GROOT-N1.5). also achieves the highest average success rate (93.6%) among state-of-the-art policies in challenging real-world bimanual manipulation tasks. Further, when en- hanced with model-based planning, we observe a 12.5 percent higher task completion rate on aver- age in two challenging real-world manipulation tasks. In these experiments, we show that Cosmos Policy can incorporate past experiences from policy rollouts to refine its world model and value function and plan more effectively. Lastly, we compare our model-based planning approach model-free variant and study their relative advantages.
VLg,
2 RELATED WORK
Video-based robot policies. Recent works have made great strides in leveraging video models for manipulation. Some methods first fine-tune video models on robot data and then train separate action modules to predict robot actions from generated video frames (Liang et al., 2025; Zhong et al., 2025; Hu et al., 2024; Liao et al., 2025; Unitree, 2025; Feng et al., 2025; Yang et al., 2025; Wang et al., 2025; He et al., 2024). Other works train unified video-action models that jointly predict future frames and actions (Li et al., 2025a; Zhu et al., 2025). but these approaches do not leverage pretrained video models and thus do not benefit from their spatiotemporal priors. In contrast to these
2

---

Published as a conference paper at ICLR 2026
works, we propose single-stage fine-tuning approach that directly adapts pretrained video models to generate actions (as well as other modalities such as robot proprioceptive state and state values) within their native latent diffusion process.
Vision-languagg--attioo models. State-of-the-arr robotic manipulation policies increasingly lever- age large pretrained backbones. Vision- Aasion-language-attiion (VLA) models such as RT-2 (Brohan al., 2023), OpenVLA (Kim et al., 2024), T0.5 (Intelligence et al., 2025), UniVLA (Bu et al., 2025), and CogVLA (Li al., 2025b) fine-tune vision-language models on large-scale robotic imitation data, achieving strong performance across diverse manipulation tasks. While these methods exhibit strong generalization to various semantic concepts unseen in robotic interaction data, they leverage pretrained models that have mostly been trained on static image-text pairs rather than videos. In con- trast to these VL.As, we leverage pretrained video model that has learned spatiotemporal dynamics and implicit physics from predicting future frames for Internet-scale datasets. We hypothesize that this different pretrained backbone can serve as a strong foundation for low-level control policies.
World models and value functions. World models have been used in various ways in robotics and reinforcement learning, from classical model-predictive control to modern neural approaches. Influential works such as Dyna (Sutton, 1991), MBPO (Janner et al., 2019), TD-MPC (Hansen et al., 2022: 2023), and the Dreamer family of works (Hafner et al., 2019; 2020; 2023) demonstrate the benefits of integrating planning with learning, using learned dynamics models to improve decision making in various control tasks. Recent works have explored different paradigms: FLARE (Zheng al., 2025) adds learnable future tokens to diffusion transformer sequences to predict compact representations of future state, SAILOR (Jain et al., 2025) uses separate world and reward models with MPPI planning iteratively search for better actions and refine the base policy, and Latent Policy Steering (Wang et al., 2025) pretrains world models using optical flow as an embodiment- agnostic action representation and subsequently trains separate value function to steer the policy towards states with higher rewards. In contrast to these prior works that rely on separate modules for the policy, world model, and value function and typically train from models from scratch, we use a single unified architecture that serves simultaneously as the policy, world model, and value function and initialize from a pretrained video model.
3 PRELIMINARIES
Cosmos video model. The pretrained video model that serves as the initialization for Cosmos Policy is Cosmos-Predictt-2 2B-Vidd22World (NVIDIA et al., 2025), a latent video diffusion model that receives a starting image and textual description as input and predicts subsequent frames to create a short video. The model operates over continuous tokens encoded by the Wan2.1 spatiotemporal VAE tokenizer (Wan et al., 2025) and is trained using the EDM denoising score matching formulation (Karras al., 2022). The core training objective for the denoiser network Dg at noise level o is: L(Do, o) Exp,c,n [Do(xo + tn; c) xoll]] where Xo is clean VAE-encoded image sequence, c represents the textual description encoded as T5-XXL embeddings (Raffel et al., 2020), ~~w [N, I² is i.i.d. Gaussian noise used to corrupt xo, and Dy is diffuuson transformer (Peebles & Xie, 2023) that learns to recover the clean sample given the corrupted one. DA conditions on e via cross-attention and on a via adaptive layer normalization (Perez et al., 2018; Peebles & Xie, 2023). The Wan2.1 tokenizer compresses a video sequence of size (1 + 7')
x H x W x 3 into latent g W W.
x these resulting allow for conditioning on a single input image. During training, a conditioning mask is used to ensure that the first latent frame corresponding the input image remains clean (without noise) while subsequent frames are corrupted with noise.
sequence of size (1+ x H' x W' x 16, where 7 = I. H' H latent frames compose Xy above. The first frame undergoes no temporal compression
MDP formulation and imitation learning. We frame robotic manipulation tasks as finite-horizon
Markov decision processes (MDPs) defined by the tuple (S, A, T, R, H), where S is a set of states,
A is a set of actions, T : S x A + II(5') is the state transition function, R : S x A + I is the reward function, and H E N is the time horizon, with time steps t E {1,
H1. We
train a policy π : S + ((A)) to maximize rewards, using sparse rewards where R(s,, ax) = for t < H and terminal rewards R(SH aн) E Jo, 1]. We train policies via imitation learning on expert demonstrations containing state-action pairs. All policies predict action chunks sequences of actions for multiple timesteps- improve motion smoothness and success rates.
3

---

Published as a conference paper at ICLR 2026
conditioning subsequence
target subsequence
syd
images
latent injection of additional modalities: robot proprioception, actions, and value
WE
latent frammss
nolsed latant
frames
a
V(s)
Figure The latent diffusion sequence of Cosmos Policy. We illustrate latent frame injection the primary mechanism for adapting the pretrained Cosmos Cosmos-Predict2 into a policy that can predict robot actions, future states, and values without architectural changes. First, raw images are tokenized into latent frames (first row). Then, additional modalities (second row). The model then tasked denoise the noised latent frames conditioned the clean frames (third row). See Section 4.1 for more details. (Note: For simplicity, this figure does not depict certain imple- mentation details; see Figure for more detailed visualization.)
inserted directly into the latent frame sequence of the video diffusion model
World models and value functions. A world model T S A II(S) learns to predict the future state given current state and action, approximating the true environment dynamics. The value func- tion for policy æ at state represents expected discounted returns from s under æ. defined as (*)(s) ET>r ΣH E > a&) s; = E,-r (h aн) 8 = in the sparse reward setting, where is a discount factor that backpropagates the terminal reward through time. We simply use Monte Carlo returns in this work, labeling each transition in rollout with the ob- served return y RsH, ag) Note: To be precise, we acknowledge that the true state is not fully observable, and the world model predicts future observations (robot proprioception and camera im- ages). However, for notational simplicity and readability, we opt to use the term "state" and treat observations as approximations of the state.
COSMOS POLICY: ADAPTING VIDEO MODEL FOR CONTROL PLANNING
In this section, we discuss how adapt Prennst2 into unified model that predicts actions, future states, and values. We also discuss leveraging policy rollout data to enable effective planning.
4.1 LATENT FRAME INJECTION: INCORPORATING NEW MODALITIES
The original Dosmos-Prediche model takes as input an image and textual description to generate a short video for single camera view. It does not support robot proprioception as input, robot actions or state values as output, nor multiple camera views-all all of which are desired or required for manipulation policies.
Rather than designing new model components or making architectural modifications as done in prior works, we propose encode additional modalities as new latent frames that are directly injected into the video model's s latent diffusion sequence. Given a (1 + T") x H' x W x sequence of latent frames, which originally correspond to images in video, interleave new modalities (robot state, action chunk, and state values) and images from additional camera views by inserting new latent frames between existing image latent frames. For multiple camera viewpoints, the process is simpler: we simply insert the additional camera images at the image sequence level as shown the top row of Figure! (and the model is subsequently fine-tuned to handle these viewpoints).
We now discuss an illustrative example of latent injection for incorporating new modalities. For robotic platform with two static third-person cameras and wrist-mounted camera, our latent
4

---

Published as a conference paper at ICLR 2026
sequence contains 11 latent frames: (1) blank placeholder, (2) robot proprioception (e.g., end- effector pose or joint angles), (3) wrist camera image, (4) first third-person camera image, (5) second third-person camera image, (6) action chunk, (7) future robot proprioception. (8) future wrist camera image, (9) future first third-person camera image, (10) future second third-person camera image, and (11) future state value. Among these, (2), (6), (7), and (11) represent new modalities while (3), (5), (8), and (10) represent additional camera views (assuming that the first person camera is the "primary" camera). To encode the new modalities as latent frames, we fill each H W' C' latent volume with normalized and duplicated copies of the robot proprioception, action chunk, or value (where normalization simply consists of rescaling to [-1, +1). See Figure for an illustration. This ordering of modalities in the sequence represents (s, V(s'). and it allows for autoregressive decoding of actions, future state, and future state value from left to right (see Section 4.2 for further discussions on this). Note that and s' only consist of the observations at time L and t + K, respectively, where K is the action chunk size. In other words, we do not use input history nor predict future frames across multiple subsequent timesteps. Lastly, latent injection is flexible and can be adapted for any particular robot setup: for example, for a robot with only one third-person camera, one can remove the latent frames corresponding to additional camera viewpoints, and this would result in only seven total latent frames. See latent injection implementation details in Appendix A.
4.2 JOINT TRAINING OF POLICY, WORLD MODEL, & VALUE FUNCTION
Implementing joint training objectives. Now that we have a latent diffusion scheme that incorpo- rates additional modalities and camera views that are compatible with robotic policy learning, we can adapt the video model into a policy by training on robot data. For each training step, we sample a batch of (s, a, s', (s')) tuples.' 50 percent of the batch is sampled from the demonstrations dataset and is used to train the policy (p(a, s', ((s')(s), while the other 50 percent is sampled from the roll- outs dataset and is split into two halves: one half for training the world model (p(s', V(s')ls, a)) and the other half for training the value function (p(V(s') |s, a, s')). The conditioning scheme -ie., which part of the latent diffusion sequence is used as conditioning and which part is used as the tar- get to generate-deterningg details). Initially, the rollouts dataset is simply superset of the demonstrations dataset that also includes failed demonstrations. if they exist. (Failed demonstrations are those that do not success- fully complete the task when replayed in the environment due to human error during data collection, e.g., in the LIBERO and RoboCasa simulations where roughly 10 to 20 percent of demonstrations fail when replayed. In certain environments where teleoperation data is collected more carefully, such as our real-world ALOHA environment, failed demonstrations do not exist; in this case, the demonstrations dataset and rollouts dataset are equal.)
which of these three functions is being trained (see Figure 12 for more
Note that policy and world model training involves auxiliary targets, i.e., the policy is trained to model not just p(als) but rather p(a, 8,, (s*))ss, and the world model learns not just p(s'|s,a) but rather p(s', ((s))ss,a). a). We find in Section 5.2 that the auxiliary supervision improves policy performance. Also, note that the Vss' predictions are conditioned on the full latent prefix (i.c., all of (s, ",.a,")) during initial Cosmos Policy training. However, when we later fine-tune this base checkpoint on policy rollout data to produce a model with more accurate future state and value pre- dictions, we can choose to condition the value generation on a subset of (s, a, s') via input masking. The choice of the input mask determines whether the value function represents the state value V V(s") or state-action value Q(s,, a); we compare these variations in planning experiments (Section 5.3).
Parallel vs. autoregressive decoding, Since Cosmos Policy learns to both jointly and condition- ally predict the targets (a, s', Vss)) based on apportioned training samples, can generate actions, future states, and values either jointly in parallel or autoregressively from left to right. Parallel de- coding offers greater speed, while autoregressive decoding provides higher-quality predictions and allow for separate checkpoints to be used for the policy versus the world model and value function. For direct policy evaluation without planning, only actions are required for task execution while the latter two outputs can be discarded, so we use parallel decoding in this case. For evaluations with planning, we enable autoregressive decoding for higher quality future state and value predictions.
'This is implementation detail that is necessary due to the VAE's (1+ F) temporal compression scheme discussed in Section 3, which is commonly used in models such as Cosmos-Predictt2 and Wan2.1. 1. See Appendix A.1 for details.
We technically sample the empirical return G. "T R(sH, an) but write "V(s')" for notational sim- plicity and readability.
5

---

Published as a conference paper at ICLR 2026
Cosmos Policy
put x on plate
fold shirt
put bawl
put candy ziploc bag)
Cosmos Policy in the ALOHA robot tasks. Cosmos Policy can successfully execute real-world robotic control tasks that require long-horizon, high-precision manipulation and have high action multimodality.
Figure 3:
4.3 PLANNING WITH COSMOS POLICY'S WORLD MODEL AND VALUE FUNCTION
Cosmos Policy can be deployed as (1) direct policy without planning or (2) a planning policy using future state and value predictions to search for higher-quality actions. However, training on demon- strations alone is insufficient for effective planning since the data only covers successful outcomes,' which means that the world model and value function see narrow state-action distribution and may struggle to generalize beyond that distribution. We thus find it critical to collect policy rollout data and learn from these experiences.
Learning from rollouts. We collect rollout data by deploying Cosmos Policy in diverse initial conditions and recording the trajectory and episode outcome (success/fail or fractional score). Given the rollout dataset, we fine-tune our Cosmos Policy checkpoint with heavier weighting on the world model and value function predictions: 90 percent of each training batch is split evenly world model and value function training, and only 10 percent is used to train the policy.
Once we have the fine-tuned checkpoint for refined world modeling and policy learning, we propose dual deployment: the original Cosmos Policy checkpoint serves as the policy (we thus call the "policy model"), while the refined checkpoint serves as the world model and value function (we thus call it the "planning model"). This ensures that the refined world model and value function are trained on on-policy data collected by the original policy.
Model-based planning. Given the policy model and the planning model, we implement best-of- sampling as follows: (1) sample multiple action proposals from the policy, (2) use the planning model to predict the future state and value for each proposal, (3) select and deploy the action that leads to the predicted state with the highest predicted value. For greater accuracy and better mod- eling of potentially multimodal future state and value distributions, we ensemble the predictions by querying the world model three times per action and the value function five times per future state, resulting in fifteen total value predictions for each action proposal. We aggregate these via "major- ity mean": we determine whether the majority predict success or failure (via a fixed threshold) and then average values within the majority group. This approach is more robust outliers than naive averaging when value predictions are bimodal or exhibit high variance.
To speed up the search process, use parallelized inference, using N GPUs in best-of-N sampling. We also execute the full action chunk (rather only part of it, as done in receding- control) to avoid further increases in computational cost.
5 EXPERIMENTS
We evaluate Cosmos Policy to answer four key questions: (Q1) How does Cosmos Policy compare with sfate-of-the-arr imitation learning policies when used as a direct policy? (Q2) How important different components of Cosmos Policy? ? (Q3) Can Cosmos Policy leverage rollout experiences and learn an accurate world model and value function for effective planning? (Q4) Is it more ef- fective to search using world model and state value function Q-value function (a model-free
'Soee demonstration datasets, including the LIBERO simulation benchmark training set, includes subop- timal behaviors that may not lead to success when replayed in the training environment, due to human errors during teleoperation. Typically, however, most (if not all) trajectories in demonstration dataset used for imi- tation learning are successful.

---

Published as a conference paper at ICLR 2026
variation)? We answer these through simulated and weal-world evaluations spanning single-arm and dual-arm manipulation tasks.
5.1 EXPERIMENTAL. SETUP
We now describe the three task suites used in our evaluations. Note that further training and evalua- tion details are available in Appendices A.2 and A.3.
LIBERO simulation benchmark. The LIBERO benchmark (Liu et al., 2024) consists of a variety of environments and tasks featuring single Franka Emika Panda robot arm. The four primary task suites include LIBERO-Spatial LIBERO-Object, LIBERO-Goal, and LIBERO-Long (also called LIBERO-10); these assess policy's s ability to handle different spatial layouts, objects, language- specified goals, and long-horizon tasks, respectively. Each task suite provides a training dataset of 500 total demonstrations (10 tasks and 50 demonstrations each). Following Kim et al. (2024), we filter unsuccessful demonstrations for policy training but use the full unfiltered set for world model and value function training.
RoboCasa simulation benchmark. The RoboCasa benchmark (Nasiriany et al., 2024) consists of 24 static kitchen manipulation tasks featuring a single Franka Emika Panda robot arm. We follow the evaluation protocol of several prior works (Nasiriany et al., 2024; Bjorck et al., 2025; Zheng et al,, 2025; Han et al., 2024; Jang et al., 2025; Liang et al., 2025). Specifically, for each task, success rate is evaluated over 50 trials across five evaluation scenes with different floor plans and styles (10 trials per scene), and the average success rate is computed across all 24 tasks over 3 random seeds (3600 trials total). Unlike LIBERO evaluations, the RoboCasa evaluations only consist of unseen object instances, and two of the five scenes per task include styles never encountered in the training data.
The benchmark provides set of 50 Juman-telcoperated demonstrations for each task and an ad- ditional set of 1000 demonstrations generated via MimicGen (Mandlekar et al., 2023), and prior works have shown clear increases in success rates from using larger training datasets (Nasiriany et al., 2024; Bjorck et al., 2025; Zheng et al., 2025; Liang et al., 2025; Jang et al., 2025). However, to assess the relative data efficiency of Cosmos Policy compared prior works, we train our method on the 50 human- -emmannteleppertteed demonstrations alone. Similar to LIBERO data preprocessing, we fil- ter unsuccessful demonstrations for policy training but use the full unfiltered dataset for world model and value function training.
Real-world ALOHA robot tasks. The ALOHA platform (Zhao et al., 2023) consists of two ViperX 300 S robot arms with three cameras: one top-down and two wrist-mounted. We reduce the con- troller frequency from 50 Hz. to 25 Hz for computational efficiency. All policies take as input robot proprioceptive state (14 joint angles), three camera images, and task descriptions, predicting action chunks of 50 timesteps (2 seconds). We deploy the full action chunk before requerying the policy.
Our evaluation suite consists of four challenging bimanual manipulation tasks (shown in Figure 3): (1) "put X on plate" (80 demos): place objects on a plate based on language instructions, testing language following: (2) "fold shirt" (15 demos): fold one of three T-shirts in multiple steps, testing long-horizon contact-rich manipulation; (3) "put candies in bowl" (45 demos): collect scattered candies, testing ability handle multimodal grasp sequences; and (4) "put candy in ziploc bag" (45 demos): open and place items in a ziploc slider bag, testing high-precision manipulation with millimeter tolerance.
The evaluations consist of both in-distribution and out-of-distribution testing conditions, with 101 trials total per method across all tasks. We ensure fair comparison between methods by using the same fixed set of initial states for each method.
5.2 COMPARING AGAINST STATE-OF-THE-AN IMITATION POLICIES WITHOUT PLANNING
Here we aim to answer questions Q1 and Q2 posed in the beginning of this section. We answer Q1 by comparing Cosmos Policy as direct policy (without planning) with sfate-of-the-an imita- tion learning policies and assessing their relative effectiveness. We answer Q2 by ablating various components of Cosmos Policy and analyzing the resulting effects on task performance.
Diffusioo Policy is an exception, as it predicts 48-timestep action chunks since the implementation requires multiple of 4.
7

---

Published as a conference paper at ICLR 2026
Table 1: LIBERO simulation beuchmark results.
Table 2: RoboCasa simulation benchmark results.
Success rates (SR) across four LIBERO benchmark
Success rates (SR) across 24 kitchen manipulation tasks (Nasiriany et al., 2024). Cosmos Policy success rates are averaged over 50 trials for each task and three
task suites (Liu et al., 2024). Cosmos Policy success
rates are averaged over 500 trials for cach suite (10
tasks x 50 episodes) and three random seeds (6000 tri-
random seeds (3600 trials total). Our method achieves
als total). Our method achicves highest performance overall, even outperforming line-tuned state-of-lhe-art
ttate-of-thh-ang average success rate of 67. 1% while requiring significantly fewer training demonstrations (50 versus >300).
lisionlanguaage-ction (VLA) models.
Sponia (R))
Goal (72)
Average .SR (54)
Triining Beess
Average (5)
SR (95
SR: (16)
GRAMTNNI oBjorck al, 2009)
100
Difiesien Policy (Chi 2001)
76.5
92.5
68.1
$0.5
12.4
201532
UniVLA
To tillack 2024)
Dioi (fors 2025)
UVA 2005gg
97.4
96.8
94.8
98.8
95.5
95.6
99.2
85.2
9010
83.6
02.3
9482
DPVLL (iaan 2000
OR Decumfire Glagg st 2025)
GROGENE NL DUST cta 2023)
1230
2025)
300 10000 sysahetic)
1030
5320
200
58.5
57.6
57.3
Te. timethggnne
(RAATEEN NI. 5 cilyock (,205
100)
611
OpenVLA OFT (im
97.6
94.5
PLARR pohing 2013)
Vidte Polley (1 [linng et 2000)
110)
66-1
66
CogNLA 2025b9
96.5
95.4
GRIOTN HAMLET DSoe et
Ciamos Policy (nuns)
98.1
100
98.2
97.66
98.5
Diftusion Folicy
OpenVLA GFT+
Cosmos Policy (ours)
100
.22
60
36.
20
on plaze
put candies bowl (45 demos)
put candy ziploc bag (45 demos)
average
100 demos)
115 demos)
Figure 4: Real-world ALOHA robot evaluation results. We evaluate state-of-the-a art policies on a suite of four tasks and measure the score, which represents average percent completion of each task. Cosmos Policy achieves highest overall score, outperforming all other methods in three of four tasks.
Methods in comparison. In LIBERO and Robocasa, we compare against recent top-performing methods including diffusion-barse policies trained from scratch (Diffusion Policy (Chi et al., 2023), Dita (Hou et al., 2025)), video model-based policies (UVA (Li et al., 2025a). UWM (Zhu et al., 2025), Video Policy (Liang et al., 2025)), and fine-tuned VIAA models (To- no.5. OpenVLA-OFT,, CogVLA, UniVLA, DP-VLA (Han al., 2024), GROOT-N1.5 (Bjorck et al., 2025)). In real-world ALOHA evaluations, we compare against policies that have demonstrated strong performance in real-world bimanual manipulation tasks: Diffusion Policy, OpenVLA-OFT+, To. and To.5.
Results. Tables 1 and 2 show the performance of Cosmos Policy and prior works in LIBERO and RoboCasa, respectively, while Figure 4 shows performance on the ALOHA robot. We find that Cosmos Policy achieves highest overall performance in all three domains, while establishing new state of the art in the LIBERO and RoboCasa benchmarks with 98.5% and 67.1% average suc- cess rates, respectively. These results demonstrate Cosmos Policy's strong multi-task manipulation performance in both in-distribution and out-of-distribution generalization scenarios. ALOHA robot evaluations, we find that Cosmos Policy outperforms fine-tuned VLAs Wo.5 and DpenYLA-OFT+-iin
addition,
have been pretrained on large amounts of robotic imitation data despiee not having benefited from similar large-scale action supervision. This finding suggests that video model priors provide a strong initialization for control policies without requiring additional action- labeled robot data. Sample Cosmos Policy rollouts are visualized in Figure 3.
Qualitatively, we find that while the fine-tuned VLAs show strong performance on the first two tasks, they encounter difficulties in the last two tasks-"put candies in bowl" and "put candy in zi- ploc which require handling high action multimodality and executing high-precision grasps, respectively. Figure visualizes two common failure modes of no.5 and Open VLA-OFF: (1) np.5. despite showing highly competitive performance the ziploc bag. often missing the initial grasp of the slider with the right arm or not grasping the left side of the bag securely enough with the left arm. (2) VLA- two candies rather than directly going for one; we hypothesize that its LI regression of actions leads to inaccurate modeling of the action distribution in tasks with high multimodality. Compared to
the first three tasks, struggles to reliably handle
often reaches in between
8

---

Published as a conference paper at ICLR 2026
OpenVLA-OFT+
X
X
SUNG
Figure Common failure modes of To.5 and Open VheeVLA-OFFT two challenging ALOHA robot fasks. Left:
To.5 struggles to execute a high-precision grasp and loses grip the ziploc hag. Right: Vpe OFT+ reaches between two candies rather than towards one, suggesting difficulty with modeling the highly multimodal action distribution.
current state
predicted future state
actual future state
WEE Wili whil
int unsd
right
base model
Stwerrp
agghteed amaye
fine- tuned
model
The base Cosmos Policy's 's world model may fail to predict errors such losing grasp of the ziplac bag slider, is only trained on demonstrations. Bottom: After fine-tuning on policy rollout data, the world model more accurately predicts the resulting state, enabling more effective planning and eventual episode success.
Figure 6: World model predictions: base Cosmos Policy vs. fine-tuned checkpoint. Top:
these methods, Cosmos Policy handles both high multimodality and high precision with substan- tially greater reliability.
Ablation experiments. Recall from Section 4.2 that Cosmos Policy's 's policy and world model train- ing involves additional targets which provide additional supervision: the policy learns to jointly pre- dict p(a,st s', (ss) instead of p(a]s), and the world model learns to jointly predict p(s', ((s))s,,)) instead of p(s'|ss a). To evaluate the effect these auxiliary learning objectives, we train a version of Cosmos Policy without them by masking the loss the additional targets. In addition, sess the importance of the video model priors by training Cosmos Policy from randomly initialized weights. We use the same number of gradient steps as the full policy for both of these variants. As shown in Table 4, removing the auxiliary losses leads to 1 1.5% absolute drop in average suc- cess rate while training from scratch leads 3.9% drop, suggesting that these components important for maximal performance. We further evaluate Cosmos Policy trained from scratch on the ALOHA robot for additional supporting evidence and find that obtains an average score of 80.8 on the "fold shirt" task, which is 18.7 points lower than the full Cosmos Policy. Qualitatively, the from-scratch variant exhibits jerky motions that may damage the robot over prolonged deployment, so we halt further evaluations with it. Additional ablation studies on the Cosmos Policy design and joint training scheme are discussed Appendix 1.
5.3 EVALUATIONS OF COSMOS POLICY WITH MODEL-BASED PLANNING
Here we aim to answer Q3 by evaluating Cosmos Policy when deployed with model-based planning (as described in Section 4.3), and Q4 by analyzing how the proposed model-based based approach com- pares different variants of planning, such as directly learning Q-value function without a world model. Since our base Cosmos Policy already obtains high success rates in LIBERO and on the first two ALOHA robot tasks, we focus our study on the last two more challenging ALOHA robot tasks ("put candies in bowl" and "put candy in ziploc bag"), where there is more room for improvement. Further, we focus on a more challenging set of initial conditions (difficult in-distribution conditions and OOD conditions) and assess whether planning enhance performance in these settings.

---

Published as a conference paper at ICLR 2026
Cosmess Policy
Cosmos Policy + Model Free Plrninng (Qls.
Cosmos Policy Model Based Planning (Vis1)
100
80
70.0 2677
60
20.3
48.0
12.0
40
20
put candies in bowl
put candy in ziploc bag
average
Figure Masel-based planning results. We evaluate the base Cosmos Policy on challenging initial states for the last two ALOHA robot tasks, and compare it with two planning variants (model-basee and model-free). We find that the model-hased variant (V('') ()) leads to highest overall performance.
Rollout data collection. To refine Cosmos Policy's world model and value function predictions and enable more effective planning, we gather a rollout dataset that we use for post-training. Con- veniently, by running the prior direct policy evaluations, rollouts across all policies. Adding to this, we collect 143 more rollouts from Cosmos Policy for the "put candy in ziploc bag" task. The additional episodes are important for this task since train- ing an accurate world model for it is particularly challenging due to low camera observability from the robot's self-occlusion and highly stochastic environment dynamics where even millimeter dif- ferences in control can dictate success or failure. We fine-tune the base Cosmos Policy checkpoint on this pool of 648 rollouts to produce refined "planning model" for world modeling and value prediction, as described in Section 4.3.
have already aggregated 505 policy
Comparing different value function formulations. When fine-tuning the base Cosmos Policy checkpoint on the rollout dataset, we use three independent formulations for value function training by using input masks to condition the value predictions on different subsets of inputs: Vs5') (mask out (s, a)) or (s,, a) (mask out s'). The V(s') variant requires world model to predict the future state before the value can be estimated, while the Q(s, a) variant enables model-free planning by directly predicting Q-values without future state predictions.
Results. We observe that model- planning using the Vss') formulation consistently improves success rates over the base Cosmos Policy without planning, shown in Figure 7. In ALOHA tasks, we observe 12.5-point average score increase in the two challenging manipulation tasks which in- volve multimodal grasp sequences and high-precision manipulation. This is notable improvement given the limited amount of rollout data available for refining the planning model. Qualitatively, we find that the fine-tuned planning model predicts future states more accurately (see Figure 6) and can plan more effectively, ultimately avoiding making mistakes that the base Cosmos Policy makes, such as losing grasp of the slider while opening the ziploc bag. When comparing model- based (s'))) versus model-free (Q(s.a) planning variants, we observe higher performance with the former, which we attribute its ability leverage learned environment dynamics for more ef- fective and sample-efficient planning. Given a limited amount of rollout data, expect difficulty with learning an accurate Q-function and suspect that the model may also overfit given higher input dimensionality.
DISCUSSION
We presented Cosmos Policy, a state-of-the-art robot policy fine-tuned from the Cosmos-Predich video foundation model that demonstrates strong performance in LIBERO. RoboCasa, and ALOHA robot environments. We also show that incorporating policy rollout data to refine world model and value function predictions enables effective model-based planning. Limitations and future work: We observe substantially lower inference speed when using model-based planning (e.g., around 5 seconds to produce one action chunk), which may limit applicability to dynamic tasks. How to speed up the search is important direction for future study. In addition, effective planning requires sub- stantial rollout data to achieve accurate predictions beyond the demonstration distribution. Learning from fewer rollouts would increase the accessibility of our approach. Lastly, focus on best-of- planning with one layer in the search tree; extending the world model's prediction horizon and planning to greater depths could potentially lead to more effective search.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We release model checkpoints, training data, and code (including training and evaluation scripts) on our project website. Further training and evaluation details are provided in Appendix A.2 and A.3.
ACKNOWLEDGMENTS
We thank Yu- Wei Chao, Lars Ankile, Alexander Swerdlow, Max Li, and the anonymous reviewers for their constructive comments which have helped improve various elements of this paper. We also thank Dan Blick, Sophia Huang, Mohammad Harrim, Yuzhu Dong, and Pooya Jannaty for their assistance with the OSS release of this project. This work was completed during internship at NVIDIA in collaboration with Stanford University, and the first author was partially supported by the Roboties and AI Institute.
REFERENCES
Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu. Vidu: a highly consistent, dynamic and skilled text-to- video generator with diffusion models. arXiv preprint arXiv:2405.04233 2024.
Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan,
Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, al. GrOOt nl: An open foundation model for generalist humanoid robots. arXiv preprint 25Xiv22533.44734 2025.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. piO: A vision- laann-annggggattttee flow model for general robot control. arXiv preprint 22iv:2410.241644 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision- lann-ale-accttiinnn models transfer web knowledge to robotic control. arXiv preprint arXiv:2307. 23t.::2307//8888.. 2023.
Qingwen Bu, Yanting Yang. Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.061 11, 2025.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783649241273666, 2023.
Yao Feng, Hengkai Tan, Xinyi Mao, Guodong Liu, Shuhe Huang, Chendong Xiang, Hang Su, and Jun Zhu. Vidar: Embodied video diffusion model for generalist bimanual manipulation. arXiv preprint arXiv:2507.12800 2025.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 2019.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04400 2023.
ByungOk Han, Jaehong Kim, and Jinhyeok Jang. A dual process vla: Efficient robotic manipulation leveraging vim. arXiv preprint arXiv:2410. 2024.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955. 2022.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control. arXiv preprint 1xXi:23300168220 2023.

---

Published as a conference paper at ICLR 2026
Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Learning an action- able discrete diffusion policy via large-scale actionless video pre-training. Advances in Neural Information Processing Systems, 17:31124-311533 2024.
Zhi Hou, Tianyi Zhang, Yuwen Xiong, Haonan Duan, Hengjun Pu, Ronglei Tong, Chengyang Zhao, Xizhou Zhu, Yu Qiao, Jifeng Dai, et al. Dita: Scaling diffusion transformer for generalist vision- language-action policy. arXiv preprint arXiv:2503. 19757, 2025.
Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.14800 2024.
Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. To.s: a Saiio-Language-Attion Model with Open- World Generalization. arXiv preprint arXhv:2504.1605, 2025.
Arnav Kumar Jain, Vibhakar Mohta, Subin Kim, Atiksh Bhardwaj. Juntao Ren, Yunhai Feng, San- jiban Choudhury, and Gokul Swamy. A smooth sea never made a skilled sailor: Robust imitation via learning to search. arXiv preprint arXiv:2506.088 05294, 2025.
Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, al. Dreamgen: Unlocking generalization in robot learning through video world models. arXiv preprini arXiv:2505. 12705, 2025.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Tero Karras, Miik Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 14-26565-267777 2022.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-Language-acttinn model. arXiv preprint arXiv:2406. 09246, 2024.
Moo Jin Kim, Chelsea Finn, and Perey Liang. Fine-tuning vision- laooon-lnguagg-acee models: Opti- mizing speed and success. arXiv preprint arXiv:2502.19645, 2025.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:2412.03600 2024.
Myungkyu Koo, Daewon Choi, Tacyoung Kim, Kyungmin Lec, Changycon Kim, Youngyo Seo, and Jinwoo Shin. Hamlet: Switch your iision-language-action model into a history-aware policy. arXiv preprint 2X.v:2550000099) 2025.
Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song. Unified video action model. arXiv preprint arXiv:2503. 00200, 2025a.
Wei Li, Renshan Zhang, Rui Shao, Jie He, and Liqiang Nie. Cogvla: Cognition-aligned vision-language-actinn model via instruction-driien routing & sparsification. arXiv preprint 2rXiv:2508.21046, 2025b.
Junbang Liang, Pavel Tokmakov, Ruoshi Liu, Sruthi Sudhakar, Paarth Shah, Rares Ambrus, and Carl Vondrick. Video generators are robot policies. arXiv preprint XXX::2508.00795, 2025.
Yue Liao, Pengfei Zhou, Siyuan Huang, Donglin Yang, Shengcong Chen, Yuxin Jiang, Yue Hu, Jingbin Cai, Si Liu, Jianlan Luo, al. Genie envisioner; A unified world foundation platform for robotic manipulation. arXiv preprint arXiv:2508.05635, 2025.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2026
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang. Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310.17596, 2023.
Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for gener- alist robots. arXiv preprint urXiv:2406.0224 02523,, 2024.
NVIDIA, :, Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jin- wei Gu, Siddharth Gururani, Ethan He, Jiahui Huang, Jacob Huffman, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal-Taixe, Anqi L., Zhaoshuo Li, Chen-Hsuan Lin, Tsung-Yi Yi Lin, Huan Ling, Ming- Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page. De- spoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang, Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang. Heng Wang. Ting-Chun Wang. Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Yen-Chen, Xiaohui Zeng, Yu Zeng, Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, and Artur Zolkowski. Cosmos world foundation model platform for physical ai, 2025. URL https: arxiv. org/ alss 2501. 03575.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision. pp.
2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 31(140):1-611 2020.
Richard S. Sutton. Dyna, integrated architecture for learning, planning, and reacting. SIGART Bull., 164) July 1991. ISSN 0163-5719. doi: 10. 10.1145/122344112331 URL https:
///doi.org/// 1111111222222 122377.
Unitree. Umiaolm-wma-0: A world-model- action (wma) framework under unifolm family, 2025.
Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen-Woe Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng. Jiayu Wang, Jingfeng Zhang. Jingren Zhou, Jinkai Wang, Jixuan Chen, Kai Zhu, Kang Zhao, Keyu Yan, Lianghua Huang, Mengyang Feng. Ningyi Zhang, Pandeng Li, Pingyu Wu, Ruihang Chu, Ruili Feng, Shiwei Zhang, Siyang Sun, Tao Fang, Tianxing Wang, Tianyi Gui, Tingyu Weng, Tong Shen, Wei Lin, Wei Wang. Wei Wang. Wenmeng Zhou, Wente Wang. Wenting Shen, Wenyuan Yu, Xianzhong Shi, Xiaoming Huang, Xin Xu, Yan Kou, Yangyu Lv, Yifei Li, Yijing Liu, Yiming Wang. Yingya Zhang, Yitong Huang, Yong Li, You Wu, Yu Liu, Yulin Pan, Yun Zheng, Yuntao Hong, Yupeng Shi, Yutong Feng. Zeyinzi Jiang, Zhen Han, Zhi-Fan Wu, and Ziyu Liu. Wan: Open and advanced large-scale video generative models. arXiv preprint orXiv:2503 20002.2..34..... 2025.
Yiqi Wang, Mrinal Verghese, and Jeff Schneider. Latent policy steering with embodiment-agnostie pretrained world models. arXiv preprint rrXiv:2507.13340, 2025.
John Won, Kyungmin Lee, Huiwon Jang. Dongyoung Kim, and Jinwoo Shin. Dual-stream diffusion for world-mode augmented mision-language-aan model. arXiv preprint arXiv:2510.27607, 2025.
Liudi Yang, Yang Bai, George Eskandar, Fengyi Shen, Mohammad Altillawi, Dong Chen, Soumajit Majumder, Ziyuan Liu, Gitta Kutyniok, and Abhinav Valada. Roboenvision: A long-horizon video generation model for multi-task robot manipulation. arXiv preprint arXiv:2506.22007, 2025.
13

---

Published as a conference paper at ICLR 2026
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint (rXXx:2408800072,, 2024.
Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
Ruijie Zheng, Jing Wang, Scott Reed, Johan Bjorek, Yu Fang, Fengyuan Hu, Joel Jang, Kaushil Kundalia, Zongyu Lin, Loic Magne, et al. Flare: Robot learning with implicit world modeling. arXiv preprint arXiv:2505. 2025.
Zangwei Zheng, Xiangyu Peng, Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all. arXiv preprint 2rXiv:2412.00444 2024.
Zhide Zhong, Haodong Yan, Junfeng Li, Xiangchen Liu, Xin Gong, Wenxuan Song, Jiayi Chen, and Haoang Li. Flowvla: Thinking in motion with a visual chain of thought. arXiv preprint arXiv:2508. 18269, 2025.
Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining large robotic datasets. arXiv preprint drXiv:250400299, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 LATENT INJECTION IMPLEMENTATION DETAILS
As discussed in Section 4.1, Cosmos Policy learns to condition and generate new non-image modalities (such as actions, robot proprioception, and values) through a mechanism called latent injection. Figure 8 provides detailed visualization of how latent injection operates within Cosmos Policy's s latent diffusion sequence.
The process begins with sequence of images from multiple camera viewpoints, along with blank (all-zero) images that serve as placeholders for the new modalities to be injected. Once the video model's VAE tokenizer converts this image sequence into latent frames, we perform latent injection by overwriting the latent frames corresponding to the blank placeholder images with normalized and duplicated copies of the robot proprioception, action chunk, and value. Normalization rescales each modality to the range [-1, +1], while duplication resolves the shape mismatch between the low-dimensional vectors and the target latent volumes.
For instance, consider an action chunk with shape K x duct. We first normalize each action di- mension to [-1,+]], and then flatten the array into a (K x dlact) vector. This vector is duplicated (H'xW" xC'] times (where (H'' W C*) represents the shape of single latent frame), reshaped into (Kxdact)
a (H' x W' C²) volume, and used to overwrite the corresponding target latent frame. We apply an analogous process for robot proprioception and value, though their initial shapes differ.
Following latent injection, we corrupt portions of the latent sequence by adding randomly sampled Gaussian noise scaled by randomly sampled noise level (see Appendix A.2.1 for details). Cosmos Policy then trained to denoise these corrupted portions of the latent diffusion sequence while con- ditioning the clean (uncorrupted) portions. Although latent injection introduces new modalities that were not present during the base video model's pretraining, we find that the model can still learn to generate them effectively through fine-tuning.
At inference time, Cosmos Policy generates clean (denoised) latent frames. Extracting the new modalities involves reversing the latent injection process described above. For example, to extract the action chunk prediction from the generated action latent, we compute the average action chunk across all (H' xW"""""") (K (Kxd) copies in the latent volume. We then un-normalize the action chunk back its original scale and deploy either the full action chunk or a portion of it on the robot. On the other hand, extracting the value is much simpler since it is a scalar (represented by single floating point number): we simply take the average across the full latent volume and then un-normalize to the original range (0, 1)). Note that extracting non-image modalities like these does not require any VAE decoding since these elements were directly injected into the latent space during training.
A.2 TRAINING DETAILS
A.2.1 COSMOS POLICY NOISE DISTRIBUTION
Changes to noise levels during training and inference. We find that the hase Cosmos-Predict2 model's a (noise level) sampling scheme is suitable for video generation tasks but not the most effective for robot policy training. The latter requires generations be very precise since the gen- erated actions are used to directly control a robot and small imprecisions can lead to catastrophic failures. Therefore, improve the accuracy of the generations at test time, we modify the noise level sampling scheme at both train and test time.
The base model's original noise distribution a log-normal normal distribution, similar to the EDM formu- lation (Karras et al., 2022): In(or) ~ N p(a) where Pamam = 1.39, Pas = 1.2 for Cosmos- Predict2-2B (see Figure 9, left). For action generation, we observe that the low weight on higher noise levels causes inaccurate action predictions during sampling. Diffusion generation begins with pure noise scaled by Fmax = 80 and iteratively denoises over multiple steps as a decreases to miin O. At each step, the network predicts the noise at the current level to progressively re- cover a clean sample. Since the log-normal distribution concentrates training weight at lower noise levels, the model has insufficient signal at the high-o regime where generation begins, causing poor initial denoising and cascading errors. While this may not be critical for image and video generation, find it is harmful for action generation, where predictions must be precise.
15

---

Published as a conference paper at ICLR 2026
conditioning subsequence
target subsequence
images
latent injection additional modalities: robot proprioception, actions, and value
latent
frames
lasent
frames
V(s)
Figure Detailed view of latent diffusion sequence Cosmos Policy. This is more detailed version of Figure which shows implementation details. As illustrated in the top row, blank (all zera) images are inserted into the input image sequence, and these images are encoded into latent frames by the VAE tokenizer. Then, in the middle these placeholder latent frames completely overwritten by latent injection of new modalities (robot proprioception. actions, and value). Note that a placeholder image is apart the beginning of sequence due to the vidco model's tokenization scheme, which encodes the first image by itself while the rest of the images are temporally compressed in groups of four (as discussed in Section 3). Therefore, to ensure that current timestep observations and future timestep observations have similarly structured latent representations, place them after the blank first latent frame in the video diffusion sequence. In addition, since wish have one latent frame each modality camera viewpoint, we construct four identical copies of each image, as shown in the top row. Each group of four identical images corresponds to single timestep rather than four timesteps).
Figure Base model noise distribution Cosmos Policy's adjusted noise distribution. We change the base Cosmos-Predictt video model's log-normal noise distribution (left) into hybrid log-normal-unife dis- tribution (right) with more weight on higher noise levels. This modification empirically leads to more accurate generations test time. See detailed explanation in Appendix A.2.1.
Therefore, for Cosmos Policy training, we use a hybrid log-normaluunfform distribution with greater weight larger noise levels (see Figure 9, right). To implement this, sample from the original log-normal distribution with probability 0.7 and from uniform distribution over [1.0, 85.0] with probability 0.3, creating a log-normal distribution with an extended tail higher values. We find this empirically improves action prediction accuracy and overall success rate. We chose the 0.7/0.3 split to stay close the original distribution while extending the high-a tail; these probability values were not tuned.
At test time, we find that final denoising steps with o are less accurate than earlier steps at larger values, likely due to the low signal-to-noise ratio at very small noise levels. Therefore, when sampling from Cosmos Policy, we set a higher lower bound with Fmin = 4 (rather than Fmin = 0.002 in the original EDM formulation (Karras et al., 2022)) while keeping dmax 80. This higher lower bound empirically improves prediction accuracy at inference time for actions, future states, and values, measured by lower I.1 loss training and validation samples.
16

---

Published as a conference paper at ICLR 2026
A.2.2 LIBERO TRAINING DETAILS
We train Cosmos Policy for 40K gradient steps using 64 H100 GPUs with global batch size 1920 (laking 48 hours total). The base Cosmos-Prediet2 model is fully fine-tuned (all model weights). We train the policy to predict an action chunk size of 16 timesteps and execute the full chunk at test time before requerying. After 40K gradient steps, the policy's 's action L1 training loss is 0.012, future proprio LI training loss is 0.007, future wrist image latent L1 training loss 0.068, future third-person image latent 1.1 training loss is 0.036, and value function L.1 loss is 0.007. All non- image modalities are normalized to the range [-1,+1] for training. (Note that these are single-step training losses given varying (noise levels) as input, rather than losses generations from the multi-step diffusion sampling used during policy inference.) The magnitudes of these losses imply that the model more quickly learns to predict actions, proprioception, and values than images.
A.2.3 ROBOCASA TRAINING DETAILS
We train Cosmos Policy for 45K gradient steps using 32 H100 GPUs with global batch size 800 (taking 48 hours total). The base Cosmos-Predict2 model fully fine-tuned (all model weights). We train the policy to predict an action chunk of 32 timesteps and execute 16 steps before requerying- After 45K gradient steps, the policy's action 1.1 training loss is 0.016, future proprio I.1 training loss is 0.007, future wrist image latent LI training loss is 0.084, future
images latents LI training loss is 0.048, and value function LI loss is 0.007. All non-image modalities are normalized to the range [-1, +1] for training.
A.2.4 ALOHA TRAINING DETAILS
For Cosmos Policy and all other methods in our ALOHA robot evaluations, we train single policy on all four tasks combined (185 total demonstrations). Cosmos Policy is trained for 50K gradi- ent steps using 8 H100 GPUs with a global batch size of 200, taking 48 hours total. The base model is fully fine-tuned (all model weights). We train with an action chunk size of 50 timesteps (spanning seconds given 25 Hz controller) and execute the full chunk before requerying. After 50K gradient steps, the policy's 's action L1 training loss is 0.010, future proprio LI training loss is 0.008, future wrist images latents LI training loss is 0.097, future third-pern peridd-ppoonnn image latent L1 training loss is 0.085, and value function L1 loss is 0.007. All non-image modalities are normalized to the range [-1,+1] for training.
For fair comparison with Cosmos Policy, we fine-tune m0.5. To, and OpenVLA--OFT: using the same computational budget: 48 hours 8 H100 GPUs. Despite identical wall-clock time for fine-tuning, differences in training iteration speed across models and training frameworks result in substantial variation in the number of gradient steps completed: To.5 and To are fine-tuned for 400K gradient steps (batch size 256), while OpenVLA-OFT given gradient accumulation 4). We train for this seemingly large number of gradient steps across all methods because we observe that training loss continues to decrease and task execution quality generally improves with extended training. Finally, for the significantly smaller Diffusion Policy, which only contains approximately 150M parameters (as opposed to 2-7B for the other methods), we train from scratch on H100 GPU for 48 hours, resulting in 190 epochs, equivalent to 72K gradient steps (batch size 350). Like Cosmos Policy, all policies
is fine-tuned for 32K gradient steps (batch size 96
trained to predict an action chunk of 50 timesteps (except Diffusion Policy, which predicts 48 steps since requires a multiple of 4), and the full chunk is executed before the model is requeried at test time.
A.3 EVALUATION DETAILS
A.3.1 GENERAL COSMOS POLICY EVALUATION DETAILS
Number of denoising steps and parallel vs. autoregressive generation. When evaluating Cosmos Policy as direct policy, we predict actions, future state, and future state value altogether in parallel. Since direct policy evaluations do not use planning, we simply discard the future state and value predictions (though these can be saved for visualizations of the policy's 's intent and estimates of task progress). We use 5 denoising steps for LIBERO and RoboCasa and 10 denoising steps for ALOHA. On the other hand, when evaluating Cosmos Policy with model-basee planning, we autoregressively generate actions, then future state, and then future state value. In the rorl-world evaluations with planning, 10 denoising steps for the actions, 5 for the future state, and 5 for the value. In
17

---

Published as a conference paper at ICLR 2026
addition, ensemble of 3 future state predictions and an ensemble of 5 value predictions to robustify the estimates.
A.3.2 WORLD ALOHA ROBOT EVALUATION DETAILS
Number of trials and initial conditions. We evaluate every method in the real ALOHA robot experiments with 101 total trials: 30 trials for "put X on plate". 20 trials for "fold shirt", 25 trials for "put candies in bowl", and 26 trials for "put candy in ziploc bag". For each task, we test on both in-distribution and out-of-distribution (OOD) generalization scenarios with respect to the training demonstration dataset (see Figures 10 and 11 for visualizations). We describe the details of the test conditions below:
"put X on plate": 20 in-distribution 10 OOD trials. For in-distribution trials, the two objects (purple eggplant and brown chicken wing) are placed in the same row along the horizontal axis of the table, with varying position along the vertical axis. For OOD trials, the objects are not constrained to be in the same row and the positions are more varied. Max time limit is 350 timesteps (14 seconds).
"fold shirt": 12 in-distribution + OOD trials. For in-distribution trials, we use three T-shirts of different colors (blue with white stripes, plain white, and brown camouflage) with four fixed initial positions on the table (back, front, left middle, right middle). These three T-shirts were seen at data collection time (five demos for each of the three shirt, for 15 training demos total). For OOD trials, we use an unseen pink T-shirt with the same four initial positions for the first four trials, and the seen flue-with-vee stite-stripes shirt with unseen distractor objects placed on the table (other shirts folded up and placed on the sides corners of the table, serving as visual distractors). Max time limit is 1600 timesteps (64 seconds).
"put candies in bowl": 15 in-distribution + 10 OOD trials. For in-distribution trials, place candies in front of a large metal bowl, with a roughly even balance of candies on the left versus right half of the table. For OOD trials, we either (a) place 5 candies in front or behind a large metal bowl with uneven amounts of candies on the left versus right half, (b) place 5 candies in front behind an unseen smaller orange bowl. Max time limit is 1100 timesteps (44 seconds).
"put candy in ziploc bag": 20 in-distribution trials + 6 OOD trials. For in-distribution trials, we place a pink ziploc bag filled half full with candies in the middle of the table with varying initial position along the vertical axis, and one piece of candy to its right (also varying along the vertical axis). For OOD trials, we replace the pink ziploc bag with unseen blue ziploc bag that is filled to about 75 percent full (more than seen during training), which requires more firm grasps when grasping the bag so that it does not slip out of grasp the robot manipulates it. Max time limit is 1 1100 timesteps (44 seconds).
Scoring rubrics. Here we describe how each policy rollout is scored between 0 and 100 points. Each score represents what percent of the task has been successfully completed. We use score instead of success rate since a binary metric does not capture fine-grained details.
"put X on plate": 50 points for touching the correct target object. 50 points for putting the correct target object on the plate.
"fold shirt": 10 points for each of the following stages: (1) grasped bottom edge of shirt, (2) folded in half, (3) grasped both sleeves, (4) folded sleeves in, (5) grasped bottom edge of shirt, (6) folded in half again, (7) grasped right edge of shirt, (8) folded in half again, (9) released the shirt from grasp, (10) pushed shirt to center of table. In the end, 10 points are deducted if a part of the shirt (e.g. sleeve) sticks out due to an incomplete fold.
"put candies in bowl": 20 points for each of 5 candies grasped and placed into the bowl.
"put candy in ziploc bag": 20 points for each of the following stages: (1) grasped ziploc bag opener with right arm, (2) grasped ziploc bag's upper-left corner with left arm, (3) opened bag at least halfway by pulling the opener with right arm, (4) grasped candy with right arm, (5) placed candy inside bag.
18

---

Published as a conference paper at ICLR 2026
Initial Positions for In-Distribution Trials
"put X on plate"
"fold shirt"
AAAM
"put candies in bowl"
"put candy in ziploc bag"
Figure 10: In-distribution initial conditions for ALOHA robot evaluations. Here we show sample initial positions used for evaluating policies conditions similar to training demonstrations.
Detailed in-distribution vs. OOD evaluation scores. Table shows breakdown of in- distribution and out-of-distribution scores for every method across the ALOHA robot task suite. Cosmos Policy achieves highest aggregate success rates, though no.s shows slightly better perfor- mance specifically in OOD test scenarios.
Table 3: In-distribution vs. OOD evaluation scores for ALOHA robot evaluations. This table a a more
detailed summary of the results shown in Figure 4.
"put on plate" Score
Score
"put candies
Score
"pat candy in ziploc hag" Score
Average Score
In-listribution
Diflusion Policy
85.0
23.3
57.3
11.0
39.2
OPT
87.5
99.2
59.0
67.1
No
98.3
Ko.s
97.5
99.2
98.7
56.0
87.8
Conmos Policy
100.0
99.2
100.0
86.0
96.3
00D
Policy OPtTVLA-OFT+
30.0
100.0
20.0
56.7
51.7
60.0
98.8
68.0
60.0
71.7
100.0
100.0
80.0
92.5
Coumos Policy
Full (In-distrbbation OOD)
Diffusion Policy
63.3
23.5
32,8
14.6
33.6
OpenVLA- OFT+
68.3
99.5
21.6
58.5
62.0
98.5
Xa.s
98.3
99.5
95.2
61.5
88.6
Coumes Policy
100.0
99.5
89.6
85.4
93.6
19

---

Published as a conference paper at ICLR 2026
Initial Positions for Out-of-Distribution Trials
"put X on plate"
"fold shirt"
"put candies in bowl"
"put candy in ziploc bag"
Figure 11: Out-of-distribution initial conditions for ALOHA robot evaluations. Here show sample initial positions used for evaluating policies in unseen test conditions.
Table 4: Cosmos Policy ablations in LIBERO. Here we report the results of two independent ablations:
(1) In Section 4.2, we discussed that Cosmos Policy's policy and world model training involves additional targets which provide auxiliary supervision: the policy learns to jointly predict p(a, s', v(s')) ((ss)))s) instead of p(a[s), and the world model learns jointly predict p(s', ((s')ls, a) instead of p(s' [s, a). To evaluate the effect of this joint learning objective, we train a version of Cosmos Policy without by masking the loss on the additional targets: the policy predicts only p(a[s) s) and the world model predicts only p(s" a) (the value function continues to predict p( (s')sss,, a, s') as before). We find that removing the auxiliary objectives leads 1.5 point drop in average success rate. (2) We also train version of Cosmos Policy from scratch and observe a 3.9 point drop from the full version.
Spatial SR (%)
Object SR (%)
Goal SR (%)
Long SR (%)
Average SR (%)
Cosmos Policy (ours)
98.1
100.0
98.2
97.6
98.5
(I) w/o auxiliary losses
97.6
99.8
96.7
94.0
97.0
(2) w/o pretrained model
94.7
98.9
96.3
88.6
94.6
A.4 ADDITIONAL EXPERIMENTS AND DETAILS
A.4.1 ADDITIONAL ABLATION EXPERIMENTS
To further study the effects of individual components of the Cosmos Policy design, as well as the joint training objectives discussed in Section 4.2, we conduct series of additional ablation exper- iments that remove individual components or objectives one at a time, until we ultimately have a barebones policy that only predicts actions (no future state or value). Recall from Section 4.2 that Cosmos Policy is trained with a joint objective that optimizes the policy, world model, and value function altogether in one architecture. Specifically, each batch of training samples is split 50/25/25: 50 percent of the samples are used for training the policy, 25 percent for the world model, and 25 percent for the value function. This balanced splitting determines which parts of the latent diffusion are used as conditioning and which parts are used as the target, which in turn determines whether we are training the policy, the world model, or the value function (see Figure 12 for an illustration). In addition to these balanced batches, the policy and world model are trained with auxiliary objectives: the policy learns to jointly predict p(a, s', u(s' (ss)))s) instead of p(als) s)), and the world model learns to jointly predict p(s', V(s')ls instead of ( 8/)8)a)
20

---

Published as a conference paper at ICLR 2026
Clean (No Noise) Latent Diffusion Sequence
CLANETS SPO3TO
CLRPETS Amai
corneclat..o iTg
CuinkEiggtt 7g
Acion chank
Falaw pagato
Iaaas veril Irg
Sular ating
Mur in
edue
V(s)
50% of Training Batches Policy Training x(a, s', v(s)ss)
conditioning
target
CumTenT DEGOO
Camentt wrisd Pig
CLATERT mo
cueent 'iit Pig
Edtion chack
i
right
a
V(s)
25% of Training Batches
World Model Training p(s', ((s)ss,))
conditioning
target
wrbsi
chank
5'
V(s)
Training Batches Value Function Training p(,(s)(ssa,s)
conditioning
target
cameel right
1nm
Famid
V(s))
Figure 12: Cosmos Policy balanced batches training scheme. We illustrate the joint objectives training scheme discussed in Section While each batch of training samples is split 50/25/25 for policy, world model, and value function training, respectively, the full latent diffusion sequence remains fixed, and the conditioning scheme determines which of the three functions being optimized. During training, the model is tasked denoise the target noised latent frames conditioncd on the clean latent frames. Note that the above depicts the initial base policy training scheme. When optionally refining the world model and value function on policy rollouts (in preparation for model-based planning). mask out the current state and action during value function training so that the value predictions Q v-vauue function,
only function of the future state s'. Separately, to train can instead mask out the future state so that the value predictions are only a function of the current state and action, (s, a); this variant can be used for model-fres planning (though we find that the model- based planning variant performs better, as shown in Figure 7).
A.4.2 COSMOS POLICY INFERENCE LATENCY
Here we discuss measurements of inference latency in terms of wall-clock time. Cosmos Policy generates action, future state, and value predictions in parallel in 0.61 seconds total HI00 GPU when sampling with 5 denoising steps (for LIBERO and RoboCasa). Inference latency increases 0.95 seconds on H100 GPU when sampling with 10 denoising steps (for ALOHA). Inference latency decreases to O. seconds on H100 GPU when sampling with denoising step. We observe competitive performance in RoboCasa (66.4% average success rate) even with denoising step, as shown in Table 5. Note that these latencies are the time spent generating action chunk. For example, on the ALOHA robot, the robot pauses for 0.95 seconds to generate an action chunk that spans 2 seconds of execution.
For model-based plamning experiments on the real ALOHA robot, we do best-of-N search with N using parallel H100 GPUs (1 for each branch in the search). Cosmos Policy first generates N candidate action chunks with 10 denoising steps each, then generates an ensemble future state predictions per action proposal with 5 denoising steps each, and then generates ensemble of
21

---

Published as a conference paper at ICLR 2026
Table 5: Cosmos Policy ablations and additional experiments in RoboCasa. Top: We ablate individual components of the joint objectives training scheme and auxiliary supervision discussed Section 4.2 and visualized in Figure 12. Each version of the policy is trained with the exact same training hyperparameters and compute as the original Cosmos Policy in RoboCasa, and evaluated across all 24 tasks with 50 trials per time until we arrive at barebones policy that only predicts actions. Here details describing cach ablation: (1) We ablate value function training samples (last row of Figure 12) and instead use 50% batches for policy training and 50% for world model training. (2) We ablate both world model and value function training samples (last two rows of Figure 12) and instead use 100% of each training batch for policy training only. (3) Adding to the prior ablation, also ablate the value target when training the policy. thus optimizing the policy learn just (a, s' (s) rather than I7 (a, s', (s))ss) (4) Adding the prior ablation, we ablate both future state and value targets when training the policy. thus training a barebones policy that only models 77 Taass). We observe the most significant drop in performance from the final ablation, suggesting that training the policy to also predict future state is crucial to the effectiveness of Cosmos Policy. Bottom: We also report the performance of Cosmos Policy with just denoising step per action chunk inference time. We observe 66.4% average success rate with just 0.16 second inference latency per action chunk 32 timesteps on one H100 GPU, which nearly 4x faster than the 0.61 second latency observed with 5 denoising steps with only a small hit (-0.5%) on success rate.
task and random seeds (3600 trials total). We remove one component
Average
Cosmos Policy (ours) (5 denoising steps)
67.1
(1) wio value finction training samples
66.6
(2) wio world model and value function training samples
64.0
(3) wio woeld model
value flancion triniing samples & anxiliary value supervision for policy tmaining samples (4) wio woeld model and value function training samples & mixiliary furure state and value supervision for policy maannnng samples
44.4
Cosmos Policy w/ dencising step
66.1
5 value predictions per future state prediction with 5 denoising steps each. In total, this search takes 4.9 seconds on 8 parallel H100 GPUs.
Though the high inference latency did not prevent effective task execution in our evaluations, we acknowledge the limitations caused by slow inference, such diffieulty in adapting the method to dynamic manipulation tasks or locomotion tasks. Optimizing the inference speed for the policy and the model-based search process would be a promising avenue for future work in order enable adoption to a wide variety of robotics applications.
22


Reviews:
[{"review_id": "Iaa9AyafBQ", "reviewer": "Reviewer_nsat", "summary": "This paper presents Cosmos Policy, a method for adapting a large pretrained video diffusion model (Cosmos-Predict2) for \"world modeling\" in visuomotor policy learning. Specifically, they use a pretrained video diffusion model which takes in a sequence of latent frames to produce RGB frames as output to also take in new types of latent frames (corresponding to robot actions, states, and value function) and produce the corresponding future outputs. Instead of adding new networks or heads (which most prior works do), Cosmos Policy reuses the exact same video diffusion transformer and pretends that these new signals (actions, proprioception, values) are just additional “frames” in the video sequence. Cosmos Policy thus jointly learns to denoise actions, future states, and expected returns, leveraging the pretrained model’s spatiotemporal priors for physically grounded control. \n\nThe paper further demonstrates a model-based planning extension, where a fine-tuned world model and value function are used to evaluate multiple action proposals in a “best-of-N” search procedure. Experiments in simulation and one some real-world manipulation tasks shows the method outperforming both diffusion-based policies trained from scratch and fine-tuned vision-language action (VLA) models", "questions": "1. Unless I missed it, I didn't see any ablations on other ways of encoding the state, action, value information as latent frames. For example, could you combine those in different ways? How much would the training/finetuning pipeline have to changes if we were to change the embodiment, for example?\n\n2. I would like to see more discussion on the computational overhead. How much does this add during inference time and how realistic is that for control in real-time on hardware?\n\n3. How well would this generalize for longer-horizon tasks?\n\n4. Do you have any tools or visualizations to understand how the injected latent frames influence downstream denoising? For example, does the diffusion noise schedule or attention pattern shift when action/value latents are introduced?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I like the idea of repurposing a video generation model (that has already learned spatio-temporal predictions) for other spatio-temporal prediction tasks, in this case robot action/value data. The evidence provided by the paper that this simple idea works is noteworthy beyond just the numbers. The numbers themselves are impressive where fine-tuning the 1.2B-parameter Cosmos-Predict2 model on just a few hundred robot demonstrations yields 98.5% task success on the LIBERO benchmark, outperforming both diffusion-based and VLA baselines (e.g., Pi0, OpenVLA) trained from scratch. \n\nI also like the extension to model-based planning without requiring architectural changes. I think this idea can be further exploited by predicting more spatio-temporal signals useful for model-based planning.", "weaknesses": "1. It would have been nice to see specific numbers on how well the state, actions, and value functions are predicted using this model.\n\n2. It is also not clear how well this would generalize standard tasks (that may be in the pretraining of the underlying model.. even though I fully agree that the action data is not). But seeing more generalizability experiments would have been nice.\n\n3. It was not clear how well does this work for longer-horizon tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m8j8FjHRVE", "reviewer": "Reviewer_i7f5", "summary": "This paper introduces Cosmos Policy, which adapts a pretrained video generative model into a robot policy though post-training without architectural modifications. In order to adapt the pretrained video model (image+text -> video) to a robot policy (requires additional modalities including robot proprioception, robot actions, state values, multiple camera views....), the authors propose to encode the additional modalities as additional latents. This simple design enables the joint training of policy, world model and value function, which can be leveraged appropriately to perform the demanded robot task e.g., direct policy evaluation, or evaluation with planning. Experiments show strong performances compared to existing VLAs or robot policy models, across the single-arm LIBERO simulation and two-arm ALOHA platform.", "questions": "- What happens if the same post-training scheme of Cosmos Policy is applied to CogVideo/CogVideoX models? \n\n- How exactly are the additional modalities being encoded into latents are are being interleaved with the original latents?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Strong empirical performances across the evaluation benchmarks, even existing methods which already rely on video generative models.\n\n- The proposed idea is simple yet effective, and ebles joint training of policy, world model and value function within the same design.\n\n- Sufficient analyses and ablation results (e.g., w/o auxiliary losses, Q_sa and V_s variants) which show interesting and significant results.", "weaknesses": "- I understand that the additional modalities are encoded as additional latents, but I still can't understand exactly how. I can understand from Figure 1 that the different modalities are interleaved, and the current state frames are given as conditioning inputs - but it is hard to understand where the original latents remain, and where the additional modalities are input. \n\n- Without the pretrained model, the performance of Cosmos Policy falls below that of CogVLA. What happens if the same post-training scheme of Cosmos Policy is applied to CogVideo/CogVideoX models? This would be a fairer comparison, as the newer Cosmos-Predict2-2B-Video2World is generally considered to be a stronger model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "08exaDinFv", "reviewer": "Reviewer_nGa7", "summary": "The authors propose a way to use a video generation model for robot control through finetuning the model without multi-stage post-training or architectural modifications to the model.  Future latent frames are decoded into future images and cumulative reward values.  This enables test-time planning of action trajectories. Simulation results are presented on the LIBERO sim benchmark.", "questions": "- The authors claim that they can predict videos for new camera views as well as states and value functions from the same model without architectural changes.  This is done through latent frame injection.  Sec. 4.1 does not provide sufficient detail on how this is done.  In particular, what should the latent state injection be.  Blank or copies of current latents should not work.  Can the authors add more details on how this is enabled - is the finetuning on the entire network rather than just a few layers?  Would the latent injection not cause an increase to the size of the latent input and an increase to the network size for the subsequent layers?  If so, is this not considered an architectural change?\n\n- Since the work depends significantly on the pre-trained video model, Cosmos-Predict2, it might be good to provide an overview and key features and capabilities of this model.  This can be added as an appendix.  Particularly, it will be interesting to point out the differences between this video model and other video models - this will enable readers to see if they can start with their own existing video models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The main strength of this work is in using the video generation model to output actions with only a single fine-tuning stage without any architectural changes to the model - this is in contrast to other approaches that employ architectural changes like adding inverse dynamics model or doing multiple stages of post-training to generate actions.\n\n- The proposed approach enables model-based planning where multiple action proposals can be sampled from the policy and resulting states / values can be predicted for each action sequence and the action sequence with the highest value can be selected.", "weaknesses": "- It's unclear why making architectural changes to the video generation model is seen as a weakness in other approaches.\n\n- Are the authors assuming the world model is on the state s (in Sec. 3)? They authors claim the world model predicts the state.  World models predict observations and not states.  Clearly making this distinction is important as the state is not completely observable. The method in the paper predicts value function as a function of the state.  However, since we are only predicting the observation, the value function can't be a function of this observation but rather a function of the (unknown) state.  Does this break things in the formulation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "19N2OwMtco", "reviewer": "Reviewer_pF5s", "summary": "This paper proposes a Cosmos policy framework that is built upon the Cosmos pre-trained video model. To add required modalities to the pre-trained model, this paper constructs additional latent frames containing the information. Experiments are conducted on the Libero tasks and the real Aloha task, where the proposed policy obtains better performance on average.", "questions": "- Can the proposed framework also work well for other pre-trained video models, especially considering different architectures? \n- Please address my concerns in Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper proposes fine-tuning a pre-trained video model to a policy that can execute and interact with the real environment. The idea and direction are very promising and interesting for the community.\n- Instead of adding an additional network for integrating modalities like actions, the paper constructs latent frames to better leverage the prior learned in the pre-trained model. While this idea is not novel, it is valuable for robotic policy learning.\n- Experiments include both simulation environments and real tasks.", "weaknesses": "- While I like the research topic studied in this paper, the proposed method is not novel and not elegant enough. For example, an executable policy requires high frequency, which makes small policy networks and action chunking useful. However, this paper requires the Cosmos pre-trained model to output executable, low-level actions directly, which can cause high latency and make it difficult to adapt to other tasks like locomotion. The second point is that the latent frames contain duplicate information, which can cause learning inefficiency. It is not necessary for the robot's proprio states and actions to occupy several latent frames. Overall, the proposed fine-tuning pipeline is not easy to reproduce, and still needs to be improved.\n- While the proposed policy gets the best success rates on average, its performance on o.o.d. tasks is not demonstrated clearly. Can the policy generalize to different tasks with the same action space? Can the policy generalize to different backgrounds and unseen objects? Since the policy is built upon a pre-trained model, I expect it to generalize better.\n- Missing citations related to fine-tuning the video prediction model to an executable policy, e.g., Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training, NeurIPS 2024. \n- Missing Reproducibility statement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)