Title: Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planning
Primary area: applications to robotics, autonomy, planning
Keywords: world models, robotics, manipulation, model-based planning, imitation learning, video generation

Abstract: Recent video generation models demonstrate remarkable ability to capture complex physical interactions and scene evolution over time. To leverage their spatiotemporal priors, recent robotics works have adapted video models for policy learning but introduce complexity by requiring multiple stages of post-training and new architectural components for action generation. In this work, we introduce Cosmos Policy, a simple approach for adapting a large pretrained video model (Cosmos-Predict2) into an effective robot policy through a single stage of post-training and with no architectural modifications, fine-tuning only on the robot demonstration data collected on the target platform. Cosmos Policy learns to directly generate robot actions encoded as latent frames within the video model's latent diffusion process, harnessing the model's rich priors and core learning algorithm to capture complex action distributions. Additionally, Cosmos Policy generates future state images and values (expected total cumulative rewards), which are similarly encoded as latent frames, enabling test-time planning of action trajectories with higher likelihood of success. In our evaluations, Cosmos Policy achieves state-of-the-art performance on the LIBERO simulation benchmark (98.5\% average success) and the highest average success rate in challenging real-world bimanual manipulation tasks, outperforming strong diffusion-based policies trained from scratch, video model-based policies, and state-of-the-art vision-language-action models fine-tuned on the same robot demonstration dataset. Furthermore, given policy rollout data, Cosmos Policy can learn from experience to refine its future state and value predictions and leverage model-based planning to achieve even higher success rates on challenging tasks.

TL;DR: Cosmos Policy is a simple framework for adapting a large pretrained video model (Cosmos-Predict2) into a state-of-the-art robot policy that can generate actions, future states, and values and plan action trajectories with high success rates.