Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning
Authors: name: NVIDIA, name: Alisson Azzolini, name: Hannah Brandon, name: Prithvijit Chattopadhyay, name: Huayu Chen, name: Jinju Chu, name: Yin Cui, name: Jenna Diamond, name: Yifan Ding, name: Francesco Ferroni, name: Rama Govindaraju, name: Jinwei Gu, name: Siddharth Gururani, name: Imad El Hanafi, name: Zekun Hao, name: Jacob Huffman, name: Jingyi Jin, name: Brendan Johnson, name: Rizwan Khan, name: George Kurian, name: Elena Lantz, name: Nayeon Lee, name: Zhaoshuo Li, name: Xuan Li, name: Tsung-Yi Lin, name: Yen-Chen Lin, name: Ming-Yu Liu, name: Andrew Mathau, name: Yun Ni, name: Lindsey Pavao, name: Wei Ping, name: David W. Romero, name: Misha Smelyanskiy, name: Shuran Song, name: Lyne Tchapmi, name: Andrew Z. Wang, name: Boxin Wang, name: Haoxiang Wang, name: Fangyin Wei, name: Jiashu Xu, name: Yao Xu, name: Xiaodong Yang, name: Zhuolin Yang, name: Xiaohui Zeng, name: Zhe Zhang

Physical AI systems need to perceive, understand, and perform complex actions
in the physical world. In this paper, we present the Cosmos-Reason1 models that
can understand the physical world and generate appropriate embodied decisions
(e.g., next step action) in natural language through long chain-of-thought
reasoning processes. We begin by defining key capabilities for Physical AI
reasoning, with a focus on physical common sense and embodied reasoning. To
represent physical common sense, we use a hierarchical ontology that captures
fundamental knowledge about space, time, and physics. For embodied reasoning,
we rely on a two-dimensional ontology that generalizes across different
physical embodiments. Building on these capabilities, we develop two multimodal
large language models, Cosmos-Reason1-8B and Cosmos-Reason1-56B. We curate data
and train our models in four stages: vision pre-training, general supervised
fine-tuning (SFT), Physical AI SFT, and Physical AI reinforcement learning (RL)
as the post-training. To evaluate our models, we build comprehensive benchmarks
for physical common sense and embodied reasoning according to our ontologies.
Evaluation results show that Physical AI SFT and reinforcement learning bring
significant improvements. To facilitate the development of Physical AI, we will
make our code and pre-trained models available under the NVIDIA Open Model
License at https://github.com/nvidia-cosmos/cosmos-reason1.