RoboDream (Cosmos-Predict2-2B, DROID)

Weights of RoboDream (paper, project page, code).

RoboDream is an embodiment-centric video world model. Given a rendered robot-only motion video, a scene prior (background image) and an object prior (task objects on a blank canvas), plus a task instruction and the end-effector trajectory, it synthesizes photorealistic two-view (third-person + wrist) robot demonstrations, zero-shot for unseen objects, scenes and viewpoints.

File	Description
model.pt	DiT weights (2B) fine-tuned from Cosmos-Predict2-2B-Video2World on ~40k DROID episodes (100k iterations, 16×A100)
default_text_emb.pt	T5 embedding the inference server falls back to for instructions outside the DROID T5 cache (unless it is started with --t5)

The architecture and all training / sampling constants are in robodream/config.py of the code repo; robodream.pipe.load_dit loads model.pt without key remapping.

Inputs: 189-frame clips, two views (third-person + wrist) at 176×320; 35 sampling steps.

huggingface-cli download Jay-Ye/RoboDream --local-dir ${ROBODREAM_CKPT_DIR}/robodream
bash shell/inference_server.sh 0 5000   # GPU 0, port 5000

notebooks/robodream_demo.ipynb in the code repo downloads these weights and runs retrieval & rebirth, prop-free teleoperation and compositional generation on the inputs shipped in demos/. The training data (robot-only renders, priors, T5 embeddings, splits) is in Jay-Ye/RoboDream-Data.

The tokenizer (VAE) and T5 encoder are loaded from nvidia/Cosmos-Predict2-2B-Video2World and google-t5/t5-11b. Use of these weights is subject to the NVIDIA Open Model License of the base model.

License

The RoboDream weights are released for non-commercial use under CC BY-NC 4.0. As a derivative of NVIDIA Cosmos-Predict2-2B-Video2World, they are additionally subject to the NVIDIA Open Model License.

@article{ye2026robodream,
  title={RoboDream: Compositional World Models for Scalable Robot Data Synthesis},
  author={Ye, Junjie and Xue, Rong and Van Hoorick, Basile and Li, Runhao and Rajaprakash, Harshitha and Tokmakov, Pavel and Irshad, Muhammad Zubair and Guizilini, Vitor and Wang, Yue},
  journal={arXiv preprint arXiv:2606.02577},
  year={2026}
}