Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

Abstract

We introduce Puffin-World, a unified multimodal world model that represents the physical environment through three complementary native 3D world states — physics (gravity-aware camera understanding and physically consistent trajectory propagation), geometry (dense spatial structure for view synthesis and 3D reconstruction), and appearance (high-fidelity, spatially coherent visual content). A single integrated framework of vision encoder, LLM, and diffusion model — without task-specific external modules — supports physical-world perception, free-viewpoint spatial simulation, 3D world modeling, and closed-loop interaction. At its core is camera-centric multi-view world modeling: given one initial view and a camera trajectory, generate the remaining views jointly in RGB and depth, with 3D point-cloud reconstruction. Puffin-World is trained on Puffin-16M, 15M vision-language-camera triplets and 1M diverse camera trajectories curated from 28 public datasets. Built on Puffin (Thinking with Camera, ICLR 2026).

Links

• Project Page: https://kangliao929.github.io/projects/puffin-world/
• GitHub Repository: https://github.com/KangLiao929/Puffin
• Hugging Face Dataset: https://huggingface.co/datasets/KangLiao/Puffin-16M

Model Details

Developed by	Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy
Affiliation	S-Lab, Nanyang Technological University
First released	Preprint, 2026
Model type	Unified multimodal world model (autoregressive understanding + diffusion-based multi-view RGB-D generation with native 3D world states)
Modality	Image → Text+Camera; Text+Camera → Image; Image+Trajectory → Multi-view RGB-D + 3D point cloud

Model Zoo

Checkpoint	LLM	Vision Encoder	Diffusion	Composition
Puffin-World-Base.pth	Qwen2.5-7B	C-RADIOv3-H	SD3.5-medium	Full und+gen world model (Stage-II understanding backbone + Stage-IV asymmetric-attention RGB-D generation)
Puffin-World-Pro.pth	Qwen2.5-1.5B	C-RADIOv4-H	SD3.5-large	Full und+gen world model (Stage-II understanding backbone + Stage-IV asymmetric-attention RGB-D generation)
Puffin-World-Caption.pth	Qwen3.5-0.8B	C-RADIOv3-H	—	Understanding-only VLM expert (camera-aware captioning)

The matching pipeline configs in the GitHub repository:

Checkpoint	Config
Puffin-World-Base.pth	configs/pipelines/final_stage_4_world_all_asym_attn_qwen2_5_7b_radiov3H_sd3p5M.py
Puffin-World-Pro.pth	configs/pipelines/final_stage_4_world_all_asym_attn_qwen2_5_1_5b_radiov4H_sd3p5L.py
Puffin-World-Caption.pth	configs/pipelines/vlm_qwen3_5_0_8b_radiov3H_stage_2.py

Direct Use

• Multi-view world modeling: given one initial view and a camera trajectory, generate the remaining views jointly in RGB and depth, and reconstruct a gauge-aligned 3D point cloud.
• Physical-world perception: gravity-aware single-image camera estimation (roll, pitch, vFoV, distortion) with perspective-field visualization.
• Free-viewpoint spatial simulation: camera-controlled single-view generation and free-view world exploration (I2T → T2I → chunked multi-view I2I).
• Closed-loop interaction: self-calibration — the model estimates its own camera anomaly and generates the corrective action sequence.

Sample Usage

Download the checkpoints and organize them in a checkpoints directory of the Puffin-World code base:

Puffin/Puffin-World/
├── checkpoints
    ├── Puffin-World-Base.pth
    ├── Puffin-World-Pro.pth
    ├── Puffin-World-Caption.pth

# pip install -U "huggingface_hub[cli]"
huggingface-cli download KangLiao/Puffin-World --local-dir checkpoints --repo-type model

Multi-view world modeling (initial view + trajectory → RGB-D views + reconstruction.glb):

export PYTHONPATH=./:$PYTHONPATH
python scripts/demo/world_modeling.py \
    --model Puffin-World \
    --checkpoint checkpoints/Puffin-World-Pro.pth \
    --dataset re10k --sample_index 0 --output output/demo_world

Physical-world perception (single-image camera estimation + perspective fields):

python scripts/demo/physics_perception.py demo.jpg \
    --model Puffin-World-Pro \
    --checkpoint checkpoints/Puffin-World-Pro.pth \
    --output output/demo_camera

Camera-controlled spatial simulation:

python scripts/demo/spatial_simulation.py \
    "A cozy living room with wooden furniture and warm sunlight." \
    --model Puffin-World-Pro \
    --checkpoint checkpoints/Puffin-World-Pro.pth \
    --camera "0.05, -0.12, 1.2" --num 4 --vis_pf

Free-view world exploration and self-calibration (closed-loop):

python scripts/demo/freeview_world_exp.py \
    --checkpoint checkpoints/Puffin-World-Pro.pth \
    --input <image> --scene_dir <re10k-style scene dir> --chunk 3

python scripts/demo/self_calibration.py <image> \
    --checkpoint checkpoints/Puffin-World-Pro.pth \
    --output output/demo_self_calib

For the 7B model, pass --model Puffin-World-7B (world modeling) or --model Puffin-World-Base (perception/simulation) with --checkpoint checkpoints/Puffin-World-Base.pth. Full evaluation entry points, flags and output layout: documents/EVALUATION_World.md in the GitHub repository.

Citation

If you find Puffin-World useful for your research or applications, please cite our papers using the following BibTeX:

@article{liao2026puffinworld,
  title   = {Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States},
  author  = {Liao, Kang and Luo, Yihang and Wu, Xiao-Ming and Jin, Linyi and Wu, Size and Lin, Chunyu and Zhao, Yao and Wang, Fei and Li, Wei and Loy, Chen Change},
  journal = {Preprint},
  year    = {2026}
}

@article{liao2025puffin,
  title={Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation},
  author={Liao, Kang and Wu, Size and Wu, Zhonghua and Jin, Linyi and Wang, Chao and Wang, Yikai and Wang, Fei and Li, Wei and Loy, Chen Change},
  journal={arXiv preprint arXiv:2510.08673},
  year={2025}
}

License

This project is licensed under NTU S-Lab License 1.0.