Cosmos3-Nano-GGUF (understanding branch)

The first GGUF of nvidia/Cosmos3-Nano — the reasoning/VQA core of NVIDIA's omnimodal Physical-AI world model, extracted and converted so it runs anywhere llama.cpp runs (llama.cpp, LM Studio, Ollama, Jan, …).

What this is (and isn't)

Cosmos3-Nano is a joint omnimodal transformer: every layer contains an understanding path (a standard Qwen3-VL-8B-class VLM) and a generation path (MMDiT-style joint attention + parallel _moe_gen MLPs) driving video/audio/action generation through a diffusion stack.

This repo contains the understanding path + vision encoder only, re-keyed to the standard Qwen3VLForConditionalGeneration layout and converted with mainline convert_hf_to_gguf.py:

• ✅ Image + multi-image understanding, VQA, physical-world reasoning, chat, tool calling (ChatML), 256K context (mrope, interleaved)
• ❌ Video/audio/action generation — those weights (*_moe_gen, joint-attention adds, action/audio/diffusion heads, ≈14 GB) are dropped; they require the VAE + scheduler pipeline and cannot run in llama.cpp

Architecture of what you get: 36 layers, 4096 hidden, GQA 32/8, head_dim 128, qk-norm, 262,144 max context — plus the 27-block deepstack vision encoder (deepstack indexes 8/16/24) as an mmproj.

Files

File	Quant	Size
Cosmos3-Nano-VLM.Q4_K_M.gguf	Q4_K_M	~5.0 GB
Cosmos3-Nano-VLM.Q5_K_M.gguf	Q5_K_M	~5.9 GB
Cosmos3-Nano-VLM.Q6_K.gguf	Q6_K	~6.8 GB
Cosmos3-Nano-VLM.Q8_0.gguf	Q8_0	~8.8 GB
Cosmos3-Nano-VLM.bf16.gguf	bf16	~16.5 GB
mmproj-Cosmos3-Nano-VLM.f16.gguf	f16	vision encoder

Run it

# server with vision
llama-server -m Cosmos3-Nano-VLM.Q4_K_M.gguf \
  --mmproj mmproj-Cosmos3-Nano-VLM.f16.gguf -c 32768

# one-shot VQA
llama-mtmd-cli -m Cosmos3-Nano-VLM.Q4_K_M.gguf \
  --mmproj mmproj-Cosmos3-Nano-VLM.f16.gguf \
  --image photo.jpg -p "What will happen next in this scene?" -c 8192

Always pass an explicit -c; without it the context allocation can eat all VRAM and generation crawls.

Verified (RTX 5070 12GB, Q4_K_M + f16 mmproj, -c 8192): correct grounded scene description plus sensible physical next-event prediction on a held-out image; 213 tok/s prefill, 105 tok/s generation.

Provenance

Extraction is a pure tensor re-keying — no weights were modified. The understanding path uses diffusers-style attention naming in the source (to_q/to_k/to_v/to_out, norm_q/norm_k); these map 1:1 onto Qwen3-VL's q_proj/k_proj/v_proj/o_proj/q_norm/k_norm. The extraction script is included in this repo (extract_vlm.py) for full reproducibility.

Original model: nvidia/Cosmos3-Nano, released under the OpenMDW-1.1 license, which this repo inherits. All credit for the model itself to the NVIDIA Cosmos team — see their technical report.