JisuHann/groot-vlm-graft28

GR00T N1.5 VLM — graft28

The vision-language path of NVIDIA GR00T N1.5-3B (backbone.eagle_model), made runnable as a standalone VLM.

The released GR00T checkpoint ships only 12 of the 28 LLM decoder layers — the policy never needs the rest. This repo fills layers 12–27 from public Qwen3-1.7B so the language model is complete and can generate text.

This is not NVIDIA's VLM. Layers 12–27 are public Qwen3-1.7B weights, not GR00T's. Treat it as a research artifact for probing what GR00T's visual encoder and lower LLM layers represent, not as an official release.

What came from where

Part	Source
Vision tower (SigLIP2, 27 layers, 1152-d, patch 14, 224px)	GR00T N1.5-3B checkpoint
Connector mlp1.0 — a single Linear(1152 → 2048)	GR00T N1.5-3B checkpoint
LLM layers 0–11	GR00T N1.5-3B checkpoint
LLM layers 12–27	public Qwen3-1.7B
Final norm, lm_head, embed_tokens	GR00T N1.5-3B checkpoint

28 LLM layers, 2.15B parameters, bfloat16. tie_word_embeddings=True, so lm_head.weight is not stored separately (it equals embed_tokens.weight in the source checkpoint, verified).

Build script: build_graft28.py. Sources and verification: provenance.json. All 760 saved tensors are byte-identical (torch.equal) to the tensor they came from — no conversion, no re-quantization.

Usage — transformers

from transformers import AutoModel, AutoProcessor

D = "JisuHann/groot-vlm-graft28"
model = AutoModel.from_pretrained(D, trust_remote_code=True, torch_dtype="bfloat16").cuda().eval()
proc = AutoProcessor.from_pretrained(D, trust_remote_code=True)

Tested with transformers 4.51.3 / torch 2.7.1. Two gotchas:

• radio_model.py (vendored GR00T code) calls torch.cuda.get_device_capability() at import time, so at least one GPU must be visible even for a CPU-only run.
• The config pins flash_attention_2, so CPU forward passes fail. For CPU, pass attn_implementation="eager" for both the vision and text configs.

smoke_test.py loads the model from this repo and re-asks one sample from the experiment below.

Image tokens

No pixel shuffle (use_pixel_shuffle: false): all 256 patches of a 224px tile go through mlp1.0, giving 256 image tokens per tile (GR00T's own preprocessor_config.json says tokens_per_tile: 256). A 640×480 frame at native resolution tiles into 12 + thumbnail = 13 tiles = 3,328 tokens.

Usage — vLLM

vLLM registers Eagle2_5_VLForConditionalGeneration, but its implementation follows NVIDIA's public Eagle2.5-VL, which differs from this checkpoint in three ways:

	this model (GR00T)	vLLM's built-in Eagle2.5-VL
connector	one Linear(1152 → 2048)	LayerNorm + Linear + GELU + Linear
pixel shuffle	never	always, scale 0.5 (256 → 64 tokens/tile)
image normalization	SigLIP, mean/std 0.5	ImageNet, 0.485… / 0.229…

vllm_plugin/ is a small plugin that fixes exactly those three things and nothing else. Setting downsample_ratio: 1.0 already makes vLLM's pixel_shuffle a no-op (the two permutes cancel, every view() keeps its shape) and sets image_seq_length to 256, so the plugin only overrides the connector shape and the normalization constants.

pip install -e vllm_plugin/          # registers GrootEagle2_5_VLForConditionalGeneration
bash make_vllm_dir.sh ./graft28-vllm # weights symlinked, config from vllm/config.json
vllm serve ./graft28-vllm --served-model-name groot-vlm-graft28 \
  --max-model-len 8192 --gpu-memory-utilization 0.30

vllm/config.json is this repo's config with three keys changed: downsample_ratio: 1.0, min_dynamic_patch / max_dynamic_patch (vLLM reads the _patch spelling; GR00T writes _tiles), and the architecture name the plugin registers. The weights are untouched and shared with the transformers copy.

Where it came from

Built for a probe of what GR00T N1.5 "sees" in RoboCasa safe-navigation scenes: 50 samples (5 layouts × 5 obstacles × blocking/non-blocking), asked to name the obstacle on the floor, at policy resolution (224px) and native 640×480.

Sample result, egocentric view, native resolution: the obstacle is in the model's top-3 for 24/31 visible cases (77%, chance 16.7%), mean rank 2.5, AUROC 0.973 for separating visible from occluded; at 224px that drops to 45% top-3 and mean rank 6.4. Layers 12–27 being Qwen3 rather than GR00T is a confound for anything that depends on the upper LLM.

License

The GR00T-derived weights follow NVIDIA's license for GR00T-N1.5-3B; the vendored modeling code is NVIDIA's. Layers 12–27 come from Qwen3-1.7B (Apache-2.0). Check both before redistributing.