nemo-vision-9b-v2 projector

A 129 M-parameter projector that gives nvidia/NVIDIA-Nemotron-Nano-9B-v2 vision through a LLaVA-style embedding splice. The LLM weights stay bit-identical to the base checkpoint — the projector is the only trained component.

The base model is a hybrid Mamba-2/Transformer LLM that was width- and depth-pruned with Minitron from the 12B Nemotron (56 layers, hidden 4480, only 4 attention layers). NVIDIA ships an official VL only for the unpruned 12B sibling. This adapter is the measured answer to whether the pruned hybrid still holds enough capacity to absorb a vision modality: it does, through the splice path alone.

Full process, training code, and ablations: github.com/benthecarman/nemo-vision.

Usage

git clone https://github.com/benthecarman/nemo-vision
cd nemo-vision && uv sync
CUDA_VISIBLE_DEVICES=0 uv run python scripts/infer.py \
    --image img.jpg --prompt "Describe this image briefly."

scripts/infer.py pulls this projector from the hub by default. First run downloads the LLM (~18 GB) and the RADIO tower; inference needs ~24 GiB VRAM in bf16.

Pipeline: image @ 512 px → RADIOv2.5-B (frozen) → 32x32 patch grid → pixel shuffle f=2 → 256 tokens of 3072 dims → 2-layer MLP projector → spliced into inputs_embeds at marker id 900. Prompt format [INST] {prompt} <IMG> [/INST]; greedy decode (the repo's manual loop — NemotronH lacks generate(inputs_embeds=)).

Artifacts

file	what it is
projector_stage2_p0.pt	The shipped model. Stage-1 alignment (1 epoch LLaVA-558K) + stage-2 instruction tuning (1 epoch LLaVA-Instruct-150K), projector-only.
projector_stage1.pt	Alignment only, after the LLaVA-558K epoch.
config.json	Marker id, factor, dims, prompt format.

Results

Vision — held-out COCO val2017 captions, teacher-forced CE (lower is better), same 200 images and prompt for every row:

model	caption CE
frozen projector (random init)	3.850
this model (projector-only, both stages)	3.363
LoRA on 4 attention layers (54.7 M params)	4.536
LoRA + mamba in_proj (54.6 M params)	4.177
official 12B VL (unpruned, 7x vision tokens, full vendor training)	2.864

The LoRA rows are the rejected arms: every adapter parameter added text damage with no vision gain over the projector-only path.

Text — lm-eval 0.4.12, completion mode, one pass (the LLM is untouched, so these are the base model's own scores):

benchmark	setting	score
MMLU	5-shot	74.63
GPQA-Diamond	0-shot	35.35
IFEval	0-shot	prompt-strict 46.21
MATH500	4-shot	31.0 (math_verify)

Card numbers for the base model (NeMo-Skills, reasoning-on) are not comparable to this completion-mode column.

Limitations

• Close-up ambiguity confabulates: on a single zebra grooming itself, this model and the official 12B VL both describe a mother and nursing foal. Treat outputs as plausible, not verified.
• Behind the official 12B VL on every vision instrument; this is a capacity experiment, not a production VL model.
• Single-turn [INST] prompt only; no chat mode, no reasoning-mode integration, no safety tuning beyond the base model's.
• 256 fixed vision tokens; no dynamic tiling.

License

• Projector weights (this repo): MIT. Trained on LLaVA-558K and LLaVA-Instruct-150K (CC-BY-4.0 data; images belong to their owners) and COCO val2017 captions (CC-BY-4.0).
• Use requires the LLM — NVIDIA Open Model License.
• Use requires the RADIO tower — NVIDIA Source Code License, non-commercial. Practical use of this adapter inherits that restriction.