Nemotron-Labs-Diffusion-3B-GGUF

Audited community GGUF conversion for research with AR, 32-token block diffusion, and linear self-speculation through the supplied runner. This is not an official NVIDIA or Ollama release. The non-AR modes require attention and cache control provided by the accompanying runner.

Language-only 3B-family checkpoint. Actual stored language parameter count is listed in the tensor audit.

Model weights retain the NVIDIA Nemotron Open Model License. See the complete license and the unchanged source model card.

Files

File	Size (decimal GB)	Storage
3B-F16.gguf	7.672	F16
3B-Q8_0.gguf	4.080	Q8_0
3B-Q5_K_M.gguf	2.750	Q5_K_M
3B-Q4_K_M.gguf	2.373	Q4_K_M

Choose one language GGUF. The full VLM additionally needs mmproj-F16.gguf for images. Tokenizer files and the chat template are included. F16 baselines are preserved; no importance matrix or further training was used.

Runtime support

Runtime	AR text	Block diffusion	Self-speculation	Images
Supplied pinned runner	Tested	Tested	Tested, sequential verifier	Not included
Stock Ollama 0.33.3	Q8_0 import/raw/chat/multi-turn tested	Not implemented	Not implemented	Not claimed

Ollama runs only the AR path. It cannot acquire diffusion or self-speculation from a Modelfile. Image input is not included in this release.

Run

Build the small bridge using runtime/README.md, with pinned llama.cpp b10760. From the downloaded repository directory:

python runtime/generate.py --model 3B-Q8_0.gguf --tokenizer . \
  --mode diffusion --prompt 'Explain how rain forms.' --max-new-tokens 128
python runtime/generate.py --model 3B-Q8_0.gguf --tokenizer . \
  --mode self-speculation --prompt 'What is 2 + 2?' --max-new-tokens 32
ollama create nemotron-3b-ar -f Modelfile.Q8_0
curl http://localhost:11434/api/chat -d '{"model":"nemotron-3b-ar","messages":[{"role":"user","content":"What is 2 + 2?"}],"think":false,"stream":false}'

The runner and tested Ollama server use GGML_CUDA_CUBLAS_COMPUTE_TYPE=f32. Set it in the server environment for Ollama; it is not a Modelfile parameter. The runner sets it by default. F16 weights/KV remain F16. Read the runtime documentation before overriding this measured precision requirement. Ollama requests must explicitly set think: false for the tested non-thinking behavior; the default thinking mode is not covered by this compatibility result. The supplied Modelfiles use PARSER passthrough to preserve exact non-thinking output, including leading whitespace. Tool calls and thinking-output parsing are outside this configuration.

Validation and limits

All original language tensors match after documented Q/K permutation and dtype conversion. The full VLM's vision tensors are audited separately. Tokenizer and positional metadata checks pass. FP16 language checks have a maximum relative L2 logit error of 0.006608 against the pinned NVIDIA FP16 eager reference under the recorded test environment.

FP16 generation across 4 selected text/image fixtures matches the NVIDIA reference for all three modes: true. This is a limited regression result, not a general cross-backend equality claim. Detailed outputs and all numerical checks are in reports.

The 973-token, 12-probe causal regression suite is author-created, with prose, code, mathematics, and multiple languages. It is not a standardized benchmark or a downstream accuracy estimate.

Storage	Mean KL vs F16	NLL increase	Top-1 agreement
F16	0.000000	+0.000000	100.0000%
Q8_0	0.003846	+0.009235	96.3001%
Q5_K_M	0.021133	+0.034068	92.8058%
Q4_K_M	0.057498	+0.030288	88.6948%

Quantized outputs may differ. Sequential self-speculation is checked against each quant's own AR output, not required to reproduce F16 text. The optional batched verifier has separate equality measurements in each validation report.

Important boundaries:

• Greedy decoding only. Self-speculation defaults to one-token AR verification, retaining exactly the verified prefix. No throughput improvement is claimed.
• --verification batched is an experiment; changing batch shape may change close floating-point token decisions.
• This linear verifier does not reproduce the upstream VLM quadratic SBD lattice, draft LoRA, or distribution-preserving stochastic speculation.
• Positional checks near 16384/32768 use synthetic consecutive short prefixes, not a full long-context benchmark. The example context is 4096.
• Image tests use simple synthetic fixtures, not an OCR/VQA benchmark.
• Thinking, tools, audio, video, streaming, and other GPUs/CPUs are not covered by these release claims. The Ollama template targets plain non-thinking chat.

Provenance

• Upstream: nvidia/Nemotron-Labs-Diffusion-3B@0d51902da1f8869f83413ce642fab402fa5641e0.
• llama.cpp: 0f3a71be15af836d277c9f918adfafb45732677e.
• Conversion/runtime source commit: 1392f98ac1d0be69baf6e425c06af98d20910aa1.
• Hardware: NVIDIA A100-SXM4-40GB; full dependencies in reports/environment.json.
• Full file hashes: SHA256SUMS, manifest.json.

Model weights and upstream tokenizer assets retain NVIDIA's terms. The original IntellAgents runner/conversion additions use MIT; that does not relicense the model. Conversion changes tensor layout, storage type, metadata, and optional quantization; it performs no training.