d3 · NVFP4

An NVFP4 quantization of vllm-sr/d3 (revision v3.1.0), the 27B multimodal Decision 3.0 model, made with llm-compressor and served by stock vLLM nightly with no patches through the /v1/systemone endpoint. The sibling quant is W4A16.


Weights	17.2 GB (`model.safetensors`, one file), against 52.2 GB for BF16
Scheme	NVFP4 (W4A4): FP4 (E2M1) weights and activations in groups of 16 with FP8 (E4M3) group scales and an FP32 global scale, GPTQ-calibrated
Quantized	every language-model Linear layer (attention q/k/v/o, Gated DeltaNet `in_proj_qkv`/`in_proj_z`/`out_proj`, MLP)
Kept in BF16	vision tower, Gated DeltaNet `in_proj_a`/`in_proj_b` (48-wide gates), embeddings, norms; `readout.safetensors` is the original FP32 readout, unchanged
GPUs	NVIDIA Blackwell for native FP4 (B200/B300, RTX PRO 6000/4500 Blackwell, RTX 50-series). Tested only on Blackwell (RTX PRO 4500); on older GPUs use the W4A16 quant
Format	compressed-tensors (`nvfp4-pack-quantized`), loadable by vLLM

Accuracy against the original

Every number below was measured through vLLM's /v1/systemone endpoint with the same prompts, one question per request. BF16 is the original vllm-sr/d3 checkpoint served the same way. Deltas are against BF16; agreement is how often the quant picks the same option as BF16 on MMLU-Pro (10 options).

Model	GPQA Diamond, 4 options (198)	MMLU-Pro, 10 options (2,000)	MMLU-Pro, 1 correct + 4 random (2,000)	Top-1 agreement with BF16
vllm-sr/d3 BF16 (original)	58.08	71.90	80.35	—
NVFP4, bidirectional (as trained)	52.53 (-5.56)	69.95 (-1.95)	78.85 (-1.50)	84.2%
NVFP4, causal (prefix caching)	45.45 (-12.63)	65.25 (-6.65)	76.65 (-3.70)	77.3%

GPQA Diamond: the 198 questions with their 4 options. MMLU-Pro 10: 2,000 test questions with all 10 options. MMLU-Pro 5: the same 2,000 questions reduced to the correct option plus 4 options drawn at random (seed 1234). Each question is the state, Which option correctly answers the question? is the instruction, and the options are the choice criteria.
Calibration data never overlaps the evaluations (below).
Causal mode is not how d3 was trained (see the next section); its loss comes from the attention change, not from quantization.

Running with vLLM

Requires vLLM nightly (tested on 0.32.1rc1.dev60, CUDA 13, PyTorch 2.14) and transformers>=5.17:

pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

vLLM detects decision_config.json, serves the checkpoint as D3ForSequenceClassification and answers typed questions at POST /v1/systemone (request format: the Quickstart below, or the vLLM Structured Decisions docs). Two modes are available; pick by workload.

1. Bidirectional (default, best accuracy)

This is how d3 was trained: the full-attention layers read the whole prompt in both directions. vLLM therefore turns prefix caching and chunked prefill off, and every question is one full forward pass over its prompt. Do not pass --enable-chunked-prefill: chunking a bidirectional prompt changes the answers (near-uniform probabilities once a prompt is longer than one chunk).

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True vllm serve crazymartian/d3-NVFP4 \
  --max-model-len 81920 --max-num-batched-tokens 81920 \
  --max-num-seqs 16 --max-cudagraph-capture-size 16 \
  --kv-cache-memory-bytes 2G --gpu-memory-utilization 0.97

--max-num-batched-tokens must equal --max-model-len (no chunked prefill). The limit is activation memory, not KV cache: one forward holds the whole prompt.
81,920 tokens is the longest prompt that serves reliably on one 32 GB GPU. On 2 GPUs within 48 GB in total, add --tensor-parallel-size 2 --disable-custom-all-reduce and use 90112 for both lengths, --kv-cache-memory-bytes 1G, --gpu-memory-utilization 0.75 (measured peak 43.9 GB with an 89.5k-token prompt).
--max-model-len auto picks 262,144 and fails to start: vLLM sizes it from the KV cache, which d3's encoder-style layers barely use.
--max-cudagraph-capture-size 16 lets CUDA graphs fit next to a long-prompt forward; it was the fastest setting tested.
Leave out --kv-cache-dtype fp8: vLLM does not support a quantized KV cache for encoder attention.

2. Causal, with prefix caching (agents with repeated context)

Switch attention_mode in decision_config.json to causal and vLLM serves d3 like a causal model: chunked prefill and prefix caching work, so a long agent state is prefilled once and each further question, or each turn the agent appends, prefills only the new tokens. The weights are unchanged; the accuracy cost is in the table above.

hf download crazymartian/d3-NVFP4 --local-dir d3-NVFP4-causal
python - <<'EOF'
import json
p = "d3-NVFP4-causal/decision_config.json"
c = json.load(open(p)); c["attention_mode"] = "causal"; json.dump(c, open(p, "w"), indent=2)
EOF

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True vllm serve ./d3-NVFP4-causal \
  --max-model-len 153600 \
  --enable-chunked-prefill --max-num-batched-tokens 8192 \
  --enable-prefix-caching \
  --max-num-seqs 16 --max-cudagraph-capture-size 16 \
  --gpu-memory-utilization 0.95

150k-token context with prefix caching fits on one 32 GB GPU (about 200k tokens of KV cache left after the weights).
--max-num-batched-tokens 8192 bounds activation memory per step; long prompts are prefilled in 8k chunks.
Without prefix caching (states that never repeat), drop --enable-prefix-caching and keep the rest. Each request then costs what the cold rows below show.

Measured speed

One NVIDIA RTX PRO 4500 Blackwell (32 GB), one request at a time unless stated; the long state is ~61.5k tokens (causal) or ~63.6k tokens (bidirectional) of text with one question.

	Bidirectional	Causal + prefix cache
Short questions (~300 tokens), 64 concurrent	24.7 req/s, p50 57 ms	not measured
64k-token state, cold	24.2 s	15.9 s
Same 64k state, a new question	24.2 s (no reuse)	0.46 s
Same state + ~1.5k appended tokens	24.2 s (no reuse)	0.74 s
143k-token state, cold	not servable on one 32 GB GPU	62 s

Every long-context request above answered a question about a fact placed in the middle of the state correctly.

Using the bundled Transformers runtime

AutoModel.from_pretrained(..., trust_remote_code=True) loads the original runtime (modeling_d3.py, d3_runtime.py), which builds a BF16 backbone and does not read compressed-tensors weights. Use vLLM for this checkpoint; the runtime files are kept unchanged from the original repository for reference and for d3_format.py, the prompt contract every server follows.

How it was made

llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, PyTorch 2.14; GPTQModifier, scheme NVFP4, block size 128, dampening 0.01, sequential pipeline over the 64 decoder layers. recipe.yaml holds the recipe.
Calibration ran the backbone with its full-attention layers bidirectional (is_causal=False), as d3 serves.
1,024 calibration prompts (297k tokens) rendered exactly as d3_format.render writes them (system prompt, state, question, coded options): AG News, dair-ai/emotion, IMDB, CommonsenseQA, ARC-Challenge (train splits) and UltraChat (test_sft), as choice, yes/no and score questions. None of them is in the evaluations.
config.json lists score (vLLM's name for the readout head) in the quantization ignore list, so vLLM keeps the readout in full precision.

Limitations

Same as the original model, plus quantization error: answers differ from BF16 on the share of questions shown above, mostly near-ties.
Full 262,144-token context does not fit in bidirectional mode on 32 or 48 GB with stock vLLM (activation memory of a single forward pass), and causal mode trades accuracy for it.
NVFP4 needs Blackwell tensor cores for its speed; it loses more accuracy than W4A16 (most on GPQA).
vLLM Semantic Router (vllm_srun) does not load NVFP4 (it supports compressed-tensors int4 and FP8 only); use W4A16 there. MODEL_MANIFEST.json here lists the file hashes for integrity checks.

Everything below is the original model card of vllm-sr/d3, unchanged. Its evaluations are of the BF16 model.

d3

d3 is the 27B multimodal foundation decision model of Decision 3.0, the decision models of vLLM Semantic Router. Give it an input (text or JSON, optionally with images and videos) and the questions you need answered: pick one of several options, say yes or no, or rate on a scale. It answers them all in one call and returns a probability for every answer, without generating text.


Parameters	26.09B, including the 0.46B vision encoder
Inputs	Text or JSON, plus images and videos (several per request)
Decision types	Choice · Yes / No · Score
License	Apache-2.0

Highlights

Jev Decision Index 0.3, public suite: 65.45, measured with the official 0.3 kit on the released weights: all 140,178 public requests answered, none unsupported.
+8.5 on the public suite over Decision 2.0 (its 27B model: 56.97 on the board), ahead in all five areas.
Reads images: multiple images per request (PNG, JPEG or WebP), given as paths, URLs, PIL images or base64 data URLs; every question of the request sees all of them.
Reads videos: multiple videos per request (MP4, WebM, MOV or MKV), given as paths, URLs, base64 data URLs or frame arrays, read at 2 frames per second; Perception Test (multiple-choice video QA): 77.4 (internal evaluation).
Speed: text requests take a median of 55 ms, requests with an image a median of 273 ms, and requests with a 10-second video a median of 553 ms, on one AMD Instinct MI325X, one request at a time.
Many questions, one call: Choice, Yes / No and Score questions about the same input are answered together, each from its own forward pass over the input, with a probability for every option.

Quickstart

pip install "transformers==5.17.0" torch torchvision pillow opencv-python-headless safetensors accelerate
pip install flash-linear-attention  # optional: fast GPU kernels for the linear-attention layers

import json

from huggingface_hub import hf_hub_download
from transformers import AutoModel

model = AutoModel.from_pretrained("vllm-sr/d3", trust_remote_code=True)

# Text
result = model.system_one(
    state="The order arrived damaged yesterday. The customer has a receipt and asks for a replacement today.",
    questions={
        "route": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "returns": "Refunds, replacements and damaged deliveries",
                "billing": "Payments, invoices and charges",
                "technical": "Product setup and faults"
            }
        },
        "receipt": {
            "type": "noul",
            "instructions": "Does the customer have a receipt?"
        },
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this request?",
            "criteria": [
                "Routine",
                "Soon",
                "Today"
            ]
        }
    },
)
print(json.dumps(result["answers"], indent=2))

# Text and an image (or several)
receipt = hf_hub_download("vllm-sr/d3", "assets/example-receipt.png")
result = model.system_one(
    state="The customer says the blender arrived cracked and attached the receipt.",
    images=[receipt],  # local paths, http(s) URLs, PIL images or base64 data URLs
    questions={
        "route": {
            "type": "choice",
            "instructions": "Which team should handle this request?",
            "criteria": {
                "returns": "Refunds, replacements and damaged deliveries",
                "billing": "Payments, invoices and charges",
                "technical": "Product setup and faults"
            }
        },
        "on_receipt": {
            "type": "noul",
            "instructions": "Does the receipt list the blender?"
        },
        "payment": {
            "type": "choice",
            "instructions": "How was the order paid?",
            "criteria": {
                "card": None,
                "cash": None,
                "gift card": None
            }
        }
    },
)
print(json.dumps(result["answers"], indent=2))

# Text and a video (or several)
clip = hf_hub_download("vllm-sr/d3", "assets/example-video.mp4")
result = model.system_one(
    state="A short clip from a test camera.",
    videos=[clip],  # local paths, http(s) URLs, base64 data URLs or frame arrays
    questions={
        "direction": {
            "type": "choice",
            "instructions": "Which way does the square move?",
            "criteria": {
                "right": "From left to right",
                "left": "From right to left",
                "still": "It does not move"
            }
        },
        "color_change": {
            "type": "noul",
            "instructions": "Does the square change color?"
        }
    },
)
print(json.dumps(result["answers"], indent=2))

# Or as a pipeline:
# transformers.pipeline("decision", model="vllm-sr/d3", trust_remote_code=True)(state=..., questions=..., images=..., videos=...)

Images go before the text of the request, each read at up to 1.6 megapixels; videos follow the images, read at 2 frames per second (at most 32 frames spread over the whole video, each at up to 0.2 megapixels). Every question of the request sees all of them.

Evaluation

Text: Jev Decision Index 0.3.1

Model	Jev Decision Index ↑	Public ↑	Same-skill tests ↑	New-domain tasks ↑
d3	64.7	65.5	62.8	56.6
Perplexity Decider v1.1 (27B)	62.8	62.3	61.1	55.6
Fastino GLiDE no-thinking (28B)	60.2	59.1	59.5	52.9
Jev	60.1	58.0	58.0	55.0
Torchcast Decision 27B	59.9	65.1	58.1	50.8
Decision 2.0 (27B)	55.9	57.0	55.7	47.9

Images: Jev Decision Index vision board 0.3.1

Model	Vision Index ↑	Public ↑	Private ↑
d3	71.6	74.2	68.9
Perplexity Decider v1.1 (27B)	70.6	73.2	67.9
JEV-27B-VL	69.6	72.8	66.4
Solomon v1.1 (27B)	66.8	69.9	63.8

d3 on the public vision benchmarks († approximate rebuild):

Benchmark	d3
CV-Bench	76.2
BLINK	59.9
RealWorldQA	73.4
CharXiv †	91.7
InfographicVQA †	96.7
Mind2Web †	90.0
Winoground	85.8
KIE (CORD+FUNSD) †	97.7
Moderation (Hateful Memes)	43.9
R-Bench-M	33.0
MMMU-Pro vision	47.0

Videos: Perception Test

Perception Test (validation)	d3
All 19,140 questions	77.4
Memory	80.3
Abstraction	71.2
Physics	72.5
Semantics	88.8

Multiple-choice video question answering (three options per question, chance 33.3): top-1 accuracy (%), videos read with the defaults above.

_{d3: internal evaluation. Others: live board data, text 2026-10-10, vision 2026-10-09.}

License

Apache-2.0 (LICENSE). Built on Qwen/Qwen3.8-27B (Apache-2.0).

Citation

@misc{d3_2026,
  title        = {{d3}: A Multimodal Foundation Decision Model},
  author       = {{vLLM Semantic Router Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/vllm-sr/d3}}
}

Trained on AMD Instinct MI325X GPUs.

Downloads last month: 4

Safetensors

Model size

26B params

Tensor type

BF16

Model tree for crazymartian/d3-NVFP4

Base model

Qwen/Qwen3.8-27B

Finetuned

vllm-sr/d3

Quantized

(17)

this model