Muse-Glimmer-30B-NVFP4

NVFP4 (4-bit weights, 4-bit activations) quantization of meta-models/Muse-Glimmer-30B, made with llm-compressor. 55.5 GiB to 21.8 GiB.

Quantized and tested on a single NVIDIA GB10 (DGX Spark, sm_121, 121 GiB unified memory).

Requirements

Muse Glimmer is not in any released vLLM. Support comes from PR #51655, which is still open. As of 2026-08-11 that PR is enough to serve this checkpoint.

Usage

vllm serve abstract-extraordinary/Muse-Glimmer-30B-NVFP4 \
  --max-model-len 131072 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.45 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
  --reasoning-parser muse_glimmer \
  --tool-call-parser muse_glimmer \
  --enable-auto-tool-choice

Both parsers are required, not optional. The model emits ATEM XML tool calls on channel-scoped messages and both parsers key off that framing. On GB10, cudagraph_mode: PIECEWISE avoids a silent hang. Multi-modal warmup failed at startup is harmless; image requests work.

What is quantized

416 Linear modules in the text decoder: 52 layers of q,k,v,o,gate plus mlp.gate,up,down. Weights are 4-bit float at group size 16 with FP8 group scales and an FP32 per-tensor global scale; activations are 4-bit float, quantized dynamically per group at runtime. Format is nvfp4-pack-quantized (compressed-tensors 0.15.1).

	precision	params
Text decoder linears	NVFP4	25.16 B
lm_head	bf16	1.35 B
Vision tower, adapter, projection	bf16	1.92 B
Embeddings, norms	bf16	1.35 B

The vision tower is bf16 because vLLM's Muse Glimmer implementation constructs every vision linear without a quant_config, so a quantized ViT cannot be loaded at all. It costs about 3.6 GiB. lm_head is bf16 by convention.

Evaluation

GSM8K, first 250 test questions, zero-shot with an explicit #### <number> answer format, greedy, exact match. Both checkpoints were served by the same vLLM build with identical settings and scored by the same harness.

	accuracy	95% CI
bf16	94.0% (235/250)	91.1 – 96.9
NVFP4	96.0% (240/250)	93.6 – 98.4

The two-point gap is not significant and should not be read as an improvement. Paired over the same questions: 234 correct under both, 9 wrong under both, 6 correct only under NVFP4, 1 only under bf16, McNemar exact p = 0.125. Greedy decoding in vLLM is also not bit-reproducible between runs. The result supports no measurable quantization loss, and nothing stronger.

Five NVFP4 and six bf16 generations hit the 2048-token cap and score as wrong, which pulls both numbers down about equally. No request errors on either side. At n=250 the eval resolves differences of roughly three points.

Text, image input and tool calling were each checked after quantization and behave as before.

Performance

Single GB10, TP=1, no speculative decoding. Both columns come from the GSM8K runs above.

	NVFP4	bf16
weights resident	22.2 GiB	55.8 GiB
weight load	157 s	351 s
KV cache at 8k context	1,322,319 tok	704,268 tok
generation, 32 concurrent	260.9 tok/s	98.8 tok/s
250 questions, wall clock	604 s	1530 s

At 131k context the KV cache is 29.3 GiB (1,645,658 tokens) for 12.56x maximum concurrency, with about 62 GiB resident against roughly 90 GiB for bf16.

Single-stream decode measures 12.7 tok/s. There is no paired bf16 single-stream figure, so use the batch numbers above for comparison.

Method

QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["re:.*lm_head", "re:.*vision_tower.*",
            "re:.*vision_adapter.*", "re:.*vision_projection.*"],
)

Calibration used 32 samples from neuralmagic/calibration (LLM split) at 1024 tokens, rendered through the model's own chat template so the channel framing is present. Text only, since the vision tower is excluded.

llm-compressor traced the model into 53 sequential subgraphs, the 52 decoder layers plus one. The 50 vision layers do not trace under text-only calibration, which is expected and harmless given they are excluded anyway.

Building the model requires transformers from main. muse_glimmer is in no released version and the base repo ships no remote code.

Limitations

• Requires a vLLM build carrying PR #51655, which is still unmerged.
• Vision output was checked qualitatively but not benchmarked. The vision tower is bf16 and bit-identical to the original; only the decoder it feeds changed.
• Speculative decoding with the DFlash drafter is untested against this target. The drafter was trained against the bf16 model.
• Accuracy comes from one task and 250 samples. It says nothing about long-context, multilingual or coding behaviour.

License

Apache 2.0, inherited from the base model, whose LICENSE and USAGE_POLICY.md are included here and apply unchanged. All credit for the model belongs to its original authors; this repository holds a quantized copy of their weights.