Minachist/Muse-Glimmer-30B-INT8-AutoRound

Muse-Glimmer-30B-W8A16-AutoRound

This is an unofficial mixed-precision INT8 quantized version of Muse-Glimmer-30B. It was created using AutoRound.

Quantization details

Some layers are not quantized as quantizing them causes a heavy downgrade, or there is no calibration signal available to quantize them with. The model shrinks from 59.5 GB to 33.5 GiB.

Precision	Layer Count	Target Layers
INT8 (group_size=-1, symmetric)	260	o_proj, gate_proj of self_attn, gate/up/down_proj of mlp
FP16 (unquantized)	461	q_proj / k_proj / v_proj of self_attn, embed_tokens, lm_head, vision_tower, vision_adapter, vision_projection

• k_proj and v_proj are kept unquantized because this model uses GQA with a 16:1 ratio (32 query heads, 2 KV heads), so an error in one KV element is amplified across 16 query heads. They are also only [256, 6656] each, so keeping all 104 of them in FP16 costs about 354 MB.
• q_proj is unquantized for a mechanical reason rather than an accuracy one. vLLM's native implementation fuses q/k/v into a single QKVParallelLinear, and one packed tensor cannot hold two different quantization schemes, so the loader rejects any checkpoint where the three disagree. Raising q_proj to FP16 costs 1.32 GiB; dropping k_proj and v_proj to INT8 instead would have been cheaper but would have discarded the reason above. Revisions of this repo before 2026-08-11 had q_proj at INT8 and fail to load on that path.
• The vision tower is kept unquantized.
• Calibration configuration: iters=250, nsamples=512, batch_size=4, with gradient_accumulate_steps=2.

A note for anyone reproducing this with a different layer configuration: the shared-dict aliasing bug in AutoRound is still present in 0.15, it has only moved from compressors/utils.py to compressors/layer_config/resolver.py. The regex expansion still assigns the same dict object to every matched layer, and the later shape check that forces bits=16 on layers whose weights are not divisible by 32 mutates that shared dict, silently dragging every sibling layer to 16 bit with it. Change

layer_config[match] = val

to

layer_config[match] = copy.deepcopy(val)

if any of your keywords can also match a shape-incompatible layer. It did not affect this model, since every layer selected for INT8 here has both dimensions divisible by 32, and the only shape-incompatible layer in the model is the vision patch embedding, which is excluded anyway.

Evaluation Results

1. KL Divergence (KLD)

The output distribution of the quantized model was compared directly to that of the base model across 122,640 tokens.

Metric @ctx 512	Value
Mean KLD	0.00130 ± 0.000015
Median KLD	0.000644
P90 / P95 / P99 KLD	0.00275 / 0.00403 / 0.00928
P99.9 / Max KLD	0.0338 / 0.8118
Top-1 Agreement	98.53%
PPL (Base → Quantized)	7.8194 → 7.8241
ln(PPL Ratio)	0.000608

Evaluation set: WikiText-2 (test), 240 sequences × 512 tokens = 122,640 scored next-token positions. Values are in nats.

• The tail is light for an INT8 quant, with a maximum per-token KLD of about 0.81 nats. Whether the untouched KV projections deserve the credit is a guess on my part, not something I isolated with an ablation.
• Raising q_proj from INT8 to FP16 is the only difference from the previous revision, and both runs were scored against the same cached reference distribution, so this one is a clean single-factor comparison. It moved mean KLD from 0.00136 to 0.00130, max KLD from 1.049 to 0.8118, and ln(PPL ratio) from 0.000750 to 0.000608. It says nothing about k_proj and v_proj, which were FP16 in both runs.

2. Benchmarks

Not run. you have to do it yourself.

How to Use

There are two ways to serve this model as of 2026-08-11 and they behave differently.

Native support lives in vLLM PR #51655, which is still open and is in neither a release nor main. The preview image vllm/vllm-openai:muse-glimmer carries it, resolves the model to MuseGlimmerForCausalLM, and ships the muse_glimmer tool-call and reasoning parsers. This is the faster path and the one this quant is now aligned with.

Released and nightly vLLM have no native muse_glimmer implementation and fall back to TransformersMultiModalForCausalLM. That path still works, with these caveats, none of which are caused by the quantization.

• --tool-call-parser muse_glimmer and --reasoning-parser muse_glimmer do not exist there and will abort startup if you pass them. Without them the model still generates fine, but the reasoning channel is not stripped from the response.
• With tensor parallelism the vision tower fails during multimodal memory profiling with mat1 and mat2 shapes cannot be multiplied (16384x768 and 1536x1536), where 768 is 1536 split across 2 ranks. This happens inside default_unquantized_gemm, on weights this quant never touched. Pass --language-model-only. Setting mm_encoder_tp_mode=data does not help, because the Transformers backend reports supports_multimodal_encoder_tp_data=False and falls back to weight splitting.
• The Transformers backend does not fuse q/k/v, which is why revisions of this repo before 2026-08-11 loaded there despite mixing INT8 q_proj with FP16 k_proj and v_proj. The native implementation does fuse them, and that is what forced the layer table above to change.

Measured on 2x RTX 3090 with --tensor-parallel-size 2 --language-model-only --gpu-memory-utilization 0.95 --max-model-len auto --max-num-seqs 1 -O3 --async-scheduling, without kv-cache quantization.

Path	Weights + non-torch per rank	KV cache	Concurrency @ 131,072 ctx
Native, vllm/vllm-openai:muse-glimmer	15.54 GiB	822,190 tokens	6.27x
Transformers fallback, vllm/vllm-openai:nightly	18.92 GiB	460,638 tokens	3.51x

You can append --kv-cache-dtype fp8 to optimize memory further.

Speculative decoding with the DFlash drafter that the vLLM recipe advertises does not currently work, for reasons unrelated to this quant. Four separate defects are present at the PR head: the draft architecture name that EAGLEConfig derives, DFlashMuseGlimmerAssistantModel, is absent from the model registry, which only has the unprefixed MuseGlimmerAssistantModel; the default SupportsEagle3 hooks assume a two-level language_model.model layout that MuseGlimmerForCausalLM does not have; vLLM parses the drafter config as Qwen3Config, which discards sliding_window unless use_sliding_window is also set; and the Qwen3 DFlash loader does not recognise the drafter's encoder. weight prefix. The first of these fires before any weight is read, so the feature cannot have been exercised on any hardware.

Acknowledgements

• Meta Superintelligence Lab for the base Muse-Glimmer-30B model.
• Intel AutoRound team for the quantization framework.
• vLLM project for the inference engine.