aisquared/Muse-Glimmer-30B-W4A16-LLMCompressor

Muse Glimmer 30B — W4A16 (llm-compressor / compressed-tensors)

Uniform 4-bit weight-only quantization of meta-models/Muse-Glimmer-30B built with llm-compressor (GPTQ). Output format: compressed-tensors (pack-quantized, group 128) — loaded natively by vLLM. Not GGUF.

Scheme	W4A16 (INT4 weights, BF16 activations), group size 128, symmetric
Method	GPTQ Hessian rounding
Quantized	language-decoder Linear layers
Kept BF16	lm_head (202k vocab), vision tower, multimodal projector
Calibration	HuggingFaceH4/ultrachat_200k, chat template, 512 samples @ 2048 tokens

Serve with vLLM

Use a vLLM build containing the Muse Glimmer implementation (merged PR #51655, 2026-08-14), e.g. the vllm/vllm-openai:muse-glimmer or :nightly image. The compressed-tensors config is auto-detected — no --quantization flag.

docker run --rm --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:muse-glimmer \
  aisquared/Muse-Glimmer-30B-W4A16-LLMCompressor \
  --served-model-name muse-glimmer \
  --generation-config auto \
  --enable-auto-tool-choice \
  --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer \
  --max-model-len 32768

Notes:

• Use both parsers together (muse_glimmer, underscores). Don't run greedy; the published sampling settings are temp 1.0 / top_p 0.95 / top_k 64.
• Multiple tool calls arrive as consecutive assistant messages.
• Add --enforce-eager when serving wNa16 on Intel GPU/CPU.

Siblings

• aisquared/Muse-Glimmer-30B-bpw4-AutoRound — AutoRound AutoScheme mixed-bit build (~4 bpw avg, sensitivity-driven)
• aisquared/Muse-Glimmer-30B-bpw2.5-AutoRound — AutoRound AutoScheme mixed-bit build (~2.5 bpw avg, aggressive)