Muse-Glimmer-30B-w8a8-llmcompressor

Model Overview

• Model Architecture: MuseGlimmerForConditionalGeneration
  • Input: Text
  • Output: Text
• Source Model: Muse-Glimmer-30B
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.28.0
• Quantization Framework: LLM Compressor v0.13.0
• Quantization Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• Compatible Stack:
  • ZenDNN v6.1.0
  • ZenTorch v2.13.0
  • PyTorch v2.13.0
  • Transformers v5.15
  • LLM Compressor v0.13.0
  • vLLM v0.28.0
• Published with: LLM Compressor v0.13.0

This is a quantized version of Muse-Glimmer-30B created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Muse-Glimmer-30B using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 55.5 GiB to 32.0 GiB on disk (~42% reduction).

• Method: 8-bit Weight, 8-bit Dynamic Activation Quantization (W8A8)
• Config: compressed-tensors, num_bits=8, type=int, symmetric=true
• Weights: INT8, symmetric, per-channel (static)
• Activations: INT8, symmetric, per-token (dynamic)
• Quantized: the dense text tower across all 52 layers — self_attn.{q,k,v,o}_proj plus the gated self_attn.gate_proj, and mlp.{gate,up,down}_proj.
• Kept in BF16: the vision tower (model.vision_tower), the vision adapter and projector (model.vision_adapter, model.vision_projection), lm_head, embed_tokens, and the layer norms.

The vision path stays in BF16 because this is a data-free quantization pass and the vision encoder's activation statistics are not represented at all. That, together with the large untouched lm_head and embed_tokens (202,048 x 6,656 each), is why the reduction lands at ~42% rather than the ~50% a pure text-only INT8 model would give.

Note that every attention block carries a self_attn.gate_proj, which is a real projection and is quantized. It is not a router and must not be confused with an MoE gate: Muse-Glimmer is a dense model.

import torch
from transformers import AutoProcessor, AutoTokenizer, MuseGlimmerForConditionalGeneration

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

model_id = "RedHatAI/Muse-Glimmer-30B"
output_dir = "./Muse-Glimmer-30B-w8a8-llmcompressor"

# Step 1: Load the BF16 model and tokenizer.
# Load the top-level MuseGlimmerForConditionalGeneration rather than
# AutoModelForCausalLM, which would demote config.json to the inner text-only LM
# and produce a checkpoint vLLM rejects.
model = MuseGlimmerForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Define the W8A8 recipe. The text tower (gated attention + dense MLP)
# is quantized; the vision tower, adapter and projector stay BF16 because a
# data-free pass has no vision activation statistics to work from.
recipe = QuantizationModifier(
    scheme="W8A8",
    targets=["Linear"],
    ignore=[
        "lm_head",
        r"re:.*lm_head",
        r"re:.*vision_tower.*",
        r"re:.*vision_adapter.*",
        r"re:.*vision_projection.*",
    ],
)

# Step 3: One-shot quantize and save in compressed-tensors format.
# W8A8 here is data-free (RTN), so no calibration dataset is needed.
oneshot(
    model=model,
    recipe=recipe,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# oneshot does not save the processor; multimodal checkpoints need it for vLLM.
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
processor.save_pretrained(output_dir)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Muse-Glimmer-30B-w8a8-llmcompressor",
    dtype="bfloat16",
    trust_remote_code=True,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.13.0
zentorch==2.13.0
transformers==5.15
vllm==0.28.0
llmcompressor==0.13.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W8A8 (this model)	Recovery
GSM8K (5-shot)	0.5830	0.6073	104.17%

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Muse-Glimmer-30B-w8a8-llmcompressor,dtype=bfloat16,language_model_only=True \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

• Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0 / PyTorch v2.13.0. It may not load correctly on other versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• Vision Path Unquantized: The vision tower, adapter and projector remain in BF16, so the memory saving is smaller than for a text-only W8A8 model and image preprocessing cost is unchanged. Evaluation was run with language_model_only=True.

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.