Instella-MoE-16B-A3B-SFT-w4a16-llmcompressor

Model Overview

• Model Architecture: InstellaMoEForCausalLM
  • Input: Text
  • Output: Text
• Source Model: Instella-MoE-16B-A3B-SFT
• Supported Hardware: AMD EPYC (CPU inference)
• Preferred Operating System: Linux
• Inference Engine: vLLM v0.28.0
• Quantization Framework: LLM Compressor v0.13.0
• Quantization Method: 4-bit Weight-Only Quantization (W4A16)
• Compatible Stack:
  • ZenDNN v6.1.0
  • ZenTorch v2.13.0.0
  • PyTorch v2.13.0.0
  • LLM Compressor v0.13.0
  • vLLM v0.28.0
• Published with: LLM Compressor v0.13.0

This is a quantized version of Instella-MoE-16B-A3B-SFT created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.

Quantization

The model was quantized from Instella-MoE-16B-A3B-SFT using LLM Compressor with the GPTQ algorithm. This reduces the model weights from 29.5 GiB to 28.9 GiB on disk (~2% reduction).

• Method: 4-bit Weight-Only Quantization (W4A16)
• Config: compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=32, actorder=static
• Weights: INT4, symmetric, group-wise, stored as pack-quantized
• Activations: BF16 (unquantized)
• Group Size: 32, not the preset's 128. Every Instella Linear input dimension (2048, 512, 544, 1408, 10944, ...) divides by 32, which gives vLLM's CPU WNA16 kernel friendlier group counts than 64 or 128 would.
• Calibration: 128 examples from HuggingFaceH4/ultrachat_200k at a max sequence length of 2048

Instella is built on DeepSeek-V3 with FarSkip-Collective MoE and MLA gated attention, across 27 layers: layer 0 is a dense MLP and layers 1-26 are MoE with 64 routed experts (top-6) plus 2 shared experts.

• Quantized: the dense mlp.{gate,up,down}_proj in layer 0, and mlp.shared_experts.{gate,up,down}_proj in the 26 MoE layers. The shared expert is fused as a plain MLP in vLLM rather than going through the MoE runner, so W4A16 linears are safe there.
• Kept in BF16: the 64 routed experts per layer (mlp.experts.*), the MLA attention block (self_attn.*), the router (mlp.gate), lm_head, embed_tokens, and the layer norms.

The ~2% footprint reduction is expected, not a failure: the routed experts hold nearly all the weight mass and they stay BF16 because vLLM's CPU path has no INT4 MoE backend. MLA attention stays BF16 too, because a quantized q_proj trips a cpu_gemm_wna16 alignment failure under the vLLM Transformers fallback. The benefit here is in the quantized dense and shared-expert paths, not in capacity.

Note the two distinct names in a MoE layer: the router is mlp.gate, while mlp.gate_proj is a SwiGLU FFN weight. The mlp\.gate$ anchor matches only the router, and mlp\.experts\. with its trailing dot skips the routed experts while leaving mlp.shared_experts quantizable.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

from compressed_tensors.quantization import (
    QuantizationArgs,
    QuantizationScheme,
    QuantizationStrategy,
    QuantizationType,
)
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization.gptq import GPTQModifier

model_id = "amd/Instella-MoE-16B-A3B-SFT"
output_dir = "./Instella-MoE-16B-A3B-SFT-w4a16-llmcompressor"
CALIB_SIZE = 128
MAX_SEQ_LENGTH = 2048

# Step 1: Load the BF16 model and tokenizer. Instella ships custom modeling
# code, so trust_remote_code is required.
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cpu",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Step 2: Build the GPTQ calibration set.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")
ds = ds.map(
    lambda ex: {"text": "\n".join(m["content"] for m in ex["messages"] if m.get("content"))},
    remove_columns=ds.column_names,
)
if not getattr(tokenizer, "pad_token", None):
    tokenizer.pad_token = tokenizer.eos_token
calib_ds = ds.map(
    lambda ex: tokenizer(
        ex["text"], truncation=True, max_length=MAX_SEQ_LENGTH, add_special_tokens=False
    ),
    remove_columns=["text"],
)

# Step 3: Define the W4A16 recipe with an explicit group_size=32 instead of the
# preset's 128, for vLLM CPU WNA16 alignment.
w4a16_g32 = QuantizationScheme(
    targets=["Linear"],
    weights=QuantizationArgs(
        num_bits=4,
        type=QuantizationType.INT,
        symmetric=True,
        strategy=QuantizationStrategy.GROUP,
        group_size=32,
    ),
)
recipe = GPTQModifier(
    config_groups={"group_0": w4a16_g32},
    ignore=[
        "lm_head",
        r"re:.*lm_head",
        r"re:.*mlp\.gate$",       # router only, not mlp.gate_proj
        r"re:.*mlp\.experts\.",   # routed experts; shared_experts still quantized
        r"re:.*self_attn\.",      # MLA stays BF16 for vLLM CPU
    ],
)

# Step 4: One-shot quantize with calibration data and save in
# compressed-tensors format.
oneshot(
    model=model,
    dataset=calib_ds,
    recipe=recipe,
    max_seq_length=MAX_SEQ_LENGTH,
    tokenizer=tokenizer,
    output_dir=output_dir,
    trust_remote_code_model=True,
)

# Smoke test
inputs = tokenizer("What are we having for dinner?", return_tensors="pt")
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quick Start

Use with vLLM

from vllm import LLM, SamplingParams

model = LLM(
    model="amd/Instella-MoE-16B-A3B-SFT-w4a16-llmcompressor",
    dtype="bfloat16",
    trust_remote_code=True,
    enforce_eager=True,
)

sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)

Requirements

torch==2.13.0.0
zentorch==2.13.0.0
vllm==0.28.0
llmcompressor==0.13.0

OpenMP Setup

For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):

# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)

# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)

Note: Set LD_PRELOAD before launching vLLM or any inference script.

Evaluation

The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.

Benchmark	BF16 Baseline	W4A16 (this model)	Recovery
GSM8K (5-shot)	0.8400	0.8044	95.76%

Evaluation Command

lm_eval \
    --model vllm \
    --model_args pretrained=amd/Instella-MoE-16B-A3B-SFT-w4a16-llmcompressor,dtype=bfloat16,enforce_eager=True \
    --tasks gsm8k \
    --batch_size auto \
    --trust_remote_code \
    --num_fewshot 5 \
    --apply_chat_template \
    --log_samples \
    --gen_kwargs "max_gen_toks=2048" \
    --output_path .

Limitations

• Version Lock: This model is compatible with ZenDNN v6.1.0 / ZenTorch v2.13.0.0 / PyTorch v2.13.0.0. It may not load correctly on other versions.
• CPU Only: This model is optimized for AMD EPYC CPU inference via ZenDNN. It is not intended for GPU inference.
• Eager Mode Required: Run with enforce_eager=True. Instella uses custom remote modeling code, and the compiled graph path is not supported for this checkpoint.
• Minimal Footprint Saving: The routed experts and the MLA attention block stay in BF16, so the on-disk size drops only ~2%. Expect faster dense and shared-expert matmuls, not a smaller memory budget.
• Accuracy Trade-off: 4-bit weight-only quantization costs about 3.6 points of GSM8K accuracy here (95.76% recovery).

License

This model is distributed under the same license as the source model. See the LICENSE file for details.

Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.