Qwen3.8-9B-Distill (W4A16 Quantized via AutoRound)

This repository contains a W4A16 (4-bit weights, 16-bit activations) quantized version of empero-ai/Qwen3.8-9B-Distill, quantized using Intel's AutoRound algorithm.

⚡ Quantization Details

Calibrated and quantized with fine-grained group sizes and high iteration depth to preserve reasoning traces (<think> blocks) and multimodal capabilities:

Algorithm: Intel AutoRound
Precision / Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 32 (fine-grained reconstruction fidelity)
Symmetric (sym): True
Calibration Samples (nsamples): 512
Sequence Length (seqlen): 4096
Tuning Iterations (iters): 1000 (Production-grade accuracy)
Vision Tower (quant_nontext_module): False (Kept in BF16 to preserve visual reasoning and OCR precision)
Special Modules (layer_config): Multi-Token Prediction (mtp, mtp.fc) kept in native bfloat16

📦 Available Formats

Depending on your inference engine, choose the appropriate repository:

AutoRound Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound
AutoGPTQ Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-GPTQ
LLM-Compressor / Compressed-Tensors Format: Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound-LLM-Compressor

🚀 Usage & Quickstart

1. High-Throughput Serving via vLLM

# Using the LLM-Compressor / Compressed-Tensors build
vllm serve Vishva007/Qwen3.8-9B-Distill-W4A16-AutoRound \
    --dtype bfloat16 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90

📊 VRAM & Performance Benefits

Original Model (BF16): ~8–10 GB VRAM required for full context inference
Quantized Model (W4A16 Group 32): ~2.5–3.5 GB VRAM (runs comfortably on 4GB/6GB consumer GPUs, laptops, and edge devices)
Throughput: Lowers memory bandwidth pressure, accelerating token generation speeds during extended chain-of-thought (<think>) reasoning.