Qwen3-Coder-30B-A3B-Instruct-AWQ

AWQ W4A16 (4-bit weights, group size 128) quantization of Qwen/Qwen3-Coder-30B-A3B-Instruct, produced with llm-compressor for serving on a single 24 GB GPU with vLLM.

~15.5 GB — fits an RTX 4090 with room for a large KV cache.

Quantization

Method	AWQ, W4A16, group size 128
Format	compressed-tensors (pack-quantized)
Ignored	lm_head, MoE router (mlp.gate)
Calibration	64 samples of HuggingFaceH4/ultrachat_200k, seq len 1024
MoE	moe_calibrate_all_experts=True — every calibration token is routed through all 128 experts, so each expert is calibrated

Qwen3-Coder-30B-A3B is a Mixture-of-Experts model (128 experts, 8 active per token) using standard GQA attention — so it serves on consumer Ada GPUs with the stock attention backends, no special kernels required.

Serving with vLLM

vllm serve dark-side-of-the-code/Qwen3-Coder-30B-A3B-Instruct-AWQ \
    --max-model-len 65536 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.95 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --override-generation-config '{"temperature":0.7,"top_p":0.8,"top_k":20}'

Notes:

• At --max-model-len 65536 on a 24 GB card the KV pool holds 118K tokens (1.8x concurrency). The base model's native context is 262K; 64K is the practical cap at 24 GB.
• Tool calling works via vLLM's first-class qwen3_coder parser.
• Non-thinking model — no reasoning parser needed.
• Pass --override-generation-config if your client doesn't send sampling params, or vLLM will default to temperature=1.0, which is far too hot for coding.

Validation

Verified after quantization: correct code generation, working tool_calls through the qwen3_coder parser, and coherent prose output.

Selected as the local agentic coding model in a head-to-head evaluation driven by the Pi coding agent, where it reliably executed a read/edit/test tool loop.