Qwen3-8B-W4A16-GPTQ
Qwen3-8B (pinned revision b968826) quantized with GPTQ (llm-compressor, W4A16 scheme: 4-bit weights, 16-bit activations), produced for sm120-quant-bench: a study of where 4-bit damage actually lands, measured with frozen splits, capability probes, and bootstrap 95% CIs on an RTX 5090. Write-up: "NVFP4 on a 5090 names two different machines".
Most quantized checkpoints ship with no damage documentation. This card tells you what this quantization costs, per capability, against the BF16 baseline measured on identical hardware with identical frozen evals.
| Metric | BF16 baseline | This checkpoint |
|---|---|---|
| Perplexity (wikitext-2 slice, lower better) | 17.92 | 19.66 |
| GSM8K strict-match | 91.8 [89.4, 94.2] | 88.4 [85.6, 91.2] |
| HumanEval pass@1 | 63.4 [56.1, 70.7] | 64.6 [57.3, 72.0] |
| MMLU STEM | 71.8 [68.6, 74.8] | 69.0 [65.8, 72.1] |
| MMLU humanities | 62.3 [58.9, 65.6] | 62.6 [59.2, 65.9] |
| Numeric fidelity probe | 78.7 [74.0, 83.3] | 76.7 [71.7, 81.3] |
| Long-context multi-needle (overall) | 97.8 [95.6, 99.4] | 96.7 [93.9, 98.9] |
| Tool-calling end-to-end (300 items) | 87.7 [84.0, 91.3] | 86.0 [82.0, 89.7] |
| Weights on disk (GB) | 16.4 | 6.1 |
| TTFT p50 ms (single-stream, RTX 5090) | 14.8 | 7.7 |
| ITL p50 ms/token | 11.1 | 4.7 |
Damage pools in computation, not structure. Full five-variant table with bootstrap CIs: the study.
Every number traces to a results JSON in the repo, and every eval run recorded which GPU kernel actually served it. This checkpoint's serving receipt on our stack (vLLM 0.26.0, sm_120):
Using MarlinLinearKernel for CompressedTensorsWNA16
Part of sm120-quant-bench
One of three checkpoints quantized and evaluated together in sm120-quant-bench; siblings: Qwen3-8B-W4A16-AWQ and Qwen3-8B-NVFP4.
These two checkpoints sit 0.06 perplexity apart and 6.7 points apart on a numeric-fidelity probe. That gap is the study.
Studies using these checkpoints
How it was made
recipes/w4a16_gptq.py (512 chat-templated ultrachat samples, max_seq 2048, seed 3407). Calibration data is disjoint from every eval split by construction. Full recipe, harness, and raw results: the repo. Reproduction gets byte-identical eval items (seed-pinned, hashed manifest).
Serve
vllm serve <this-repo> --max-model-len 16384
Check the log for the kernel-selection line before benchmarking. If a number matters, its serving receipt matters.
- Downloads last month
- -