Tech2wild/Nanbeige4.3-33B-W4A16-AutoRound

Nanbeige4.3-33B W4A16 (AutoRound int4)

INT4 weight-only quant of Nanbeige/Nanbeige4.3-33B for vLLM on Ampere and newer GPUs. 20.8 GB (from 66.2 GB BF16), GPTQ format, loads with vLLM's GPTQ Marlin kernels. Fits two 24 GB cards with a 213K-token fp8 KV pool.

Recipe, launcher, benchmark script and raw results: https://github.com/tonyd2wild/Nanbeige4.3-33B-W4A16-2x-RTX-3090

Quant

• Intel AutoRound 0.16.1, scheme W4A16: INT4 symmetric, group size 128, all 364 linear layers in the 52 decoder blocks.
• lm_head, embeddings and norms kept in FP16.
• 100 tuning iterations per block, 128 calibration samples x 2048 tokens (NeelNanda/pile-10k).
• Exported with --format auto_gptq (quant_method: gptq, desc_act: false). 87 minutes on 2x RTX 3090.

Measured on 2x RTX 3090 (NVLink), TP2, vLLM 0.27.1

Single stream decode	61 to 66 tok/s
6 concurrent streams	327 to 354 tok/s total, 58 to 61 tok/s each
TTFT	0.05 to 0.08 s (C1), 0.26 to 0.41 s (C6)
Prefill	~1,770 tok/s on a 17K-token prompt
KV pool	213,264 tokens (fp8 e5m2, 131K context, gmu 0.92)
69-scenario tool eval, thinking on	134/138, Quality 97.1, Deployability 93.3
69-scenario tool eval, thinking off	131/138, Quality 94.9, Deployability 95.3

Serve

vllm serve Tech2wild/Nanbeige4.3-33B-W4A16-AutoRound \
  --served-model-name nanbeige4.3-33b --tensor-parallel-size 2 \
  --quantization gptq_marlin --dtype float16 --max-model-len 131072 \
  --gpu-memory-utilization 0.92 --kv-cache-dtype fp8_e5m2 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

Thinking is on by default; send "chat_template_kwargs": {"enable_thinking": false} to turn it off per request.

Credits

The Nanbeige team for Nanbeige4.3-33B (Apache 2.0), Intel for AutoRound, the vLLM team for vLLM and the Marlin kernels. Quantized by Tony DeAngelo (Tech2Wild / 2Wild) with Kai. Same license as the base model: Apache 2.0.