Ttimms/zaya1-8b-nvfp4-w4a4

ZAYA1-8B NVFP4 W4A4

NVFP4 W4A4 inference for Zyphra's ZAYA1-8B on consumer Blackwell (RTX 5070 Ti, SM120) — 4-bit weights and 4-bit activations running on native CUTLASS FP4 tensor-core kernels, at 102.6 tok/s single-stream / 407.4 tok/s batch-8 in under 6 GB of VRAM.

One other public NVFP4 W4A4 ZAYA1 checkpoint exists — built with NVIDIA ModelOpt and validated on a 96 GB workstation Blackwell card, with no published accuracy or throughput numbers. This checkpoint is, as far as we know, the first built with compressed-tensors on a 16 GB consumer card, with its own measured accuracy and throughput. Full build/debug writeup, quantization scripts, and reproduction steps: github.com/t-timms/zaya1-godspeed.

Status: active project. This checkpoint is a snapshot from the quantization phase, not a final release — the next phase (agentic fine-tuning via QLoRA SFT + GRPO) is in progress and may produce an updated checkpoint. Watch github.com/t-timms/zaya1-godspeed for the latest status.

Highlights

Result	Detail
102.6 tok/s single / 407.4 tok/s batch-8	vLLM + CUDA graphs on RTX 5070 Ti (12.8× over eager mode)
5.99 GB checkpoint	936 Linears in packed NVFP4 W4A4, 384 outlier-sensitive Linears kept BF16 (mixed precision)
Checkpoint verified healthy	Budget-forced GPQA-Diamond rises monotonically with reasoning budget — 45.8% → 62.5% at a 12k-token think budget, within CI of Zyphra's BF16 CoT 71.0%
vLLM SM120 source build	TORCH_CUDA_ARCH_LIST=12.0 build enabling cutlass_scaled_fp4_mm_sm120a + FP4 group MoE GEMM — kernels that ship in vLLM source but not in wheels

Quantization details

Field	Value
Base model	Zyphra/ZAYA1-8B
Method	compressed-tensors, float-quantized
Scheme	NVFP4, w4a4 (4-bit weights, 4-bit activations)
Group size	16 (tensor-group strategy)
Ignored / kept BF16	lm_head, router, norm, qkv, cca modules, plus 12 MoE layers with max_abs > 500 (384 Linears)
Calibration	977 samples, ARC-mix calibration set, 936 Linears calibrated
Compressor	NVFP4PackedCompressor (weight_packed_dtype=uint8, weight_scale_dtype=float8_e4m3fn)
Built on	RTX 5070 Ti, 16 GB VRAM, CUDA 13.0

See quantization_manifest.json in this repo for the full machine-readable config, including the exact outlier-layer list and mixed-precision threshold.

Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA (compressed convolutional attention) — no stock quantization path works out of the box. Notably: W4A4 requires calibrating activation scales (not just weights), the compressed-tensors calibration path has a silent NaN-producing trap if you calibrate through the fake-quant nn.Linear.forward wrapper, and the NVFP4 global-scale convention (2688 / max_abs, divisor form, block scales pre-multiplied) is undocumented — getting it wrong produces silent pad-token collapse, not an error. Full root-cause writeup in RESEARCH.md on the GitHub repo.

Usage

This checkpoint requires vLLM built from source with SM120 NVFP4 CUTLASS kernels (stock wheels don't include them) — see the reproduce steps in the GitHub repo. Inference must run in bfloat16 (not fp16/fp32).

# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4 --dtype bfloat16

Evaluation

Budget-forced GPQA-Diamond (n=24, paired), using an s1-style harness that caps the reasoning trace and scores only the closed answer — stock lm-eval harnesses score near-random on this model because ZAYA never closes its <think> block within a normal budget:

think budget	GPQA-Diamond	traces self-closing </think>
2,500	45.8%	1/24
5,000	45.8%	2/24
12,000	62.5%	9/24

The gap to Zyphra's BF16 CoT (71.0%) is the 16 GB context/reasoning-budget ceiling on the source hardware — not quantization damage.

License

Apache 2.0 — matches the ZAYA1-8B upstream license.