Qwythos-27B-v1-NVFP4

NVFP4 quantisation of empero-ai/Qwythos-27B-v1 built with NVIDIA TensorRT Model Optimizer (ModelOpt). ~55.6 GB bf16 → 19.8 GB NVFP4, quantised and load-verified on an NVIDIA DGX Spark (GB10, sm_121a).

All credit for the model belongs to Empero — Qwythos-27B-v1 is their open-weight, full-parameter reasoning model, released un-ablated. Read their card.

This build has NO speculative-decoding head. Read this before choosing it.

ModelOpt's export dropped all 15 mtp.* tensors, so multi-token prediction is not available in this repo and --speculative-config '{"method":"mtp",...}' will not work.

If you want speculative decoding, take PassingByPixels/Qwythos-27B-v1-NVFP4-MTP instead — same quantisation, with all 15 MTP tensors restored, measured at 0.905 acceptance on short context and 28.1 tok/s against this build's 13.6. It is 849 MB larger and materially faster.

This repo exists for people who will never enable speculation and want the smaller download. For most people the MTP repo is the better choice.

There is also CocaKova/Qwythos-27B-v1-NVFP4, an earlier llm-compressor (compressed-tensors) build which does include a working MTP head. Different toolchain, different kernel path in vLLM.

What is quantised, and what is not

Component	Precision
Text backbone Linear layers (64 layers, hybrid Gated-DeltaNet + full attention)	NVFP4 (FP4 weights + activations, group size 16)
lm_head	bf16
Vision tower (all 333 model.visual.* tensors)	bf16
Gated-DeltaNet conv1d layers	bf16
KV cache	FP8 (e4m3) scales exported

1699 quantizers inserted; none landed on the vision tower, lm_head or the conv1d layers. Calibrated on 512 samples at 2048 tokens (cnn_dailymail).

Measured

Load-verified on a DGX Spark (GB10, 128 GB unified) under vLLM 0.25.2: loads clean, no restarts, and generates coherent text — 13.6 tok/s single-stream without speculation, token count from usage.completion_tokens.

That figure is a smoke test, not a benchmark: one short single-stream generation, no concurrency sweep, no quality evaluation against the bf16 source. For the fuller measurement set (three context points, acceptance figures) see the MTP repo's card.

Context length

Ships the native 262,144-token configuration (max_position_embeddings 262144, no rope_scaling). Upstream declares 1,048,576 via static YaRN over an original 262144; static YaRN applies at every length and slightly degrades short-context quality, and 262,144 is comfortably servable on this class of box (a 2,138,983-token KV cache = 8.1 concurrent requests at 262k, against 2.0 at 1M).

Get the full 1M window at serve time without editing anything:

--hf-overrides '{"rope_scaling":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":262144},"max_position_embeddings":1048576}'

Usage (vLLM)

vllm serve PassingByPixels/Qwythos-27B-v1-NVFP4 \
    --trust-remote-code \
    --max-model-len 131072 \
    --kv-cache-dtype fp8 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3

Sampling, from the upstream generation_config.json: temperature 0.6, top_p 0.95, top_k 20, repetition_penalty 1.05.

Limitations

• 4-bit quantisation is lossy. No quality evaluation against the bf16 source was run here.
• No MTP head — see the box above.
• Vision was frozen upstream and the tower is preserved in bf16, but image behaviour is untested post-quantisation.
• The VL processor configs and chat template are not emitted by ModelOpt and were copied verbatim from Empero's own checkpoint, so they match the source exactly.
• Qwythos is intentionally uncensored for technical and research use — see the original card. Apply your own application-level controls.
• This is the pre-RL v1 checkpoint.

Credits

• Empero — the model.
• Qwen team — the base.
• CocaKova — the first public NVFP4 of this model.
• NVIDIA — TensorRT Model Optimizer.

Quantised and load-verified by PassingByPixels on a DGX Spark. Apache-2.0, same as the original.