CaptainArni/Swift-Qwen3.8-27B-NInfer

Swift-Qwen3.8-27B — NInfer v3 artifact

A single-file NInfer v3 artifact of ukisai/Swift-Qwen3.8-27B-NVFP4, UkisAI's reasoning-efficient fine-tune of Qwen3.8-27B in NVIDIA ModelOpt's NVFP4/FP8 quantization. Converted with upstream's own converter and the recipe included here; weight bytes are imported, not requantized.

file	qwen3_8_27b_nvfp4swift.ninfer
size	22,783,241,220 bytes (21.2 GiB)
sha256	5412a0e7ad7a670bb653a8363785257fe970b6930ffe9f0213f78b696299cf7f
components	Text, Vision, MTP, DFlash2, indexed proposal head
weights in VRAM	18.9 GiB (MTP) / 20.5 GiB (DFlash2)
model id served	qwen3.8-27b
built	2026-09-16, NInfer 6cc95cc5

What is inside: all 64 MLP layers and nothing else at NVFP4 (block size 16); the 144 GDN and 64 full-attention projections at FP8 E4M3 with one BF16 scale per row; an FP8 output head and FP8 embeddings; Q4/Q5/Q6 Vision; Q8 MTP; z-lab's DFlash2 draft at W8G32. Norms, convolutions, the small GDN projections and the draft codebooks stay BF16.

Quick start

Build NInfer (Linux + NVIDIA, CUDA 12.8+; tested on an RTX 5090) and serve the file directly — a v3 artifact carries its own config, tokenizer, chat template and bindings, so there is nothing else to download:

git clone https://github.com/Neroued/ninfer && cd ninfer && ./build.sh
hf download CaptainArni/Swift-Qwen3.8-27B-NInfer \
  qwen3_8_27b_nvfp4swift.ninfer --local-dir ~/ninfer-models

./build/apps/ninfer-serve ~/ninfer-models/qwen3_8_27b_nvfp4swift.ninfer \
  --host 127.0.0.1 --port 8080 \
  --max-context 64000 --kv-capacity 64000 --kv-dtype int8 \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 --lm-head-draft

It serves an OpenAI-compatible API:

curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.8-27b","messages":[{"role":"user","content":"What is 17*23?"}]}'

Speculative decoding — pick one at startup. --spec mtp --draft-tokens 3 uses Swift's own MTP head. --spec dflash2 --draft-tokens 7 uses the DFlash2 draft, which is faster on code and structured output and costs ~1.6 GiB more weights plus ~0.9 GiB of runtime. Only the selected backend is materialized.

Context and VRAM, measured on a 32 GiB card with Vision off:

context	KV	spec	weights	runtime	free
64,000	int8	MTP	18.9 GiB	3.05 GiB	8.44 GiB
64,000	int8	DFlash2	20.5 GiB	3.93 GiB	6.03 GiB
224,000	int8	MTP	18.9 GiB	8.40 GiB	2.47 GiB

Add --vision for images and video (~0.25 GiB of weights plus a media arena). The architecture ceiling is 262,144 tokens; --kv-dtype nvfp4 roughly halves the pool. The model id is qwen3.8-27b because the artifact was named to drop into a setup already serving that id — the weights are Swift's.

Measured against the official NVFP4 artifact

Against neroued/Qwen3.8-27B-nvfp4-NInfer (base Qwen3.8 in unsloth's NVFP4/FP8), RTX 5090, 224,000 context, Vision off, KV int8, MTP, one request at a time, both runs back to back. This compares two artifacts: the fine-tune and a different quantizer at once. Tokens and quality belong to the fine-tune; the speed mostly belongs to this file quantizing all 64 MLP layers where the official one leaves the last eight FP8.

Tokens — 14 tasks × 4 seeds, 32,768-token cap, temp 1.0 / top_p 0.95 / top_k 20, thinking at the template default (xhigh):

	official NVFP4	this artifact
output tokens, 56 runs	208,309	108,628 (−48%)
of which thinking	196,697	99,028 (−50%)
mean / median per run	3,720 / 1,500	1,940 / 940
graded correct	41/44	43/44
runs truncated at the cap	0	0

Fewer tokens on 14 of 14 tasks (one-sided sign test p = 0.00006), per-task ratio 0.57 geometric mean. UkisAI claim 41% fewer thinking tokens at xhigh; this measured 50% on a smaller, easier set. At tighter caps (4k/6k) the same prompts finished 17/18 against 8/18.

Speed and footprint:

	official NVFP4	this artifact
prefill, 10.7k / 43.8k / 172k tokens	7,380 / 5,390 / 2,680 tok/s	8,650 / 6,410 / 3,000 tok/s
decode, model steps/s (acceptance divided out)	57.5	60.6
weights / free VRAM at 224k	19.7 / 1.24 GiB	18.9 / 2.47 GiB

What this does not show. The quality suite is a smoke test — six graded task types, two to four seeds — able to catch damage, not to resolve a point on a public eval. Decode figures come from one greedy pass per topic. In a 12-run multi-turn agent loop this artifact failed to end the turn twice under one agent where the official artifact did not, and passed 12/12 under another; at that sample size the difference is not significant (Fisher p = 0.22), but it is the open question on this file. The DFlash2 draft was trained against base Qwen3.8's hidden states, not Swift's, and still accepted 78% on a code prompt.

How it was built

swift_nvfp4.py in this repo is the recipe. Upstream reads compressed-tensors; this checkpoint is ModelOpt, which stores the same NVFP4 bits under other names and inverts two scales, so the recipe adds a reader for it. Conversion took 4 min 48 s on CPU and imports every NVFP4 and FP8 code word unchanged.

python3 -m tools.convert \
  --model /path/to/Swift-Qwen3.8-27B-NVFP4 \
  --recipe swift_nvfp4.py \
  --source dflash2=/path/to/Qwen3.8-27B-DFlash2 \
  --components text,vision,mtp,dflash2 \
  --resource chat_template.jinja=tools/chat_templates/qwen3_8.jinja \
  --name qwen3.8-27b --device cpu \
  --out qwen3_8_27b_nvfp4swift.ninfer

Two things differ from upstream's qwen3_8_27b_nvfp4 recipe, both forced: every MLP layer is NVFP4 (upstream stops at 56 because unsloth's checkpoint leaves the rest FP8), and the output head is re-encoded from NVFP4 to FP8 because linear_topk takes a Q8 or an FP8 head only. ModelOpt's per-tensor FP8 scale is carried as a BF16 per-row multiplier: measured against Swift's own matrices that is 0.06–0.30% relative error, where re-encoding each row from decoded values costs 2.4–2.8%.

Sources: Swift NVFP4 at revision 4cf10191, z-lab/Qwen3.8-27B-DFlash2 at 50307d4c, NInfer at 6cc95cc5.

License and credits

The weights are UkisAI's and this artifact inherits the Swift Open License v1.0 from them: free for personal, research, educational, evaluation and commercial use for individuals and organizations with annual recurring revenue up to US$1,000,000; above that, commercial use needs a Swift Enterprise License. See the original model for the terms and for access.

Thanks to UkisAI for Swift, to Neroued for NInfer and its converter, and to z-lab for the DFlash2 drafter.