Qwen3.5-4B-FP8

Block-wise FP8 (e4m3) quantization of Qwen/Qwen3.5-4B in the exact format of Qwen's own official *-FP8 releases (quant_method: fp8, weight_block_size [128,128], dynamic activations, fp32 weight_scale_inv scales).

Qwen published official FP8 weights for Qwen3.5-27B/35B-A3B/122B/397B — but not for the 4B, and it kept being asked for (#13, #24, #27). So here it is.

• Size: 5.65 GB (from ~8.8 GB BF16)
• No calibration — this is a deterministic weight transform. The conversion script is included in this repo (quantize_fp8_block.py), as requested in #24.

Run it

Serve (tested — vLLM 0.19.0, RTX 5070 12 GB; vLLM downloads the weights automatically):

vllm serve Hob-forge/Qwen3.5-4B-FP8 --max-model-len 4096 --gpu-memory-utilization 0.75

Or in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="Hob-forge/Qwen3.5-4B-FP8", max_model_len=4096, gpu_memory_utilization=0.75)
out = llm.chat(
    [{"role": "user", "content": "Hello!"}],
    SamplingParams(temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5, max_tokens=512),
)
print(out[0].outputs[0].text)

To just fetch the weights:

hf download Hob-forge/Qwen3.5-4B-FP8

Hardware: FP8 GEMM needs a recent NVIDIA GPU (Ada / Hopper / Blackwell, sm89+); ~6 GB VRAM for weights plus KV cache — a 12 GB card runs it comfortably at 4K context. On an 8 GB card, CPU, or modest RAM? FP8 is not the right artifact for you — use our GGUF edition instead: Hob-forge/Qwen3.5-4B-Instruct-GGUF.

Module treatment

Mirrors Qwen/Qwen3.5-27B-FP8 exactly:

	modules
FP8 block-128	language self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj, linear_attn.{in_proj_qkv, in_proj_z, out_proj}, MTP-block attention + MLP projections
kept BF16	embeddings (tied lm_head), linear_attn.{conv1d, in_proj_a, in_proj_b}, the entire vision tower, mtp.fc, all norms

207 weight tensors quantized, 531 kept bit-identical to upstream.

Verification

• Every quantized tensor dequantized and compared against upstream BF16: max block-dequant relative error 3.6%, mean 2.8% — the expected e4m3 envelope.
• All passthrough tensors verified bit-identical to upstream.
• Smoke-tested end-to-end with vLLM 0.19.0 on an RTX 5070 12 GB (Blackwell, WSL2): greedy completions and chat-template generation are coherent; thinking mode works.

A/B against BF16 (same box, same inputs, vLLM 0.19.0, teacher-forced):

measurement	BF16	FP8	delta
PPL, prose (Austen excerpt, 4776 tok)	15.141	15.174	+0.22%
PPL, prose (Shelley excerpt, 3741 tok)	10.173	10.256	+0.81%
PPL, code (Python, 2800 tok)	2.859	2.864	+0.19%
12-item exact-answer QA (greedy)	12/12	12/12	parity

Greedy 64-token continuations match BF16 token-for-token 61% of the time (7/20 prompts identical throughout) — normal for any quantization: near-tie top-2 tokens flip and paths diverge. Every diverged continuation was manually inspected and is coherent. This is a small probe, not a benchmark suite — for production decisions run your own evals.

Sampling parameters

Use upstream's recommendations (from the Qwen3.5-4B card):

• Thinking, general: temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5
• Thinking, precise coding: temperature=0.6, top_p=0.95, top_k=20
• Instruct, general: temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5

Tested only on Blackwell (RTX 5070) under vLLM; other stacks (SGLang, transformers fine-grained-FP8) load this format but are untested by us — reports welcome.

NVFP4

Also requested in #27 — under evaluation. NVFP4 needs a calibration pass and currently serves only on Blackwell-class hardware; if we can produce and validate one honestly, it will appear as a sibling repo.

Who we are

Hob Forge — small household spirits for small machines. We set up small models correctly for small GPUs and modest RAM.

Quantized with the included script; Apache-2.0, upstream LICENSE retained.