Qwen3.5-4B-Heretic FP8
FP8 build of darrellbest/Qwen3.5-4B-Heretic for vLLM. 6.79 GB instead of 9.35 GB. Qwen/Qwen3.5-4B with its refusal behaviour removed by Heretic using full-weight Arbitrary-Rank Ablation: 6/100 refusals (original: 99/100) at KL divergence 0.0220. See the main repository for how it was made and measured.
What is quantized
| Part | Precision |
|---|---|
| MLP linear layers, and the attention projections of the full-attention layers | FP8 E4M3 weights with per-channel scales, dynamic per-token FP8 activations (W8A8) |
| Everything else | unchanged |
Unchanged: the vision encoder, the Gated DeltaNet (linear_attn) layers, the multi-token-prediction block, the embeddings (tied to lm_head) and norms stay bf16 (the DeltaNet A_log/norm parameters float32, as in the original); the recurrent DeltaNet state is sensitive to low precision, and the 248k-token embedding table is a large share of a model this size. Made with llm-compressor 0.13.0
(scheme="FP8_DYNAMIC"). The multi-token-prediction
weights, which the quantized save drops, were copied back unchanged into model-auxiliary.safetensors.
Checked
Loaded in vLLM 0.30.0 on an RTX PRO 6000 Blackwell: ordinary prompts answered correctly and the test image (a red circle and a blue square) described correctly. Thinking-mode reasoning, 4 arithmetic and word problems x 10 seeds with Qwen's recommended sampling: 40/40 finished and correct (bf16 Heretic 40/40, original 40/40). Throughput: ~140 tok/s single-stream, ~3,000 tok/s aggregate at batch 32 (512-token generations; bf16: ~100 / ~2,500).
Use
vllm serve darrellbest/Qwen3.5-4B-Heretic-FP8
The FP8 weights were not re-measured for refusals.
Reduced safety guardrails by design. You are responsible for what you do with it.
The family
| Repository | Format | Size | Use it with |
|---|---|---|---|
| Qwen3.5-4B-Heretic | bf16 safetensors | 9.35 GB | transformers, vLLM, SGLang |
| Qwen3.5-4B-Heretic-GGUF | GGUF BF16 / Q8_0 / Q4_K_M + vision mmproj | 8.67 / 4.61 / 2.78 GB + 0.67 GB | llama.cpp, Ollama |
| Qwen3.5-4B-Heretic-FP8 | FP8 W8A8, compressed-tensors | 6.79 GB | vLLM |
| Qwen3.5-4B-Heretic-NVFP4 | NVFP4, compressed-tensors | 5.67 GB | vLLM on Blackwell |
- Downloads last month
- -