Instructions to use Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP")
model = AutoModelForMultimodalLM.from_pretrained("Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP

SGLang

How to use Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP with Docker Model Runner:
```
docker model run hf.co/Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP
```

Qwen3.5-9B-heretic-v2 — NVFP4 + MTP

NVFP4 (4-bit floating-point, group_size=16) quantization of trohrbaugh/Qwen3.5-9B-heretic-v2, a Heretic-decensored Qwen/Qwen3.5-9B, with the native MTP (multi-token prediction) head restored so vLLM can run speculative decoding out of the box.

heretic-v2 is also the base of several popular finetunes (e.g. DavidAU's Qwen3.5-9B-Claude-4.6-HighIQ-HERETIC series), but had no NVFP4 release. Built and tested on an NVIDIA DGX Spark (GB10).

Speculative decoding is optional: without --speculative-config the model serves as plain NVFP4 and the MTP tensors are simply not loaded.

Quick facts


Base model	trohrbaugh/Qwen3.5-9B-heretic-v2 (Heretic v1.2.0 of Qwen/Qwen3.5-9B; KL 0.0793, refusals 6/100 per its card)
Architecture	`Qwen3_5ForConditionalGeneration` — 32 layers, hybrid 3× Gated DeltaNet + 1× Gated Attention per block, hidden 4096, vocab 248320, vision encoder, 262K context
Original size	~19 GB (BF16)
Quantized size	~11.1 GB (10.6 GB weights + 0.49 GB MTP head)
Quant format	NVFP4 via nvidia-modelopt 0.43.0, `group_size=16`, stored as modelopt checkpoint (`quant_method: modelopt`)
Quantized modules	152 Linear layers: `mlp.{gate,up,down}_proj` (32 layers), `self_attn.{q,k,v,o}_proj` (8 full-attention layers), `linear_attn.out_proj` (24 DeltaNet layers)
Kept BF16	`lm_head`, `embed_tokens`, entire vision tower, all norms, DeltaNet `conv1d` / `A_log` / `dt_bias` / *all `in_proj_`**, MTP head
MTP head	1 layer, 15 tensors, BF16, from `Qwen/Qwen3.5-9B@c2022362`
Calibration	none (weight-only quantization, `forward_loop=None`; dynamic input scales)
Conversion date	2026-09-16

Why the MTP head had to be restored

Qwen3.5-9B ships with a one-layer MTP head (mtp.*, 15 tensors) that vLLM uses as a built-in draft model. trohrbaugh/Qwen3.5-9B-heretic-v2 still declares mtp_num_hidden_layers: 1 in config.json, but its safetensors index contains zero mtp.* keys: the head was dropped when the abliterated model was saved through transformers, which does not load or save MTP weights.

Without the tensors, vLLM either refuses the speculative config or drafts with an uninitialized head (≈0 % acceptance). This repo re-attaches the original head:

the 15 mtp.* tensors are copied byte-exact from Qwen/Qwen3.5-9B (revision c202236235762e1c871ad0ccb60c8ee5ba337b9a) into model-mtp.safetensors
they are wired into model.safetensors.index.json
mtp* is added to quantization_config.ignore (and hf_quant_config.json exclude_modules), otherwise vLLM's modelopt loader would build NVFP4 linears for the BF16 MTP layer

Heretic only edits attn.o_proj and mlp.down_proj in the main decoder via directional ablation. The embeddings, lm_head and hidden-state geometry the MTP head depends on stay close to the original, and the measured acceptance rates below confirm the head still drafts well.

Why the DeltaNet `in_proj_*` stay BF16

vLLM's Qwen3.5 implementation fuses in_proj_qkv + in_proj_z into one in_proj_qkvz module (and in_proj_a + in_proj_b likewise). A fused module must have uniform precision — quantizing only part of the group makes vLLM reject the checkpoint ("some but not all shards … quantized"). The whole in_proj_* group is therefore kept BF16; only the standalone linear_attn.out_proj is quantized in DeltaNet layers.

Post-export fixes

modelopt 0.43's HF export needs three adjustments for vLLM, all applied to this checkpoint:

input_scale = 1.0 injected for every quantized linear (152×; modelopt omits them for weight-only export, vLLM otherwise registers an uninitialized parameter)
config_groups.*.input_activations.dynamic = true
quantization_config.ignore completed (lm_head, model.visual, model.language_model.embed_tokens, DeltaNet in_proj_* / conv1d, mtp*)

Benchmarks (DGX Spark, single GB10)

vLLM 0.26.0, FlashInfer attention, FP8 KV cache. Greedy decoding, thinking disabled, 4 prompts (code, German prose, English prose, summary) × 2 rounds, 600 max tokens. seq = one request at a time, end-to-end incl. prefill. par4 = mean per-request speed with 4 concurrent requests.

Config	seq tok/s	speedup	acceptance rate	mean accept length	par4 tok/s per request
no MTP (same weights, no `--speculative-config`)	23.6	1.00×	–	–	25.9
MTP, k=1	32.5	1.38×	71.8 %	1.72	31.8
MTP, k=2	32.8	1.39×	60.1 %	2.20	30.1
MTP, k=3	30.5	1.29×	49.9 %	2.50	29.7

Recommendation: num_speculative_tokens: 1. k=1 and k=2 are within noise single-stream, k=1 is fastest per request under concurrency and has the smallest KV overhead. With the MTP head reused for k>1 (only one MTP layer exists), acceptance drops with every additional draft token.

With thinking enabled and sampling (temp 0.6, top_p 0.95, top_k 20), acceptance at k=1 measured 74–88 % in two short probes.

Accuracy benchmarks (MMLU-Pro and others, NVFP4 vs. the BF16 source on the same harness) are running and will be added here.

Verification

On a single DGX Spark (GB10), vLLM 0.26.0:

loads with Detected MTP model. Sharing target model embedding weights / lm_head weights with the draft model. and no missing-parameter warnings
model memory 10.47 GiB (10.02 GiB without MTP)
coherent English and German output, code generation, correct arithmetic in thinking mode
tool calling (--tool-call-parser qwen3_xml) returns well-formed tool_calls; reasoning is split correctly by --reasoning-parser qwen3
at --gpu-memory-utilization 0.38 (45.9 GiB of the 128 GB UMA pool): 30.8 GiB KV cache = 1,727,305 tokens, i.e. 6.6× full 262K context. Only 8 of 32 layers (plus MTP) carry a KV cache.

Vision was not re-tested after quantization. The vision tower is untouched BF16.

Usage

vLLM (serve)

vllm serve Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --attention-backend flashinfer \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-seqs 6 \
  --gpu-memory-utilization 0.38 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching

No --quantization flag needed; vLLM detects the modelopt NVFP4 config.
On vLLM ≥ 0.26, "method":"qwen3_5_mtp" still works but is deprecated in favour of "mtp".
With prefix caching, vLLM sets mamba_cache_mode=align for this architecture automatically. all is not supported together with MTP.
--gpu-memory-utilization is a fraction of total memory. 0.38 fits 6 × 262K on a 128 GB Spark; scale it to your GPU.
To run without speculative decoding, just drop --speculative-config; the MTP tensors are then ignored.

Sampling

Qwen's recommendations for Qwen3.5-9B apply (all with top_k=20, min_p=0, repetition_penalty=1.0):

Mode	temperature	top_p	presence_penalty
Thinking, general	1.0	0.95	1.5
Thinking, precise coding	0.6	0.95	0.0
Non-thinking, general	0.7	0.8	1.5
Non-thinking, reasoning	1.0	0.95	1.5

Toggle thinking per request with chat_template_kwargs: {"enable_thinking": false}.

min_p and logit_bias have no effect while speculative decoding is active (vLLM limitation). Use top_k / top_p instead, or serve without --speculative-config if you rely on min_p.

Files

model-00001-of-00002.safetensors, model-00002-of-00002.safetensors — NVFP4 weights + scales, BF16 vision tower / embeddings / norms
model-mtp.safetensors — MTP head, 15 BF16 tensors (0.49 GB), unmodified from Qwen/Qwen3.5-9B
model.safetensors.index.json — 1231 keys incl. mtp.*
config.json — upstream heretic-v2 config + quantization_config (modelopt NVFP4, ignore list incl. mtp*)
hf_quant_config.json, generation_config.json
tokenizer.json, tokenizer_config.json, vocab.json, chat_template.jinja, preprocessor_config.json, video_preprocessor_config.json — unchanged from heretic-v2

Content notice

The base model is decensored (abliterated): it answers requests the original Qwen3.5-9B would refuse. Quantization does not add any safety layer. You are responsible for how you deploy and use it.

Acknowledgments

trohrbaugh for Qwen3.5-9B-heretic-v2
p-e-w for Heretic
Qwen team for Qwen3.5-9B and its MTP head
NVIDIA for DGX Spark / GB10, NVFP4 and modelopt
vLLM project for modelopt NVFP4 and Qwen3.5 MTP support

License

Apache 2.0, inherited from Qwen/Qwen3.5-9B via trohrbaugh/Qwen3.5-9B-heretic-v2.

Downloads last month: 391

Safetensors

Model size

7B params

Tensor type

BF16

Model tree for Kaleto/Qwen3.5-9B-heretic-v2-NVFP4-MTP

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

trohrbaugh/Qwen3.5-9B-heretic-v2

Quantized

(12)

this model