Qwen3.5-4B-MTP-Heretic

Qwen/Qwen3.5-4B with its refusal behaviour ablated using heretic v1.4.0, and with the multi-token-prediction head left intact.

Why this model exists

I wanted an abliterated Qwen3.5-4B that was still a complete vision-language model and still had its MTP head working, and that documented clearly enough to be checked. Neither property is guaranteed by the toolchain. heretic's merge export drops the MTP head silently (see below), and the published Qwen3.5-4B abliterations I found gave no clear account of how they had been done — which refusal direction, what weight range, which trial, what the utility cost — so there was no way to tell a careful ablation from a reckless one.

So this is an attempt at a variant you can actually check: the exact heretic configuration, the trial that was selected, the measured refusal rate before and after, and a note on every step that is easy to get silently wrong.

⚠️ Risk of Sensitive or Controversial Outputs — This model's safety filtering has been significantly reduced. It will comply with requests that a stock Qwen3.5-4B refuses, including harmful ones. You are responsible for what you do with it.

⚠️ No Default Safety Guarantees — Unlike stock Qwen3.5-4B, this model has not undergone safety optimization. Do not expose it to untrusted input without a policy layer of your own.

What was done

heretic searches for a low-rank "refusal direction" in the residual stream and orthogonalises the attn.o_proj and mlp.down_proj output projections against it, so the model can no longer represent a refusal. It optimises a weight vector over 200 trials against mlabonne/harmless_alpaca (good prompts, preserve utility) and mlabonne/harmful_behaviors (bad prompts, remove refusal), maximising refusal reduction subject to a KL-divergence budget against the original model.

The published checkpoint is trial 109 — the lowest-refusal point on the Pareto front:

	value
refusals (heretic, in-run)	19 / 100
KL divergence vs base	0.0459
direction_index	20.24
attn.o_proj weights	1.36 – 1.49, centred at layer 20.64
mlp.down_proj weights	1.33 – 1.40, centred at layer 21.71
trials / seed	200, seed=0

The edit is a weight orthogonalisation, not retraining. Nothing was fine-tuned, and no data was trained on.

Results

Refusal rate over the first 100 prompts of mlabonne/harmful_behaviors (temperature 0, max_tokens 100, system prompt "You are a helpful assistant."):

model	refusals
Qwen/Qwen3.5-4B (stock)	99 / 100
this model	13 / 100

Measured with llama-server and a keyword-based refusal classifier, not a proper capability benchmark — treat it as a relative signal only. A second variant of this model that suppresses the thinking block reached 10 / 100 and is published separately.

The MTP head is preserved

This is the part that distinguishes it from other published Qwen3.5-4B abliterations.

Qwen3.5-4B ships a multi-token-prediction head (mtp_num_hidden_layers = 1, 15 tensors, visible as block 32 in the GGUF). heretic's --export-strategy merge does not write those tensors — it saves the 723 tensors it loaded and the MTP head is silently gone, with no warning. Converted naively you get block_count = 32 and speculative decoding quietly stops working.

model	GGUF tensors	block_count
Qwen/Qwen3.5-4B (stock)	441	33
this model	441	33
Qwen3.5-4B-Deckard-HERETIC-UNCENSORED-Thinking	426	32

The 15 missing tensors in that last row are the MTP head. This checkpoint has them re-injected and the index corrected, so the drafter is intact.

Format

Standard transformers format, 3 safetensors shards (9.3 GB total):

config.json  generation_config.json  chat_template.jinja
model-00001-of-00003.safetensors  model-00002-of-00003.safetensors
model-00003-of-00003.safetensors  model.safetensors.index.json
preprocessor_config.json  processor_config.json  video_preprocessor_config.json
tokenizer.json  tokenizer_config.json  LICENSE

738 tensors: 297 vision, 426 language, 15 MTP.

Architecture is unchanged (Qwen3_5ForConditionalGeneration), so it is still a vision-language model and the processor configs are the stock ones. Only the o_proj / down_proj weights of the language model differ from the base.

Mixed precision, deliberately. 723 tensors are float16 (heretic was run with --dtypes float16, which is what was stable on ROCm for this model). The 15 re-injected MTP tensors are still bfloat16, copied verbatim from the base checkpoint. config.json declares dtype: float16, so from_pretrained casts them on load and this is transparent — but if you load the shards by hand, expect the two dtypes.

GGUF

A Q8_0 GGUF plus the f16 vision projector are published in the companion repository RBergBauer/Qwen3.5-4B-MTP-Heretic-GGUF for LM Studio / llama.cpp. general.name is set to Qwen3.5-4B-MTP-Heretic.

# with the vision projector (required for image input)
llama-server -m Qwen3.5-4B-MTP-Heretic-Q8_0.gguf --mmproj mmproj-F16.gguf -ngl 99

The chat template is the stock Qwen3.5 one and is still parameterised on enable_thinking, so the thinking block behaves as upstream.

Intended use

Research on refusal directions and model alignment; local experimentation with reduced safety alignment; red-teaming and evaluation harness development. Not intended for deployment where untrusted users can reach the model without a filtering layer.

Reproduction

heretic 1.4.0, llama.cpp b10689, seed=0, n_trials=200, fp16. The one step that is easy to miss: the MTP tensors must be re-injected from the base model after every heretic save, or they are silently absent from the export.

License

Apache 2.0, inherited from Qwen/Qwen3.5-4B. See LICENSE.