Cudecnik/Qwen3.8-Flash-Next-refusal-projection

Qwen3.8-Flash-Next — refusal abliteration as a control vector (llama.cpp)

98% fewer refusals (1/50 harmful prompts vs 50/50 for the stock model) at 0.186 KL divergence, as a 480 KB file applied at load time to any GGUF of this model.

This repo contains a pre-generated refusal direction in GGUF control-vector format, plus the two llama.cpp patches needed to apply it to the qwen4exp architecture. There are no weights here: the direction is projected out of the residual stream while the model runs, so it works with any quantization of Qwen3.8-Flash-Next (measured on unsloth UD-Q2_K_XL), costs no extra disk next to the model you already have, and is switched off by removing a flag.

The operator is a projection, h -= s · (h·v) v per layer, not llama.cpp's usual additive steering h += s · v: same refusal removal, about 7× less drift on everything else.

Performance

Metric	Projection (this repo)	Additive control vector	Stock model
KL divergence (mean over neutral text, nats)	0.186	1.376	0 (by definition)
Refusals, harmful prompts	✅ 1/50	✅ 0/50	❌ 50/50
Refusals, harmless prompts	2/50	0/50	2/50
Decode speed	93% of stock	93%	100%

Scored on 50 harmful + 50 harmless held-out prompts, with thinking off (256 tokens) and again with thinking on (2048 tokens, judged on the body after </think>, with truncated answers excluded — a truncated thinking trace otherwise reads as compliance). Broken or empty output counts as not compliant. KL is measured token by token against the stock model's logits on neutral text. The single remaining harmful "refusal" is a capability caveat ("I can't verify…"), counted against us.

The two harmless "refusals" are not the same two prompts in both columns: one is shared (a request for news from the last month), the other is a knowledge-cutoff decline in the stock model and an "I have no personal experience" decline with the vector. Neither column contains a safety refusal on the harmless set.

Parameters

Parameter	Value
mode	project (h -= s·(h·v)v)
direction	per-layer (one unit vector per layer)
scale	1.0
layers	4 … 44 (of 48)
direction source	mean difference of harmless vs harmful activations over all prompt positions

How to run it

Stock llama.cpp silently ignores --control-vector* for this architecture: the qwen4exp graph never calls build_cvec, and there is no projection mode. Build llama.cpp with the two patches in this repo (or use the cvec-hooks branch of Cudila/llama.cpp):

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git apply /path/to/01-qwen4exp-cvec-hooks.patch /path/to/02-cvec-projection-mode.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

The patches are against llama.cpp master 159b74142 and are MIT-licensed, like the code they apply to.

A complete working command line for a 32 GB card with 96 GB of RAM (the memory flags are not optional — without -ncmoe and -ot this model does not fit on any consumer card; see Hardware configurations for your row):

llama-server -m Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf \
    --jinja -fa on -ngl 999 -c 262144 -np 1 -ub 1024 \
    -ncmoe 37 -ot per_layer_token_embd=CPU \
    -ctk q4_0 -ctv q4_0 \
    -lm none -lzm off \
    --control-vector-scaled Qwen3.8-Flash-Next-refusal-projection.gguf:1.0 \
    --control-vector-layer-range 4 44 \
    --cvec-mode project --cvec-dir per-layer

Of that line, only the last four flags belong to this repo — --control-vector-scaled, --control-vector-layer-range 4 44, --cvec-mode project, --cvec-dir per-layer. Add them to whatever command you already run the model with; everything else above is the stock recipe for fitting it.

Behind llama-swap, as a second entry next to the untouched stock one:

  qwen3.8-flash-next-abliterated:
    cmd: |
      /path/to/patched/llama-server --port ${PORT} --host 127.0.0.1 --jinja -fa on -ngl 999
      --model ${m}/unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-00003.gguf
      -c 262144 -np 1 -ub 1024
      -ncmoe 37 -ot per_layer_token_embd=CPU
      -ctk q4_0 -ctv q4_0 -lm none -lzm off
      --control-vector-scaled /path/to/Qwen3.8-Flash-Next-refusal-projection.gguf:1.0
      --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer
    ttl: 1800

If you would rather have an agent do this, AGENTS.md in this repo is a step-by-step setup script for one (Linux and Windows, with a check after every step): point Claude Code, Codex, Cursor or aider at it with "follow AGENTS.md from this repo step by step and set the model up on this machine".

On startup the server prints apply: control vector mode = project, dir = per-layer, layers 4..44. No such line means the flags were dropped and you are running the stock model.

Notes that cost time if you miss them:

• One request at a time (-np 1). The qwen4exp lightning indexer asserts on batched attention, with or without the vector.
• --cvec-mode add keeps llama.cpp's stock additive behaviour; for that mode use Qwen3.8-Flash-Next-refusal-additive.gguf at scale 0.05–0.15 on layers 16..32 (--cvec-mode add --control-vector-layer-range 16 32, scale ≈ 0.1) — not the 4 44 range above. The vectors are unit-normalized per layer, and scale 1.0 destroys the model in both signs.
• Remove the four flags and you are running the stock model again; nothing on disk changed.

Tested hardware

Everything measured in this card was measured on exactly one machine, with one quant:

GPU	NVIDIA RTX 5090, 32 GB (driver 610.43.03, CUDA 13.3)
CPU	Intel Core i7-14700KF, 8P + 12E, 28 threads
RAM	128 GB DDR5-5200
Storage	NVMe SSD (ADATA SX8200PNP) — matters only for the PLE-on-SSD chapter
OS	Linux (Arch, kernel 7.1)
Model	unsloth Qwen3.8-Flash-Next UD-Q2_K_XL, 73.5 GB, 3 shards
llama.cpp	master 159b74142 + the two patches in this repo; speed numbers additionally on the sm120-5090-prod fork
Context	262144, -np 1, -ncmoe 37, q4_0 KV

Windows was not tested by me at all — the Windows half of AGENTS.md is written from the documented behaviour of the same tools, not from a run. Nothing here was tested on AMD, Apple, multi-GPU, or CPU-only builds, and no quant other than UD-Q2_K_XL was scored. Every row of the table below except the 32 GB one is arithmetic, not a measurement.

Hardware configurations

The vector costs 480 KB on disk and ~7% of decode speed. What decides whether this model runs at all is where its parts live. For UD-Q2_K_XL the split is:

part	size	where
per-layer n-gram embeddings (PLE)	27 GB	always RAM (-ot per_layer_token_embd=CPU)
attention, hyper-connections, norms	3.3 GB	GPU
48 expert layers	0.92 GB each	split: -ncmoe N keeps N of them in RAM
KV cache at 262144, q4_0	~1.6 GB	GPU
compute buffer, -ub 1024 at 262144	~2.9 GB	GPU

So: GPU ≈ 3.3 + 0.92·(48−N) + KV + compute, RAM ≈ 27 + 0.92·N plus a few GB of runtime. Q3_K_XL is the same arithmetic with ~1.0 GB per expert layer and 27 GB of PLE.

Two things about the numbers in the table below. They were measured on a speed fork of llama.cpp (branch sm120-5090-prod: fused lightning indexer, block top-k, gather attention, a GPU-resident LRU cache of host experts) — a stock build with the two patches from this repo gives ~33 t/s decode at a 2k prompt on the same box, not 54. And the measured VRAM peak is that fork's slot, which additionally holds ~8 GB of expert cache and the vision projector; the formula above is for a stock build and lands at ~18 GB for the 32 GB row.

VRAM	example cards	-ncmoe	KV	context	RAM for the CPU side	notes
32 GB	5090	37	q4_0	262144	62 GB	measured: ~33 t/s decode on a stock build with these patches; 54 ± 3 t/s decode and 594 ± 108 t/s prefill on the speed fork over 981 real requests (i7-14700KF, DDR5-5200), VRAM peak 30.5 GB with its expert cache
24 GB	4090, 3090	41	q4_0	262144	65 GB	~7 expert layers on the GPU; expect 10–15% less decode than the 32 GB row
16 GB	5080, 5070 Ti, 4060 Ti 16G	44	q4_0	131072	68 GB	4 expert layers on the GPU, -ub 512 to shrink the compute buffer; estimate 30–40% less decode
12 GB	5070, 3080 12G	46	q4_0	65536	69 GB	almost everything on the CPU; the GPU is doing attention and the PLE lookups
8 GB	5060, 3070	47	q4_0	32768	70 GB	works, but the CPU is the model at this point

The patches need no particular GPU generation — any CUDA card llama.cpp supports will do. (The speed fork is built for sm_120a and is therefore 50-series only.)

The "RAM for the CPU side" column is 27 + 0.92·N and it is what the process actually holds: the 32 GB row measured 61.7 GiB resident, against 68.5 GiB (73.5 GB) for the file on disk: the expert layers that go to the GPU are never held in RAM, and what is left over of the difference is runtime buffers. 64 GB of RAM fits the 32 GB-VRAM row, but only just: 62 GiB of model plus the runtime leaves no page cache and no room to grow, so treat 64 GB as the floor and 96 GB as the comfortable configuration. Every row below the first one needs 96 GB, or the PLE on an SSD (next chapter). Q3_K_XL adds ~4 GB to each row.

Only the 32 GB row was actually run (see Tested hardware); the others are derived from the sizes above and from the per-layer cost measured on that machine, so treat them as starting points — raise -ncmoe by one if you OOM during a long request rather than at load, that is the usual way this model fails.

Flags that matter more than the quant choice:

• -ot per_layer_token_embd=CPU — the 27 GB PLE table has no business on the GPU.
• -lzm off (--lazy-mode off) — since llama.cpp #27837 the PLE is mmapped even with -lm none, and every prefill token then faults 48 rows in from disk. Keeping it resident is worth 3.7× prefill here; if your RAM cannot hold it, see the chapter below.
• -ctk q4_0 -ctv q4_0 — at 262144 this frees ~4.8 GB against f16 for +0.4% perplexity; spend the VRAM on expert layers.
• -np 1 — mandatory here, see above.
• -fa on, -ub 1024 (-ub 512 on 16 GB and below).

Running it with less RAM: the PLE on an SSD

The 27 GB n-gram table is the reason this model wants a lot of RAM, and it is also the one part you can leave on disk: each token looks up one row per layer, 90 bytes each. Dropping -lzm off (and using -lm auto, i.e. mmap) lets those rows stream from the SSD instead. Measured on the machine of Tested hardware with the server confined to a cgroup whose memory limit is too small to cache the table, page cache dropped before each run, prompt 16k tokens, 128 tokens generated, NVMe (ADATA SX8200PNP), -ncmoe 37. These three rows are the speed fork again, so read them as ratios — a stock build is slower in every row by about the same factor:

RAM available to the process	PLE	flags	load	prefill 16k	decode	major faults	resident
120 GB	in RAM	-lm none -lzm off	61 s	977 t/s	54.5 t/s	58	61.7 GiB
60 GB (a 64 GB box)	on SSD	-lm auto -lzm auto	20 s	260 t/s	46.5 t/s	146 k	33.8 GiB
44 GB (a 48 GB box)	on SSD	-lm auto -lzm auto	20 s	220 t/s	42.5 t/s	225 k	34.1 GiB

Generation barely cares: −15% at 64 GB, −22% at 48 GB. One row per layer per token is ~200 KB of random reads, and an NVMe serves that in a few milliseconds spread over a 20 ms token. Prefill is what you pay with: 3.7–4.4× slower, because a single 1024-token ubatch needs ~200 MB of random reads. In practice that means a 100k-token document takes ~7 minutes to ingest instead of ~1.7, while chatting at short prompts feels nearly the same as with the table in RAM. Loading is 3× faster, since nothing is read up front.

So: 48–64 GB of RAM runs this model perfectly well for conversation and agent loops, and badly for feeding it large documents. If you have 96 GB or more, keep -lm none -lzm off and forget this chapter exists.

Speculative decoding (MTP) and this vector

Qwen3.8-Flash-Next ships a multi-token-prediction head, and unsloth publishes it as GGUF draft heads. Measured on the machine of Tested hardware, same quant and flags, with their prebuilt build (upstream PR #28243), --spec-draft-n-max 2, shared-Q8_0 head, two rounds each:

prompt	MTP off	MTP on	speedup	draft acceptance
2k	33.5 / 32.9 t/s	40.6 / 39.8 t/s	1.21×	73%
16k	32.8 / 35.2 t/s	45.9 / 49.7 t/s	1.41×	73%
239k	21.0 / 21.9 t/s	36.0 / 28.8 t/s	1.51×	82%

Prefill is unaffected, acceptance rises with context depth, and verification is exact, so the text is identical to non-speculative decoding.

Combining it with this vector. MTP needs a build with PR #28243; the projection needs the two patches in this repo. The two are independent — the vector steers the main model, the head only drafts, and verification is always done by the steered model, so the output stays abliterated even though the draft head itself is not steered (on refusal-adjacent prompts the drafts miss more often and you lose some of the speedup, but never the abliteration). Applying both patches to one build is straightforward and does not conflict.

One caveat if you also run the speed fork (branch sm120-5090-prod, write-up in unsloth discussion #3) rather than a stock build: MTP and a GPU-resident expert cache target the same bottleneck and do not stack — a speculative batch bypasses the single-token cache, so once the cache is on, MTP costs more than it saves (measured: −13 to −21% at 262144). MTP is the right lever on a plain llama.cpp build, which is what the numbers above assume; the expert cache is the right lever if you have the VRAM for it. Use one, not both.

Why this is not shipped as abliterated weights

The usual recipe (Arditi et al., heretic, and the "uncensored" weight releases) orthogonalizes every matrix that writes into the residual stream against the refusal direction, then re-quantizes. That was tried here first, on the BF16 weights, with a tool that rewrites attn_output, ssm_out, all 512 experts of ffn_down_exps, ffn_down_shexp, ple_value and token_embd, and it does not work on this architecture:

intervention	harmful refusals
stock model	50/50
runtime projection (this repo)	1/50
baked: per-layer direction into that layer's own writers	30/50
baked: one global direction everywhere, embeddings included	30/50
baked: 8-dimensional subspace (96% of the energy of the 41 per-layer directions)	31/50
baked: one direction, overcorrected (α = 2)	33/50

Published weight-space "uncensored" releases of this model were not scored with this harness, so this is a statement about the recipe as reproduced here, not a claim about anyone else's files. The projection itself lands correctly every time (|Vᵀ W| on a target tensor drops from 4.13 to 0.087 in a Q6_K file, from 0.37 to 0.002 in BF16, and non-target tensors stay byte-identical) — the model just refuses anyway.

The reason is the per-layer embedding (PLE) path. build_ple adds hidden + gated + conv_out to the residual, and conv_out = silu(Σₖ shifted_normalized · wₖ): a grouped RMSNorm with per-channel weights, a depthwise conv with per-channel taps, and finally a SiLU. A per-channel multiply already rotates a vector back out of any subspace the weights were cleared of, and an elementwise nonlinearity removes the invariant entirely — no edit of ple_value, ple_conv1d or the norms can make that term orthogonal to a direction for every input. So the refusal component is recreated in the residual on every token, downstream of every matrix you are allowed to touch, and weight-space abliteration plateaus at ~60% refusals. A larger subspace and overcorrection do not help: they cannot cancel a term that is regenerated nonlinearly after the writers have run.

Projecting the residual at runtime happens after that injection, at every layer, which is why the same direction that fails in weight space works here. For this architecture, a weights-only "uncensored" release would need a different direction (derived from block outputs rather than the residual) or actual fine-tuning.

Files

File	Description
Qwen3.8-Flash-Next-refusal-projection.gguf	the refusal direction, one unit vector per layer (use with --cvec-mode project)
Qwen3.8-Flash-Next-refusal-projection.json	the selected configuration and its scores
Qwen3.8-Flash-Next-refusal-additive.gguf	the raw mean-difference vector, for llama.cpp's stock additive mode
01-qwen4exp-cvec-hooks.patch	build_cvec on the hyper-connection residual + l_out-<il> export so llama-cvector-generator works on this architecture
02-cvec-projection-mode.patch	--cvec-mode {add,project}, --cvec-dir {per-layer,single:L}, and the llama-cvec-probe diagnostics tool
AGENTS.md	step-by-step setup instructions for a coding agent (Linux and Windows)

How it was made

1. llama-cvector-generator (method mean) over harmless/harmful prompt pairs, collecting the residual at all prompt positions. A last-token-only direction from the same prompts does not abliterate under projection, even though the two directions are nearly orthogonal — the averaging matters.
2. A grid over mode (add/project), scale, layer range and per-layer vs single-layer direction, each configuration scored on refusals in both thinking modes and on KL.
3. Winner: projection, per-layer, scale 1.0, layers 4..44. The additive vector needs scale ≈0.1 on layers 16..32 to reach the same refusal rate and pays 1.38 nats of KL for it; a single-layer direction removes only 6–16% of refusals.

Caveats

• Derived and measured on the Q2_K_XL quant, on one machine (Tested hardware); other quants share the residual geometry and should behave the same, but were not scored, and no other hardware was tried.
• KL is a heavy-tailed mean (individual tokens reach 18 nats); expect terser answers on some prompts.
• Requires the patched build. The patches are against llama.cpp master 159b74142 (September 2026) and are MIT-licensed, as is the llama.cpp code they modify.
• Removing refusals removes a safety behaviour. You own what the model writes with this applied.

Credits

Base model by Qwen, GGUF quantizations by unsloth, control-vector tooling from llama.cpp. Weights derived from Qwen3.8-Flash-Next are subject to the Qwen Community License 1.0.