DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated

Qwen3.8-Flash-Next FP4-Local — Abliterated

NVIDIA's FP4-Local Qwen3.8-Flash-Next with the refusal direction removed, in the same 96 GiB format, at about the same speed.

This checkpoint keeps NVIDIA's FP4-Local storage format, including its compressed NVFP4 scales. Like the base, it needs the B12X vLLM container used in serve-spark.sh. Stock vLLM and Transformers cannot load it.

Highlights

• Refusals removed: 0 of 100 development and 6 of 100 holdout test prompts were refused, against 99 and 100 for the NVIDIA base.
• Same format as the base: every tensor has the base checkpoint's name, dtype and shape. Routed experts stay NVFP4, and the attention and shared-expert projections MXFP8.
• About the same speed: 38.0 tok/s on prose and 54.3 on code on one DGX Spark, against 39.4 and 50.7 for the base with the same serve profile.
• Nothing else removed: image input, tool calling, the thinking toggle and the built-in MTP draft head all pass the same checks as the base, with the same 262K context setting.
• Self-contained: the edit is baked into the weights. There is no adapter or runtime hook, and one script serves it on a DGX Spark.

Model overview

Original model	Qwen/Qwen3.8-Flash-Next by Qwen
Quantized base	nvidia/Qwen3.8-Flash-Next-FP4-Local by NVIDIA, revision b9ce18a
Parameters	125B total, 6B active, plus a 51B n-gram embedding table and a 4B MTP layer (from NVIDIA's card)
Input / output	Text and images in, text out
Context	262,144 tokens as served by serve-spark.sh
Weights	NVFP4 routed experts; MXFP8 attention projections and shared experts; about 96 GiB in 21 shards
What changed	Refusal-direction ablation baked into the 48 attention output projections and the expert down projections
Tested on	One NVIDIA DGX Spark (GB10, 128 GB)

Quick start (DGX Spark)

Requirements: a DGX Spark with Docker and NVIDIA GPU support, and nothing else large running. The script pulls the pinned image eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595.

hf download DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated --local-dir ./qwen-abliterated
bash ./qwen-abliterated/serve-spark.sh ./qwen-abliterated

The first start compiles and autotunes kernels for your machine, about 23 minutes on ours. Later starts reuse the cache and take about 5 minutes. The API is then at http://127.0.0.1:8000/v1, with model ID qwen-trial117.

curl http://127.0.0.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "qwen-trial117",
  "messages": [{"role": "user", "content": "Explain how a heat pump works in three sentences."}],
  "chat_template_kwargs": {"enable_thinking": false}
}'

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")
reply = client.chat.completions.create(
    model="qwen-trial117",
    messages=[{"role": "user", "content": "Write a haiku about unified memory."}],
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(reply.choices[0].message.reasoning)
print(reply.choices[0].message.content)

• Thinking is on by default and set per request with chat_template_kwargs.enable_thinking. Reasoning is returned in the message's reasoning field.
• Tools work through the OpenAI tools field, parsed with qwen3_xml.
• Sampling defaults come from generation_config.json: temperature 1.0, top-p 0.95, top-k 20.

The profile follows NVIDIA's published DGX Spark settings for the base:

• 262,144-token context and up to 8 concurrent sequences;
• FP8 KV cache and prefix caching;
• MTP speculative decoding with 4 draft tokens;
• the B12X MoE, linear and GDN kernels;
• a 104 GiB container memory cap.

Variable	Default	Effect
PORT	8000	API port
HOST	127.0.0.1	Bind address. Put an authenticating proxy in front before exposing it beyond the host.
MTP_TOKENS	4	Speculative draft tokens; 0 disables MTP
CACHE_DIR	~/.cache/qwen-trial117	Compiled kernels and autotune results

Evaluation

All measurements ran on one DGX Spark with serve-spark.sh, against NVIDIA's base checkpoint served the same way.

Refusals and drift

Refusal prompts were 100 development and 100 holdout prompts from mlabonne/harmful_behaviors. Answers were greedy, seed 3741, thinking off and capped at 100 tokens, and a refusal regex marked the refusals.

	Refusals, development	Refusals, holdout	First-token KL to base	Top-1 agreement with base
NVIDIA base	99 / 100	100 / 100	—	—
This model	0 / 100	6 / 100	0.207 / 0.219	81% / 84%

KL and agreement were measured on the first answer token for 100 development and 100 holdout prompts from mlabonne/harmless_alpaca. KL is in nats over each prompt's top-20 candidates, renormalised.

These are counts from a regex on short answers. They are not a measure of how often answers are correct or useful.

Capabilities

• Functional checks, 11 of 11 passed:
  • model listing and streamed usage;
  • thinking on and off;
  • automatic and forced tool calls, and a tool-result follow-up returning exact JSON;
  • image colour recognition;
  • exact retrieval of three planted markers from documents of about 8,000 and 30,000 tokens.
• Vision: 10 of 10 solid-colour images named correctly.
• Speculative decoding: MTP acceptance matches the base, so NVIDIA's draft layer still fits these weights. A greedy audit found every emitted token was the target model's top choice.
• Writing: a fixed 20-case editing and composition set, scored by an AI reviewer rather than people. 13 of 20 cases passed fully, against 14 for the base, and the two changed cases also vary between runs of the base.
  • On short arithmetic questions it tends to explain step by step and can run out of a tight token budget before stating the result.
  • Asked to tighten a long note, it edits more lightly than the base.

Speed

Single request, greedy, thinking off, 512 output tokens, MTP 4. Each figure is the median of six probes over two rounds, with the range in brackets.

	This model	NVIDIA base
Prose, tok/s	38.0 (37.7–39.6)	39.4 (39.1–40.9)
Code, tok/s	54.3 (48.5–56.0)	50.7 (49.3–52.4)
Decode step	67.7–67.9 ms	66.8–67.2 ms
KV cache capacity	947,989 tokens	902,668 tokens
Time to first token, short prompt	0.05–0.27 s	—
Time to first token, ~30,000-token prompt, cold	12.7 s	—

Speed depends on content: speculative decoding accepts more draft tokens on predictable text such as code.

On a more heavily tuned single-Spark stack with MTP 2, based on diffbot's DGX Spark recipe and not included here, prose reached 45.3–45.7 tok/s against 46.1–47.1 for the base.

How it was made

1. Find the refusal direction. This follows the method of Arditi et al. (2024) and the parameterization of Heretic, reimplemented for this checkpoint's native vLLM backend.
   • Per-layer directions come from attention-output activations on harmful and harmless prompt sets.
   • Each layer's strength comes from a distance-based weight kernel, with a fractional direction index and row-norm normalization on attention.
   • An Optuna TPE search traded refusals against KL, and trial 117 was chosen. Its parameters, directions and row norms are in bake-selection.json, frozen-directions.json and frozen-row-norms.json.
2. Bake it into the weights. The projection was applied to the 48 attention output projections and to the routed and shared expert down projections.
   • Expert weights were re-rounded in place in NVFP4 and MXFP8, with their scales unchanged.
3. Re-encode attention to MXFP8. Plain rounding of the edited attention weights would undo about a tenth of the edit, so reencode/mxfp8_reencode.py uses constrained rounding instead.
   • Each weight rounds to one of its two nearest E4M3 values, using the base's scale rule of one exponent per 32 inputs.
   • The roundings are chosen to cancel the error along each layer's removed direction.
   • Across the 48 layers, error along that direction is at most 0.002% of the edit, and overall weight error is the normal 1.5–1.9% for MXFP8. Per-layer figures are in reencode/reencode-report.json.

The configuration, tokenizer, chat template, vision encoder and MTP layer are NVIDIA's. The only addition to config.json is a standalone_provenance metadata entry. provenance.json records the full build and validation details.

Files

File	Contents
tensors/, model.safetensors.index.json	The weights: 21 shards in the base layout
config.json, tokenizer, chat_template.jinja, preprocessor configs	NVIDIA's files, unchanged apart from the metadata entry above
serve-spark.sh	One-command DGX Spark server
bake-selection.json, frozen-directions.json, frozen-row-norms.json	The trial 117 intervention
reencode/	MXFP8 re-encode script and per-layer report
provenance.json	Build and validation record
SHA256SUMS, release-files.json	Checksums for every file
LICENSE, NOTICE, upstream/	NVIDIA Open Model Agreement, notices, Qwen Community License 1.0 and NVIDIA's original card

Limitations

• Narrow evaluation: only the checks above were run. Broad benchmarks, video input, contexts beyond about 30,000 tokens, multi-user throughput and other hardware were not tested.
• Targets not met: the project aimed for zero refusals and a KL of 0.20 or less, and did not reach them. The model is published as measured.
• Still refuses sometimes: 6 of 100 holdout prompts were refused.
• Same base weaknesses: it can be wrong, inconsistent or misleading in the ways the base model can.

Responsible use

Abliteration removes the model's learned tendency to refuse. It will attempt requests the original declines, and it can produce harmful, offensive, false or illegal content.

• You are responsible for how you use it and for complying with the law and the licenses below.
• Do not deploy it to the public without your own safeguards.
• Do not use it for anything the Qwen or NVIDIA terms prohibit.

There is no warranty of any kind.

License

This is a derivative of NVIDIA's checkpoint and of Qwen's model, and both sets of terms apply. The base weights are not relicensed.

• NVIDIA Open Model Agreement: the governing terms for NVIDIA's checkpoint. A copy is in LICENSE.
  • Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement.
• Qwen Community License 1.0: covers the original model. A copy is in upstream/QWEN-LICENSE. Among its conditions:
  • Keep the license notice with all copies.
  • Products or services with more than 100 million monthly active users or US$20 million monthly revenue must show the model name in their user interface.
  • Model-as-a-Service and AI Work Assistant businesses need a separate license from Qwen for commercial use.

Read the full texts before you use or redistribute this model.

Citation

@misc{developingdad2026qwen38abliterated,
  title        = {Qwen3.8-Flash-Next FP4-Local -- Abliterated},
  author       = {DevelopingDad},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/DevelopingDad/Qwen3.8-Flash-Next-FP4-Local-Abliterated}}
}

@article{arditi2024refusal,
  title   = {Refusal in Language Models Is Mediated by a Single Direction},
  author  = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
  journal = {arXiv preprint arXiv:2406.11717},
  year    = {2024}
}

@misc{heretic,
  author       = {Weidmann, Philipp Emanuel},
  title        = {Heretic: Fully automatic censorship removal for language models},
  year         = {2025},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/p-e-w/heretic}}
}

Credits

This release stands on other people's work.

Model

• Qwen: Qwen3.8-Flash-Next, the original model, under the Qwen Community License 1.0.
• NVIDIA: the FP4-Local checkpoint, quantized with Model Optimizer (Apache-2.0), under the NVIDIA Open Model Agreement. NVIDIA's published DGX Spark profile is the basis of serve-spark.sh.

Method

• Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee and Neel Nanda: the refusal-direction research behind abliteration.
• Philipp Emanuel Weidmann (p-e-w) and the Heretic contributors: Heretic (AGPL-3.0). Its search parameterization was reimplemented here, and no Heretic code is included. The Heretic-DGX fork was consulted while adapting it.
• Maxime Labonne (mlabonne): the abliteration write-up and the two prompt sets used to find directions and measure refusals. Neither prompt set is redistributed here.
  • mlabonne/harmful_behaviors, revision 01cead0, comes from AdvBench (llm-attacks, MIT).
  • mlabonne/harmless_alpaca, revision 02c6a92, comes from Stanford Alpaca (CC BY-NC 4.0).
• Open Compute Project: the Microscaling (MX) format specification that MXFP8 follows.

Software

• vLLM (Apache-2.0): the serving engine.
• FlashInfer (Apache-2.0) and B12X by Local Inference Lab (Apache-2.0): the kernels.
• eugr/spark-vllm-docker (MIT): the DGX Spark vLLM image used for building and serving.
• Optuna (MIT), PyTorch (BSD-3-Clause) and safetensors (Apache-2.0).
• diffbot (MIT): the tuned-stack recipe behind the optional speed figure above.

None of this software is redistributed here; each project keeps its own license. This release is not endorsed by Qwen, NVIDIA or any of the projects above.