Refusal in Language Models Is Mediated by a Single Direction
Paper • 2406.11717 • Published • 17
Access this experimental checkpoint
Accept the applicable model licenses and acknowledge the experimental status to receive automatic download access.
This derivative inherits the NVIDIA Open Model Agreement and the upstream GLM MIT license. Review the licenses and evaluation limitations on this page before downloading. Hugging Face manages this access request and shares your profile information according to its gate settings.
Log in or Sign Up to review the conditions and access this model content.
GLM-5.3-Flash FP4-Local — Abliterated
A standalone weight-only abliteration with native NVFP4/FP8 storage, evaluated on two DGX Sparks.
Quick start · Serving profile · Evaluation · Method · Files · Credits
A derivative of NVIDIA's GLM-5.3-Flash-FP4-Local, published by DevelopingDad. The selected per-layer direction-removal edits are baked into the checkpoint. Downloaded weights run independently of the search-time projection hooks and external adapters.
This is the retained native-format V1, selected on November 5, 2026 and published on November 9, 2026. The source payload occupies approximately 177.5 GB / 165.3 GiB. Local text-serving validation used two DGX Sparks, each with 128 GB unified memory.
NVIDIA's compressed NVFP4 scale format requires a compatible B12X vLLM runtime. The pinned container and two-node serving profile below identify the tested stack. Configure the interconnect and container environment using NVIDIA's upstream instructions before launching the model.
Highlights
- Standalone baked checkpoint. The release includes all 40 model shards, auxiliary tensors, tokenizer, processor and native chat template.
- Native mixed precision. Routed experts use edited NVFP4 encodings. The 90 non-routed output writers retain FP8/MXFP8 storage and their original quantization recipes/scales.
- Documented task checks. The selected MTP3 serving profile passed 12/12 fresh acceptance cases. A later 24-session parent/V1 coding comparison and observed regressions are reported below.
- Measured serving tradeoffs. Reducing MTP depth from five to three improved completed-request throughput by 10.37–21.46% across four synthetic workloads, with a measured first-token latency tradeoff.
- Traceable release. The repository includes source and publication manifests, intervention parameters, source hashes, licenses and checksums.
These results describe specific local tests and serving configurations. The evaluation section retains the failed fidelity target, repeated coding regression and incomplete multi-turn comparison.
Model overview
| Item | Details |
|---|---|
| Original model | GLM-5.3-Flash by Z.AI |
| Quantized parent | NVIDIA GLM-5.3-Flash-FP4-Local, revision 76348ddb2be2e35e39ecfc00e75dc07703f4f3d5 |
| Architecture | Glm5NextForConditionalGeneration / glm5_next, hybrid sparse and linear attention with manifold-constrained hyper-connections |
| Parameters | 320B total, 18B activated, as stated in NVIDIA's pinned parent card |
| Input / output | Inherited text, image and video interfaces, with local derivative validation focused on text and tool workflows |
| Source payload | 177,519,332,182 bytes / approximately 165.3 GiB across 53 source-manifest files |
| Weight files | 40 numbered safetensors shards, model-inputscales.safetensors and tensors/native-QB.safetensors |
| Intervention | Per-layer output-writer projections across the 45-layer text decoder, followed by native-format requantization |
| Retained representation | Edited NVFP4 routed experts, 90 FP8/MXFP8 non-routed writers and preserved protected tensors |
| Local selection | Original proposal 66 / study trial 107, native-format V1 |
| Local serving context | Configured for 262,144 tokens, with separate exact-retrieval checks reaching 110,662 input tokens |
| Tested deployment | Two DGX Sparks with TP2, FP8 KV cache and three-token MTP |
The payload size measures stored files. Runtime memory also includes KV cache, workspaces and compilation state. NVIDIA's original card supplies the architecture and parameter description. Pinned parent card.
Quick start (two DGX Sparks)
1. Accept access and download
Open the model page while signed in, accept the license/experimental-status acknowledgement, and authenticate the Hugging Face CLI. Access requests are approved automatically.
hf auth login
hf download DevelopingDad/GLM-5.3-Flash-FP4-Local-Abliterated \
--local-dir ./glm53-abliterated
Run the download on both nodes, preserving the complete tensors/ directory and all auxiliary files. Allow at least the 177.5 GB payload size on each node, plus container and compilation-cache storage. Install the CLI using Hugging Face's instructions if needed. CLI installation.
2. Prepare the compatible runtime
The inspected local deployment uses this registry digest, also recorded by Docker for the running image.
eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595
Its Docker image ID is sha256:a7f76ae9f18c7701c9cde5f72c606393d9a21ca94dd8e53b27589935f64fcbe2. The captured runtime reports Python 3.12.3, vLLM 0.1.dev21830+g8244d1c44.d20261015, PyTorch 2.13.0+cu130, FlashInfer 0.7.1 and safetensors 0.8.0.
Use NVIDIA's two-Spark container/network setup with that image and bind this downloaded checkpoint read-only at /model on each node. Configure host networking, matching RoCE/socket interfaces, HEAD_IP and the interconnect environment before starting vLLM. The local profile uses the upstream B12X activation settings, including VLLM_B12X_MOE_FP4_FORCE_A16=1, B12X_W4A16_A4_PREFILL=1 and CUTE_DSL_ARCH=sm_121a. NVIDIA two-Spark setup.
Reference vLLM commands inside the configured containers
Set the same head interconnect address and an available shared master port in both containers. The /model mount must already contain the derivative downloaded above.
export HEAD_IP="<head-interconnect-ip>"
export MASTER_PORT=29513
COMMON_ARGS=(
--served-model-name glm53-abliterated
--host 127.0.0.1 --port 8000
--nnodes 2 --master-addr "$HEAD_IP" --master-port "$MASTER_PORT"
--tensor-parallel-size 2 --pipeline-parallel-size 1
--decode-context-parallel-size 1
--dtype bfloat16 --quantization modelopt_mixed --load-format b12x
--attention-backend B12X --moe-backend b12x --linear-backend b12x
--mamba-cache-mode align --kv-cache-dtype fp8 --block-size 256
--enable-prefix-caching --enable-chunked-prefill
--max-model-len 262144 --max-num-seqs 4
--max-num-batched-tokens 4096 --gpu-memory-utilization 0.87
--no-enable-flashinfer-autotune
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"humming","attention_backend":"B12X"}'
--reasoning-parser glm45 --tool-call-parser glm47
--enable-auto-tool-choice
)
On the worker, start rank 1.
vllm serve /model "${COMMON_ARGS[@]}" --node-rank 1 --headless
On the head, start rank 0.
vllm serve /model "${COMMON_ARGS[@]}" --node-rank 0
The commands express the captured inference profile with a public example model alias. They require the upstream container, model mounts and interconnect setup. This documentation revision was checked against the captured configuration. A fresh portable deployment from these examples has not been exercised.
3. Check readiness and send a request
On the head host, wait for health before sending inference requests.
curl --fail http://127.0.0.1:8000/health
The reference commands expose an OpenAI-compatible API at http://127.0.0.1:8000/v1 with model ID glm53-abliterated.
curl --fail http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "glm53-abliterated",
"messages": [
{"role": "user", "content": "Explain how a heat pump works in three sentences."}
],
"reasoning_effort": "low",
"max_tokens": 1024,
"temperature": 0.7
}'
For Python, install the OpenAI client in your own environment.
python3 -m pip install openai
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="none")
reply = client.chat.completions.create(
model="glm53-abliterated",
messages=[{"role": "user", "content": "Write a haiku about unified memory."}],
reasoning_effort="low",
max_tokens=1024,
)
message = reply.choices[0].message
reasoning = getattr(message, "reasoning", None)
if reasoning:
print(reasoning)
print(message.content)
- Reasoning controls. Native template effort values are
low,highandmax. Unrecognized or omitted values render Max in the inherited template. Set the effort explicitly when controlling a request's reasoning budget. - Tools. Use OpenAI-compatible
toolsandtool_choicefields with the configuredglm47tool parser. Theglm45reasoning parser separates reasoning from answer content. - Sampling. The inherited generation file specifies temperature 1.0 and top-p 0.95. Historical greedy evaluations used temperature 0 and fixed seeds. The curl example chooses its own temperature explicitly.
Serving profile
| Setting | Captured local value |
|---|---|
| Nodes / tensor parallelism | Two / TP2 |
| Compute / loading | BF16 compute, modelopt_mixed, B12X load format |
| Attention / MoE / linear backends | B12X |
| Maximum configured context | 262,144 tokens |
| Active sequences | Up to four |
| KV cache | FP8, block size 256 |
| Prefix caching / chunked prefill | Enabled |
| Maximum batched prefill tokens | 4,096 |
| Speculative decoding | MTP, three draft tokens |
| GPU memory fraction | 0.87 |
| Reasoning / tool parsers | glm45 / glm47 |
Available memory and the effective KV layout determine concurrent context capacity. The four scheduler slots share that capacity.
Evaluation
The results below summarize retained local evidence from November 5–7, 2026. Different rows measure different interventions and workloads. The model-card revision adds documentation and source checks.
Coding, tool use and long context
| Check | Result | Scope |
|---|---|---|
| Native-format export diagnostics | 40/40 automatic cases passed | Earlier bounded export checks |
| Fresh MTP serving acceptance | MTP5 12/12, MTP3 12/12 | Same V1 weights, seed 3761, arithmetic, schema, code, retrieval and native tool cases |
| Exact long retrieval | 2/2 on the retained MTP3/4096 profile | Fresh documents with 32,710 and 110,662 actual input tokens, 12 requested keys and a last-record correction |
| Paired native coding repairs | Parent 10/12, V1 11/12 | Six fresh multi-file tasks, two seeds and both checkpoints, 24 actual CLI sessions |
| Same coding comparison, excluding ambiguous root-route fixture | Parent 10/10, V1 10/10 | Post-hoc ambiguity sensitivity, original scores retained |
| Strict complete-workflow score | Parent 9/12, V1 9/12 | The same 24-session trial, including reporting/completion requirements |
The native coding comparison found that all failing repairs concerned an ambiguously worded root-route fixture. Both checkpoints ran the current test suites in all eight normal sessions and respected all four source-only controls.
Earlier matched development tasks also exposed a repeated payment receipt-isolation regression, with parent 2/2 correct and V1 0/2. An inventory omission cleared on a repeat. Both checkpoints passed the eight primary legitimate-sensitive behavior cases, so that panel established no over-refusal benefit.
The longer multi-turn comparison was paused after 23 of 36 planned turns, with an additional interrupted, unscored turn. Its unequal partial totals provide no final model ranking. Long native tool workflows have also shown scope drift and loops. Local client/bridge recovery work has separate identities and sits outside this weight release.
Refusal behavior and numerical drift
This checkpoint was selected through a refusal-direction search. A final baked-checkpoint 100-prompt refusal table comparable to the earlier Qwen release is unavailable. Search-hook phrase counts and short automatic checks retain their original scope.
Historical native-format export measurements used 100 benign prompts per split and the first answer-token distribution.
| V1 numerical comparison | Development | Holdout |
|---|---|---|
| Full-vocabulary first-token KL relative to NVIDIA parent | 0.0744780 | 0.0615110 |
| First-token KL relative to selected runtime projection | 0.0472034 | 0.0466650 |
| Top-1 token matches relative to selected runtime projection | 92/100 | 89/100 |
The original 0.01 projection-fidelity target failed. KL measures probability-distribution drift in nats. It supplies a numerical comparison at one position. These GLM full-vocabulary measurements use a different method from the Qwen card's renormalized top-20 metric.
Serving speed
Measured on November 6, 2026, using the same V1 checkpoint, pinned runtime, TP2, FP8 KV, 4,096-token prefill chunks and four scheduler slots. Only MTP depth changed. Values pool six scored waves per arm/workload over two clean rounds. Seed was 3741, with fixed input/output lengths and greedy decoding. A contaminated candidate round was excluded and retained in the local record.
| Input / output tokens | Concurrency | MTP5 aggregate tok/s | MTP3 aggregate tok/s | Gain |
|---|---|---|---|---|
| 1,024 / 512 | 1 | 27.00 | 32.79 | +21.46% |
| 1,024 / 512 | 4 | 54.29 | 62.18 | +14.53% |
| 8,192 / 256 | 1 | 20.68 | 23.04 | +11.40% |
| 8,192 / 256 | 4 | 43.53 | 48.04 | +10.37% |
Throughput is aggregate output tokens divided by request-wave wall time, including queueing and prefill. Both columns measure this edited checkpoint under different MTP settings. These figures provide a serving-profile comparison.
In the second short-prompt, four-request round, median first-token wait increased 1.111 → 1.571 seconds, while median completion time fell 37.128 → 32.675 seconds. MTP3 failed the original conservative first-token latency gate and was subsequently deployed under an explicit net-benefit decision after fresh acceptance passed 12/12 for both arms. Greedy output strings also differed across the MTP settings. The throughput result carries these latency and output-equivalence limits.
How it was made
- Measure per-layer directions. A custom native-backend campaign measured 45 GLM directions using harmful and harmless calibration cohorts, with attention-pre coordinates and frozen attention row norms.
- Select a projection. The reconstructed search used separate attention and MLP layer envelopes, with an Optuna TPE continuation minimizing first-token drift under the phrase-screen constraints. Original proposal 66 / study trial 107 supplied the retained selection.
- Bake the writers. Apply the selected output-writer projections and requantize into native formats. Export accounting records 12,186 writer matrices, 84,852 modified tensors and 210,189 protected tensors. The retained V1 includes the 90 non-routed FP8/MXFP8 writers.
- Reload and evaluate the serialized artifact. Numerical fidelity, task outcomes and serving configurations were checked separately. The selected checkpoint retains the failed historical fidelity target and the observed task limitations above.
The direction-removal research and Heretic's search parameterization informed this approach. The GLM campaign used a reconstructed implementation. Its original Qwen search implementation/cohort and the complete NVIDIA BF16-teacher/QAD recipe remain incompletely recovered. Arditi et al., Heretic.
Calibration and evaluation slices
| Cohort role | Harmless Alpaca | Harmful Behaviors |
|---|---|---|
| Direction calibration | train[:300] |
train[:300] |
| Search development | test[:100] |
train[300:400] |
| Search holdout | test[100:200] |
test[:100] |
The pinned sources are mlabonne/harmless_alpaca@02c6a92cfcf11bb0c387334f8146d149d65b587f and mlabonne/harmful_behaviors@01cead01398926d81f7c52bdb790ee8cf77ebba7. Search evaluations used seed 3741 and greedy decoding. The short behavioral search outputs had a 100-token cap. These calibration/search cohorts are separate from the subsequent coding, retrieval and MTP qualification fixtures. Harmless Alpaca, Harmful Behaviors.
Provenance and release identity
| Item | Identity |
|---|---|
| NVIDIA parent revision | 76348ddb2be2e35e39ecfc00e75dc07703f4f3d5 |
| Original source artifact SHA-256 identity | 0f52b8c5cb5fca482caf7b9f601515cf08e0b04699ae697d461304b3ea82f371 |
| Direction payload SHA-256 | 9031cbebafdd3de9aa8e89b505a45cdf8f72cdb8eebf78a64aca610eeee75f3a |
| Selection SHA-256 | 84237ae0c4250043df6e0cab567bb0d7ec828539cf9eda3a7824ebb5d64196e2 |
The unchanged source export-manifest.json includes hashes for the inherited parent README. That card is preserved under upstream/README.md, while the root README describes this derivative. The source identity above stays tied to that original manifest. The separate publication manifest and checksums describe this repository's release layout and current card.
Detailed projection envelopes, direction/norm hashes, exporter/math source hashes and a packaging-only recovery history are preserved in the original provenance. The tokenizer, template, processor and model configuration retain the parent bytes. The underlying direction tensors and raw search response corpus remain in the local experiment archive. Abliteration provenance, source manifest.
Files
| File or directory | Contents |
|---|---|
model-00001-of-00040.safetensors … model-00040-of-00040.safetensors |
Main numbered weight shards |
model-inputscales.safetensors, tensors/native-QB.safetensors |
Required auxiliary tensors |
model.safetensors.index.json |
Complete tensor-to-file index |
config.json, hf_quant_config.json |
Model and mixed-quantization configuration |
Tokenizer files, chat_template.jinja, processor_config.json, generation_config.json |
Inherited processing, formatting and generation files |
abliteration-provenance.json, export-manifest.json |
Original intervention and source artifact identities |
publication-manifest.json, SHA256SUMS |
Publication file hashes, sizes and card revision |
LICENSE.txt, LICENSE-NVIDIA.pdf, LICENSE-GLM.txt, NOTICE |
Governing terms and attribution |
upstream/README.md |
Preserved NVIDIA model card |
Download the complete repository for inference. The November 9 publication verification read and hashed all 53 source-manifest files, matched all 42 published safetensors hashes/sizes and downloaded their complete headers, and fully downloaded and hashed the supporting files. That publication check establishes file identity and availability. Publication manifest, checksums.
Limitations
- Fidelity target failed. The frozen selected-hook KL ceiling of 0.01 remains unmet after baking.
- Known coding regression. Repeated receipt-isolation failures occurred in a matched development panel. Later native coding checks have their own fixture scope and ambiguity sensitivity.
- Incomplete longer comparison. The multi-turn study was paused before all planned turns completed. Long native tool workflows have exhibited loops and scope drift.
- Refusal-removal benefit undermeasured. A broad post-export refusal/over-refusal improvement has not been established for this serialized V1.
- Modality and context scope. The inherited model exposes image/video interfaces. Broad derivative multimodal quality and complete 262K workload qualification remain open.
- Runtime scope. Evidence covers the pinned B12X two-Spark stack. The documentation examples have not undergone a new clean-room deployment trial.
The model can generate inaccurate, harmful or otherwise unsuitable outputs. Users are responsible for applying the governing terms and evaluating their intended applications. The model is distributed with the warranty limitations stated in its licenses.
License
The NVIDIA parent is governed by the NVIDIA Open Model Agreement released April 2, 2026. The original Z.AI GLM model carries the MIT License. Both texts and attribution notices accompany this release. NVIDIA agreement text, official agreement PDF, GLM MIT license, NOTICE.
Licensed by NVIDIA Corporation under the NVIDIA Open Model Agreement.
Citation
@misc{developingdad2026glm53abliterated,
title = {GLM-5.3-Flash FP4-Local -- Abliterated},
author = {DevelopingDad},
year = {2026},
howpublished = {\url{https://huggingface.co/DevelopingDad/GLM-5.3-Flash-FP4-Local-Abliterated}}
}
@article{arditi2024refusal,
title = {Refusal in Language Models Is Mediated by a Single Direction},
author = {Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Gurnee, Wes and Nanda, Neel},
journal = {arXiv preprint arXiv:2406.11717},
year = {2024}
}
Credits
- Z.AI developed GLM-5.3-Flash, its architecture, tokenizer and multimodal components. Original GLM.
- NVIDIA supplied the FP4-Local quantization-aware-distilled parent, Model Optimizer and the published two-Spark runtime guidance. NVIDIA parent, Model Optimizer.
- Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee and Neel Nanda authored the refusal-direction research. Philipp Emanuel Weidmann and the Heretic contributors developed the search approach that informed the local reconstruction. Research, Heretic.
- Maxime Labonne supplied the harmful/harmless datasets used in direction calibration and search. Dataset publisher.
- vLLM, Local Inference Lab/B12X, FlashInfer and eugr/spark-vllm-docker supplied the serving engine, kernels and compatible Spark image. vLLM, B12X, FlashInfer, Spark image project.
- Optuna, PyTorch and Hugging Face/safetensors supplied the search, tensor and storage tooling. Optuna, PyTorch, safetensors.
- DevelopingDad created and published this local derivative. OpenAI Codex assisted with evaluation, packaging and documentation.
Related release
Our earlier Qwen3.8-Flash-Next-FP4-Local-Abliterated card documents a separate checkpoint, its own refusal/capability measurements, a single-Spark launcher and a two-Spark recipe. Its results and hardware profiles retain their own scope. Qwen3.8-Flash-Next-FP4-Local-Abliterated.
- Downloads last month
- 4