Qwen3.5-9B-RANA-abliterated-MLX

MLX builds of abliterant/Qwen3.5-9B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.5-9B released by Abliterant, for Apple Silicon. The repository has 4-, 5-, 6- and 8-bit builds (5.97 GB to 10.45 GB) and a BF16 build in separate folders, the 4-bit build at the root, and an MTP drafter for speculative decoding. Vision is included in every build.

Quick start · Intended use · What changed · Evaluation · Limitations · Available files

Try RANA in Abliterant Chat · abliterant.ai · Research notes · All formats · Abliterant models

Research model with safety alignment removed. It will produce content the base model refuses. Read Intended use and Limitations before using it. The method, full evaluation and release gates are on the BF16 card.

At a glance

Field	Value
Developed by	Abliterant
Base checkpoint	abliterant/Qwen3.5-9B-RANA-abliterated, derived from Qwen/Qwen3.5-9B
Release type	MLX conversion and AWQ-calibrated quantization of the BF16 reference; not additional training
Weight formats	4-, 5-, 6- and 8-bit (affine, group size 64) and BF16; MTP drafter in BF16
Runtime	mlx-vlm 0.7.12 with MLX 0.32.4; tested on Linux with MLX's CUDA backend (one RTX 5090)
Context	262,144 tokens configured; not established for MLX
Closeness to BF16	KL 0.0009 (8-bit) to 0.0539 (4-bit)
Compared with a public abliteration	Against bowmanslayer/Qwen3.5-9B-Uncensored: +14.9 pp useful answers to harmful requests (67.4% vs 52.5%), 4.2× smaller capability loss (0.79 vs 3.31 pp) and 2.7× closer to the base model (KL 0.109 vs 0.294); measured on the BF16 build (BF16 card)
License	Apache-2.0
Intended use	Refusal-behavior research; see Intended use

Release family

Format	Repo	Size
BF16 (reference; Transformers, vLLM)	abliterant/Qwen3.5-9B-RANA-abliterated	19.3 GB
FP8 (block e4m3, Qwen's official FP8 layout; vLLM, SGLang)	abliterant/Qwen3.5-9B-RANA-abliterated-FP8	12.2 GB
GGUF (llama.cpp; Q8_0 to IQ2_M, plus vision and MTP files)	abliterant/Qwen3.5-9B-RANA-abliterated-GGUF	3.6–9.5 GB per quantization
MLX (Apple Silicon; 4-, 5-, 6-, 8-bit and BF16, plus MTP drafter) (this repo)	abliterant/Qwen3.5-9B-RANA-abliterated-MLX	6.0–18.8 GB per folder
AWQ 4-bit (compressed-tensors; vLLM)	abliterant/Qwen3.5-9B-RANA-abliterated-AWQ	11.7 GB

Quick start

Download one precision folder, then load it by local path:

pip install -U mlx-vlm    # mlx-vlm >= 0.7.12, mlx >= 0.32.4
hf download abliterant/Qwen3.5-9B-RANA-abliterated-MLX --include "4-bit/*" \
  --local-dir ./Qwen3.5-9B-RANA-abliterated-MLX

python -m mlx_vlm.generate --model ./Qwen3.5-9B-RANA-abliterated-MLX/4-bit \
  --enable-thinking --max-tokens 8192 --temperature 1.0 --top-p 0.95 --top-k 20 \
  --prompt "Explain how a hash map handles collisions."

With the MTP drafter (download mtp/* as well):

python -m mlx_vlm.generate --model ./Qwen3.5-9B-RANA-abliterated-MLX/4-bit \
  --draft-model ./Qwen3.5-9B-RANA-abliterated-MLX/mtp --draft-kind mtp \
  --enable-thinking --max-tokens 8192 --temperature 1.0 --top-p 0.95 --top-k 20 \
  --prompt "Explain how a hash map handles collisions."

Recommended settings

Setting	Value
Runtime	mlx-vlm 0.7.12 or later with MLX 0.32.4 or later
Thinking	--enable-thinking
Sampling	T = 1.0, top_p = 0.95, top_k = 20 (Qwen's thinking-mode settings)
Output length	Allow a large --max-tokens for long coding or technical answers
Vision	--image path/to/image.png
Server	python -m mlx_vlm.server --model ./Qwen3.5-9B-RANA-abliterated-MLX/4-bit (add --draft-model …/mtp --draft-kind mtp)

• Do not pass the repository ID itself to --model: mlx-vlm then downloads every folder. The root repeats the 4-bit/ files for apps that open a repository's top level.

Choosing a precision

8-bit (KL 0.0009) and 6-bit (0.0037) are near-lossless; 5-bit (0.0129) is a small step down; 4-bit (0.0539) has a visible quality cost for the smallest size. Per-folder numbers are under Available files; the functional checks ran on the 4-bit build.

Intended use

• Research only: interpretability, red-teaming, and robustness evaluation of refusal behavior.
• Responsibility: You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen, and the terms of any platform where outputs are used.

Out of scope

• Not for public or end-user deployment without a separate moderation layer. The model's own refusals have been largely removed, so any safety filtering has to happen outside it.

What changed

RANA (Reasoning-Anchored, Norm-preserving Ablation) is Abliterant's refusal-ablation method for reasoning models: it anchors the edit to the model's own reasoning, keeps the edited weights at their original norms, and keeps the reference model's vision, tool calling and speculative decoding working. Its design, full BF16 evaluation and release gates are documented on the BF16 card. This conversion is not additional training.

• Source: the BF16 reference, converted with mlx_vlm.convert (mlx-vlm 0.7.12, MLX 0.32.4).
• Quantization: affine quantization with group size 64. Before quantizing, AWQ (activation-aware weight quantization) rescales the attention and MLP weights from calibration activations so that the most important weights lose the least precision; the rescaling is folded into existing weights, so the layout is unchanged. AWQ was calibrated on 128 samples (up to 2,000 characters each) of the same calibration set as the release's other calibrated builds (half ultrachat_200k conversations, half the BF16 model's own thinking-mode answers).
• AWQ versus round-to-nearest: both were built and measured at every bit width; AWQ was closer to BF16 at every width and is the version shipped (table under Evaluation).
• Kept in BF16: the vision tower in every build; the MTP head is split into mtp/ with --mtp and kept in BF16.
• Layout: in every folder, tensor names, shapes and dtypes are identical to the matching mlx-community build of Qwen/Qwen3.5-9B (4-bit/ to 8-bit/: mlx-community -4bit to -8bit; bf16/: -bf16; mtp/: -MTP-bf16); config.json differs only by a dtype entry that this mlx-vlm version records.

Evaluation

Every precision was measured for closeness to the BF16 build, and the 4-bit build was served for functional checks. The harmful-request, capability and benign-task evaluations of the BF16 card were not repeated on this format.

Setup

• Compared: each quantized folder against bf16/, on MLX's CUDA backend on one RTX 5090 (Linux).
• Data: the release's 64 held-out conversations (63,959 answer tokens), disjoint from the calibration data.
• Closeness: teacher-forced; KL over the BF16 build's 64 most likely tokens, with each build's exact probabilities.
• Functional checks: the 4-bit build served by mlx_vlm.server with the MTP drafter.
• Raw results: results/.

Closeness to BF16

KL is the mean KL divergence from BF16's next-token probabilities over BF16's 64 most likely tokens (how far the probabilities have moved; 0 means identical), top-1 agreement is how often both pick the same most likely token, and ΔNLL is the change in the answers' mean negative log-likelihood.

Bits	AWQ: KL	AWQ: top-1	AWQ: ΔNLL	Round-to-nearest: KL	Round-to-nearest: top-1	Round-to-nearest: ΔNLL	Shipped
4	0.0539	91.7%	+0.0603	0.0755	90.1%	+0.0841	AWQ
5	0.0129	95.9%	+0.0130	0.0166	95.5%	+0.0167	AWQ
6	0.0037	97.7%	+0.0027	0.0045	97.6%	+0.0037	AWQ
8	0.0009	99.0%	+0.0003	0.0010	99.0%	+0.0004	AWQ

• AWQ is closer to BF16 than round-to-nearest at every width; the gap is largest at 4 bits (KL 0.0539 vs 0.0755).

Functional checks

Served by mlx_vlm.server with 4-bit/ and the mtp/ drafter:

• Text: arithmetic answered correctly: pass.
• Vision: reads the code word and shape from a synthetic image: pass.
• Tool calling: call → result → second call with a new argument: pass.
• MTP speculative decoding: the server ran with the drafter; it does not report acceptance counts, so the acceptance rate was not measured.

Limitations

• Refusal behavior was not re-measured on this format. The harmful-request evaluation ran on the BF16 build only; closeness to BF16 and the functional checks under Evaluation are what was measured here.
• Everything listed under Limitations on the BF16 card applies here too: safety alignment removed, judge-rated usefulness, longer coding answers and the TruthfulQA change measured there.
• Apple Silicon was not measured. All checks ran on Linux with MLX's CUDA backend.
• The MTP drafter's acceptance rate was not measured on MLX.

Recommendations

• Precision choice: 8-bit or 6-bit when memory allows; see Choosing a precision.
• Safety filtering: keep it in a separate moderation layer outside the model; see Out of scope.

Available files

Other formats are listed under Release family.

Folder	Precision	Size	KL vs BF16	Top-1 agreement
(root)	same files as 4-bit/	5.97 GB	0.0539	91.7%
4-bit/	4-bit, AWQ-calibrated	5.97 GB	0.0539	91.7%
5-bit/	5-bit, AWQ-calibrated	7.09 GB	0.0129	95.9%
6-bit/	6-bit, AWQ-calibrated	8.21 GB	0.0037	97.7%
8-bit/	8-bit, AWQ-calibrated	10.45 GB	0.0009	99.0%
bf16/	BF16, unquantized	18.84 GB	reference	reference
mtp/	MTP drafter, BF16	0.51 GB	–	–

• Each folder holds model-*.safetensors, model.safetensors.index.json, config.json, tokenizer and processor files and chat_template.jinja.
• LICENSE: the Apache-2.0 license inherited from Qwen.
• assets/abliterant-model-card-header.png: the card banner.
• results/: fidelity_mlx_*.json (closeness of every AWQ and round-to-nearest build), mlx_select.json (the per-width choice), smoke_mlx_4bit.json (functional checks), verify_mlx_layout.json (layout checks).

Provenance and license

• Publisher: Abliterant.
• Source weights: abliterant/Qwen3.5-9B-RANA-abliterated, the BF16 reference of this release, derived from Qwen/Qwen3.5-9B.
• Method: RANA (Reasoning-Anchored, Norm-preserving Ablation); see the BF16 card.
• Conversion: mlx-vlm 0.7.12 with MLX 0.32.4.
• Evidence: results/.
• License: Apache License 2.0, inherited from Qwen. Upstream attribution and obligations remain applicable.

Credits

• Qwen team: base model Qwen/Qwen3.5-9B.
• Arditi et al., 2024: "Refusal in Language Models Is Mediated by a Single Direction".
• Jim Lai (grimjim): prior work on norm-preserving abliteration.
• Apple MLX team and mlx-vlm (Prince Canuma): MLX and the conversion and runtime tooling.
• Datasets: HuggingFaceH4/ultrachat_200k.