Instructions to use abliterant/Qwen3.5-9B-RANA-abliterated-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use abliterant/Qwen3.5-9B-RANA-abliterated-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="abliterant/Qwen3.5-9B-RANA-abliterated-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("abliterant/Qwen3.5-9B-RANA-abliterated-FP8")
model = AutoModelForMultimodalLM.from_pretrained("abliterant/Qwen3.5-9B-RANA-abliterated-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use abliterant/Qwen3.5-9B-RANA-abliterated-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "abliterant/Qwen3.5-9B-RANA-abliterated-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abliterant/Qwen3.5-9B-RANA-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/abliterant/Qwen3.5-9B-RANA-abliterated-FP8

SGLang

How to use abliterant/Qwen3.5-9B-RANA-abliterated-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "abliterant/Qwen3.5-9B-RANA-abliterated-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abliterant/Qwen3.5-9B-RANA-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "abliterant/Qwen3.5-9B-RANA-abliterated-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "abliterant/Qwen3.5-9B-RANA-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use abliterant/Qwen3.5-9B-RANA-abliterated-FP8 with Docker Model Runner:
```
docker model run hf.co/abliterant/Qwen3.5-9B-RANA-abliterated-FP8
```

Qwen3.5-9B-RANA-abliterated-FP8

FP8 weights of abliterant/Qwen3.5-9B-RANA-abliterated, a refusal-ablated Qwen/Qwen3.5-9B released by Abliterant, for vLLM and SGLang. The checkpoint uses Qwen's official block-FP8 format and is 12.2 GB, against 19.3 GB for BF16.

Quick start · Intended use · What changed · Evaluation · Limitations · Available files

Try RANA in Abliterant Chat · abliterant.ai · Research notes · All formats · Abliterant models

Research model with safety alignment removed. It will produce content the base model refuses. Read Intended use and Limitations before using it. The method, full evaluation and release gates are on the BF16 card.

At a glance

Field	Value
Developed by	Abliterant
Base checkpoint	abliterant/Qwen3.5-9B-RANA-abliterated, derived from Qwen/Qwen3.5-9B
Release type	Quantization of the BF16 reference; not additional training
Weights	FP8 (e4m3) safetensors in 128 × 128 blocks with BF16 scales, 12.21 GB
Runtime	vLLM 0.30.0; tested on one RTX 5090 32 GB (Blackwell)
Context	262,144 tokens configured; tested at a 20,480-token serving limit
Closeness to BF16	KL 0.0045; same most likely token at 97.7% of positions
Compared with a public abliteration	Against `bowmanslayer/Qwen3.5-9B-Uncensored`: +14.9 pp useful answers to harmful requests (67.4% vs 52.5%), 4.2× smaller capability loss (0.79 vs 3.31 pp) and 2.7× closer to the base model (KL 0.109 vs 0.294); measured on the BF16 build (BF16 card)
License	Apache-2.0
Intended use	Refusal-behavior research; see Intended use

Release family

Format	Repo	Size
BF16 (reference; Transformers, vLLM)	`abliterant/Qwen3.5-9B-RANA-abliterated`	19.3 GB
FP8 (block e4m3, Qwen's official FP8 layout; vLLM, SGLang) (this repo)	`abliterant/Qwen3.5-9B-RANA-abliterated-FP8`	12.2 GB
GGUF (llama.cpp; Q8_0 to IQ2_M, plus vision and MTP files)	`abliterant/Qwen3.5-9B-RANA-abliterated-GGUF`	3.6–9.5 GB per quantization
MLX (Apple Silicon; 4-, 5-, 6-, 8-bit and BF16, plus MTP drafter)	`abliterant/Qwen3.5-9B-RANA-abliterated-MLX`	6.0–18.8 GB per folder
AWQ 4-bit (compressed-tensors; vLLM)	`abliterant/Qwen3.5-9B-RANA-abliterated-AWQ`	11.7 GB

Quick start

Tested with vLLM 0.30.0 on one RTX 5090 32 GB (Blackwell).

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve abliterant/Qwen3.5-9B-RANA-abliterated-FP8 \
  --max-model-len 20480 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image": 1, "video": 0}' \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Recommended settings

Setting	Value
Thinking mode	On by default
Sampling	T = 1.0, top_p = 0.95, top_k = 20, presence_penalty = 1.5 (Qwen's thinking-mode settings)
Context	`--max-model-len 20480`, the serving limit of the BF16 evaluation; raise it for long coding or technical answers
MTP speculative decoding	`--speculative-config` as above; the MTP layers are stored in BF16
Blackwell GPUs	`VLLM_USE_FLASHINFER_SAMPLER=0`, as in the command above

vLLM reads the quantization format from config.json; no extra quantization flag is needed.

Intended use

Research only: interpretability, red-teaming, and robustness evaluation of refusal behavior.
Responsibility: You are responsible for complying with applicable law, the Apache-2.0 license inherited from Qwen, and the terms of any platform where outputs are used.

Out of scope

Not for public or end-user deployment without a separate moderation layer. The model's own refusals have been largely removed, so any safety filtering has to happen outside it.

What changed

RANA (Reasoning-Anchored, Norm-preserving Ablation) is Abliterant's refusal-ablation method for reasoning models: it anchors the edit to the model's own reasoning, keeps the edited weights at their original norms, and keeps the reference model's vision, tool calling and speculative decoding working. Its design, full BF16 evaluation and release gates are documented on the BF16 card. This conversion is not additional training.

Format: the same as Qwen's official FP8 checkpoints (for example Qwen/Qwen3.5-27B-FP8): 207 linear layers (attention, linear-attention input and output projections, MLP, and the MTP layer's attention and MLP) are stored as FP8 e4m3 in 128 × 128 blocks, each block with one BF16 scale (weight_scale_inv); activations are quantized dynamically. The tensor types, dtypes and quantization_config match the official checkpoint.
Kept in BF16: token embeddings, lm_head, the linear-attention convolution and gate projections, the vision tower and all norms. All 568 BF16 tensors are byte-identical to the BF16 reference.
Same arithmetic as Qwen: re-quantizing three BF16 tensors of Qwen/Qwen3.5-27B with this conversion reproduces Qwen's own FP8 file byte for byte: 173,015,040 of 173,015,040 FP8 values and 10,560 of 10,560 scales.
Rounding error: the dequantized FP8 weights differ from BF16 by 2.7% (relative Frobenius norm) in every one of the 207 layers.
Published files: re-sharded for this repository into about 5 GB files; every tensor is byte-identical to the evaluated build (SHA-256 checked per tensor).

Evaluation

This build was compared with the BF16 reference for closeness and checked for vision, tool calling and MTP speculative decoding. The harmful-request, capability, benign-task, vision and tool evaluations of the BF16 card were not repeated on this format.

Setup

Compared: this build against the BF16 reference, both served by vLLM 0.30.0 on one RTX 5090 32 GB (Blackwell).
Data: 64 held-out conversations: ultrachat_200k test prompts with the BF16 model's own thinking-mode answers (63,959 answer tokens), disjoint from the calibration data.
Closeness: each build scores the same answer tokens (teacher forcing) with its top 64 next-token log-probabilities.
Functional checks: the build served with the command in Quick start.
Raw results: results/.

Closeness to BF16

KL is the mean KL divergence from BF16's next-token probabilities over BF16's 64 most likely tokens (how far the probabilities have moved; 0 means identical), top-1 agreement is how often both pick the same most likely token, and ΔNLL is the change in the answers' mean negative log-likelihood.

Build	Top-1 agreement	KL	99th pct KL	ΔNLL
Qwen3.5-9B-RANA-abliterated-FP8	97.7%	0.0045	0.039	+0.0048

FP8 tracks BF16 closely: the same most likely token at 97.7% of positions and a perplexity 0.5% higher on the held-out answers.

Functional checks

Served on one RTX 5090 32 GB (Blackwell) with the command in Quick start (results/smoke_FP8.json):

Text: thinking-mode arithmetic answered correctly: pass.
Vision: reads the code word and shape from a synthetic image: pass.
Tool calling: call → result → second call with a new argument: pass.
MTP speculative decoding (1 drafted token per step): 478 of 546 drafted tokens accepted (87.5%) on one generation at temperature 0: pass.
Published files: served again from this repository's files in vLLM 0.30.0 on one A40 48 GB (Ampere): text pass, vision pass, tool calling pass, MTP pass (results/layout_smoke_FP8.json).

Limitations

Refusal behavior was not re-measured on this format. The harmful-request evaluation ran on the BF16 build only; closeness to BF16 and the functional checks under Evaluation are what was measured here.
Everything listed under Limitations on the BF16 card applies here too: safety alignment removed, judge-rated usefulness, longer coding answers and the TruthfulQA change measured there.
Tested on two GPUs. Closeness was measured on one RTX 5090 32 GB (Blackwell); the published files were also served on one A40 48 GB (Ampere). Ada and Hopper GPUs were not tested.

Recommendations

Use this build when BF16 does not fit; it keeps 97.7% top-1 agreement with BF16 at 63.2% of its size.
Keep any safety filtering in a separate moderation layer (see Out of scope).

Available files

Other formats are listed under Release family.

model-0000X-of-0000N.safetensors (about 5 GB shards), model.safetensors.index.json, config.json (with quantization_config), tokenizer and processor files, chat_template.jinja, LICENSE: the model.
assets/abliterant-model-card-header.png: the card banner.
results/: verify_fp8.json (format, conversion and arithmetic checks), fidelity_vllm_FP8.json and fidelity_vllm_BF16.json (closeness), smoke_FP8.json (functional checks).

Provenance and license

Publisher: Abliterant.
Source weights: abliterant/Qwen3.5-9B-RANA-abliterated, the BF16 reference of this release, derived from Qwen/Qwen3.5-9B.
Method: RANA (Reasoning-Anchored, Norm-preserving Ablation); see the BF16 card.
Conversion: block-FP8 quantization reproducing Qwen's arithmetic (scale = block maximum × float32(1/448), round to nearest even); checks in results/verify_fp8.json.
Evidence: results/.
License: Apache License 2.0, inherited from Qwen. Upstream attribution and obligations remain applicable.

Credits

Qwen team: base model Qwen/Qwen3.5-9B.
Arditi et al., 2024: "Refusal in Language Models Is Mediated by a Single Direction".
Jim Lai (grimjim): prior work on norm-preserving abliteration.
Qwen team (FP8 format): Qwen/Qwen3.5-27B-FP8.
vLLM: serving and evaluation.