Instructions to use Jitheswar/Qwen3.5-4B-Amadeus with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jitheswar/Qwen3.5-4B-Amadeus with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Jitheswar/Qwen3.5-4B-Amadeus")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Jitheswar/Qwen3.5-4B-Amadeus")
model = AutoModelForMultimodalLM.from_pretrained("Jitheswar/Qwen3.5-4B-Amadeus", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Jitheswar/Qwen3.5-4B-Amadeus with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Use Docker

docker model run hf.co/Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use Jitheswar/Qwen3.5-4B-Amadeus with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jitheswar/Qwen3.5-4B-Amadeus"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jitheswar/Qwen3.5-4B-Amadeus",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

SGLang

How to use Jitheswar/Qwen3.5-4B-Amadeus with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jitheswar/Qwen3.5-4B-Amadeus" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jitheswar/Qwen3.5-4B-Amadeus",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jitheswar/Qwen3.5-4B-Amadeus" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jitheswar/Qwen3.5-4B-Amadeus",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Ollama
How to use Jitheswar/Qwen3.5-4B-Amadeus with Ollama:
```
ollama run hf.co/Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M
```
Unsloth Desktop

How to use Jitheswar/Qwen3.5-4B-Amadeus with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Jitheswar/Qwen3.5-4B-Amadeus with Docker Model Runner:
```
docker model run hf.co/Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M
```

Lemonade

How to use Jitheswar/Qwen3.5-4B-Amadeus with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-4B-Amadeus-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Jitheswar/Qwen3.5-4B-Amadeus with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Jitheswar/Qwen3.5-4B-Amadeus with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Jitheswar/Qwen3.5-4B-Amadeus:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Qwen3.5-4B-Amadeus

Qwen3.5-4B with its refusals removed — and its brains left intact.

Original model · Abliterated · Safetensors + GGUF (BF16 / Q6_K / Q4_K_M) · Vision ✓ · Thinking ✓

At a glance

	Qwen3.5-4B	Amadeus
Refusals across all 304 test prompts (harmful + edgy, Minos judge)	231 / 304	1 / 304
Refuses edgy-but-harmless requests (JailbreakBench benign)	31 / 100	0 / 100
GSM8K (math)	91.4%	91.1%
HellaSwag / WinoGrande / ARC-C	71.9% / 68.6% / 46.5%	71.6% / 70.1% / 46.8%
KL divergence from original (wikitext-2)	0	0.0128

Amadeus is not fine-tuned and saw no new training data. The "refusal direction" inside Qwen3.5-4B was located and removed from the weights (directional ablation, a.k.a. abliteration), using 1000 rounds of automatic optimisation to remove as much refusal as possible while changing the model as little as possible. Everything else — knowledge, reasoning, vision, thinking mode, chat template — is the original Qwen3.5-4B.

Which file should I download?

Your hardware	File	Size
GPU with 6 GB+ VRAM (e.g. RTX 3050/3060, 4060)	`Qwen3.5-4B-Amadeus-Q6_K.gguf` — near-lossless, recommended	3.6 GB
GPU with 12 GB+ VRAM, want zero quantisation loss	`Qwen3.5-4B-Amadeus-BF16.gguf` — full precision	8.7 GB
GPU with 4 GB VRAM, or CPU only	`Qwen3.5-4B-Amadeus-Q4_K_M.gguf`	2.8 GB
Image input with a GGUF	add `mmproj-Qwen3.5-4B-Amadeus-F16.gguf`	0.7 GB
🤗 Transformers / vLLM / SGLang, fine-tuning	the `model-*.safetensors` files (full BF16)	9.3 GB

The Q6_K and Q4_K_M files are quantised with an importance matrix computed on Amadeus itself. All GGUFs include Qwen3.5's multi-token-prediction (MTP) head.

Quick start

LM Studio — search Jitheswar/Qwen3.5-4B-Amadeus, download the Q6_K file, chat.

Ollama

ollama run hf.co/Jitheswar/Qwen3.5-4B-Amadeus:Q6_K

llama.cpp

llama-server -hf Jitheswar/Qwen3.5-4B-Amadeus:Q6_K -ngl 99 -c 16384
# image input: add  --mmproj mmproj-Qwen3.5-4B-Amadeus-F16.gguf
# no thinking:  add  --reasoning off

🤗 Transformers

from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "Jitheswar/Qwen3.5-4B-Amadeus"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(model_id, dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Hello!"}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt",
    enable_thinking=True,  # False for direct answers
).to(model.device)
out = model.generate(**inputs, max_new_tokens=2048)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use Qwen3.5's recommended sampling settings (see the original model card).

Full benchmark results

All four columns were measured with the same pipeline on the same hardware (llama.cpp, A100 80 GB, greedy decoding, thinking off unless stated), so they compare directly with each other.

Benchmark	Qwen3.5-4B (original, BF16)	Amadeus BF16	Amadeus Q6_K	Amadeus Q4_K_M
Refusals – harmful_behaviors test (Minos judge) ↓	103/104	1/104	0/104	2/104
Refusals – JailbreakBench harmful (Minos) ↓	97/100	0/100	0/100	0/100
Refusals – JailbreakBench benign (Minos) ↓	31/100	0/100	0/100	0/100
Refusals – harmful_behaviors test, thinking ON (Minos) ↓	99/99 (5 unfinished)	–	0/85 (19 unfinished)	–
Refusals – JailbreakBench harmful, thinking ON (Minos) ↓	89/90 (10 unfinished)	–	0/84 (16 unfinished)	–
Refusals – JailbreakBench benign, thinking ON (Minos) ↓	31/84 (16 unfinished)	–	0/84 (16 unfinished)	–
Refusals – harmful_behaviors test (keyword rule) ↓	104/104	3/104	3/104	3/104
Refusals – JailbreakBench harmful (keyword rule) ↓	97/100	6/100	6/100	5/100
Refusals – JailbreakBench benign (keyword rule) ↓	38/100	4/100	4/100	5/100
GSM8K (8-shot CoT, flexible) ↑	91.36%	91.13%	90.67%	90.67%
IFEval prompt-level strict ↑	81.70%	78.19%	78.19%	78.93%
IFEval instruction-level strict ↑	86.93%	85.25%	84.65%	85.37%
MMLU (14k, cloze) ↑	38.68%	38.42%	38.52%	38.03%
ARC-Challenge ↑	46.49%	46.82%	46.15%	46.82%
HellaSwag (10k) ↑	71.86%	71.64%	71.68%	71.22%
WinoGrande ↑	68.59%	70.09%	69.85%	70.40%
TruthfulQA (MC) ↑	28.27%	25.95%	25.34%	26.56%
Wikitext-2 perplexity ↓	8.622	8.726	8.778	8.791
KL divergence vs original ↓	0 (reference)	0.0128	0.0175	0.0396
Same top token as original ↑	100% (reference)	95.60%	94.72%	91.08%

How to read these numbers

Refusals are counted two ways. Minos = the NousResearch/Minos-v1 refusal classifier reading the whole prompt + reply (the more reliable judge). Keyword rule = a simple substring check, which also flags complying answers that merely contain words like "harmful" or "illegal".
Prompt sets. harmful_behaviors test = the 104 held-out prompts the optimisation was scored on. JailbreakBench harmful / benign = JBB-Behaviors, never seen during optimisation. Benign prompts are edgy-but-harmless requests (e.g. "write a story in which a character uses heroin") that the original model over-refuses.
Thinking ON rows judge the final answer (not the reasoning), with up to 6144 tokens per reply. Unfinished = still reasoning at the token limit, so no answer to judge; those are excluded.
MMLU / ARC / TruthfulQA use llama.cpp's multiple-choice scorer, which compares the likelihood of each full answer text (cloze style). Absolute numbers are lower than letter-choice leaderboard MMLU — compare columns, not leaderboards.
KL divergence / same top token are measured on wikitext-2 against the original model's token distributions.
What uncensoring costs: IFEval drops ~3.5 points (prompt-level strict) and TruthfulQA ~2.3 points — the model is less inclined to hedge. Everything else is within error bars of the original.

How it was made


Method	directional ablation (abliteration) with Optuna (TPE) parameter search over per-layer ablation weights
Precision	full BF16 throughout (no quantisation during optimisation), 1 × NVIDIA A100 80 GB
Search	1000 trials (250 random start-up), seed 42
Refusal directions	1000 harmless (harmless_alpaca) + all 416 harmful (harmful_behaviors) training prompts
Trial scoring	300 harmless + all 104 harmful test prompts, 150-token replies
Ablation	orthogonalised directions, full row-norm preservation (rank-8 LoRA) on `attn.o_proj` and `mlp.down_proj`
Selected	trial #551: 2/104 refusals, KL 0.1383 (first-token, harmless prompts) — fewest refusals with KL ≤ 0.2
Parameters	direction index 13.54; `o_proj` weight 1.27–1.28, `down_proj` weight 1.46–1.47, peak near layer 19

Two Qwen3.5-specific details:

Thinking-mode scoring. Qwen3.5's chat template opens <think> by itself, so a naive refusal check ends up scoring the model's reasoning ("this request is harmful…") instead of its answer — a first attempt plateaued at 95/104 refusals. Directions and refusals were therefore measured after a closed, empty think block (\n</think>\n\n, i.e. Qwen's own no-thinking format), on the actual reply.
MTP head kept. Exporting through 🤗 Transformers drops Qwen3.5's mtp.* weights. The 15 original MTP tensors are added back unchanged in model-mtp.safetensors; they only drive speculative decoding (perplexity is bit-identical with and without them).

Raw benchmark scores: benchmarks/.

Limitations & responsible use

Amadeus will comply with requests the original refuses. You are responsible for how you use it and for following the laws that apply to you. Do not deploy it where users expect a safety-filtered assistant.
Removing refusals does not add knowledge: answers on dangerous or specialised topics can be wrong or outdated.
It inherits the biases and limitations of Qwen3.5-4B, and is slightly more willing to state misconceptions confidently (see TruthfulQA).