Instructions to use Archsloth/Qwen3.5-9B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Archsloth/Qwen3.5-9B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Archsloth/Qwen3.5-9B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Archsloth/Qwen3.5-9B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Archsloth/Qwen3.5-9B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Ollama
How to use Archsloth/Qwen3.5-9B-GGUF with Ollama:
```
ollama run hf.co/Archsloth/Qwen3.5-9B-GGUF:Q4_K_M
```
Unsloth Desktop

How to use Archsloth/Qwen3.5-9B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Archsloth/Qwen3.5-9B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Archsloth/Qwen3.5-9B-GGUF with Docker Model Runner:
```
docker model run hf.co/Archsloth/Qwen3.5-9B-GGUF:Q4_K_M
```

Lemonade

How to use Archsloth/Qwen3.5-9B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-9B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Archsloth/Qwen3.5-9B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Archsloth/Qwen3.5-9B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Archsloth/Qwen3.5-9B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Archsloth/Qwen3.5-9B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

ARCHSLOTH — the summit of quantization

Most quantized weights ship with an adjective. Ours ship with a table.

Head to head with unsloth/Qwen3.5-9B-GGUF, same Q4_K_M rung — KL divergence from bf16, lower is better:
Korean −25.4 % · English −11.4 %
_{Two axes measured, two ahead. What we did not win is further down, with its number.}

📚 Collection

▶ Archsloth MODELs — every build in the ARCHsloth Q4 grade Qwen3-4B · Qwen3.5-9B · Qwen3.8-27B

Qwen3.5-9B · ARCHsloth Q4

Same rung. Same tensor types. Only the rounding differs.

Which file do I want?

File	Size	What it is
`Qwen3.5-9B-AS-Q4_K_M.gguf` ⭐	5.78 GB	The weights. This is the model.
`mmproj-F32.gguf`	1.82 GB	The vision tower. Only needed if you want the model to see images.

Qwen3.5-9B is a vision-language model. Text-only chat needs the first file alone; pass --mmproj mmproj-F32.gguf to llama-server to turn the eyes on. The vision tower is not quantized — it is carried at full precision, exactly as the base model shipped it.

`[measured]` Q4_K_M — the same rung, both sides

KL divergence from the bf16 original. Lower is better. llama-perplexity --kl-divergence, ctx 512, 60 chunks, held-out Korean and English text that is not in the calibration set.

KL divergence, ARCHsloth against unsloth, Korean and English

	bytes	Korean	English
ARCHsloth `Q4_K_M`	5,780,090,240	0.019255	0.019237
unsloth `Q4_K_M`	5,680,522,464	0.025821	0.021718
unsloth `UD-Q4_K_XL`	5,966,095,584	0.013740	0.012546
stock `llama-quantize`	5,780,090,304	0.043475	0.037879

Against	Korean	English
unsloth `Q4_K_M` — same rung	−25.4 %	−11.4 %
stock `llama-quantize`	−55.7 %	−49.2 %

Same-top-p moves the same direction on every row (93.974 % vs 92.784 % on Korean). Full statistics: EVAL.md. Raw per-run logs: eval/logs/.

Quickstart

# text only
llama-cli -m Qwen3.5-9B-AS-Q4_K_M.gguf -p "안녕하세요" -ngl 0 -t 8

# OpenAI-compatible server, with vision
llama-server -m Qwen3.5-9B-AS-Q4_K_M.gguf --mmproj mmproj-F32.gguf \
  --host 127.0.0.1 --port 8080 --jinja

No fork. No custom format. The layout is llama.cpp's own, which is the point.

What is actually different

--scheme GGUF:Q4_K_M     optimize the rounding for the quantizer that is used at export
--enable_alg_ext         sign-gradient rounding search (SignRoundV2, arXiv:2512.04746)
--dataset cal_archsloth.jsonl
--iters 200  --nsamples 128

Nothing else. Bit widths and tensor types are llama.cpp's standard Q4_K_M. Nothing below 4 bits anywhere.

The calibration set — it ships in this repo

cal_archsloth.jsonl, 200 samples of exactly 9,000 characters each, Korean and English interleaved at the sample level. Interleaving is what mattered, not the ratio.

This is the lever. The same model built with an English-only corpus scored 0.027336 on Korean — worse than the file we are comparing against. Same code, same flags, same bit widths; only the calibration text changed, and it moved Korean by 29.6 %. That build is not in this repository. Details: CALIBRATION.md.

What we did not win

[measured] unsloth's UD-Q4_K_XL is ahead of this file. 0.013740 Korean and 0.012546 English against our 0.019255 / 0.019237 — they are 40.1 % and 53.3 % better. It is a 3.2 % larger file with a different tensor map, and on this model that map is worth more than our rounding search. We publish the row because a number you omit is a number someone else produces. Closing it is the open work on this model.
[measured] Two axes, not ten. Korean and English are measured here. The ten-axis sweep that the 4B card carries has not been run on this model. We publish only what we ran ourselves.
[pending] Q6_K and Q8_0 rungs, long-context retrieval, agentic tool use, and the vision axis. Not measured on this model.

Reproduce this

# 1. the comparison file, straight from its own repository
hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-Q4_K_M.gguf --local-dir rival

# 2. the reference the distance is measured from
python llama.cpp/convert_hf_to_gguf.py Qwen/Qwen3.5-9B --outtype bf16 --outfile ref-BF16.gguf

# 3. teacher logits, once per evaluation text
llama-perplexity -m ref-BF16.gguf -f eval/eval_ko.txt \
  --kl-divergence-base base_ko.dat -ngl 99 -c 512 --chunks 60

# 4. every candidate against the same teacher, same text, same chunks
llama-perplexity -m <candidate>.gguf -f eval/eval_ko.txt \
  --kl-divergence --kl-divergence-base base_ko.dat -ngl 99 -c 512 --chunks 60

If a number here does not replicate on your hardware, open a discussion and we will correct it.

Lineage

Component	Origin
Weights	Qwen/Qwen3.5-9B, unmodified — quantized from the original, not from anyone else's quantization
Layout, bit widths, tensor types	llama.cpp standard `Q4_K_M`, unchanged
Vision tower	carried at full precision, not quantized
Runtime	upstream llama.cpp, unmodified
Rounding search	Intel AutoRound with `enable_alg_ext` (SignRoundV2)
Calibration corpus, evaluation set, and the measurement	ours — all three ship in this repo

License

Apache-2.0, inherited from the base model. Comparison files are named so you can fetch the exact bytes we measured.

Downloads last month: -

GGUF

Model size

9B params

Architecture

qwen35

Hardware compatibility

4-bit

Model tree for Archsloth/Qwen3.5-9B-GGUF

Base model

Qwen/Qwen3.5-9B-Base

Finetuned

Qwen/Qwen3.5-9B

Quantized

(515)

this model

Collection including Archsloth/Qwen3.5-9B-GGUF

Archsloth MODELs

Collection

3 items • Updated about 22 hours ago • 30

Paper for Archsloth/Qwen3.5-9B-GGUF

SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs

Paper • 2512.04746 • Published Dec 4, 2025 • 17

📚 Collection

Qwen3.5-9B · ARCHsloth Q4

Same rung. Same tensor types. Only the rounding differs.

Which file do I want?

[measured] Q4_K_M — the same rung, both sides

Quickstart

What is actually different

The calibration set — it ships in this repo

What we did not win

Reproduce this

Lineage

License

Model tree for Archsloth/Qwen3.5-9B-GGUF

Collection including Archsloth/Qwen3.5-9B-GGUF

Paper for Archsloth/Qwen3.5-9B-GGUF

`[measured]` Q4_K_M — the same rung, both sides