Instructions to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF
# Run inference directly in the terminal:
llama cli -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF
# Run inference directly in the terminal:
llama cli -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF
# Run inference directly in the terminal:
./llama-cli -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Use Docker

docker model run hf.co/EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

LM Studio
Jan
Ollama
How to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with Ollama:
```
ollama run hf.co/EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF
```
Unsloth Desktop

How to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with Docker Model Runner:
```
docker model run hf.co/EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF
```

Lemonade

How to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Run and chat with the model

lemonade run user.ShiftAddQwen-3.5-9B-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

ShiftAddQwen-3.5-9B-GGUF

Qwen3.5-9B, with its MTP head, quantized to 3–4.4 bits per decoder weight (ShiftAddQwen-3.5-9B-{3.0,3.2,3.6,4.0,4.4}bits) using the ShiftAddLLM weight form. Each weight group is a sum of binary planes times per-group scales.

The scales are fitted with output-aware GPTQ error feedback, bits are allocated per layer by measured KLD, and the scales are then distilled end to end. ShiftAddQwen keeps ShiftAddLLM's weight representation and replaces its fitting pipeline and kernels (the decode kernel is LUT-free: sign-select with dp4a byte masks); see docs/shiftadd_matvec.pdf for what is kept and what is replaced.

Code, method and full results: https://github.com/Purdue-EcoAI/ShiftAddQwen

These files need the patched llama.cpp. They use new ggml quant types (sa2 … sa6_256). Stock llama.cpp will refuse to load them. See Run.

Files

Only checkpoints that beat an UnslothAI/Qwen-3.5-9B-* quant of equal or larger size are published here. To qualify, a checkpoint must be smaller or equal in size, have lower KLD, and not be worse on tasks within noise.

file	GB	KLD mixed ↓	tasks ↑	decode tok/s	beats (UnslothAI/Qwen-3.5-9B-…)
`ShiftAddQwen-3.5-9B-4.4bits.gguf`	5.10	0.0303	78.2	130	UD-Q3_K_XL (5.24 GB): KLD 0.030 vs 0.036, tasks 78.2 vs 78.9, 0.14 GB smaller
`ShiftAddQwen-3.5-9B-4.0bits.gguf`	4.76	0.0384	77.4	135	Q3_K_M (4.83 GB): KLD 0.038 vs 0.067, tasks 77.4 vs 76.2
`ShiftAddQwen-3.5-9B-3.6bits.gguf`	4.41	0.0564	75.3	137	UD-Q2_K_XL (4.44 GB): KLD 0.056 vs 0.137, tasks 75.3 vs 72.8
`ShiftAddQwen-3.5-9B-3.2bits.gguf`	4.07	0.0857	74.8	143	UD-IQ3_XXS (4.30 GB): KLD 0.086 vs 0.088, tasks 74.8 vs 74.8, 0.23 GB smaller
`ShiftAddQwen-3.5-9B-3.0bits.gguf`	3.90	0.1052	73.2	145	UD-IQ2_M (3.97 GB): KLD 0.105 vs 0.150, tasks 73.2 vs 70.7

KLD: to the BF16 model on held-out mixed-domain text (web, chat, code, multilingual, math), 40×512 tokens.
Tasks: mean of GSM8K (250), MMLU-Pro (350), HumanEval (164) and MGSM (80). Greedy decoding, thinking off. Differences under ~1.5 points are noise.
Decode: llama-bench tg128 on an A100 with CUDA.

All files include the MTP head for speculative decoding. The embedding and LM head use llama.cpp k-quants.

Tool calling

The two drop-in candidates for the reference 4-bit deployment, evaluated on agentic benchmarks through the same patched llama-server (--jinja, thinking on, temperature 0.6 / top-p 0.95), against UnslothAI/Qwen-3.5-9B-Q4_K_M and -Q3_K_M. BFCL v4: accuracy (%) of the non-live (1,390 items), live (2,251) and multi-turn (800, mean of two runs) splits. τ-bench: pass^1 (%) over 2 trials of the retail (115 tasks) and airline (50 tasks) environments, with the same model (thinking off) as user simulator. ▲ / ▼ / ≈: better / worse than / within 2 points of Q4_K_M, with the per-item difference.

model	GB	BFCL non-live ↑	BFCL live ↑	BFCL multi-turn ↑	τ-bench retail ↑	τ-bench airline ↑
UnslothAI/Qwen-3.5-9B-Q4_K_M	5.87	86.9	79.7	46.9	47.4	55.0
ShiftAddQwen-3.5-9B-4.4bits	5.10	85.9 ≈−1.1	79.9 ≈+1.8	43.8 ▼−3.2	51.3 ▲+3.9	53.0 ≈−2.0
UnslothAI/Qwen-3.5-9B-Q3_K_M	4.83	86.7 ≈−1.7	79.2 ≈−0.4	43.4 ▼−3.6	50.4 ▲+3.0	46.0 ▼−9.0
ShiftAddQwen-3.5-9B-4.0bits	4.76	84.8 ≈−1.8	79.3 ≈+1.9	43.6 ▼−3.4	48.7 ≈+1.3	50.0 ▼−5.0

Against Q4_K_M, the 4.4-bit model is 1 win / 3 ties / 1 loss, Q3_K_M 1 / 2 / 2, and the 4.0-bit model 0 / 3 / 2. Single-turn tool calling and τ-bench task completion are preserved; the one consistent loss is BFCL multi-turn, about 3 points below Q4_K_M for every model below it in size, including Q3_K_M. Per-category tables, pass^2, protocol and raw numbers: docs/results.md.

Hardware compatibility

Requirements. An NVIDIA GPU and the patched llama.cpp from the repository (scripts/setup_llama.sh, CUDA toolkit 12.x). The decode kernels use dp4a, so any GPU of compute capability 6.1 or newer can run them; tested on V100 (CUDA_ARCH=70) and A100 (CUDA_ARCH=80), set CUDA_ARCH to your GPU when building. A CPU reference implementation is included (correct, but slow). Not supported: stock llama.cpp and the tools that bundle it (Ollama, LM Studio, llama-cpp-python wheels), Metal, Vulkan and ROCm backends.

GPU memory ≈ file size + KV cache + ~0.3 GB of compute buffers. Qwen3.5-9B has 8 full-attention layers (the other 24 are Gated-DeltaNet layers with a small fixed-size state), 4 KV heads and 256-dim heads, so the f16 KV cache costs 32 KB per token of context: 1 GB at 32K, 4.2 GB at 128K, 8.4 GB at the full 262K. -ctk q8_0 -ctv q8_0 halves it. Estimated GPU memory in GB by context length (measured on a V100: 13.7 GB for the 4.0-bit file at 262K, against 14.6 GB for Q4_K_M):

file	GB	8K	32K	128K	262K
`ShiftAddQwen-3.5-9B-4.4bits.gguf`	5.10	5.7	6.5	9.7	14.0
`ShiftAddQwen-3.5-9B-4.0bits.gguf`	4.76	5.3	6.1	9.4	13.6
`ShiftAddQwen-3.5-9B-3.6bits.gguf`	4.41	5.0	5.8	9.0	13.3
`ShiftAddQwen-3.5-9B-3.2bits.gguf`	4.07	4.6	5.4	8.7	13.0
`ShiftAddQwen-3.5-9B-3.0bits.gguf`	3.90	4.5	5.3	8.5	12.8

So every file fits an 8 GB GPU (RTX 3070 / 4060 / 2080) at 32K context, a 12 GB GPU (RTX 3060 12 GB / 4070) at 128K, and a 16 GB GPU (RTX 4060 Ti 16 GB / 4080 / V100 16 GB) at the full 262K; the 3.0- and 3.2-bit files fit a 6 GB GPU (RTX 2060 / 3050 / GTX 1660) at 32K. Add about 0.2 GB for the MTP draft head with --spec-type draft-mtp, and the KV cache is per slot: -np N multiplies the context term by N.

Run

git clone https://github.com/Purdue-EcoAI/ShiftAddQwen && cd ShiftAddQwen
scripts/setup_llama.sh            # llama.cpp b11538 + ShiftAdd patch, CUDA build (set CUDA_ARCH)
hf download EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF ShiftAddQwen-3.5-9B-4.0bits.gguf --local-dir models
work/llama.cpp/build/bin/llama-server -m models/ShiftAddQwen-3.5-9B-4.0bits.gguf -ngl 99 -fa on -c 32768 \
    --jinja --spec-type draft-mtp --spec-draft-n-max 3

Kernels are provided for CUDA (fast decode and prefill) and CPU (reference).

Citation

ShiftAddQwen builds on ShiftAddLLM but is not the same method; please cite both.

@misc{you2026shiftaddqwen,
  title={ShiftAddQwen: ShiftAdd reparameterization of Qwen3.5-9B for llama.cpp},
  author={You, Haoran},
  year={2026},
  howpublished={\url{https://github.com/Purdue-EcoAI/ShiftAddQwen}},
  note={Models: \url{https://huggingface.co/EcoAI-Lab/ShiftAddQwen-3.5-9B-GGUF}}
}

@inproceedings{you2024shiftaddllm,
  title={ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization},
  author={You, Haoran and Guo, Yipin and Fu, Yichao and Zhou, Wei and Shi, Huihong and Zhang, Xiaofan and Kundu, Souvik and Yazdanbakhsh, Amir and Lin, Yingyan Celine},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2024}
}