Instructions to use TheStageAI/Qwen3.5-2B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use TheStageAI/Qwen3.5-2B-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="TheStageAI/Qwen3.5-2B-GGUF",
	filename="Qwen3.5-2B-L-TS-Q8_0.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use TheStageAI/Qwen3.5-2B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use TheStageAI/Qwen3.5-2B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "TheStageAI/Qwen3.5-2B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "TheStageAI/Qwen3.5-2B-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Ollama
How to use TheStageAI/Qwen3.5-2B-GGUF with Ollama:
```
ollama run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
```

Unsloth Studio

How to use TheStageAI/Qwen3.5-2B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheStageAI/Qwen3.5-2B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for TheStageAI/Qwen3.5-2B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for TheStageAI/Qwen3.5-2B-GGUF to start chatting

How to use TheStageAI/Qwen3.5-2B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use TheStageAI/Qwen3.5-2B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use TheStageAI/Qwen3.5-2B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use TheStageAI/Qwen3.5-2B-GGUF with Docker Model Runner:
```
docker model run hf.co/TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M
```

Lemonade

How to use TheStageAI/Qwen3.5-2B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull TheStageAI/Qwen3.5-2B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.5-2B-GGUF-Q4_K_M

List all available models

lemonade list

TheStageAI Edge Models: the right model at every memory budget

Qwen3.5 2B

Four GGUF checkpoints for llama.cpp, from 738 MB to 2.01 GB.
Start with M: 1.07 GB and ≈100% of the BF16 instruction-strict IFEval score.

Choose a checkpoint

Tier	Size	Best for	File
XS	738 MB	Minimum footprint	Download
S	967 MB	Compact	Download
M	1.07 GB	Recommended	Download
L	2.01 GB	High-precision Q8	Download

Other Qwen 3.5 sizes: 0.8B · 4B · 9B

Quickstart

llama-cli \
  --hf-repo TheStageAI/Qwen3.5-2B-GGUF \
  --hf-file Qwen3.5-2B-M-TS-Q4_K_M.gguf

Why we recommend M

At 1.07 GB, M matches the BF16 instruction-strict IFEval score within evaluation variation. It uses 47% less disk than L.

Tier	IFEval P / I (%)	MMLU-Pro (%)
BF16 reference	65.43 / 74.70	—
L	65.80 / 74.94	—
M	66.54 / 75.54	—
S	63.22 / 73.38	—
XS	52.68 / 64.15	—

P / I means prompt-strict / instruction-strict. IFEval uses deterministic non-thinking decoding; MMLU-Pro uses sampled long-form reasoning. Only complete model-level scores are reported. A dash means not reported.

Reasoning: XS is intended for non-thinking use. Run it with --reasoning off. Choose S, M, or L for long-form reasoning.

Evaluation protocol

IFEval: 541 prompts, native chat template, enable_thinking=false, temperature 0.
MMLU-Pro: 12,032 questions, vLLM, 0-shot, native chat template qwen_mc_json_v1, enable_thinking=true; temperature=1, top_p=0.95, top_k=20, min_p=0, presence_penalty=1.5, frequency_penalty=0, repetition_penalty=1, seed=42; max_model_len=40960, max_new_tokens=32768; dataset revision b189ec765aa7ed75c8acfea42df31fdae71f97be.
The headline BF16 comparison uses instruction-strict IFEval; the raw scores are shown in the table.

In matched long-form reasoning diagnostics, XS produced longer trajectories and reached the 32,768-token output limit more often than S. For Qwen 0.8B and 2B, a complete comparable MMLU-Pro matrix was not available at release time. The table omits incomplete subject runs.

How the checkpoints are built

All four checkpoints use the same production PTQ pipeline. The precision map is the only tier-specific part.

1. Fit native GGUF codes

Calibration activations produce a curvature objective weighted by true Fisher information for each quantized projection. We adapt the scale and minimum initialization from NeUQI to that objective, then solve integer codes on the target GGUF grid with a guarded cyclic coordinate-descent solver inspired by QuantEase. A final K-quant pass tunes stored scales and minima while keeping packed codes fixed.

2. Reconstruct the deployed trajectory

Layers are calibrated in execution order against activations from the already-quantized prefix. A dense reference path measures accumulated drift. Quantization Error Propagation (QEP) adds that drift to the next reconstruction target, so later layers optimize for the inputs they receive at inference time.

3. Allocate the byte budget

XS and S can choose Q2_K through Q8_0 for each quantizable group. The optimizer trades changes in the teacher distribution against the actual encoded byte cost, including scale and minimum metadata. ANNA provides constrained configuration search. RCO provides an exact-budget route (code). For this model, ANNA selected both the XS and S precision maps. M and L keep the decoder qtypes at Q4_K_M and Q8_0, respectively, and use the same reconstruction and scale-tuning stages.

After schedule selection, PTQ runs again from the source weights. Each layer is then calibrated with the final upstream precision choices.

4. Align the full model

A short affine distillation pass tunes native FP16 scales and minima while qtypes, packed codes, dense weights, and tensor layouts stay fixed. The loss matches the teacher's next-token distribution without changing file size or runtime layout.

We load-test the shipping GGUF and evaluate it on a held-out set of 3,072 sequences with next-token KL. release-manifest.json records its SHA-256, downstream evaluation IDs, and tensor metadata. Final recommendations use complete-model benchmarks.

File details

Tier	Hub selector	GGUF file type	Whole-file BPW
XS	`Q3_K_S`	`MOSTLY_Q2_K`	3.140
S	`Q4_K_S`	`MOSTLY_Q2_K`	4.109
M	`Q4_K_M`	`MOSTLY_Q4_K_M`	4.562
L	`Q8_0`	`MOSTLY_Q8_0`	8.558

The Hub selector controls sidebar grouping and download discovery. For XS and S it approximates the whole-file size class; release-manifest.json contains the exact tensor mix. M and L keep their decoder qtypes at Q4_K_M and Q8_0 within the same production PTQ pipeline.

Runtime memory also includes KV cache and buffers, which grow with context length.

TheStageAI deployment stack

These GGUF files target llama.cpp-compatible runtimes. edge-lm runs compressed MLX models on Macs and iPhones. ANNA searches compression configurations under size or compute constraints. The TheStageAI Platform and documentation cover compression, compilation, and serving workflows.

For a specific device, latency target, or memory budget, talk to our team.

Reproducibility

Release: July 21, 2026.
Base model: Qwen/Qwen3.5-2B at revision 15852e8c.
Manifest: release-manifest.json records the exact base revision, byte sizes, GGUF file types, whole-file BPW, tensor inventories, SHA-256 digests, held-out KL values, and evaluation IDs.
Runtime gate: export and load checks used llama.cpp revision bec4772f.

Citation

If you use this checkpoint, cite this release and follow the upstream model's citation guidance:

@misc{thestageai2026qwen3p52bgguf,
  author       = {{TheStageAI}},
  title        = {Qwen3.5 2B: TheStageAI GGUF Release},
  year         = {2026},
  month        = {jul},
  howpublished = {Hugging Face model release},
  url          = {https://huggingface.co/TheStageAI/Qwen3.5-2B-GGUF},
  note         = {XS, S, M, and L deployment tiers},
}

References