Instructions to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS
# Run inference directly in the terminal:
llama cli -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS
# Run inference directly in the terminal:
./llama-cli -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Use Docker

docker model run hf.co/ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

LM Studio
Jan
Ollama
How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with Ollama:
```
ollama run hf.co/ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS
```

Unsloth Studio

How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ByteOtter/Qwen3.5-4B-CADA-IQ2_XS to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ByteOtter/Qwen3.5-4B-CADA-IQ2_XS to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ByteOtter/Qwen3.5-4B-CADA-IQ2_XS to start chatting

How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with Docker Model Runner:
```
docker model run hf.co/ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS
```

Lemonade

How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Run and chat with the model

lemonade run user.Qwen3.5-4B-CADA-IQ2_XS-IQ2_XS

List all available models

lemonade list

Hermes Agent

How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ByteOtter/Qwen3.5-4B-CADA-IQ2_XS with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ByteOtter/Qwen3.5-4B-CADA-IQ2_XS:IQ2_XS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

Qwen 3.5 4B — reasoning-directed mixed-precision IQ2_XS GGUF

---
base_model: Qwen/Qwen3.5-4B
language:
- multilingual
pipeline_tag: text-generation
license: apache-2.0
tags:
- qwen3.5
- gguf
- llama.cpp
- mixed-precision
- task-aware-quantization
- reasoning
- cada
---

Qwen 3.5 4B — reasoning-directed mixed-precision IQ2_XS GGUF

This is an experimental, reasoning-directed mixed-precision GGUF derived from Qwen3.5-4B.

It was produced by QLAB, which measures quantization damage and redistributes precision at the tensor level under an approximately fixed byte budget.

This release is important because it is the first QLAB result outside the Gemma family.

On the reasoning evaluation, the stock IQ2_XS + category-imatrix quant scored 46.875%. The QLAB allocation using the same calibration strategy scored 54.688%.

That is:

+7.813 percentage points
+16.67% relative reasoning performance
at only +0.412% model size

The BF16 source scored 78.125%, giving this artifact 70.0% reasoning retention versus 60.0% for the stock IQ2_XS comparator.

This is a targeted quantization experiment, not a claim that the resulting model is universally better. Eight of eleven measured suites improved over the stock comparator, while knowledge QA, structured output, and coherence regressed.

See Interpretation and limitations before using this model.

Model details

Property	Value
Architecture	Qwen3.5 4B
Format	GGUF
Quantization	Per-tensor mixed precision searched around an IQ2_XS byte budget
File size	1,637,318,816 bytes (1.525 GiB)
Stock comparator	1,630,594,336 bytes (1.519 GiB)
BF16 source	8,424,393,632 bytes (7.846 GiB)
Size reduction from BF16	80.56%
Candidate vs stock size	+0.412%
Candidate SHA-256	`[CANDIDATE SHA-256]`
Stock SHA-256	`[STOCK SHA-256]`
Source SHA-256	`[SOURCE SHA-256]`
Target category	Reasoning
Base model	Qwen/Qwen3.5-4B

The upstream Qwen3.5-4B model is released under the Apache License 2.0.

What QLAB changed

This is not a uniform IQ2_XS quantization.

QLAB begins with a category-calibrated quantization and then searches for tensor-level precision reallocations intended to recover capability lost during quantization.

Some tensors are assigned more precision while others give precision back so that the resulting artifact remains near the original byte budget.

For this experiment:

Stock IQ2_XS + imatrix: 1,630,594,336 bytes
QLAB allocation + imatrix: 1,637,318,816 bytes
Difference: 6,724,480 bytes
Relative increase: 0.412%

There is no LoRA, post-training, pruning, distillation, or weight optimization involved.

The measured difference comes from changing where the quantization budget is spent.

Usage

Use a recent build of llama.cpp:

llama-cli \
  -m qwen3.5-4b-CADA-IQ2_XS.gguf \
  -cnv

Or start an OpenAI-compatible server:

llama-server \
  -m qwen3.5-4b-CADA-IQ2_XS.gguf \
  --host 127.0.0.1 \
  --port 8080

Qwen3.5 is natively multimodal upstream. This release contains the GGUF language-model artifact only unless a matching projector is explicitly included in this repository.

Multimodal behavior was not evaluated as part of this QLAB release.

Evaluation

Target result: reasoning

Artifact	Bytes	Reasoning score	Retention vs BF16
BF16 source	8,424,393,632	78.125%	100.0%
Stock IQ2_XS + imatrix	1,630,594,336	46.875%	60.0%
QLAB allocation + imatrix	1,637,318,816	54.688%	70.0%

Allocation effect

54.688 − 46.875 = +7.813 percentage points

Relative improvement over the stock comparator:

+16.67%

The candidate uses only 0.412% more bytes than the stock comparator.

Absolute scores across all measured suites

Suite	BF16	Stock IQ2_XS + imatrix	QLAB allocation + imatrix	QLAB − stock
Reasoning	78.125	46.875	54.688	+7.813
Math	32.812	4.688	13.281	+8.594
Coding	44.531	2.344	4.688	+2.344
Knowledge QA	65.625	54.688	39.062	−15.625
Instruction following	26.562	19.048	23.438	+4.390
Structured output	57.812	51.562	42.188	−9.375
Summarization / extraction	68.750	32.812	45.312	+12.500
General fidelity	36.500	30.515	30.813	+0.299
Coherence	81.250	85.938	67.188	−18.750
Stability	41.546	35.888	38.139	+2.251
Context	100.000	86.458	92.708	+6.250

The QLAB candidate improved 8 of 11 measured suites by point estimate.

What improved

The largest positive allocation effects were:

Summarization / extraction: +12.500 points
Math: +8.594
Reasoning: +7.813
Context: +6.250
Instruction following: +4.390
Stability: +2.251
Coding: +2.344
General fidelity: +0.299

Reasoning was the optimization target.

The improvements outside reasoning are secondary observations and should not be interpreted as independently optimized capabilities.

What regressed

Three measured suites scored below the stock comparator:

Knowledge QA: −15.625 points
Structured output: −9.375
Coherence: −18.750

These are meaningful tradeoffs.

This artifact should therefore be treated as a reasoning-directed specialized quant, not as a universally superior IQ2_XS replacement.

Why this result matters

Earlier QLAB experiments demonstrated allocation effects on Gemma-family models.

This run demonstrates the same basic mechanism on Qwen3.5, a different model family and architecture.

At an essentially fixed IQ2_XS storage budget, tensor-level allocation increased reasoning performance from:

46.875 → 54.688

while increasing model size by only:

0.412%

This reduces the likelihood that the allocation behavior observed in earlier experiments is specific to Gemma.

It does not establish universal transfer across architectures. Additional models and larger scales still need to be tested.

Evaluation protocol

Property	Value
Target	Reasoning
Evaluation system	MLAB
Source artifact	`[SOURCE ARTIFACT]`
Stock comparator	IQ2_XS + same category imatrix
Candidate	QLAB tensor-level allocation + same imatrix
MLAB run	`[MLAB RUN ID]`
Evaluation pack	`[PACK NAME / VERSION]`
Bootstrap resamples	`[BOOTSTRAP COUNT]`
Bootstrap seed	`[BOOTSTRAP SEED]`
MLAB report SHA-256	`[REPORT SHA-256]`
llama.cpp commit	`[LLAMA.CPP COMMIT]`

The candidate and comparator should be interpreted as a matched-budget comparison. The candidate is 0.412% larger, which remains inside QLAB's ±0.5% byte-parity window.

Interpretation and limitations

This is an experimental research artifact.

The strongest supported claim from this release is narrow:

At approximately the same IQ2_XS byte budget, QLAB tensor-level allocation improved measured reasoning performance from 46.875% to 54.688%, a +7.813-point / +16.67% relative improvement over the stock category-imatrix comparator.

Do not interpret this as:

IQ2_XS matching BF16 generally
a universally better Qwen3.5 quant
proof that the same allocation strategy works on every architecture
proof that every category benefits
evidence of multimodal preservation

Known weaknesses

Coding remains heavily degraded.

The candidate improves coding from 2.344% to 4.688%, but the BF16 source scores 44.531%. The relative improvement should not be confused with practical coding parity.

Intended use

This model is intended for:

quantization research
tensor-level mixed-precision experimentation
studying capability recovery near aggressive quantization regimes
reasoning-focused local inference where a roughly 1.5 GiB model is useful
community evaluation of QLAB/CADA

Evaluate the model against your own workload before relying on it.

It should not be used as the sole source of truth for medical, legal, financial, safety-critical, or other high-stakes decisions.

License and attribution

This derivative is distributed under the Apache License 2.0, consistent with the upstream Qwen3.5-4B release.

Upstream model:

Qwen/Qwen3.5-4B

License:

Apache License 2.0

Qwen3.5 is developed by the Qwen team.

QLAB produced the task-directed tensor-level mixed-precision allocation described in this model card.

MLAB performed the independent evaluation.

Project links

QLAB / ByteOtter updates:

https://x.com/byteotter

Support larger QLAB experiments:

https://buymeacoffee.com/byteotter

Community support goes toward compute for larger experimental runs, including Qwen 27B-class models. Funding an experiment does not guarantee a successful result. Failed, neutral, and negative experiments are part of the research process.

Citation

For the upstream model, follow the citation guidance provided by the Qwen team:

@misc{qwen3_5,
  title        = {Qwen3.5},
  author       = {{Qwen Team}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Qwen/Qwen3.5-4B}}
}

If referencing this quantization artifact specifically, please also link directly to this Hugging Face repository.

Downloads last month: -

GGUF

Model size

4B params

Architecture

qwen35

Hardware compatibility

2-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support