ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit

Qwen3.6-35B-A3B CompQuant-MLX-3bit

Post-training, training-free compression of Qwen/Qwen3.6-35B-A3B for Apple MLX. Combines 3-bit affine quantization, bias compensation, and Ban&Pick expert routing into a single pipeline.

Text-only model. The vision tower from the base model was removed during compression. For multimodal (image+text) inference, use mlx-community/Qwen3.6-35B-A3B-4bit instead.

TL;DR: 71.9 GB → 15 GB (4.8× compression). Perplexity 8.65. GSM8K 35%. Runs on Apple Silicon with ≥16 GB unified memory.

Key Features

• 4.8× compression — 71.9 GB FP16 → 15 GB 3-bit
• Training-free — no fine-tuning, no gradient updates, calibration data only
• Bias compensation — per-layer output error correction (arXiv:2404.01892)
• Ban&Pick routing — smarter expert selection for quality + speed (arXiv:2509.06346)
• All 256 experts retained — no pruning, no expert dropping
• MLX-native — runs directly on Apple Silicon with mlx-lm
• Thinking mode — supports enable_thinking for reasoning chains

---

Model Summary

Property	Value
Base model	Qwen/Qwen3.6-35B-A3B
Architecture	MoE (Mixture-of-Experts), 35B total / 3B active
Quantization	3-bit affine (group_size=64)
Original size (FP16)	71.90 GB
Compressed size	15 GB
Compression ratio	4.8×
Bits per weight	~3.0
Experts	256 per layer (all retained)
Framework	mlx-lm ≥ 0.31.0
Target platform	Apple Silicon (M-series)

Architecture Details

Property	Value
Total parameters	35B
Active parameters per token	3B
Layers	40
Experts per layer	256 (8 routed per token)
Hidden dimension	2048
Attention heads	16 (GQA: 2 KV heads)
Context length	262,144 tokens
Attention type	DeltaNet (linear attention)
Vocab size	248,320

---

Hardware Requirements

Mac Chip	Unified Memory	Status
M4 (24 GB+)	24 GB+	✅ Recommended
M3 Pro (18 GB)	18 GB	⚠️ Tight — short prompts only
M3 Max (36 GB+)	36 GB+	✅ Comfortable
M2 Pro (16 GB)	16 GB	⚠️ Minimal — may OOM with long context
M2 Max (32 GB+)	32 GB+	✅ Comfortable
M1 (16 GB)	16 GB	⚠️ Minimal — may OOM

Model weights occupy ~15 GB. Remaining memory needed for KV cache + activations. Long context (32K+ tokens) requires ≥24 GB unified memory.

---

Compression Pipeline

Four-stage post-training pipeline. No fine-tuning, no gradient updates — all stages run on calibration data only.

┌─────────────────────────────────────────────────────────────────┐
│                   FP16 Base Model (71.9 GB)                      │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  Stage 0: Expert Weight Tying                                    │
│  Cosine similarity > 0.85 → tie experts                          │
│  Result: 0 pairs tied (experts distinct)                         │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  Stage 1: 3-bit Affine Quantization (group_size=64)              │
│  Expert FFN + Attention → 3-bit                                  │
│  Embeddings/LM head → 8-bit · Router/DeltaNet → FP16             │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  Stage 2: Bias Compensation                                      │
│  Per-layer bias = argmin ||W_fp·x - (W_q·x + b)||²              │
│  39/40 layers compensated (8 calibration samples)                │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│  Stage 3: Ban&Pick Expert Routing                                │
│  Ban bottom 10% · Boost top 10% by +0.3                          │
│  All 40 layers, 256 experts retained                             │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────────┐
│              Compressed Model (15 GB, 4.8× smaller)              │
└─────────────────────────────────────────────────────────────────┘

Stage 0: Expert Weight Tying

Pairwise cosine similarity across all 40 layers (256 experts each). Layers with similarity > 0.85 are tied to reduce storage. For this model, no pairs exceeded the threshold — experts are sufficiently distinct, so no tying was applied.

Technical details

• Similarity metric: cosine similarity on flattened expert weight matrices
• Threshold: 0.85
• Layers scanned: 40 (all)
• Pairs found: 0
• Related work: Tying the Loop — Tied Expert Layers in MoE, MoRE: Mixture of Reused Experts

Stage 1: 3-bit Affine Quantization

MLX-native 3-bit affine quantization with group_size=64. Applied to all linear layers except DeltaNet attention.

Layer type	Bits	Group size	Notes
Expert FFN (gate_proj, up_proj, down_proj)	3	64	All 256 experts per layer
Attention (Q, K, V, O)	3	64	Standard attention layers
Embeddings & LM head	8	—	Higher precision for vocab
Router / gate	16 (FP16)	—	Must stay precise for routing
DeltaNet (linear_attn)	16 (FP16)	—	MLX lacks quantized matmul for this op

Stage 2: Bias Compensation

Per-layer bias vectors computed by comparing FP16 vs quantized activations on calibration data. A bias vector is added to each layer's output to minimize the quantization-induced output error — a convex optimization that requires no gradient updates.

• Calibration samples: 8
• Bias vectors applied: 39 / 40 layers
• Layer 0 skipped: DeltaNet reshape incompatibility

Based on Minimize Quantization Output Error with Bias Compensation (Gong et al., 2024).

Method details

For each quantized layer l, compute:

b_l = argmin_b || W_fp16 @ x - (W_quant @ x + b) ||²

where x are activations from calibration data. This is a least-squares problem with a closed-form solution. The bias b_l is stored in FP16 and added at inference time.

Code: github.com/GongCheng1919/bias-compensation

Stage 3: Ban&Pick Expert Routing

Profiles expert activation frequency on calibration data. Bans the bottom 10% least-used experts (sets their routing weight to −∞) and boosts the top 10% by +0.3. This improves both quality (concentrates compute on useful experts) and speed (fewer expert evaluations).

• Ban threshold: bottom 10%
• Pick boost: +0.3 (top 10%)
• Applied to: all 40 layers
• Experts retained: 256 (banning affects routing, not storage)

Based on Ban&Pick: Achieving Free Performance Gains and Inference Speedup via Smarter Routing in MoE-LLMs (Chen et al., 2025).

---

Evaluation Results

Evaluated on Apple Silicon M4 with mlx-lm.

Metric	Value
Perplexity	8.654
GSM8K accuracy	35% (7/20)
Model size	15 GB
Compression ratio	4.8×

Evaluation details

• GSM8K: 20 problems, max_tokens=512, chat template, step-by-step prompting. Answer extracted via regex on final number in response.
• Perplexity: 10 diverse English texts (~1,000 tokens total). Computed as exp(mean(cross_entropy_loss)) over all tokens.
• Hardware: Apple M4, mlx-lm
• Note: GSM8K sample size is small (20). The model is a "thinking" model that generates long reasoning chains; answer extraction may occasionally miss the final number.

---

Comparison

Model	Size	Bits/w	GSM8K	Perplexity	Notes
unsloth/Qwen3.6-35B-A3B-UD-MLX-3bit	17.4 GB	~3.5	—	—	Unsloth Dynamic
This model	15 GB	~3.0	35%	8.654	bias comp + Ban&Pick, all 256 experts
ala-la/Qwen3.6-35B-A3B-MLX-3bit	11 GB	3.51	—	—	192 experts (25% pruned)
mlx-community/Qwen3.6-35B-A3B-4bit	20.4 GB	~4.0	—	—	Standard mlx-vlm conversion

Key tradeoffs:

• vs mlx-community 4-bit: 26% smaller, 3-bit vs 4-bit (some quality loss expected)
• vs ala-la 3-bit (pruned): 36% larger but retains all 256 experts (no pruning) + bias compensation + Ban&Pick routing
• vs unsloth 3-bit UD: 14% smaller, uses additional bias compensation and expert routing optimization

---

Usage

Installation

pip install mlx-lm

Basic generation

from mlx_lm import load, generate

model, tokenizer = load("ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit")

prompt = "Explain quantum computing in simple terms."
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

Chat mode

from mlx_lm import load, generate

model, tokenizer = load("ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the square root of 144?"}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

CLI

mlx_lm.generate --model ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit --prompt "Hello!" --max-tokens 256

Thinking Mode

Qwen3.6 is a "thinking" model — it generates reasoning chains before answering. Control this via enable_thinking:

from mlx_lm import load, generate

model, tokenizer = load("ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit")

messages = [{"role": "user", "content": "What is 15 * 17?"}]

# Thinking ON (default) — model reasons step by step
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    chat_template_kwargs={"enable_thinking": True}
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=2048)

# Thinking OFF — direct answer, faster
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    chat_template_kwargs={"enable_thinking": False}
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)

For math/reasoning tasks, keep thinking ON (max_tokens ≥ 2048). For simple Q&A, thinking OFF is faster and uses less memory.

OpenAI-compatible server

mlx_lm.server --model ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit

Then call from any OpenAI-compatible client:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit", "messages": [{"role": "user", "content": "Hello!"}]}'

---

Use Case Recommendations

Use case	Thinking	max_tokens	Notes
Math / reasoning	ON	2048+	Let model reason fully
Code generation	ON	2048+	Better with reasoning
Quick Q&A	OFF	512	Faster, less memory
Chat / conversation	OFF	512	Natural conversation flow
Long document analysis	OFF	4096	Save tokens for context
Creative writing	ON	2048+	Reasoning helps coherence

---

Limitations

• Text-only: Vision tower removed during compression — model cannot process images
• GSM8K sample size: 20 problems — small sample, confidence interval is wide
• DeltaNet layers not quantized: MLX lacks quantized matmul for linear attention architecture; these layers remain FP16
• Bias compensation on layer 0: Skipped due to DeltaNet reshape incompatibility
• Weight tying: Found 0 eligible pairs (experts are distinct) — stage effectively no-op for this model
• Thinking model: Generates long reasoning chains; answer extraction may miss final number in some cases
• No full benchmark suite: MMLU, HumanEval, HellaSwag etc. not yet evaluated

---

Reproducibility

Pipeline configuration

Stage 0: Expert Weight Tying
  - similarity_threshold: 0.85
  - metric: cosine
  - result: 0 pairs tied

Stage 1: 3-bit Affine Quantization
  - bits: 3
  - group_size: 64
  - quantized: expert FFN, attention Q/K/V/O
  - kept FP16: router/gate, DeltaNet linear_attn
  - kept 8-bit: embeddings, LM head

Stage 2: Bias Compensation
  - calibration_samples: 8
  - layers_compensated: 39/40
  - layer_0: skipped (DeltaNet reshape)

Stage 3: Ban&Pick
  - ban_threshold: 0.10 (bottom 10%)
  - pick_boost: 0.30 (top 10%)
  - applied_layers: 40 (all)

Reproduce evaluation

pip install mlx-lm

python eval.py --model ala-la/Qwen3.6-35B-A3B-CompQuant-MLX-3bit \
  --gsm8k --gsm8k-samples 20 \
  --perplexity --perplexity-texts 10

GSM8K prompt format:

{question}
Let's think step by step.

Perplexity: exp(mean(cross_entropy_loss)) over 10 diverse English texts.

---

References

Methods used in this pipeline:

1. Bias Compensation — Gong et al., "Minimize Quantization Output Error with Bias Compensation," CAAI AIR 2024. arXiv:2404.01892 · Code

2. Ban&Pick — Chen et al., "Ban&Pick: Achieving Free Performance Gains and Inference Speedup via Smarter Routing in MoE-LLMs," 2025. arXiv:2509.06346

3. Expert Weight Tying (related) — Jaggi et al., "Tying the Loop — Tied Expert Layers in Mixture-of-Experts Language Models," 2026. arXiv:2606.16825

4. MLX — Apple Machine Learning Research, mlx-lm

5. Base model — Qwen/Qwen3.6-35B-A3B

BibTeX

@article{gong2024biascomp,
  title={Minimize Quantization Output Error with Bias Compensation},
  author={Gong, Cheng and Zheng, Haoshuai and Hu, Mengting and Lin, Zheng and Fan, Deng-Ping and Zhang, Yuzhi and Li, Tao},
  journal={CAAI Artificial Intelligence Research},
  year={2024},
  doi={10.26599/AIR.2024.9150036}
}

@article{chen2025banpick,
  title={Ban\&Pick: Achieving Free Performance Gains and Inference Speedup via Smarter Routing in MoE-LLMs},
  author={Chen, Yuanteng and Wang, Peisong and Shao, Yuantian and Cheng, Jian},
  year={2025},
  eprint={2509.06346},
  archivePrefix={arXiv}
}

@article{jaggi2026tyingloop,
  title={Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models},
  author={Jaggi, Martin and others},
  year={2026},
  eprint={2606.16825},
  archivePrefix={arXiv}
}

---

License

Apache 2.0 — inherited from base model.

Scientific References

1. Frantar, E., et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." ICLR 2023. — Group-wise weight quantization with second-order error compensation.

2. Lin, J., et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." MLSys 2024. — Activation-aware weight scaling to protect salient channels.

3. Dettmers, T., et al. "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale." NeurIPS 2022. — Mixed-precision decomposition for 8-bit quantization.

4. Xiao, G., et al. "SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models." ICML 2023. — Activation smoothing for outlier mitigation.

5. Ashkboos, S., et al. "QuaRot: Outlier-free 4-bit Quantization of Large Language Models." arXiv 2403.02747. — Hadamard-based rotation to remove outlier features.

6. Chee, J., et al. "QuIP: 2-Bit Quantization of Large Language Models with Incoherence Processing." arXiv 2307.13304. — Incoherence processing and vector quantization.

7. Frantar, E. & Alistarh, D. "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot." ICML 2023. — One-shot pruning with second-order information.

8. Fedus, Z., et al. "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." JMLR 2022. — MoE architecture and load balancing.

9. Jiang, A.Q., et al. "Mixtral of Experts." arXiv 2401.04088. — Sparse MoE inference and expert routing.

10. Rajbhandari, S., et al. "DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training." MLSys 2022. — MoE system optimization.

11. Xue, F., et al. "HARC: Hessian-Aware Router Calibration for Mixture-of-Experts." — Router recalibration after pruning.

12. Liu, Z., et al. "LLM-QAT: Data-Free Quantization Aware Training of Large Language Models." arXiv 2405.06031. — Calibration data generation for quantization.