Qwen3.5-397B-911MTP

A 4-bit quantized Qwen3.5-397B-A17B Mixture-of-Experts model with an integrated Multi-Token Prediction (MTP) drafter for speculative decoding on Apple Silicon. Built for oMLX VLM MTP inference.

Architecture

	Base Model	MTP Drafter
Layers	60 MoE decoder layers	1 MoE decoder layer (cloned from layer 59)
Hidden Size	4096	4096 (shared)
Attention	Hybrid: GatedDeltaNet (linear) + full attention in ~3:1 pattern	Full attention only (1 layer)
Experts	512 total, 10 active per token	512 total, 10 active per token
Vocab	248,320	Shared embed_tokens + lm_head from base
Quantization	Uniform 4-bit affine (group_size=64)	Dynamic precision: 2-bit MoE experts, 4-bit attention, bfloat16 fc/norms
Size on Disk	~209 GB (91 shards)	~2.15 GB (1 shard)
Total	~211 GB unified memory

The MTP drafter predicts the next token by fusing the base model's pre-norm hidden state with the current token embedding through a projection layer (fc), then running a single full MoE decoder layer and projecting through the shared LM head. During inference, the drafter proposes candidate tokens that the base model verifies in a single forward pass, accepting correct predictions for free.

Key Features

• Speculative Decoding: The integrated MTP drafter generates candidate tokens at a fraction of the base model's cost. Accepted tokens skip full 60-layer verification entirely.
• Dynamic Precision Drafter: MoE expert weights are aggressively compressed to 2-bit while attention projections remain at 4-bit and the fusion layer (fc) stays at full bfloat16 precision. This reduces drafter memory bandwidth by 43%. The quality impact of the 2-bit expert requantization has not been measured with a controlled before/after accept-rate comparison.
• Streak-Weighted Self-Distillation: The drafter was trained on real hidden states extracted from the quantized base model using a streak-weighted cross-entropy loss (weight = 1/(k+1)) that prioritizes first-token accuracy -- a rejection at position 0 wastes all subsequent draft compute.
• Apple Silicon Optimized: Designed for unified memory systems (512 GB recommended). The base model and drafter share embed_tokens and lm_head via oMLX's bind() mechanism, avoiding duplicate storage of the 1 GB LM head.

File Structure

.
|-- config.json                         # Base model config (mtp_num_hidden_layers=1)
|-- generation_config.json
|-- model.safetensors.index.json        # Weight map for 91 shards
|-- model-00001-of-00091.safetensors
|-- ...
|-- model-00091-of-00091.safetensors
|-- model-mtp.safetensors               # Lightning MTP shard (language_model.mtp.* prefix)
|-- tokenizer.json
|-- tokenizer_config.json
|-- vocab.json
|-- preprocessor_config.json
|-- processor_config.json
|-- video_preprocessor_config.json
+-- drafter/                            # Standalone VLM MTP drafter
    |-- config.json                     # Drafter config (model_type=qwen3_5_mtp, block_size=2)
    |-- model.safetensors               # 2.15 GB mixed-precision weights (format=mlx)
    |-- tokenizer.json
    |-- tokenizer_config.json
    +-- vocab.json

Usage with oMLX

VLM MTP Mode (Recommended)

Add to ~/.omlx/model_settings.json:

{
  "models": {
    "Qwen3.5-397B-911MTP": {
      "vlm_mtp_enabled": true,
      "vlm_mtp_draft_model": "/path/to/this/repo/drafter",
      "vlm_mtp_draft_block_size": 2,
      "mtp_enabled": false,
      "turboquant_kv_enabled": false,
      "dflash_enabled": false,
      "specprefill_enabled": false
    }
  }
}

Lightning MTP Mode (Alternative, Text-Only)

The model-mtp.safetensors shard enables built-in Lightning MTP without an external drafter. Note: model_type_override: "llm" drops the vision tower -- this mode is text-only and does not support image/video inputs.

{
  "models": {
    "Qwen3.5-397B-911MTP": {
      "model_type_override": "llm",
      "mtp_enabled": true,
      "vlm_mtp_enabled": false
    }
  }
}

Block Size Tuning

Acceptance Rate	Recommended block_size
< 50%	2 (1 draft token)
50-80%	3 (2 draft tokens)
> 80%	4 (3 draft tokens)

Monitor acceptance in oMLX logs: MTP[0] ... accept=X/Y (Z%)

Training Details

Base Model Quantization

• Source: Qwen/Qwen3.5-397B-A17B (8-bit MLX)
• Method: Uniform 4-bit affine quantization with group_size=64
• LoRA router fine-tuning: 4.42M params, fused after training

MTP Drafter Distillation

1. Hidden State Collection: 500 sequences from flytech/python-codes-25k processed through the full 4-bit base model. Pre-norm hidden states (layer 59 output before final RMSNorm) captured at every position.
2. Architecture: Single MoE decoder layer cloned from layer 59, with a learned fusion projection (fc: Linear(8192 -> 4096)) initialized as hidden-state pass-through.
3. Training: LoRA rank-16 on fc, q_proj, k_proj, v_proj, o_proj. MoE forward pass preserved with stop_gradient at MLP boundary for non-differentiable router. Inf/NaN filtering for 4-bit quantization outliers.
4. Loss: Streak-weighted cross-entropy: loss_k = CE(logits_k, target_k) / (k + 1), prioritizing early-position accuracy.
5. Dynamic Precision: Post-training, MoE expert weights re-quantized from 4-bit to 2-bit. Attention and fusion layer preserved at higher precision.

Hardware Requirements

Configuration	Memory
Base model only	~209 GB
Base + VLM MTP drafter	~211 GB
Minimum recommended	512 GB unified memory (M4 Ultra / M3 Ultra)

Benchmarks

Intelligence: MMLU (0-shot, Chain of Thought)

Overall accuracy: 91.3% (913/1000) with internal CoT reasoning enabled.

Category	Accuracy	Notes
Abstract Algebra	100%	Perfect across all STEM/logic categories
College Mathematics	100%
College Physics	100%
Machine Learning	100%
Conceptual Physics	100%
Medical Genetics	100%
Human Sexuality	100%
Prehistory	100%
Sociology	100%
22 categories total at 100%
Security Studies	70.6%	Factual retrieval weakness
College Chemistry	71.4%
Electrical Engineering	60.0%
Anatomy	60.0%
Virology	50.0%
Global Facts	28.6%	Lowest — niche memorization

Strengths: STEM, logic, mathematics, and reasoning-heavy domains benefit from CoT architecture. 22 categories scored 100%.

Weaknesses: Factual retrieval and niche memorization (Global Facts, Virology, Anatomy). CoT reasoning is applied uniformly — spending 60+ seconds reasoning about a statistic does not overcome missing training data.

Performance: oMLX Inference (Apple Silicon, 512 GB)

Benchmarked on oMLX with Python code context, MTP disabled (baseline).

Test	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak Memory
pp1024 / tg128	480.9	38.6	2.13s	209.7 GB
pp4096 / tg128	540.3	38.3	7.58s	211.5 GB

Continuous batching (pp1024/tg128):

Batch	Decode (tok/s)	Scaling
1x	38.6	1.00x
2x	61.5	1.59x
4x	84.3	2.18x

Key observations:

• 38.6 tok/s single-request decode (26ms per token) — bandwidth-bound on unified memory
• Prefill scales well with length (480 → 540 tok/s) due to better GPU utilization
• Decode stable across context lengths — no KV cache degradation at 4K
• Memory headroom: ~300 GB available for KV cache and activations
• Batch scaling sublinear (2.18x at batch 4) — approaching memory bandwidth ceiling

Intelligence vs. Performance Trade-off

Metric	Value
MMLU accuracy	91.3%
Avg time per MMLU question	33.5 seconds
Total MMLU eval time	33,502 seconds (~9.3 hours)
Fastest question	10.7s (HS Geography)
Slowest question	75.0s (Global Facts — answered incorrectly)

The CoT architecture drives high accuracy on reasoning tasks but applies uniformly to all questions, including factual recall where extended reasoning provides no benefit. This represents a latency-accuracy trade-off inherent to the thinking mode, not a quantization artifact.

Known Limitations

1. Drafter training data is narrow. The MTP drafter was distilled on 500 sequences from flytech/python-codes-25k. Expect strong draft acceptance on Python/code completions but lower acceptance on chat, tool-use JSON, Chinese text, and vision-prefixed sequences. A broader calibration corpus covering diverse domains would improve generalization.

2. MTP accept rates unpublished. Base model MMLU and inference speed benchmarks are provided above. MTP drafter acceptance rates and speculative decode speedups have not yet been measured. Community benchmarks are welcome.

3. 2-bit expert requantization is unmeasured. The drafter's MoE experts were re-quantized from 4-bit to 2-bit post-training for bandwidth reduction. No controlled accept-rate comparison between the 4-bit and 2-bit drafter versions has been conducted.

4. Hybrid attention architecture. The base model uses Qwen3.5's GatedDeltaNet + full attention hybrid pattern (~3:1 ratio). These layers are preserved as-is from the 8-bit MLX source. The MTP drafter uses full attention only (1 layer cloned from a full-attention layer 59).

5. oMLX-specific. This is an oMLX inference artifact. The base 4-bit model loads in standard mlx-lm, but the MTP drafter integration (VLM MTP and Lightning MTP) requires oMLX's speculative decoding plumbing. Users with NVIDIA/AMD GPUs should use official Qwen3.5 + vLLM/SGLang MTP instead.

Compatibility

Runtime	Base Model	VLM MTP Drafter	Lightning MTP
oMLX (recommended)	Yes	Yes	Yes (text-only)
mlx-lm	Yes (no MTP)	No	No
vLLM / SGLang	No (use official Qwen3.5)	No	No
llama.cpp / GGUF	No	No	No

Model Specifications

• Parameters: 397B total, ~17B active per token
• Context Length: 262,144 tokens
• Attention: Hybrid GatedDeltaNet (linear, ~3:1) + full attention; 32 query heads, 2 KV heads (16:1 GQA ratio)
• RoPE: Multimodal RoPE (mrope_interleaved, sections [11, 11, 10]), rope_theta=10M, partial_rotary_factor=0.25
• Norm: RMSNorm with QK-Norm on attention
• MoE: 512 experts, top-10 routing, 1 shared expert with gating

Citation

Based on Qwen3.5 by Alibaba Cloud.

MTP drafter training informed by:

• FastMTP (arXiv:2509.18362)
• MTP-D Self-Distillation (arXiv:2603.23911)
• SpecDiff-2 Streak Distillation (arXiv:2511.00606)

License

Apache 2.0 (following the Qwen3.5 license)

---

Acknowledgments

Model creator: Qwen Original model: Qwen3.5-397B-A17B 8-bit MLX quantization: provided by LM Studio team using mlx_vlm 4-bit re-quantization + MTP drafter: apolloransom

🙏 Special thanks to the Apple Machine Learning Research team for creating MLX.

👾 The 8-bit base weights were sourced from the LM Studio Community models program. Join the conversation on Discord.