Swift-Qwen3.8-27b-oQ6-mtp — mixed 6/8-bit MLX quant of Swift-Qwen3.8-27b (MTP head kept)

Unofficial Apple Silicon quantization of ukisai/Swift-Qwen3.8-27b, produced with oMLX 0.6.4 (MLX affine quantizer, group size 64) on an Apple M5 Max / 128 GB.

Two things are preserved on purpose:

• Swift weights. This variant quantizes the upstream Swift checkpoint; task-level quality after quantization has not been measured here.
• The MTP head. The -mtp suffix means the multi-token-prediction head from the base checkpoint ships intact (29 tensors, mtp_num_hidden_layers: 1), so oMLX can run self-speculative decoding instead of wasting the weights.

I am not affiliated with UkisAI. All upstream weights, benchmarks and license terms belong to UkisAI, and the upstream license governs this repository too (see License).

Format note: these are MLX safetensors, not GGUF. They will not load in llama.cpp / Ollama / LM Studio. Use oMLX or MLX runtimes.

Pick a variant

	oQ4-mtp	oQ6-mtp	oQ8-mtp
Weights on disk	15.81 GiB (16.97 GB), 4 shards	22.09 GiB (23.72 GB), 5 shards	27.94 GiB (30.00 GB), 6 shards
Weight precision	mixed 4/5-bit	mixed 6/8-bit	uniform 8-bit
Quantization	affine, group size 64	affine, group size 64	affine, group size 64
Choose when	prioritize memory headroom	balance size and weight precision	prioritize weight precision and have more memory

All three retain the MTP head. Task-level accuracy and speed after quantization have not been measured here. Runtime memory also depends on context length and KV cache.

What is inside (read straight from the shipped config.json)

Field	Value
Architecture	Qwen3_5ForConditionalGeneration (model_type: qwen3_5)
Parameters	27.78 B total, 27.27 B quantized (98.1%)
Text layers / hidden	64 layers, hidden_size 5120, intermediate_size 17408
Attention	hybrid: 1 full-attention layer every 4 (full_attention_interval: 4), 24 heads / 4 KV, head_dim 256, attn_output_gate: true; the rest are gated linear-attention (linear_attn)
Context	max_position_embeddings: 262144
Vocab	248,320 (tokenizer and chat_template.jinja copied from upstream, unchanged)
MTP	mtp_num_hidden_layers: 1, mtp_use_dedicated_embeddings: false, weights included
Vision	Qwen vision tower kept in BF16 (~0.92 GB), depth 27, patch 16, spatial merge 2
Metadata	{"format": "mlx"} in every safetensors header

Quantization recipe

Precision is mixed per module and recorded verbatim in config.json → quantization_config, so any MLX loader reproduces the layout without guessing:

• default: mixed, group_size: 64, mode: affine
• 472 modules @ 6-bit + 33 modules bumped to 8-bit
• bumped modules: late layers and embed_tokens (linear_attn.in_proj_a/b, self_attn.q/k/v/o_proj, mlp.gate/up/down_proj)
• never quantized: vision tower (BF16), all scales/biases (1.68 GB BF16), norms, A_log, dt_bias, convolutions (≈5 MB), MTP non-linear weights (128 MB)

Requirements

• Apple Silicon, macOS 15+ (built and smoke-tested on M5 Max, 128 GB)
• oMLX ≥ 0.6.4, or a recent mlx / mlx-lm / mlx-vlm build with qwen3_5 support

Usage

oMLX (the runtime these were made for)

# 1. drop the folder into the oMLX model dir
git clone https://huggingface.co/TokenAI-zer/Swift-Qwen3.8-27b-oQ6-mtp ~/.omlx/models/Swift-Qwen3.8-27b-oQ6-mtp

# 2. start the multi-model server (model id = folder name)
omlx serve --model-dir ~/.omlx/models --port 8000

# 3. talk to it
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Swift-Qwen3.8-27b-oQ6-mtp",
       "messages": [{"role": "user", "content": "Explain speculative decoding in two sentences."}]}'

In oMLX model settings, enable the speculative head and the matching reasoning parser:

{
  "mtp_enabled": true,
  "reasoning_parser": "qwen_3_5",
  "max_context_window": 262144,
  "model_type_override": "vlm"
}

MLX directly

pip install -U mlx-lm mlx-vlm
python -m mlx_lm.server --model TokenAI-zer/Swift-Qwen3.8-27b-oQ6-mtp --port 8000

Only oMLX 0.6.4 is verified by me; if you get mlx_lm running this architecture, please open an issue and I will document it.

Not supported

llama.cpp, GGUF, vLLM and SGLang paths in the upstream card do not apply here — this repo has no GGUF and no PyTorch weights. For BF16/server deployments use ukisai/Swift-Qwen3.8-27b.

Recommended sampling

Shipped generation_config.json (unchanged from upstream) is the tuning target for thinking mode:

temperature 1.0 · top_p 0.95 · top_k 20 · min_p 0.0 · repetition_penalty 1.0
eos_token_id [248046, 248044]

The upstream chat template supports tool calling, image/video inputs and an enable_thinking switch, so you can trade reasoning length per request; upstream reports Swift's token savings hold at xhigh, medium and low reasoning effort.

Benchmarks

I publish no numbers I have not measured myself. This table is the honest state of the repository:

Benchmark	oQ4-mtp	oQ6-mtp	oQ8-mtp	BF16 upstream (reference)
GPQA-Diamond	not measured	not measured	not measured	88.28%
AIME 2026	not measured	not measured	not measured	94.00%
LiveCodeBench v6	not measured	not measured	not measured	81.55%
IFBench	not measured	not measured	not measured	71.80%

Upstream Swift vs Qwen3.8-27B results (GPQA-Diamond −0.1 pt for ~41% fewer mean thinking tokens, ~1.95× faster) are reported by UkisAI in the base model card and are not measurements of these quantized weights. What you should realistically expect from a quantization of a "think less" fine-tune: token savings largely survive (they come from behaviour, not precision), while the hardest math splits and long-horizon tool chains degrade slightly — most visibly at 4-bit.

Measured throughput/acceptance-length data and issue reports (especially "quant X broke task Y") are welcome and will be merged into this table.

Known caveats

• Quantization is lossy. If you see a regression, compare with a higher precision variant and the BF16 source before filing a bug.
• oQ4-mtp can amplify repetition on degenerate loops; keep repetition_penalty at 1.0 first and only then nudge it.
• Vision works through the BF16 tower, but I have not benchmarked VQA accuracy post-quantization.
• 262k context is the architecture's limit, not a promise: keep KV cache within your memory budget or the system swaps.
• MTP decoding only helps when the speculative draft is enabled in the runtime; without it you pay for the head and get nothing.

License

This repository is distributed under the Swift Open License v1.0. A quantization is a derivative work: it inherits the upstream terms in full and cannot be released under a more permissive license.

• Free personal, research, educational, evaluation and commercial use for individuals and organizations with annual recurring revenue (including affiliates) up to US$1,000,000.
• Above that threshold, commercial use requires a separate Swift Enterprise License from UkisAI.
• The base Qwen3.8 checkpoint and the ThinkingCap-Qwen3.6-27B transfer component (BottleCap AI) contribute their own terms, which apply to you as well — read the LICENSE files in ukisai/Swift-Qwen3.8-27b and the BottleCap repository before commercial deployment.
• Keep this attribution, the upstream citation and the base_model metadata intact when you redistribute.

Citation

@misc{swift-qwen3.8-27b-mlx-quants,
  title   = {Swift-Qwen3.8-27b-oQ6-mtp}: oMLX/MLX quantization of Swift-Qwen3.8-27B with MTP head retained,
  author  = {TokenAI-zer},
  year    = {2026},
  howpublished = {\url{https://huggingface.co/TokenAI-zer/Swift-Qwen3.8-27b-oQ6-mtp}},
  note    = {Unofficial quantization of ukisai/Swift-Qwen3.8-27b}
}

@misc{swift-qwen3.8-27b,
  title  = {Swift-Qwen3.8-27B},
  author = {UkisAI},
  year   = {2026},
  url    = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
}

Acknowledgements

• UkisAI for Swift-Qwen3.8-27B and the public evaluation harness.
• Qwen team for the Qwen3.8-27B base model.
• BottleCap AI for the ThinkingCap-Qwen3.6-27B transfer component used upstream.
• oMLX for the Apple Silicon server and quantizer that made these builds possible.