OsaurusAI/Qwen3.8-27B-JANG_6D

The near-lossless tier — 6/8-bit mixed allocation for maximum quality below the fp-reference footprint.

A calibrated JANG bundle of Qwen/Qwen3.8-27B — Qwen's 27B dense hybrid (GatedDeltaNet + gated attention) vision-language model with flexible thinking control — quantized for Apple Silicon / MLX and runnable today with stock mlx_vlm. Text, image, and video understanding all verified working in this exact bundle, with the model's native multi-token-prediction head preserved.

Why this quant

This is not a uniform one-bit-number-for-everything conversion:

• Measured allocation, not name rules — every one of the 590 quantized modules got its bit width from a Hessian-trace sensitivity capture (tr(H)·‖W‖²_F per module, from activation statistics over a calibration corpus), so the bits go where this model actually needs them. Attention stays high-precision; insensitive FFN blocks give the budget back.
• imatrix refit on every sub-8-bit module — after allocation, each affine module below 8 bits is re-fit with activation-weighted (imatrix) least squares against the calibration capture, at zero size cost. This is the transferable half of llama.cpp's IQ recipe, applied natively to MLX affine storage.
• fp16 where quantization would lie — the 27 vision-block linear_fc2 projections (in_features 4304, indivisible by any MLX quant group) pass through in fp16 instead of being force-fit. No silently broken vision tower.
• The full serving contract is stamped, not documented-elsewhere — sampling presets, reasoning-effort tiers, thinking defaults, EOS pair, context guidance: all machine-readable in the bundle (see below).
• Honest method record — imatrix + Hessian allocation applied; AWQ and GPTQ deliberately not applied (AWQ's norm-fold is unsafe on this family's zero-centered-norm convention; GPTQ needs off-diagonal Hessians — logged as follow-up), and the bundle metadata says exactly that.

The lineup

Four bundles of the same calibrated build, by size/quality trade: JANG_2D (10.9 GiB) · JANG_4D (17.0 GiB) · JANG_6D (24.1 GiB) · MXFP8 (26.8 GiB, reference tier).

Model + bundle facts

Field	Value
Base model	Qwen/Qwen3.8-27B (dense 27B VLM)
Layout	64 layers — 48 GatedDeltaNet + 16 gated full-attention (partial RoPE dim 64)
Vision	native image + video tower (verified in this bundle)
MTP	native multi-token-prediction head preserved (31 tensors, own shard)
Context	262,144 native, extensible to 1M
Quantization	2x4-bit / 199x6-bit / 389x8-bit
On disk	24.1 GiB
Runs on	Apple Silicon Macs with ≥ 32 GB unified memory

Serving contract (stamped in the bundle)

Everything below is stamped in generation_config.json + jang_config.json — runtimes should read it, not re-derive it:

• Thinking mode ON by default — sampling temperature=1.0, top_p=0.95, top_k=20 (the agentic preset; also the correct preset for coding agents). Instruct / non-thinking preset: temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5.
• reasoning_effort: low / medium / xhigh (default xhigh), carried as a chat-template kwarg.
• preserve_thinking ON by default (Qwen3.8 retains reasoning context across turns — new in this generation, and prefix-cache friendly).
• Reasoning OFF = prefilled closed <think>\n\n</think>\n\n — never plain omission. Reasoning parser qwen3; tool-call parser qwen3_coder.
• Stop on both EOS ids 248046 and 248044.
• Recommended output budget: up to 262,144 reasoning + 131,072 final tokens.

Use it

pip install -U mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.utils import load_config

model, processor = load("OsaurusAI/Qwen3.8-27B-JANG_6D")

# text + image
out = generate(model, processor, "Describe this image.", image=["photo.png"],
               max_tokens=512, temperature=1.0, top_p=0.95)

Video note: render video prompts through the bundle's own chat template ({"type": "video"} content item → <|vision_start|><|video_pad|><|vision_end|>); mlx_vlm.prompt_utils.apply_chat_template silently drops video items.

MTP head

The bundle ships the model's trained multi-token-prediction head (Qwen trained it with multiple steps) plus a vmlx_mtp_tuning.json sidecar recommending 1 draft token/step as the starting depth. Speculative decoding is opt-in for runtimes that support it; the head is skipped harmlessly everywhere else.

---

Quantized and calibrated by Jinho Jang — eric@osaurus.ai — for Osaurus, the native local-AI runtime for macOS.