angelsbrood/Qwen3.8-27B-FP8-CoreAI-Fused-MTP-Multimodal

Qwen3.8-27B FP8 + native MTP + multimodal Core AI

Experimental native Core AI package for the official Qwen/Qwen3.8-27B-FP8 checkpoint at revision 017b9c7af6b5689d5dd426a76e0bc077eb5ca20a.

This is the original Hugging Face checkpoint, not an MLX conversion. The target retains the checkpoint's official blockwise FP8 weights and activation-quantized matmul path. It is not a post-conversion integer quantization of a BF16 model.

The checked-in AOT graph is the accepted v5 conversion. The v6 package adds a complete native Swift host, stricter preprocessing parity, runtime hardening, and additional acceptance evidence without duplicating or altering the model weights.

What is fused

One Core AI parameter store exposes three functions:

• main: all 64 target layers plus the checkpoint's native one-layer MTP module in a K=2 draft-and-verify round;
• vision: the official vision encoder with dynamic, unpadded inputs from 4 through 4,096 patches per temporal group;
• prefill3: an experimental three-token target prefill function over the same shared target weights.

The MTP tensors exist in the official checkpoint but ordinary target inference does not execute them as a speculative drafter. main is the conversion that wires those tensors into drafting, target verification, and coherent target/MTP cache deltas. The host maintains this invariant after every committed round:

mtpPast == targetPast - 1

Native runtime

The Swift package now implements the full tested host path, not only model loading and ABI inspection:

• official tokenizer, chat template, thinking controls, and reasoning effort;
• standard message histories and recursive tool schemas;
• image, ordered multi-image, and local-video content blocks;
• exact official smart-resize dimensions, UInt8 antialiased bicubic image resizing, normalization, patchification, visual token expansion, and mRoPE;
• dynamic vision calls and embedding insertion into the fused text/MTP graph;
• cache allocation/growth, memory admission, EOS handling, streaming token events, typed cancellation, and structured completion reports;
• native Qwen XML tool-call parsing;
• greedy fused MTP, target-only greedy, and deterministic target sampling.

The resized-image regression is byte-exact with the Transformers 5.8.1 / Torchvision 0.24.0 reference tensor. Its full native chat run also exactly matches the Python reference prompt IDs, generated IDs, prefill commits, mRoPE, and final cache positions.

Generation modes

Mode	Status	Selection path
greedy-mtp	Accepted for tested greedy use	Fused native MTP drafts and target verifies up to two proposals
target-greedy	Accepted reference path	Target selects every token
official-sampling	Accepted target-only sampler	Official temperature/top-k/top-p defaults with a deterministic native SplitMix64 inverse-CDF RNG
experimental-sampling-mtp	Rejected for production fidelity	Cache-safe experiment; one of five tested seeds diverged from sequential target-only sampling

official-sampling describes the official filtering configuration, not PyTorch RNG identity. The same seed is reproducible in this Swift runtime, but it is not expected to produce PyTorch's exact seeded token stream.

The sampled-MTP experiment remains explicitly available for research. It is not a drop-in mode: compiled multi-row verifier logits can differ slightly from sequential one-row logits, and seed 3 crossed a sampling boundary. Use target-only sampling when fidelity matters.

Artifacts

assets/qwen3_8_27b_fp8_native_mtp_multimodal_main_prefill3_singleasset_dynamic262144ctx_patches4096_v5.h17s.aimodelc
bin/h17s/qwen38-coreai-runner
bin/h17s/qwen38-coreai-memory-audit
RELEASE_MANIFEST.json
MANIFEST.sha256

The AOT asset targets h17s / M5 Pro, GPU, and macOS 27.0. It was built with coreai-build 3600.82.1 and validated on an M5 Pro with 64 GB unified memory, macOS 27.0 build 26A5421a, and Xcode 27.0 build 27A5237l.

This Hub release is the approximately 29 GB AOT-only M5 Pro deployment. The separate portable representation is also about 29 GB and describes the same logical model, but it is intentionally not duplicated here. It remains a recompilation archive for other supported Apple architectures; those builds must rerun acceptance on their target chip.

RELEASE_MANIFEST.json identifies the accepted modes, exact representations, packaged binary hashes, and decisive validation reports. MANIFEST.sha256 checks every shipped file.

Quick start

PACKAGE=/path/to/package
ASSET="$PACKAGE/assets/qwen3_8_27b_fp8_native_mtp_multimodal_main_prefill3_singleasset_dynamic262144ctx_patches4096_v5.h17s.aimodelc"
TOKENIZER="$PACKAGE/resources/tokenizer"
RUNNER="$PACKAGE/bin/h17s/qwen38-coreai-runner"

"$RUNNER" "$ASSET" "$TOKENIZER" \
  --prompt "Describe this image." \
  --image /path/to/image.png \
  --mode greedy-mtp \
  --disable-thinking \
  --max-new-tokens 64 \
  --output result.json

Build the executable from swift/ if a prebuilt binary is not present. See RUNTIME.md for Swift build commands, message/tool JSON, multiple images, video, sampling, cancellation, context, and the Python oracle.

Acceptance summary

• Source accounting covers all 64 target layers, one native MTP layer, and 333 vision tensors from the pinned official checkpoint.
• The portable asset is only 1.000029x the prior target+MTP+vision baseline, demonstrating shared weights across all three entry points.
• Reachable compiled main decisions matched eager; target-logit PSNR was 43.56 dB on a visual state and 47.32 dB on a full-prompt state.
• Dynamic vision passed 256, 512, 1,024, 1,536, mixed-image, four-frame video, and 4,096-patch cases. Compiled-vs-official PSNR was 88.78-97.41 dB.
• Native Swift and Python matched exactly on tested text, image, resized image, two-image, video, and tool-history chat contracts.
• The resized mixed-axis image matched the official Float16 patch tensor byte-for-byte and passed all 11 end-to-end parity checks.
• The release Swift suite passes 19 ABI, preprocessing, tokenizer, mRoPE, cache, admission, video, resize, and tool tests.
• Independent Python and native Swift active-history gates both passed at the declared 262,144-token ceiling. The native call exposed history through 262,141 and returned all 13 fused outputs.
• Ordinary native runs peak around 1.35-1.63 GB private physical footprint. Their 41-53 GB resident-size readings are dominated by file-backed model mappings and are not equivalent to private RAM. Long active contexts are materially heavier; the ceiling call needs a clean 64 GB machine and is not representative of ordinary generation.

Status

This is a strong research/runtime artifact for the validated M5 Pro setup, not a production-certified Transformers server. The remaining material gaps are the prerelease Core AI toolchain, chip-specific AOT, slow one-to-three-token fidelity prefill for long prompts, the 4,096-patch compiled vision ceiling, native-vs-PyTorch RNG identity, and unimplemented serving features such as batching and scheduling. Read KNOWN_GAPS.md before deployment.

Source and license

Model configuration and tokenizer files derive from the pinned official Qwen checkpoint. Its Apache 2.0 license is included as LICENSE-QWEN. Conversion sources, the Python oracle, native Swift runtime, metadata, and validation evidence are included for audit and recompilation.