Swift-Qwen3.8-27b-oQ4e-mtp

Model architecture

A 27B-class dense multimodal model stored in MLX format. It contains a text backbone, a vision encoder, and one multi-token prediction (MTP) layer.

Component	Structure
Text backbone	64 layers; hidden size 5,120; feed-forward size 17,408
Attention layout	48 linear-attention layers and 16 full-attention layers, with full attention every fourth layer
Full attention	24 query heads, 4 key/value heads, head dimension 256
Vocabulary	248,320 tokens
Configured context limit	262,144 tokens; usable length depends on runtime settings and available memory
Vision encoder	27 layers; hidden size 1,152; 16 attention heads; 16 × 16 image patches
Vision-to-text connection	Vision features are projected to the text hidden size of 5,120
MTP	One additional prediction layer with attention and feed-forward projections

Weight precision

The oQ4e checkpoint uses mixed precision rather than uniform 4-bit weights:

• The default quantization is 4-bit affine, with 64 values per group.
• 187 modules have explicit 5-bit overrides in config.json.
• The MTP layer's seven large attention and feed-forward matrices use 4-bit weights.
• The MTP fusion matrix (mtp.fc) and normalization weights remain BF16. The fusion matrix maps 10,240 input features to 5,120 output features.
• The MTP quantization scales and offsets are stored in BF16.

This checkpoint retains BF16 floating-point weights in the MTP fusion and normalization layers alongside its quantized matrices. Exact per-module settings are recorded in config.json.

Source and license

Source revision: 54e66d6c81439bd4fda5ef9a690fa571e3b0d272.

Original model by UkisAI. This conversion does not change the upstream Swift Open License v1.0 terms. Consult the source model license and access information.