cloudyu/gemma4-31B-Fable-5-Distilled-MLX-mxfp4

Gemma-4-31B-Fable-5-Distilled — MLX (mxfp4)

Model ID: cloudyu/gemma4-31B-Fable-5-Distilled-MLX-mxfp4

This is the MLX-converted, mxfp4-quantized version of autotrust/gemma4-31B-Fable-5-Distilled, converted using mlx_vlm.

Model Origin

• Base model: google/gemma-4-31B-it
• Fine-tuned by: AutoTrust AI Lab
• Trained by: Hai Yu
• Method: LoRA (r=16) on Glint-Research/Fable-5-traces
• LoRA applied to layers 30–59 only (layers 0–29 + vision encoder frozen), preserving multimodal vision
• Trainable parameters: 61.2M (0.20% of 31.27B)

Quantization

Property	Value
Format	MLX safetensors
Quantization	mxfp4 (4-bit)
Group size	32
Original dtype	bfloat16
Total size	~17.5 GB on disk

Benchmark

Model	HumanEval pass@1
This model (ours)	92.7% (152/164)
google/gemma-4-31B-it (base)	76.8%

Evaluation: HumanEval 164 problems, vLLM 0.22, T=0.1, thinking=off.

Usage

Requires mlx-vlm and mlx:

uv pip install mlx-vlm mlx

Text (code generation with thinking)

python3.12 -m mlx_vlm.generate \
  --model cloudyu/gemma4-31B-Fable-5-Distilled-MLX-mxfp4 \
  --prompt "Write a Python function to reverse a linked list." \
  --max-tokens 512

Image understanding

python3.12 -m mlx_vlm.generate \
  --model cloudyu/gemma4-31B-Fable-5-Distilled-MLX-mxfp4 \
  --image photo.jpg \
  --prompt "Describe this image in detail." \
  --max-tokens 256

Video understanding (requires PR fix)

python3.12 -m mlx_vlm.video_generate \
  --model cloudyu/gemma4-31B-Fable-5-Distilled-MLX-mxfp4 \
  --video video.mp4 \
  --fps 1.0 \
  --prompt "Describe this video" \
  --max-tokens 10000

Python API

from mlx_vlm import load, generate
from mlx_vlm.utils import load_image

model, processor = load("cloudyu/gemma4-31B-Fable-5-Distilled-MLX-mxfp4")

# Text
messages = [{"role": "user", "content": "Write a Python function to reverse a linked list."}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
output = generate(model, processor, prompt, max_tokens=512)
print(output)

# Image
image = load_image("photo.jpg")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image."}
]}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
output = generate(model, processor, prompt, image=image, max_tokens=256)
print(output)

Important: Video Processing Bug Fix

Upstream mlx-vlm has a bug in Gemma 4 video processing that causes hallucinated output (e.g., "a gray horizontal line on a dark background" instead of actual video content). The fix is submitted at:

PR #1431 — fix: Gemma4 video processing - float32 overflow and wrong token scatter

Two bugs fixed by this PR:

1. Float32 overflow in _resize_frames: fetch_video() returns float32 frames in [0,255] range, but _resize_frames treats them as [0,1] and multiplies by 255 again, causing overflow/clipping. Also, the output dtype was float32 instead of uint8, preventing the rescale step from triggering.

2. Wrong token scatter: Video pixel values were passed as pixel_values (scattered at image_token_id positions) instead of pixel_values_videos (scattered at video_token_id positions), so vision features were never injected into video prompt slots.

Until this PR is merged, apply the changes manually:

• mlx_vlm/models/gemma4/processing_gemma4.py: Fix _resize_frames output dtype and value range
• mlx_vlm/video_generate.py: Change kwargs["pixel_values"] to kwargs["pixel_values_videos"]

Layer-Freezing Strategy

┌─────────────────────────────────────────────────────────────┐
│  Layers 30–59  │  🟢 LoRA-adapted (language head)           │
│   (30 layers)  │     Q/K/V/O + gate/up/down projections     │
├─────────────────────────────────────────────────────────────┤
│  Layers 0–29   │  🔒 FROZEN (multimodal fusion)             │
│   (30 layers)  │     Visual feature processing untouched    │
└─────────────────────────────────────────────────────────────┘
            ▲
    Vision encoder (mmproj) — fully frozen

This strategy ensures image description quality matches the base model exactly while delivering +15.9 point HumanEval improvement.

Model Details

Property	Value
Architecture	Gemma4ForConditionalGeneration
Parameters	31.27B (bfloat16 original)
Hidden size	5376
Layers	60 (27 vision + 60 text)
Attention heads	32 (16 KV)
Vocab size	262,144
Max position embeddings	262,144
Sliding window	1024
Vision encoder	27-layer Gemma4Vision, patch_size=16
Vision soft tokens	280 per image

Training

Hyperparameter	Value
Optimizer	AdamW
Learning rate	2e-4
LR schedule	Cosine
Warmup steps	50
Batch size (per device)	1
Gradient accumulation	16
Precision	bfloat16
Epochs	1
Training examples	23,325

Known Limitations

• 308 training examples: May not generalize to all coding domains.
• Thinking-mode dependency: Trained with enable_thinking=True; responses without thinking may be suboptimal.
• Tool calls: JSON-serialized, not bound to a runtime. You provide the execution layer.
• Missing stdlib imports: Occasionally omits imports like re, math — a distillation artifact from Fable 5 traces.
• Inherits Gemma base limitations: factual errors, occasional hallucination.

Citation

@misc{autotrust2026gemma4fable5,
  title        = {Gemma-4-31B-Fable-5-Distilled: Layer-Frozen LoRA Distillation
                  Preserving Multimodal Vision},
  author       = {{AutoTrust AI Lab} and Yu, Cloud},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/autotrust/gemma4-31B-Fable-5-Distilled}},
}

Related Models

• 🤗 cloudyu/gemma4-31B-Fable-5-Distilled-MLX-mxfp4 — this model, MLX mxfp4 quantized
• 🤗 autotrust/gemma4-31B-Fable-5-Distilled — original bfloat16 weights
• 🤗 autotrust/gemma4-31B-Fable-5-Distilled-GGUF — GGUF quantized variants (F16 / Q8_0)

About AutoTrust AI Lab

AutoTrust AI Lab builds open foundation models and agentic systems for scientific research and coding.

• 🌐 autotrust.ai
• 🤗 huggingface.co/autotrust
• 📧 andy@autotrust.ai