ToPo-ToPo/gemma-4-26B-A4B-it-mlx-6bit

MLX 6bit conversion of google/gemma-4-26b-a4b-it (Gemma 4 26B-A4B, MoE / 128 experts), made with mlx-vlm 0.6.6.

Provenance (self-converted, re-quantized)

• Source: ToPo-ToPo/gemma-4-26B-A4B-it-mlx-8bit, dequantized to bf16 and quantized again to 6 bits. It was not made from the original bf16 weights, so the weights went through two rounding steps.
• Quantization: 6-bit affine (group_size=64, ~6.72 bpw). The MoE router projections stay at 8 bits, as in the 4bit / 8bit conversions.
• Tool: mlx-vlm 0.6.6 (mlx_vlm convert -d, then -q --q-bits 6 --q-group-size 64). The conversion stopped at the processor save step (the copied tokenizer.json was read-only) before writing the quantization block, so the quantization / quantization_config entries in config.json were added by hand (same layout as the 4bit conversion, with bits: 6).
• Validated: loads and answers in Japanese under mlx-vlm 0.6.6.

Measurements (Apple Silicon, mlx-vlm 0.6.6, same prompt)

	8bit	6bit
Peak memory	28.0 GB	21.7 GB
Generation	11.6 tok/s	6.1 tok/s

In our agent evaluation (tool calling, multi-step CAD/CAE jobs) this 6bit build completed the tasks but was about half as fast as the 8bit build and made one path-copying mistake in a report, so we kept using 8bit.

Usage

from mlx_vlm import load
model, processor = load("ToPo-ToPo/gemma-4-26B-A4B-it-mlx-6bit")