NaorBonomo/gemma-4-e4b-it-4bit-ple

Gemma 4 E4B (4-bit, PLE-split) for on-device inference

This is Google's Gemma 4 E4B instruction-tuned model, quantized to 4-bit (group size 64) and repackaged so an iPhone can run it. It is a repackaging of mlx-community/gemma-4-e4b-it-4bit. No weights were retrained or altered, and the split is verified bit-for-bit against the source. Gemma's original license and terms of use apply.

It runs with SwiftPLE, a Swift package built on mlx-swift. It will not load in stock mlx-lm or other MLX loaders, which don't know about the split table.

Why the split

Gemma 4 uses Per-Layer Embeddings (PLE). Every token ID indexes a 256-dim vector for each of the 42 layers, which is 10,752 values per token. At a 262,144-token vocabulary, that table alone is about 1.5 GB. Standard loaders put it in GPU memory with the rest of the model, which pushes an iPhone past the iOS memory-kill (Jetsam) limit.

PLE is looked up by token ID before the attention layers run, so it can live outside the main weights. This repo ships the model in parts:

File	Size	Purpose
model.safetensors	~2.5 GB	Transformer weights, loaded normally
gemma4_ple_table.bin	~1.5 GB	PLE table, memory-mapped from flash (mmap, MADV_RANDOM)
ple_manifest.json	<1 KB	Row layout, quantization schema, SHA-256
config.json, tokenizer files, chat_template.jinja	small	Upstream configuration

Binary layout: each token is one contiguous 6,048-byte row: 5,376 bytes of packed 4-bit weights, then 336 bytes of bfloat16 scales and 336 bytes of biases. Row offset is tokenID × 6048, so a decode step touches one row.

Usage

import SwiftPLE

let engine = PLEModelClient(modelRepo: "NaorBonomo/gemma-4-e4b-it-4bit-ple")
try await engine.prepare()
let result = try await engine.generate(prompt: "Hello", maxNewTokens: 256) {
    print($0, terminator: "")
}

Add SwiftPLE via Swift Package Manager: https://github.com/naorbonomo/SwiftPLE.git. A complete iOS/macOS demo app is in Example/SwiftPLEDemo.

Requirements: iOS 17+ / macOS 14+, Apple Silicon. On iOS, enable the Increased Memory Limit entitlement (com.apple.developer.kernel.increased-memory-limit).

Performance (Gemma 4 E4B, decode)

Device	Decode	Prefill
iPhone 17 Pro	up to 21 tok/s (~16 tok/s sustained under load)	~578 tok/s
M4 MacBook Air	~26 tok/s	n/a

Wired memory stays under 2.5 GB. Numbers are at empty context, and decode speed drops as context grows.

Convert your own

The ple-splitter tool in the SwiftPLE repo (Tools/ple-splitter) produces this exact format from an upstream checkpoint.

Limitations

• Text generation only. The vision and audio towers were removed.
• Needs the SwiftPLE runtime.
• Inherits the capabilities and limitations of the base Gemma 4 model.