cardbud/gemma-4-E2B-it-coreml-ane

Gemma 4 E2B IT — Core ML for the iPhone Neural Engine, with logits

A Core ML conversion of google/gemma-4-E2B-it (by Google DeepMind), text only, derived from the original model to run on the iPhone Neural Engine. The final chunk returns the scores of every token, so the caller can sample with any temperature, top-k or seed.

What was changed

These files are modified versions of the original model:

• Converted to Core ML with CoreML-LLM v1.9.0 (MIT): stateful 3-chunk layout, context 2048, sliding window 512, weights palettized to 4 bits, Linear projections.
• chunk_1 and chunk_2 are taken unchanged from mlboydaisuke/gemma-4-E2B-stateful-coreml at revision 157dda4, as are the embedding and RoPE sidecar files.
• chunk_3 (layers 25–34, final norm, output layer with soft-capping) was converted again from the original weights with the same settings, returning logits ([1, 1, 262144], float16) instead of the in-graph argmax.

Files

• chunk_1.mlmodelc — layers 0–7; owns its KV cache (MLState), computes the per-layer inputs. Functions infer (one token) and prefill_b8 (eight prompt tokens).
• chunk_2.mlmodelc — layers 8–24; owns its KV cache, returns the shared KV (kv13_*, kv14_*). Functions infer and prefill_b8.
• chunk_3.mlmodelc — layers 25–34 and the output layer; stateless, reads the shared KV, returns logits.
• embed_tokens_q8.bin, embed_tokens_per_layer_q8.bin and their _scales.bin — int8 embedding tables with one float16 scale per row (scaled by embed_scale and per_layer_embed_scale from model_config.json).
• cos_*.npy, sin_*.npy — RoPE tables for sliding (256) and full (512) attention.
• hf_model/ — tokenizer and chat template.

Load with computeUnits = .cpuAndNeuralEngine. Per token, run chunk_1 and chunk_2 with their states (a prompt can go through prefill_b8 eight tokens at a time, as long as it doesn't wrap the sliding window); run chunk_3 only where the next token is needed. CoreML-LLM's Gemma4StatefulEngine shows the inputs (masks, positions, RoPE rows); it reads token_id, where this build returns logits.

Prompt format (Gemma 4):

<bos><|turn>system
{instructions}<turn|>
<|turn>user
{message}<turn|>
<|turn>model

Add <|think|>\n at the start of the system turn for thinking; the thinking comes before the answer as <|channel>thought\n…<channel|>. Stop at token 1 (<eos>) or 106 (<turn|>).

License

Same license as the original model: Apache 2.0, as published by Google for Gemma 4.