aufklarer/FunctionGemma-270M-CoreML

FunctionGemma 270M — CoreML (fp16)

A Core ML export of google/functiongemma-270m-it, optimized for the Apple Neural Engine on iOS 18 / macOS 15. The 18-layer transformer is reshaped into Apple's BC1S layout ((B, C, 1, T) channel-last with 1×1 Conv2d projections and per-head split attention) and the K/V cache lives in MLState slots, so token-by-token decode sends no tensor I/O back to the host.

This is the fp16 reference build — full Float16 weights, no quantization. For the same accuracy with ~½ the disk and faster decode, see the Palettize-8 variant.

Model

Parameters	270M
Architecture	Gemma 3 (18 layers, 4 query heads, 1 KV head, head_dim 256, hidden 640, MLP 2048)
Quantization	none (Float16 weights, fp16 compute)
Format	Core ML .mlmodelc (ML Program)
Cache layout	BC1S MLState, fixed cache length 128
Shapes	T_q ∈ {1, 128} via EnumeratedShapes
File size	513 MB model + 33 MB tokenizer ≈ 546 MB total
Min target	iOS 18 / macOS 15
Compute units	cpuAndNeuralEngine (required — CPU-only emulation diverges)

Files

File	Size	Description
FunctionGemmaANEUnifiedStateful.mlmodelc/	513 MB	Compiled Core ML model. Load with MLModel(contentsOf:).
config.json	~2 KB	Architecture metadata (state names, input/output names, deployment target).
chat_template.jinja	~1 KB	Jinja chat template used by tokenizer.apply_chat_template.
tokenizer.json	~33 MB	Hugging Face tokenizers fast SentencePiece model.
tokenizer_config.json	~1 KB	Tokenizer settings.

Performance

Measured on Apple M-series Mac via cpuAndNeuralEngine, on the canonical "Convert 23 USD to EUR" tool-call prompt (91-token prompt → 31-token function call), warmed.

	Value
Prefill (128 tokens)	8.5 ms
Decode	5.65 ms/token (177 tok/s)
End-to-end (32 tokens)	~185 ms
Swift peak RSS (warm)	~37 MB private + ~510 MB mmap'd from disk (evictable)
Compute-plan device	96 %+ of ops prefer neuralEngine
Output	Byte-identical to PyTorch fp32 reference on the canonical prompt

Usage

Swift (iOS 18 / macOS 15)

import CoreML

let url = URL(fileURLWithPath: "FunctionGemmaANEUnifiedStateful.mlmodelc")
let config = MLModelConfiguration()
config.computeUnits = .cpuAndNeuralEngine
let model = try MLModel(contentsOf: url, configuration: config)
let state = model.makeState()

// Build prefill inputs (input_ids, cos/sin tables, attention mask,
// write_mask=ones, logits_mask one-hot at the last prompt position),
// then for decode call repeatedly with T_q=1 inputs and a one-hot
// write_mask at the current cache slot.
let output = try await model.prediction(from: prefillInputs, using: state)
let logits = output.featureValue(for: "logits")!.multiArrayValue!

The full prefill + decode driver is published as part of the speech-swift SDK.

Python (coremltools, macOS only)

import coremltools as ct
import numpy as np

model = ct.models.MLModel(
    "FunctionGemmaANEUnifiedStateful.mlpackage",
    compute_units=ct.ComputeUnit.CPU_AND_NE,
)
state = model.make_state()
out = model.predict(prefill_inputs, state=state)
next_id = int(out["logits"][0].argmax())

Source

Upstream model: google/functiongemma-270m-it — Gemma 3 270M instruction-tuned for structured function calls.

Links

• speech-swift — Apple SDK
• soniqo.audio — website
• blog