aufklarer/EmbeddingGemma-300M-MLX-8bit

EmbeddingGemma 300M — MLX 8-bit

MLX conversion of google/embeddinggemma-300m for on-device transcript search on Apple Silicon. It produces the checkpoint's own 768-dimensional sentence embeddings: prompt, bidirectional Gemma 3 encoder, mean pooling, dense head, L2 normalization.

Exported from the upstream checkpoint at revision 57c266a740f537b4dc058e1b0cda161fd15afa75, not from a third-party MLX conversion.

Usage (Swift / MLX)

Used by speech-runtime's TextEmbedding module:

import TextEmbedding

let embedder = try await EmbeddingGemma.load(directory: bundleDirectory)
let passages = embedder.embed(documents: ["The deadline moved to March."], batchSize: 16)
let query = embedder.embed(query: "when is the deadline")

A query and the documents that answer it are embedded under different prompts, which embed(query:) and embed(documents:) apply.

Model details

• Architecture: Gemma 3 bidirectional encoder — 24 layers, hidden size 768, 3 attention heads over 1 key/value head, head dimension 256
• Head: mean pooling, dense 768 → 3072 → 768, L2 normalization
• Embeddings: 768 dimensions, float32
• Precision: 8-bit, group 64
• Size: 327 MB
• Context: 512 tokens including <bos> and <eos>
• Prompts: title: none | text: for a document, task: search result | query: for a query

Fidelity

Cosine against the float32 source through sentence-transformers, on identical token ids, over 240 meeting-transcript passages and 800 utterances: 0.9989 minimum, 0.9996 mean.

What a runtime has to match

Three details decide whether a port reproduces this model, and each was measured against the source:

• Bidirectional window 257. Full layers see every token; sliding layers see tokens fewer than 257 positions away, which is what transformers derives from the stored sliding_window: 512 for a bidirectional Gemma 3. It is written into config.json as bidirectional_window. Using 512 fell to 0.9985 cosine on a 294-token passage, and a causal mask scored 0.54–0.65.
• Padding is masked in attention, not only in pooling.
• Norm weights include Gemma's +1 (norm_weights_include_offset in config.json), so every norm is a plain RMSNorm.

Conversion

From soniqo/speech-models, models/embeddinggemma/export:

poetry run python convert_mlx.py --output EmbeddingGemma-300M-MLX-8bit \
    --bits 8

The exporter's tests hold the MLX implementation to the source checkpoint through sentence-transformers in float32.

License

The base model is Gemma, and this is a derivative of it: use is governed by the Gemma Terms of Use and the Gemma Prohibited Use Policy, which apply to these weights as they apply to the original. The weights are converted and quantized; nothing else about the model is changed.