AmarettoLabs/amaretto-embed-148m-ONNX

amaretto-embed-148m — ONNX

ONNX builds of AmarettoLabs/amaretto-embed-148m — a 148M-parameter EmbeddingGemma specialised for 8 Latin-script languages + code — for running text embeddings on CPU with onnxruntime.

For the full model description, benchmarks, and the vocabulary-slicing + distillation method, see the source model card and the code repository.

Not using onnxruntime? There are also GGUF builds for llama.cpp (113–293 MB, down to Q5_K_M), and the source model itself for PyTorch / sentence-transformers.

These are self-contained, end-to-end graphs

Not the backend="onnx" layout. These graphs take token ids and return the final embedding — mean pooling, both Dense projections, and L2 normalization are all inside the graph. That is deliberate: it means no sentence-transformers, no optimum, and no transformers are needed at inference time, only onnxruntime and tokenizers.

Because of this they do not work with SentenceTransformer(..., backend="onnx"), which expects an onnx/ subfolder containing a transformer-only graph and applies pooling itself. If you want that workflow, use the source model with PyTorch instead.

inputs	input_ids [batch, seq] int64, attention_mask [batch, seq] int64
output	sentence_embedding [batch, 768] float32, already L2-normalized
dynamic axes	batch and sequence are both dynamic
max sequence length	2048 — the graph does not enforce this; truncate before feeding it
opset	18 (ai.onnx)

Files

Fidelity is measured against the original PyTorch model — cosine similarity of the output embeddings over a fixed prompt set spanning nearly the full context.

file	size	fidelity vs PyTorch	notes
model.onnx	612 MB	1.000000 at every length	float32. Standard ONNX ops only — portable to any runtime.
model_int8.onnx	297 MB	0.9999 → 0.9993	8-bit weight-only; activations stay float32. onnxruntime only (see below).

Per sequence length, cos(PyTorch, ONNX):

tokens	model.onnx	model_int8.onnx
15	1.000000	0.999891
29	1.000000	0.999854
45	1.000000	0.999815
135	1.000000	0.999777
978	1.000000	0.999453
1822	1.000000	0.999324

Both builds reproduce the source model within a padded batch as exactly as alone — batching does not change a row's embedding.

Portability caveat for model_int8.onnx. It uses MatMulNBits, an operator in the com.microsoft domain. It ships inside the core onnxruntime binary, so no extra install is needed — but it is not standard ONNX, and other runtimes (TensorRT, OpenVINO, CoreML converters, non-ORT ONNX implementations) will reject it, as may onnxruntime builds older than ~1.20. If you are not on onnxruntime, use model.onnx.

Why weight-only, and not int8 activations

EmbeddingGemma carries large activation outliers — the same reason Google notes that fp16 activations are unusable for it. Quantizing activations to 8 bits therefore costs a great deal of accuracy, and the error compounds with sequence length. For reference, on the same prompt set: standard dynamic int8 reaches only 0.880 at 1822 tokens, and statically calibrated int8 is worse still. Weight-only quantization leaves activations in float32 and avoids the problem entirely, which is why it is the only quantized build published here.

Memory

File size is not the memory budget. Peak resident memory is dominated by float32 activations, so it scales with your longest input and batch size, not with the weights:

	file	RSS after load	peak, 1822-token input
model.onnx	612 MB	~920 MB	~2,120 MB
model_int8.onnx	297 MB	~555 MB	~1,880 MB

If you are memory-capped, limiting sequence length is a far stronger lever than quantization: a 512-token cap costs roughly half the peak of a 2048-token one.

model_int8.onnx is also slower per query than model.onnx on some hardware, because MatMulNBits dequantizes weights on the fly rather than running integer kernels. Choose it for download size and idle footprint, not for latency. Measure on your own hardware.

Usage

Only onnxruntime and tokenizers are required.

import numpy as np, onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")   # adds <bos>/<eos> itself
# Truncate the SEQUENCE so the post-processor still appends <eos>. Slicing the resulting
# ids instead (ids[:2048]) would drop it on any input at the limit.
tok.enable_truncation(max_length=2048)
sess = ort.InferenceSession("model_int8.onnx", providers=["CPUExecutionProvider"])

def embed(texts):
    ids = [tok.encode(t).ids for t in texts]
    n = max(len(x) for x in ids)
    input_ids = np.zeros((len(ids), n), dtype=np.int64)      # pad id 0
    mask = np.zeros((len(ids), n), dtype=np.int64)
    for i, row in enumerate(ids):
        input_ids[i, :len(row)] = row
        mask[i, :len(row)] = 1
    return sess.run(["sentence_embedding"],
                    {"input_ids": input_ids, "attention_mask": mask})[0]

# Task prefixes are REQUIRED — same as EmbeddingGemma.
queries   = ["task: search result | query: how do I sort a list in python?"]
documents = ["title: none | text: Use list.sort() to sort in place, or sorted() for a new list."]

q, d = embed(queries), embed(documents)
print((q @ d.T).item())        # already normalized, so a dot product is the cosine

Task prefixes matter. Use task: search result | query: for queries and title: none | text: for documents. The graph cannot add them — they are text, prepended before tokenization. Omitting them measurably degrades retrieval.

Embeddings are 768-dimensional and Matryoshka-trained: truncate to 512/256/128 and re-normalize.

Intended use & limitations

A general-purpose text-embedding model — retrieval, semantic search, RAG, clustering, classification, STS — for English, Spanish, Portuguese, French, German, Italian, Dutch, Polish, and source code.

Not multilingual. The vocabulary for non-target languages was removed; text in other scripts (Chinese, Japanese, Korean, Arabic, Cyrillic, Greek, Indic, Thai, …) degrades severely. Use the original google/embeddinggemma-300m for broad multilingual coverage.

Benchmarks (as % of EmbeddingGemma): ≥99.3% on retrieval and STS, ≈98.7% on code retrieval, ≈97.9% on classification, ≈96% on long-document retrieval. Full tables on the source model card.

License — Gemma Terms of Use

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms

These ONNX files are a format conversion of amaretto-embed-148m, itself a Gemma Model Derivative of google/embeddinggemma-300m. Your use, reproduction, and distribution are governed by the Gemma Terms of Use, a copy of which is distributed here in the LICENSE file. By using these files you accept those terms.

• Use restrictions (§3.2): you may not use these models in violation of the Gemma Prohibited Use Policy.
• If you redistribute these or a derivative, the Gemma Terms (§3.1) require you to pass on the use restrictions, include the LICENSE, mark modified files, and ship a NOTICE with the required Gemma notice.
• Trademarks (§4.2): not affiliated with, endorsed by, or sponsored by Google. "Gemma" / "EmbeddingGemma" are used descriptively to identify the upstream model.

Built by AmarettoLabs. Converted with onnxruntime and optimum.