bdrgon/embeddinggemma2-audio-tower-onnx

Google EmbeddingGemma 2 audio encoder exported to ONNX.

Experimental: every artifact in this repository, including all FP32, INT8, INT4 graphs and the FP32 weight sidecar, is experimental. Numerical parity on smoke inputs does not establish retrieval quality or production suitability.

Filename	Quantization	File size	Output	Status
audio_embedding.onnx (4.88 MB) audio_embedding.data (2.31 GB)	FP32	2.31 GB total	Final embedding; weights are split across both files	Experimental
audio_embedding_int8.onnx	INT8 dynamic	583.1 MB	Final embedding	Experimental
audio_embedding_int4.onnx	INT4 weight-only	777.1 MB	Final embedding	Experimental
audio_tower.onnx	FP32	1.22 GB	Intermediate states	Experimental
audio_tower_int8.onnx	INT8 dynamic	307.7 MB	Intermediate states	Experimental
audio_tower_int4.onnx	INT4 weight-only	164.5 MB	Intermediate states	Experimental

Input and output

• Final-embedding input: input_ids and attention_mask int64 [1, text_tokens]; input_features float32 [1, feature_frames, 128]; input_features_mask bool [1, feature_frames]. From the pinned AutoProcessor for mono 16 kHz audio; decoding and preprocessing are external.
• Tower input: only input_features float32 [1, feature_frames, 128] and input_features_mask bool [1, feature_frames].
• Output: audio_embedding*.onnx returns an L2-normalized embedding, float32 [1, 512]. audio_tower*.onnx returns unpooled audio_features float32 [1, audio_tokens, 1536] and audio_mask bool [1, audio_tokens]; token count varies with duration.
• Runtime: Opset 18; INT4 requires ONNX Runtime support for com.microsoft::MatMulNBits.

Independent exporters

python -m pip install -r requirements.txt
python export_tower.py
python export_embedding.py

export_tower.py exports only the raw audio tower. export_embedding.py exports only the final 512-dimensional embedding, including its text encoder and pooling. Both use export_common.py for shared export primitives. The former combined export_onnx.py entry point has been replaced by these two scripts.

Reference outputs are computed with untouched upstream Transformers code. ONNX-specific attention patches are scoped to tracing and restored even if tracing fails, so running either exporter cannot contaminate the other's reference. The pinned reference is google/embeddinggemma-2 at 914f7f89142e33e77833254d9c9b90c3cef7303b, with Python 3.13, torch 2.8.0, and Transformers 5.19.0. Splitting the scripts does not modify existing weights.

Experimental status for every graph and weight sidecar is also recorded in artifacts.json.

Validation

Each exporter now runs model-only corpus validation after export. To evaluate the existing artifacts without regenerating them:

python export_tower.py --validate-only
python export_embedding.py --validate-only

The corpus is the entire train split of yojul/one-shot-hip-hop-drums, pinned to 07be604e729fcf8f7904bca5328d6b09b1ceed7b: 19,673 rows, seven drum categories, Apache-2.0. Its Parquet shard is downloaded to the Hugging Face cache; --dataset-cache PATH selects another HF cache. --offline uses cached files. No dataset audio is copied into this model repository or its published reports.

The validations are:

1. Untouched upstream PyTorch output versus FP32 ONNX output, with exact shape and mask agreement, cosine similarity, maximum absolute error, and RMSE.
2. FP32 versus INT8/INT4 output divergence and exact cosine-neighbor agreement across the corpus. Final embeddings search all successful rows. Raw towers search only rows with identical tensor shapes and masks; no pooling or padding is introduced to make incompatible outputs comparable. The reports state eligible query counts. Labels are not used to restrict searches.
3. Ten frozen one-parameter synth continuity triples, evaluated independently for untouched upstream, FP32, INT8, and INT4. Both adjacent distances must be no larger than the endpoint distance, with 1e-6 tolerance; 9/10 is the existing expectation. These Python results do not imply native-runtime parity.

Deferred: Python-versus-native preprocessing parity belongs to a later Grain Elevator evaluation. These scripts have no dependency on Grain Elevator.

Audio is decoded with SoundFile, averaged to mono in float32, limited to its first 30 seconds at the source rate, and resampled to 16 kHz with SciPy if necessary. The same pinned Python processor inputs feed upstream and all three ONNX graphs. Every row is attempted. Empty/invalid inputs and inference errors remain explicit in the JSON report; they are excluded from numerical and neighbor aggregates. Numerical summaries weight each successful clip equally; P95 absolute error is the percentile of per-clip maximum errors, not a pooled tensor percentile.

The scripts print Markdown divergence/neighbor/continuity tables to stdout and write JSON plus Markdown under validation-results/. JSON includes dataset, artifact and evaluator hashes, dependency versions, and failures. The Markdown reports state the numerical thresholds. Exit status is nonzero for input errors, inference errors, numerical violations, or unmet synth expectations, after saving the complete report. These experimental artifacts are not declared validated merely because evaluation ran.

--workers N runs independent CPU workers (each loads its own model copies); --threads N sets inference threads per worker. Budget RAM accordingly. --resume reuses only checkpoints with matching artifacts, code, dataset, environment, and configuration. --limit N is for diagnostics and labels its reports PARTIAL. --skip-validation exports with the original smoke checks only. Per-clip checkpoints and raw neighbor vectors stay in .validation-cache/ and are not committed.

Run the model-only regression tests with:

python -m unittest discover -s tests -v

Measured full-corpus results

Both runs completed on October 13, 2026 (America/New_York), using CPU inference, three workers and one inference thread per worker. Every one of the 19,673 rows was accounted for in each run. The same three rows (zero-based indices 4770, 9910, and 13049) contain zero audio bytes; 19,670 clips were evaluated successfully with no inference errors. Of these, 26 clips were truncated to their first 30 seconds. Full corpus therefore means every row was attempted, not that every input was usable or every numerical gate passed.

Machine-readable provenance and results: tower JSON and embedding JSON. Standalone tables: tower Markdown and embedding Markdown.

Findings: final FP32 embeddings pass the frozen parity tolerances on all 19,670 valid clips (worst absolute error 3.64394e-05). Raw-tower FP32 outputs have near-perfect cosine agreement but fail the absolute-error gate on 3,348 clips (worst absolute error 0.0177279). Cosines shown as 1 below are rounded: the minimum upstream/FP32 cosine is 0.999999992022 for the tower and 0.999999973258 for final embeddings.

Quantization substantially changes final-embedding neighbor rankings: top-1 agreement is 29.71% for INT8 and 32.35% for INT4, with about 42.5% mean top-10 overlap for both. These are agreement measurements, not labeled retrieval quality or listening judgments. INT4's looser cosine gate does not establish that it is better than INT8. Tower neighbor results use restricted shape/mask groups and cannot be compared directly with the global final-embedding search.

Upstream and FP32 each pass 9/10 synth triples. INT8 passes 6/10 in both approaches; INT4 passes 7/10 for the tower and 9/10 for final embeddings. The reports preserve these failures, and the full validation commands return a failing exit status. All artifacts remain experimental; native preprocessing and Grain Elevator integration remain deferred.

Tower — FULL CORPUS

Dataset yojul/one-shot-hip-hop-drums at 07be604e729fcf8f7904bca5328d6b09b1ceed7b. Attempted 19,673/19,673 rows; successful 19,670; input errors 3; inference errors 0.

Comparison	Clips	Within tolerance	Mean cosine	Min cosine	P95 clip max abs	Worst abs	P95 RMSE
upstream -> fp32	19,670	16,322/19,670	1	1	0.00204446	0.0177279	8.16176e-05
fp32 -> int8	19,670	15,373/19,670	0.983244	0.796764	34.8828	52.3205	2.50718
fp32 -> int4	19,670	19,670/19,670	0.985616	0.829793	31.8087	48.6736	2.05439

Quantization vs FP32	Top-1 eligible queries	Top-1 agreement	Top-10 eligible queries	Mean overlap@10
INT8	19,571	0.586071	18,745	0.730083
INT4	19,571	0.643197	18,745	0.765831

Synth continuity	Passing triples	Meets 90% expectation
upstream	9/10	Yes
fp32	9/10	Yes
int8	6/10	No
int4	7/10	No

Raw-tower neighbor searches use only identical shapes and masks; no pooling is applied. Final embeddings search all successful clips. Self matches are excluded; exact ties use dataset row order. Overlap@10 includes only groups with at least 11 clips.

FP32 tolerance: cosine >= 0.99999 and max absolute error <= 0.0005. INT8/INT4 cosine tolerances are 0.97/0.80, matching the existing smoke gates; they are numerical diagnostics, not perceptual acceptance criteria.

Python-versus-native preprocessing parity is deferred and was not measured here.

Embedding — FULL CORPUS

Dataset yojul/one-shot-hip-hop-drums at 07be604e729fcf8f7904bca5328d6b09b1ceed7b. Attempted 19,673/19,673 rows; successful 19,670; input errors 3; inference errors 0.

Comparison	Clips	Within tolerance	Mean cosine	Min cosine	P95 clip max abs	Worst abs	P95 RMSE
upstream -> fp32	19,670	19,670/19,670	1	1	1.01617e-06	3.64394e-05	3.04553e-07
fp32 -> int8	19,670	19,064/19,670	0.986291	0.940703	0.0389784	0.112514	0.010319
fp32 -> int4	19,670	19,670/19,670	0.962251	0.898048	0.0521914	0.126151	0.015006

Quantization vs FP32	Top-1 eligible queries	Top-1 agreement	Top-10 eligible queries	Mean overlap@10
INT8	19,670	0.297051	19,670	0.425338
INT4	19,670	0.323488	19,670	0.424408

Synth continuity	Passing triples	Meets 90% expectation
upstream	9/10	Yes
fp32	9/10	Yes
int8	6/10	No
int4	9/10	Yes

Raw-tower neighbor searches use only identical shapes and masks; no pooling is applied. Final embeddings search all successful clips. Self matches are excluded; exact ties use dataset row order. Overlap@10 includes only groups with at least 11 clips.

FP32 tolerance: cosine >= 0.99999 and max absolute error <= 0.0005. INT8/INT4 cosine tolerances are 0.97/0.80, matching the existing smoke gates; they are numerical diagnostics, not perceptual acceptance criteria.

Python-versus-native preprocessing parity is deferred and was not measured here.

SHA-256 hashes

Filename	SHA-256
audio_embedding.onnx	41ae9b77cf7c0add5843887ce4ed44d71e8d3558959226868faf25b870642f0b
audio_embedding.data	a9e6f1aee2489d02eb7d4953e9bc2bb2e03cc557bf4f01654f1af9dcf5d7d4dd
audio_embedding_int8.onnx	41441cf3cccd9c755ca7efbdb6b69c726632a516d200d8230daa030e9fb840bd
audio_embedding_int4.onnx	d4c269c7a024d1d2d495d121a2b608c4612833fa52e5eda5d7a9345ee493a3fe
audio_tower.onnx	08f2eb80d794d4cdef5e8d16224ac1685893cad2e6cb7e68d8bed614a7c3a023
audio_tower_int8.onnx	0fee17fc7ba12788122634d1780ae15bcce48b6c2e3943e47d98e92b8cace6ae
audio_tower_int4.onnx	b0a3d2e2a0f3177322bab1947577c2e7abd408e063e3c6d7619f8aef8a96eded

The FP32 embedding graph stores small Constant tensors inline; only audio_embedding.data is required alongside it. Its hash was updated on 2026-10-12 to remove references to 12 exporter tensor sidecars, without changing their tensor values or the consolidated weights file.