EmbeddingGemma 2 — Persian speech ↔ text

google/embeddinggemma-2 fine-tuned so that a Persian speech clip and its transcription land next to each other in the embedding space.

Use it to search Persian audio with text, match a spoken utterance to the closest sentence or intent in a list you provide, or find duplicate/similar audio. It is not a speech-to-text model. It ranks candidate texts you provide and cannot write out sentences it hasn't been given. For transcription use a generative ASR model (e.g. Whisper).

Results

Task: given a clip, rank every distinct sentence in the test set; the metric is the share of clips whose own transcription ranks first (R@1). Sentences are deduplicated after Persian normalization (ي/ی, ك/ک, ZWNJ, diacritics, punctuation).

Test set	Clips / candidate sentences	Base R@1	Fine-tuned R@1	Fine-tuned R@5	Text→audio R@1 (base → ft)
Common Voice 17 fa test (speakers unseen in training)	10,558 / 10,457	64.0%	98.5%	99.3%	74.1% → 98.0%
YouTube, 82 videos fully held out	1,326 / 1,326	93.9%	99.5%	99.7%	93.6% → 98.9%
Filimo test (5k sample, movie dialogue, ~2 s clips)	5,000 / 4,856	34.4%	90.7%	94.9%	46.8% → 89.7%
FLEURS fa test	871 / 324	99.2%	100%	100%	100% → 100%

On clips whose sentence never occurs in the training data: Common Voice 98.5%, YouTube 99.5%, filimo 92.8% (base: 64.2 / 93.9 / 34.7). About 28% of filimo test sentences also appear in filimo train. Those are mostly short stock phrases such as «بله» or «سلام», and they score lower than novel sentences, so the overlap isn't inflating the filimo number.

The model still works outside Persian. Zero-shot English Speech Commands (35 words) is 91.1%, vs 93.0% for the base model.

Usage

import torch
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MohammadGholizadeh/embeddinggemma-2-persian-speech",
                            model_kwargs={"torch_dtype": torch.bfloat16},   # bf16 or fp32, never fp16
                            config_kwargs={"vision_config": None})          # audio + text only

audio = model.encode(["clip1.wav", "clip2.wav"])        # paths, or {"array": np.ndarray, "sampling_rate": 16000}
texts = model.encode(["سلام، حالت چطوره؟", "قیمت این محصول چنده؟"], prompt_name="SearchQuery")
print(model.similarity(audio, texts))

Audio: mono, 16 kHz, no prompt. Trained on 0.5–20 s clips.
Text: use prompt_name="SearchQuery", the prompt used in training. Normalizing ي→ی and ك→ک in your texts helps.

Training

Data: 479k (audio, transcription) pairs, ~534 h. Filimo train (322k), YouTube (128k, held-out videos excluded), Common Voice 17 fa train (29k). All resampled to 16 kHz.
Loss: CachedMultipleNegativesRankingLoss (audio anchor → transcription positive, in-batch negatives), batch 256 with no duplicate texts per batch, mini-batch 32.
Optimization: 1 epoch (1,872 steps), lr 2e-5 cosine with 5% warmup, bf16 autocast over fp32 weights. About 3 h on one GH200.
Validation R@1 (1k-sentence pools) plateaued after ~1,000 steps at 99.8% CV / 99.6% YouTube / 93.2% filimo.

Limitations

Retrieval and matching only; no transcription.
Short, generic utterances (“بله”, “نه”) are inherently ambiguous across speakers and recordings.
Evaluated on read speech, YouTube talks and movie dialogue. Noisy phone-call audio and strong dialects weren't tested.
General text–text retrieval quality after fine-tuning wasn't measured.