TembraAIPlatform/tembra-georgian-asr

Tembra Georgian ASR

Speech-to-text for Georgian (ქართული) from Tembra AI — November 2026 model.

• Lowest average error rate on the public Georgian benchmarks (FLEURS + Common Voice): 6.57 % WER, vs ElevenLabs Scribe v2 6.86 %, Google Chirp 3 7.11 %, NVIDIA stt_ka 7.71 %, Meta Omnilingual ASR 17.16 %, Meta MMS 23.18 %, OpenAI Whisper large-v3 67.76 %.
• Common Voice 22: 4.48 % WER — 31 % fewer errors than ElevenLabs Scribe v2 (6.45 %) and 41 % fewer than Google Chirp 3 (7.66 %). On the 1,501 test clips that are absent from NVIDIA's published training data, Tembra has the lowest WER of all systems: 4.14 % vs 4.69 % (NVIDIA), 5.75 % (ElevenLabs), 7.01 % (Google).
• Real-world speech (Georgian YouTube — lectures, interviews, podcasts, news): 11.55 % WER — on par with ElevenLabs Scribe v2 (11.00 %, the difference is within the confidence interval), 28 % fewer errors than Google Chirp 3 (16.13 %) and less than half of NVIDIA stt_ka (24.11 %).
• FLEURS (read sentences, clean studio-like audio): 8.66 % WER; Google Chirp 3 (6.56 %) and ElevenLabs Scribe v2 (7.27 %) are ahead on this set.
• 22 % fewer errors than our previous Georgian release (October 2026), averaged over the three test sets.

Results

Word error rate, %, lower is better; 95 % confidence intervals in parentheses. All systems transcribed the same audio files and were scored with the same text normalization. Systems are sorted by the average over the two public benchmarks; «—» — not evaluated.

system	FLEURS	Common Voice 22	Common Voice 22, clips unseen by NVIDIA training	YouTube, real-world	Average FLEURS + CV
Tembra Georgian ASR (2026-11)	8.66 (7.8–9.5)	4.48 (3.9–5.1)	4.14 (3.6–4.7)	11.55 (8.7–15.6)	6.57 (6.1–7.1)
ElevenLabs Scribe v2 (API)	7.27 (6.6–8.0)	6.45 (5.7–7.3)	5.75 (5.1–6.5)	11.00 (8.6–14.5)	6.86 (6.3–7.4)
Google Chirp 3 (Cloud Speech-to-Text v2 API)	6.56 (5.8–7.3)	7.66 (6.7–8.7)	7.01 (6.1–7.9)	16.13 (13.7–18.6)	7.11 (6.5–7.7)
NVIDIA stt_ka_fastconformer_hybrid_large_pc	11.12 (10.2–12.1)	4.29 (3.8–4.9)	4.69 (4.1–5.3)	24.11 (16.4–34.9)	7.71 (7.2–8.3)
Meta Omnilingual ASR CTC 1B v2	17.00 (15.9–18.1)	17.33 (16.2–18.6)	16.74 (15.5–18.0)	19.89 (14.6–27.9)	17.16 (16.4–18.0)
Meta MMS-1B-all (kat adapter)	23.57 (22.4–24.8)	22.78 (21.5–24.1)	21.61 (20.3–22.9)	33.48 (25.6–44.7)	23.18 (22.3–24.1)
OpenAI Whisper large-v3	66.53 (64.8–68.1)	69.00 (67.4–70.6)	67.90 (66.2–69.6)	—	67.76 (66.6–68.9)
Tembra Georgian ASR (2026-10, previous release)	10.09 (9.2–10.9)	6.71 (6.1–7.4)	6.98 (6.3–7.7)	14.84 (11.1–20.0)	8.40 (7.9–8.9)

test set	what it is	size
FLEURS	google/fleurs, config ka_ge, test split: sentences from the FLoRes benchmark read aloud, clean audio	725 of 979 utterances, 2.16 h, 259 sentences
Common Voice 22	Common Voice 22.0 ka test split: crowd-sourced read speech recorded on volunteers' own microphones	random sample of 2,000 of 13,104 clips, 3.28 h, 699 speakers
Common Voice 22, unseen by NVIDIA training	the same sample without the clips and sentences that occur in the Common Voice 17 train/dev/other portions used to train NVIDIA stt_ka	1,501 clips
YouTube, real-world	our internal test: lectures, interviews, podcasts, news and talk shows from Georgian YouTube; references are the channels' own manually created subtitles; none of these channels is in our training data	787 segments, 2.37 h, 14 videos from 7 channels

How we evaluated

• Normalization (same for every system): Georgian letters only — Mtavruli and other historical capital forms folded to Mkhedruli, apostrophes removed, all other punctuation replaced by spaces, whitespace collapsed. WER and CER via jiwer.
• Numbers. Systems write numbers differently (digits vs words, Georgian case endings such as «მე-5»), so — as in other multilingual evaluations — utterances whose reference contains digits or Latin-script words are not scored. This removes 254 of 979 FLEURS utterances and none of the Common Voice sample; the rule is the same for all systems.
• Confidence intervals: bootstrap with 2,000 resamples over groups — sentence id for FLEURS (several speakers read each sentence), speaker for Common Voice, video for YouTube. The average column resamples both test sets.
• Commercial APIs were called once per file on 5–6 November 2026 with the language set to Georgian and every other option at its default: ElevenLabs speech-to-text model_id=scribe_v2, language_code=kat, tag_audio_events=false; Google Cloud Speech-to-Text v2 chirp_3, synchronous Recognize, location us, language_codes=["ka-GE"], automatic decoding.
• Open models were run locally with default decoding: NVIDIA stt_ka (default decoder, greedy), Meta Omnilingual ASR CTC 1B v2 and MMS-1B-all (CTC argmax; MMS with the kat adapter), OpenAI Whisper large-v3 (transformers, language="georgian", task="transcribe", greedy, 30-second windows).
• Tembra Georgian ASR: one model with two decoding profiles, read speech and conversational. For each benchmark the profile was chosen on that benchmark's own dev split (FLEURS dev, Common Voice dev, YouTube dev) — never on test data: read speech for FLEURS and Common Voice, conversational for YouTube.

Fairness and data hygiene

• No test audio and no test text was used to train or tune Tembra Georgian ASR. Training transcripts that match any test or dev sentence of these benchmarks were removed, and text resources were also filtered for any 8 consecutive words shared with a test or dev sentence.
• NVIDIA stt_ka was trained on Common Voice 17 train, dev and "other" (per its model card). 10.5 % of the clips in our Common Voice 22 test sample were in Common Voice 17 "other", and 24.9 % of its sentences occur in those portions. The "unseen" column scores every system on the remaining 1,501 clips.
• The training data of the commercial APIs is not public; they may or may not have seen these public test sets.
• Common Voice is scored on a 2,000-clip random sample (fixed seed) of the 13,104-clip test split to keep API costs reasonable; the clip list is in eval/.
• Whisper large-v3 often repeats a phrase in a loop on Georgian audio; such output is scored as is.

Contact

Tembra AI — https://tembra.ai