VieGem-ASR 🇻🇳💎

Fast Vietnamese and Vietnamese–English speech recognition: a 359M-parameter streaming CTC model on Google's EmbeddingGemma 2 audio encoder. Inspired by Quan Nguyen's QUAN-Transcribe-1.0.

• Fast: about 3,500× real time on one H100 (batched, greedy); streams in 0.48 s chunks with no look-ahead.
• Vietnamese: mean WER 6.34 over 8 common test sets (5.29 with the included n-gram LM).
• Code-switching: ViMedCSS WER 14.3; 66% of English terms written exactly.

Demo

Results

Vietnamese test sets

WER %, lower is better, greedy unless marked "+ LM". VieGem-ASR, QUAN-Transcribe-1.0 and Qwen3-ASR-0.6B were run by us with one normalizer; † = reported in the QUAN-Transcribe-1.0 model card (our scorer matches its numbers within 0.1).

Test set	What it contains	VieGem-ASR	VieGem-ASR + LM	QUAN-Transcribe-1.0	Qwen3-ASR-0.6B	Gemini-Flash-3.8 †	ElevenLabs Scribe v2 †	PhoWhisper-large †	Whisper large-v3 †
VIVOS	read speech, quiet room	6.63	4.78	7.52	10.22	9.36	9.14	4.52	9.18
FLEURS vi	read Wikipedia sentences	7.33	5.85	7.88	8.40	5.90	5.60	5.31	7.09
Common Voice 17	read, crowd-sourced microphones	8.61	6.21	9.69	16.07	10.30	10.50	9.12	18.43
PhoAudiobook	audiobooks	1.39	1.04	1.39	3.23	1.60	1.45	0.87	1.85
LSVSC	large mixed-domain corpus	5.16	4.71	5.35	7.78	5.70	6.25	6.81	13.96
Bud500	mixed domains, tightly cut clips	3.08	2.75	1.82	6.79	1.80	3.15	9.51	64.03
GigaSpeech2 vi	YouTube, spontaneous	9.50	8.52	11.60	11.37	10.70	10.50	10.14	14.19
ViMD	dialects of 63 provinces	9.05	8.46	9.84	13.98	10.00	10.05	11.38	15.56
Mean of the 8		6.34	5.29	6.89	9.73	6.92	7.08	7.21	18.04

Bold = best greedy result per row. Text is lowercased, without punctuation, numbers as words.

Vietnamese–English code-switching (ViMedCSS)

System	Test WER	Hard WER
Whisper-large-v3 (paper)	34.47	39.35
PhoWhisper-large (paper)	31.24	37.37
PhoWhisper-small + AG, fine-tuned (paper)	23.67	33.73
QUAN-Transcribe-1.0	22.26	26.56
VieGem-ASR	14.30	21.08
VieGem-ASR + LM	13.00	19.84

"Paper" = numbers from the ViMedCSS paper (its own normalization, so approximate).

Streaming and speed

• Streaming (0.48 s chunks): VIVOS 8.73, CV17 10.84, FLEURS 9.02, LSVSC 5.84, ViMD 10.00, ViMedCSS 15.99.
• A 10-minute video is transcribed in 0.49 s on one H100 (median of 20 runs, ~1,220× real time).

Usage

import torch, soundfile as sf
from transformers import AutoTokenizer
from asr.model import CTCModel, ModelConfig, greedy_decode     # asr/ folder of this repo
from asr.text import Units

ck = torch.load("viegem_asr.pt", map_location="cpu")
cfg = ModelConfig.from_dict(ck["model_config"])
units = Units(ck["gemma_ids"])
tok = AutoTokenizer.from_pretrained(cfg.encoder_id)               # google/embeddinggemma-2 (accept its terms)
model = CTCModel(cfg, units.gemma_ids)
model.load_state_dict(ck["model"], strict=False)
model = model.cuda().eval()

wav, sr = sf.read("audio.wav", dtype="float32")                  # 16 kHz mono, ≤ 30 s
x = torch.from_numpy(wav)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    logits, lens = model(x, torch.tensor([x.shape[1]], device="cuda"), streaming=False)  # streaming=True: 0.48 s chunks
print(units.decode(tok, greedy_decode(logits, lens)[0]))

With the LM (pip install kenlm):

from asr.lm_decode import LMDecoder

words = open("lm/viegem_lm.unigrams.txt", encoding="utf-8").read().split()
dec = LMDecoder("lm/viegem_lm.bin", unigrams=words, alpha=0.35, beta=1.5)
pieces = ["", ""] + tok.convert_ids_to_tokens(units.gemma_ids)
lp = torch.log_softmax(logits[0, : lens[0]].float(), -1).cpu().numpy()
print(units.decode(tok, dec.decode(lp, pieces, beam_width=32)))

Model

• Encoder: EmbeddingGemma 2 audio tower (12 conformer layers, 25 frames/s); top 4 layers fine-tuned.
• Reader: layer mix + 6 transformer blocks (d=768) with chunked attention, so one model serves offline and streaming.
• Head: 21,031 Gemma tokens (Vietnamese, English, single characters), initialized from Gemma's text embeddings.
• Files: viegem_asr.pt (fp16 weights, 311 MB), asr/ (code), lm/ (4-gram KenLM from our training transcripts, 71 MB), demo/.

Training data

About 5,800 hours, every set filtered against all test and dev sentences.

Type	Sources	Hours
Audiobooks	PhoAudiobook, InfoRe2	~1,900
Social media and YouTube	VietSpeech, viVoice, GigaSpeech2 (subset), VietSuperSpeech	~2,820
Mixed domains	Bud500, VLSP2020, LSVSC, ViMD, FPT FOSD, InfoRe1, FLEURS, Common Voice 17	~780
English	LibriSpeech-100	~100
Vietnamese–English (medical)	ViMedCSS	~23

Limitations

• Output is lowercase, without punctuation; numbers are written as words.
• Clips cut in the middle of a word (as in Bud500) lose the last word more often.
• English is meant for terms mixed into Vietnamese; plain English is weak (LibriSpeech test-clean 10.7).

License

Non-commercial research use (CC-BY-NC-SA-4.0), because the training data includes Bud500, viVoice and PhoAudiobook. Use must also follow the Gemma Terms of Use and Prohibited Use Policy.

Credits

Google DeepMind (EmbeddingGemma 2); Quan Nguyen (QUAN-Transcribe-1.0, the inspiration for this work); the authors of VIVOS, Common Voice, FLEURS, LSVSC, ViMD, Bud500, PhoAudiobook, VietSpeech, viVoice, GigaSpeech2, VietSuperSpeech, VLSP2020, FPT FOSD, InfoRe, ViMedCSS and LibriSpeech. ViMedCSS baselines: Nguyen et al., LREC 2026.