FM-1976/eg2-voice-reader-a4

EG2 Voice reader (a4) — a 13.3 M-parameter CTC head for EmbeddingGemma 2 audio

A tiny CTC reader that turns EmbeddingGemma 2 audio soft tokens into English text — trained from scratch, fully on one laptop CPU (63.6 h wall, no GPU), as a local, CPU-only reproduction of the eg2-transcribe recipe (Apache-2.0).

Test WER (greedy, no language model): 6.65 % clean / 15.05 % other — slightly better than the upstream released v1 reader (6.81 % / 15.10 %; numbers as reported upstream, same greedy protocol).

	test-clean	test-other
released v1 reader	6.81	15.10
this model (a4, epoch 7)	6.65	15.05

You also need the EmbeddingGemma 2 base model — a one-time ≈1.5 GB download. This repo ships only the reader. See Usage.

How it works

16 kHz mono waveform
   └─ EmbeddingGemma 2 audio encoder  (base model, loaded separately)
        -> soft tokens [T, 512] float32, 25 tokens per second
   └─ this reader (13,268,994 params)
        -> unit vectors [T, 768] + per-frame blank logit
        -> score = scale · cos(vector, word_table[w])   (scale = 76.49, learned)
        -> per-frame argmax over 200,000 words + blank
   └─ greedy CTC read-out (merge repeats, drop blanks) -> words

The reader is a pre-LN bidirectional transformer: LayerNorm → Linear 512→512 → 4 blocks (d 512, 8 heads, FFN 2048, GELU, rotary positions base 10000, full O(T²) attention over the token window) → LayerNorm → Linear 512→768, plus a linear blank head. Word vectors are the EmbeddingGemma 2 text-embedding space: word_table.npy is [200000, 768] (stored float16, L2-normalized at load), the same table the base model uses for text embeddings — which is why scores are cosine similarities and why the vocabulary is exactly the base model's vocabulary.

Usage

pip install torch numpy soundfile safetensors "transformers>=5.19" scipy
huggingface-cli download google/embeddinggemma-2              # one-time, ≈1.5 GB, then cached
git clone https://huggingface.co/FM-1976/eg2-voice-reader-a4   # this repo (weights included)
python example.py clip.flac --eg2 google/embeddinggemma-2     # or --eg2 /path/to/local/snapshot

example.py is standalone (self-contained model definition, no repo imports) and prints the transcript plus encode/decode timings. Everything runs locally; nothing leaves the machine.

• Fully offline after setup — pass a local snapshot to --eg2 or set HF_HUB_OFFLINE=1. No serving infrastructure: one Python process, load once (~2.5 GB RAM: float32 encoder + 200k×768 word table + reader), transcribe files in a loop.
• CPU throughput: encoder ≈ 60 ms per second of audio, reader ≈ 0.8 s per ~10 s window; a 54-minute file took 8.3 min end to end (≈ 6.5× realtime).
• Deployments that include the base weights follow Google's Gemma terms of use; the reader itself is Apache-2.0.

Example outputs (from samples/, LibriSpeech test-clean clips):

sample transcripts — reference vs model

clip	reference	model output
1089-134686-0000	… fat mutton pieces … fattened sauce	… fat muttered pieces … fatten sauce
1089-134686-0002	… the squalid quarter of the brothels	… the squalid quarter of the broaches
1188-133604-0000	… four men titian holbein …	… four men tion holbein …

Full lines in samples/transcripts.tsv. These are the reader's characteristic errors: rare words and proper nouns inside a fixed 200k vocabulary, decoded greedily with no language model.

Long audio

The reader's attention is quadratic in the token count, so long files are transcribed with chunked decoding: overlapping 25 s windows (2.5 s of context on each side, seams at the overlap midpoints), per-frame CTC streams concatenated across windows and collapsed once — a word spanning a seam merges through the CTC repeat rule instead of duplicating. On a 54-minute single-speaker test file this reached 5.7 % WER in 8.3 min wall on CPU (163 windows, ≈ 6.5× realtime), better than the per-utterance test-clean figure — long, coherent single-speaker audio is this setup's sweet spot.

Training recipe (fully reproducible on CPU)

data	LibriSpeech train-clean-100, in-vocabulary utterances only: 26,422 utts / 92.8 h
dev / early stopping	dev-clean WER, best epoch kept
objective	CTC over 200,001 classes (200k words + blank), gathered-column implementation
optimizer	AdamW, batch 16, lr 3e-4 → 0 linear with 500-step warmup, weight decay 0.01, grad clip 1.0
schedule	8 epochs × 1652 steps = 13,216 steps; epoch 7 kept (dev WER 6.49 %)
compute	63.6 h wall on one Zen-4 class laptop CPU (8C/16T), torch 2.14.1+cpu, float32

Dev WER plateaus at epoch 6–8 (6.51 / 6.49 / 6.49 %); epoch 7 had the best dev score and is what reader.safetensors contains. Per-epoch transcripts and the full training log are in eval/.

Files

file	what
reader.safetensors	the trained reader weights (50.6 MB, loads with strict=True)
config.json	architecture + decode contract, training/eval summary, provenance
vocab.txt	200,000 words, one per line, index = class id
word_table.npy	[200000, 768] float16 word vectors (L2-normalize after loading)
example.py	standalone end-to-end demo (see Usage)
eval/	wer.json (full confusion counts), per-set transcripts, training metrics
assets/	the charts above
samples/	3 test-clean clips + reference/model transcripts

Not included: the EmbeddingGemma 2 base model (audio encoder) — load it from google/embeddinggemma-2 or a local snapshot.

Limitations

• No language model — greedy CTC only; expect characteristic proper-noun/rare-word errors.
• Fixed 200k vocabulary — out-of-vocabulary words can never be emitted (they count as errors).
• English, read-speech domain — trained and evaluated on LibriSpeech (16 kHz mono, audiobook reading); noisy, spontaneous, or multi-speaker audio is untested territory.
• Lowercase, unpunctuated output — normalization is lower case + whitespace collapse, nothing else.
• The encoder processes windows bidirectionally; for audio longer than ~a minute, use chunked decoding with context overlap (see above) rather than one giant pass.

Provenance and credits

• Recipe and training/eval code adapted from john-rocky/eg2-transcribe (Apache-2.0) — this is an independent CPU-only reproduction of its released v1 reader, not a copy of its weights. The upstream released readers and their reported numbers live on mlboydaisuke/EmbeddingGemma-2-Transcribe-Reader.
• Base model: google/embeddinggemma-2 (Apache-2.0) — audio encoder, vocabulary, and word vectors.
• Data: LibriSpeech (CC-BY-4.0, Vassil Panayotov et al.).
• Trained and packaged as part of the eg2-voice local-on-device speech project (reproduction log and app: see the project repository).