CavaAlign v1
Audio + text โ word timings, small enough for a phone. 35 languages. 16 MB.
Give a recording and its text; get back when every word is spoken.
Highlights
- Precise. On 70 LibriSpeech test speakers, 88% of word starts land within 50 ms of forced-alignment truth. That is close to the 89.5% where two independent references agree with each other.
- Small and fast. 16 MB for all thirty-five languages, int8 weights, standard ONNX ops. Five minutes of audio align in 12 s on a 4-core laptop CPU, with 0.4 GB peak memory.
- Every word timed. Each word in the text gets a time, and 0.3% are off by more than 250 ms. Speech recognisers with word timestamps leave 5โ20% of words untimed or far off.
- Built for books. Handles text that starts before the audio (front matter, chapter titles), unread footnotes and spoken announcements, without transcribing.
Quick start
pip install onnxruntime numpy soundfile regex num2words
hf download cava-reader/cavaalign-v1 --local-dir cavaalign-v1
python cavaalign-v1/align.py speech.wav speech.txt --lang en
import sys; sys.path.insert(0, "cavaalign-v1")
import soundfile as sf
from align import load, align
sess, cfg = load()
audio, sr = sf.read("speech.wav", dtype="float32") # 16 kHz mono
with open("speech.txt", encoding="utf-8") as f:
text = f.read()
for w in align(sess, cfg, audio, text, lang="de")[:5]:
print(f"{w['start']:7.2f} {w['end']:7.2f} {w['word']}")
pip install numba is optional and makes long recordings about 4ร faster. Audio of any length
works: align.py runs the model in short overlapping windows, so memory stays flat.
num2words is used to read digits in languages other than English; regex is optional.
Android (Kotlin)
kotlin/ contains the same romanizer and aligner as the Python code, plus an ONNX Runtime
wrapper. Given the same model output, the Kotlin and Python aligners produce the same timings.
- Add
com.microsoft.onnxruntime:onnxruntime-android(1.17 or newer) and copy the three files inkotlin/into your project (packagedev.micr0.cavaalign). - Ship
cavaalign.onnxin your assets or download it on first use. - Align:
val model = CavaAlignModel(context.assets.open("cavaalign.onnx").readBytes())
val emissions = model.logProbs(audio) // FloatArray, 16 kHz mono, about [-1, 1]
val result = CavaAligner.fromText(text).align(emissions)
for (w in result.words) {
// w.wordIndex, w.startSec, w.endSec, w.measured
}
wordIndex counts words in the text you passed in: runs of letters and digits, with
apostrophes inside a word ("don't" is one word, "well-known" is two). measured is false for
words the audio did not cover. Their times are inherited from the previous word.
The aligner's memory grows with the length of the recording. Align long books one chapter or section at a time, passing the text for that stretch of audio.
Benchmark
English, 70 LibriSpeech test-clean / test-other speakers (about 60 s each, 12,195 words). Truth is Montreal Forced Aligner word starts. Every system gets the exact text and runs the way its users run it. Words a system leaves untimed count as misses. Systems with a constant timing delay are scored with that delay removed, so they are not penalised for a fixed offset.
| Model | Size | Words within 50 ms | Off by > 250 ms or untimed |
|---|---|---|---|
| CavaAlign v1 | 16 MB | 88.0% | 0.3% |
| wav2vec2-base (WhisperX) | 378 MB | 84.4% | 0.4% |
| wav2vec2-large | 1.26 GB | 83.9% | 0.3% |
| HuBERT-large | 1.26 GB | 83.9% | 0.3% |
| data2vec-audio-base | 373 MB | 83.8% | 0.4% |
| PocketSphinx | 10 MB | 83.8% | 8.6% |
| Vosk en lgraph | 214 MB | 81.4% | 9.7% |
| Vosk small-en | 71 MB | 80.4% | 12.7% |
| MMS-300m aligner | 1.26 GB | 77.0% | 7.2% |
| NeMo Conformer-CTC S | 46 MB | 67.3% | 5.7% |
| NeMo Conformer-CTC M | 68 MB | 60.5% | 4.8% |
| Zipformer-CTC | 70 MB | 55.2% | 3.9% |
| Whisper small.en | 967 MB | 48.3% | 10.8% |
| Whisper tiny.en | 151 MB | 45.5% | 13.2% |
| NeMo Citrinet-512 | 38 MB | 44.0% | 7.0% |
| Whisper base.en | 290 MB | 42.4% | 19.9% |
Two independent MFA alignments of the same audio agree on 89.5% of words within 50 ms, so this benchmark cannot separate systems above roughly 90%. On the words where the two references agree with each other, CavaAlign v1 is within 50 ms on 95.1%.
Languages
One model covers all thirty-five. Text is romanized per word (Cyrillic, umlauts and accents โ aโz), so there is no per-language download. Listed: every language tested with 70% or more of word starts within 50 ms.
| Language | Words within 50 ms | > 250 ms off or untimed |
|---|---|---|
| Afrikaans โ | 79.8% | 5.6% |
| Belarusian | 74.2% | 11.1% |
| Bosnian | 92.9% | 1.5% |
| Bulgarian | 94.9% | 2.0% |
| Catalan โ | 85.8% | 3.4% |
| Croatian | 82.2% | 3.6% |
| Czech | 90.8% | 1.9% |
| Danish โ | 82.2% | 3.9% |
| Dutch | 96.1% | 0.1% |
| English | 89.7% | 0.1% |
| Estonian โ | 81.1% | 6.2% |
| Finnish โ | 90.9% | 2.6% |
| French | 97.3% | 0.0% |
| Galician โ | 90.6% | 2.2% |
| German | 96.0% | 0.0% |
| Hausa | 79.2% | 3.6% |
| Hungarian | 81.4% | 7.3% |
| Indonesian โ | 70.6% | 9.6% |
| Italian | 80.9% | 6.4% |
| Kyrgyz | 82.5% | 4.3% |
| Latvian โ | 89.4% | 1.9% |
| Lithuanian โ | 85.5% | 4.9% |
| Polish | 98.9% | 0.0% |
| Portuguese | 93.7% | 0.0% |
| Romanian | 79.4% | 7.2% |
| Russian | 92.6% | 1.1% |
| Serbian | 85.3% | 1.5% |
| Slovak โ | 87.0% | 1.9% |
| Slovenian โ | 90.2% | 3.9% |
| Spanish | 95.6% | 0.4% |
| Swedish | 89.5% | 1.5% |
| Turkish | 88.1% | 1.7% |
| Ukrainian | 84.4% | 5.1% |
| Vietnamese | 82.6% | 5.0% |
| Welsh โ | 75.0% | 7.6% |
Average over all thirty-five (plain mean of the rows): 86.7% of word starts within 50 ms, 3.4% more than 250 ms off or untimed. Read speech with MFA reference timing; words missing from MFA's pronunciation dictionaries get generated pronunciations (G2P), so every word is timed. โ MFA has no model for these languages, so their reference is an MFA model trained on Common Voice audio with spelling-based pronunciations. On Czech, which has both, CavaAlign scores 88.4% against that kind of reference and 90.8% against MFA's own.
Test audio: FLEURS for Afrikaans, Belarusian, Bosnian, Bulgarian, Catalan, Croatian, Czech, Danish, Estonian, Finnish, Galician, Hausa, Hungarian, Indonesian, Kyrgyz, Latvian, Lithuanian, Romanian, Serbian, Slovak, Slovenian, Swedish, Turkish, Ukrainian, Vietnamese and Welsh; LibriSpeech, Multilingual LibriSpeech and Russian LibriSpeech for Dutch, English, French, German, Italian, Polish, Portuguese, Russian and Spanish.
Model
| Architecture | raw-waveform front end + 8-layer streaming conformer (d 256), CTC head |
| Parameters | 14.6 M |
| Weights | int8 (dynamic quantisation), fp32 compute |
| Input | waveform float32 [batch, samples] at 16 kHz mono, lengths int64 [batch] |
| Output | log_probs [batch, frames, 29] at 50 frames/s, out_lengths |
| Vocabulary | blank, aโz, apostrophe, unk (vocab.json) |
| Aligner | monotonic dynamic program with text-skip, audio-skip and front-matter arcs (cavaalign/) |
Running it yourself:
- Feed the model 6.4 s windows with 3.2 s of preceding context, cut on 0.64 s chunk boundaries
counted from the start of the audio (
align.py::log_probs). Attention memory grows with the square of the input length, so whole chapters in one pass will not fit. - Create the ONNX Runtime session with the CPU memory arena and memory pattern disabled.
- Romanize the text before tokenizing (
cavaalign/romanize.py); a space maps to the blank id. - Frames are 20 ms (
frame_shift_secinalign_config.json).
Files
| file | |
|---|---|
cavaalign.onnx |
the model |
align.py, cavaalign/ |
Python reference: romanizer and aligner (numpy) |
align_config.json, vocab.json |
aligner settings, vocabulary, frame shift |
kotlin/ |
the same romanizer and aligner in Kotlin, plus an ONNX Runtime wrapper, for Android |
reference/ |
English, German and Russian clips with expected outputs, to check a port |
SHA256SUMS |
checksums (sha256sum -c SHA256SUMS) |
LICENSE |
GNU Affero General Public License v3 |
To check a port, run each reference/*.wav with its .txt and compare with .expected.json:
romanized_text must match exactly, log_probs_frames must be equal, and word times should match
to the frame (an occasional one-frame difference from int8 rounding on ARM is normal).
Limitations
- It aligns given text; it does not transcribe. Speech that is not in the text is skipped.
- Built for read speech (audiobooks).
- Precision drops when the text window is much wider than the audio. Hand it the text for the stretch being played, with modest slack.
- Digits are read with language-specific number words in Python; the Kotlin port uses English readings.
- Latin and Cyrillic scripts only. Turkish รง and รผ are romanized approximately (as "s" and "ue"), and Hausa's hooked letters (ษ ษ ฦ) are dropped; the words still align from their other letters.
Reference clips
The clips in reference/ are 30 s excerpts of LibriVox recordings:
en_*: LibriSpeech test-clean, speaker 1089, chapter 134686 (OpenSLR 12, CC BY 4.0).de_*: Multilingual LibriSpeech German, speaker 3363, book 2124 (OpenSLR 94, CC BY 4.0).ru_*: Russian LibriSpeech (OpenSLR 96, public domain in the USA).
License
Copyright (C) 2026 Cava Reader.
CavaAlign (the model weights in cavaalign.onnx and all code in this repository) is free
software: you can redistribute it and/or modify it under the terms of the GNU Affero General
Public License as published by the Free Software Foundation, either version 3 of the License,
or (at your option) any later version. See LICENSE. The reference clips keep their own
licenses, listed above.
It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.