CavaAlign v1

Audio + text โ†’ word timings, small enough for a phone. 35 languages. 16 MB.

Give a recording and its text; get back when every word is spoken.

16 MB 35 languages ONNX Runtime 20 ms frames

CavaAlign v1: 16 MB, 88.0% of word starts within 50 ms, 0.3% of words more than 250 ms off or untimed, compared with wav2vec2-base, PocketSphinx, Vosk small-en, MMS-300m, NeMo Conformer-CTC S and Whisper small.en

Highlights

  • Precise. On 70 LibriSpeech test speakers, 88% of word starts land within 50 ms of forced-alignment truth. That is close to the 89.5% where two independent references agree with each other.
  • Small and fast. 16 MB for all thirty-five languages, int8 weights, standard ONNX ops. Five minutes of audio align in 12 s on a 4-core laptop CPU, with 0.4 GB peak memory.
  • Every word timed. Each word in the text gets a time, and 0.3% are off by more than 250 ms. Speech recognisers with word timestamps leave 5โ€“20% of words untimed or far off.
  • Built for books. Handles text that starts before the audio (front matter, chapter titles), unread footnotes and spoken announcements, without transcribing.

Quick start

pip install onnxruntime numpy soundfile regex num2words
hf download cava-reader/cavaalign-v1 --local-dir cavaalign-v1
python cavaalign-v1/align.py speech.wav speech.txt --lang en
import sys; sys.path.insert(0, "cavaalign-v1")
import soundfile as sf
from align import load, align

sess, cfg = load()
audio, sr = sf.read("speech.wav", dtype="float32")   # 16 kHz mono
with open("speech.txt", encoding="utf-8") as f:
    text = f.read()
for w in align(sess, cfg, audio, text, lang="de")[:5]:
    print(f"{w['start']:7.2f}  {w['end']:7.2f}  {w['word']}")

pip install numba is optional and makes long recordings about 4ร— faster. Audio of any length works: align.py runs the model in short overlapping windows, so memory stays flat. num2words is used to read digits in languages other than English; regex is optional.

Android (Kotlin)

kotlin/ contains the same romanizer and aligner as the Python code, plus an ONNX Runtime wrapper. Given the same model output, the Kotlin and Python aligners produce the same timings.

  1. Add com.microsoft.onnxruntime:onnxruntime-android (1.17 or newer) and copy the three files in kotlin/ into your project (package dev.micr0.cavaalign).
  2. Ship cavaalign.onnx in your assets or download it on first use.
  3. Align:
val model = CavaAlignModel(context.assets.open("cavaalign.onnx").readBytes())
val emissions = model.logProbs(audio)          // FloatArray, 16 kHz mono, about [-1, 1]
val result = CavaAligner.fromText(text).align(emissions)
for (w in result.words) {
    // w.wordIndex, w.startSec, w.endSec, w.measured
}

wordIndex counts words in the text you passed in: runs of letters and digits, with apostrophes inside a word ("don't" is one word, "well-known" is two). measured is false for words the audio did not cover. Their times are inherited from the previous word.

The aligner's memory grows with the length of the recording. Align long books one chapter or section at a time, passing the text for that stretch of audio.

Benchmark

English, 70 LibriSpeech test-clean / test-other speakers (about 60 s each, 12,195 words). Truth is Montreal Forced Aligner word starts. Every system gets the exact text and runs the way its users run it. Words a system leaves untimed count as misses. Systems with a constant timing delay are scored with that delay removed, so they are not penalised for a fixed offset.

Model Size Words within 50 ms Off by > 250 ms or untimed
CavaAlign v1 16 MB 88.0% 0.3%
wav2vec2-base (WhisperX) 378 MB 84.4% 0.4%
wav2vec2-large 1.26 GB 83.9% 0.3%
HuBERT-large 1.26 GB 83.9% 0.3%
data2vec-audio-base 373 MB 83.8% 0.4%
PocketSphinx 10 MB 83.8% 8.6%
Vosk en lgraph 214 MB 81.4% 9.7%
Vosk small-en 71 MB 80.4% 12.7%
MMS-300m aligner 1.26 GB 77.0% 7.2%
NeMo Conformer-CTC S 46 MB 67.3% 5.7%
NeMo Conformer-CTC M 68 MB 60.5% 4.8%
Zipformer-CTC 70 MB 55.2% 3.9%
Whisper small.en 967 MB 48.3% 10.8%
Whisper tiny.en 151 MB 45.5% 13.2%
NeMo Citrinet-512 38 MB 44.0% 7.0%
Whisper base.en 290 MB 42.4% 19.9%

Two independent MFA alignments of the same audio agree on 89.5% of words within 50 ms, so this benchmark cannot separate systems above roughly 90%. On the words where the two references agree with each other, CavaAlign v1 is within 50 ms on 95.1%.

Languages

One model covers all thirty-five. Text is romanized per word (Cyrillic, umlauts and accents โ†’ aโ€“z), so there is no per-language download. Listed: every language tested with 70% or more of word starts within 50 ms.

Language Words within 50 ms > 250 ms off or untimed
Afrikaans โ€  79.8% 5.6%
Belarusian 74.2% 11.1%
Bosnian 92.9% 1.5%
Bulgarian 94.9% 2.0%
Catalan โ€  85.8% 3.4%
Croatian 82.2% 3.6%
Czech 90.8% 1.9%
Danish โ€  82.2% 3.9%
Dutch 96.1% 0.1%
English 89.7% 0.1%
Estonian โ€  81.1% 6.2%
Finnish โ€  90.9% 2.6%
French 97.3% 0.0%
Galician โ€  90.6% 2.2%
German 96.0% 0.0%
Hausa 79.2% 3.6%
Hungarian 81.4% 7.3%
Indonesian โ€  70.6% 9.6%
Italian 80.9% 6.4%
Kyrgyz 82.5% 4.3%
Latvian โ€  89.4% 1.9%
Lithuanian โ€  85.5% 4.9%
Polish 98.9% 0.0%
Portuguese 93.7% 0.0%
Romanian 79.4% 7.2%
Russian 92.6% 1.1%
Serbian 85.3% 1.5%
Slovak โ€  87.0% 1.9%
Slovenian โ€  90.2% 3.9%
Spanish 95.6% 0.4%
Swedish 89.5% 1.5%
Turkish 88.1% 1.7%
Ukrainian 84.4% 5.1%
Vietnamese 82.6% 5.0%
Welsh โ€  75.0% 7.6%

Average over all thirty-five (plain mean of the rows): 86.7% of word starts within 50 ms, 3.4% more than 250 ms off or untimed. Read speech with MFA reference timing; words missing from MFA's pronunciation dictionaries get generated pronunciations (G2P), so every word is timed. โ€  MFA has no model for these languages, so their reference is an MFA model trained on Common Voice audio with spelling-based pronunciations. On Czech, which has both, CavaAlign scores 88.4% against that kind of reference and 90.8% against MFA's own.

Test audio: FLEURS for Afrikaans, Belarusian, Bosnian, Bulgarian, Catalan, Croatian, Czech, Danish, Estonian, Finnish, Galician, Hausa, Hungarian, Indonesian, Kyrgyz, Latvian, Lithuanian, Romanian, Serbian, Slovak, Slovenian, Swedish, Turkish, Ukrainian, Vietnamese and Welsh; LibriSpeech, Multilingual LibriSpeech and Russian LibriSpeech for Dutch, English, French, German, Italian, Polish, Portuguese, Russian and Spanish.

Model

Architecture raw-waveform front end + 8-layer streaming conformer (d 256), CTC head
Parameters 14.6 M
Weights int8 (dynamic quantisation), fp32 compute
Input waveform float32 [batch, samples] at 16 kHz mono, lengths int64 [batch]
Output log_probs [batch, frames, 29] at 50 frames/s, out_lengths
Vocabulary blank, aโ€“z, apostrophe, unk (vocab.json)
Aligner monotonic dynamic program with text-skip, audio-skip and front-matter arcs (cavaalign/)

Running it yourself:

  • Feed the model 6.4 s windows with 3.2 s of preceding context, cut on 0.64 s chunk boundaries counted from the start of the audio (align.py::log_probs). Attention memory grows with the square of the input length, so whole chapters in one pass will not fit.
  • Create the ONNX Runtime session with the CPU memory arena and memory pattern disabled.
  • Romanize the text before tokenizing (cavaalign/romanize.py); a space maps to the blank id.
  • Frames are 20 ms (frame_shift_sec in align_config.json).

Files

file
cavaalign.onnx the model
align.py, cavaalign/ Python reference: romanizer and aligner (numpy)
align_config.json, vocab.json aligner settings, vocabulary, frame shift
kotlin/ the same romanizer and aligner in Kotlin, plus an ONNX Runtime wrapper, for Android
reference/ English, German and Russian clips with expected outputs, to check a port
SHA256SUMS checksums (sha256sum -c SHA256SUMS)
LICENSE GNU Affero General Public License v3

To check a port, run each reference/*.wav with its .txt and compare with .expected.json: romanized_text must match exactly, log_probs_frames must be equal, and word times should match to the frame (an occasional one-frame difference from int8 rounding on ARM is normal).

Limitations

  • It aligns given text; it does not transcribe. Speech that is not in the text is skipped.
  • Built for read speech (audiobooks).
  • Precision drops when the text window is much wider than the audio. Hand it the text for the stretch being played, with modest slack.
  • Digits are read with language-specific number words in Python; the Kotlin port uses English readings.
  • Latin and Cyrillic scripts only. Turkish รง and รผ are romanized approximately (as "s" and "ue"), and Hausa's hooked letters (ษ“ ษ— ฦ™) are dropped; the words still align from their other letters.

Reference clips

The clips in reference/ are 30 s excerpts of LibriVox recordings:

  • en_*: LibriSpeech test-clean, speaker 1089, chapter 134686 (OpenSLR 12, CC BY 4.0).
  • de_*: Multilingual LibriSpeech German, speaker 3363, book 2124 (OpenSLR 94, CC BY 4.0).
  • ru_*: Russian LibriSpeech (OpenSLR 96, public domain in the USA).

License

Copyright (C) 2026 Cava Reader.

CavaAlign (the model weights in cavaalign.onnx and all code in this repository) is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. See LICENSE. The reference clips keep their own licenses, listed above.

It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support