chordai/mms-fa-aligner-coreml

MMS-FA Multilingual Forced Aligner (CoreML)

On-device CTC forced aligner for word-level lyric timestamps, multilingual (1000+ languages via romanization). Alignment half of a WhisperX-style pipeline: a transcript is aligned to audio by Viterbi-decoding CTC emissions.

Converted from torchaudio.pipelines.MMS_FA (Meta MMS ctc_alignment_mling_uroman).

⚠️ LICENSE: CC-BY-NC-4.0 (non-commercial). Inherited from Meta's MMS release. Hosting/attribution is fine; shipping in a commercial app requires resolving the non-commercial restriction first. For commercial production use the Apache-2.0 wav2vec2-base960h-aligner-coreml (English) instead, or obtain a commercial MMS license / a differently-licensed multilingual model.

I/O contract

input	waveform float32 [1, 960000] — a fixed 60 s window @ 16 kHz mono
output	emissions float32 [1, 2999, 28] — log-probs; 20 ms / frame (stride 320)
dict	mms_fa_dict.json = {label: index}, blank -=0, then lowercase romanized a–z + apostrophe (no word-separator)

Text must be romanized (uroman) before tokenization for non-Latin scripts. Long audio: overlapping 60 s windows (hop 50 s), center-crop, single CTC Viterbi over concatenated emissions. A synthetic star (garbage) token wraps the transcript to absorb instrumental intros/outros — align on a separated vocal stem for heavily-produced tracks. Precision FLOAT16; target iOS 17.