ELF-Voice

One 106M-parameter transformer that does both speech recognition (speech → text) and zero-shot speech synthesis (text + a voice prompt → speech) with the same weights. It is built on ELF (Embedded Language Flows), a flow-matching language model, and initialised from embedded-language-flows/ELF-B-owt-torch.

Code, training and evaluation scripts: https://github.com/WanZhengyan/TTS-ELF

How it works, in one paragraph

Speech (as a 100-band mel spectrogram, 93.75 frames per second) and text (as frozen T5-small embeddings) sit side by side in one sequence of 1,648 positions. The model learns to turn noise into whichever part is missing. Give it audio and it writes the text. Give it text plus a few seconds of a speaker's voice and it writes the mel spectrogram, which Vocos turns into a waveform. For recognition, the audio also passes through a frozen Whisper-large-v3 encoder; for synthesis, a speaker x-vector of the voice prompt is added.

Checkpoints

Each folder holds ckpt.pt (EMA weights, training arguments and the character table) and info.json (training setup).

Folder	Synthesis	Recognition data	Synthesis data	Training
ms_unified_100k	multi-speaker, zero-shot (voice prompt + speaker x-vector)	LibriSpeech 960 h (886 h after dropping clips over 16 s)	LibriTTS train-clean-100/360 + train-other-500 at 24 kHz (479.5 h after the same filter)	100K steps, 16 recognition + 32 synthesis rows per step, 4 H100
dual_lj_100k	single speaker (LJSpeech voice), prompt-continued	LibriSpeech 960 h (same filter)	LJSpeech, all 13,100 clips (24 h)	100K steps, 16 recognition + 16 synthesis rows per step

The results below are for ms_unified_100k. Results for dual_lj_100k under the same protocols will be added here.

Results

Evaluated with the papers' own protocols. Synthesis is generated following F5-TTS's inference rules and scored with F5-TTS's official scoring code; our ground-truth row reproduces the one in F5-TTS's Table 9. Recognition uses ELF-S2T's sampler and its jiwer scoring.

Zero-shot TTS, LibriSpeech-PC test-clean (1,127 pairs, unseen speakers)

Model	WER ↓	Speaker similarity ↑	UTMOS ↑
ELF-Voice (this checkpoint)	12.5	0.239	2.19
F5-TTS small, 100K steps (paper)	29.5	0.53	3.78
F5-TTS small, 200K steps (paper)	4.58	0.59	4.07
Real recordings	2.22	0.695	4.10

This model saw about 10% of the audio F5-TTS small saw in its 100K steps. Intelligibility (WER) is already better than F5-TTS small at 100K steps. Voice similarity and audio quality are clearly worse.

Speech recognition, LibriSpeech test-clean (all 2,620 utterances)

Model	WER ↓
ELF-Voice (this checkpoint), ELF-S2T sampler	14.24
ELF-Voice (this checkpoint), our ODE sampler	13.68
ELF-S2T, ELF-B (paper, recognition only)	10.50

Use

import torch
from eval_lj import load_model          # from the GitHub repo
ck = torch.load("ms_unified_100k/ckpt.pt", map_location="cpu", weights_only=False)
model, vocab = load_model(ck, torch.device("cuda"))

Generation and evaluation scripts are in the GitHub repo (gen_f5_protocol.py for synthesis, eval_s2t_full.py for recognition). The guidance mode is picked from the checkpoint automatically: ms_unified_100k uses joint classifier-free guidance (text, prompt and x-vector dropped together) with w = 3 and Sway sampling −1; dual_lj_100k uses text-only guidance with w = 2.

Limitations

• English only, read audiobook speech. dual_lj_100k speaks only in the LJSpeech voice.
• Zero-shot voice similarity is low: the voice is roughly right (pitch, gender) but does not sound like the reference speaker.
• Sequences hold at most 16.38 s of audio; longer utterances are cut at silences for recognition.
• Training recognition alongside synthesis costs synthesis quality: a synthesis-only control trained the same way reached WER 9.88, similarity 0.281 and UTMOS 2.73.

Training data

LibriSpeech and LibriTTS (both CC BY 4.0), LJSpeech (public domain).