TurnWave — end-of-turn detection for voice agents

Decides whether a caller has finished speaking or is only pausing, so a voice agent neither interrupts them nor leaves an awkward silence. It replaces the fixed 300–700 ms silence timeout most pipelines still use.

Trained from scratch — no pretrained weights anywhere. A causal transformer (RoPE, RMSNorm, SwiGLU) over the transcript tail, and a CNN over log-mel spectrograms for prosody. Even the log-mel front end is hand-built on torch.stft, so there is no torchaudio or librosa dependency.

Benchmark

Scored by LiveKit's eot-bench harness on real human-to-agent conversation, using their code and published baselines. Lower is better; bold marks the best per column.

model	false cutoffs @300 ms ↓	@600 ms ↓	latency @5% cutoff ↓
VAD baseline	55.6%	21.7%	1600 ms
TurnWave audio branch (this model)	42.1%	17.2%	1195 ms
SmartTurn v3.2	35.2%	14.8%	1051 ms
LiveKit Turn Detector v1	9.9%	4.5%	543 ms

TurnWave beats the VAD baseline on every metric the harness reports.

Models in this repo

file	licence	training data
audio_eot_v2.onnx	apache-2.0	trained on smart-turn conversational clips
audio_eot.onnx	cc-by-4.0	Phase 4; trained on semantic-vad-eot (CC BY 4.0)
text_eot.int8.onnx	cc-by-nc-sa-4.0	trained on DailyDialog (CC BY-NC-SA 4.0) — non-commercial
fusion_eot.onnx	cc-by-nc-sa-4.0	contains the text branch, so it inherits the same terms

Each model's licence follows its training data, so they differ. audio_eot_v2 is the one the benchmark above measures and the one to use.

Usage

from huggingface_hub import hf_hub_download
from turnwave.infer import TurnDetector   # pip install git+https://github.com/Nikhils-G/turnwave

detector = TurnDetector(hf_hub_download("Nikhil-09/turnwave", "audio_eot_v2.onnx"))
if detector.predict(audio=wav_16k) > 0.5:
    respond()

16 kHz mono. The model reads the last 2 seconds ending at the decision point, which sits 0.2 s into the pause — where a live agent decides, and where eot-bench scores.

model	variant	CPU latency	size
audio_eot_v2	fp32	4.81 ms	14.0 MB
audio_eot	fp32	4.52 ms	14.0 MB
text_eot	int8	5.02 ms	7.2 MB
fusion_eot	fp32	9.35 ms	42.4 MB

INT8 is not applied blindly: dynamic quantization rewrites MatMul, so it speeds up the transformer and slows down the conv-heavy branches. Each model ships whichever variant measured faster.

What this project found

The first version of this model scored AP 0.945 on its own held-out test set and AUC 0.563 on eot-bench — barely above random. The policy sweep chose thresholds of 0.0 and 1.0, meaning ignore the model entirely.

The cause was the training corpus, not the architecture. It derived from a dataset whose own card declares task_categories: [text-to-speech] — read speech, whose pauses are reading hesitations rather than conversational turn-yields. The model had learned "has this sentence finished being read aloud."

Retraining on conversational data, changing nothing else, lifted AUC to 0.770. The in-domain score could never have revealed this; only a benchmark on data we did not build could.

Limitations

• English only. Other languages are in the training data but untested here.
• Behind the production models, and not a fair comparison: SmartTurn starts from a pretrained Whisper encoder, LiveKit's is a fine-tuned 0.5B LLM distilled from a 7B teacher. This is 3.49M parameters from random initialisation.
• The fusion model is stale. It was trained on the read-speech corpus, which the benchmark showed to be the wrong task. The conversational corpus has no transcripts, so retraining fusion needs ASR first.
• Non-commercial models included. The text and fusion models derive from DailyDialog (CC BY-NC-SA 4.0). Only the audio branches are permissively licensed.

Code, training scripts, and the full write-up: https://github.com/Nikhils-G/turnwave