Scicom-intl/semantic-vad-eot-whisper-base

Semantic VAD — Whisper-base end-of-turn detector (audio only)

The whisper-base sibling of Scicom-intl/semantic-vad-eot-whisper-tiny: same recipe, same input contract, same data, a 2.5× larger encoder. Given the last 8 seconds of a caller's 16 kHz audio it returns p(end of turn) — finished speaking vs paused mid-sentence — with no transcript.

20 M parameters · int8 ONNX 24 MB · roughly twice the compute of the tiny model (int8 69 ms vs 34 ms per prediction measured back to back on the same busy CPU; the tiny model runs ≈30 ms idle on one thread). It ranks turns better than the tiny model (AUC 0.88 vs 0.84 in the pipeline, 0.85 / 0.87 / 0.98 vs 0.80 / 0.81 / 0.97 offline at 0 / 0.2 / 0.6 s into a pause) — worth it when CPU is not the constraint or when you want a stricter threshold; otherwise use the tiny model, which we recommend for production.

Results

Compared with other open detectors (eot-bench, private telephony)

Same eot-bench harness for every model (100 ms causal grid over every pause ≥ 0.1 s, threshold × action_delay × timeout policy sweep, scalar metrics scored 0.2 s into each pause), random private telephony test turns, both language tags pooled. Third-party models run through eot-bench's own adapters with the language gate widened to Malay; the text detector on transcripts from our Whisper STT (segment timestamps interpolated to words, ~30 % of these short turns have no transcript); ultraVAD without the assistant context it was designed for (this set has none); LiveKit's cloud Turn Detector v1 streamed the 300-turn set once with the data owner's approval (292 of 300 turns scored, 8 failed the gateway handshake; LiveKit Cloud caps this project at ~5 streaming turns per minute, so the run was paced).

The 300 benchmark turns (300 eot / 191 hold spans) — every detector, LiveKit cloud v1 included:

model	cutoff @ 300 ms	cutoff @ 600 ms	latency @ 5 % cutoff	latency @ 10 % cutoff	AUC
Scicom Semantic VAD (enterprise model, GPU-served, private)	38.6 %	20.0 %	1 444 ms	1 034 ms	0.87
Semantic-VAD whisper-small v6	48.6 %	22.1 %	1 381 ms	1 099 ms	0.86
Semantic-VAD whisper-base v6	45.0 %	23.6 %	1 592 ms	1 260 ms	0.84
Semantic-VAD whisper-tiny v6	57.9 %	28.6 %	1 685 ms	1 332 ms	0.78
LiveKit turn-detector v1-mini (audio-only, local)	55.7 %	28.6 %	1 633 ms	1 292 ms	0.76
LiveKit Turn Detector v1 (cloud, audio)	66.7 %	27.8 %	1 568 ms	1 348 ms	0.70
ultraVAD (no text context)	65.7 %	32.1 %	1 880 ms	1 384 ms	0.65
smart-turn v3.2	73.6 %	31.4 %	1 863 ms	1 367 ms	0.64
smart-turn v2	77.1 %	32.1 %	2 000 ms	1 420 ms	0.64
LiveKit text turn-detector v0.4.1-intl (on STT transcripts)	–	–	1 952 ms	1 769 ms	0.45
VAD baseline (silence timer)	77.9 %	32.1 %	1 900 ms	1 510 ms	–

LiveKit's cloud v1 lands between ultraVAD and the silence timer on this Malay-heavy telephony audio (no Malay, no telephony in its training); its local v1-mini does better. Scicom Semantic VAD is the enterprise member of this family, served from a GPU with dynamic batching, and is not public.

1 000 random test turns (1 010 eot / 569 hold spans) — the larger sample; the cloud detector was not run here:

model	cutoff @ 300 ms	cutoff @ 600 ms	latency @ 5 % cutoff	latency @ 10 % cutoff	AUC
Scicom Semantic VAD (enterprise model, GPU-served, private)	43.8 %	24.5 %	1 663 ms	1 215 ms	0.87
Semantic-VAD whisper-small v6 (repo)	45.2 %	24.5 %	1 839 ms	1 226 ms	0.86
Semantic-VAD whisper-base v6 (repo)	47.3 %	25.4 %	1 812 ms	1 280 ms	0.85
Semantic-VAD whisper-tiny v6 (repo)	52.2 %	30.8 %	2 042 ms	1 503 ms	0.81
LiveKit turn-detector v1-mini (audio-only, livekit-local-inference)	63.6 %	35.7 %	2 156 ms	1 720 ms	0.74
ultraVAD (fixie-ai/ultraVAD, 0.7 B, no text context)	71.6 %	39.6 %	2 212 ms	1 784 ms	0.65
smart-turn v3.2 (pipecat-ai/smart-turn-v3)	73.7 %	36.6 %	2 296 ms	1 860 ms	0.65
LiveKit text turn-detector v0.4.1-intl (on STT transcripts)	–	–	2 381 ms	1 894 ms	0.45
smart-turn v2 (pipecat-ai/smart-turn-v2, 95 M wav2vec2)	74.1 %	39.6 %	2 500 ms	2 000 ms	0.62
VAD baseline (silence timer)	78.1 %	43.6 %	2 250 ms	1 770 ms	–

In the pipeline and at fixed cut points

In a real LiveKit Agents 1.8 pipeline (Silero VAD → turn detector → endpointing, no STT, 300 recorded telephony turns, LiveKit defaults: VAD silence 0.55 s, min_delay 0.5 s, max_delay 3.0 s):

turn detector	latency p50 / p90	turns cut off	finished turns on the fast path	AUC (eot vs hold)
VAD only	0.63 / 0.71 s	14.3 %	–	–
smart-turn-v3, threshold 0.5	0.65 / 3.04 s	10.0 %	82 %	0.74
tiny variant, threshold 0.5	0.64 / 0.74 s	10.0 %	95 %	0.84
this model, threshold 0.3	0.64 / 0.74 s	9.7 %	96 %	0.88
this model, threshold 0.5	0.65 / 2.93 s	9.3 %	90 %	0.88

The int8 export's scores sit a little lower than the tiny model's (recall at 0.5 is 0.87 vs 0.92 at the pause start), so 0.3 is this model's equivalent of the tiny model's 0.5; at 0.5 it is stricter — one more cut-off avoided, but 10 % of finished turns wait for max_delay. One cut-off turn in 300 separates it from the tiny model at matched fast-path share, which is within noise.

Offline, at fixed cut points relative to the start of each pause (AUC, same 300 turns, every pause):

cut relative to pause start	−0.4 s	−0.2 s	0.0 s	+0.2 s	+0.6 s
smart-turn-v3	0.60	0.62	0.63	0.65	0.69
tiny variant (int8)	0.72	0.78	0.80	0.81	0.97
this model (int8)	0.77	0.82	0.85	0.87	0.98

Score smoothness along a pause matches the tiny model (local std 0.044 over 200 ms, threshold flips 1.7 % per 20 ms step; smart-turn-v3 0.124 / 9.8 %).

Under LiveKit's eot-bench harness (100 ms causal grid over every pause ≥ 0.1 s, threshold × action_delay × timeout policy sweep; same adapter for all audio models, scored 0.2 s into each pause for the scalar metrics):

set	model	cutoff @ 300 ms budget	cutoff @ 600 ms	latency @ 5 % cutoff	latency @ 10 % cutoff	AUC
telephony test, 1 000 turns, English (510 eot / 260 hold spans)	this model	47.3 %	25.0 %	1 722 ms	1 261 ms	0.84
	tiny variant	50.8 %	30.0 %	2 039 ms	1 529 ms	0.80
	smart-turn-v3	69.6 %	35.4 %	2 269 ms	1 756 ms	0.66
	VAD baseline	77.3 %	41.9 %	2 020 ms	1 610 ms	–
telephony test, 1 000 turns, Malay (485 / 169)	this model	49.1 %	25.4 %	1 903 ms	1 494 ms	0.86
	tiny variant	55.0 %	32.5 %	2 019 ms	1 423 ms	0.81
	smart-turn-v3	78.1 %	39.1 %	2 635 ms	2 116 ms	0.63
	VAD baseline	79.3 %	46.2 %	2 540 ms	2 060 ms	–
telephony, the 300 benchmark turns, English (188 / 105)	this model	45.7 %	21.9 %	1 530 ms	1 151 ms	0.83
	tiny variant	58.1 %	27.6 %	1 636 ms	1 198 ms	0.77
	smart-turn-v3	74.3 %	30.5 %	1 649 ms	1 164 ms	0.64
	VAD baseline	77.1 %	31.4 %	1 800 ms	1 510 ms	–
telephony, the 300 benchmark turns, Malay (112 / 35)	this model	48.6 %	25.7 %	1 761 ms	1 238 ms	0.85
	tiny variant	57.1 %	31.4 %	1 843 ms	1 482 ms	0.78
	smart-turn-v3	68.6 %	34.3 %	2 357 ms	1 603 ms	0.63
	VAD baseline	80.0 %	34.3 %	2 410 ms	1 830 ms	–

Where the tiny model only ties the VAD timer on latency at a 5 % cutoff budget, this one is ahead of it on every operating point of every subset, and ahead of the tiny model everywhere except latency at 10 % on the Malay 1 000-turn set. The harness asks within the first 100–300 ms of every pause, before an audio model has silence evidence; the extra encoder capacity buys the most exactly there (AUC 0.85 vs 0.80 at the pause start). In the LiveKit pipeline, which asks after the VAD's 0.4–0.55 s of silence, the two are one cut-off turn apart.

Files

file	what
onnx/model.int8.onnx	MatMul-only dynamic int8, 24 MB
onnx/model.fp32.onnx	fp32 export, 81 MB; max abs Δp vs PyTorch 1e-6
onnx/export_report.json	sizes, parity vs PyTorch, latency at export time
encoder/	fine-tuned Whisper-base encoder, HF format (config.json, model.safetensors, bf16)
eot_head.pt	{"state_dict": LayerNorm→Linear(512,256)→GELU→Linear(256,1), "pooling": "last5"}
eot_window.json / preprocessor_config.json	the input contract: 8 s window, 80 mel bins, 16 kHz, no mel normalisation, mean of the last 5 encoder frames
training_summary.json	best step, validation AUC history

Input: input_features [batch, 80, 800] float32 — Whisper log-mel of the last 8 s of audio, left-padded with zeros when shorter, do_normalize=False. Output: probability [batch, 1], already through the sigmoid.

Usage

Identical to the tiny model — substitute the repo id. In short (ONNX, no torch):

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor

REPO, SR, WINDOW = "Scicom-intl/semantic-vad-eot-whisper-base", 16000, 8 * 16000
opts = ort.SessionOptions(); opts.intra_op_num_threads = 1
sess = ort.InferenceSession(hf_hub_download(REPO, "onnx/model.int8.onnx"), opts, providers=["CPUExecutionProvider"])
fe = WhisperFeatureExtractor(feature_size=80, sampling_rate=SR, chunk_length=8)

def p_end_of_turn(pcm: np.ndarray) -> float:
    """pcm: float32 in [-1, 1] at 16 kHz, the caller's audio up to *now* (any length)."""
    pcm = np.asarray(pcm, dtype=np.float32)
    if pcm.size and np.abs(pcm).max() > 1.5:   # int16-scale samples -> unit float
        pcm = pcm / 32768.0
    pcm = pcm[-WINDOW:] if len(pcm) >= WINDOW else np.pad(pcm, (WINDOW - len(pcm), 0))
    feats = fe([pcm], sampling_rate=SR, return_tensors="np", padding="max_length", max_length=WINDOW,
               truncation=True, do_normalize=False)["input_features"].astype(np.float32)
    return float(sess.run(None, {"input_features": feats})[0].reshape(-1)[0])

For LiveKit Agents use it as the backend of STT-API's SemanticVAD through a three-line predict(pcm) -> p(eot) backend around the ONNX snippet, exactly as on the tiny model's card.

Use p ≥ 0.3 as "the turn is over" for the operating point in the table above. The PyTorch loading snippet (a WhisperEncoder subclass that accepts the 8 s window + the 3-layer head) is on the tiny model's card and works unchanged with this repo id (d_model 512).

License

Apache-2.0 (the Whisper encoder it fine-tunes is Apache-2.0).