Scicom-intl/semantic-vad-eot-whisper-tiny

Semantic VAD — Whisper-tiny end-of-turn detector (audio only)

An audio-native end-of-turn (EOT) detector for voice agents: given the last 8 seconds of a caller's 16 kHz audio, it returns p(end of turn) — the probability that the person has finished speaking rather than paused mid-sentence. No transcript is needed, so it can answer as soon as the VAD has seen a short silence instead of waiting for an STT result.

Same recipe and input contract as pipecat smart-turn-v3 (Whisper encoder + small classification head on a fixed 8 s left-padded window), trained on real Malaysian call-centre telephony (Malay and English, both the customer and the agent channel), where turn ends are observed — the other party took the floor — rather than inferred from alignment gaps.

8 M parameters · int8 ONNX 10 MB · ≈30 ms per prediction on one CPU thread (≈65 ms for smart-turn-v3 on the same machine). A whisper-base variant with more ranking headroom is Scicom-intl/semantic-vad-eot-whisper-base.

Results

In a real LiveKit Agents 1.8 pipeline (Silero VAD → turn detector → endpointing, no STT, 300 recorded telephony turns, LiveKit defaults: VAD silence 0.55 s, min_delay 0.5 s, max_delay 3.0 s). Latency is measured from the caller's true end of speech to the pipeline's commit; a cut-off is a commit during a mid-turn pause.

turn detector	latency p50 / p90	turns cut off	finished turns on the fast path	AUC (eot vs hold)
VAD only	0.63 / 0.71 s	14.3 %	–	–
smart-turn-v3, threshold 0.5	0.65 / 3.04 s	10.0 %	82 %	0.74
this model, threshold 0.5	0.64 / 0.74 s	10.0 %	95 %	0.84
this model, threshold 0.6	0.65 / 0.80 s	9.7 %	92 %	0.84
this model, threshold 0.6, VAD silence 0.4 s	0.49 / 0.66 s	14.0 %	92 %	0.84
whisper-base variant, threshold 0.3	0.64 / 0.74 s	9.7 %	96 %	0.88

Same interruption rate as smart-turn-v3, but 95 % of finished turns are released on the fast path, so the 3 s "not sure" tail disappears (p90 0.74 s, within 0.03 s of VAD-only). With the VAD silence lowered to 0.4 s the agent answers 0.49 s after the caller stops at the stock pipeline's interruption rate (VAD-only at 0.4 s: 21.7 % cut-offs).

Offline, at fixed cut points relative to the start of each pause (AUC, same 300 turns, every pause):

cut relative to pause start	−0.4 s	−0.2 s	0.0 s	+0.2 s	+0.6 s
smart-turn-v3	0.60	0.62	0.63	0.65	0.69
this model (int8)	0.72	0.78	0.80	0.81	0.97

The score is smooth and monotone along a pause (local std 0.04 over 200 ms, threshold flips 1.5 % per 20 ms step, vs 0.12 / 9.8 % for smart-turn-v3), which is what a pipeline that asks at VAD-tick times needs.

Under LiveKit's eot-bench harness (100 ms causal grid over every pause ≥ 0.1 s, threshold × action_delay × timeout policy sweep, operating points at fixed latency / cutoff budgets; the VAD baseline is the harness's own silence-timer policy on the same spans). All audio models scored with the same adapter, 0.2 s into each pause for the scalar metrics:

set	model	cutoff @ 300 ms budget	cutoff @ 600 ms	latency @ 5 % cutoff	latency @ 10 % cutoff
telephony test, 1 000 turns, English (510 eot / 260 hold spans)	this model	50.8 %	30.0 %	2 039 ms	1 529 ms
	smart-turn-v3	69.6 %	35.4 %	2 269 ms	1 756 ms
	VAD baseline	77.3 %	41.9 %	2 020 ms	1 610 ms
telephony test, 1 000 turns, Malay (485 / 169)	this model	55.0 %	32.5 %	2 019 ms	1 423 ms
	smart-turn-v3	78.1 %	39.1 %	2 635 ms	2 116 ms
	VAD baseline	79.3 %	46.2 %	2 540 ms	2 060 ms
telephony, the 300 benchmark turns, English (188 / 105)	this model	58.1 %	27.6 %	1 636 ms	1 198 ms
	smart-turn-v3	74.3 %	30.5 %	1 649 ms	1 164 ms
	VAD baseline	77.1 %	31.4 %	1 800 ms	1 510 ms
telephony, the 300 benchmark turns, Malay (112 / 35)	this model	57.1 %	31.4 %	1 843 ms	1 482 ms
	smart-turn-v3	68.6 %	34.3 %	2 357 ms	1 603 ms
	VAD baseline	80.0 %	34.3 %	2 410 ms	1 830 ms

On telephony it beats smart-turn-v3 on every operating point and the VAD baseline on cutoff rate at both latency budgets and on latency at a 10 % cutoff budget; at a 5 % budget it ties the VAD timer. The harness asks within the first 100–300 ms of every pause, before an audio model has silence evidence (AUC 0.80 at the pause start vs 0.97 at +0.6 s), and its 5 % budget over all pauses ≥ 0.1 s forces a threshold (≈0.86) that a calibrated, non-saturating score rarely reaches — so most turns run to the timeout there. The LiveKit pipeline asks after the VAD's 0.4–0.55 s of silence, which is the operating point production runs at and where this model does its work (95 % fast path at smart-turn's interruption rate).

Compared with other open detectors (eot-bench, 1 000 telephony turns)

Same harness, same 1 000 random Scicom-intl/semantic-vad-eot-emgs test turns (both language tags pooled), every open audio-native end-of-turn detector we could run locally; cloud services were not run because the call audio may not leave our infrastructure. Third-party models run through eot-bench's own adapters with the language gate widened to Malay; the text detector on transcripts from our Whisper STT (segment timestamps interpolated to words, ~30 % of these short turns have no transcript); ultraVAD without the assistant context it was designed for (this set has none).

model	cutoff @ 300 ms	cutoff @ 600 ms	latency @ 5 % cutoff	latency @ 10 % cutoff	AUC
Semantic-VAD whisper-small v6 (repo)	45.2 %	24.5 %	1 839 ms	1 226 ms	0.86
Semantic-VAD whisper-base v6 (repo)	47.3 %	25.4 %	1 812 ms	1 280 ms	0.85
Semantic-VAD whisper-tiny v6 (repo)	52.2 %	30.8 %	2 042 ms	1 503 ms	0.81
LiveKit turn-detector v1-mini (audio-only, livekit-local-inference)	63.6 %	35.7 %	2 156 ms	1 720 ms	0.74
ultraVAD (fixie-ai/ultraVAD, 0.7 B, no text context)	71.6 %	39.6 %	2 212 ms	1 784 ms	0.65
smart-turn v3.2 (pipecat-ai/smart-turn-v3)	73.7 %	36.6 %	2 296 ms	1 860 ms	0.65
LiveKit text turn-detector v0.4.1-intl (on STT transcripts)	–	–	2 381 ms	1 894 ms	0.45
smart-turn v2 (pipecat-ai/smart-turn-v2, 95 M wav2vec2)	74.1 %	39.6 %	2 500 ms	2 000 ms	0.62
VAD baseline (silence timer)	78.1 %	43.6 %	2 250 ms	1 770 ms	–

Files

file	what
onnx/model.int8.onnx	serve this — MatMul-only dynamic int8, 10 MB, ≈30 ms / CPU thread; AUC identical to fp32 within noise
onnx/model.fp32.onnx	fp32 export, 32 MB, ≈56 ms; max abs Δp vs PyTorch 1e-6
onnx/export_report.json	sizes, parity vs PyTorch, latency at export time
encoder/	fine-tuned Whisper-tiny encoder, HF format (config.json, model.safetensors, bf16)
eot_head.pt	{"state_dict": LayerNorm→Linear(384,256)→GELU→Linear(256,1), "pooling": "last5"}
eot_window.json / preprocessor_config.json	the input contract: 8 s window, 80 mel bins, 16 kHz, no mel normalisation, mean of the last 5 encoder frames
training_summary.json	best step, validation AUC history

Input: input_features [batch, 80, 800] float32 — Whisper log-mel of the last 8 s of audio, left-padded with zeros when shorter, do_normalize=False. Output: probability [batch, 1], already through the sigmoid.

Usage

ONNX (recommended for serving — no torch)

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor

REPO, SR, WINDOW = "Scicom-intl/semantic-vad-eot-whisper-tiny", 16000, 8 * 16000
opts = ort.SessionOptions(); opts.intra_op_num_threads = 1
sess = ort.InferenceSession(hf_hub_download(REPO, "onnx/model.int8.onnx"), opts, providers=["CPUExecutionProvider"])
fe = WhisperFeatureExtractor(feature_size=80, sampling_rate=SR, chunk_length=8)

def p_end_of_turn(pcm: np.ndarray) -> float:
    """pcm: float32 in [-1, 1] at 16 kHz, the caller's audio up to *now* (any length)."""
    pcm = np.asarray(pcm, dtype=np.float32)
    if pcm.size and np.abs(pcm).max() > 1.5:   # int16-scale samples (e.g. LiveKit buffers) -> unit float
        pcm = pcm / 32768.0
    pcm = pcm[-WINDOW:] if len(pcm) >= WINDOW else np.pad(pcm, (WINDOW - len(pcm), 0))
    feats = fe([pcm], sampling_rate=SR, return_tensors="np", padding="max_length", max_length=WINDOW,
               truncation=True, do_normalize=False)["input_features"].astype(np.float32)
    return float(sess.run(None, {"input_features": feats})[0].reshape(-1)[0])

Call it once the VAD has seen ≥ 0.2–0.3 s of silence; treat p ≥ 0.5 (or 0.6 for fewer interruptions) as "the turn is over". The model expects to be asked during a pause — scores taken mid-word are not meaningful.

LiveKit Agents (as the backend of STT-API's SemanticVAD)

STT-API ships SemanticVAD, an audio-native turn detector for LiveKit Agents (the streaming inference.eot interface — asked after ~200 ms of VAD silence, no transcript needed) that takes any backend with predict(pcm) -> p(eot); WhisperEoTOnnx in the Semantic-VAD repo wraps this model for it.

from stt_api.livekit_plugin.semantic_vad import SemanticVAD          # github.com/Scicom-AI-Enterprise-Organization/STT-API
from whisper_head.onnx_backend import WhisperEoTOnnx                 # github.com/Scicom-AI-Enterprise-Organization/Semantic-VAD

backend = WhisperEoTOnnx(hf_hub_download(REPO, "onnx/model.int8.onnx"))
session = AgentSession(
    vad=silero.VAD.load(min_silence_duration=0.4),
    turn_handling={"turn_detection": SemanticVAD(backend=backend, unlikely_threshold=0.5),
                   "endpointing": {"min_delay": 0.4, "max_delay": 1.5}},
    ...)

PyTorch (fine-tuning, research)

The stock WhisperEncoder insists on a 30 s mel; the model was trained on an 8 s window, so a 20-line subclass narrows the position table for the call. Weights are a plain WhisperEncoder state dict.

import numpy as np, torch, torch.nn as nn
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor
from transformers.models.whisper.modeling_whisper import WhisperEncoder

REPO, SR, WINDOW = "Scicom-intl/semantic-vad-eot-whisper-tiny", 16000, 8 * 16000

class VariableLengthWhisperEncoder(WhisperEncoder):
    def forward(self, input_features, attention_mask=None, **kw):
        n = input_features.shape[-1] // 2
        prev = (self.config.max_source_positions, self.embed_positions.num_embeddings)
        self.config.max_source_positions = self.embed_positions.num_embeddings = n
        try:
            return super().forward(input_features, attention_mask, **kw)
        finally:
            self.config.max_source_positions, self.embed_positions.num_embeddings = prev

class EoTHead(nn.Module):
    def __init__(self, d, hidden=256):
        super().__init__()
        self.norm, self.fc1, self.act, self.fc2 = nn.LayerNorm(d), nn.Linear(d, hidden), nn.GELU(), nn.Linear(hidden, 1)
    def forward(self, x):
        return self.fc2(self.act(self.fc1(self.norm(x)))).squeeze(-1)

encoder = VariableLengthWhisperEncoder.from_pretrained(REPO, subfolder="encoder").float().eval()
payload = torch.load(hf_hub_download(REPO, "eot_head.pt"), map_location="cpu")
head = EoTHead(encoder.config.d_model); head.load_state_dict(payload["state_dict"]); head.eval()
fe = WhisperFeatureExtractor(feature_size=80, sampling_rate=SR, chunk_length=8)

@torch.inference_mode()
def p_end_of_turn(pcm):
    pcm = np.asarray(pcm, dtype=np.float32)
    pcm = pcm[-WINDOW:] if len(pcm) >= WINDOW else np.pad(pcm, (WINDOW - len(pcm), 0))
    feats = fe([pcm], sampling_rate=SR, return_tensors="pt", padding="max_length", max_length=WINDOW,
               truncation=True, do_normalize=False)["input_features"]
    hidden = encoder(feats).last_hidden_state        # [1, 400, 384]
    pooled = hidden[:, -5:, :].mean(1)               # "last5": mean of the last 100 ms
    return torch.sigmoid(head(pooled)).item()

Training

• Data: Scicom-intl/semantic-vad-eot-emgs (private; stereo call-centre recordings split into the customer's and the agent's own channel, one row per turn with the observed end of turn and the mid-turn pauses as hold spans). All train + validation files of both speakers; a fixed random 4 000-cut sample of the test split for validation and early stopping (patience 3). The 300-turn pipeline benchmark is drawn from that same test split — model selection touched benchmark data, so read the numbers above as in-distribution.
• Examples: each silence span is cut at 6 offsets drawn uniformly in [−0.4, +1.2] s around the pause start (clipped to the span end); the final pause is labelled 1, earlier pauses 0. The audio after the cut is dropped; the model sees the last 8 s before the cut. Do not train this with a fixed grid of offsets: the amount of trailing silence is trivially readable from the mel, the grid gets memorised (p ≈ 1 exactly at the trained silence durations, 0.05–0.5 in between), and a LiveKit pipeline — which asks ~0.3 s into a pause — lands between the grid points. Five earlier versions of this model failed that way while scoring 0.98 AUC offline.
• Model: openai/whisper-tiny encoder (4 layers, d 384), fully fine-tuned, bf16, + EoTHead on the mean of the last 5 encoder frames. Batch 128, AdamW lr 5e-5, constant after warm-up. Early-stopped at step 8 000 (validation AUC 0.859 on uniformly sampled cuts).
• Export: torch.onnx.export at the fixed 800-frame input, then onnxruntime dynamic quantization of MatMul only (quantizing the convolutions as well was slower and less accurate).

Limitations

• Trained on Malaysian call-centre telephony (narrowband, Malay/English code-switching, customer and agent voices). Untested on other domains and languages; on Malay read speech (ms_dialects of Scicom-intl/semantic-vad-eot) it matches the VAD baseline, it does not beat it.
• Noise cancellation in front of it (GTCRN) lowered the fast-path share in our pipeline: the model expects the raw phone channel. Measure before adding any pre-processing.
• Feed unit-scale float audio. int16-scale samples shift the log-mel by ~90 dB and turn the output into a coin flip — the snippet above rescales, so should any other integration.
• A hold decision within the first 200 ms of a pause is genuinely hard from audio alone (AUC 0.80 at the pause start vs 0.97 at +0.6 s); the model is meant to be queried after a short VAD silence, not on every frame.
• Only a 300-turn pipeline sample and a 1 000-turn offline sample stand behind the numbers; differences of one or two cut-off turns are noise.

License

Apache-2.0 (the Whisper encoder it fine-tunes is Apache-2.0). The training data is not released.