voice-isolator-cpu

A causal, streaming primary-speaker isolator for voice agents, on the CPU: it keeps the caller and removes other voices and noise before the VAD, the turn detector and the recogniser hear the call. 16 kHz in, 16 kHz out, 10 ms frames, 10 ms of delay. onnxruntime and numpy only: no torch, no GPU.

Paper: The Voice on the Line (arXiv:2610.35272).

Speech-to-text word error rate after each system, the mean of the 9 test sets in Results, every system live except VoxLock's benchmark protocol, which takes its sample from the first 15 s of the call. Live VoxLock is ahead on two of them: Krisp VI benchmark (265 recordings); Call-centre calls, caller alone.

A test meeting streamed through a real LiveKit agent with this model on one CPU thread: first the call as LiveKit hands it to the agent, then what the agent's VAD and speech-to-text hear after the isolator. Bars: the call's level, coloured by who is talking (the corpus annotations). Words: what speech-to-text writes, the caller's in blue and other people's in orange. More below.

It is our GPU release, Scicom-intl/voice-isolator (M + causal WavLM, 555M parameters), distilled into a 22M-parameter transformer of the same design (8 layers of width 384 that see the last 10 s, a frame LSTM, band LSTMs).

Use in a LiveKit agent

How to use it in a LiveKit agent: STT-API's voice_isolator plugin (this model inside the agent as LocalIsolator, the GPU release on a server as RemoteIsolator, prewarming, and a streaming benchmark through real LiveKit agents). The same plugin is in this repository's voice_isolator/ folder.

from livekit.agents.voice import room_io
from stt_api.livekit_plugin.voice_isolator import LocalIsolator, VoiceIsolator   # or this repository's voice_isolator/

await session.start(
    agent=MyAgent(), room=ctx.room,
    room_options=room_io.RoomOptions(audio_input=room_io.AudioInputOptions(
        sample_rate=16000,
        noise_cancellation=lambda params: VoiceIsolator(backend=LocalIsolator("Scicom-intl/voice-isolator-cpu")))),
)

Without LiveKit: voice_isolator.OnnxStream(model_dir).push(chunk) takes 16 kHz float32 audio in any amounts and returns the isolated audio of every whole 10 ms hop. example.py isolates a file.

Examples

AMI test meeting ES2004c, 70 s: the caller stops for 38 s while the others talk, then comes back

Speech-to-text (parakeet-tdt-0.6b-v3) writes 120 words from other people without the isolator and 0 with it, and gets 38 of the caller's 38 right without it and 38 with it. before.wav · after.wav

AMI test meeting EN2002c, 60 s, a hard one: the others are faint here, and the isolator can cost the caller words

Speech-to-text (parakeet-tdt-0.6b-v3) writes 6 words from other people without the isolator and 1 with it, and gets 36 of the caller's 49 right without it and 36 with it. before.wav · after.wav

AMI test meeting TS3003b, 60 s: 33 s of the others talking on their own

Speech-to-text (parakeet-tdt-0.6b-v3) writes 64 words from other people without the isolator and 0 with it, and gets 46 of the caller's 48 right without it and 45 with it. before.wav · after.wav

AliMeeting R1019_M4035 (Mandarin), 13 s: another voice after the caller

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 42 of the caller's 47 right without it and 44 with it. before.wav · after.wav

Hard: AMI test meeting EN2002d, 60 s: the others talk over the caller for 24 s, almost as loud as the caller in the caller's own headset

Speech-to-text (parakeet-tdt-0.6b-v3) writes 24 words from other people without the isolator and 0 with it, and gets 35 of the caller's 67 right without it and 58 with it. before.wav · after.wav

Hard: AMI test meeting EN2002a, 14 s: in the caller's own headset the others are louder than the caller, and the isolator lets them through

Speech-to-text (parakeet-tdt-0.6b-v3) writes 0 words from other people without the isolator and 0 with it, and gets 21 of the caller's 24 right without it and 21 with it. before.wav · after.wav

Hard: AliMeeting R0008_M0058 (Mandarin), 17 s: another voice as loud as the caller talks over 6 s of the caller's speech

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 45 of the caller's 59 right without it and 46 with it. before.wav · after.wav

Hard: AliMeeting R0004_M0018 (Mandarin), 9 s: the other voice is 12 dB louder than the caller

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 22 of the caller's 25 right without it and 23 with it. before.wav · after.wav

Hard: AMI test meeting ES2004b, 12 s: a caller nearly alone, talking from the first second, before the live enrollment exists

Speech-to-text (parakeet-tdt-0.6b-v3) writes 0 words from other people without the isolator and 0 with it, and gets 34 of the caller's 35 right without it and 34 with it. before.wav · after.wav

All recorded inside a LiveKit agent: the caller streams the recording over WebRTC in real time, and this model runs as the agent's noise_cancellation (one CPU thread, 50 ms frames). Chosen by rules on the input, not the result: the longest stretches of others talking alone; the most words from others before processing; the most speech overlapped by a loud voice, and the loudest other voices (one window per meeting not already shown); a caller nearly alone from the first second. From AMI (CC BY 4.0) and AliMeeting (CC BY-SA 4.0) test meetings.

Results

Word error rate (%) after processing; Parakeet TDT 0.6B v2 (Krisp, calls) or v3 (meetings), Whisper large-v3 for AliMeeting. Live: the enrollment comes from the call itself (from 3 s in, the loudest 3 s heard so far).

test set	no processing	VoxLock live (CPU)	this model, live (CPU)	our GPU release, live
Krisp VI benchmark (265 recordings)	23.19	5.63	5.66	5.74
AMI headset, VoxLock's windows	33.00	11.71	11.23	10.19
AMI test meetings	42.59	12.07	10.29	10.18
AMI test meetings, 60 s windows	121.79	23.66	15.55	16.13
NOTSOFAR	34.70	8.84	7.58	7.67
AliMeeting (character error rate)	29.40	16.28	15.82	15.69
Call-centre calls, caller alone	0.04	2.92	5.32	5.58
Call-centre calls + another voice	69.79	56.65	49.90	47.52
ai-coustics test calls	66.36	24.55	18.03	14.26

Chosen on development sets only (AMI and NOTSOFAR development meetings, 60 s AMI windows, clean development calls): their mean word error rate is 10.66% for this model, 13.51% for VoxLock live and 10.49% for the GPU release it was distilled from.

The agent's VAD

A noise filter leaves other voices in, and LiveKit's VAD takes them for the caller: it starts turns for them, holds the caller's turn open while they talk, and the agent answers late or not at all. Behind the isolator it hears far less of them. LiveKit's Silero VAD with LiveKit's own rule on 120 one-minute test calls with another voice and 120 with the caller alone, against the same VAD on the caller's own channel:

	speech heard in the caller's pauses	false starts / min	turn ends held	turn end p90 (s)	caller's speech missed	caller's speech missed, caller alone
no processing	47.9%	3.33	28%	2.43	0.1%	0.0%
GTCRN (noise only)	48.3%	3.58	26%	2.40	0.3%	0.6%
VoxLock live	27.5%	2.28	24%	1.51	11.0%	2.6%
this model, live	26.2%	2.27	20%	1.34	6.7%	4.9%
our GPU release, live	22.8%	1.94	18%	1.00	8.4%	5.2%

The same in a running LiveKit agent: 32 of the test calls with another voice streamed at once into rooms of a LiveKit server, each answered by an AgentSession with LiveKit's Silero VAD, the filter on its audio input in LiveKit's 50 ms frames, against the same rooms streaming each caller's own channel with no filter:

filter	speech heard in the caller's pauses	false starts / min	turn ends held	turn end p90 (s)	caller's speech missed
no filter	47.5%	2.84	23%	2.04	0.3%
GTCRN (noise only)	46.8%	2.88	22%	2.03	0.7%
this model	27.7%	2.12	18%	1.03	3.0%
this model, 8-bit	28.2%	2.25	16%	1.04	2.9%
our GPU release, through its server	24.9%	1.75	18%	1.06	6.9%

CPU

Share of one CPU core per live call, one thread, every system through its own streaming entry point on the same calls, timed in one run (lower is better):

frames	this model	this model, int8 (quantized=True)	GTCRN (noise only)	VoxLock, its default int8	VoxLock, float32
10 ms	54%	34%	5%	227%	264%
20 ms	33%	23%	5%	132%	158%
50 ms (LiveKit's default)	17%	12%	5%	65%	88%

GTCRN, the noise suppressor STT-API ships, is the cheapest, and removes only noise: another voice is speech, and it keeps it (see the VAD tables above). Each model runs once per frame, so larger frames cost less.

The 8-bit model sets its activation scales at every step, so its output depends on how the call is cut into frames (the same call in 20 ms and in 50 ms frames agrees to 40 dB; the float model to rounding); on the development sets it makes 10.52% word errors against 10.66% in float.

In a LiveKit agent carrying 32 calls at once on one machine (STT-API's streaming benchmark: LiveKit server, Agents worker, Silero VAD, callers in real time, 50 ms frames), the isolator took 15 ms of a frame at the median and 25 ms at the 99th percentile in float, 9 and 17 ms in int8: wall clock on the agent's audio loop, with every call's agent and caller sharing the machine, so 32% of a core per call rather than the figure above. Its output stood 10 ms behind the call.

Citation

@misc{zolkepli2026voice,
  title         = {The Voice on the Line},
  author        = {Zolkepli, Husein},
  year          = {2026},
  eprint        = {2610.35272},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2610.35272}
}