voice-isolator

A causal, streaming primary-speaker isolator for voice agents, served on a GPU: it keeps the caller and removes other voices and noise before the VAD, the turn detector and the recogniser hear the call. 16 kHz in, 16 kHz out, 10 ms frames, chunks of 20 ms or more, every call on a GPU in one batched step. It enrolls the caller from the call itself (live: from 3 s in, the loudest 3 s heard so far, fixed once the call is 15 s old), or from a clip of the caller.

Paper: The Voice on the Line (arXiv:2610.35272).

Speech-to-text word error rate after each system, the mean of the 9 test sets in Results, every system live except VoxLock's benchmark protocol, which takes its sample from the first 15 s of the call. Live VoxLock is ahead on two of them: Krisp VI benchmark (265 recordings); Call-centre calls, caller alone.

A test meeting streamed through a real LiveKit agent whose audio input sends the call to this model on a GPU server: first the call as LiveKit hands it to the agent, then what the agent's VAD and speech-to-text hear. Bars: the call's level, coloured by who is talking (the corpus annotations). Words: what speech-to-text writes, the caller's in blue and other people's in orange. More below.

A causal transformer that sees the last 10 s, with WavLM-Large made causal beside it: 555M parameters, 47.7 GMAC per second of audio, 115 MB of state per live call, no look-ahead beyond the 20 ms STFT window. Distilled to 22M parameters, the same design runs inside the agent on one CPU thread: Scicom-intl/voice-isolator-cpu.

Use in a LiveKit agent

How to use it in a LiveKit agent: STT-API's voice_isolator plugin (this model on a GPU server as RemoteIsolator, the CPU model inside the agent as LocalIsolator, prewarming, and a streaming benchmark through real LiveKit agents). The same plugin is in this repository's voice_isolator/ folder.

One GPU server carries every call (voice_isolator/Dockerfile builds it):

python -m vi.server_mp --ckpt Scicom-intl/voice-isolator --workers 8 --enroll-loud --chunk-hops 4 --max-streams 128 --port 8765

and each agent streams its calls to it through the plugin, as the audio input's noise cancellation:

from livekit.agents.voice import room_io
from stt_api.livekit_plugin.voice_isolator import RemoteIsolator, VoiceIsolator   # or this repository's voice_isolator/

backend = RemoteIsolator("ws://gpu-host:8765")           # one per process, shared by its calls
await session.start(
    agent=MyAgent(), room=ctx.room,
    room_options=room_io.RoomOptions(audio_input=room_io.AudioInputOptions(
        sample_rate=16000,
        noise_cancellation=lambda params: VoiceIsolator(backend=backend))),
)

Through real LiveKit agents (STT-API's streaming benchmark), with LiveKit's 50 ms frames and the server on --chunk-hops 4 (40 ms steps), 32 calls at once came back 40 ms behind the call at the median (50 ms at the 99th percentile) and 64 calls 50 ms (50 ms): each frame waits on the agent's audio loop for its own reply (14 ms at the median, 30 ms at most, the plugin's default), so the reply goes out with its frame instead of the next; 0.13% of the audio of the 32-call run and 1.00% of the audio of the 64-call run came back past the plugin's 40 ms cap on its delay and went through unfiltered. With AudioInputOptions(frame_size_ms=20) and --chunk-hops 2 the delay is 30 ms (50 ms), for fewer calls per GPU.

Streaming without LiveKit

One call on a GPU, in Python, in this process:

from vi.stream import Stream

# Stream(..., enrollment=clip) gives it the caller's voice; without one the call enrolls itself
s = Stream("Scicom-intl/voice-isolator")
for chunk in call:               # 16 kHz float32, any length
    caller = s.push(chunk)       # the caller of every whole 20 ms so far, s.delay samples behind

Any language: one WebSocket per call to vi.server or vi.server_mp. Send binary 16 kHz mono int16 PCM in any amounts; the isolated caller comes back as binary int16 PCM, in order, 10 ms behind plus the server's chunk. A text message {"enroll": "<base64 int16 PCM>"} sets the caller's voice; without one the call enrolls itself.

Examples

AMI test meeting ES2004c, 70 s: the caller stops for 38 s while the others talk, then comes back

Speech-to-text (parakeet-tdt-0.6b-v3) writes 111 words from other people without the isolator and 0 with it, and gets 38 of the caller's 38 right without it and 38 with it. before.wav · after.wav

AMI test meeting EN2002c, 60 s, a hard one: the others are faint here, and the isolator can cost the caller words

Speech-to-text (parakeet-tdt-0.6b-v3) writes 20 words from other people without the isolator and 1 with it, and gets 25 of the caller's 49 right without it and 36 with it. before.wav · after.wav

AMI test meeting TS3003b, 60 s: 33 s of the others talking on their own

Speech-to-text (parakeet-tdt-0.6b-v3) writes 64 words from other people without the isolator and 0 with it, and gets 46 of the caller's 48 right without it and 45 with it. before.wav · after.wav

AliMeeting R1019_M4035 (Mandarin), 13 s: another voice after the caller

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 40 of the caller's 47 right without it and 44 with it. before.wav · after.wav

Hard: AMI test meeting EN2002d, 60 s: the others talk over the caller for 24 s, almost as loud as the caller in the caller's own headset

Speech-to-text (parakeet-tdt-0.6b-v3) writes 38 words from other people without the isolator and 0 with it, and gets 38 of the caller's 67 right without it and 58 with it. before.wav · after.wav

Hard: AMI test meeting EN2002a, 14 s: in the caller's own headset the others are louder than the caller, and the isolator lets them through

Speech-to-text (parakeet-tdt-0.6b-v3) writes 0 words from other people without the isolator and 0 with it, and gets 21 of the caller's 24 right without it and 21 with it. before.wav · after.wav

Hard: AliMeeting R0008_M0058 (Mandarin), 17 s: another voice as loud as the caller talks over 6 s of the caller's speech

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 45 of the caller's 59 right without it and 47 with it. before.wav · after.wav

Hard: AliMeeting R0004_M0018 (Mandarin), 9 s: the other voice is 12 dB louder than the caller

Speech-to-text (whisper-large-v3) writes 0 characters from other people without the isolator and 0 with it, and gets 22 of the caller's 25 right without it and 23 with it. before.wav · after.wav

Hard: AMI test meeting ES2004b, 12 s: a caller nearly alone, talking from the first second, before the live enrollment exists

Speech-to-text (parakeet-tdt-0.6b-v3) writes 0 words from other people without the isolator and 0 with it, and gets 34 of the caller's 35 right without it and 34 with it. before.wav · after.wav

All recorded inside a LiveKit agent: the caller streams the recording over WebRTC in real time, and the agent's noise_cancellation sends the call to this model's server (RemoteIsolator, LiveKit's 50 ms frames, the server stepping 40 ms at a time). Chosen by rules on the input, not the result: the longest stretches of others talking alone; the most words from others before processing; the most speech overlapped by a loud voice, and the loudest other voices (one window per meeting not already shown); a caller nearly alone from the first second. From AMI (CC BY 4.0) and AliMeeting (CC BY-SA 4.0) test meetings.

Results

Word error rate (%) after processing, Parakeet TDT 0.6B v2 (Krisp, calls) or v3 (meetings), Whisper large-v3 for AliMeeting; for the call-centre calls, the change against recognising the clean caller. Live = causal enrollment from the call; * = looks up to 15 s ahead for its enrollment sample (VoxLock's benchmark protocol), so not live.

test set	no processing	VoxLock live	VoxLock, loudest 3 s *	this model, live	this model, loudest 3 s *
Krisp VI benchmark (265 recordings)	23.19	5.63	5.19	5.74	5.93
AMI headset, VoxLock's windows	33.00	11.71	11.26	10.19	11.37
AMI test meetings	42.59	12.07	9.70	10.18	10.35
AMI test meetings, 60 s windows	121.79	23.66	19.52	16.13	16.22
NOTSOFAR	34.70	8.84	7.61	7.67	7.67
AliMeeting (character error rate)	29.40	16.28	17.11	15.69	15.40
Call-centre calls, caller alone	0.04	2.92	3.37	5.58	5.66
Call-centre calls + another voice	69.79	56.65	47.05	47.52	47.87
ai-coustics test calls	66.36	24.55	20.31	14.26	12.54

On Krisp's benchmark, word errors against the words that leak through per minute where only the background talks:

The model was chosen on development sets only (AMI and NOTSOFAR development meetings, 60 s AMI development windows and clean development calls; their mean word error rate: 10.49% for this model, 11.44% for the same transformer without WavLM, 11.91% for VoxLock). The test sets above were never used to choose, and the evaluation-only data (Krisp's benchmark, AppTek's call-centre recordings, ai-coustics' test calls) was never trained on.

The agent's VAD

A noise filter leaves other voices in, and LiveKit's VAD takes them for the caller: it starts turns for them, holds the caller's turn open while they talk, and the agent answers late or not at all. Behind the isolator it hears far less of them. LiveKit's Silero VAD with LiveKit's own rule on 120 one-minute test calls with another voice and 120 with the caller alone, against the same VAD on the caller's own channel:

	speech heard in the caller's pauses	false starts / min	turn ends held	turn end p90 (s)	caller's speech missed	caller's speech missed, caller alone
no processing	47.9%	3.33	28%	2.43	0.1%	0.0%
GTCRN (noise only)	48.3%	3.58	26%	2.40	0.3%	0.6%
VoxLock live	27.5%	2.28	24%	1.51	11.0%	2.6%
our CPU model, live	26.2%	2.27	20%	1.34	6.7%	4.9%
this model, live	22.8%	1.94	18%	1.00	8.4%	5.2%

The same in a running LiveKit agent: 32 of the test calls with another voice streamed at once into rooms of a LiveKit server, each answered by an AgentSession with LiveKit's Silero VAD, the filter on its audio input in LiveKit's 50 ms frames, against the same rooms streaming each caller's own channel with no filter:

filter	speech heard in the caller's pauses	false starts / min	turn ends held	turn end p90 (s)	caller's speech missed
no filter	47.5%	2.84	23%	2.04	0.3%
GTCRN (noise only)	46.8%	2.88	22%	2.03	0.7%
our CPU model, in the agent	27.7%	2.12	18%	1.03	3.0%
this model, through its server	24.9%	1.75	18%	1.06	6.9%
this model, 20 ms frames	24.1%	1.59	18%	1.03	6.8%
this model, 64 calls on one GPU	22.8%	1.61	21%	1.02	8.5%

Serving

One GPU runs every open call in one batched step per chunk (fp16, CUDA graphs, the LSTMs and their states in fp32): 160 concurrent calls in real time with 20 ms chunks, 256 with 40 ms chunks. vi/server.py is a WebSocket server around that engine; vi/server_mp.py spreads the WebSocket side over processes: on one GPU, 128 live calls with 20 ms chunks got each chunk back in 27 ms (median) and 38 ms (99th percentile) over localhost, once every call had enrolled. Over 30 minutes at 64 calls (20 ms chunks) the server lost no call (0 errors); the 99th percentile of the round trip was 33 ms over the first minute, while every call enrolled, and between 32 and 33 ms over every later one.

Citation

@misc{zolkepli2026voice,
  title         = {The Voice on the Line},
  author        = {Zolkepli, Husein},
  year          = {2026},
  eprint        = {2610.35272},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SD},
  url           = {https://arxiv.org/abs/2610.35272}
}