bsye/byosm-turn-completion

Turn completion

Turn completion is a small English text classifier that reads the partial ASR transcript of a user's current turn and decides whether the speaker has finished their turn or is still speaking. It runs on CPU and is meant to be called again each time new tokens arrive in a voice agent pipeline.

On the turn-completion-builtin test set (734 rows), it reaches 0.895 recall at 98% precision (95% confidence interval 0.861 to 0.924). None of the rivals reaches the 98% precision bar.

Table of Contents

• Overview
• How It Works
• Architecture and Training
• Benchmarks
• Usage
• Deployment Requirements
• Limitations
• License
• Resources
• Reproduce

Overview

A voice agent has to decide when the user has stopped talking. If it replies too early, it interrupts the user. If it waits too long, the conversation feels slow. Silence alone is a weak signal, because people pause in the middle of a sentence, use fillers, and trail off on words like "and" or "so".

This model handles one narrow decision: given the text of the user's turn so far, is that turn finished or unfinished? A small specialist model fits this job because the check runs many times per turn and has to finish in a few milliseconds on CPU. A general-purpose LLM is too slow and too costly to call on every token.

Use it alongside a voice activity detector. Once speech pauses, call the model on the current transcript, and only let the agent reply when the model returns finished.

Labels

• finished: the user has completed a self-contained turn that the other party can act on or respond to. Examples are an imperative, a declaration, a complete question, a thanks or refusal, or a complete answer. Turns that end on a filler or a trailing comma also count when the intent is clearly complete ("Okay." / "Sure, fine").
• unfinished: the utterance is syntactically or pragmatically incomplete. Examples are a missing object or verb, a trailing "and", "but" or "so", a question word without its clause, a pause or filler with more clearly coming, or a mid-word ASR cut. Mid-question ("what does the") and mid-list ("red and blue") fragments are unfinished.

How It Works

1. Input. A raw ASR fragment of the user's current turn, beginning at the moment the user started speaking after the assistant's previous turn ended. It is case-preserved and lightly punctuated, with no speaker IDs and no timestamps. It may end mid-word, contain fillers ("um", "uh", "you know"), or lack punctuation the ASR has not yet emitted. It never includes the assistant's speech. Examples: "I think we should", "I need to reschedule my flight to San Francisco on the 14th".
2. Prediction. The text is tokenized (up to 128 tokens) and passed through the encoder and a classification head, which outputs a probability that the turn is finished.
3. Decision threshold. The probability is compared with a fixed threshold of 0.966, chosen to hold precision on the finished label high, so the agent rarely cuts the user off.
4. Output. A single binary decision: finished = true or finished = false.

Architecture and Training

Base Model

The model is fine-tuned from sentence-transformers/all-MiniLM-L6-v2 and has 22.7M parameters. The released checkpoint is experiment v15.

Training Data

The labeled training set has 6086 rows from two sources, plus a separate dev set of 391 rows.

• anyreach-ai/semantic-turn-taking-benchmark: 318 rows.
• LLM-labeled rows produced with claude-opus-5-5 via claude-code: 5768 rows.
• Unlabeled train inputs: 5547 rows.

Pretraining and Refinement

The model was pretrained for 200 steps and then fine-tuned for 8 epochs. Training went through refinement rounds. Each round inspected errors on the dev set and added targeted rows:

Rows before round	Dev score
5547	0.672
5817	0.859
6086	0.808

The error patterns targeted during refinement were:

• Unpunctuated complete wh- and yes/no questions predicted unfinished.
• Disfluent long questions with fillers and comma chains predicted unfinished.
• Unpunctuated short answers and confirmations predicted unfinished.
• Long technical or narrative declaratives, with or without a period and with fillers, predicted unfinished.
• Bare short imperatives predicted unfinished.
• False "finished" on dangling transitive verbs, mid-word truncation, and self-repair.

Quantization and Export

The model is exported to ONNX at fp32 precision as a single folder with a one-file Python runtime. The export is 90.4 MB.

Benchmarks

The headline metric is recall at 98% precision on the finished label, measured on the turn-completion-builtin test set (734 rows). Rivals are LiveKit's English and multilingual end-of-turn models, videosdk-live/Namo-Turn-Detector-v1-English, and claude-haiku-5-5 run through claude-code. Each threshold-based rival was evaluated with its threshold calibrated on the same calib split. LiveKit's models were also evaluated at LiveKit's own tuned thresholds. The table also reports the share of each label that every system classifies correctly.

Results, measured by code

Every number in this section was computed by BYOSM's scorer on frozen splits; none was written by an LLM. This model and its rivals were scored by the same code, each with its decision threshold calibrated on the same calib split. Bold = this model.

model	params	builtin recall at 98% precision (primary)	llm-written recall at 98% precision	builtin ROC-AUC	llm-written ROC-AUC
v15 (this model)	22.71M	0.8946	0 (reaches 76.2% precision)	0.989	0.988
livekit-turn (baseline)	–	0 (reaches 97.4% precision)	0.6250	0.964	0.953
llm (baseline)	–	0 (reaches 94.7% precision)	1.0000	0.952	1.000
hf:videosdk-live/Namo-Turn-Detector-v1-English#LABEL_1=finished,LABEL_0=unfinished (baseline)	135.33M	0 (reaches 91.8% precision)	0 (reaches 84.6% precision)	0.877	0.926
llm:rival-claude-haiku-5-5 (baseline)	–	0 (reaches 95.9% precision)	1.0000	0.955	1.000
livekit-turn-default (baseline)	–	0 (reaches 91.3% precision)	0 (reaches 92.3% precision)	0.886	0.844
livekit-turn-multilingual (baseline)	–	0 (reaches 94.4% precision)	0 (reaches 87.5% precision)	0.956	0.922
livekit-turn-multilingual-default (baseline)	–	0 (reaches 79.0% precision)	0 (reaches 75.0% precision)	0.857	0.812

This model, per suite

suite	dataset	test items	recall at 98% precision	95% CI	precision	recall	ROC-AUC	avg precision	decision threshold
builtin	turn-completion-builtin	734	0.8946	0.861-0.924	0.9940	0.8946	0.9891	0.9915	0.966
llm-written	turn-completion-llm-written	32	0.0000	0.000-0.000	0.7619	1.0000	0.9883	0.9883	0.093

Model details

task	text / classify
labels	finished, unfinished
parameters	22,714,754
backbone	hf:sentence-transformers/all-MiniLM-L6-v2 · embedding dim 384
objective	recall at 98% precision, primary suite builtin
decision policy	threshold, threshold 0.966 (calibrated on builtin's calib split)

This model is the only system that reaches 98% precision, at 0.895 recall with 0.994 precision, so every rival scores 0.0 on the headline metric. Some rivals catch more finished turns. livekit-turn-multilingual-default catches 0.978, the LLM 0.959 and claude-haiku-5-5 0.951, compared with 0.895 here. They do so by letting more unfinished turns through as finished: those three correctly label 0.736, 0.945 and 0.959 of unfinished turns, compared with 0.995 for this model.

Results on test sets written by an LLM

Inputs an LLM wrote and labelled for this task, scored by the same code. They are not a public benchmark, and an LLM rival from the same family as the writer may have an edge. Every input, label and answer is in bsye/byosm-turn-completion-data.

llm-written: 16,060 inputs written by claude-code:claude-sonnet-5-5, claude-code:claude-haiku-5-5

model	right	accuracy
claude-haiku-5-5	15,562 / 16,060	0.969
v15 (this model)	15,202 / 16,060	0.947
videosdk-live/Namo-Turn-Detector-v1-English	12,848 / 16,060	0.800
livekit-turn	11,934 / 16,060	0.743

workbench: 44 inputs written by claude-code:claude-opus-5-5, hf:anyreach-ai/semantic-turn-taking-benchmark

model	right	accuracy
v15 (this model)	41 / 43	0.953
the LLM	40 / 43	0.930
livekit-turn	35 / 43	0.814

Usage

Load the exported folder with its bundled Python runtime and pass the current transcript to get a finished or unfinished decision.

pip install huggingface_hub numpy onnxruntime tokenizers

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("bsye/byosm-turn-completion")
sys.path.insert(0, path)
from byosm_rt import load

m = load(path)
print(m.predict('User: Archaeological evidence increasingly suggests that human cognitive modernity preceded anatomical modernity challen'))
emb = m.encode(...)                # the learned representation (JEPA-style), for probes / retrieval / other heads

Local HTTP server

python byosm_rt.py serve . --port 8800
# GET /v1/models · POST /v1/turn-completion/predict · POST /v1/turn-completion/encode

Deployment Requirements

• CPU only: no GPU required.
• Size: 90.4 MB on disk (ONNX, fp32), 22.7M parameters.
• Latency: about 6.174 ms per call on CPU.
• Input length: up to 128 tokens.
• Dependencies: numpy, onnxruntime, tokenizers.

Limitations

• Evaluated on builtin (734 test items) and llm-written (32 test items). Small suites have wide confidence intervals.
• English only.
• Text only. The model does not use audio, prosody or pause length, so it should be combined with a voice activity detector.
• The input never includes the assistant's previous turn. Whether a short reply is a complete answer to the last question has to be inferred from the user's text alone.
• Most labeled training rows (5768 of 6086) were labeled by an LLM, so label noise and LLM-specific phrasing may carry over.
• The dev score fell from 0.859 to 0.808 in the last refinement round, measured on a 391-row dev set.
• At the 0.966 threshold, the model favors precision over recall: it misses some finished turns (recall 0.895), which shows up as added delay rather than interruptions.

License

This model is released under CC BY-NC 4.0 (cc-by-nc-4.0).

Resources

• BYOSM on GitHub
• anyreach-ai/semantic-turn-taking-benchmark
• sentence-transformers/all-MiniLM-L6-v2
• livekit/turn-detector (English and multilingual end-of-turn models)
• videosdk-live/Namo-Turn-Detector-v1-English
• Claude models (claude-haiku-5-5)

Reproduce

git clone https://github.com/bsye/BYOSM && cd BYOSM && pip install -e .
# put spec.yaml from this repo at specialists/turn-completion/spec.yaml, then:
byosm build turn-completion && byosm board turn-completion && byosm export turn-completion

Written by an LLM

Everything in this section was written by qwen3.8:27b from the one-sentence task above, when the spec was compiled. It shapes what the model learns but was not checked by hand; the numbers above were computed by code.

Task. From the incremental transcript of a speaker's current turn (text so far, may end mid-word), decide whether the speaker has finished their conversational turn (finished = true) or is still speaking / likely to say more (finished = false). The model is called repeatedly as tokens arrive; this spec is a single binary classification of a partial-utterance.

Inputs. A raw ASR transcript fragment of the user's current in-progress turn, starting at session/session-turn boundary (the moment the user began speaking after the assistant's previous turn ended). May be one word, a few words, a partial clause, or a near-complete sentence. May contain trailing whitespace, a trailing partial token (mid-word), punctuation the ASR has or has not yet emitted, or filler words ("um", "uh", "you know"). It is case-preserved, lightly punctuated, no speaker IDs, no timestamps. Never includes the assistant's speech. Examples: "I think we should", "I think we should schedule a call next", "I think we should schedule a call next week.", "um so basically the thing is", "I need to reschedule my flight to San Francisco on", "I need to reschedule my flight to San Francisco on the 14th".

Labels.

• finished: The user has completed a self-contained, communicative turn: an intent that the other party can act on or respond to without further words. Typically ends in a period, question mark, or a clear syntactic/semantic terminal (imperative, declaration, complete question, complete thanks/refusal, complete answer to the last question). Includes turns that end on a filler or trailing comma when the intent is clearly complete ("Okay." / "Sure, fine").
• unfinished: The user is still producing the turn: the utterance is syntactically or pragmatically incomplete (missing object, missing verb, trailing "and", "but", "so", relative-pronoun gap, question word without interrogative clause), OR it ends on a pause/filler with more clearly coming, OR the ASR fragment cuts mid-word. Includes mid-question ("what does the"), mid-list ("red and blue").

Hard cases the data was asked to cover.

• "I think" -- unfinished (almost always more follows) but could close as a complete assertion in casual talk; default unfinished
• "So basically, the thing is" -- unfinished (expectant frame)
• "I need help" -- finished (complete request)
• "I need help with" -- unfinished (trailing preposition)
• "What does this do?" -- finished
• "What does this" -- unfinished (verb missing)
• "Yeah" -- finished (backchannel acknowledgment)
• "Yeah, um, and then so" -- unfinished
• "I would like to book a flight to" -- unfinished
• "I would like to book a flight to Paris" -- finished even with no period
• "The number is 123" -- UNFINISHED (trailing number often not the end of a phone/ID; more digits expected) -- but finished if the prior question was "give me the last 3 digits"; model must use context cue; default unfinished when length is short
• "Okay fine." -- finished
• "Okay fine, and" -- unfinished
• "Um, wait, I mean" -- unfinished (self-repair in progress)
• "I mean, the red one" -- finished (answer completed)
• "I mean the" -- unfinished
• "Actually, no" -- finished (clear refusal)
• "Actually" -- UNFINISHED (usually a hedge before the real content)
• "Actually, no thanks" -- finished
• "Can you hear me" -- finished (yes/no question)
• "Can you hear" -- UNFINISHED (object missing; though colloquially closable, default unfinished)
• "Thank you very much" -- finished
• "Thank you very" -- UNFINISHED
• "The total is" -- UNFINISHED (amount gap is the most common real hard case -- user is about to say a number)
• "The total is 42" -- finished
• "I live in" -- UNFINISHED (place gap)
• "I live in 5th Street" -- finished even though no city
• "So what I am thinking is" -- UNFINISHED
• "So what I am thinking is we ship on Friday" -- finished
• "Hmm" -- ambiguous: treat as UNFINISHED (hesitation, more likely to follow) unless the prior context is a yes/no question and only "hmm" is given, in which case finished; in a zero-context classifier default to finished only if length >= 2
• "Yeah, I guess so" -- finished
• "Yeah, I guess" -- finished even without "so" (colloquial close)
• "I, uh, the thing" -- UNFINISHED (mid-repair)
• "I, uh, the thing is, it is broken" -- finished
• "Give me the address of" -- UNFINISHED
• "Give me the address of the blue one" -- finished
• "I am" -- UNFINISHED (tense expects complement)
• "I am fine" -- finished
• "Are you" -- UNFINISHED (predicate missing)
• "Are you free" -- finished (predicate implied),"", -- finished (yes/no question),
• "I want" -- UNFINISHED
• "I want it" -- finished
• "Um, what about" -- UNFINISHED
• "Um, what about Tuesday" -- finished

---

Prose by claude-opus-5-5 from the facts above; every number is checked against them.