byojl-vad

Voice activity detection for a realtime voice agent. For every 10ms of microphone audio, say whether a human is speaking. Must ignore background noise, coughs, laughter, door slams, keyboard, crying babies and other non-speech human sounds. Runs on-device next to the agent, streaming, in well under 1ms per frame.

A tiny, single-purpose System 1 model built with BYOJL: a JEPA-style encoder trained only for this task, with a calibrated decision policy, shipped as a 0.10M-parameter ONNX bundle that runs on any CPU with just numpy + onnxruntime.

task	audio / frame (one decision every 10 ms)
labels	non_speech, speech
parameters	101,458
backbone	tiny-crnn · embedding dim 96
objective	f1, primary suite ava
decision policy	hysteresis, threshold 0.325 (calibrated on ava's calib split)
CPU, streaming	0.173 ms per 30 ms chunk, 1 thread (PyTorch; the ONNX bundle is faster)

Labels

• non_speech: silence, noise, music, and non-verbal human sounds (coughing, laughing, breathing, crying)
• speech: a frame inside an utterance of human speech, including pauses up to ~300ms inside an utterance

Comparison

Every model below, this one and the baselines, was scored by the same code on the same frozen test splits, with its decision policy calibrated on the same calib splits. Bold = this model.

model	params	builtin f1	ava f1 (primary)	builtin ROC-AUC	ava ROC-AUC
v2 (this model)	0.10M	0.8794	0.8763	0.976	0.939
silero (baseline)	0.55M	0.8850	0.8555	0.981	0.923
v1 (BYOJL experiment)	0.10M	0.8941	0.8485	0.982	0.900
v0 (BYOJL experiment)	0.10M	0.8906	0.8451	0.983	0.900
energy (baseline)	–	0.7281	0.7275	0.828	0.694

Test evaluation (this model)

suite	dataset	test items	f1	precision	recall	ROC-AUC	avg precision	decision threshold
builtin	vad-builtin	600	0.8794	0.8517	0.9089	0.9760	0.9684	0.750
ava	nccratliri/vad-human-ava-speech	20	0.8763	0.8456	0.9094	0.9391	0.9494	0.325

Usage

pip install huggingface_hub numpy onnxruntime soundfile

from huggingface_hub import snapshot_download
import sys
path = snapshot_download("bsye/byojl-vad")
sys.path.insert(0, path)
from byojl_rt import load

m = load(path)
import soundfile as sf
wav, sr = sf.read("clip.wav", dtype="float32")   # 16 kHz mono
print(m.predict(wav)["segments"])
emb = m.encode(...)                # the learned representation (JEPA-style), for probes / retrieval / other heads

Streaming

s = m.stream()
for chunk in mic_chunks():         # float32 mono, any size
    out = s.step(chunk)            # {"probs": [...per 10ms...], "speaking": bool, "events": [{"type": "start"|"end", "t_s": ...}]}

Local HTTP server

python byojl_rt.py serve . --port 8800
# GET /v1/models · POST /v1/vad/predict · POST /v1/vad/encode

How it was built

1. Spec: labels, definitions, objective and data plan in spec.yaml (included in this repo).
2. Data (8,400 training / 1,000 dev examples):

• openslr/librispeech_asr: role positive
• ashraq/esc50: role negative
• nccratliri/vad-human-ava-speech: role labeled, partition train

3. LeJEPA pretraining (800 steps): multi-view prediction + SIGReg (isotropic-Gaussian regulariser on random projections), no EMA teacher, no stop-gradient.
4. Supervised training: tiny head on the backbone, class-weighted, checkpoint chosen on dev (best epoch 18).
5. Evaluation: frozen eval suites; thresholds calibrated on each suite's calib split; baselines scored identically.
6. Improvement lineage (each step proposed by byojl improve from the previous board):

• v1: initial spec
• v2 ← real-data-ava (from v1): ava: objective 0.848 vs silero 0.855, 0.046 below builtin. The model has never seen this domain; train on the source's train partition (hash-disjoint from the suite's calib/test files).
• v2 ← utterance-labels (from v1): ava: calibration had to move the decision threshold to 0.20 (P=0.824, R=0.875) -> the suite's label convention differs from training's. Training labels only voiced frames (gaps > 60ms are negative); segment suites usually label whole utterances. Fill gaps up to 300ms in synthetic labels.

Limitations

• Trained on the train partition of the source behind ava (file-level hash split, disjoint from that suite's calib/test files). Results there are in-domain; baselines may not have seen that domain.
• Training data includes ESC-50 (CC BY-NC 3.0): non-commercial. Check licences before commercial use.
• Evaluated on builtin (600 test items), ava (20 test items). Small suites have wide confidence intervals.

Reproduce

git clone https://github.com/bsye/BYOJL && cd BYOJL && pip install -e .
# put spec.yaml from this repo at specialists/vad/spec.yaml, then:
byojl build vad && byojl board vad && byojl export vad