aufklarer/VAP-OTO-CoreML

VAP otoSpeech — Compiled Core ML .mlmodelc

Unofficial format export of the public oto.ckpt from viks66/VAP_checkpoints, using Erik Ekstedt's VoiceActivityProjection architecture. No fine-tuning or weight quantization was applied. The original authors and checkpoint distributor do not endorse this export.

VAP listens to synchronized user and agent channels and forecasts their speech activity. It provides inputs to a turn policy; it does not directly decide when an agent should answer.

Property	Value
Parameters	5,785,117
Format	Compiled Core ML .mlmodelc
Precision / quantization	Float32 / none
Graph size	39,282,957 bytes
Audio	16 kHz stereo Float32 PCM, channel order user then agent
Input	audio, shape [1, 2, 320000] (20 seconds)
Outputs	p_now, p_future, vad, each [1, 1000, 2]
Projection bins	0.2, 0.4, 0.6 and 0.8 seconds
Weights licence	CC BY 4.0
Architecture code licence	MIT

p_now aggregates the first two projection bins; p_future aggregates the last two. Both give relative speaker forecasts in [user, agent] order. vad gives current voice-activity probabilities. All 1,000 output frames are available; SDKs expose the latest frame. Prepend silence for a shorter context and retain the latest 20 seconds of a longer conversation. An inactive agent needs a zero channel of the same length. Each call recomputes the window; this graph does not cache streaming network state.

Files

File	Bytes	Description
vap.mlmodelc	39,282,957	Compiled graph directory
config.json	771	Input/output geometry and pinned source provenance
LICENSE	19,773	Licence text
NOTICE	1,039	Attribution and conversion notice
validation.json	3,267	All-frame parity and timing against the original checkpoint
benchmark.json	5,486	Matched TurnBench pilot and original-checkpoint controls
SHA256SUMS	—	SHA-256 for each release file
README.md	—	This model card

Validation and compute samples

All outputs over all 1,000 frames were compared with the original PyTorch checkpoint on four 20-second fixtures: silence, user-only signal, agent-only signal and simultaneous signals. Maximum absolute error for this format was 0.00000477, below the 0.0005 tolerance. Native model tests additionally check channel order and fixed original-checkpoint reference values.

Apple M5 Pro, 48 GiB, CPU only. Each row is one validation request with model loading excluded:

Fixture	20-second window compute time
silence	155.99 ms
user	111.54 ms
agent	114.72 ms
both	111.23 ms

These desktop samples do not measure phone performance or a complete turn-detection policy. The matched TurnBench pilot below evaluates this export with a fixed readout and original-checkpoint controls.

TurnBench development pilot

Two fixed English conversations, 20 and 21: 148 turn endings and 94 labelled mid-turn pauses. No threshold was selected on this pilot. The original TurnBench scorer and labels were retained.

Backend	Turn endings caught	False alarms	Audio-time delay, median	Request compute, median
Core ML	83.8% (124/148)	2.1% (2/94)	945.5 ms	113.05 ms
ONNX Runtime	83.8% (124/148)	2.1% (2/94)	945.5 ms	377.93 ms
Original PyTorch checkpoint	83.8% (124/148)	2.1% (2/94)	945.5 ms	409.97 ms

The same 20-second stereo window is read every 500 ms, taking only its latest frame. The end-of-turn threshold is fixed at 0.9161 with the original 2 s refractory rule. Both exports and the original checkpoint use identical windows and readout.

Audio-time delay follows audio availability in offline replay; it excludes the separately measured request computation. These values do not measure complete deployed-agent latency or phone performance. Full counts, policies, checksums and reference comparisons are in benchmark.json and the leaderboard method.

Usage

Tested with Python 3.12. Install the inference dependencies and download CLI:

pip install coremltools==9.0 numpy==2.2.6 huggingface-hub==2.2.0

hf download aufklarer/VAP-OTO-CoreML --local-dir vap-coreml

Python on macOS:

import json
from pathlib import Path
import numpy as np

bundle = Path("vap-coreml")
config = json.loads((bundle / "config.json").read_text())
audio = np.zeros((1, 2, config["window_samples"]), np.float32)
import coremltools as ct
model = ct.models.CompiledMLModel(str(bundle / config["compiled_model"]),
                                  compute_units=ct.ComputeUnit.CPU_ONLY)
outputs = model.predict({"audio": audio})
latest = {name: np.asarray(outputs[name])[0, -1] for name in config["output_names"]}
print(latest)  # Each value is [user, agent].

Replace the zero input with synchronized microphone and playback PCM. The application combines these forecasts with its response and interruption policy. Core ML requires macOS 15+ / iOS 18+.

Source and attribution

• Weights: viks66/VAP_checkpoints, oto.ckpt, revision b9aa0ba1718221153e04ef343c7f3b8cf84bfbc3, CC BY 4.0.
• Architecture: ErikEkstedt/VoiceActivityProjection, revision f39a78b23a6dccdbedd106e00b48c410b8739f5d, MIT.
• Changes: fixed-window waveform and forecast export, format packaging and runtime adapters. Preserve LICENSE and NOTICE; the two components retain their respective licences. The checkpoint checksum and channel geometry are recorded in config.json.

Links

• speech-swift — Apple SDK
• Installation and guides
• Soniqo · Blog