soniqo/VAP-OTO-ONNX

VAP otoSpeech — ONNX, opset 17

Unofficial format export of the public oto.ckpt from viks66/VAP_checkpoints, using Erik Ekstedt's VoiceActivityProjection architecture. No fine-tuning or weight quantization was applied. The original authors and checkpoint distributor do not endorse this export.

VAP listens to synchronized user and agent channels and forecasts their speech activity. It provides inputs to a turn policy; it does not directly decide when an agent should answer.

Property	Value
Parameters	5,785,117
Format	ONNX, opset 17
Precision / quantization	Float32 / none
Graph size	39,564,833 bytes
Audio	16 kHz stereo Float32 PCM, channel order user then agent
Input	audio, shape [1, 2, 320000] (20 seconds)
Outputs	p_now, p_future, vad, each [1, 1000, 2]
Projection bins	0.2, 0.4, 0.6 and 0.8 seconds
Weights licence	CC BY 4.0
Architecture code licence	MIT

p_now aggregates the first two projection bins; p_future aggregates the last two. Both give relative speaker forecasts in [user, agent] order. vad gives current voice-activity probabilities. All 1,000 output frames are available; SDKs expose the latest frame. Prepend silence for a shorter context and retain the latest 20 seconds of a longer conversation. An inactive agent needs a zero channel of the same length. Each call recomputes the window; this graph does not cache streaming network state.

Files

File	Bytes	Description
vap.onnx	39,564,833	Float32 inference graph
config.json	761	Input/output geometry and pinned source provenance
LICENSE	19,773	Licence text
NOTICE	1,039	Attribution and conversion notice
validation.json	3,267	All-frame parity and timing against the original checkpoint
benchmark.json	5,486	Matched TurnBench pilot and original-checkpoint controls
SHA256SUMS	—	SHA-256 for each release file
README.md	—	This model card

Validation and compute samples

All outputs over all 1,000 frames were compared with the original PyTorch checkpoint on four 20-second fixtures: silence, user-only signal, agent-only signal and simultaneous signals. Maximum absolute error for this format was 0.00000253, below the 0.0005 tolerance. Native model tests additionally check channel order and fixed original-checkpoint reference values.

Apple M5 Pro, 48 GiB, CPU only; ONNX Runtime with two threads. Each row is one validation request with model loading excluded:

Fixture	20-second window compute time
silence	361.85 ms
user	359.69 ms
agent	351.98 ms
both	353.39 ms

These desktop samples do not measure phone performance or a complete turn-detection policy. The matched TurnBench pilot below evaluates this export with a fixed readout and original-checkpoint controls.

TurnBench development pilot

Two fixed English conversations, 20 and 21: 148 turn endings and 94 labelled mid-turn pauses. No threshold was selected on this pilot. The original TurnBench scorer and labels were retained.

Backend	Turn endings caught	False alarms	Audio-time delay, median	Request compute, median
Core ML	83.8% (124/148)	2.1% (2/94)	945.5 ms	113.05 ms
ONNX Runtime	83.8% (124/148)	2.1% (2/94)	945.5 ms	377.93 ms
Original PyTorch checkpoint	83.8% (124/148)	2.1% (2/94)	945.5 ms	409.97 ms

The same 20-second stereo window is read every 500 ms, taking only its latest frame. The end-of-turn threshold is fixed at 0.9161 with the original 2 s refractory rule. Both exports and the original checkpoint use identical windows and readout.

Audio-time delay follows audio availability in offline replay; it excludes the separately measured request computation. These values do not measure complete deployed-agent latency or phone performance. Full counts, policies, checksums and reference comparisons are in benchmark.json and the leaderboard method.

Usage with the native SDKs

Build speech-android or speech-core from source for these APIs. The Android model manager downloads the pinned ONNX bundle and its attribution files.

val directory = ModelManager.ensureVapModels(context)
withContext(Dispatchers.Default) {
    VoiceActivityProjection(VoiceActivityProjectionConfig(directory)).use { model ->
        val userPcm = FloatArray(16_000)
        val agentPcm = FloatArray(userPcm.size)
        val forecast = model.forecast(userPcm, agentPcm)
        println(forecast.userNearTerm)
        println(forecast.agentNearTerm)
    }
}

Replace the silence with synchronized microphone and playback PCM at 16 kHz. Use equal-length channels and zeros for an inactive agent. The SDK uses the latest 20 seconds and pads shorter inputs with leading silence. Your application chooses the response and interruption policy.

For C++, build with SPEECH_CORE_WITH_ONNX=ON and download the release files:

mkdir -p vap-onnx
for file in vap.onnx config.json LICENSE NOTICE; do
    curl --fail --location "https://huggingface.co/soniqo/VAP-OTO-ONNX/resolve/v1.0.0/$file" --output "vap-onnx/$file"
done

#include <speech_core/models/onnx_vap.h>
#include <vector>

std::vector<float> user(16'000, 0.0f), agent(user.size(), 0.0f);
speech_core::OnnxVap model("vap-onnx/vap.onnx");
auto forecast = model.forecast(user.data(), agent.data(), user.size(), 16'000);
// p_now[0]: user; p_now[1]: agent.

For Apple platforms, use VAP CoreML with speech-swift. The integration guide covers all three SDKs.

Source and attribution

• Weights: viks66/VAP_checkpoints, oto.ckpt, revision b9aa0ba1718221153e04ef343c7f3b8cf84bfbc3, CC BY 4.0.
• Architecture: ErikEkstedt/VoiceActivityProjection, revision f39a78b23a6dccdbedd106e00b48c410b8739f5d, MIT.
• Changes: fixed-window waveform and forecast export, format packaging and runtime adapters. Preserve LICENSE and NOTICE; the two components retain their respective licences. The checkpoint checksum and channel geometry are recorded in config.json.

Links

• speech-android — Android SDK
• Android setup
• speech-core — C++ runtime
• C++ runtime documentation
• Soniqo · Blog