Vibrato-ai/vibrato-v2

Model Card: Vibrato v2.2

Model Details

• Name: Vibrato
• Version: 2.2 (current; v2.0 remains available in this repo — see Versions below)
• Type: Multi-task 1D CNN for vocal analysis (269,464 parameters)
• Framework: PyTorch (training), CoreML FP16 ML Program (deployment, iOS 17+)
• License: Apache-2.0
• Repository: Vibrato-ai/vibrato-v2

Project

Vibrato is an open vocal-AI model for the singing community, maintained by Anycompany LLC. It powers Toney, a free iOS vocal-training app, and is published for anyone to use, study, and improve. This repository contains the CoreML model, the PyTorch checkpoint, the complete training pipeline (training/), and the full evaluation reports (V2_EVAL.md, FAIR_BASELINE.md, SELECTION_STUDY.md).

To our knowledge, Vibrato is the only publicly released singing-technique classifier whose entire chain — training data, labels, code, and weights — is permissively licensed for commercial use. (A 2026-08 survey found every other public technique/register model to be trained on non-commercial corpora.)

What changed in v2.2

Nothing about training — only checkpoint selection. v2.0 selected its checkpoint by validation quality-MSE alone; a pre-registered study (SELECTION_STUDY.md) reran the identical training (bit-identical losses over all shared epochs) and selected instead by a balanced criterion (technique macro-F1 + vowel accuracy + mean quality R²)/3. The two criteria correlate only weakly (Spearman ρ = 0.402). The balanced pick (epoch 47 vs 19) passed all five pre-registered GO gates — nothing regressed beyond noise, and every quality dimension improved.

v2.2 also ships with a fair v1 baseline (FAIR_BASELINE.md): the v1 recipe retrained on v2's singer-disjoint splits, removing the training-set bias from v1's published numbers. Result: v1's vowel accuracy drops 0.716 → 0.669 and voice-type 0.534 → 0.467 once evaluated fairly, so v2's vowel score — previously described as "parity" — is a clear win. The catastrophic v1 quality finding is unchanged by the leak (−2.78 leaked, −2.68 fair): it is a labeling defect, not a generalization one.

Evaluation (held-out singers)

head	v2.2	v2.0	v1 recipe (fair¹)	majority baseline
technique accuracy / macro-F1	0.696 / 0.463	0.690 / 0.451	0.398 / 0.395	0.704 / ≈0.207
vowel accuracy / macro-F1	0.747 / 0.745	0.709 / 0.703	0.669 / 0.676	0.201
quality mean R² (5 dims)	0.566	0.528	−2.682	0 by definition

Per-dimension quality R² (v2.2 / v2.0): power 0.973 / 0.962, brightness 0.787 / 0.730, breathiness 0.643 / 0.582, strain 0.377 / 0.325, stability 0.050 / 0.043.

¹ v1-recipe numbers are the best per metric across 6 fair retrained checkpoints (3 seeds × 2 selection rules) — deliberately generous, so every v2 margin is a lower bound. v1's originally published numbers were training-set numbers (18/20 singers, including the whole test set); see FAIR_BASELINE.md.

Known limitations — read before deploying

These outputs should be suppressed or de-emphasised in user-facing products:

• vibrato recall remains near zero. The 0.25 s input window holds only 1–2 cycles of a 5–7 Hz modulation — an architectural limit, not a labeling problem. (The Toney app measures vibrato from a pitch contour instead.)
• stability R² is 0.050 — near-zero; not reliable. Measure f0 dispersion directly instead.
• falsetto, nasal, mixed, schwa logits are untrained (0 samples in the corpus; VocalSet contains no falsetto recordings at all).
• Voice type / fach is not learnable from VocalSet. Only the female/male split is real. The 6-class head exists for interface compatibility and should not be surfaced.
• Technique accuracy (0.696) remains marginally below the majority baseline (0.704) on this 70%-straight corpus; macro-F1 (0.463 vs ≈0.207) is the meaningful number.
• Trained on 20 professional singers, one corpus, 8.8 hours. No coverage of amateur voices or non-Western styles beyond synthetic augmentation.

Model Architecture

Identical to v1 and v2.0 (drop-in compatible):

Input: audio_input [1, 4000] raw un-normalized audio (0.25s @ 16kHz)
  -> SharedEncoder (4x ConvBlock, BatchNorm, stride-2, AdaptiveAvgPool)
  -> 256-dim feature vector
  -> voice_logits [6]      (softmax at inference; see limitations — do not surface)
  -> technique_logits [7]  (belt, falsetto, vibrato, straight, breathy, nasal, mixed)
  -> vowel_logits [6]      (a, e, i, o, u, schwa)
  -> quality_scores [5]    (brightness, breathiness, strain, power, stability; sigmoid in-model)

Class orders are fixed and must match labels.plist — never sort them.

Versions & Files

file	description
vibrato-v2_2.mlpackage	Current. CoreML FP16 ML Program (548 KB), epoch 47, balanced selection
vibrato-v2_2.pt	Current. PyTorch checkpoint (epoch 47)
vibrato-v2.mlpackage / vibrato-v2.pt	v2.0 (epoch 19, quality-MSE selection) — kept for reproducibility
labels.plist	Class-order label map (identical across v2.0/v2.2)
model.py	Architecture (unchanged since v1)
training/	Full pipeline: cache builder, Praat labeling, augmentation, training, eval, export
V2_EVAL.md	v2.0 training & evaluation report
FAIR_BASELINE.md	Fair v1-recipe baseline on singer-disjoint splits
SELECTION_STUDY.md	Pre-registered checkpoint-selection study (v2.2's GO decision)
export_report.json / export_report_v2_2.json	CoreML export parity reports
parity_full_v2_2.json	Full-split PyTorch↔CoreML parity (softmax deltas ≤ 0.012, argmax agreement ≥ 0.9986 over 4,352 crops)

CoreML parity note: raw logit deltas grow with logit magnitude (max 0.080 on v2.2's larger logits) while softmax/quality deltas remain ≤ 0.012 / ≤ 0.0033 — compare probabilities, not unbounded logits.

To reproduce v2.2: the v2 pipeline plus the selection study — python training/build_cache.py → python training/train_v2.py → selection per SELECTION_STUDY.md (seed 1337; ≈41 min additional training on an Apple M4 Pro, MPS).

Provenance

component	source	license
Training audio	VocalSet (Wilkins et al., ISMIR 2018)	CC BY 4.0
Quality labels	Measured acoustics (Praat/parselmouth CPPS, jitter+shimmer, f0-σ, RMS) computed from VocalSet	derived, this project
Technique/vowel labels	VocalSet's own human annotations	CC BY 4.0
Architecture & weights	This project, trained from scratch	Apache-2.0

No scraped, non-commercial, or unlicensed audio was used at any stage, and no other model's outputs were used as training labels.

Companion-model disclosure: the Toney app pairs Vibrato with SwiftF0 (MIT) for pitch tracking. SwiftF0 is not part of this repository or its Apache-2.0 grant; its upstream weights were trained partly on corpora that are non-commercial or unlicensed (MIR-1K, MDB-stem-synth), which we note here for transparency.

Dataset & Attribution

Trained on VocalSet (Wilkins et al., ISMIR 2018), CC BY 4.0:

Julia Wilkins, Prem Seetharaman, Alison Wahl, Bryan Pardo. "VocalSet: A Singing Voice Dataset." ISMIR 2018.

Out of Scope

Speech recognition, language identification, speaker identification, medical diagnosis of vocal disorders. The model analyzes singing exercises; it is not a biometric system and must not be used as one.