Scicom-intl/rmvpe-onnx

RMVPE — ONNX

RMVPE (Wei et al., RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music, Interspeech 2023) as shipped by the RVC project (lj1995/VoiceConversionWebUI, rmvpe.pt). A deep U-Net + BiGRU on a 128-band log-mel; robust to noise and accompaniment.

Exported to ONNX with the model's own front end inside the graph: feed 16 kHz mono audio, get the model's own output — no torch, librosa or reimplemented STFT/mel needed, and nothing that can quietly differ from the original. Every export was checked frame by frame against the original PyTorch pipeline (below), and measured on PTDB-TUG speech against the other exports in this family and against pyin.

file	model	parameters	size
rmvpe.onnx	RMVPE	90.5 M	185.8 MB

Same family: rmvpe-onnx · fcpe-onnx · crepe-onnx · pesto-onnx.

Use

# pip install onnxruntime numpy soundfile soxr huggingface_hub
import soundfile as sf, soxr
from huggingface_hub import hf_hub_download
import importlib.util

repo = "Scicom-intl/rmvpe-onnx"
spec = importlib.util.spec_from_file_location("pitch_onnx", hf_hub_download(repo, "pitch_onnx.py"))
pitch_onnx = importlib.util.module_from_spec(spec); spec.loader.exec_module(pitch_onnx)

model = pitch_onnx.PitchModel(hf_hub_download(repo, "rmvpe.onnx"))   # threshold= to override voicing
y, sr = sf.read("speech.wav", dtype="float32", always_2d=True)
y = soxr.resample(y.mean(1), sr, 16000)
f0, voiced, confidence = model(y)   # one per 10 ms frame (frame i at i * 10 ms; FCPE + 5 ms); f0 = 0 unvoiced

pitch_onnx.py is numpy + onnxruntime only: it cuts CREPE's frames, decodes the 360-bin salience the way the models ship (salience-weighted mean of the 9 bins around the peak; CREPE/RMVPE bins are 20 cents from 32.70 Hz, FCPE's are evenly spaced 32.7–1975.5 Hz) and applies the voicing threshold.

Details

RMVPE

• Input: audio float32 [1, N] — 16 kHz mono
• Output: salience float32 [1, T, 360], T = N // 160 + 1 frames, frame i centred at i × 10 ms.
• Front end: RVC's mel (centred reflect STFT, n_fft 1024, hop 160, 128 HTK mels 30–8000 Hz, log clamp 1e-5) and the U-Net's pad to a multiple of 32 frames are inside the graph.
• Exactness vs the original PyTorch pipeline: 4 clips (speech of three lengths + 3 s of noise, 2696 frames): max |ONNX − PyTorch| salience 5.66e-06, max pitch difference 0.00020 cents (1 cent = 1/100 semitone), voicing agreement 100.0 %.
• Voicing threshold: 0.35 (chosen on PTDB-TUG's dev speakers, lowest F0 frame error; the upstream default is 0.03).

Accuracy on PTDB-TUG's 16 test speakers (400 utterances per condition), in %:

condition	RPA ↑	RCA ↑	GPE ↓	VDE ↓	FFE ↓	VR ↑	VFA ↓
clean	91.15	92.09	1.36	2.78	3.09	94.34	1.89
white noise 10 dB	90.98	92.04	1.51	2.84	3.18	94.50	2.01
white noise 0 dB	89.04	90.26	1.76	3.17	3.56	93.14	2.01
pink noise 10 dB	89.41	90.59	1.70	3.10	3.48	93.10	1.91
pink noise 0 dB	84.31	85.57	1.94	4.05	4.47	88.98	1.87

CPU speed (onnxruntime, x86 server CPU; RTF = compute seconds per second of audio; peak RSS is the whole Python process, librosa and numpy included):

threads	audio	RTF (compute s / audio s)	peak RSS
1	10 s	0.0559	1079 MB
4	10 s	0.0166	1052 MB
1	30 s	0.0539	1089 MB

Comparison on PTDB-TUG (F0 frame error, %, lower is better)

FFE = (voicing-decision errors + gross pitch errors > 20 %) / all frames — the single number that counts both "is this frame voice" and "how wrong is the pitch". Reference: PTDB-TUG's laryngograph-derived f0 (RAPT, 10 ms). Speakers F01 F02 M01 M02 were used ONLY to pick each method's voicing threshold and frame lag (clean audio, lowest FFE); the table is the other 16 speakers, 400 utterances per condition, scored once. Noise is added at the stated SNR over the whole utterance. Best per row in bold.

condition	RMVPE	FCPE	CREPE full	CREPE tiny	PESTO	pyin (librosa 0.11)
clean	3.09	2.77	6.43	6.33	5.58	7.72
white noise 10 dB	3.18	2.95	5.98	6.07	5.99	8.32
white noise 0 dB	3.56	3.39	6.92	7.21	7.46	16.92
pink noise 10 dB	3.48	3.30	6.59	6.41	6.54	8.36
pink noise 0 dB	4.47	4.78	9.48	9.31	8.99	17.18

Metrics per condition for every method, the chosen thresholds/lags and the CPU timings are in benchmark.json; the harness is export/evaluate.py.

How it was exported

export/export.py (torch 2.8, legacy ONNX exporter, opset 17). The STFT is two conv1ds with fixed DFT kernels (torch.stft's complex output doesn't export), equal to torch.stft to float32 rounding. export/decode.py is the decoder the parity check uses. RMVPE ships fp16 weights: the graph stores every fp16-exact weight as fp16 with a Cast to fp32 in front, so it computes in fp32 with bit-identical results at half the file size.

License and credits

Weights: RVC-Project rmvpe.pt (MIT, Copyright (c) 2023 liujing04, 源文雨, Ftps), from the original Dream-High/RMVPE (Apache-2.0, included as LICENSE-RMVPE-original). Network code rmvpe_net.py is RVC's, verbatim. ONNX conversion, evaluation and packaging by Scicom.