restore, ONNX (float16 weights)

A voice restored to a studio take: speech or singing, English, French or Russian, recorded in a room, in noise, on a phone or through a codec, back as 48 kHz full band. One network, one pass, in the STFT. Trained by audiojs for @audio/neural-restore, which computes the STFT, the chunks and the level in JavaScript and runs this file through onnxruntime (Node, or a browser's onnxruntime-web on WebGPU or wasm); audio's restore() runs it.

File Size SHA-256
restore.fp16.onnx 35.9 MB (35,929,213 bytes) 46eb6ebb05cc50bb6c3b1af87e3c4542bf30cedbcc356e2c68646d3bbf9a65f1

Float16 weights, float32 compute: every weight tensor of 1024 values or more stored as float16 and Cast to float32 in the graph (scripts/export.py). The float32 export it is made from is 71,671,387 bytes.

Model

Vocos's ConvNeXt stack over STFT frames (dim 512, 8 blocks) with two self-attention layers for a room's tail, reading each frame's log magnitude and phase and writing a change of log magnitude (unbounded: a band can be taken down 60 dB or raised from nothing) and a rotation of the input's phase (the identity at init: what is there keeps its phase unless the net turns it); 17.9M parameters, the STFT outside the graph.

name shape
input spec [1, 2, 1025, T] float32 STFT of one channel at 48 kHz: n_fft 2048, hop 512, periodic Hann, centered with reflect padding, unnormalized; re, im
output spec_out [1, 2, 1025, T] float32 the restored STFT, the same layout

T is dynamic; opset 17. The network hears its input at βˆ’20 dBFS RMS: @audio/neural-restore brings the whole input to that level, runs it in 10 s runs overlapping by 1 s, raised-cosine crossfaded, and undoes the gain, as scripts/infer.py does; other rates are resampled to 48 kHz and back.

import restore from '@audio/neural-restore'
let y = await restore(samples, { sampleRate: 44100 })   // mono Float32Array; the same rate and length back

Training

Trained from scratch: 300,000 steps on an Apple M4 Max GPU (PyTorch 2.14 on MPS, batch 8 Γ— 4 s), 2026-10-15 to 2026-10-22. These are step 290,000's weights: best SI-SDR and mel distance on the validation draw; 270k–300k are within each other's noise on every measure. A regressor first (multi-resolution STFT, log-mel, a compressed complex L1 on the model's grid for the phase, WavLM Base+ features: FINALLY's recipe), then adversarially against HiFi-GAN's multi-period and UnivNet's multi-resolution discriminators with feature matching. Every loss sees target and estimate through a low-pass at the target's own measured bandwidth, so 24 kHz corpora train beside full-band ones. The inputs are made on the fly from clean voices: a room (a measured response or a synthetic one), noise, a cheap microphone, a compressor, clipping, a band limit, MP3 / AAC / Opus, 8–16-bit words. The code is scripts/ in @audio/neural-restore.

Set Hours Rate Licence Role
VCTK 0.92, mic1, speakers p232 and p257 held out 44 48 kHz CC BY 4.0 English, full band
LibriTTS-R train-clean-100 54 24 kHz CC BY 4.0 English, to 12 kHz
SIWIS 10 44.1 kHz CC BY 4.0 French, one professional voice
Dialogs (langswap) 17.4 44.1 kHz OpenRAIL (commercial use allowed; its use restrictions flow down to derived models) Russian, three actors, studio; its manifest is separate so a licence-clean variant is one flag away
Supertonic 3 (Supertone, OpenRAIL-M weights), ten voices, public-domain text ru 12, fr 8, en 4 44.1 kHz the synthesised audio is ours the only full-band Russian beyond Dialogs; DNSMOS OVRL 3.43 / 3.40 / 3.39 (ru / fr / en) against 3.39 for VCTK's clean test set and 3.20 for Dialogs
Pocket TTS (Kyutai, CC BY 4.0), eight voices fr 6, en 3 24 kHz ours more voices, to 12 kHz
VocalSet (the local subset) 1.8 44.1 kHz CC BY 4.0 singing, full band
MIT IR Survey, the even-numbered rooms 135 rooms 32 kHz unstated: training only, never redistributed rooms; the odd-numbered ones are audio's bench rooms and stay out
MUSAN noise 6 16 kHz CC BY 4.0 noise

Texts read by the synthetic voices: Project Gutenberg (Austen, Dickens, Melville, Doyle; Hugo, Flaubert, Dumas, Proust; four Russian titles) and Dialogs' own transcripts. WavLM Base+ (Microsoft, MIT) is a feature loss, not data. A measured room or noise recorded at 16 or 32 kHz is continued to 24 kHz with synthetic content of the same decay or colour, so a measured room never low-passes the voice by its own rate. Validation pairs come from VoiceBank's clean test set (p232, p257), never trained on.

Measured

24 validation pairs: VoiceBank's test speakers through the chain (rooms, noise, microphones, codecs, band limits), each output at the reference's level, scored through the pair's band limit. The baselines are audio's own ops on the same inputs (DeepFilterNet3 through deepfilter(); auto('speech')).

System LSD ↓ SI-SDR ↑ PESQ ↑ DNSMOS SIG BAK OVRL band, Hz
input 3.30 βˆ’5.2 1.77 2.50 2.41 1.97 16 947
deepfilter() 2.85 βˆ’2.1 2.13 2.93 3.75 2.59 17 104
deepfilter(0) 2.71 βˆ’2.1 2.07 2.91 3.90 2.63 16 366
auto('speech') 2.91 βˆ’4.0 1.88 2.91 3.29 2.45 16 265
these weights (step 290k) 0.79 +6.0 2.40 3.21 3.91 2.89 24 000
reference 0 4.64 3.29 3.89 2.95 24 000

SI-SDR against a dry reference is a hard measure for this chain (rooms and codec lags put the input itself at βˆ’5 dB); the listen decides.

This file against PyTorch: the float32 export is 108–121 dB from PyTorch on five real takes, one of them 16 s in chunks; this file 68–74 dB, the same scores to 0.1 dB SI-SDR. @audio/neural-restore's JavaScript pipeline with this file is 73 dB from scripts/infer.py on its fixture in Node, 70 dB on WebGPU and wasm in a browser; 10.5 s restored in 0.36 s on the M4 Max's WebGPU, 1.3 s on wasm.

Limits

  • Measured on the 24 English pairs above only. French, Russian, singing, real recordings, lost or invented words (WER) and Adobe Podcast Enhance are not measured.
  • Above an input's band limit the network writes content the input does not hold: plausible, not the original.
  • Full-band Russian comes from three Dialogs actors and synthetic voices, French from one professional voice (SIWIS) and synthetic voices, singing from 1.8 h of VocalSet.
  • One channel a run, 48 kHz in and out of the graph.

Licence and attribution

OpenRAIL. These weights are trained on Dialogs, whose licence names "any models or other works trained on or derived from it together with their outputs" as Derivatives and requires that a Derivative be distributed with that licence, its Section 2 use restrictions included, and its recipients bound to them. LICENSE is that licence verbatim (langswap/dialogs-ru-emotional-conversations at a74ff44): whoever uses these weights, or audio they make, agrees to its Section 2.

The MIT IR Survey states no licence: its responses trained the model and are not redistributed, here or elsewhere. The CC BY 4.0 sets: VCTK (CSTR, University of Edinburgh), LibriTTS-R (Koizumi et al.), SIWIS (Honnet et al.), VocalSet (Wilkins et al.), MUSAN (Snyder, Chen, Povey), and Pocket TTS's weights (Kyutai). Dialogs by Ilya Shigabeev and Ilya Latyshev (Langswap); Supertonic 3 by Supertone.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Space using audiojs/restore 1