restore, ONNX (float16 weights)
A voice restored to a studio take: speech or singing, English, French or Russian, recorded in a room, in noise, on a
phone or through a codec, back as 48 kHz full band. One network, one pass, in the STFT. Trained by audiojs for
@audio/neural-restore, which computes the STFT,
the chunks and the level in JavaScript and runs this file through onnxruntime (Node, or a browser's onnxruntime-web on
WebGPU or wasm); audio's restore() runs it.
| File | Size | SHA-256 |
|---|---|---|
restore.fp16.onnx |
35.9 MB (35,929,213 bytes) | 46eb6ebb05cc50bb6c3b1af87e3c4542bf30cedbcc356e2c68646d3bbf9a65f1 |
Float16 weights, float32 compute: every weight tensor of 1024 values or more stored as float16 and Cast to float32 in
the graph (scripts/export.py). The float32 export it is made from is 71,671,387 bytes.
Model
Vocos's ConvNeXt stack over STFT frames (dim 512, 8 blocks) with two self-attention layers for a room's tail, reading each frame's log magnitude and phase and writing a change of log magnitude (unbounded: a band can be taken down 60 dB or raised from nothing) and a rotation of the input's phase (the identity at init: what is there keeps its phase unless the net turns it); 17.9M parameters, the STFT outside the graph.
| name | shape | ||
|---|---|---|---|
| input | spec |
[1, 2, 1025, T] float32 | STFT of one channel at 48 kHz: n_fft 2048, hop 512, periodic Hann, centered with reflect padding, unnormalized; re, im |
| output | spec_out |
[1, 2, 1025, T] float32 | the restored STFT, the same layout |
T is dynamic; opset 17. The network hears its input at β20 dBFS RMS: @audio/neural-restore brings the whole input to
that level, runs it in 10 s runs overlapping by 1 s, raised-cosine crossfaded, and undoes the gain, as
scripts/infer.py does; other rates are resampled to 48 kHz and back.
import restore from '@audio/neural-restore'
let y = await restore(samples, { sampleRate: 44100 }) // mono Float32Array; the same rate and length back
Training
Trained from scratch: 300,000 steps on an Apple M4 Max GPU (PyTorch 2.14 on MPS, batch 8 Γ 4 s), 2026-10-15 to
2026-10-22. These are step 290,000's weights: best SI-SDR and mel distance on the validation draw; 270kβ300k are within
each other's noise on every measure. A regressor first (multi-resolution STFT, log-mel, a compressed complex L1 on the
model's grid for the phase, WavLM Base+ features: FINALLY's recipe), then adversarially against HiFi-GAN's
multi-period and UnivNet's multi-resolution discriminators with feature matching. Every loss sees target and estimate
through a low-pass at the target's own measured bandwidth, so 24 kHz corpora train beside full-band ones. The inputs
are made on the fly from clean voices: a room (a measured response or a synthetic one), noise, a cheap microphone, a
compressor, clipping, a band limit, MP3 / AAC / Opus, 8β16-bit words. The code is scripts/ in @audio/neural-restore.
| Set | Hours | Rate | Licence | Role |
|---|---|---|---|---|
| VCTK 0.92, mic1, speakers p232 and p257 held out | 44 | 48 kHz | CC BY 4.0 | English, full band |
| LibriTTS-R train-clean-100 | 54 | 24 kHz | CC BY 4.0 | English, to 12 kHz |
| SIWIS | 10 | 44.1 kHz | CC BY 4.0 | French, one professional voice |
| Dialogs (langswap) | 17.4 | 44.1 kHz | OpenRAIL (commercial use allowed; its use restrictions flow down to derived models) | Russian, three actors, studio; its manifest is separate so a licence-clean variant is one flag away |
| Supertonic 3 (Supertone, OpenRAIL-M weights), ten voices, public-domain text | ru 12, fr 8, en 4 | 44.1 kHz | the synthesised audio is ours | the only full-band Russian beyond Dialogs; DNSMOS OVRL 3.43 / 3.40 / 3.39 (ru / fr / en) against 3.39 for VCTK's clean test set and 3.20 for Dialogs |
| Pocket TTS (Kyutai, CC BY 4.0), eight voices | fr 6, en 3 | 24 kHz | ours | more voices, to 12 kHz |
| VocalSet (the local subset) | 1.8 | 44.1 kHz | CC BY 4.0 | singing, full band |
| MIT IR Survey, the even-numbered rooms | 135 rooms | 32 kHz | unstated: training only, never redistributed | rooms; the odd-numbered ones are audio's bench rooms and stay out |
| MUSAN noise | 6 | 16 kHz | CC BY 4.0 | noise |
Texts read by the synthetic voices: Project Gutenberg (Austen, Dickens, Melville, Doyle; Hugo, Flaubert, Dumas, Proust; four Russian titles) and Dialogs' own transcripts. WavLM Base+ (Microsoft, MIT) is a feature loss, not data. A measured room or noise recorded at 16 or 32 kHz is continued to 24 kHz with synthetic content of the same decay or colour, so a measured room never low-passes the voice by its own rate. Validation pairs come from VoiceBank's clean test set (p232, p257), never trained on.
Measured
24 validation pairs: VoiceBank's test speakers through the chain (rooms, noise, microphones, codecs, band limits),
each output at the reference's level, scored through the pair's band limit. The baselines are audio's own ops on the
same inputs (DeepFilterNet3 through deepfilter(); auto('speech')).
| System | LSD β | SI-SDR β | PESQ β | DNSMOS SIG | BAK | OVRL | band, Hz |
|---|---|---|---|---|---|---|---|
| input | 3.30 | β5.2 | 1.77 | 2.50 | 2.41 | 1.97 | 16 947 |
deepfilter() |
2.85 | β2.1 | 2.13 | 2.93 | 3.75 | 2.59 | 17 104 |
deepfilter(0) |
2.71 | β2.1 | 2.07 | 2.91 | 3.90 | 2.63 | 16 366 |
auto('speech') |
2.91 | β4.0 | 1.88 | 2.91 | 3.29 | 2.45 | 16 265 |
| these weights (step 290k) | 0.79 | +6.0 | 2.40 | 3.21 | 3.91 | 2.89 | 24 000 |
| reference | 0 | 4.64 | 3.29 | 3.89 | 2.95 | 24 000 |
SI-SDR against a dry reference is a hard measure for this chain (rooms and codec lags put the input itself at β5 dB); the listen decides.
This file against PyTorch: the float32 export is 108β121 dB from PyTorch on five real takes, one of them 16 s in
chunks; this file 68β74 dB, the same scores to 0.1 dB SI-SDR. @audio/neural-restore's JavaScript pipeline with this
file is 73 dB from scripts/infer.py on its fixture in Node, 70 dB on WebGPU and wasm in a browser; 10.5 s restored in
0.36 s on the M4 Max's WebGPU, 1.3 s on wasm.
Limits
- Measured on the 24 English pairs above only. French, Russian, singing, real recordings, lost or invented words (WER) and Adobe Podcast Enhance are not measured.
- Above an input's band limit the network writes content the input does not hold: plausible, not the original.
- Full-band Russian comes from three Dialogs actors and synthetic voices, French from one professional voice (SIWIS) and synthetic voices, singing from 1.8 h of VocalSet.
- One channel a run, 48 kHz in and out of the graph.
Licence and attribution
OpenRAIL. These weights are trained on Dialogs, whose licence names "any models or other works trained on or derived
from it together with their outputs" as Derivatives and requires that a Derivative be distributed with that licence,
its Section 2 use restrictions included, and its recipients bound to them. LICENSE is that licence verbatim
(langswap/dialogs-ru-emotional-conversations
at a74ff44): whoever uses these weights, or audio they make, agrees to its Section 2.
The MIT IR Survey states no licence: its responses trained the model and are not redistributed, here or elsewhere. The CC BY 4.0 sets: VCTK (CSTR, University of Edinburgh), LibriTTS-R (Koizumi et al.), SIWIS (Honnet et al.), VocalSet (Wilkins et al.), MUSAN (Snyder, Chen, Povey), and Pocket TTS's weights (Kyutai). Dialogs by Ilya Shigabeev and Ilya Latyshev (Langswap); Supertonic 3 by Supertone.