audiojs/dereverb-room

anvuew's room de-reverb, ONNX (float16 weights)

A voice in a room in, the voice dry out: anvuew's dereverb_room, a mono Band-Split RoFormer (Lu, Wang, Kong, Hung, "Music Source Separation with Band-Split RoPE Transformer", ICASSP 2024, arXiv:2309.02612) trained on voices in rooms simulated with pyroomacoustics: axial transformers over time and 63 contiguous bands of the complex spectrogram, dim 128, depth 16, 29.4 M parameters. The network between its STFT and iSTFT, as @audio/neural-separate runs it (model: 'dereverb-room') and audio's dereverb() uses it.

File	Size	SHA-256
dereverb-room.fp16.onnx	60.2 MB (60,241,089 bytes)	99797e086a71d1cb57d50f3b5be93dcb7c89c49d1537c60487a488a24aaee9ee

Float16 weights, float32 compute (onnxruntime casts them at load); the float32 export it is made from is 121.8 MB.

Source

• Checkpoint: dereverb_room_anvuew_sdr_13.7432.ckpt (SHA-256 2edec521f09e26341c1923dc82c8c52dbc86478b42b9999f679535743c970cb3) and its config dereverb_room_anvuew.yaml, anvuew/dereverb_room at 0b85f5b80b7f779b2dfe80f33a1b35b38af9376d (GPL-3.0).
• Model code: ZFTurbo/Music-Source-Separation-Training (MIT) at 84b1eac0887756b4f1a9d7a1ff49105939749ed2, models/bs_roformer/bs_roformer.py (from Phil Wang's BS-RoFormer, MIT).
• Export: scripts/export-roformer.py --model dereverb-room --verify --fp16 in @audio/neural-separate.

Licence and attribution

GPL-3.0 (LICENSE), as the checkpoint's repository states: this converted copy is under the same terms. The weights by anvuew; the architecture by Lu, Wang, Kong and Hung (ByteDance); the code it was written in by Phil Wang and Roman Solovyev (ZFTurbo); ONNX export and float16 weights by audiojs.

Graph

One 8.7 s segment (384,000 samples at 44.1 kHz, 751 frames) of one channel per run:

	name	shape
input	mix_spec	[1, 2, 1025, 751]	STFT: n 2048, hop 512, Hann of 2048, unnormalized, centered with reflect padding; re, im
output	stems_spec	[1, 2, 1025, 751]	the voice dry: re, im

Segments every 4.35 s, faded over 0.87 s, the input reflected out at the ends, as Music-Source-Separation-Training's demix() runs it (num_overlap: 2); each channel apart; nothing normalized.

import separate from '@audio/neural-separate'
let { stems, residual } = await separate([voice], { sampleRate: 48000, model: 'dereverb-room' })   // stems.dry; residual: the room

The graph computes what BSRoformer.forward computes between its STFT and iSTFT, rearranged for the GPU (bands of near widths batched, norms folded into the weights they feed, the rotary embedding as products with cos and sin tables, no tensor past 100 MB). --verify: the float32 graph against the forward on noise and tones, max |diff| 2e-6 of max |y|; this file 3.4e-3; the package's pipeline over 20 s against the forward on its segments, 74 dB SNR (the float32 export 123 dB).

Quality

Measured through audio's dereverb() (bench/rx/dereverb.mjs, test split): 42 VoiceBank test utterances and four 41–60 s takes in 19 MIT IR Survey rooms (Traer & McDermott, PNAS 2016; T60 0.38–1.85 s), each response split at direct + 50 ms; PESQ and STOI against the dry take, DNSMOS P.835 OVRL; tail: the output's power where the late part stands 10 dB over the early one.

	PESQ	STOI	OVRL	tail, dB
the takes	1.39	0.774	2.19
this model	2.65	0.940	2.98	−15.6
iZotope RX 12 De-reverb, tuned	1.45	0.779	2.32	−2.3
iZotope RX 12 Dialogue Isolate, reverb off	1.79	0.837	2.70	−8.3

The long takes 1.37 → 2.58 (tail −17.5 dB); with DEMAND noise 10 dB under the voice 1.26 → 1.70 (RX Dialogue Isolate 1.33); dry voices 4.64 → 4.55, their 125 Hz octave 2.1 dB down. It takes the early reflections too, and all that is not a voice: music loses 6 dB in its worst octave as it is, 9 dB of itself in a room (audio passes music by a guard).

@inproceedings{lu2024bsroformer,
  title     = {Music Source Separation with Band-Split RoPE Transformer},
  author    = {Lu, Wei-Tsung and Wang, Ju-Chiang and Kong, Qiuqiang and Hung, Yun-Ning},
  booktitle = {ICASSP},
  year      = {2024},
  eprint    = {2309.02612},
  archivePrefix = {arXiv}
}