BigVGAN-16kHz-80band

A 16 kHz, 80 band BigVGAN vocoder fine-tuned from nvidia/bigvgan_22khz_80band. It renders the mel spectrograms of EmoSphere++, whose authors did not release their vocoder. Resynthesis measurements and the paper's reference results are reported below.

Model summary

Category	Descriptions
Input	Natural log mel spectrogram, 80 bands, fmin 0, fmax 8000 Hz, FFT 1024, window 1024, hop 256
Output	16 kHz mono waveform, 256 samples per frame
Architecture	BigVGAN: upsampling 4, 4, 2, 2, 2, 2, anti-aliased SnakeBeta activations
Parameters	about 112M
Precision	FP32 checkpoint storage, BF16 inference

Usage

uv add voicestudio==5.18.0

The repository holds the configuration and weights of voicestudio's BigVGANModel; its remote code files only import the classes from voicestudio. The vocoder has no NVFP4 layers and does not require torchao. The example normalizes each waveform to a peak of 0.95 before mel extraction.

import torch
from transformers import AutoFeatureExtractor, AutoModel

device = "cuda"
vocoder = AutoModel.from_pretrained(
    "RetentionLabs/BigVGAN-16kHz-80band", trust_remote_code=True, dtype=torch.bfloat16,
).to(device).eval()
feature_extractor = AutoFeatureExtractor.from_pretrained(
    "RetentionLabs/BigVGAN-16kHz-80band", trust_remote_code=True, normalize_volume=True,
)

mel = feature_extractor(waveform_16khz, sampling_rate=16000, return_tensors="pt").input_features
mel = mel.to(device=device, dtype=torch.bfloat16)
with torch.no_grad():
    audio = vocoder(input_features=mel).audio_values.float().cpu()  # (batch_size, samples)

Training

• Start: generator and discriminators of nvidia/bigvgan_22khz_80band (commit b8d974a), with the mel front end changed to 16 kHz and fmax 8000 Hz. The upsampling factors are unchanged, so the hop is 256.
• Data: ESD English seen speaker train and evaluation recordings, LibriTTS train-clean-100 and VCTK 0.92 (mic1, without p315), resampled to 16 kHz. ESD test recordings and the unseen ESD speakers are held out.
• Recipe: BigVGAN's (MPD and MRD discriminators, mel loss weight 45, AdamW 1e-4 with betas 0.8 and 0.99, learning rate decay 0.999 per epoch, segments of 8192 samples), 40,000 updates, global batch 96 (48 per GPU), two GPUs with DistributedDataParallel.

Evaluation

Our 40k-step resynthesis results below were measured with FP32 inference. Resynthesis of ESD English test recordings (200 seen speaker and 200 unseen speaker recordings): the ground truth mel is vocoded and compared with the recording. Metrics as in the EmoSphere++ paper (WER avg of Whisper large-v3 and facebook/wav2vec2-large-960h-lv60-self, SECS avg of Resemblyzer and WavLM, ECA and EECS of emotion2vec+ base, SVAS of audEERING arousal and valence). Paper rows are the BigVGAN rows of Tables II and III of EmoSphere++, on the paper's 1,500 pairs with the authors' vocoder.

Seen speakers

Row	WER avg (%)	SECS avg	ECA (%)	SVAS	EECS
Paper, BigVGAN	13.26	0.8251	94.25	0.9815	0.9389
Ours, 40k steps	2.99	0.8419	92.50	0.9827	0.9326

Unseen speakers

Row	WER avg (%)	SECS avg	ECA (%)	SVAS	EECS
Paper, BigVGAN	15.09	0.8439	99.32	0.9844	0.9903
Ours, 40k steps	4.24	0.8439	100.00	0.9756	0.9916

Our WER setup reads far lower than the paper's even on ground truth recordings, so WER compares only within our rows.

Limitations

• Trained on read English speech at 16 kHz; other languages, singing and noisy input are untested.
• Expects the mel front end above (natural log, clipped at 1e-5); other mel definitions need retraining.

License

Research and non-commercial use only, because of the ESD training data. See LICENSE and NOTICE.

• BigVGAN code and the starting weights: MIT, Copyright (c) 2024 NVIDIA CORPORATION.
• LibriTTS (Zen et al., 2019) and VCTK 0.92 (Yamagishi et al., 2019): CC BY 4.0.
• ESD (Zhou et al., 2021): research use only.

Citation

@inproceedings{lee2023bigvgan,
  title={BigVGAN: A Universal Neural Vocoder with Large-Scale Training},
  author={Lee, Sang-gil and Ping, Wei and Ginsburg, Boris and Catanzaro, Bryan and Yoon, Sungroh},
  booktitle={International Conference on Learning Representations},
  year={2023}
}

Fine-tuning by b-re-w and TICTAP-11.