Hugging Face's logo

Scicom-intl
/
WideCodec

Audio-to-Audio
PyTorch
neucodec
neural-audio-codec
widecodec
44100hz
decoder-finetune
speech
Model card Files
xet
 Community

WideCodec β€” 44.1 kHz decoder (scaled NeuCodec depth-20 finetune)

A SOTA 0.8 kbps, 44.1 kHz audio tokenizer.

WideCodec is a 44.1 kHz decoder finetune of neuphonic/neucodec. Only the decoder (Vocos backbone + ISTFT head, transformer depth 20) is trained to reconstruct 44.1 kHz audio. The 16 kHz acoustic/semantic encoders and the ResidualFSQ codebook are frozen and completely unchanged β€” the discrete codes are identical to base NeuCodec, so this model is a drop-in higher-fidelity decoder.

  • Sample rate: 44 100 Hz Β· hop: 882 Β· 50 tokens/sec
  • Trainable: decoder only (GAN: multi-period + spec discriminators; losses: multi-res mel + STFT + feature-matching + adversarial)
  • Frozen: w2v-BERT semantic encoder, acoustic CodecEnc, fc_prior/fc_post_a, FSQ codebook

Inference (self-contained)

Everything needed to run inference is in this repo β€” the neucodec/ package and infer_widecodec.py. No other source code required.

pip install torch transformers huggingface_hub local-attention einops librosa soundfile
huggingface-cli login          # if this repo is private (or export HF_TOKEN=hf_...)

huggingface-cli download Scicom-intl/WideCodec --local-dir WideCodec
cd WideCodec
python infer_widecodec.py --input my.wav --out-dir out      # one file
python infer_widecodec.py --input folder/  --out-dir out    # a directory

Or load it directly in Python:

import sys; sys.path.insert(0, "WideCodec")   # the downloaded repo dir
import torch, librosa
from neucodec import NeuCodec

model = NeuCodec._from_pretrained(model_id="Scicom-intl/WideCodec", decoder_depth=20).eval().cuda()
wav16, _ = librosa.load("my.wav", sr=16000, mono=True)   # encoder ingests 16 kHz mono
x = torch.from_numpy(wav16).float().view(1, 1, -1).cuda()
with torch.no_grad():
    codes  = model.encode_code(x)      # frozen FSQ codes β€” 0.8 kbps, 50 tok/s
    wav44  = model.decode_code(codes)  # 44.1 kHz reconstruction

Files

file what
pytorch_model.bin decoder weights for inference (load with NeuCodec._from_pretrained(..., decoder_depth=20))
last.ckpt full PyTorch-Lightning checkpoint with optimizer states + LR schedulers for resuming training

How WideCodec compares (44 kHz-class, by bitrate)

Offline benchmark on a heavily multilingual 9,291-clip wideband set (50 clips Γ— 188 clean β‰₯44.1 kHz datasets, 400+ language/dialect tags), faster-UTMOSv2 + spectral vs ground truth. Read this by bitrate, not raw MOS β€” these codecs span a 30Γ— bitrate range. DAC/NeMo/EnCodec are high-bitrate multi-codebook RVQ acoustic codecs; WideCodec is a single-codebook, 50-tokens/s code at **0.8 kbps** β€” the only one here whose token stream an autoregressive LLM/TTS model can practically predict.

bitrate vs UTMOSv2

codec native SR tokens/s codebooks ~bitrate UTMOSv2 mel-L1 ↓ HFβ‰₯11k rolloff
ground truth β€” β€” β€” β€” 2.822 β€” 0.00244 8263
nvidia nemo44k 44.1k ~86 RVQ (many) ~6–9 kbps 2.903 0.379 0.00196 8372
WideCodec (ours, gs-2.0M) 44.1k 50 1 ~0.8 kbps 2.788 0.571 0.00223 9011
dac 44.1k 86 9 ~8 kbps 2.672 0.341 0.00156 8185
snac44k 44.1k multi-scale 3–4 ~2.6 kbps 2.340 0.493 0.00158 8402
encodec48k 48k 150 RVQ 24 kbps 2.042 0.458 0.00169 8800

Bitrate = frame-rate Γ— Ξ£ logβ‚‚(codebook size). WideCodec's FSQ is levels=[4]Γ—8, num_quantizers=1 β†’ 8Β·logβ‚‚4 = 16 bits/frame Γ— 50 frames/s = 800 bps = 0.8 kbps (vs DAC's 9 codebooks Γ— 10 bits Γ— 86 fps β‰ˆ 8 kbps).

The WideCodec row is the final gs-2.0M checkpoint (9,291-clip set). At ~0.8 kbps / 1 codebook it beats DAC on naturalness (UTMOSv2 2.79 vs 2.67) using ~10x fewer bits, and lands within ~0.12 of NeMo's ~6-9 kbps RVQ. It carries genuine high-band content (rolloff 9.0 kHz; HF energy ~91% of ground truth), and genuine high-band content (reference-based LSD-HF 0.96, on par with the 8-kbps RVQ codecs), trading in-band fidelity for a ~10x bitrate advantage. (PESQ/SI-SDR are excluded β€” they structurally penalize a low-bitrate generative decoder regardless of naturalness.) Full report: evaluation/REPORT_44k_class_codecs.md.

Training stages / which revision to pull

This model is trained in two stages. main always points at the latest stage.

stage data how to load
Stage 1 β€” noisy mix (steps 0β†’1.58M, epoch=9-step=1580000) 8 base corpora + scale44k (~6,500 h, mostly noisy crowdsourced/podcast) pin revision d2c612f4319c1c1e0c65a9431a7d6db1b7955ce0
Stage 2 β€” clean finetune (from 1.58M, clean β‰₯44.1 kHz only; in progress) TTS-Clean44k + Clean-Podcast + clean-teacher pool + EARS/Expresso main (default) 
# the `neucodec/` package is bundled in this repo (see 'Inference (self-contained)')
from neucodec import NeuCodec

# Stage 1 (first-stage noisy training) β€” pin the exact revision:
model = NeuCodec._from_pretrained(
    model_id="Scicom-intl/WideCodec",
    decoder_depth=20,
    revision="d2c612f4319c1c1e0c65a9431a7d6db1b7955ce0",
)

Training data

Diverse, predominantly high-sample-rate human speech (no synthetic TTS). The scale-up group is SR-verified β‰₯ 44 kHz (probed before inclusion); the base corpora are long-form conversational, expressive and anechoic speech. Two groups:

Base corpora (Malaysia-AI + expressive/anechoic speech)

dataset content
malaysia-ai/malaysian-podcast-youtube Malay long-form conversational podcasts (~2,234 h)
malaysia-ai/singaporean-podcast-youtube Singaporean English podcasts (~1,255 h)
malaysia-ai/Multilingual-TTS CommonVoice-sidon multilingual short clips
malaysia-ai/malaysian-cartoons-youtube 48 kHz cartoon dialogue (subset)
malaysia-ai/malaysian-movie-youtube 48 kHz Malay movie dialogue
ylacombe/expresso 48 kHz expressive read speech
nytopop/expresso-conversational 48 kHz improvised dialogue
EARS 48 kHz anechoic fullband (emotional / conversational / reading)

Scale-up: 336 verified β‰₯ 44 kHz datasets

To maximise acoustic diversity for the decoder (language-agnostic β€” the codebook is frozen), training is scaled with 336 additional datasets (162 at 48 kHz, 174 at 44.1 kHz), mirrored as <name>*audio.zip in malaysia-ai/Multilingual-TTS and SR-verified β‰₯ 44 kHz. Original sources:

Full list of 336 β‰₯ 44 kHz source datasets

Usage

See the canonical Load from Hugging Face section in the source repo. decoder_depth=20 is required β€” the weights are a depth-20 decoder, so loading with any other depth mismatches the architecture.

import soundfile as sf

# the `neucodec/` package is bundled in this repo (see 'Inference (self-contained)')
from neucodec import NeuCodec

# decoder_depth=20 MUST match this repo; pass token=... (or hf login) for access
model = NeuCodec._from_pretrained(model_id="Scicom-intl/WideCodec", decoder_depth=20)
model = model.eval().cuda()

# encode (16 kHz path, frozen) -> codes -> decode (44.1 kHz, this finetune)
codes = model.encode_code("input.wav")                   # [1, 1, T], identical to base NeuCodec
recon_44k = model.decode_code(codes).squeeze().cpu().numpy()
sf.write("recon.wav", recon_44k, model.sample_rate)      # 44100

The discrete codes are identical to base neuphonic/neucodec β€” only the reconstruction sample rate and fidelity differ.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
Audio-to-Audio
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including Scicom-intl/WideCodec