MuCodec: Ultra Low-Bitrate Music Codec
Authors: name: Yaoxun Xu; name: Hangting Chen; name: Jianwei Yu; name: Wei Tan; name: Rongzhi Gu; name: Shun Lei; name: Zhiwei Lin; name: Zhiyong Wu
Published: 2024-09-20 05:06:49

Abstract: Music codecs are a vital aspect of audio codec research, and ultra
low-bitrate compression holds significant importance for music transmission and
generation. Due to the complexity of music backgrounds and the richness of
vocals, solely relying on modeling semantic or acoustic information cannot
effectively reconstruct music with both vocals and backgrounds. To address this
issue, we propose MuCodec, specifically targeting music compression and
reconstruction tasks at ultra low bitrates. MuCodec employs MuEncoder to
extract both acoustic and semantic features, discretizes them with RVQ, and
obtains Mel-VAE features via flow-matching. The music is then reconstructed
using a pre-trained MEL-VAE decoder and HiFi-GAN. MuCodec can reconstruct
high-fidelity music at ultra low (0.35kbps) or high bitrates (1.35kbps),
achieving the best results to date in both subjective and objective metrics.
Code and Demo: https://xuyaoxun.github.io/MuCodec_demo/.