WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio
  Language Modeling
Authors: name: Shengpeng Ji; name: Ziyue Jiang; name: Xize Cheng; name: Yifu Chen; name: Minghui Fang; name: Jialong Zuo; name: Qian Yang; name: Ruiqi Li; name: Ziang Zhang; name: Xiaoda Yang; name: Rongjie Huang; name: Yidi Jiang; name: Qian Chen; name: Siqi Zheng; name: Wen Wang; name: Zhou Zhao
Published: 2024-08-29 13:43:36

Abstract: Language models have been effectively applied to modeling natural signals,
such as images, video, speech, and audio. A crucial component of these models
is the codec tokenizer, which compresses high-dimensional natural signals into
lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer,
which offers several advantages over previous SOTA acoustic codec models in the
audio domain: 1)extreme compression. By compressing the layers of quantizers
and the temporal dimension of the discrete codec, one-second audio of 24kHz
sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved
subjective quality. Despite the reduced number of tokens, WavTokenizer achieves
state-of-the-art reconstruction quality with outstanding UTMOS scores and
inherently contains richer semantic information. Specifically, we achieve these
results by designing a broader VQ space, extended contextual windows, and
improved attention networks, as well as introducing a powerful multi-scale
discriminator and an inverse Fourier transform structure. We conducted
extensive reconstruction experiments in the domains of speech, audio, and
music. WavTokenizer exhibited strong performance across various objective and
subjective metrics compared to state-of-the-art models. We also tested semantic
information, VQ utilization, and adaptability to generative models.
Comprehensive ablation studies confirm the necessity of each module in
WavTokenizer. The related code, demos, and pre-trained models are available at
https://github.com/jishengpeng/WavTokenizer.