WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling

## Abstract

Abstract Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer , which offers several advantages over previous state-of-the-art (SOTA) acoustic codec models in the audio domain: 1) extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens.
2) improved subjective reconstruction quality. Despite the reduced number of tokens, WavTokenizer achieves SOTA reconstruction quality with outstanding UTMOS scores and also inherently contains richer semantic information .
Specifically, we achieve these results by designing a broader VQ space, extending contextual windows, improving attention networks, and introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conduct extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibits competitive to superior performance across various objective and subjective metrics compared to SOTA models. We also evaluate WavTokenizer on semantic representation, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The code is available at https://github.com/jishengpeng/WavTokenizer .

## 1 Introduction

Recently, significant achievements have been made by large language models (LLMs) (Brown et al., ) in audio generative tasks, including multiple-speaker speech syntheses (Wang et al., ; Kharitonov et al., ; Jiang et al., ; ; Ji et al., ), music generation (Agostinelli et al., ), and audio generation (Kreuk et al., ). Furthermore, the integration of the speech modality into unified large multimodal models also has garnered significant attention, such as SpeechGPT (Zhang et al., ), AnyGPT (Zhan et al., ), GPT-4o(^1^11[https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/)), and Moshi (Défossez et al., ). These successes can be largely attributed to the utilization of discrete acoustic codec representations
produced by neural codec models (Zeghidour et al., ; Défossez et al., ; Kumar et al., ). These discrete acoustic codec models bridge the gap between continuous speech signal and discrete-token-based language models, by discretizing high-rate audio signals into a finite set of tokens, hence enabling the application of LLM architectures to audio data.

Most end-to-end discrete codec models (Défossez et al., ; Wu et al., ) adopt a three-stage structure consisting of an encoder, a Residual Vector Quantization (RVQ) module, and a decoder. The encoder performs downsampling of the audio signal in the time domain to obtain compressed audio frames. Each compressed audio frame is then quantized by a series of quantizers, with each quantizer operating on the residual of the previous one. The number of quantizers determines the overall bitrate. The decoder, on the other hand, performs upsampling in the time domain to reconstruct the audio signal from the quantizer outputs. Existing acoustic codec models (Kumar et al., ) demonstrate impressive reconstruction quality, and generative models based on discrete codecs are now capable of synthesizing speech at near-human levels. However, two important directions are worth exploring beyond the current acoustic codec models, namely, high bitrate compression and semantic richness.

Higher Bitrate Compression. The compression level of current codec models still warrants exploration to achieve higher compression. Two aspects merit optimization: the number of quantizers and the temporal dimension of the codec. While some efforts have reduced the quantity of quantizers from eight to four (Yang et al., ; Ji et al., ), we argue that a single quantizer layer fundamentally differs from multiple quantizers. When the number of quantizers exceeds one, downstream models require additional design efforts, such as VALL-E’s (Wang et al., ) AR and NAR structures, SoundStorm’s (Borsos et al., ; Ji et al., ) parallel generation, MusicGen’s (Copet et al., ; Peng et al., ) slanted autoregressive structure, and UniAudio’s (Yang et al., ) global and local attention structures. Conversely, with a single quantizer, speech modalities can be directly autoregressively embedded into large multimodal models (Touvron et al., ). Additionally, high temporal dimensions of codecs, such as DAC’s (Kumar et al., ) requirement of 900 tokens per second, substantially degrade language model generation quality and increase resource consumption.

Richer Semantic Information. Considering the gap between the codec’s reconstruction paradigm and the generative paradigm of downstream models (SpeechTeam, ; Chu et al., ), incorporating more semantic information in codec can facilitate weakly supervised text-to-speech generation. Moreover, many large multimodal models (Chu et al., ; Tang et al., ) adopt the Whisper paradigm for understanding tasks (the target for generation is acoustic codec); hence, incorporating additional semantic information into acoustic codec models could help unify the understanding and generation processes in multimodal models. While many approaches attempt to introduce semantic information through distillation, additional pre-trained semantic modules (Zhang et al., ) can interfere with the unified modeling of music and audio. Moreover, this distillation-based strategy may limit the potential of codec models. Exploring more elegant approaches to directly integrate semantic information into the codec remains an open question.

In this paper, we introduce WavTokenizer, a discrete acoustic codec model capable of reconstructing 24kHz speech, music, and audio using only 40 or 75 tokens per second. WavTokenizer achieves high reconstruction quality with extreme compression while enhancing the semantic richness of the codec. Specifically, WavTokenizer enhances audio reconstruction quality by employing a multi-scale discriminator and an inverse Fourier transform upsampling structure from the vocoder in the decoder. To compress the codec from multiple quantizers to a single one, we discover that expanding the VQ space, alongside employing recent K-means clustering initialization and random awakening strategies, can significantly compress audio representations while maintaining high codebook utilization. Additionally, expanding the contextual window for speech modeling and incorporating attention networks in the decoder not only balance reconstruction quality and compression but also enrich the
semantic information. Our contributions can be summarized as follows:

- •
Conceptual Contributions. We introduce the concept of compressing the quantizer layers of acoustic codec models to a single quantizer for the first time and enhancing semantic information of the codec without disrupting the codec paradigm for modeling music and audio. Inspired by a detailed analysis of the codebook space in Section 3.2, we propose aligning the large speech space with the textual vocabulary and show the potential of large speech space as a latent form of a unique language.
- •
Methodological Contributions.
Utilizing K-means clustering initialization and random awakening strategies on the VQ codebook space, we design an expanded VQ space for compressing the codec model to a single quantizer. Furthermore, we design extended contextual modeling windows and add attention mechanisms in the decoder. The integration of an inverse Fourier transform module and multi-scale discriminator in the vocoder also contributes to improved reconstruction.
- •
Experimental Contributions.
WavTokenizer surpasses the current state-of-the-art (SOTA) models’ subjective reconstruction performance on speech, music, and audio, with only 75 tokens per second. It achieves comparable results with 40 or 75 tokens per second across broader metrics. Additional experiments demonstrate the superiority of WavTokenizer over competitive baseline models regarding semantic information, codebook utilization, and performance in generative models. Rigorous ablation studies confirm the necessity of each component in WavTokenizer. We will open-source the entire codebase and pretrained models for WavTokenizer.

## 2 Related Work

In recent times, neural acoustic codecs (Zeghidour et al., ; Défossez et al., ; Kumar et al., ) have demonstrated remarkable capabilities in reconstructing high-quality
audio at low bitrates. Typically, these methods employ
an encoder to extract deep features in a latent space, which
are subsequently quantized before being fed into the decoder. Given that acoustic tokens, compared to semantic tokens, can support audio, speech, and music domains, and their rich acoustic details can eliminate the need for cascading architectures in downstream generative models (Kharitonov et al., ; Huang et al., ) or large multimodal models (SpeechTeam, ; Anastassiou et al., ), current optimization directions for acoustic codec models can be categorized as follows:
Pursuing Better Reconstruction Quality. AudioDec (Wu et al., ) demonstrates the importance of discriminators. PromptCodec (Pan et al., ) enhances representation capabilities through additional input prompts. DAC (Kumar et al., ) significantly improves reconstruction quality through techniques like quantizer dropout and a multi-scale Short-Time Fourier Transform (STFT) based discriminator. Vocos (Siuzdak, ) eliminates codec noise artifacts using a pre-trained Encodec with an inverse Fourier transform vocoder. HILCodec (Ahn et al., ) introduces the MFBD discriminator to guide codec modeling. APCodec (Ahn et al., ) further enhances reconstruction quality by incorporating ConvNextV2 modules in the encoder and decoder.
Enhancing Compression. HiFi-Codec (Yang et al., ) proposes a parallel GRVQ structure and achieves good speech reconstruction quality with just four quantizers. Language-Codec (Ji et al., ) introduces the MCRVQ mechanism to evenly distribute information across the first quantizer and only requires four quantizers to achieve excellent performance across various generative models. Regarding achieving high compression with a single quantizer, Single-Codec (Li et al., ) is most related to our work. Throughout the progression of our work on WavTokenizer, Single-Codec designs additional BLSTM, hybrid sampling, and resampling modules to ensure basic performance with a single quantizer; however, different from the impressive reconstruction performance of our WavTokenizer, Single-Codec’s reconstruction quality is uncompetitve (with UTMOS only 3.0).
Deepening Understanding of the Codec Space. TiCodec (Ren et al., ) models the codec space by distinguishing between time-independent and time-dependent information. FACodec () decouples the codec space into content, style, and acoustic-detail modules.
Additionally, recognizing the importance of semantic information in generative models, recent efforts start integrating semantic information into codec models. RepCodec (Huang et al., ) learns a vector quantization codebook by reconstructing speech representations from speech encoders like HuBERT (Hsu et al., ) or Data2vec (Baevski et al., ). SpeechTokenizer (Zhang et al., ) enriches the semantic content of the first quantizer through semantic distillation. FunCodec (Du et al., ) makes semantic tokens optional and explores different combinations. SemanticCodec (Liu et al., ) uses quantized semantic tokens and reconstructs acoustic information using an audio encoder and diffusion model. Although the semantic codecs achieve good audio reconstruction quality, they disrupt the encoder-VQ-decoder paradigm of acoustic codec models and introduce additional training costs.

Compared to the aforementioned approaches, WavTokenizer achieves impressive reconstruction results with only one quantizer and through 40 or 75 tokens. In contrast, for one second of speech, DAC (Kumar et al., ) requires 900 tokens, with 9 quantizers. Furthermore, WavTokenizer explores enhancing semantic information by strengthening the capabilities of the Codec itself.

## 3 WavTokenizer

Our model is built on the framework of VQ-GANs, following the same paradigm as SoundStream (Zeghidour et al., ) and EnCodec (Défossez et al., ). Specifically, WavTokenizer passes the raw audio $X$ through three modules: 1) A full convolution encoder network that takes the input audio and generates a latent feature representation $Z$; 2) A single quantizer discretizes $Z$ to generate a discrete representation $Z_{q}$. 3) An improved decoder that reconstructs the audio signal $\tilde{X}$ from the compressed latent representation $Z_{q}$. The model is trained end-to-end, optimizing a reconstruction loss applied over both time and frequency domains, along with a perceptual loss in the form of discriminators operating at different resolutions.

Considering that WavTokenizer is designed as a discrete token representation for large audio language models, the focus should be on the subjective reconstruction quality of the codec (audio fidelity) and semantic content information. In Figure 1, we visualize the relationship between bitrates and UTMOS metrics (Saeki et al., ) across different codec models. As shown in the Figure 1, WavTokenizer achieves SOTA reconstruction quality with only 75 tokens. Notably, WavTokenizer facilitates extreme compression bitrates and achieves a fair UTMOS score of 3.6 at 0.48 kpbs.

### 3.1 Encoder

Following Encodec (Défossez et al., ), the encoder
consists of a 1D convolution with $C$ channels and a kernel
size of 7 followed by $B$ convolution blocks. Each convolution
block is composed of a single residual unit followed by a
downsampling layer consisting of a strided convolution,
with a kernel size twice of the stride $S$. The residual unit
contains two convolutions with kernel size of 3 and a skip-connection. The number of channels is doubled whenever
downsampling occurs. The convolution blocks are followed
by a two-layer LSTM for sequence modeling and a final
1D convolution layer with a kernel size of 7 and $D$ output
channels. Following Encodec, we set
$C=32$, $B=4$, $D=512$, and use ELU (Clevert et al., ) as a non-
linear activation function. For the stride
$S$, we employ two configurations, (2, 4, 5, 8) and (4, 5, 5, 6), to ensure that WavTokenizer can downsample 24 kHz speech by factors of 320 and 600 along the time dimension.

### 3.2 Rethinking the vector quantization space

WavTokenizer aims to compress speech representations into the codebook space of a single quantizer. This allows for the seamless serialization of speech and elimination of the need of hierarchical design in downstream models across channel dimensions (Wang et al., ; Yang et al., ; Borsos et al., ). Initially, we attempt to rely solely on a single quantizer for reconstruction during training, without changing any structure; however, we find the results suboptimal. Considering the vast vocabulary space in natural language, we hypothesize that treating speech as a unique language might yield better results. Motivated by this hypothesis, we conduct the following analysis. First, we expand the codebook space from $2^{10}$ to $2^{14}$. Next, we train on 585 hours of LibriTTS and then visualize the probability distribution of codebooks on the LibriTTS test-clean dataset, as shown in Figure 2 (a). We observe a concentration of the speech vocabulary to the left of $2^{12}$, indicating the potential merit of utilizing the larger $2^{12}$
speech vocabulary space since the current codec codebooks $2^{10}$ may not fully represent the speech space. More analyses in Appendix D verify that increasing the training dataset size does not lead to higher codebook space utilization and the codebook space trained with 4000 hours multilingual data remains concentrated to the left of $2^{12}$.

Figure: Figure 2: The visualization analysis of WavTokenizer’s quantized codebook space. Figure (a) illustrates the probability distribution of each codebook index (1-16384) on the LibriTTS test-clean across different codebook spaces. Figure (b) examines the relationship between reconstruction quality in terms of UTMOS and codebook utilization rate across different codebook spaces.
Refer to caption: x2.png

Simply expanding the quantized codebook space could lead to lower utilization rates. To mitigate this issue, leveraging recent advancements in codec models (Défossez et al., ; ), we use k-means clustering to initialize the codebook vectors. We adjust the number of cluster centers to 200 to align with the larger codebook space. During training, for each input, the code is selected, assigned, and updated using an exponential moving average with a decay of 0.99, and codes unassigned for several batches are replaced with input vectors randomly sampled from the current batch. This forced activation strategy (Dhariwal et al., ) helps ensure effective utilization of the large codebook space. We analyze the relationship between codebook utilization rate and reconstruction result, after applying the aforementioned k-means clustering initialization and random awakening strategies. Figure 2 (b) confirms that expanding the corresponding codebook space appropriately can reduce information loss caused by compressing the hierarchical RVQ structure into a single quantizer. Speech can be effectively reconstructed under a serialized quantizer structure, with a codebook space of $2^{12}$ achieving a favorable balance between codebook utilization and reconstruction quality. Furthermore, the experiments in Section 4.3 explore the potential of a large codebook space in discrete acoustic codecs as a specialized form of language representations.

### 3.3 Improved decoder

As in FACodec (), we believe that the decoder plays a more crucial role than the encoder during the acoustic codec reconstruction process. Upsampling and reconstructing audio from the highly compressed information in WavTokenizer is particularly challenging. Notably, WavTokenizer does not employ a mirrored decoder upsampling structure (Kumar et al., ), a standard practice that uses a stack of dilated convolutions to increase the receptive field, and uses transposed convolutions to sequentially upsample the feature sequence to the
waveform. Since this standard design is known to be susceptible to aliasing artifacts, following Vocos (Siuzdak, ), we maintain consistent feature resolution at all depths and achieve waveform upsampling through inverse Fourier transform. In the decoder, the target audio signal $\tilde{X}$ is represented using STFT:

$$ $STFT(\tilde{X}_{\left[m,k\right]})=\sum_{n=0}^{N}\tilde{X}\left[n\right]w\left [n-m\right]e^{-j2\pi kn/K}$ (1) $$

where $K$ denotes the number of frequency points after performing Discrete Fourier Transform (DFT), $k$ denotes the frequency index, $N$ denotes the number of points in the sampled sequence, with $n$ denoting a particular sample point, and $m$ denoting the index length. In the practical implementation, STFT is performed by applying a series of Fast Fourier Transforms (FFTs) to overlapping and windowed frames of data. The window function advances or hops through time to create these frames.

Moreover, to directly enhance the semantic modeling capability of the acoustic codec model, rather than adding various semantic tokens (Zhang et al., ), we hypothesize that incorporating an attention network module (Rombach et al., ) into the decoder may enhance information reconstruction and semantic modeling. Although attention models have proven their scalability and high performance in broader tasks (Dosovitskiy et al., ; Fang et al., ; Huang et al., ; Yang et al., ), the encoder-decoder structure in the acoustic codec model remains fully convolutional. Concerns may arise about the potential extrapolation issues of attention models when modeling long sequences during inference, given that acoustic codec models often train on randomly selected short one-second audio clips. However, our experiments show that WavTokenizer achieves good reconstruction even for long audio sequences during inference. Additionally, we
find that simply expanding contextual modeling windows to three seconds for WavTokenizer with attention modules could further improve codec reconstruction during training and in turn the reconstruction quality during inference. This is probably because one-second clips, including silence, may contain insufficient semantic information; hence, increasing the contextual modeling window size helps the Codec model better capture context. We validate these findings through detailed ablation studies (Section 4.3). We investigate various configurations of introducing attention modules to the encoder, the decoder, or both, and find that adding the attention module to the decoder only is beneficial, and adding it before the ConvNext module (Liu et al., ) appears to be optimal.

Therefore, for the representation of the intermediate signals $Z_{q}$ after quantization, WavTokenizer only needs to input $Z_{q}$ into the conv1D layer, attention block, ConvNeXt blocks, which serves as the fundamental backbone. Subsequently, a Fourier transform is performed on the real-valued signals. Following Vocos (Siuzdak, ), the ConvNeXt Block first embeds the input features into a hidden dimensionality and then applies a sequence of convolutional blocks. Each block is composed of a large-kernel-sized depthwise convolution, followed by an inverted bottleneck that projects features into a higher dimensionality using pointwise convolution. GELU activations (Hendrycks & Gimpel, ) are used within the bottleneck, and Layer Normalization is employed between the blocks. Regarding the transformation of real-valued signals, we utilize a single side band spectrum, resulting in $n_{fft}/2+1$ coefficients per frame. Since we parameterize the model to output both phase and magnitude values, the activations of the hidden dimensions are projected into a tensor $h$ with $n_{fft}+2$ channels. Finally, the inverse Fourier transform $\mathcal{F}^{-1}$ is used to directly reconstruct the final audio.

### 3.4 The advanced discriminator and the loss functions

We use the adversarial loss to promote perceptual quality. Following Vocos (Siuzdak, ), We employ the open-source multi-period discriminator (MPD) (Kong et al., ), the single band amplitude only and multi band complex multi-resolution discriminator (MRD) (Jang et al., ). Furthermore, to learn discriminative features about a specific sub-band and provide a stronger gradient signal to the generator, following (Kumar et al., ), we use a complex STFT discriminator (Zeghidour et al., ) at multiple time-scales (Défossez et al., ). We adopt a hinge loss formulation instead of the least squares GAN objective, as suggested by (Zeghidour et al., ). The discriminator training loss $\mathcal{L}_{dis}(X,\tilde{X})$ is as follows:

$$ $\frac{1}{K}\sum_{k=1}^{K}max(0,1-D_{k}(X))+max(0,1+D_{k}(\tilde{X}))$ (2) $$

$K$ denotes the number of discriminators, $D_{k}$ denotes the $k$-th discriminator. The training loss for the generator of WavTokenizer consists of four components: quantizer loss, mel-spectrum reconstruction loss, adversarial loss, and feature matching loss. The quantizer loss is defined as follows:

$$ $\mathcal{L}_{q}(Z,Z_{q})=\sum_{i=1}^{N}\left\|Z_{i}-\hat{Z}_{i}\right\|_{2}^{2}$ (3) $$

The mel-spectrum reconstruction loss is defined as follows:

$$ $\mathcal{L}_{mel}(X,\tilde{X})=\left\|Mel(X)-Mel(\tilde{X})\right\|_{1}$ (4) $$

Furthermore, we define the adversarial loss as a hinge loss over the logits of these discriminators:

$$ $\mathcal{L}_{adv}=\frac{1}{K}\sum_{k=1}^{K}max(0,1-D_{k}(\tilde{X}))$ (5) $$

The feature matching loss, denoted as $\mathcal{L}_{feat}$ , is calculated as the mean of the distances between the $l$th feature maps of the $k$th subdistriminator:

$$ $\mathcal{L}_{feat}=\frac{1}{K*L}\sum_{k}\sum_{l}\left\|D_{k}^{l}(X)-D_{k}^{l}( \tilde{X})\right\|_{1}$ (6) $$

The total training loss of the generator, $\mathcal{L}_{gen}$, is computed as:

$$ $\mathcal{L}_{gen}=\lambda_{q}\mathcal{L}_{q}+\lambda_{mel}\mathcal{L}_{mel}+ \lambda_{adv}\mathcal{L}_{adv}+\lambda_{feat}\mathcal{L}_{feat}$ (7) $$

where $\lambda_{q}$, $\lambda_{mel}$, $\lambda_{adv}$, $\lambda_{feat}$ are hyper-parameters.

## 4 Experiments

### 4.1 Experimental Setup

Dataset.
Due to constrained computation resources, we conduct the training process only on a subset of common publicly available datasets. WavTokenizer is trained on approximately 8K hours of data. For the speech domain, we use LibriTTS (Zen et al., ), VCTK (Veaux et al., ), and a subset of CommonVoice (Ardila et al., )(3000 hours are randomly selected). For the audio domain, we utilize a subset of AudioSet (Gemmeke et al., )(2000 hours are randomly selected); and for the music domain, we employ the Jamendo (Bogdanov et al., ) and MusicDB (Rafii et al., ) datasets. We evaluate speech reconstruction performance of codec in clean and noisy environments using the LibriTTS test-clean and test-other sets respectively, and assess audio and music reconstruction performance using the AudioSet eval and MusicDB test sets respectively. On the other hand, for most confirmatory experiments, such as the ablation experiments, we evaluate the results with the WavTokenizer trained only on LibriTTS.
Baselines.
We select the state-of-the-art (SOTA) codec models as the baselines for WavTokenizer. To ensure fair comparisons, we employ the official weights provided by Encodec (Défossez et al., )(^2^22[https://github.com/facebookresearch/encodec](https://github.com/facebookresearch/encodec)), HiFi-Codec (Yang et al., )(^3^33[https://github.com/yangdongchao/AcademiCodec](https://github.com/yangdongchao/AcademiCodec)), Vocos (Siuzdak, )(^4^44[https://github.com/gemelo-ai/vocos](https://github.com/gemelo-ai/vocos)), SpeechTokenizer (Zhang et al., )(^5^55[https://github.com/ZhangXInFD/SpeechTokenizer](https://github.com/ZhangXInFD/SpeechTokenizer)), and DAC (Kumar et al., )(^6^66[https://github.com/descriptinc/descript-audio-codec](https://github.com/descriptinc/descript-audio-codec)) frameworks.
Evaluation metrics and Implementation Details
For objective evaluation of discrete codec models,
following Vocos (Siuzdak, ), we employ the UTMOS (Saeki et al., ) automatic Mean Opinion Score (MOS) prediction system. UTMOS can yield scores highly correlated with human evaluations, closer to human perception than PESQ (Perceptual Evaluation of Speech Quality) (Rix et al., ), but it is restricted to 16 kHz sample rate. we also adopt the metrics in speech enhancement fields, such as PESQ, STOI (Short-time Objective Intelligibility), and the F1 score for voiced/unvoiced classification (V/UV F1). In addition to these objective metrics, following Encodec (Défossez et al., ), we also employ the subjective MUSHRA evaluation to assess the reconstruction performance of the codec. We employ the common subjective CMOS evaluation metrics to assess the performance of the downstream TTS model with the codec models. Details of the subjective evaluations are in Appendix C. Training and Inference Settings are detailed in Appendix A.

Table: Table 1: Objective reconstruction results of different codec models on LibriTTS test-clean (clean environment), LibriTTS test-other (noisy environment), and LJSpeech dataset (out-of-domain environment). Nq denotes the number of quantizers. GT denotes ground truth waveforms. Best results from models with a single quantizer (hence directly comparable to WavTokenizer) are in bold.

### 4.2 Main Results

Evaluation on Reconstruction. We compare the speech reconstruction performance of WavTokenizer with a broad selection of SOTA and competitive codec models as baselines on LibriTTS test-clean (4837 samples), LibriTTS test-other (5120 samples), and LJSpeech (13100 samples), which correspond to audio reconstruction in clean, noisy, and out-of-domain environments, respectively. Notably, RVQ-based codec models often select quantizers with varying bandwidths during training. To ensure fair comparisons, we use the quantizers that the baseline models are trained on. The results are shown in Table 1. We observe the following: 1) WavTokenizer achieves impressive results on the UTMOS metric, with WavTokenizer at 0.9 kbps surpassing the current SOTA DAC model at 9 kbps on all test sets. Since the UTMOS metric closely aligns with human perception of audio quality (Saeki et al., ), these results validate that WavTokenizer maintains excellent reconstruction quality under extreme compression. 2) When compared to the SOTA DAC model with a single quantizer (hence directly comparable to WavTokenizer, shown in the shaded regions of Table 1), WavTokenizer with 40 and 75 tokens remarkably outperforms DAC with 100 tokens across all metrics. To the best of our knowledge, WavTokenizer is the first codec model capable of effectively reconstructing audio with a single quantizer. 3) On objective metrics STOI, PESQ, and F1 score, WavTokenizer also performs comparably to the Vocos model with four quantizers and the SpeechTokenizer model with eight quantizers. 4) In noisy environments and out-of-domain scenarios, WavTokenizer demonstrates strong robustness and generalizability across all metrics.

We further evaluate the reconstruction performance of the codec models on the broader range of music and audio domains. Following Encodec, we used MUSHRA as the metric for subjective evaluation. As shown in Table 2, WavTokenizer at 0.9 kbps outperforms the SOTA DAC model at 9 kbps in reconstruction quality on the speech, music, and audio domains. These results further demonstrate that WavTokenizer is capable of maintaining high subjective reconstruction performance on speech, music, and audio, with an extremely limited number of tokens.

Evaluation on Semantic Representation.
We evaluate the semantic richness of different codec models on the ARCH benchmark (La Quatra et al., ). Notably, we opt not to utilize the conventional Superb benchmark (Yang et al., ) due to its exclusive focus on the speech domain, while ARCH enables further assessment of a Codec model in music and audio realms. The ARCH benchmark comprises 12 datasets in speech, music, audio domains (details in Appendix B). We extract embeddings corresponding to the discrete codebooks of an acoustic codec model as its respective representations and evaluate the classification accuracy of the codec model on ARCH datasets using its representations. For fair comparisons, we evaluate Encodec and DAC models on semantic representation, as they are under the same paradigm as WavTokenizer.
The experimental results, as shown in Table 3, demonstrate that WavTokenizer substantially outperforms DAC and Encodec configured with a single quantizer or two quantizers on classification accuracy. Remarkably, on the AM and SLURP datasets in the speech domain, the MTT and IRMAS datasets in the music domain, and the FSD50K and VIVAE datasets in the audio domain, WavTokenizer surpasses DAC with nine quantizers and Encodec with eight quantizers on classification performance. These findings validate that WavTokenizer effectively captures rich semantic information.

Table: Table 2: The subjective reconstruction results using MUSHRA (comparative scoring of samples) of codec models on speech, music and audio domains. Nq denotes the number of quantizers.

Table: Table 3: The semantic representation (speech, music, audio) evaluation of different codec models on ARCH Benchmark in terms of classification accuracy. Nq represents the number of quantizers.

Evaluation on Downstream Generative Tasks.
We evaluate WavTokenizer’s performance on downstream generative tasks exemplified by text-to-speech synthesis (TTS) (Ren et al., ). We adopt an autoregressive language model backbone. Specifically, we follow the MusicGen paradigm (Copet et al., ), which expands the acoustic codec sequence through autoregressive prediction, and modify the open-sourced ParlerTTS model (Lyth & King, ) accordingly. We use the ParlerTTS 600M model configuration and its default hyperparameters, and train TTS models based on DAC and WavTokenizer representations on the LibriTTS dataset. Each model is trained for 40 epochs on 8 A800 80GB GPUs.

The results are shown in Table 4. In terms of audio quality (CMOS-Q) and audio prosody (CMOS-P), the speech synthesis model trained on WavTokenizer’s single-layer quantizer representations outperforms that trained on DAC’s 9-layer quantizer representations. This demonstrates that: 1) Acoustic models with a single-layer quantizer show significant potential in downstream autoregressive generative models, and 2) speech synthesis models using large-codebook acoustic representations can also synthesize high-quality audio, suggesting that large-codebook acoustic spaces have the potential to model speech as a special form of text. We will validate these findings on large multimodal models (Zhan et al., ) based on acoustic codec models that are trained on larger datasets in the future work.

Table: Table 4: The subjective evaluations of various acoustic codec models for downstream speech synthesis models. GT denotes ground truth waveforms.

Table: Table 5: Impact of codebook scale. Utilization rate reflects codebook’s usage efficiency.

### 4.3 Ablation Study

Due to limited compute resources, we use 585 hours LibriTTS training data for training WavTokenizer and conduct ablation studies on reconstruction performance on the LibriTTS test-clean subset. For all ablation experiments, we use WavTokenizer with a single quantizer operating at 0.9 kbps.
Codebook size. we evaluate varying codebook sizes on the performance of WavTokenizer. We record the frequency of each codebook entry on LibriTTS test-clean. As shown in Table 6 and discussed in Section 3.2, we observe significant potential for expansion in the typical codebook space, even under extreme compression with a single quantizer, when combined with advanced training strategies (Section 3.2). We find that expanding the codebook size from the typical 1024 to 4096 significantly enhances audio quality, with the UTMOS gain by 0.55, PESQ gain by 0.6, and STOI gain by 0.5. However, excessively large codebook spaces (16384) can lead to reduced codeboook utilization.
Contextual window size. Most Codec models are trained on randomly selected one-second audio clips. With an attention module in WavTokenizer’s decoder, as shown in Table 6, using a three-second contextual window further enhances the reconstruction quality. We hypothesize that a one-second window may contain insufficient information and be more affected by silence. Longer contextual window may enhance attention module’s ability to capture relevant semantics.
Multi-scale STFT discriminator. As shown in Table 7, the multi-scale STFT discriminator (MSTFTD) enhances the reconstruction quality, albeit increasing the training time. The improvement is probably due to the fact that MSTFTD splits STFT into sub-bands and learns discriminative features for a specific sub-band, hence providing a stronger gradient signal to WavTokenizer’s generator, in turn improving high-frequency prediction and mitigating aliasing artifacts.
Attention module. We remove the attention module from WavTokenizer’s decoder, resulting in WavTokenizer w/o attention. As shown in Table 7, removing the attention module degrades WavTokenizer’s performance more than MSTFTD. Ablation results on impact of the attention module on semantic representation are in Appendix E.
Improved decoder. We replace the improved decoder based on the inverse Fourier transform with an up-sampling structure mirroring the encoder, denoted as WavTokenizer w/ mirror decoder. As shown in Table 7, a purely mirrored structure substantially compromises the reconstruction performance of WavTokenizer under extreme compression. This underscores the importance of a robust decoder in ensuring the reconstruction performance of codec models under high compression.

**Table 7: Ablation on the multi-scale STFT discriminator (MSTFTD), the atention module, and switching from our improved decoder to a mirror decoder, in WavTokenizer.**
| Model | UTMOS $\uparrow$ | PESQ $\uparrow$ | STOI $\uparrow$ | V/UV F1 $\uparrow$ |
| --- | --- | --- | --- | --- |
| WavTokenizer | 4.0486 | 2.3730 | 0.9139 | 0.9382 |
| w/ mirror decoder | 2.7782 | 1.5007 | 0.8249 | 0.8820 |
| w/o attention module | 3.6020 | 1.9332 | 0.8734 | 0.9067 |
| w/o MSTFTD | 3.7806 | 2.1270 | 0.9008 | 0.9269 |

## 5 Conclusion

In this paper, we introduce WavTokenizer, a codec model capable of quantizing one second speech, music and general audio into 75 or 40 tokens with a single quantizer. Compared to SOTA acoustic codec models, WavTokenizer maintains high subjective reconstruction quality and preserves rich semantic information even under extreme compression. The limitation of WavTokenizer and future work are discussed in Appendix F.

## Appendix A Training and Inference Settings

We train WavTokenizer up to 2 million iterations, with 1 million iterations allocated to training the generator and the discriminator respectively, on 8 NVIDIA A800 80G GPUs. Throughout the entire training process, all input speech, music, and audio samples are resampled to 24 kHz, and the batch size is 40. We uniformly truncate excessively long segments in the training data to a fixed length of 10 seconds and subsequently perform a random crop of the waveform to obtain audio snippets of 3-second duration to be fed into WavTokenizer. WavTokenizer is optimized using the AdamW optimizer with an initial learning rate of 2e-4 and betas set to (0.9, 0.999). The learning rate is decayed based on a cosine schedule.

## Appendix B The ARCH benchmark

The ARCH benchmark comprises twelve datasets within the speech, music, audio domain. Emotional Speech and Song (RAVDESS) (Livingstone & Russo, ), Audio-MNIST (AM) (Becker et al., ), Spoken Language Understanding Resource Package (SLURP) (Bastianelli et al., ), and EMOVO dataset (Costantini et al., ) assess performance in the Speech domain.
ESC-50 (Piczak, ), US8K (Salamon et al., ), FSD50K (Fonseca et al., ),
and VIVAE (Holz et al., ) assess performance on Acoustic Events. FMA (Defferrard et al., ), MTT (Law et al., ), IRMAS (Bosch et al., ), and MS-DB (Rafii et al., ) assess performance in the Music domain.

## Appendix C Subjective Evaluations

For the subjective evaluations, we follow the MUSHRA protocol (Series, ), using both a hidden reference and a low anchor. Annotators are recruited using a crowd-sourcing platform, in which they are asked to rate the perceptual quality of the provided samples in a range between 1 to 100. We randomly select 50 samples from each category of the test set and ensure at least 10 annotations per sample. To filter out noisy annotations and outliers, we remove annotators who rate the reference recordings less then 90 in at least 20$\%$ of the cases, or rate the low-anchor recording above 80 more than 50$\%$ of the time.

For CMOS-Q and CMOS-P evaluations, we randomly choose 40 samples from the LibriTTS test-set for the subjective evaluation, and each audio is listened to
by at least 10 testers. We analyze the CMOS in two aspects: CMOS-Q (quality, clarity and
high-frequency details), CMOS-P (Speech rate, pauses, and pitch). We instruct the testers to focus on the aspect in question and ignore the other aspect when scoring the aspect being considered.

## Appendix D Ablation experiments on more Training data and codebook space

We further evaluate whether a larger training dataset would elevate the upper bound of the codebook space. The results, shown in Table 8, present codebook utilization on the LibriTTS test-clean dataset. We find that increasing the training dataset size from 585 hours to 4000 hours does not lead to higher codebook space utilization. Moreover, through visualizing the probability distribution, we observe that the codebook space trained with larger datasets remains concentrated on the left side of the 4096 range.

Table: Table 8: The ablation study investigates the impact of dataset size on codebook utilization.

## Appendix E Ablation on the attention module and the contextual model size on the ARCH benchmark

We conduct ablation study on the impact of the attention module and the extended context modeling window in WavTokenizer on semantic information, with experiments performed on the ARCH (La Quatra et al., ) speech-domain datasets. The experimental results, as shown in Table 9, indicate that adding an attention module to the decoder and also extending the context window in the codec model improve the preservation of semantic information within the codec.

Table: Table 9: The ablation study of the attention module and the contextual window size on the semantic information in WavTokenizer.

## Appendix F Limitation and Future work

While WavTokenizer is capable of reconstructing high-quality audio using only 75 tokens and demonstrates the potential of a single-layer quantizer with a large codebook space in downstream generative models, current acoustic codec models lack the understanding capabilities (ASR) found in semantic models (Hsu et al., ). This limitation constrains the development of codec models within unified multimodal understanding and generation frameworks, such as the GPT-4o paradigm. At present, distillation methods used in acoustic codec models serve more as temporary solutions. Unlike the powerful decoder we design in WavTokenizer, in future work, we aim to explore a more robust encoder module that can further improve compression, reconstruction, and semantic information retention.

Additionally, we will train WavTokenizer on hundreds of thousands of hours of speech data, and will verify whether acoustic codec models with single-layer quantizers and large codebook spaces, trained on a large amount of speech data, can truly align speech as a special form of language to the text space within unified large multimodal models.

## Appendix G acknowledgement

This work was supported by Alibaba Research Intern Program.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 10
presentation: 3
questions: refer to weakness
soundness: 3
strengths: - Achieves state-of-the-art performance using only a single codebook with 40 or 75 tokens, demonstrating remarkable efficiency.
- Covers multiple domains, including audio, speech, and music.
- Its single codebook capability supports efficient training of large language models (LLMs).
- Provides a comprehensive analysis of different settings, along with detailed ablation studies that validate the impact of each component.
- The paper is well-written and easy to follow, with a comprehensive analysis included.
summary: This paper introduces WavTokenizer, a codec tokenizer for audio that achieves extreme compression and superior reconstruction quality compared to previous state-of-the-art models. WavTokenizer requires only a single quantizer with 40-75 tokens per second for 24kHz audio, while still preserving high subjective quality and rich semantic content. Extensive experiments demonstrate WavTokenizer's effectiveness across speech, audio, and music, with competitive results in both objective and subjective metrics, and ablation studies confirm the impact of each component.
weaknesses: Here’s a refined version of the weaknesses:

- The evaluation could be more thorough by incorporating existing benchmarks such as Codec-Superb and DASB to enable a more comprehensive comparison of the proposed method against existing models under standardized settings.
- The model currently supports only a 24kHz sampling rate; I wonder if you anticipate any challenges in adapting WavTokenizer to different sampling rates. It would be valuable to study its performance across different sampling rates, such as lower (16kHz) and higher (44kHz or 48kHz) rates.

---

confidence: 5
contribution: 1
initial_rating: 5
presentation: 3
questions: 1, Missing baselines: SemantiCodec[1], Single-Codec[2], Mimi (Moshi) [3].

2, Weak Semantic Performance: Compared to the ARCH benchmark (https://huggingface.co/spaces/ALM/ARCH), the semantic performance of WavTokenizer is far worse than models like HuBERT base. Therefore, the claim of "rich semantic information" does not hold up well. Moreover, it is unclear why WavTokenizer's semantic capabilities were compared only against acoustic codec models, rather than semantic codecs. Why not compare against semantic codecs like SpeechTokenizer[4], SemantiCodec[1], or Mini[3]?

3, Single-VQ Assumption: The entire premise of the paper is based on the assumption that using a single-VQ codec is better for audio language modeling compared to RVQ. But is this correct? The paper mentions that RVQ-based codecs like DAC require 900 tokens per second, which is true, but it fails to acknowledge that RVQ codecs typically have a temporal resolution of only 50Hz, and the most advanced models like Mini have even lower resolution at 12.5Hz. In practice, language models typically need to model at frequencies of 50Hz or less. So, what is the real advantage of a single-VQ codec with 40 or 75 tokens per second? Moshi and Mini have already demonstrated success in low-latency speech dialogue applications, but what about single-VQ? Is reconstructing all acoustic details truly beneficial for language modeling?  Does an LLM truly need to model timbre and detailed acoustic features, or is focusing solely on semantic content sufficient?  Therefore, you must at least demonstrate the effectiveness of single-VQ in practical applications, as stated in the weakness section, rather than constructing the paper solely based on this assumption.

[1]: Liu H, Xu X, Yuan Y, et al. SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound[J]. arXiv preprint arXiv:2405.00233, 2024.

[2]: Li H, Xue L, Guo H, et al. Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation[J]. arXiv preprint arXiv:2406.07422, 2024.

[3]:Défossez A, Mazaré L, Orsini M, et al. Moshi: a speech-text foundation model for real-time dialogue[J].

[4]:Zhang X, Zhang D, Li S, et al. Speechtokenizer: Unified speech tokenizer for speech large language models[J]. arXiv preprint arXiv:2308.16692, 2023.
soundness: 2
strengths: The overall approach is sound.  The paper is straightforward and easy to understand, with a clear structure that makes it accessible to readers. Additionally, the introduction provides a good overview of the context and background, helping readers to understand the problem.
summary: The paper introduces WavTokenizer, a single-vq codec aimed at simplifying and improving the current audio language modeling approaches by replacing codec with multiple-vq. It claims to achieve competitive reconstruction quality while enhancing integration with language models.
weaknesses: The paper introduces WavTokenizer, a single-VQ codec designed to replace the current codec + LLM audio generation systems. However, there is no experimental evidence to support that WavTokenizer can effectively achieve this. Existing codec + LLM systems can generally be categorized into RVQ-style generation (e.g., VALL-E, MusicGen, Moshi) or semantic-to-acoustic approaches (e.g., SpearTTS, SeedTTS, MusicLM), all of which demonstrate strong performance.

Firstly, there are no experiments showing that WavTokenizer combined with an LLM outperforms any of the existing systems in practical applications such as TTS or music generation. Secondly, when considering the codec itself, WavTokenizer's reconstruction quality is significantly worse than RVQ-based codecs, and its semantic performance falls short compared to semantic token like HuBERT or WavLM.

Given these shortcomings, it is difficult to see any tangible improvements or significant contributions that WavTokenizer offers to the current field of audio generation.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: - How was VQ utilization calculated in Figure 2(b) of the paper?
- It seems that WavTokenizer is not inherently streamable. Can WavTokenizer be extended to support streaming encoding and decoding?
soundness: 3
strengths: - The acoustic codec representation model is a crucial technology in the current speech domain. WavTokenizer addresses one of the core challenges in the field by achieving high-quality audio reconstruction with only 40 or 75 tokens.
- WavTokenizer introduces a novel single-layer quantizer concept, demonstrating its potential in TTS tasks and offering a promising single-layer solution for codec-LLM architectures.
- From a methodological standpoint, WavTokenizer revisits vector quantization (VQ) in the speech domain and proposes a larger codebook space, a more powerful decoder (with attention mechanisms), and an extended context modeling window. These innovations appear to be effective.
- The model achieves strong experimental results across reconstruction tasks, semantic understanding tasks, and downstream TTS tasks.
- The open-sourcing of the complete training and inference code, along with model weights, will contribute to the development of the research community.
summary: This paper introduces WavTokenizer, while preserving the classical acoustic codec model paradigm, achieves high-quality audio reconstruction using only 40 or 75 tokens per second. By proposing a larger codebook space, integrating attention mechanisms, and extending the context window, WavTokenizer demonstrates impressive results in audio reconstruction, semantic understanding, and downstream TTS tasks.
weaknesses: Overall, this work does not present significant weaknesses. However, the design of a highly powerful decoder in WavTokenizer raises some concerns. Specifically, I am concerned that the increased model parameters and the introduction of attention mechanisms may potentially slow down the codec's reconstruction speed?

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Did the authors try other techniques to improve VQ codebook utilization? The current approach closely mirrors EnCodec, but DAC demonstrates that low-dimensional code lookups and L2-normalization can significantly improve RVQ scalability. A useful reference to consider is [1], which shows effective scaling strategies in image reconstruction that could be also valuable for audio codecs. I recommend continuing work on the paper, as a single-quantizer audio codec has potential, but the paper’s current scientific contribution is limited.


2. What is the motivation for evaluating semantic representation? Neural audio codecs are expected to encode low-level acoustic features rather than abstract semantic concepts. Comparing audio codecs on semantic representation could be misleading, especially given the results e.g. all codecs score below 10% on the SLURP dataset, while self-supervised models like HuBERT achieve nearly 50% (not shown in this paper). This gap calls into question the statement on line 445 that "WavTokenizer effectively captures rich semantic information" which may be an overclaim. That said, audio codecs might have a significant impact on representation learning, as shown by EnCodecMAE [2], so it may be more appropriate to treat semantic representation as a downstream task. WavTokenizer's single codebook could be particularly useful for discrete targets in BERT-like setups.

[1] Zhu, Lei, et al. "Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%."

[2] Pepino, Leonardo, Pablo Riera, and Luciana Ferrer. "EnCodecMAE: Leveraging neural codecs for universal audio representation learning."
soundness: 3
strengths: - The paper effectively motivates and addresses an important problem in neural audio codecs - quantizing audio into a single sequence of tokens rather than multiple sequences, which complicates modeling for downstream tasks.
- It presents useful findings regarding decoder design, which are supported by the ablation study. In particular, a Fourier-based decoder combined with an attention layer yields better results, while a time-domain decoder with attention performs surprisingly worse.
summary: This paper introduces WavTokenizer, a GAN-based neural audio codec that uses a single vector quantization (VQ) codebook, in contrast to previous methods that rely on multiple residual vector quantization (RVQ) codebooks. This results in a bitrate as low as 480 bits per second, and to recover quality, the authors propose replacing the time-domain decoder with a Fourier-based decoder, preceded by attention layers. The ablation study confirms these design choices, and a comprehensive evaluation, including both subjective and objective metrics, shows that WavTokenizer maintains competitive reconstruction quality with state-of-the-art models.
weaknesses: While the motivation for scaling the single VQ codebook to many entries is clear, the paper falls short of achieving high codebook utilization when expanding beyond a size of 4096. What the authors list as contributions in VQ, such as k-means initialization and random restarts, are in fact well-established techniques in neural audio compression, and this paper doesn’t offer any novel methods to improve codebook usage. This is somewhat disappointing, given that a key focus of the paper is to provide a single quantizer. More experimentation to scale the VQ codebook is needed, as the current contribution feels more incremental and may be better suited for a different venue.

Meta-review: **Paper Summary:**

This paper describes a new neural audio codec based on the VQ-VAE, focusing on low bitrates and tokens/second (important for language modeling). The proposed codec is more efficient than any current audio codec, and experiments show that the quality of encoded audio is competitive with much higher bitrate neural codecs.

**Strengths:**

Thorough empirical evaluation, comparing to many baseline codec models on speech, music, and general audio data.

The single-codebook quantization is a conceptual simplification over common RVQ and acoustic+semantic quantization strategies. 

Reviewers h5oK and 3XJA remark on the value of the empirical ablation study of architectural components.

The authors statement regarding artifacts related to this work:

> We will open-source the entire codebase and pretrained models for WavTokenizer.

**Weaknesses:**

Reviewer VpBu questions whether the single-VQ codec approach is indeed better for audio language modeling compared to RVQ. This is not convincingly addressed, because no language model is trained on top of the codec. I don't think that training an LM is necessary for presenting work on codecs, but this paper leans strongly into the framing of codecs "for Audio Language Modeling."

Reviewer h5oK is concerned that paper doesn’t offer any novel methods to improve codebook usage. And more broadly, that the paper is lacking a significant methodological contribution. I tend to agree that the performance of WavTokenizer stems more from careful engineering than a significant methodological advancement. That said, careful engineering underpins many of the modern successes of our field.

**Conclusions**

There is value in this work, and the paper has continued to improve over the discussion period. I am somewhat concerned that the paper oversells its findings, and in places it reads more like a product advertisement than a sober scientific discussion. The weaknesses identified by VpBu and h5oK (highlighted above) are legitimate, but not fatal flaws.

Decision: Accept (Poster)