WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling
Authors: Shengpeng Ji
Ziyue Jiang
Wen Wang
Yifu Chen
Minghui Fang
Jialong Zuo
Qian Yang
Xize Cheng
Zehan Wang
Ruiqi Li
Ziang Zhang
Xiaoda Yang
Rongjie Huang
Yidi Jiang
Qian Chen
Siqi Zheng
Zhou Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: speech representation
discrete codec
audio language model
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the codec tokenizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous SOTA acoustic codec models in the audio domain: 1) extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2) improved subjective quality. Despite the reduced number of tokens, WavTokenizer achieves state-of-the-art reconstruction quality with outstanding UTMOS scores and inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extended contextual windows, and improved attention networks, as well as introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conducted extensive reconstruction experiments in the domains of speech, audio, and music. WavTokenizer exhibited strong performance across various objective and subjective metrics compared to state-of-the-art models. We also tested semantic information, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in WavTokenizer. The code is available at https://github.com/jishengpeng/WavTokenizer.

Published as a conference paper at ICLR 2025
WAVTOKENIZER: AN EFFICIENT ACOUSTIC DISCRETE CODEC TOKENIZER FOR AUDIO LANGUAGE MODELING
Shengpeng Ji
a Ziyue Jiang Wen Wang
Yifu Chen
Minghui Fang
Jialong Zuo
Qian Yang
Xize Cheng
Zehan Wang
Ziang Zhang Qian Chen & Siqi Zheng v
Ruiqi Li
Xiaoda Yang
Rongjie Huang +
Yidi Jiang
Zhou Zhao
Zhejiang University & Alibaba Group & + Fundamental A1 Research (FAIR), Meta
Code and Checkpoint:
ABSTRACT
Language models have been effectively applied to modeling natural signals, such as images, video, speech, and audio. A crucial component of these models is the tok- enizer, which compresses high-dimensional natural signals into lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer, which offers several advantages over previous state-of-the-art (SOTA) acoustic codec models in the audio domain: 1) extreme compression. By compressing the layers of quantizers and the temporal dimension of the discrete codec, one-second audio of 24kHz sampling rate requires only a single quantizer with 40 or 75 tokens. 2) improved subjective reconstruction quality. Despite the reduced number of tokens, WavTo- kenizer achieves SOTA reconstruction quality with outstanding UTMOS scores and also inherently contains richer semantic information. Specifically, we achieve these results by designing a broader VQ space, extending contextual windows, improving attention networks, and introducing a powerful multi-scale discriminator and an inverse Fourier transform structure. We conduct extensive reconstruction experiments in the domains of speech, audio, and music. WayTokenizer exhibits competitive to superior performance across various objective and subjective metrics compared to SOTA models. We also evaluate Wav TavTenenieer semantic repre- sentation, VQ utilization, and adaptability to generative models. Comprehensive ablation studies confirm the necessity of each module in
INTRODUCTION
Recently, significant achievements have been made by large language models (LLMs) (Brown et al. 2020) in audio generative tasks, including multiple-speaker speech syntheses (Wang et al. 2023 Kharitonov al. 2023 Jiang et al. 2023; 2024; Ji et al., (024c), music generation (Agostinelli ct 2023), and audio generation (Kreuk et al. 2022). Furthermore, the integration the speech modality into unified large multimodal models also has garnered significant attention, such as SpeechGPT (Zhang et al.) 2023a), AnyGPT (Zhan et al., 2024), GPT-4o, and Moshi (Défossez) ct al. 2024 These successes can be largely attributed to the utilization of discrete acoustic codec representations produced by neural codec models (Zeghidour et al.) 2021 Défossez et al., 2022 Kumar et al.) 2023). These discrete acoustic codec models bridge the gap between continuous speech signal and discrete-token -baeed language models, by discretizing high-rate audio signals into a finite set of tokens, hence enabling the application of LLM architectures to audio data.
Most end-to-end discrete codec models DDéosseez et al. 2022 Wu et al. 2023) adopt a three-stage structure consisting of an encoder, a Residual Vector Quantization (RVQ) module, and decoder. The encoder performs downsampling of the audio signal in the time domain to obtain compressed audio frames. Each compressed audio frame then quantized by series of quantizers, with each quantizer
"Equal contribution. Corresponning author.

---

Published as a conference paper at ICLR 2025
4.4
600 token's
300 tokenis
400 token''s
4.0
75 tokenis
3.6
40 token's
300 tokenis
3.2
token's
UTMOS
400 token's
2.8
600 token's
2.4
GT
Encodec
2.0
300 tokents
HiFiCodec
SpeechTokenizer
1.6
DAC
100
WavTokenizer
1.2
3
5
6
8
Kbps
Figure 1: Comparison between different acoustic codec models. The y-axis UTMOS reflects reconstruction quality (UTMOS highly correlates with human evaluations), the x-axis kbps represents audio compression levels. The size of circles represents the number of discrete tokens per second.
operating on the residual of the previous one. The number of quantizers determines the overall bitrate. The decoder, the other hand, performs upsampling in the time domain to reconstruct the audio signal from the quantizer outputs. Existing acoustic codec models (Kumar et al. 2023) demonstrate impressive reconstruction quality, and generative models based on discrete codecs now capable of synthesizing speech at hear-human levels. However, two important directions are worth exploring beyond the current acoustic codec models, namely, high bitrate compression and semantic richness.
Higher Bitrate Compression. The compression level of current codec models still warrants explo- ration to achieve higher compression. Two aspects merit optimization: the number of quantizers and the temporal dimension of the codec. While some efforts have reduced the quantity of quantizers from eight to four (Yang et al.] 2023] Ji et al., 2024a), we argue that single quantizer layer fundamentally differs from multiple quantizers. When the number of quantizers exceeds one, downstream models require additional design efforts, such as VALL-E's (Wang et al. 2023) AR and NAR structures, SoundStorm's 's (Borsos al. 2023 Ji al. 2024b) parallel generation, Music- Gen's ' (Copet et al.) 2024 Peng et al. (2024) slanted autoregressive structure, and UniAudio 's (Yang al. global and local attention structures. Conversely, with single quantizer, speech modalities can be directly autoregressively embedded into large multimodal models (Touvron et al., (003). Additionally, high temporal dimensions of codees, such as DAC's (Kumar al. (023)) requirement of 900 tokens per second, substantially degrade language model generation quality and increase resource consumption.
Richer Semantic Information. Considering the gap between the codec's reconstruction paradigm and the generative paradigm of downstream models (SpeechTeam [024] Chu al. 2024), incorporating
more semantic information in codec can facilitate weakly supervised texteto-speech generation.
Moreover, many large multimodal models (Chu et al. 2023 Tang et al. 2023) adopt the Whisper paradigm for understanding tasks (the target for generation is acoustic codec); hence, incorporating additional semantic information into acoustic codec models could help unify the understanding and generation processes in multimodal models. While many approaches attempt to introduce semantic information through distillation, additional pre- pre-trained semantic modules (Zhang et al., 2023b) can interfere with the unified modeling of music and audio. Moreover, this distillation based strategy may limit the potential of codec models. Exploring more elegant approaches to directly integrate semantic information into the codec remains open question.
2

---

Published as a conference paper at ICLR 2025
In this paper, introduce Wav WoTTonenierr discrete acoustic codec model capable of reconstructing 24kHz speech, music, and audio using only 40 or 75 tokens per second. WayTokenizer achieves high reconstruction quality with extreme compression while enhancing the semantic richness of the codec. Specifically, WavTokkenize enhances audio reconstruction quality by employing a multi-scale discriminator and an inverse Fourier transform upsampling structure from the vocoder in the decoder. To compress the codec from multiple quantizers to a single one, we discover that expanding the VQ space, alongside employing recent K-means clustering initialization and random awakening strate- gies, can significantly compress audio representations while maintaining high codebook utilization. Additionally, expanding the contextual window for speech modeling and incorporating attention networks in the decoder not only balance reconstruction quality and compression but also enrich the semantic information. Our contributions can be summarized as follows:
Conceptual Contributions. We introduce the concept of compressing the quantizer layers of acoustic codec models to a single quantizer for the first time and enhancing semantic information of the codec without disrupting the codec paradigm for modeling music and audio. Inspired by detailed analysis of the codebook space in Section 3.2 we propose aligning the large speech space with the textual vocabulary and show the potential of large speech space latent form of
unique language.
Methodological Contributions. Utilizing K-means clustering initialization and random awakening strategies on the VQ codebook space, we design an expanded VQ space for compressing the codec model a single quantizer. Furthermore, we design extended contextual modeling windows and add attention mechanisms in the decoder. The integration of an inverse Fourier transform module and multi seale discriminator in the vocoder also contributes to improved reconstruction.
Experimental Contributions. Wav Tokenizer surpasses the current state-of-the-ar (SOTA) models' subjective reconstruction performance on speech, music, and audio, with only 75 tokens per second. It achieves comparable results with 40 or 75 tokens per second across broader metrics. Additional experiments demonstrate the superiority of WaTTokenizer over competitive baseline models regarding semantic information, codebook utilization, and performance in generative models. Rigorous ablation studies confirm the necessity of each component in WayTokenizer. We will open-source the entire codebase and pretrained models for WavTokenizer
2 RELATED WORK
In recent times, neural acoustic codecs (Zeghidour et al. 2021 Défossez et al. 2022 Kumar et al. 2023) have demonstrated remarkable capabilities in reconstructing high-quality audio at low bitrates. Typically, these methods employ an encoder extract deep features in latent space, which are subsequently quantized before being fed into the decoder. Given that acoustic tokens, compared to semantic tokens, can support audio, speech, and music domains, and their rich acoustic details can eliminate the need for cascading architectures in downstream generative models (Kharitonov et al. 2023. Huang et al. 2024b) or large multimodal models (SpeechTeem 2024 Anastassiou et al. 2024 current optimization directions for acoustic codec models can be categorized as follows:
Pursuing Better Reconstruction Quality. AudioDec (Wu et al. (023) demonstrates the importance of discriminators. PrompiCodec (Pan et al., 2024) enhances representation capabilities through
additional input prompts. DAC (Kumar et al. 2023) significantly improves reconstruction quality
through techniques like quantizer dropout and multi-scale Short-Time Fourier Transform (STFT) based discriminator. Vocos (Siuzdak 2023) eliminates codec noise artifacts using a pre-trained Encodec with an inverse Fourier transform vocoder. HILCCodec (Ahn et al. (024) introduces the MFBD discriminator to guide codec modeling. APCodec (Ahn et al. 2024) further enhances reconstruction quality by incorporating ConyNextV2 modules in the encoder and decoder.
Enhancing Compression. HiFi-Codec (Yang et al. (2023) proposes a parallel GRVQ structure and achieves good speech reconstruction quality with just four quantizers. Language- Codec (Ji et al. 2024a) introduces the MCRVQ mechanism to evenly distribute information across the first quantizer and only requires four quantizers to achieve excellent performance across various generative models. Regarding achieving high compression with a single quantizer, Single-Codec (Li et al. 2024) is most related to our work. Throughout the progression of our work on Wav Tokenize,,, Single-Codec designs additional BLSTM, hybrid sampling, and resampling modules ensure basic performance with a single quantizer; however, different from the impressive reconstruction performance of our Wav Tokeeiize,,, Single-Codec's reconstruction quality is uncompetitve (with UTMOS only 3.0).
3

---

Published as a conference paper at ICLR 2025
Deepening Understanding of the Codee Space. TiCodec (Ren et al. 2024) models the codec space by distinguishing between iineinndeenndent and Limedependent information. FACodec (Ju et al.) decouples the codec space into content, style, and acoustic-detail modules. Additionally, recognizing the importance of semantic information in generative models, recent efforts start integrating semantic information into codec models. RepCodec (Huang al.] 2024c) learns vector quantization codebook by reconstructing speech representations from speech encoders like HuBERT (Hsu et al. 2021) or Data2vec (Baevski et al. 2022). Speech Tokenzze (Zhang al. 2023b) enriches the semantic content of the first quantizer through semantic distillation. FunCodec (Du et al. 2023) makes semantic tokens optional and explores different combinations. SemanticCodee (Liu et al. (024) uses quantized semantic tokens and reconstructs acoustic information using an audio encoder and diffusion model. Although the semantic codecs achieve good audio reconstruction quality, they disrupt the paradigm of acoustic codec models and introduce additional training costs.
Compared to the aforementioned approaches, WayTokenizer achieves impressive reconstruction results with only one quantizer and through 40 or 75 tokens. In contrast, for one second of speech, DAC (Kumar et al., 2023) requires 900 tokens, with 9 quantizers. Furthermore, Wav Wakeneeizerr explores enhancing semantic information by strengthening the capabilities of the Codec itself.
3 WAVTOKENIZER
Our model is built on the framework of VQ-GANs. following the same paradigm as Sound- Stream (Zeghidour al., 202T) and EnCodee (Défossez et al. 2022). Specifically, WayTokenizer passes the raw audio X through three modules: 1) A full convolution encoder network that takes the input audio and generates a latent feature representation Z: 2) A single quantizer discretizes Z to generate a discrete representation Zg. 3) An improved decoder that reconstructs the audio signal X from the compressed latent representation Zg. The model is trained end-to-end, optimizing a reconstruction loss applied over both time and frequency domains, along with a perceptual loss in the form of discriminators operating at different resolutions.
Considering that Wav Tokenizer is designed as a discrete token representation for large audio language models, the focus should be on the subjective reconstruction quality of the codec (audio fidelity) and semantic content information. In Figure ! we visualize the relationship between bitrates and UTMOS metrics (Saeki et al., (022) across different codec models. As shown in the Figure [] Wav Tokenizer achieves SOTA reconstruction quality with only 75 tokens. Notably, Tavookeizerr facilitates extreme compression bitrates and achieves a fair UTMOS score of 3.6 at 0.48 kpbs.
3.1 ENCODER
Following Encodec DDéfossez et al. (2022), the encoder consists of a ID convolution with C' channels and a kernel size of 7 followed by convolution blocks. Each convolution block composed of single residual unit followed by a downsampling layer consisting of a strided convolution, with a kernel size twice of the stride S. The residual unit contains two convolutions with kernel size 3 and ttip-connection. The number of channels is doubled whenever downsampling occurs. The convolution blocks are followed by a two-layer LSTT for sequence modeling and a final 1I) convolution layer with kernel size of 7 and D output channels. Following Encodec, we set C 32, B = 4, D = 512, and use ELU (Clevert et al. 2015)) as non- linear activation function. For the stride S, we employ two configurations, (2, 4, 5, 8) and (4, 5, 6), to ensure that WayTokenizer can downsample 24 kHz speech by factors of 320 and 600 along the time dimension.
3.2 RETHINKING THE VECTOR QUANTIZATION SPACE
WavTokenizer aims to compress speech representations into the codebook space of a single quantizer. This allows for the seamless serialization of speech and elimination of the need of hierarchical design in downstream models across channel dimensions (Wang et al. 2023 Yang et al. a Borsos et al., 2023). Initially, we attempt to rely solely on single quantizer for reconstruction during training, without changing any structure; however, we find the results suboptimal. Considering the vast vocabulary space in natural language, we hypothesize that treating speech unique language might yield better results. Motivated by this hypothesis, we conduct the following analysis. First, expand the codebook space from 210 to 2-1 Next, we train 585 hours of LibriTTS and then
4

---

Published as a conference paper at ICLR 2025
visualize the probability distribution of codebooks the LibriTTS test-clean dataset, shown in Figure (a). We observe a concentration of the speech vocabulary to the left of 2 indicating the potential merit of utilizing the larger 2 speech vocabulary space since the current codec codebooks 20 may not fully represent the speech space. More analyses in Appendix D verify that increasing the training dataset size does not lead to higher codebook space utilization and the codebook space trained with 4000 hours multilingual data remains concentrated to the left of 2
0.01%
120%
44
Utilzzatinn Rate
UTMOS
Probability Donslty
0.03%
Utiliaation Rata
80%
4.0 80
0.02%
60%
3.8
0.01%
40%
0-
20%
3.4
0
1024
2048
3072
4006
5120
8192
16384
2
2
2 (b) Codeboolas
2
2
Codebooles
Figure The visualization analysis of WayTokenizer's quantized codebook space. Figure (a) illustrates the probability distribution of each codebook index (1-16384) on the LibriTTS test-clean across different codebook spaces. Figure (b) examines the relationship between reconstruction quality in terms of UTMOS and codebook utilization rate across different codebook spaces.
Simply expanding the quantized codebook space could lead to lower utilization rates. To mitigate this issue, leveraging recent advancements in codec models (Défossez et al. 2022 Ju et al.)., we use k-means clustering initialize the codebook vectors. We adjust the number of cluster centers 200 to align with the larger codebook space. During training, for each imput, the code is selected, assigned, and updated using exponential moving average with decay of 0.99, and codes unassigned for several batches are replaced with input vectors randomly sampled from the current batch. This forced activation strategy (Dhariwal et al., 2020) helps ensure effective utilization of the large codebook space. We analyze the relationship between codebook utilization rate and reconstruction result, after applying the aforementioned k-means clustering initialization and random awakening strategies. Figure 2 (b) confirms that expanding the corresponding codebook space appropriately can reduce information loss caused by compressing the hierarchical RVQ structure into single quantizer. Speech can be effectively reconstructed under serialized quantizer structure, with a codebook space of 2 !2 achieving a favorable balance between codebook utilization and reconstruction quality. Furthermore, the experiments in Section 4.3 explore the potential of a large codebook space in discrete acoustic codecs as a specialized form of language representations.
3.3 IMPROVED DECODER
As in FACodec UJu et we believe that the decoder plays more crucial role than the encoder during the acoustic codec reconstruction process. Upsampling and reconstructing audio from the highly compressed information in WavTokenize is particularly challenging. Notably, WayTokenizee does not employ a mirrored decoder upsampling structure (Kumar et al. 2023), a standard practice that uses stack of dilated convolutions increase the receptive field, and uses transposed convolutions to sequentially upsample the feature sequence to the waveform. Since this standard design is known to be susceptible aliasing artifacts, following Vocos (Siuzdak) 2023), we maintain consistent feature resolution at all depths and achieve waveform upsampling through inverse Fourier transform. In the decoder. the target audio signal X is represented using STFT:
ST (X
Σ ==
X [n]] In m] f22kknKK
where K denotes the number of frequency points after performing Discrete Fourier Transform (DFT), denotes the frequency index, denotes the number of points in the sampled sequence, with n denoting particular sample point, and m denoting the index length. In the practical implementation, STFT is performed by applying a series of Fast Fourier Transforms (FFTs) to overlapping and windowed frames of data. The window function advances hops through time to create these frames.
5

---

Published as a conference paper at ICLR 2025
Moreover, to directly enhance the semantic modeling capability of the acoustic codec model, rather than adding various semantic tokens (Zhang et al., 2023b). we hypothesize that incorporating an attention network module (Rombach et al. (022) into the decoder may enhance information reconstruction and semantic modeling. Although attention models have proven their scalability and high performance in broader tasks (Dosovitskiy et al., 2020 Fang et al. 2024 Huang al.) 2024a Yang et al., bb. the encoder-decoder structure in the acoustic codec model remains fully convolutional. Concerns may arise about the potential extrapolation issues of attention models when modeling long sequences during inference, given that acoustic codec models often train on randomly selected short secone audio clips. However, our experiments show that Wav WavTokenizer achieves good reconstruction even for long audio sequences during inference. Additionally, we find that simply expanding contextual modeling windows to three seconds for WayTokenizzr with attention modules could further improve codec reconstruction during training and in turn the reconstruction quality during inference. This is probably because one-second clips, including silence, may contain insufficient semantic information; hence, increasing the contextual modeling window size helps the Codec model better capture context. We validate these findings through detailed ablation studies (Section 4.3)) We investigate various configurations of introducing attention modules to the encoder, the decoder, or both, and find that adding the attention module to the decoder only is beneficial, and adding it before the ConvNext module (Liu et al. 2022) appears to be optimal.
Therefore, for the representation of the intermediate signals Z, after quantization, WaTTokeniee only needs to input Zy into the conv ID layer, attention block, ConvNeXt blocks, which serves as the fundamental backbone. Subsequently. a Fourier transform is performed on the real-valued signals. Following Vocos (Siuzdak 2023), the ConvNeXt Block first embeds the input features into a hidden dimensionality and then applies a sequence of convolutional blocks. Each block is composed of snge-kernel-sized depthwise convolution, followed by an inverted bottleneck that projects features into a higher dimensionality using pointwise convolution. GELU activations (Hendrycks & Gimpell 2016) are used within the bottleneck, and Layer Normalization employed between the blocks. Regarding the transformation of real-valued signals, utilize single side band spectrum, resulting in n + 1 coefficients per frame. Since we parameterize the model to output both phase and magnitude values, the activations of the hidden dimensions are projected into tensor h with ft + channels. Finally, the inverse Fourier transform JF is used to directly reconstruct the final audio.
3.4 THE ADVANCED DISCRIMINATOR AND THE LOSS FUNCTIONS
We use the adversarial loss to promote perceptual quality. Following Vocos Siuzdak 2023), We em- ploy the open-source multi-period discriminator (MPD) (Kong et al. 2020). the single band amplitude only and multi band complex to learn discriminative features about specific sub-band and provide stronger gradient signal the generator, following (Kumar et al. (003),, ) we use a complex STFT discriminator (Zeghidour et al. 2021) at multiple time-scales (Défossez et al. (022). We adopt a hinge loss formulation instead of the least squares GAN objective, as suggested by (Zeghidour et al. 2021). The discriminator training loss Liiss(X X) is as follows:
discriminator (MRD) (Jang et al. 202T). Furthermore,
mar 1 Dx(X)) + m0. + (x))
(2)
K
K denotes the number of discriminators, Dj denotes the k-th discriminator. The training loss for the generator of WayTokenizer consists of four components: quantizer loss, mel-spectrum reconstruction loss, adversarial loss, and feature matching loss. The quantizer loss is defined as follows:
N
Σ
2,
-
2/
(3)
1=1
The mel-spectrum reconstruction loss is defined as follows:
Linet (xnet(X,, X) = MMl(X) - Mel(x)]
(4)
Furthermore, we define the adversarial loss as a hinge loss over the logits of these discriminators:
K
Ladv =
(0, (x))
(5)
K
k=
6

---

Published as a conference paper at ICLR 2025
The feature matching loss, denoted feeat is calculated the mean of the distances between the Ith feature maps the kth subdistriminator:
K L ΣΣ -
featt
(6)
K
The total training loss of the generator, Lyen, is computed as:
Cyen AgGy + met + Mede + X Aeeat
(7)
where Ago Amel, Aado Afeat are hyper-parameters.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset. Due to constrained computation resources, we conduct the training process only on a subset of common publicly available datasets. WayTokenizer is trained approximately 8K hours of data. For the speech domain, we use LibriTTS Zen et al., 2019), VCTK (Veaux et al. 2016). and subset of Common Voice (Ardila al. 2019 (300 hours are randomly selected). For the audio domain, we utilize a subset of AudioSet (Gemmeke et al. 2017) hours are randomly selected); and for the music domain, we employ the Jamendo (Bogdanov et al. 2019) and MusicDB (Rafil et al. 2017) datasets. We evaluate speech reconstruction performance of codec in clean and noisy environments using the LibriTTS test-clean and test-other sets respectively, and assess audio and music reconstruction performance using the AudioSet eval and MusicDB test sets respectively. On the other hand, for most confirmatory experiments, such as the ablation experiments, we evaluate the results with the Wav Tokenizer trained only on LibriTTS.
Baselines. We select the state-of-the-art (SOTA) codec models as the baselines for WayTokenizer. To ensure fair comparisons, we employ the official weights provided by Encodec (Défossez et al. 2022 HiFi-Codec (Yang et al. 2023 Vocos (Siuzdak 2023
SpeechTokenizer (Zhang et al.
2023b and DAC (Kumar et al. 2023 frameworks.
Evaluation metries and Implementation Details. For objective evaluation of discrete codec models, following Vocos (Siuzdak 2023), employ the UTMOS (Saeki et al., (022) automatic Mean Opinion Score (MOS) prediction system. UTMOS can yield scores highly correlated with human evaluations, closer human perception than PESQ (Perceptual Evaluation of Speech Quality) (Rix) et al., 200T). but it is restricted to 16 kHz sample rate. we also adopt the metrics in speech enhancement fields, such PESQ, STOI "Short-time Objective Intelligibility). and the F1 score for voiced/unvoiced classification (V/UV F1). In addition to these objective metrics, following Encodec (Défossez et al. 2022), we also employ the subjective MUSHRA evaluation to assess the reconstruction performance of the codec. We employ the common subjective CMOS evaluation metrics to assess the performance of the downstream TTS model with the codec models. Details of the subjective evaluations are in Training and Inference Settings are detailed in Appendix A
Appendix C
4.2 MAIN RESULTS
Evaluation on Reconstruction. We compare the speech reconstruction performance of Wav WaTeoeeniee with a broad selection of SOTA and competitive codec models as baselines on LibriTTS test- clean (4837 samples), LibriTTS test-other (5120 samples), and LJSpeech (13100 samples), which correspond to audio reconstruction in clean, noisy, and out-of-domain environments, respectively. Notably, RVQ-based codec models often select quantizers with varying bandwidths during training. To ensure fair comparisons, we use the quantizers that the baseline models are trained on. The results are shown in Table [ We observe the following: 1) Tokenizz achieves impressive results on the UTMOS metric, with WayTokenizer at 0.9 kbps surpassing the current SOTA DAC model at 9 kbps on all test sets. Since the UTMOS metric closely aligns with human perception
http:s//aathh.cco//eeeeeee/ierer coded
7

---

Published as a conference paper at ICLR 2025
Table Objective reconstruction results of different codec models on LibriTTS test-clean (clean environment), LibriTTS test-other (noisy environment), and LJSpeech dataset (out-of-domain environment). Nq denotes the number of quantizers. GT denotes ground truth waveforms. Best
results from models with single quantizer (hence directly comparable to WayTokeniizr)) are in bold.
Dataset
Model
Bandwidth
Nq
token/s
UTMOS
PESQ
STOI
VUV FI
4,0562
DAC
9.0kpbs
900
3.9097
3.9082
0.9699
0.9781
Encodec
6.0kbps
600
3.0399
2.7202
0.9391
0.9527
Vocos
6.0kbps
3,6954
2.8069
0.9426
0,9437
Sokeniro
6.0kphs
600
3,8794
2.6121
0.9165
0.9495
DAC
4.0kbps
400
3.4329
2.7378
0.9280
0.9572
LibriTSS test-clean
HiFi-Codec
3.0kbps
3.7529
2.9611
0.9405
0.9617
HiHi-Codec
4.0khps
300
3,9135
3.0116
0.9446
0.9576
Encodee
3.0kbps
300
2,3070
2.0517
0.9007
0.9198
3.0khps
0,9358
SpeechTokenier
3.0kbps
300
3,5632
1.9311
0.8778
0.9273
DAC
1. L.0bps
100
1,4940
1.2464
0.7706
0.7941
WayTokenizer
0.5khps
1.7027
0.8615
0.9173
WayTokenizer
0.9kbps
75
4,0486
2.3730
0.9139
0.9382
GT
3,4831
DAC
9.0kpbs
900
3,3566
3.7595
0.9576
9696
6.0khps
0,9338
Vocos
6kbbss
600
3,1956
2.5590
0.9209
0,9202
SpeechTokeeizer
6.0kphs
600
3,2851
2.3269
0.8811
0.9205
DAC
4.0kbps
2.9448
2.5948
0.9083
0,9404
HiFi-Codec
4.0bpps
400
3,0750
2.5536
0.9126
0.9387
LibriTS test-atker
Hili-Codec
3.0khps
300
3. 3.31134
2.6083
0.91666
0.9318
Encodec
3.0kbps
300
2,0883
2.0520
0.8835
0.8926
Vocos
3.0bbps
300
3,0558
2.1933
0.8967
0.9051
SpeechTokemizer
3.0kbps
300
3.0183
1.7373
0.8371
0.8907
DAC
1.0kbps
100
1,2454
0.7505
0.7775
WayTokenize
0.5kbps
40
3,0545
1.6622
0.8336
0.8953
Wav Tokenizer
1kbpss
75
3,4312
2.2614
0,8907
0.9172
4.3794
DAC
9.0kpbs
4,3007
3,9022
0.9733
0.9757
Enendee
6.0khps
6010
3,2286
2,6633
0.9441
0.9555
Vocos
6.0kbps
4.0332
2.9258
0.9497
0.9459
SpeechTokenizer
6.0kpbs
600
4.2373
2.6413
0.9316
0,9452
DAC
4.0khps
400
3.8109
2.7616
0.9338
0.9524
LJSpeech
HiFi-Codec
4.0kbps
400
4.1656
2.7629
0.9446
0.9497
HiFl-Codec
3.0kbps
300
4,2692
2.9091
0.9485
0,9469
Encodec
3.0kbps
300
2,3905
2.0194
0.9058
0.9326
Vocos
3.0kbps
300
3,7880
2.5006
0.9310
0.9388
SpecchTokenieer
3.0kpss
300
3,9908
2,0458
0.9021
0.9299
DAC
1. 1.0kpps
100
1,4438
1,2084
0.7822
0,8095
Way
0.5khps
4.0186
2.1142
0.9093
0.9406
WaYTokenizer
0.9kbps
4,2580
2.4923
0.9312
0.9397
of audio quality (Saeki et al., 2022), these results validate that WayTokenize maintains excellent reconstruction quality under extreme compression. 2) When compared the SOTA DAC model with single quantizer (hence directly comparable to Way Tokenizer, shown in the shaded regions of Table [. WavTokenizer with 40 and 75 tokens remarkably outperforms DAC with 100 tokens across all metrics. To the best of our knowledge, Way Takeeizzz is the first codec model capable of effectively reconstructing audio with a single quantizer. 3) On objective metrics STOI, PESQ. and Fl score, Takeeiize also performs comparably to the Vocos model with four quantizers and the SpeechTokenizer model with eight quantizers. 4) noisy environments and oft-of-domain scenarios, WayTezenizer demonstrates strong robustness and generalizability across all metrics.
We further evaluate the reconstruction performance of the codee models on the broader range of music and audio domains. Following Encodec, used MUSHRA the metric for subjective evaluation. As shown in Table 2 WayTokenizer at 0.9 kbps outperforms the SOTA DAC model at 9 kbps in reconstruction quality the speech, music, and audio domains. These results further demonstrate that
is capable of maintaining high subjective reconstruction performance on speech, music, and audio, with extremely limited number of tokens.
Evaluation on Semantic Representation. We evaluate the semantic richness of different codec models the ARCH benchmark (La Quatra al. 2024 Notably, opt not utilize the
8

---

Published as a conference paper at ICLR 2025
conventional Superb benchmark (Yang et al. 2021) due to its exclusive focus the speech domain, while ARCH enables further assessment of a Codec model in music and audio realms. The ARCH benchmark comprises 12 datasets in speech, music, audio domains (details in Appendix B). We extract embeddings corresponding to the discrete codebooks of an acoustic codec model as its respective representations and evaluate the classification accuracy of the codec model ARCH datasets using its representations. For fair comparisons, we evaluate Encodec and DAC models on semantic representation, as they are under the same paradigm as WayTokenizer. The experimental results, as shown in Table 3 demonstrate that Wav TavTooenizer substantially outperforms DAC and Encodec configured with single quantizer or two quantizers on classification aceuracy. Remarkably, on the AM and SLURP datasets in the speech domain, the MTT and IRMAS datasets in the music domain, and the FSD50K and VIVAE datasets in the audio domain, Way TayTokenizer surpasses DAC with nine quantizers and Encodec with eight quantizers on classification performance.
Table The subjective reconstruction results using MUSHRA (comparative scoring of samples) of codec models on speech, music and audio domains. Nq denotes the number of quantizers.
Model
Bandwidth +
Nq.
token/s
LibriTTS test-clean 1
MusicDB
Audioset f
GT
96.411.2
95.311.7
95.8122.1
DAC
9.0kpbs
900
92.811.8
92.61:..4
92.7111.5
Encodec
6.0kbps
600
78.6
76.911.6
81.2.11.8
DAC
1.kbps
100
58.4
57.612.1
56.8
WavTokenizer
0.9kbps
75
96.1
92.9 2.2
94.4.1..6
Table 3: The semantic representation (speech, music, audio) evaluation of different codec models
on ARCH Benchmark in terms of classification accuracy. Nq represents the number of quantizers.
Nq.
takea's
RAVDESS
SLURP
AM1
MIT
IRMAS
ESC50
USSK
VIVAET
DAU
9033
0.3750
0.0779
0.33663
110920
013504
0.3915
0.4125
016014
0121144
01.032
02.2917
0,3680
Encades
503
0.2881
0.0536
0.2261
0.438
0.2750
0.1993
0,3671
0.3917
0.1925
0.3055
0.1091
0,3005
DAC
4
400
0.1194
017767
0.2365
01683
0.379
0.3784
0.813
0.9967
112200
115534
011391
0.3541
Encodee
300
0.2051
0.0660
0.2193
0.4301
0.228
0.934
0.3684
0.3656
0.1790
0.3097
0,1099
0,2710
100
033522
0.1295
0.299
WasTolemizer
0.3255
0UB02
0.316
0.687
0.3417
0.3835
0.4117
IL5764
0.2858
0.3975
0.1392
0.356
Evaluation on Downstream Generative Tasks. We evaluate WayTokenizer's performance on downstream generative tasks exemplified by text-to-speech synthesis (TTS) (Ren et al. 2020). We adopt an autoregressive language model backbone. Specifically, we follow the MusicGen paradigm (Copet al. 2024), which expands the acoustic codec sequence through autoregressive prediction, and modify the open-sourced ParlerTTS model (Lyth & King 2024) accordingly. We the ParlerTTS 600M model configuration and its default hyperparameters, and train TTS models based on DAC and WayTokenizer representations on the LibriTTS dataset. Each model is trained for 40 epochs 8 A800 80GB GPUs.
The results are shown in Table 4 In terms of audio quality (CMOS-Q) Q) and audio prosody (CMOS-P) the speech synthesis model trained on WayTokenizer's single-layer quantizer representations outper- forms that trained on DAC's 9-layer quantizer representations. This demonstrates that: 1) Acoustic models with single-layer quantizer show significant potential in downstream autoregressive audio generation models, and 2) Speech synthesis models using large-codebook acoustic repre- sentations can also synthesize high-quality audio, suggesting that large- codebookk acoustic spaces have the potential to model speech as a special form of text. We will validate these findings on large multimodal models (Zhan et 2024 based on acoustic codec models that are trained on larger datasets in future work.
Table 4: The Subjective Evaluations of various acoustic codec models for downstream text-to- speech synthesis models on the LibriTTS test set. GT denotes ground truth waveforms.
Model
Bandwidth 1 Ng L
CMOS-Q1
CMOS-Pt
GT
0.22
0.26
DAC
9.0kbps 0.9kbps
9
-0.35
-0.29
Way TavTooenizerr
1
0.00
0.00

---

Published as a conference paper at ICLR 2025
Table 5: Impact of codebook scale. Utiliza-
Table 6: Impact of the contextual modeling window size.
tion rate reflects codebook's 's usage efficiency.
Midel
Cadebooks
Utitiorian
UTMUS
PRSQ
STO
Codebooks
windows
UTMOS
PESQ
Tokenizer
16384
27%
3,5009
2,500
02:22
Was Tokeiies
4016
3.7448
200112
0,8944
Tokenzerr
100%
2,5730
1.7751
WarTokeuzzrr
WarTokee
4066
4,0486
4,0445
2,3730
2.3555
0.9139
4.3 ABLATION STUDY
Due to limited compute resources, we use 585 hours LibriTTS training data for training Wav Tokenizer and conduct ablation studies on reconstruction performance on the LibriTTS test-clean subset. For all ablation experiments, we use WayTokenizer with single quantizer operating at 0.9 kbps.
Codebook size. we evaluate varying codebook sizes on the performance WayTokenizer. We record the frequency of each codebook entry LibriTTS test-clean. As shown in Table and discussed in Section 3.2 we observe significant potential for expansion in the typical codebook space, even under extreme compression with single quantizer, when combined with advanced training strategies (Section 3.2). We find that expanding the codebook size from the typical 1024 to 4096 significantly enhances audio quality, with the UTMOS gain by 0.55, PESQ gain by 0.6, and STOI gain by 0.5. However, excessively large codebook spaces (1 (16384) can lead to reduced codeboook utilization. Contextual window size. Most Codec models are trained on randomly selected one-second audio clips. With an attention module in Wav WavTookenssr''s 's decoder, shown in Table using a three-second contextual window further enhances the reconstruction quality. We hypothesize that a one-second window may contain insufficient information and be more affected by silence. Longer contextual window may enhance attention module's 's ability to capture relevant semantics.
Multi-scale STFT discriminator. As shown in Table the multi-scale STFT discriminator (MSTFTD) enhances the reconstruction quality, albeit increasing the training time. 'he improvement is probably due the fact that MSTFTD splits STFT into sub-bands and learns discriminative features for specific sub-ban band, hence providing stronger gradient signal to Wav Way'okenizer's 's generator, in turn improving gegufrrey prediction and mitigating aliasing artifacts.
Attention module. We remove the attention module from Way TayTokeniser's decoder, resulting in WayTokenizer w/o attention. As shown in Table 7, removing the attention module degrades WayTok- enizer's $ performance more than MSTFTD. Ablation results on impact of the attention module on semantic representation are in Appendix Improved decoder. We replace the improved decoder based on the inverse Fourier transform with an up-sampling structure mirroring the encoder, denoted as WavTokenizer w/ mirror decoder. As shown in Table 7 purely mirrored structure substantially compromises the reconstruction performance of Wav Tokeninne under extreme compression. This underscores the importance robust decoder in ensuring the reconstruction performance of codec models under high compression.
Table 7: Ablation on the multi-scale STFT discriminator (MSTFTD), the atention module, and switching from our improved decoder to a mirror decoder, in Wav WavTokenizer.
Model
UTMOS
PESQ t
STOI
V/UV FI 1
TavTekenizer
4.0486
2.3730
0.9139
0.9382
mirror decoder
2.7782
1.5007
0.8249
0.8820
w/o attention module
3.6020
1.9332
0.8734
0.9067
w/o MSTFTD
3.7806
2.1270
0.9008
0.9269
CONCLUSION
In this paper, we introduce WayTokenizer, codec model capable of quantizing one second speech, music and general audio into 75 or 40 tokens with a single quantizer. Compared to SOTA acoustic codec models, WavTokenizer maintains high subjective reconstruction quality and preserves rich semantic information even under extreme compression. The limitation of WoTeenzzee and future work are discussed in Appendix F
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was done at Alibaba Group during the internship of Shengpeng Ji. I would like to express my sincere gratitude to my internship mentors, Qian Chen, Siqi Zheng, and Wen Wang, for their valuable guidance and support. This work was supported by the National Natural Science Foundation of China under Grant No.62222211.
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint 2r 1135,, 2023.
Sunghwan Ahn, Beom Jun Woo, Min Hyun Han, Chanyeong Moon, and Nam Soo Kim. Hilcodec: High fidelity and lightweight neural audio codec. arXiv preprint arXiv:2405.04755 2024.
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint vrXiv:2406.024380 2024.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: massively- multilingual speech corpus. arXiv preprint arXiv: 1912.0667,, 2019.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning, 1298-1312. PMLR, 2022.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. Slurp: A spoken
language understanding resource package. arXiv preprint arXiv:2011.13203 2020.
Sören Becker, Johanna Vielhaben, Marcel Ackermann, Klaus-Robert Müller, Sebastian Lapuschkin, and Wojciech Samek. Audiomnist: Exploring explainable artificial intelligence for audio analysis on simple benchmark. Journal of the Franklin Institute, (61(1):418-42 2024.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg-jamendo dataset for automatic music tagging. ICML, 2019.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.096. 2023.
Juan J Bosch, Jordi Janer, Ferdinand Fuhrmann, and Perfecto Herrera. A comparison of sound segregation techniques for predominant instrument recognition in musical audio signals. In ISMIR, pp. 559-564, 2012.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 2023.
Yunfei Chu. Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv. Jinzheng He, Junyang Lin, al. auen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024.
Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 151.0728) 2015.
11

---

dataset for Published as a conference paper at ICLR 2025
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
Giovanni Costantini, Iacopo Iaderola, Andrea Paoloni, Massimiliano Todisco, et al. Emovo corpus: an italian emotional speech database. In Proceedings of the ninth international conference on language resources and evaluation (LREC' 14), pp. 3501-3504. European Language Resources Association (ELRA), 2014.
Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. Fma: music analysis. arXiv preprint arXiv:1612.018800, 2016.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint TrXiv:2210..34555 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. Technical report, Kyutai, September 2024. URL htt.hhttp///kytaai.... http://ytttaa..o..ooo.h....... pddf
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.0034/ 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly. et al. An image is worth 16x16 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. ArXiv, 0etp2229900705 2023. URL 823065
https://api. www.:pppppeeeenittssiieee.eesess icschllar www.orrg//ooooo Corg/CorpussD/.
Minghui Fang, Shengpeng Ji, Jialong Zuo, Hai Huang, Yan Xia, Jieming Zhu, Xize Cheng, Xiaoda Yang. Wenrui Liu, Gang Wang, et al. Ace: A generative cross-modal retrieval framework with coarse-to-fie semantic modeling. arXiv preprint arXiv:2406. 175007,77777... 2024.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: open dataset of human-labeled sound events. /EEF/ACM Transactions on Audio, Speech, and Language Processing, 30:829-888 852, 2021.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780, IEEE, 2017.
Dan Hendrycks and Kevin Gimpel. 1oXiv:1606008411 2016.
Gaussian error
linear units (gelus). arXiv preprint
Natalie Holz, Pauline Marruuy,aeettr, and David Poeppel. The variably intense vocalizations of affect and emotion (vivae) corpus prompts new perspective on nonspeech perception. Emotion, 22 (1):213, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions audio, speech, and language processing, 29:3451-344 34600 2021.
Hai Huang. Yan Xia, Shengpeng Ji, Shulei Wang, Hanting Wang, Jieming Zhu, Zhenhua Dong, and Zhou Zhao. Unlocking the potential of multimodal unified discrete representation through training-free free codebook optimization and hierarchical alignment. arXiv preprint 08X 2024a.
12

---

Published as a conference paper at ICLR 2025
Rongjie Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Jinchuan Tian, Zhenhui Ye, Luping Liu. Zehan Wang. Ziyue Jiang, Xuankai Chang, et al. Make-a aviice: Revisiting voice large language models scalable multilingual and multitask learners. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), Pp- 10929-10942, 2024b.
Zhichao Huang. Chutong Meng, and Tom Ko. Repcodec: A speech representation codec for speech tokenization, 2024c. URL httttpss//axxii.orr///aaaa
0013099..169
Won Jang. Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation. arXiv preprint arXiv:2106.07889, 2021.
Shengpeng Ji, Minghui Fang, Ziyue Jiang, Rongjie Huang, Jialung Zuo, Shulei Wang, and Zhou Zhao. Language-codec: Reducing the gaps between discrete codee representation and speech language models. arXiv preprint arXiv:2402. 12208, 2024a.
Shengpeng Ji, Ziyue Jiang, Hanting Wang, Jialong Zuo, and Zhou Zhao. Mobilespeech: A fast and high-fidelity framework for mobile zero-shot text-to-speech. arXiv preprint 097777 2024b.
Shengpeng Ji, Jialong Zuo, Minghui Fang, Ziyue Jiang, Feiyang Chen, Xinyu Duan, Baoxing Huai, and Zhou Zhao. Textrolspeech: A text style control speech corpus with codec language text-to- speech models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10301-10305. IEEE, 2024c.
Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang. Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias. arXiv preprint arXiv:2306.00335 035009 2023.
Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang. Chen Zhang, Pengfei Wei, Chunfeng Wang, et al. Mega-tts 2: Boosting prompting mechanisms for zero-shot speech synthesis. In The Twelfth International Conference on Learning Representations, 2024.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In Forty-first International Conference on Machine Learning.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11:1703-7718, 2023.
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033, 2020.
Felix Kreuk, Gabriel Synnaeve. Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209. /5352,, 2022.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. arXiv preprint arXiv:2306.065.4 065446 2023.
Moreno La Quatra, Alkis Koudounas, Lorenzo Vaiani, Elena Baralis, Luca Cagliero, Paolo Garza, and Sabato Marco Siniscalchi. Benchmarking representations for speech, music, and acoustic events. arXiv preprint ar Mi1::2405.0093.. 2024.
Edith Law, Kris West, Michael Mandel, Mert Bay, and J Stephen Downie. Evaluation of algorithms using games: The case of music tagging. In ISMIR, pp. 387-392. Citeseer, 2009.
Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Yuanjun Lv. Lei Xie, Yunlin Chen, Hao Yin, and Zhifei Li. Single-codec: Single-codebook speech codec towards high-performance speech generation, arXiv preprint 02X14 2024.
13

---

Published as a conference paper at ICLR 2025
Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang. and Mark D Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound. arXiv preprint vrXiv:2405.00233 2024.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-1198, 2022.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database of emotional speech and song (ravdess). Funding Information Natural Sciences and Engineering Research Council of Canada, 341583, 2012.
Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01911,, 2024.
Yu Pan, Lei Ma, and Jianjun Zhao. Prompteodec: High-fidelity neural speech codec using disen- tangled representation learning based adaptive feature-aware prompt encoders. arXiv preprint arXiv:2404.02700 02702, 2024.
Puyuan Peng, Po- Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. Voicecraft: sero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403. 16973, 2024.
Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Zafar Rafii, Antoine Liutkus, Fabian-Rober, Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. The musdb 18 corpus for music separation. 2017.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint vrXiv22006.0455, 2020.
Yong Ren, Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. Fewer- token neural speech codec with iive-invriiant codes. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12737-12741, IEBE, 2024.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a a new method for speech quality assessment telephone networks and codecs. In 2001 IEEE international conference acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), volume 2, pp. 749-752. IEEE, 2001.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hi- roshi Saruwatari. Utmos: Utokyo-sarulab system for voicemos challenge 2022. arXiv preprint arXiv:2204.02//2, 2022.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, 2014.
Series. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommmmication Assembly, 2, 2014.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00 00814, 2023.
Tongyi SpeechTeam. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and 1lms. arXiv preprint arXiv:2407.04051. 2024.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310. 13289, 2023.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXtv:2302.1397,, 2023,
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, al. Superseded-cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit. 2016.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, al. Neural codec language models are zero-shot text speech synthesizers. arXiv preprint urXiv:2301.02111 2023.
Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Donghhao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang. Songxiang Liu, Haohan Guo. Xuankai Chang, Jiatong Shi, Jiang Bian, Zhou Zhao, et al. Uniaudio: Towards universal audio generation with large language models. In Forty-firs International Conference on Machine Learning, a.
Dongehao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.02765. 2023.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051, 2021.
Xiaoda Yang, Xize Cheng. Dongjie Fu, Minghui Fang, Jialung Zuo, Shengpeng Ji, Tao Jin, and Zhou Zhao. Synctalklip: Highly synchronized rea-read speaker generation with multi-task learning. In ACM Multimedia 2024, b.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jía, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv: 1rXiv:190..00 1904.02882.. 2019.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal 11m with discrete sequence modeling. arXiv preprint arXiv:2402.12222 2024.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprini arXiv:2305.1 11000, 2023a.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. /6692, 2023b.
15

---

Published as a conference paper at ICLR 2025
A TRAINING AND INFERENCE SETTINGS
We train WayTokenizer up to 2 million iterations, with 1 million iterations allocated to training the generator and the discriminator respectively, 8 NVIDIA A800 80G GPUs. Throughout the entire training process, all input speech, music, and audio samples are resampled to 24 kHz, and the hatch size 40. We uniformly truncate excessively long segments in the training data a fixed length of 10 seconds and subsequently perform a random crop of the waveform to obtain audio snippets of 3-second duration to be fed into Wav TavTokenzerr WayTokenizer optimized using the AdamW optimizer with an initial learning rate of 2e-4 and betas set to (0.9, 0.999). The learning rate is decayed based on a cosine schedule.
B THE ARCH BENCHMARK
The ARCH benchmark comprises twelve datasets within the speech, music, audio domain. Emotional
Speech and Song (RAVDESS) (Livingstone & Russo 2012). Audio-MNIST (AM) (Becker et al.
2024), Spoken Language Understanding Resource Package (SLURP) (Bastianelli et al. 2020). and
EMOVO dataset (Costantini et al. 2014) assess performance in the Speech domain. ESC-50 50 (iizak
2015), US8K (Salamon al. 2014), FSD50K (Fonseca et al., (222)). and VIVAE (Holz al. 2022)
assess performance on Acoustic Events. FMA (Defferrard et al. 2016), MTT (Law et al. 2009).
IRMAS (Bosch et al. (0112), and MS-DB (Rafii al. (017) assess performance in the Music domain.
C SUBJECTIVE EVALUATIONS
For the subjective evaluations, we follow the MUSHRA protocol (Series) 2014). using both a hidden
reference and low anchor. Annotators recruited using crowd-sourcing platform, in which they are asked to rate the perceptual quality of the provided samples in range between I to 100. We randomly select 50 samples from each category of the test set and ensure at least 10 annotations per sample. To filter out noisy annotations and outliers, we remove annotators who rate the reference recordings less then 90 in at least 20% of the cases, or rate the low-anchor recording above 80 more than 50% of the time.
For CMOS-Q and CMOS-P evaluations, we randomly choose 40 samples from the LibriTTS test-set for the subjective evaluation, and each audio is listened to by at least 10 testers. We analyze the CMOS two aspects: CMOS-Q (quality, clarity and high-frequency details), CMOS-P (Speech rate, pauses, and pitch). We instruct the testers to focus the aspect in question and ignore the other aspect when scoring the aspect being considered.
D ABLATION EXPERIMENTS ON MORE TRAINING DATA AND CODEBOOK SPACE
We further evaluate whether a larger training dataset would elevate the upper bound of the codebook space. The results, shown in Table 8 present codebook utilization the LibriTTS test-clean dataset. We find that increasing the training dataset size from 585 hours to 4000 hours does not lead to higher codebook space utilization. Moreover, through visualizing the probability distribution, that the codebook space trained with larger datasets remains concentrated on the left side of the 4096 range.
observe
Table 8: The ablation study investigates the impact of dataset size on codebook utilization.
Model
Dataset
Codebooks
Utilization rate
UTMOS +
PESQ t
STOI
Way Tokenizer
585 Hours
16384
27% 26.5%
3.9989
2.3600
0.8129
Way Tokenizer
4000 Hours
16384
3.9465
2.3721
0.8217
16

---

Published as a conference paper at ICLR 2025
E ABLATION ON THE ATTENTION MODULE AND THE CONTEXTUAL MODEL SIZE ON THE ARCH BENCHMARK
We conduct ablation study the impact of the attention module and the extended context modeling window in WavTokenize on semantic information, with experiments performed on the ARCH (La Quatra et al. 2024) speech-domain datasets. The experimental results, shown in Table 9 indicate that adding an attention module to the decoder and also extending the context window in the codec model improve the preservation of semantic information within the codec.
Table 9: The ablation study of the attention module and the contextual window size the semantic information in WayTokenizer.
Model
Nq
token/s +
RAVDESS
SLURP
EMOVO
AM f
WavTokenizer w/o attention w/ extended windows
1
75
0,2614
0.0643 0.0802
0.2368
0.6192
WavTokenize
75
0.3255
0.3163
0.6957
F LIMITATION AND FUTURE WORK
While WavTokenizer is capable of reconstructing high-quality audio using only 75 tokens and demonstrates the potential of a single-layer quantizer with large codebook space in downstream generative models, current acoustic codec models lack the understanding capabilities (ASR) found in semantic models (Hsu et al. 2021). This limitation constrains the development of codec models within unified multimodal understanding and generation frameworks, such as the GPT-4o paradigm. At present, distillation methods used in acoustic codec models serve more temporary solutions. Unlike the powerful decoder we design in WayTokenizer, in future work, we aim to explore a more robust encoder module that further improve compression, reconstruction, and semantic information retention. Specifically, a key focus our future work on WayTokenizer will be to explore elegant methods for significantly enhancing the semantic capacity of discrete tokens within the encoder.
Additionally, will train WavTokenizer on hundreds of thousands of hours of speech data, and will verify whether acoustic codec models with single-layer quantizers and large codebook spaces, trained on large amount of speech data, can truly align speech as a special form of language the text space within unified large multimodal models.
G WAVTOKENIZER AT 16KHZ AND 48KHZ SAMPLING RATE
We have augmented the reconstruction evaluations by training both 16kHz and 48kHz versions of Wav'okenieer using LibriTTS while maintaining a consistent downsampling factor of 320x. The hyperparameters are kept at their default settings. Experimental results on the LibriTTS test-clean dataset (4,837 samples) are presented Table [0]
We find that Wav Taveooeeizer delivers high-quality and comparable audio reconstruction performance across different sampling rates of 16kHz, 24kHz,, and 48kHz. Notably, at 16kHz sampling rate, Wav'okkenizer can effectively reconstruct audio with high fidelity using only 50 tokens. We also observed that all objective metrics, except for UTMOS, exhibit an increasing trend of increasing scores when the number of tokens increases. In contrast, UTMOS for WayTokenizer remains relatively stable. We hypothesize that the reconstruction quality of WavTokenizer should be stable across different sampling rates, to human perception of audio quality; hence, this observation of stable UTMOS aligns with prior finding Saæki et al. (2022) that UTMOS highly correlates with human subjective auditory perception.
H RECONSTRUCTION SPEED
We evaluate the reconstruction speed of Semanticodec Encodec, DAC, and WayTokenizer single NVIDIA A100 80G GPU on the LibriTTS test-clean dataset. We calculate the real-time factor (RTF) by dividing the total reconstruction time by the duration of the generated audio. The results shown
17

---

Published as a conference paper at ICLR 2025
Table 10: Objective reconstruction performance for WavTokenizer at different sampling rates on Librispeech lest-clean. Sr denotes sampling rate. The results of WacTokknizz at 24kHz are compared to baseline models in T
Methods
Sr
Tokens/s
UTMOS
PESQ
STO1 1
V/UV F1 1
WavTokenizer
16000
50
3.9606
2.3240
0.9095
0.9375
WaTTokenizer
24000
75
4.0486
2.3730
0.9139
0.9382
Wav Takeniznerr
48000
150
3,9582
2.7230
0.9350
0.9496
in Table I1 Notably, despite incorporating the attention module in the decoder of Wav its reconstruction speed remains remarkably fast. This can be attributed two factors: (1) the use of a fast inverse Fourier transform, following Vocos, and (2) the low bitrate of WayTokenizer. These results demonstrate the high reconstruction efficiency WavTokenizer.
Table Reconstruction speed (measured by RTF) of different codec models on reconstruction on the LibriTTS test-clean dataset. RTF is computed by dividing the total reconstruction time by the duration of the generated audio.
Methods
RTF 1
SemantiCodec
0.9483
Encodec
0.0175
DAC
0.0144
WavTokenizer
0.0098
MORE TTS RESULTS
Note that Section 4.2 we present the subjective evaluation results of TTS models in the acoustic codec and LLM architecture in Table 4 We further supplement these results with synthesis accuracy and speaker similarity results. Building on the pre-trained TTS model using WayTokenizer as presented in Section 4.2 further evaluate the synthesis accuracy, measured by WER, and speaker similarity SPK, computed by using WavLM to extract speaker embeddings for cosine similarity, on the zero-shot TTS task, following the same settings as the VALL-E-coninnee version (Wang al. 2023). As shown in Table 12 in the audio codec and autoregressive LLM architecture for audio generation, with identical generative model, the TTS model trained WayTokenizer (0.9kbps) yields substantially lower WER than the TTS model trained on SOTA acoustic codec model DAC (9.0kbps), as 5.1 % versus 6,9%, and also achieves better speaker similarity performance, as 0.61 versus DAC's 0.59. This further confirms that WayTokenizer demonstrates excellent performance in the downstream TTS tasks.
Table 12: Word Error Rate (WER) and Speaker Similarity (SPK) of various acoustic codec models for downstream speech synthesis models. GT denotes ground truth waveforms. To evaluate Spk between the original prompt and the synthesized speech, we utilize the base-plus-sv version of W.
Model
Bandwidth
Nq
WER
SPK*
GT
2.4
DAC
9.0kbps 0.9kbps
9
6,9
0.59
Wav Wakkonnizrrr
1
5.1
0.61
18


Reviews:
[{"review_id": "6biMFTnDge", "reviewer": "Reviewer_3XJA", "summary": "This paper introduces WavTokenizer, a codec tokenizer for audio that achieves extreme compression and superior reconstruction quality compared to previous state-of-the-art models. WavTokenizer requires only a single quantizer with 40-75 tokens per second for 24kHz audio, while still preserving high subjective quality and rich semantic content. Extensive experiments demonstrate WavTokenizer's effectiveness across speech, audio, and music, with competitive results in both objective and subjective metrics, and ablation studies confirm the impact of each component.", "questions": "refer to weakness", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Achieves state-of-the-art performance using only a single codebook with 40 or 75 tokens, demonstrating remarkable efficiency.\n- Covers multiple domains, including audio, speech, and music.\n- Its single codebook capability supports efficient training of large language models (LLMs).\n- Provides a comprehensive analysis of different settings, along with detailed ablation studies that validate the impact of each component.\n- The paper is well-written and easy to follow, with a comprehensive analysis included.", "weaknesses": "Here’s a refined version of the weaknesses:\n\n- The evaluation could be more thorough by incorporating existing benchmarks such as Codec-Superb and DASB to enable a more comprehensive comparison of the proposed method against existing models under standardized settings.\n- The model currently supports only a 24kHz sampling rate; I wonder if you anticipate any challenges in adapting WavTokenizer to different sampling rates. It would be valuable to study its performance across different sampling rates, such as lower (16kHz) and higher (44kHz or 48kHz) rates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fs8rgAiNpH", "reviewer": "Reviewer_VpBu", "summary": "The paper introduces WavTokenizer, a single-vq codec aimed at simplifying and improving the current audio language modeling approaches by replacing codec with multiple-vq. It claims to achieve competitive reconstruction quality while enhancing integration with language models.", "questions": "1, Missing baselines: SemantiCodec[1], Single-Codec[2], Mimi (Moshi) [3].\n\n2, Weak Semantic Performance: Compared to the ARCH benchmark (https://huggingface.co/spaces/ALM/ARCH), the semantic performance of WavTokenizer is far worse than models like HuBERT base. Therefore, the claim of \"rich semantic information\" does not hold up well. Moreover, it is unclear why WavTokenizer's semantic capabilities were compared only against acoustic codec models, rather than semantic codecs. Why not compare against semantic codecs like SpeechTokenizer[4], SemantiCodec[1], or Mini[3]?\n\n3, Single-VQ Assumption: The entire premise of the paper is based on the assumption that using a single-VQ codec is better for audio language modeling compared to RVQ. But is this correct? The paper mentions that RVQ-based codecs like DAC require 900 tokens per second, which is true, but it fails to acknowledge that RVQ codecs typically have a temporal resolution of only 50Hz, and the most advanced models like Mini have even lower resolution at 12.5Hz. In practice, language models typically need to model at frequencies of 50Hz or less. So, what is the real advantage of a single-VQ codec with 40 or 75 tokens per second? Moshi and Mini have already demonstrated success in low-latency speech dialogue applications, but what about single-VQ? Is reconstructing all acoustic details truly beneficial for language modeling?  Does an LLM truly need to model timbre and detailed acoustic features, or is focusing solely on semantic content sufficient?  Therefore, you must at least demonstrate the effectiveness of single-VQ in practical applications, as stated in the weakness section, rather than constructing the paper solely based on this assumption.\n\n[1]: Liu H, Xu X, Yuan Y, et al. SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound[J]. arXiv preprint arXiv:2405.00233, 2024.\n\n[2]: Li H, Xue L, Guo H, et al. Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation[J]. arXiv preprint arXiv:2406.07422, 2024.\n\n[3]:Défossez A, Mazaré L, Orsini M, et al. Moshi: a speech-text foundation model for real-time dialogue[J].\n\n[4]:Zhang X, Zhang D, Li S, et al. Speechtokenizer: Unified speech tokenizer for speech large language models[J]. arXiv preprint arXiv:2308.16692, 2023.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The overall approach is sound.  The paper is straightforward and easy to understand, with a clear structure that makes it accessible to readers. Additionally, the introduction provides a good overview of the context and background, helping readers to understand the problem.", "weaknesses": "The paper introduces WavTokenizer, a single-VQ codec designed to replace the current codec + LLM audio generation systems. However, there is no experimental evidence to support that WavTokenizer can effectively achieve this. Existing codec + LLM systems can generally be categorized into RVQ-style generation (e.g., VALL-E, MusicGen, Moshi) or semantic-to-acoustic approaches (e.g., SpearTTS, SeedTTS, MusicLM), all of which demonstrate strong performance.\n\nFirstly, there are no experiments showing that WavTokenizer combined with an LLM outperforms any of the existing systems in practical applications such as TTS or music generation. Secondly, when considering the codec itself, WavTokenizer's reconstruction quality is significantly worse than RVQ-based codecs, and its semantic performance falls short compared to semantic token like HuBERT or WavLM.\n\nGiven these shortcomings, it is difficult to see any tangible improvements or significant contributions that WavTokenizer offers to the current field of audio generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BDaDiXZQPl", "reviewer": "Reviewer_wU96", "summary": "This paper introduces WavTokenizer, while preserving the classical acoustic codec model paradigm, achieves high-quality audio reconstruction using only 40 or 75 tokens per second. By proposing a larger codebook space, integrating attention mechanisms, and extending the context window, WavTokenizer demonstrates impressive results in audio reconstruction, semantic understanding, and downstream TTS tasks.", "questions": "- How was VQ utilization calculated in Figure 2(b) of the paper?\n- It seems that WavTokenizer is not inherently streamable. Can WavTokenizer be extended to support streaming encoding and decoding?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The acoustic codec representation model is a crucial technology in the current speech domain. WavTokenizer addresses one of the core challenges in the field by achieving high-quality audio reconstruction with only 40 or 75 tokens.\n- WavTokenizer introduces a novel single-layer quantizer concept, demonstrating its potential in TTS tasks and offering a promising single-layer solution for codec-LLM architectures.\n- From a methodological standpoint, WavTokenizer revisits vector quantization (VQ) in the speech domain and proposes a larger codebook space, a more powerful decoder (with attention mechanisms), and an extended context modeling window. These innovations appear to be effective.\n- The model achieves strong experimental results across reconstruction tasks, semantic understanding tasks, and downstream TTS tasks.\n- The open-sourcing of the complete training and inference code, along with model weights, will contribute to the development of the research community.", "weaknesses": "Overall, this work does not present significant weaknesses. However, the design of a highly powerful decoder in WavTokenizer raises some concerns. Specifically, I am concerned that the increased model parameters and the introduction of attention mechanisms may potentially slow down the codec's reconstruction speed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYSTlxLOga", "reviewer": "Reviewer_h5oK", "summary": "This paper introduces WavTokenizer, a GAN-based neural audio codec that uses a single vector quantization (VQ) codebook, in contrast to previous methods that rely on multiple residual vector quantization (RVQ) codebooks. This results in a bitrate as low as 480 bits per second, and to recover quality, the authors propose replacing the time-domain decoder with a Fourier-based decoder, preceded by attention layers. The ablation study confirms these design choices, and a comprehensive evaluation, including both subjective and objective metrics, shows that WavTokenizer maintains competitive reconstruction quality with state-of-the-art models.", "questions": "1. Did the authors try other techniques to improve VQ codebook utilization? The current approach closely mirrors EnCodec, but DAC demonstrates that low-dimensional code lookups and L2-normalization can significantly improve RVQ scalability. A useful reference to consider is [1], which shows effective scaling strategies in image reconstruction that could be also valuable for audio codecs. I recommend continuing work on the paper, as a single-quantizer audio codec has potential, but the paper’s current scientific contribution is limited.\n\n\n2. What is the motivation for evaluating semantic representation? Neural audio codecs are expected to encode low-level acoustic features rather than abstract semantic concepts. Comparing audio codecs on semantic representation could be misleading, especially given the results e.g. all codecs score below 10% on the SLURP dataset, while self-supervised models like HuBERT achieve nearly 50% (not shown in this paper). This gap calls into question the statement on line 445 that \"WavTokenizer effectively captures rich semantic information\" which may be an overclaim. That said, audio codecs might have a significant impact on representation learning, as shown by EnCodecMAE [2], so it may be more appropriate to treat semantic representation as a downstream task. WavTokenizer's single codebook could be particularly useful for discrete targets in BERT-like setups.\n\n[1] Zhu, Lei, et al. \"Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99%.\"\n\n[2] Pepino, Leonardo, Pablo Riera, and Luciana Ferrer. \"EnCodecMAE: Leveraging neural codecs for universal audio representation learning.\"", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper effectively motivates and addresses an important problem in neural audio codecs - quantizing audio into a single sequence of tokens rather than multiple sequences, which complicates modeling for downstream tasks.\n- It presents useful findings regarding decoder design, which are supported by the ablation study. In particular, a Fourier-based decoder combined with an attention layer yields better results, while a time-domain decoder with attention performs surprisingly worse.", "weaknesses": "While the motivation for scaling the single VQ codebook to many entries is clear, the paper falls short of achieving high codebook utilization when expanding beyond a size of 4096. What the authors list as contributions in VQ, such as k-means initialization and random restarts, are in fact well-established techniques in neural audio compression, and this paper doesn’t offer any novel methods to improve codebook usage. This is somewhat disappointing, given that a key focus of the paper is to provide a single quantizer. More experimentation to scale the VQ codebook is needed, as the current contribution feels more incremental and may be better suited for a different venue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)