ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Authors: Dongchao Yang
Songxiang Liu
Haohan Guo
Jiankun Zhao
Yuanyuan Wang
Helin Wang
Zeqian Ju
Xubo Liu
Xueyuan Chen
Xu Tan
Xixin Wu
Helen M. Meng
Venue: ICML 2025 poster (2025)
Keywords: Audio Language Models
Audio Codec
Audio Tokenizer
Audio Understanding and Generation
Primary area: applications->language_speech_and_dialog

Abstract: Recent advancements in audio language models have underscored the pivotal role of audio tokenization, which converts audio signals into discrete tokens, thereby facilitating the application of language model architectures to the audio domain. In this study, we introduce ALMTokenizer, a novel low-bitrate and semantically rich audio codec tokenizer for audio language models. Prior methods, such as Encodec, typically encode individual audio frames into discrete tokens without considering the use of context information across frames. Unlike these methods, we introduce a novel query-based compression strategy to capture holistic information with a set of learnable query tokens by explicitly modeling the context information across frames. This design not only enables the codec model to capture more semantic information but also encodes the audio signal with fewer token sequences. Additionally, to enhance the semantic information in audio codec models, we introduce the following: (1) A masked autoencoder (MAE) loss, (2) Vector quantization based on semantic priors, and (3) An autoregressive (AR) prediction loss. As a result, ALMTokenizer achieves competitive reconstruction performance relative to state-of-the-art approaches while operating at a lower bitrate. Within the same audio language model framework, ALMTokenizer outperforms previous tokenizers in audio understanding and generation tasks.\footnote{http://dongchaoyang.top/ALMTokenizer/}

ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Dongchao Yang'
Songxiang Liu?
Haohan Guo
Jiankun Zhao
Yuanyuan Wang' Helin Wang? Zeqian Ju
Xubo Liu
Xueyuan Chen 1 Xu Tan Xixin Wu Helen Meng'
Abstract
1. Introduction
The field of generative modeling has witnessed remarkable progress, largely driven by the success of autoregressive (AR) models in the development of large language models (LLMs) (OpenAI, 2023). Inspired by the success of LLMs in the fields of natural language processing (NLP), recent works have begun to employ AR transformers for audio gen- eration (Borsos al., 2023a; Agostinelli et al., 2023; Yang et al., 2023c), such using the AR transformer paradigm to solve text-to-speech task (Wang et al., 2023), or expanding the text L.MM into multimodal LLM by integrating the audio modality into the original LLM (Défossez et al., 2024). Au- dio tokenizer plays an important role in all of these models, which converts audio signals into discrete token sequence for AR audio language modeling.
Recent advancements in audio language models have underscored the pivotal role of audio tok- enization, which converts audio signals into dis- crete tokens, thereby facilitating the application of language model architectures to the audio do- main. In this study, we introduce AL.ooeenezer a novel low-bitrate and semantically rich audio codec tokenizer for audio language models. Prior methods, such Encodec, typically encode indi- vidual audio frames into discrete tokens without considering the use of context information across frames. Unlike these methods, we introduce a novel query-based compression strategy to cap- ture holistic information with a set of learnable query tokens by explicitly modeling the context information across frames. This design not only enables the codec model capture more semantic information but also encodes the audio signal with fewer token sequences. Additionally, to enhance the semantic information in audio codec models, we introduce the following: (1) A masked au- toencoder (MAE) loss, (2) Vector quantization based on semantic priors, and (3) An autoregres- sive (AR) prediction loss. As result, ALMTT- kenizer achieves competitive reconstruction per- formance relative to state-of-the-art approaches while operating at lower bitrate. Within the same audio language model framework, ALMTo- kenizer outperforms previous tokenizers in audio understanding and generation tasks.
In the literature, audio codec models, such as SoundStream (Zeghidour et al., 2021) and Encodee (Défossez et al., 2022), have been widely adopted as audio tokenizers for audio lan- guage models. These generative models aim to represent audio data in a quantized discrete latent space, where the codec's s decoder is then used to reconstruct the audio sig- nals from the generated discrete token sequences. Recently, there has been significant interest in the audio community regarding audio codec tokenizers, leading to the proposal of several novel models (Kumar et al., 2023; Ji et al., 2024; Défossez et al., 2024; Parker et al., 2024; Zhang et al., 2023). Despite the advancements in audio codec models, an im- portant research question remains unanswered: which type of audio codec most suitable for audio language mod- eling? Inspired by previous works (Borsos al., 2023a; Parker al., 2024; Ji al., 2024; Défossez al., 2024), these studies investigate two key properties of audio codec models: low bitrate and semantic richness. We first conduct a set of evaluation experiments to explore the influence of bitrate and semantic information on audio language model- ing. Specifically, we train three audio codec models with varying bitrates, while keeping the number of vector quan- tization (VQ) layers constant and adjusting the frame rates to 50 Hz, 25 Hz, and 12.5 Hz. We then train the audio lan- guage model using different audio tokenizers on the same dataset. To assess the impact of semantic information, we
'The Chinese University of Hong Kong, Hong Kong, China 'Independent Researchers. Correspondence to: Donghaa Yang <deyang@ flcyang@@s.ccuk..duu .du.k....... Helen Meng <hmmeng @ seseccur sc..c...edu.kk...
Proceedings of the ngnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s). http://konghaaayyng..po//A...od

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
also train 12.5 Hz semantic tokenizer and incorporate it into the audio language model. Further details can be found in Appendix B. Figure 1 presents the results, which show that: (1) low-bitrate audio codec models significantly en- hance training and inference efficiency; and (2) semantic information is more easily modeled by LM-based genera- tive methods, e.g. lower PPL and loss. The experimental findings demonstrate the importance of constructing a low- bitrate and semantic-rich audio codec tokenizer for audio language modeling. Based on these results, we propose novel audio codec tokenizer that offers the following advan- tages: (I) Low-bitrate: it compresses the audio data into fewer tokens; (2) Semantic-rich: it incorporates abundant se- mantic information; (3) AR-driven latent space: it optimizes the latent space for autoregressive (AR) modeling.
Training cost (Hours)
12.5hz semantic
12.5hz codec
25hz codec
50h2 codec
24.0
PPL
Loss
To achieve this objective, propose the following novel techniques: (1) We introduce a novel query-based compres- sion strategy, which uses a set of learnable query tokens to capture holistic information by explicitly modeling the con- text information across audio frames with transformer lay- ers. This strategy effectively takes advantage of the strong modeling capabilities of transformers to achieve better com- pression and semantic modeling. It also enables dynamic control over the compression rate by adjusting the number of query tokens. (2) To enhance semantic richness in the codec model, we introduce a Masked Autoencoder (MAE) loss, which encourages the model to capture more global information. (3) Inspired by previous works (Zhu et al., 2024), we propose the integration of semantic priors into the VQ layer. Specifically, we perform k-means cluster- ing on the pre-trained wav2vec2 (Baevski et al., 2020) and BEATs (Chen et al., 2022b) encoder outputs, using the clus- ter centers to initialize the VQ layer. (4) We observe that AR models struggle to fit the distribution of the residuals in the VQ layers, with token prediction accuracy being notably lower in the second and third VQ layers compared to the first. To address this issue, we introduce an AR prediction loss to optimize the latent space.
Inference cost (RTF)
Figure I. The performance comparison when different types of tokenizer is used for audio modeling. PPL refers to perplexity.
formers and hierarchical modeling techniques to process audio data directly, learning representations that capture both linguistic and acoustic features. Inspired by AudioLM, VALL-E (Wang et al., 2023) and SPEAR-TTS (Kharitonov et al., 2023) formulate the text-to-speech task as an audio language modeling problem: generating an audio token se- quence with the help of an autoregressive transformer. Mu- sicLM (Agostinelli et al., 2023) and MusicGen (Copet et al., 2023) frame the text-to-music task modeling problem. UniSep (Wang et al., 2025) explores using audio LM to solve audio separation tasks with the help of audio tokenizer. Moshi (Défossez et al., 2024), SpiRit- LM (N (gguyen et al., 2025), and GLM4- (Zeng et al., 2024) explore speech-to-speech conversation. Furthermore, audio tokenizers can also be combined with discrete diffu- sion models (Yang et al., 2023d;a; Borsos et al., 2023b; Ju et al., 2024).In In all of these models, the audio tokenizer plays crucial role by transforming audio data into a discrete la- tent sequence, reducing computational demands compared to directly processing the audio signal, and enhancing the effectiveness and efficiency of the generation process.
audio language
To evaluate the effectiveness of the ALMTokenizer, we first compare its reconstruction and semantic performance with previous state-of-thh--a models. Using the same au- dio language model framework, we then demonstrate that ALMTokenizer achieves superior performance in L.M -ba-eded audio understanding and generation tasks, including text- to-speech (TTS), speech-to-text (ASR), audio captioning, text-to-sound, text-to-music, and music captioning.
2.2. Audio Tokenizer
In the literature, both semantic and acoustic tokenizers are widely employed in audio language models. The seman- tic tokenizer is trained using pre-trained self-supervised learning (SSL) models, such as Hubert (Hsu et al., 2021) and WavLM (Chen et al., 2022a). Applying k-means or vector quantization in these models generates semantic to- kens (Zeng et al., 2024; Du et al., 2024; Liu et al., 2024). Previous works (Borsos et al., 2023a) demonstrate that se- mantic tokens are more easily modeled by language models. However, due to the loss of significant acoustic information
2. Related Works
2.1. Audio Language Models
Recently, there has been a growing interest in bridging audio and text through multimodal learning approaches. Models such A (Borsos et al., 2023a) leverage AR trans-
2

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Patchify Module
Token Interleaving
Transformer Encoder
Token Retrieval
Encoder
Interleaving
Retrieval
RVQ
Query
RVQ
Diseriminators
Diseriminators
Decoder
Retrieval
Interleaving
UnPatchiff
Token Retrieval
Transformer
Module
Interleaving
Figure 2. The left part illustrates the framework of the previous audio codec, while the right part provides an overview of the proposed ALMTokenizer. denotes the window size. The details of M..oenzneerr can be found in Section 3.2.
in semantic tokens, they rely on additional decoder to generate high-fidelity waveform, such as a diffusion model (Ho et al., 2020) or now-matching (Lipman et al., 2022). Inevitably, this additional module results increased infer- ence complexity and poorer reconstruction.
validated for non-speech tasks, such sound and music generation. In contrast to MimiCodec, our ALMTokenizer encodes more semantic information across all VQ layers, achieves a lower bitrate, and is designed for both speech and general sound.
Acoustic tokenizer refers audio codec models, trained for acoustic- reconstruction tasks. Audio codecs (Zeghi- dour et al., 2021; Défossez et al., 2022; Yang et al., 2023b; Kumar et al., 2023) have demonstrated exceptional perfor- mance reconstructing high-quality audio. In general, these codec models consist of an encoder, a quantizer, and decoder. Both the encoder and decoder are lightweight, resulting minimal inference costs. Compared to semantic tokens, codec models can support audio, speech, and music domains, and their rich acoustic details mitigate the need for cascading architectures in downstream generative models. Recently, an increasing number of audio codec models have been proposed, focusing on (1) Better reconstruction quality, such DAC (Kumar al., 2023), Vocos (Siuzdak, 2023), SQ-Codec (Yang et al., 2024c:b) and APCodee (Ai al., 2024); (2) Low-bitrate models, such HiFiCodec (Yang et al., 2023b), wavtokenizer (Ji et al., 2024), StableCodec (Parker et al., 2024), and TS3-Codec (Wu et al., 2024); (3) Task-driven codecs, designed for text-to-speech tasks, such as FACodec (Ju et al., 2024), SokechTo (Zhang et al., 2023), Single-Codec (Li al., 2024), audio retrieval-based Tokenizers (Banerjee & Arora. 2022; van Nickerk et al., 2024). In this study, focus developing low-bitrate, semantically rich audio codec tokenizer. The most closely related work to ours is MimiCodec (Défossez et al., 2024), which provides high-quality semantic information while achieving a low bitrate (1.1 kbps). However, MimiCodec relies on knowledge distillation from WavL. (Chen al., 2022a) the first VQ layer, whereas the remaining VQ lay- do not incorporate semantic information. Furthermore, it is specifically designed for speech tasks and has not been
3. Proposed Method
This section introduces the technical details of the proposed ALMTokenizer. Section 3.1 presents the framework of pre- vious audio codee models. Section 3.2 presents the details of proposed audio codec framework. In Sections 3.3 and 3.4, present the training loss and training strategies.
3.1. Preliminary
Previous audio codecs (Défossez et al., 2022; Zeghidour et al., 2021) typically adopt an encoder-quantizer-eeooll framework. shown in the left part of Figure 2. The audio is encoded into several audio frames by the encoder. Then, residual vector quantization (RVQ) (Zeghidour et al., 2021) is used to quantize these audio frames. Lastly, the decoder is used to recover the waveform from the quantized audio frames. It can be observed that previous works treat each audio frame equally and rely on these quantized frames to recover the audio. However, such strategy (1) ignores the fact that different audio frames encode different levels of information, which results in some audio frames being difficult to recover in low-bitrate settings (c.g., encoding the audio frames at 12.5 Hz); (2) fails to utilize the context information between different frames.
3.2. Query-based Audio Compression
To construct a low-bitrate, semantically rich audio codec model, we propose a query-based compression strategy. Our
3

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
approach is inspired by the success of MAE (He et al., 2022), which applies masking operation the original image with high mask rate (75%). With the help of a transformer encoder and decoder, it is possible to recover the masked image content by utilizing the context information between different patches. Thus, we propose using a group of query tokens 2 to capture holistic audio context information from the audio frames with the assistance of transformer en- coder. Since these query tokens include rich context infor- mation, it is possible to reconstruct the audio based on them. Then, a transformer decoder and mask tokens are employed to reconstruct the audio from the quantized query tokens. This strategy leverages the powerful modeling capabilities of transformers to achieve better compression and semantic modeling. Similar query-based strategies has been widely explored in previous works, such as BLIP2 (Li et al., 2023), SALMONN (Tang et al., 2024) and TiTok Yu et al., 2024). The right part of Figure 2 illustrates the overall framework of ALMTokenizer. In the following sections, we detail each component and the associated training loss.
use it to retrieve the learnable query tokens. In the decoder part, use it to retrieve the learnable mask tokens.
Query-based Transformer Encoder As the previous part discussed, we introduce a learnable query token [c1s] E Rixd to capture holistic information from the audio frames e. As Figure 2 shows, we first combine the audio frames and query token using a token interleaving module with a window size w. Then, a transformer module is applied to model the whole sequence Ca. After that, we employ token retrieval module to extract the query tokens h E R [TT//]] xd
e - P())) Ea =Iterreavingg(e,c cls, w), Ca = En(ea), h Rectrieval(ea, w)
(1)
where P(-) denotes the Patchify module. En(-) denotes the transformer encoder.
Residual Vector Quantization To build low-bitrate audio codec, we empirically set the number of RVQ layers to 3, since we found that 3 RVQ layers suffice to build an effec- tive audio codec model: h- = Qh).. Inspired by previous works (Zhu et al., 2024; Yang et al., 2024a), we first obtain the k-means clusters of Wav2vec2 (Baevski et al., 2020) to represent the speech semantic prior, and the k-means clusters of the BEATs (Chen et al., 2022b) to represent the general sound semantic prior Assuming the codebook size is C, we set C/2 to represent speech, with the remaining portion representing general sound. We then use these se- mantic priors to initialize the codebook of the VQ layer and fix it. Next, we apply a linear layer to map the input features into the VQ layer.
Patchify and UnPatchify We explore two types of Patchify modules: (1) Following Encodec (Défossez et al., 2022), a convolution-based module, which encodes the audio data into E RTxd where T and d denote the number of frames and the vector dimension, and (2) Following Sta- bleCodec (Parker et al., 2024), which directly uses a linear layer to encode the audio data into e E R'x and adds sev- eral transformer layers. Similarly, the UnPatchify mirrors the architecture of Patchify. If we use the Encodec-style Patchify module, the UnPatchify module substitutes stride convolutions with transposed convolutions and reverses the stride order. If we use the StableCodec-style Patchify mod- ule. the UnPatchify module includes a transformer block and reshape operation. In our preliminary experiments, we find that the Encodec-style Patchify and UnPatchify mod- ules bring better reconstruction performance. We adopt the Encodec-style Patchify module as our default setting.
Query-based Transformer Decoder To recover the au- dio information, we construct reverse process using the encoder part. We first use the token interleaving module to combine the mask token m E R with h. The new sequence is then modeled by a transformer module. We expect that these mask tokens can be used to recover the audio information with the help of the Unpatchify module.
Token Interleaving The token interleaving module aims to combine two token sequences into single sequence. In the encoder part, we combine the audio frames e E RT'xd xd and the query token [CLS]. Assuming a window size of w, the query token will be inserted into the audio frame sequence at every w-intervals. In the decoder part, the token interleaving module is used combine the quantized query tokens and learnable mask tokens. We insert mask tokens before each query token. During the training stage, we dynamically choose the window size for each training iteration.
Qa = Interleavinggh, m, w), Qa -D(gg)) Rectrievallga. w), a = UnP(eo),
(2)
where Unp(() denotes the Unpatchify module. De(+) de- notes the transformer decoder.
3.3. Training Loss
Similar to previous audio codecs, our approach is based on GAN objective, where we optimize both the generator (which consists of the Patchify module, transformer encoder, quantizer, transformer decoder, and UnPatchify module) and the discriminators. For the generator, the training loss comprises four components: (1) reconstruction loss term; (2) adversarial loss term; (3) Masked AutoEncoder (MAE) loss; and (4) AR prediction loss. The reconstruction and
Token Retrieval The token retrieval module aims to retrieve the relevant tokens from sequence. In the encoder part, we
2 Query tokens are learnable embedding vectors that are updated throughout the training process.
4

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
adversarial losses typically follow previous works (Défossez et al., 2022; Zeghidour et al., 2021). In the following, we describe the MAE loss and AR prediction loss. More details of training loss refer to Appendix G.
tic information focuses on modeling global relationships. These two goals are in conflict. To resolve this conflict, we present a two-stage training strategy. In the first stage, we do not incorporate the quantization part, instead, we train directly an AutoEncoder with Patchify and UnPatchify mod- ules. To encode more semantic information in the Patchity module, we introduce MAE loss during this stage, by adding transformer-basee MAE-encoder and decoder. The encoder processes the masked frame sequence, and the decoder pre- dicts the masked part. After training, the transformer en- coder and decoder are discarded. In the second stage, we first initialize the ALMTokenizer's 's Patchify and UnPatchify modules with the checkpoint from the first stage, and freeze the parameters of the Patchify module. Then, we train the model using the training loss described in Section 3.3.
MAE Loss As we discussed in Section 1, a semantic-rich audio codec tokenizer is better suited for audio language modeling. Inspired by the success of MAE (He et al., 2022), we propose to incorporate an MAE loss during the training of the audio codec. Specifically, for the frame sequence e, we randomly choose several audio frame features and set these frames to zero, em = Mask(e). We pass the masked features c into the encoder transformer. Then, the encoded features are passed into an MAE-decoder transformer block to predict e. In our experiments, we adopt a dynamic mask rate (from 0.2 to 0.3). we found that using large mask rate will significantly influence the reconstruction performance. Following MAE (He et al., 2022), we apply the MSE loss to the masked audio frames.
4. Experiments
4.1. Dataset and Training Details
AR Loss As shown in figure 3, we find that the first layer of RVQ-based audio codec models is casier to fit for the audio language model than the other layers (e.g., layers 2 and 3). One possible reason is that the first layer encodes more semantically related information. For speech data, most of the content information can be recovered by the first VQ layer, while the residual layers primarily encode acoustic-level information, which influences speech quality. To make the tokens in the residual layer casier to fit, we in- troduce an autoregressive (AR) prediction prior (Wang et al., 2024a) in the RVQ latent space. Specifically, we introduce a lightweight continuous autoregressive (AR) transformer 3, which is used to conduct next-token prediction in the RVQ layer. For example, it is tasked with predicting the quantized feature of the third VQ layer based on the features of the first and second VQ layers. We use mean squared error (MSE) loss for optimization.
Data preparation for the audio codec ALMTokenizer is trained on approximately 4,500 hours of data. In the speech domain, we utilize LibriTTS training set (Zen et al., 2019) and subset of Multilingual LibriSpeech (MI.S) (Pratap et al., 2020), with 2,000 hours randomly selected. In the sound domain, we utilize a subset of AudioSet, with 1,000 hours randomly selected; in the music domain, we employ subset of the Million Song Dataset (Bertin-Mahiee et al., 201 I), also with 1,000 hours randomly selected. We eval- uate the codec's ' speech reconstruction performance using subset of the VCTK dataset (Veaux et al., 2017), and as- sess both audio and music reconstruction performance using the AudioCaps (Kim et al., 2019) validation set and the MusicCaps dataset (Agostinelli et al., 2023), respectively.
Data for Audio Language Models To assess the effective- ness of the proposed audio tokenizer, we construct an audio language model framework to perform six audio-related tasks. The details are provided in Appendix D.3 and D.4. For speech data, we select 2,000 hours of speech-text pairs from LibriHeavy (Kang al., 2024). For sound data, we utilize the AudioCaps training set and BBC Sound Effects. For music data, we use subset of the Million Song dataset and the caption data from LP-MusicCaps (Doh et al., 2023).
II po 2..1:;
(3)
where 0 denotes the parameter of AR transformer.
3.4. Two-stage Training Strategy
Although training the ALMTokenizer using the typical En- codec (Défossez et al., 2022) setting is feasible, we intro- duce a two-stage training paradigm to improve both recon- struction performance and semantic information. Our mo- tivation stems from the fact that audio codec quantization focuses on modeling local relationships, whereas seman-
Implementation Details ALMTokenizer first performs patchification on the audio data, we set the patch size to 320 in all of experiments, which encodes I second of 24kHz audio into 75 frames. For the Encodec-style Patchify mod- ule, we adopt the settings from Encodec (Défossez et al., 2022) encoder. To enable streaming for the codec model, a causal convolution layer is employed. For the encoder- transformer and transforre components, we use 24 self-attention layers, with latent dimensions of 256 and 512, respectively. Following StableCodec (Parker al.,
'The term continuous autoregressive (AR) transformer is used to distinguish our approach from traditional discrete AR models, which operate on discrete token sequences and are optimized using cross-entropy loss. In our study, to facilitate gradient backpropaga- tion, we apply the AR transformer directly to continuous features.
5

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table The speech reconstruction and semantic performance comparison between the MrMTokenizer and previous tokenizers. FPS
denotes that the frame number in one second. TPS denotes that the token number in one second. CS denotes the codebook size, BR denotes the bit-rate. ST denotes speechtokenizer. Bold best result underline for the second best result. Evaluation VCTK dataset.
Reconstruction
Semantic
Models
FPS/TPS
CS/BR
UTMOS (1)
DNS-MOS
VISQOL (t)
STOI
PESQ (f)
ASR (4)
ER (t)
Huhert al., 2021)
WayLM (Chen et al., 2022a)
6.2
29.0
Encodec (Défossez al., 2022)
50/150
1024/1.5khyg
2.58
3.27
3.64
0.81
2.0
35.3
26.5
DAC (Kumar et al., 2023)
50/150
1024/1.5kbrr 5kbpps
3.13
3.41
3.67
0.81
2.1
44.1
17.6
Wavtokenizer (Ji ul., 2024)
40/40
48kbps
3.67
3.50
3.72
1.9
44.6
19.8
StableCodec (Parker al., 2024)
25/25
46656/0 4kbps
4.22
3.64
3.40
0.76
98.3
15.8
ST (Zhang 2023)
50/150
1024/1.5kbpe
3.41
3.36
3.68
0.79
19.8
Mini (Défossez et al., 2024)
12.5/37.5
2048/0.41kbe
3.01
3,14
3.28
0.75
1.5
25.1
28.0
Mimi (Défossez et al., 2024)
12.5/100
2048/1 20bb/1.1kbps
3.65
3.38
3.82
0.82
2.1
23.8
28.3
ALMTokemizer (Ours)
12.5/37.5
2048/0.41kbps
3.76
3.64
3.78
0.81
2.0
18.3
29.0
Table 2. The sound reconstruction peformance comparison be- tween the proposed ALMTokenizer and previous audio tokenizer models. SC denotes the sound classification task. Evaluation on AudioCaps validation set.
2024), the self-attention mechanism uses a causal sliding at- tention window of 64 steps to restrict the receptive field and promote the generalization of the architecture to sequences of arbitrary length. Rotary Positional Embeddings (RoPE) used. Refer to Appendix G for the details of ALMTo- kenizer model training. For the audio language model, we follow the framework of Moshi (Défossez et al., 2024). For further details, refer to Appendix A.
Models
ViSQOL (1)
Mel loss ())
STFT loss (3)
SC (1)
BEATs
24%
Wav2vec2
53%
Encodee
3.05
16.3
1.23
15%
DAC
2.98
17.6
1.24
20%
Wavtokenizer
2.18
32.7
12%
Ours
2.99
15.0
1.24
44%
4.2. Evaluation Metrics
We evaluate the performance of previous SOTA audio to- kenizers, and our proposed ALMTokenizer across audio reconstruction, audio semantic information, audio under- standing, and audio generation tasks.
Table 3. The music reconstruction and semantic performance com- parison between the ALMTokenizer and previous audio tokenizers. MC denotes the music classification task. Evaluation on Musicaps dataset.
Audio Reconstruction For speech reconstruction, we DNS-MOS, UT-MOS, PESQ, STOI (Short-time Objective Intelligibiliity) and VISQOL. For sound and music data evaluation. VISQOL (audio version), STFT loss, and Mel loss are used. Furthermore, following (Kumar et al., 2023), the MUSHRA subjective test conducted for speech, sound, and music. Refer Appendix for more details.
Models
ViSQOL (1)
Mel loss (1)
STFT loss (1)
MC (t)
BEATs
54%
Wav2vec2
Encodec
4.04
34.8
1.26
45%
DAC
4.06
35.9
1.28
48%
Wavtokenizer
3.85
48.2
1.47
54%
Ours
3.96
34.4
1.32
59%
Audio Semantie Information Previous SSL models, such Hubert (Hsu et al., 2021), have shown that semantic-rich representation can be used to solve downstream recognition tasks by fine-tuning several adaptor layers. Thus, we can validate the performance of features of the audio tokenizer for downstream recognition tasks. For speech data, we conduct the automatic speech recognition (ASR) task on the LibriSpeech (Panayotov al., 2015) dataset, and the emotion classification (EC) task on the EMOVO (Costantini et al., 2014) dataset. For sound data, we conduct sound classification tasks on the ESC-50 dataset (Piczak, 2015). For music data, we conduct music classification tasks on the Medley-solos- DD dataset (Lostanlen & Cella, 2016).
tokenizer is suitable for building an audio language model, we propose to conduct an understanding task using discrete tokens. We conduct three tasks: ASR, audio caption, and music caption. For the audio data, we use the audio to- kenizer to transform into discrete tokens, and for text data, we use the BPE tokenizer of LLAMA 3.2. For audio and music caption, follow (Drossos al., 2020) and adopt BLEU-1, BLEU-2, BLEU-3, METEOR, ROUGE-L. CIDEr-D, SPICE, and SPIDEr metrics.
Audio Generation including text-to-speech, text-to-se text-to-sound, and mext-to-music. Refer to Appendix D for more details.
We also conduct audio generation tasks,
Audio Understanding To further validate whether the audio
6

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
4.3. The Reconstruction and Semantic Performance
Table 4. The L.M-based TTS and ASR results. The first three
We first compare the reconstruction and semantic perfor- mance of ALMTokenizer with previous audio tokenizers. Table 1 presents the speech reconstruction and semantic results. We observe the following: (1) In terms of recon- struction, ALMTokenizer achieves impressive results in the low-bitrate setting. For example, compared with previous SOTA models, MimiCodec and Wavtokenizer, ALMTok- enizer achieves better reconstruction performance at lower bitrate. We also note that StableCodec performs well UT- MOS. The main reason is that StableCodec has denoising capabilities, while the original audio includes some noise. This explains why StableCodec achieves good results on UT- MOS but performs poorly on PESQ and STOI. (2) In terms of semantic information, ALMTokenizer demonstrates supe- rior performance, e.g, ALMTokenizer outperforms previous SOTA models, such Wavtokenizer and StableCodec 4. Notably, in the emotion classification task, ALMTokenizer achieves performance comparable to previous SSL models, such as Hubert and WavLM. However, we also note that ALMTokknizer still lags behind these SSL models in ASR performance. We speculate that the inclusion of acoustic information may detract from ASR performance, despite ALMTokenizer containing rich semantic information. Table 2 and 3 show the sound and music experimental results. We can see that ALMTokenizer demonstrates strong reconstruc- tion performance under the low-bitrate setting. Compared to Wav Wavenniiz,,,, the reconstruction performance shows significant improvement. Furthermore, we also note that sound and music are inherently more complex than speech, and encoding them at very low-bitrate remains a challenge. In terms of semantic information, ALMTokenizer signifi- cantly surpasses previous works, such WayTokenizer and Encodec. In comparison with SSL models, BEATs (Chen et al., 2022b) and Waw2vec2-audioset version, ALMTTo-- enizer shows comparable performance. We also perform the MUSHRA subjective test for the reconstruction perfor- mance. As shown in Table 7, we find that ALMTokenizer effectively maintains strong subjective reconstruction per- formance speech, music, and audio, even with very low-bitrate setting.
metrics are used for TTS, while the last one is used for ASR.
LLLM4-oiice (Zeng et al., 2024) is single layer semantic tok- enizer. Evaluation on LibriSpeech test clean set.
Models WER (L)
DNSMOS (1)
UT-MOS (1)
ASR (J)
GLM4-voice
9.9
3.96
3.79
16.3 = 1.5
DAC
24.5
3.14
2.06
58.4 :: 1.2
Encodec
22.9
3.48
2.14
77.2 + 2.3
StableCodec
22.7
3.63
3.70
28.0 + 1.9
Wavtokenizer
18.5
3.72
3.58
45.6 + 2.7
MimiCodec
16.0
3.67
2.93
23.1 + 1.5
Ours
11.7
3.75
3.88
19.6 + 1.8
w/o loss
30-
20.30
w AR loss
19.20
15-
15.20
30
9.50
6.40
3
3.75
ACC llaver 1)
ACC Oayer 3)
dayer 3)
DNS MOS (TTS)
WER (T7S)
Figure 3. The performance comparison with or without AR loss.
for LM-based speech generation. Compared to previous audio codec tokenizers, ALMTokenizer brings significant improvement. In terms of generated speech quality, ALMTo- kenizer also shows great advantages, further demonstrating that the proposed tokenizer is more suitable for audio lan- guage modeling. Similarly, when we conduct the ASR task using discrete tokens as input, semantic information is also important. Traditional audio codec models perform poorly in this setting, such as DAC, Encodec, and WayTokenizer. StableCodec was fine-tuned by using a CTC head to pre- dict the force-aligned phoneme tags from pre-bottleneck latents. MimiCodec distills the semantic information from WavLM. Thus, they have better performance than previ- ous codec models. In ALMTokenizer, we propose novel codec framework and training loss to better encode semantic information in the codec model.
4.4. Audio Understanding and Generation Results
Speech Understanding and Generation Tasks Table 4 shows the LM-based TTS and ASR results. For the TTS task, we mainly focus on robustness and speech quality. In terms of robustness, we can see that the GLM4-voice tokenizer (Zeng et al., 2024), MimiCodec, and the pro- posed ALMTokenizer bring better performance than oth- ers, highlighting the importance of semantic information
Sound/music Understanding and Generation Results We conduct text-to-sound, text-to-music. audio caption and mu- sic caption tasks within the same audio language model framework. The experimental results shown in Table 5 indi- cate that ALM'Tokenizer shows better performance in both audio caption and audio generation tasks, further demon- strating its advantages. We put more audio tokenizer recon- struction performance experiments on Appendix F, includ- ing evaluation on LibriTTS test set, length generalization,
'StableCodec's feature dimension is 6, is hard to apply it for down-streaming task by simple fine-tuning
7

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table The I. L.M-hased sound, music understanding and generation. B1, B2, B3, RG, ME, CD, SP, and SD denote BLEU-1, BI.U-22, BLEU-3, METEOR, ROUGE-L. CIDEr-D, SPICE, and SPIDEr, respectively. Evaluation on Audiocaps and Musicaps datasets.
Understanding
Generation
Models
B1 (1)
B2(1)
(t)
ME (1)
RG (t)
CD (t)
SP (t)
(D
HD
FAD ())
KL. (L)
Sound Task
Encodee
0.25
0.15
0.08
0.11
0.24
0.57
0.14
0.35
10.03
8.22
1.73
DAC
0.26
0.15
0.08
0.11
0.26
0.51
0.13
14.14
11.7
1.55
Wavtokenizer
0.24
0.14
0.08
0.10
0.22
0.38
0.11
0.25
6.76
4.55
1.28
ALMTooknizer (Ours)
0.28
0.17
0.11
0.12
0.24
0.60
0.15
0.37
4.11
6.16
0,55
Music Task
Encodee
0.30
0.14
0.08
0.11
0.23
0.37
0.09
0.23
7.22
5.48
1.06
DAC
0,29
0.14
0.08
0.11
0.23
0.37
0.09
0.23
12.89
8.36
1.68
Wavtokenizer
0.19
0.06
0.02
0.06
0.13
0.06
0.05
0.05
4.39
11.93
0.88
ALMTokenizer (Ours)
0.34
0.15
0.07
0.13
0.25
0.44
0.10
0.27
3.55
4.58
0.43
and compared to diffusion-based audio codec models.
shows the effectiveness of our motivation and solution. The Influence of Two-stage Training As Table 6 shows, the two-stage training strategy crucial as it significantly improves reconstruction performance and semantic informa- tion in the codec model. The Influence of Patchify Mod- ule We investigate two types of Patchify modules: Encodec- style and StableCodec-style. As shown in Table 6, using Encodec-style Patchify modules yields better performance. One possible reason is that StableCodec-style Patchify mod- ules (Parker et al., 2024) may depend on larger data and model parameters, as the original paper scales their model to IB. In contrast, we use only four transformer layers to ensure fair comparison with Encodec-style modules. Due page limitations, we defer the ablation study on the influence of window size in query-based compression, codebook size, the influence of mask-rate, and model size on reconstruction to Appendix C.
4.5. Ablation Study
In order gain more comprehensive understanding of ALMTokenizer, we systematically compared each key com- ponent using controlled experimental setup, employing identical architectures and hyperparameters across all trials. The Effectiveness of Query-based Audio Compression In this study, we propose a query-based based audio compression strategy for compressing audio data in very low-bitrate setting. To validate its effectiveness, we follow previous audio codec models, such MimiCodec (Défossez et al., 2024). In the encoder part, we use stride size of [8, 6, 5, 4, 2] to compress -second, 24 kHz audio into 12.5H7, followed by applying 3 RVQ layers to quantize it. As shown in Table 6, using previous audio codec frameworks makes difficult to maintain good reconstruction performance in very low-bitrate settings. As result, the proposed query- based compression method is more effective in this setting. The Influence of Semantic Prior for VQ To explore the influence of semantic priors on the audio codec model, we conduct an experiment where we remove the semantic prior and instead train a learnable RVQ following Encodec. As shown in Table 6, we find that updating the RVQ layer im- proves reconstruction performance but reduces semantic information, demonstrating that integrating semantic priors into the VQ layer enhances semantic information.
4.6. Discussion
In this section, we discuss two fundamental questions in audio tokenization. Question 1: Is a single quantization layer better than multiple quantization layers? Question 2: Does a low-bit rate with high reconstruction perfor- mance define good audio tokenizer?
Question 1 Although Tokenize and StableCodec demonstrate the potential to build a low-bitrate audio codec tokenizer with single quantization layer, they rely on higher frame rate (e.g., 25 or 40 Hz). As shown in Figure I, a lower frame rate (e.g.. 12.5 Hz) is critical for improving training efficiency. Thanks to UniAudio (Yang et al., 2023c) and Moshi's 's (Défossez et al., 2024) audio language model framework, multiple quantization layers do not increase the sequence length. Therefore, multiple quantization layers present effective approach for building a low-bitrate, se- mantically rich audio codec.
The Influence of MAE Loss We also conduct experiments to evaluate the effectiveness of the MAE loss. As shown in Table 6, we find that the MAE loss is crucial for enhancing the semantic information the codee model. Although the MAE loss has slight negative effect on reconstruction, it crucial factor in building a better audio tokenizer.
The Influence of AR Loss From Table 6, we observe that adding the AR loss reduces reconstruction performance. In Figure 3, we compare token prediction accuracy and TTS performance with and without LM loss. We observe that using LM loss significantly improves token prediction accu- racy, particularly for the second and third VQ layers, which
Question 2 To address this question, we present two com- parisons. First, as shown in Tables 4 and 1, StableCodec ex-
hibits better reconstruction performance and a lower bit-rate compared WavTokenizer. However, when applied the
8

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table Ablation study of codee framework, training loss, and training strategy. ASR and ER used evaluate the semantic information.
The others are used to evaluate the reconstruction performance. Experiments conducts VCTK dataset.
Setting
UTMOS (t)
DNSMOS (1)
VISQOL (1)
PESQ
STOI (1)
ASR (L)
ER (1)
ALMTokenizer
3.76
3.64
3.78
2.0
0.81
18.3
29.0
Framework ablation
w/o the query-based framework
2.49
3.13
3.37
1.58
0.77
34.5
22.6
Only query-based framework
3.54
3.41
3,44
1.69
0.78
27.2
24.5
Training loss ablation
w/o semantic prior for VQ w/o MAE loss
3.79
3.66
3.78
2.12
0.83
19.2
28.4
3.70
3.76
3.83
2.10
0.82
24.5
23.2
w/o AR loss
3.72
3.81
3.80
2.08
0.82
18.8
30.2
Different Patchify module use Linear-Pates Patchily
3.47
3.36
3.27
1.78
0.78
20.3
26.7
Training strategy ablation w/o two-stage training
3.60
3.39
3.24
1.55
0.74
22.8
25.9
Table The subjective reconstruction results using MUSHRA (comparative scoring of samples) of codec models on speech, sound and music. Bold for the best result and underline for the second-best result.
Models
FPS/TPS
CS/BR
Speech (t)
Sound (1)
Music (t)
Speech
MiniCodec (3 RVQ) (Défossez et al., 2024)
12.5/37.5
2048/0.41kbps
65.61 + 5.2
MimiCodec (8 RVQ) (Défossez et al., 2024)
12.5/100
2048/1.
86.7 2.3
StableCodec (Parker al., 2024)
25/25
46656/0.4kbpp
81.7 4.4
SpeechTokenizer (Zhang et al., 2023)
50/150
1024/1.5hps
73.7 4.6
Audio
Encodec (Défossez et al., 2022)
50/150
1024/1. 1024/1.5bps
3.9
77.2 4.2
73.7
DAC (Kumar el al., 2023)
50/150
1024/1.5bps
79.3 1: 4.2
71.3 1 4.1
71.3 + 4.1
Wavtokenizer (Défossez et al., 2022)
40/40
4096/0.48bps
84.0 2.1
63.1 +
54.1 14.155 5.4
Ours
12.5/37.5
2048/0.41kbps
84.8 3.7
72.4 - 4.7
69.0 + 4.5
text-to-speech generation task, Wav demonstrates better robustness. One possible reason for this is that Sta- bleCodec uses large-scale codebook size (46,656), which may increase the modeling complexity. Second, although MimiCodec has higher bit-rate and poorer reconstruction performance than StableCodec, it demonstrates more stable TTS generation performance and better ASR performance. This phenomenon further underscores the importance of semantic information. In summary, good audio tokenizer for an audio language model should not only consider low- bitrate and reconstruction, but also account for the semantic information in the codec model.
effectivvnesss of ALMTokenizer. Within the same audio language modeling framework, ALMTokenizer exhibits su- perior performance both understanding and generation tasks. We discuss the limitation of this study in Appendix I.
Impact Statement
This paper presents audio tokenizer for audio language models, which can be applied to various audio generation tasks, such as text-to-specch and text-to-music. There is potential for misuse in generating misinformation, deepfake audio, or other harmful content. We advocate for the devel- opment of detection model identify audio produced by the codec model and generated by other generative models.
5. Conclusion
In this study, we present low-bitrate, semantically rich au- dio codec tokenizer. Specifically, we propose a query-based compression strategy to effectively compress the audio data into a low-bitrate format while incorporating more semantic information. Furthermore, we introduce several training losses enhance semantic information, including MAE loss and AR loss. Extensive experiments demonstrate the
A Acknovledgements
This study was supported in part by the Centre for Percep- tual and Interactive Intelligence (CPII Ltd., CUHK-led InnoCentre under the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government.

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
References
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y.
High fidelity neural audio compression. arXiv preprint 2xXiv:22001344:. 2022.
Agostinelli, A., Denk, T. L., Borsos, Z., Engel, J.. Verzetti, M.. Caillon, A., Huang, Q. Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301. 1132, 2023.
Défossez, A., Mazaré, L.., Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., and Zeghidour, N. Moshi: speech-text foundation model for real-time dialogue. arXiv preprint urXiv:2410.0003: 2024.
Ai, Y., Jiang, X.-H., Lu, Y.-.,, Du, H.-P., and Ling, Z.- H. Apcodec: A neural audio codec with parallel ampli- tude and phase spectrum encoding and decoding. arXiv preprint arXiv:2402. 10533, 2024.
Doh, S., Choi, K., Lee, J., and Nam, J. Lp-musiccaps: Llm-based pseudo music captioning. arXiv preprint vrXiv:2307.16372 2023.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self supervised learning of speech representations. Advances in neural information process- ing systems, 33:1 33:12449-11 11144-11260, 2020.
Drossos, K., Lipping. S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Banerjee, A. and Arora, V. wav2tok: Deep sequence tok- enizer for audio retrieval. In The Eleventh International Conference on Learning Representations, 2022.
Du, Z., Chen, Q., Zhang, S., Hu, K., Lu, H., Yang, Y., Hu, H., Zheng. S., Gu, Y., Ma, Z., et al. Cosyvoice: scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.0542 2024.
Bertin-Mahieux, T., Ellis, D. P., Whitman, B., and Lamere, P. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011), 2011.
Hao, H., Zhou, L., Liu, S., Li, J., Hu, S., Wang. R., and Wei, F, Boosting large language model for speech synthesis: An empirical study. arXiv preprint 1rXiv2240.00224 2023.
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grang- ier, D., Tagliasacchi, M., al. Audiolm: language mod- cling approach audio generation. IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 2023a.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghi- dour, N., and Tagliasacchi, M. Soundstorm: Efficient par- allel audio generation. arXiv preprint arXiv:2305.0665 09636 2023b.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840 116664--851,,,, 2020.
Chen, S., Wang, C., Chen, Z. Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Large- scale self- supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1005-151, 2022a.
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3460, 2021.
Chen, S.. Wu, Y., Wang, C., Liu, S., Tompkins, D., Chen, Z., and Wei, F. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09052 2022b.
Hu, E. J.. Shen, Y., Wallis, P., Ahuu Z., Li, Y., Wang, S., Wang, L.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 106609655. 2021.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G. Adi, Y., and Défossez, A. Simple and controllable mu- sic generation. arXiv preprint arXiv:2306.0528.. 2023.
Huang. P.-Y,, Xu, H., J., Baevski, A., Auli, M., Galuba, W., Metze, F., and Feichtenhofer. C. Masked autoen- coders that listen. Advances in Neural Information Pro- cessing Systems, 35:28708-28720, 2022.
Costantini, G., Iaderola, I., Paoloni, A., Todisco, M., al. Emovo corpus: an italian emotional speech database. In Proceedings of the ninth international conference on language resources and evaluation (LREC' 14), pp- European Language Resources Association (ELRA), 2014.
Ji, S., Jiang, Z., Wang, W., Chen, Y., Fang, M., Zuo, J., Yang, Q., Cheng, X., Wang, Z., Li, R., et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio
10

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
language modeling. arXiv preprint arXiv:2408. 2024.
Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv: 2022.
Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., Liu, Y., Leng, Y., Song, K., Tang, S., et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and dif- fusion models. arXiv preprint arXiv:2403.03106 2024.
Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., and Plumb- ley, M. D. Semanticodec: An ultra low bitrate se- mantic audio codec for general sound. arXiv preprint SeGiv:2405.002335 2024.
Kang, W., Yang, X., Yao, Z., Kuang, F., Yang, Y., Guo, L., Lin, L., and Povey, D. Libriheavy: a 50,000 hours ast cor- pus with punctuation casing and context. In ICASSP 2024-2024 IEEE International Conference on Acous- fics, Speech and Signal Processing (ICASSP), pp. 10991 10995. IEEE, 2024.
Lostanlen, V. and Cella, C.-F. Deep convolutional networks on the pitch spiral for musical instrument recognition. arXiv preprint arXiv: 06644,0644, 2016.
Mei, X., Meng, C., Liu, H., Kong, Q.. Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Waveaps: audio captioning dataset for audio-language multimodal research. arXiv preprint arXiv:2303 2023.
chatgpt-assisted weakly-
Kharitonov, E.,, Vincent, D., Borsos, Z., Marinier, R., Gir- gin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. arXiv preprint arXiv:2302.03540. 2023.
Nguyen, T. A., Muller, B., Yu, B., Costa-Jussa, M. R., El- bayad, M., Popuri, S., Ropers, C., Duquenne, P.-.,, Al- gayres, R., Mavlyutov, R., et al. Spirit-Im: Interleaved spoken and written language model. Transactions of the Association for Computational Linguistics, 13:30-52, 2025.
Kim, C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long and Short Papers), pp. 119- 132, 2019.
OpenAL. Gpt-4 technical report. arXiv:2204.06/25 2023.
arXiv preprint
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. arXiv preprint arXiv:2209.15352 2022.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain au- dio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206 5210. IEEE, 2015.
Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., and Kumar, K. High-fidelity audio compression with im- proved RVQGAN. In seirth-seventh Conference on Neu- ral Information Processing Systems, 2023. URL https: http///orummrdeeeee //openreview.o.mmmmmmrr/oo htwp:eeeeeooommmdddinn1n1o11
Parker, J. D., Smirnov, A., Pons, J., Carr, C., Zukowski, Z., Evans, Z., and Liu, X. Scaling transformers for low-bitrate high-quality speech coding. arXiv preprint arXiv:241 1. 19842, 2024.
Piczak, K. J. Esc: Dataset for environmental sound classi- fication. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
La Quatra, M., Koudounas, A., Vaiani, I.., Baralis, E., Cagliero, I.., Garza, P., and Siniscalchi, S. M. Bench- marking representations for speech, music, and acoustic events. In 2024 IEEE International Conference on Acous- tics, Speech, and Signal Processing Workshops (ICAS- SPW), pp. 505-509 2024. doi: 10.1 10.1109/CASSPPW6444. 2024.10625964
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:20/2.0344 2020.
Reddy, C. K., Gopal, V., and Cutler, R. Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2022-2022
Li, H., Xue, L.,, Guo, H., Zhu, X., Lv, Y., Xie, I.,, Chen, Y., Yin, H., and Li, Z. Single-codec: Single-codebook speech codec towards high-performance speech genera- tion. arXiv preprint arXiv:2406.07422 2024.
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
IEEE, 2022.
Li, Li, D., Savarese, S., and Hoi, S. 2lip-2: Bootstrapping language-ing -iegeemaeeeee pre- vreitrann with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742, PMLR, 2023.
Sacki, T., Xin, D.,, Nakata, W., Koriyama, T., Takamichi, S., and Saruwatari, H. Utmos: Utokyo-sarulab sys- tem for voicemos challenge 2022. arXiv preprint arXiv:2204.22/52, 2022.

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Siuzdak, H. Vocos:
Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.0 2300000144, 2023.
Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J., Zhao, S., Bian, J., Wu, X., et al. Uniaudio: An au- dio foundation model toward universal audio generation. arXiv preprint wrXiv:2310.00700 2023c.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, I.., MA. Z., and Zhang, C. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL //tp::://///reteeee/ee www.wrewwwwwwwwweeeeeeeeeeeeeee id-14rn7HpKVk.
Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text- to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023d.
Yang, D., Guo, H., Wang. Y., Huang, R., Li, X., Tan, X.,
Wu, X., and Meng, H. Uniaudio 1.5: Large language model-driven audio codec is a few-shot audio task learner. arXiv preprint arXiv:2406. 10056, 2024a.
van Nickerk, B., Zaidi, J., Carbonneau, M.-A., and Kamper, H. Spoken-term discovery using discrete speech units. arXiv preprint arXiv:2408. 14390, 2024.
Yang, D., Huang, R., Wang. Y., Guo. H., Chong, D., Liu, S., Wu, X., and Meng, H. Simplespeech 2: To- wards simple and efficient text-to text-to-speech with flow-based scalar latent transformer diffusion models. arXiv preprint arXiv:24085 13893, 2024b.
Veaux; C., Yamagishi, J., MacDonald, K., et al. Cstr vetk corpus: English multi sealti--peaee corpus for estr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 6:15, 2017.
Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. 2023.
Yang, D., Wang, D.., Guo, H., Chen, X., Wu, X., and Meng, H. Simplespeech: Towards simple and efficient text-to- speech with scalar latent transformer diffusion models. arXiv preprint arXiv:2406.0233 2024c.
arXiv preprint ar arXiv:2301.0011
Yang. S.-w., Chi, P.-.., Chuang. Y.-S., Lai, C.-L. Lakho- tia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., et al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.00
Wang, H., Suri, S., Ren, Y., Chen, H., and Shrivastava, A. Larp: Tokenizing videos with learned autoregres- sive generative prior. arXiv preprint arXi2.2410.21264, 2024a.
2021.
Yu, Q. Weber, M., Deng, X., Shen, X., Cremers, D., and Chen, L.-C. An image is worth 32 tokens for reconstruc- tion and generation. arXiv preprint arXiv:2406.07550 2024.
Wang, Y., Chen, H., Yang, D., Yu, J., Weng, C., Wu, Z and Meng, H. Consistent and relevant: Rethink the query embedding general sound separation. In ICASSP 2024-2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 961-9699 IEEE, 2024b.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 30:495-500 2021
Wang, Y., Chen, H., Yang, D., Li, W., Luo, D., Li, G., Yang, S., Wu, Z., Meng. H., and Wu, X. Unisep: Universal target audio separation with language models at scale. arXiv preprint arXiv:2503.23766 2025.
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y. Libritts: A corpus de- rived from librispeech for text-to-speech. arXiv preprint arXiv: /904 904.02882, 2019.
Wu, H., Kanda, N., Eskimez, S. E., and Li, J. Ts3-codec: Transformer- based simple streaming single codec. arXiv preprint ar 2ix::2411..880000 2024.
Zeng, A., Du Z., Liu, M., Wang. K., Jiang, S., Zhao, L., Dong, Y., and Tang, Glm-4-voice: Towards intelli- gent and human-like end-to spoken chatbot. arXiv preprint arXiv:2412.0266 2024.
Yang. D., Liu, S., Huang, R., Lei, G., Weng, C., Meng, H., and Yu, D. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. arXiv preprint arXiv:2301. 1366, 2023a.
Zhang, X., Zhang, D., Li. S., Zhou, Y., and Qiu, X. Speech- tokenizer: Unified speech tokenizer for speech large lan- guage models. arXiv preprint arXiv:2308. 16692, 2023.
Yang, D., Liu, S., Huang, R., Tian, J., Weng, C., and Zou, Y. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.0276. 2023b.
Zhu, L., Wei, F., Lu, Y., and Chen, D. Scaling the codebook size of vggan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406. 1837, 2024.
12

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Audio Token
4 1,3
1+12
1,3
Depth Transformer
Depth Transformer
add
Text prediction
s++1
1
A.+12
LORA
LLAMA 3.2
LLAMA 3.2
S,
Embedding
Audio streaming
A
Text streaming
As11
Figure The left diagram illustrates framework of audio language model, which includes pre-trained LLM, a LoRA module, and a depth transformer. The audio language model can process both text and audio streaming inputs and generate corresponding text and audio outputs. The right diagram provides details hierarchical audio modeling.
A. The details of audio language model framework
In this section, we provide details the audio language model. We follow the framework of UniAudio (Yang et al., 2023c) and Moshi (Défossez et al., 2024), which combines pre-trained LLM with smaller Transformer model predict audio tokens in a hierarchical manner. In their original paper, both the LLM and the small Transformer are updated during the training process. Due to resource limitations, and following (Hao et al., 2023), we incorporate LoRA (Hu et al., 2021) into the LLM model. For the LLM model, we use the LLAMA3.2 IB version. During training, we update only the LoRA module and the small Transformer.
LORA setting For the LoRA module, add LoRA parameters the self-attention and linear layers. We set lora, 32 and loraatpha 16.
Depth Transformer setting For the depth transformer, The attention dimension is the same as the LLAMA 3.2 1B.
6 self-attention layer. We set the attention head number as 32.
B. The details of the influence of bitrate and semantic information for audio language model.
In this section, we provide details of the validation experiments to explore the influence of bitrate and semantic information on audio language models. Following AudioLM (Borsos et al., 2023a), construct audio token pre-training task similar text pre-training, where the model is tasked with predicting the next audio token based the previous token sequence.
B.1. Training data
We conduct the experiments on 2000 hours speech data, these data is selected from MLS dataset (Pratap al., 2020).
B.2. Test data
We evaluate on LibriSpeech test clean set.
13

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table
The reconstruction performance of different frame of audio tokenizers.
Version
Bitrate ())
FPS (L)
codebook size
PESQ (1)
UT-MOS (t)
VISQOL (t)
STOI (1)
50hz
1650hps
50
2048
2.22
3.69
3.63
25hz
825bps
25
2048
2.07
3,56
3.61
0.83
12.5hz
412.5bps
12.5
2048
1.58
2.49
3.37
0.77
B.3. Framework
We use the same framework as described in Section A; the difference is that we do use text streaming.
B.4. Three Types of Audio Tokenizers
Following the structure of MimiCodec (Défossez et al., 2024), we train three versions of the audio codec tokenizer. All of the audio codec models are trained on 24kHz speech data. We train three versions of the audio codec models, as follows:
(V1) We set the down-sampling rate to [2, 5, 6, 8], resulting in 50 Hz frame rate. We use three RVQ layers, and the codebook size is 2,048. The bitrate of this audio codec is 1.65 kbps.
(V2) We set the down-sampling rate to [4, 5, 6, 8]. resulting in a 25 Hz frame rate. We use three RVQ layers, and the codebook size 2,048. The bitrate of this audio codec is 825 bps.
(V3) set the down-sampling rate |2, 4, 5, 6, 8), resulting in a 12.5 Hz frame rate. We use three RVQ layers, and the codebook size 2,048. The bitrate of this audio codec 412.5 bps.
Note that the original MimiCodec is trained with distillation loss from WavLM; do not add this loss during the training of our audio tokenizer. Therefore, these three audio tokenizers do not include any semantic information. Table 8 shows the reconstruction performance of the three audio tokenizers.
B.5. Semantic Tokenizer
The previous three audio codec tokenizers do not consider semantic information. To evaluate the importance of semantic information, we follow WhisperSpeech' to build a Whisper-based semantic tokenizer. Specifically, we follow the training code of WhisperSpeech, using two down-sampling layers to compress the Whisper encoder's features into a 12.5 Hz frame rate, and then we add three RVQ layers to quantize them. Thus, this semantic tokenizer has the same bitrate as the V3 audio tokenizer.
B.6. Evaluation metrics
evaluate the pre-training performance from the following aspects:
Training efficiency: As is well known, the space complexity transformer O(T2), where T' the sequence length. low-bitrate audio tokenizer can compress the audio signal into few token sequences, thereby improving training efficiency, For all experiments, use the same GPU machine to train the model and record the statistical training duration.
Inference efficiency: Similarly, low-bitrate audio tokenizer can improve inference efficiency, as it requires fewer inference steps. We use the Real-Time Factor (RTF) to assess inference efficiency. Note that for all experiments, we do not use any inference optimization tricks, such as KV cache.
Validation loss and perplexity: Following text LLMS (OpenAl, 2023), we use validation loss and perplexity to evaluate model performance.
14

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
4,0
UI-MSS
3.0
Score
7.0
1.5
Window size
Figure The performance comparison with different window size during inference.
Table 9.
influence of codebook size for reconstruction performance.
Codehook Size
PESQ (t)
UT-MOS (t)
VISQOL (1)
STOI (f)
STFT loss (L)
Token untilzation (1)
2048
2.0
3.76
3.78
0.81
1.20
100%
1024
1.83
3.66
3.65
0.80
1.14
100%
512
1.69
3.64
3.58
0.792
1.18
100%
C. Ablation study
C.1. The influence of window size for ALMTokenizer
As discussed in the previous section, the proposed ALMTokenizer supports a dynamic compression rate by changing the window size Figure 5 shows the comparison of reconstruction performance with different window sizes. We observe that using a smaller window size results in better reconstruction performance, but it also increases the bitrate. For exmaple, if the window size the bitrate 1237.5bps, window size the bitrate 412.5. also shows the advantages of proposed method: we can dynamically change the frame rate during the inference by setting different window size.
C.2. The influence codebook size
We explore three different codebook sizes: 512, 1024, and 2048. align with the setting MimiCodec (Défossez et al., 2024), set the max codebook size as 2048. The results, shown in Table 9, are presented. We observe that scaling the codebook size improves reconstruction performance. Furthermore, we also find that almost all tokens have been used.
C.3. The influence model size for reconstruction performance
To explore the influence of model size on reconstruction performance, we up two configurations: (1) We use 24 self-attention layers for both the transformer encoder and transformer decoder, resulting in 174M parameters. (2) We use 12 self-attention layers for both the transformer encoder and transformer decoder, resulting in 87M parameters. In both settings, we keep the Patchify module the same size, it consists of several convolutional layers, and its total parameters are small. The experimental results, as shown in Table 10, indicate that using a larger model can improve reconstruction but also increases computational resource consumption (higher RTF). Previous work, StableCodec (Parker et al., 2024), shows that scaling the codec model to IB parameters can lead to better performance. Due to computational resource limitations, we leave scaling larger model size for future work.
15

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table 10. The influence of model for reconstruction performance.
Setting
PESQ (1)
UT-MOS (f)
VISQOL (1)
STOI (t)
Model size (L)
RTF (L)
attention layer 12 attention layer
3.76
3.78
0.81
0.031
1.87
3.57
3.70
0.79
87
0.019
C.4. The influence mask-rate in MAE loss
Inspired by MAE(He et al., 2022), we tested three group of mask rates ranges: (10-20%) 20030%) and (30 (0004 The experiments as following Table shows. Results indicate that higher rates (30-40%) benefit semantics but harm reconstruction, leading adopt intermediate range (20-30%).
Table 11.
The influence of mask-rate for MAE loss.
mask rate range
UTMOS
DNSMOS
VISQOL
PESQ
STOI
ASR
ER
10-20%
3.77
3.62
3.80
2.0
0.81
18.7
27.7
20-30%
3.76
3.64
3.78
2.0
0.81
18.3
29.0
30-40%
3.36
3.06
3.31
1.58
0.77
18.1
29.6
D. Evaluation
evaluate the performance of the previous SOTA audio tokenizers and our proposed ALMTokenizer across audio reconstruction, audio semantic information, audio understanding, and audio generation tasks.
D.1. Audio Reconstruction
For speech data, we use DNS-MOS (Reddy et al., 2022), UT-MOS (Saeki et al., 2022), PESQ, STOI (Short-Time Objective Intelligibility), VISQOL (speech version), and STFT loss as metrics.
For sound and music data, al., 2023), we conduct the MUSHRA subjective test for speech, sound, and music. Specifically, we hire 10 audio-related researchers to conduct the MOS evaluation. We ask the listeners to rate each audio, with scores ranging from 0 to 100. Refer to D.5 for the details.
VISQOL (audio version), STFT loss, and Mel loss. Furthermore, following (Kumar
Evaluation Datasets: For speech data, we evaluate on a subset of VCTK (Veaux et al., 2017) (200 speech utterances) and a subset of the LibriTTS test clean set (Zen et al., 2019) (400 speech utterances). For sound data, we evaluate on subset of the AudioCaps validation set (Kim al., 2019) (200 sound utterances). For music data, we evaluate on subset of the MusicCaps (Agostinelli et al., 2023) dataset (200 music utterances).
D.2. Audio Semantic Information
Previous SSL models, such Hubert (Hsu al., 2021) and WavLM (Chen al., 2022a), have shown that semantic-rich representations can be used to solve downstream recognition tasks by fine-tuning several adaptor layers. Inspired by these works, propose evaluating the performance of the audio tokenizer for downstream recognition tasks. We use the quantized features of the audio tokenizer as the input for downstream tasks. We follow two popular benchmarks: SUPERB (Yang et al., 2021) and ARCH (L.a Quatra et al., 2024).
For speech data, we conduct the automatic speech recognition (ASR) task on the LibriSpeech (Panayotov et al., 2015) dataset and the emotion classification (EC) task on the EMOVO (Costantini al., 2014) dataset. For the ASR task, we train on the LibriSpeech 100 set and evaluate on the LibriSpeech test clean set. For the EC task, we follow ARCH (La Quatra al., 2024) to split the training and test sets.
For sound data, we conduct the sound classification task on the FSC-50 dataset (Piczak, 2015). For music data, we conduct the music classification task on the Medley-Solos-De dataset (Lostanlen & Cella, 2016). For both tasks, we follow the ARCH benchmarking settings to split the training and test sets.
For all experiments, we train for 10 epochs with the same learning rate and batch size. For the automatic speech recognition
16

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
task, we use word error rate (WER) the metric. For the other classification tasks, we use accuracy the metric.
D.3. LM-based Audio Understanding
Overview To further validate whether the audio tokenizer is suitable for building an audio language model, we propose conducting an audio understanding task using discrete tokens input. We conduct three tasks: automatic speech recognition (ASR), audio captioning, and music captioning. We use the framework introduced in Section A. For audio data, we use the audio tokenizer to encode it as discrete tokens; for text data, use the BPE tokenizer of LLAMA 3.2. We construct the sequence as [audio token, text token], then the model is asked to predict the text token based on the previous audio token.
Training Data For the ASR task, we select 2,000 hours of LibriHeavy speech data (Kang et al., 2024). For the audio captioning tasks, we use AudioCaps (Kim et al., 2019) and BBC sound effects (Mei et al., 2023). For the BBC sound effects, we cut off the first 10 seconds of audio if the utterance duration greater than 10 seconds. Finally, we obtain about 500 hours of sound data. For the music captioning task, we use subset of the Million Song dataset. We cut off the first 10 seconds of music data for each utterance, which results in about 500 hours of music data. For the corresponding captions, we use LPMusicCaps (Doh et al., 2023).
Test Data For the ASR task, we evaluate on the LibriSpeech test clean set. For the audio captioning task, we evaluate on the AudioCaps dataset (Kim et al., 2019). For the music captioning task, we evaluate on the MusicCaps dataset (Agostinellii et al., 2023).
Metries Similarly, we use WER the evaluation metrie for the ASR task. For audio and music captioning, we follow (Drossos al., 2020) and adopt BLEU-1, BLEU-2, BLEU-3, METEOR, ROUGE-L, CIDEr-D, SPICE, and SPIDEr metrics.
Inference Setting For inference, we directly use the top-k sampling strategy and set k 30 for all experiments.
D.4. LM-based Audio Generation
We also perform audio generation tasks, including text-to-speech, text-to-sound, and text-to-music generation. Similarly, we construct the sequence as [text token, audio token], then the model is asked to predict the audio token based on the previous text token.
Training and Test Data We use the same training and test data as the audio comprehension task.
Metrics For TTS evaluation, we use WER to evaluate robustness, and UTMOS and DNSMOS are used to assess speech quality. For text-to-sound and texi-to-music, we follow previous works AudioGen (Kreuk et al., 2022), using Fréchet Audio Distance (FAD), Kullback-Leibler (KL) Divergence, and Fréchet Distance (FD) for audio fidelity and similarity.
Inference Setting During the inference stage, we use the top-k sampling strategy and set k = 30 for all experiments.
D.5. Subjective Evaluations
For the subjective evaluations, we adopt the approach used in previous works (Kumar et al., 2023; Parker et al., 2024) and use the MUSHRA format without a hidden anchor. Listeners are asked to compare multiple versions of an example simultaneously, including both labeled reference and hidden reference. They are given the following instructions: "Please assess the quality similarity between an audio sample and its reference. Listen carefully to the reference audio, then rate the quality of each test clip in comparison. A score of indicates no resemblance to the reference, while a score of 100 means identical the reference." " We randomly select 10 samples from each category (speech, music, and sound) in the test set, ensuring that each sample receives 10 ratings.
E. Audio Tokenizer Baselines
To make a fair comparison, we classify the audio tokenizers into two types: (1) speech-based tokenizers, which are trained on speech datasets, and (2) audio-based tokenizers, which are trained on speech, sound, and music datasets.
E. Speech Tokenizer
For speech data, we compare with:
17

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table 12. The performance comparison on LibriTTS test clean. Bold for the best result and underline for the second-best result.
Reconstruction
Efficiency
Models
FPS/TPS
CS/BR
UTMOS (1)
DNS-MOS (1)
VISQOL (1)
STOI
PESQ (f)
Model size (M) (L)
RTF (U)
Encodec
50/400
1024/6kbps
3.30
3.76
3.95
0.94
2.72
14
0.019
Encodec
50/150
1024/1 1024/1_5kbps
2.02
3.27
3.83
0.88
1.79
0.019
50/150
1024/1 Skbps
3.36
3.85
0.89
1,96
0.026
Wavtokenizer
40/40
40khss
3.65
3.61
3,80
0.87
1.81
77
0.017
StableCodec
25/25
46656/0 4kbps
3.74
1.85
0.039
MimiCodec (3 RVQ)
12:5/37.5
21kbp
2.82
3.28
3.34
0.83
1.40
75.6
0.023
ALMTokenizer (Ours)
12,5/37.5
204B/0.41bpps
3.68
3.64
3,90
0.90
1.92
174
0.031
(1) Encodec (Défossez et al., 2022), SOTA audio codec model trained on large-scale speech, sound, and music datasets. The official open-sourced 24 kHz version used.
(2) DAC-Codec (Kumar et al., 2023), which offers very high reconstruction performance. trained on large-scale speech, sound, and music datasets. The official open-sourced 24 kHz version is used.
(3) MimiCodec (Défossez al., 2024), SOTA low-bitrate speech codec model trained large-scale speech dataset. The sampling rate is 24 kHz.
(4) Sokezzerr (Zhang et al., 2023), semantic-rich speech codec model trained on a large-scale speech dataset. The sampling rate is 16 kHz.
(5) ToTenzziee (Ji al., 2024), an audio codee tokenizer trained on large-scale speech, sound, and music datasets. The sampling rate is 24 kHz.
To make fair comparison, for Encodec, DAC-Codec, and SpeechTokenizer, the first three RVQ layers to control the bitrate during inference.
E.2. Audio Tokenizer
For sound and music data, compare with Encodec, DAC-Codec, and WavTokenizer. These three models trained on large-scale speech, sound, and music datasets.
E.3. Semantic Models
Furthermore, to evaluate the performance of semantie information, also introduce several SSL-based models. For speech, use WavLM (Chen et al., 2022a) and HuBERT (Hsu et al., 2021). For sound and music, use BEATs (Chen et al., 2022b) and Wav2Vec2-AudioSet
F. More audio tokenizer evaluation experiments
F.1 The subjective evaluation for audio tokenizer
Table 7 shows the subjective evaluation results for audio tokenizer.
F2. Evaluation results on LibriTTS test clean
We report the reconstruction performance evaluated on a subset of the LibriTTS test clean set, where we randomly select 400 speech utterances. Additionally, we calculate the Real-Time Factor (RTF) and model size to assess efficiency. For RTF evaluation,
NVIDIA A100 GPU evaluate all models.
F.3. Length generalization
StableCodec (Parker et al., 2024) highlights that the introduction of transformer-based architectures can lead the length generalization problem. For instance, the training data of ALMTokenizer consists of 5-second segments, whereas the test
18

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table 13. Objective metries for the Al.Toocnnzerr and baselines, evaluated utterances from length 4s 10s, showing generalization of
models across lengths
Model
FPS
TPS
Bitrate
PESQ (1)
UT-MOS (1)
VISQOL (t)
STOI (t)
DNSMOS (t)
4 seconds
Encodec
50
150
1.5kbps
1.97
2.64
3.62
0.80
3.26
DAC
50
150
1.5kbps
2.1
3.17
3.65
0.81
3,26
Ours
12.5
37.5
0.41kbps
1.84
3.63
0.79
3.41
seconds
Encodec
50
150
1.5kbps
1.97
2.54
0.81
3.26
DAC
50
150
1.5kbps
2.0
3.11
3.65
0.81
3.28
Ours
12.5
37.5
0.41kbps
1.89
3.66
3.75
0.81
3.62
seconds
Encodec
50
150
1.5kbps
1.96
2.52
3.63
0.81
3,34
DAC
50
150
1.5kbps
2.1
3.18
3.66
0.81
3.28
Ours
12.5
37.5
0.41kbps
1.95
3.55
3.74
0.81
3.66
10 seconds
Encodec
50
150
1.5kbps
1.95
2.53
3.65
0.81
3.32
DAC
50
150
1.5kbps
2.1
2.19
3.67
0,81
3.25
Ours
12.5
37.5
0.41khps
1.96
3.54
3.73
0.81
3.66
data comprises segments of varying durations. We evaluate the model across four distinct length levels: 4, 6, 8, and 10 seconds. Encodec and DAC are selected as baselines due to their reliance on convolutional layers, which demonstrate robustness to variable input lengths. As shown in Table 13, the evaluation results indicate that ALMTokenizer effectively handles inference across these diverse lengths. These findings suggest that ALMTokenizer exhibits strong generalization capabilities with respect to input length variation.
F4. Compared to diffusion-basee audio codec models
We compare ALMTokenizer with an alternative family of audio tokenizers that leverage discrete semantic tokens derived from self-supervised pre-trained (SSL) models (e.g., Hubert (Hsu al., 2021), WavLM (Chen et al., 2022a), AudioMAE (Huang et al.. 2022)). These models first quantize the SSL features into semantic tokens and subsequently use generative model resynthesize the waveform. Diffusion (Ho al., 2020) and Flow-Man Mott-Mathhing (Lipman al., 2022) generative models. Previous works, such GLMc-Voice tokenizer (Zeng et al., 2024) and SemantiCodec (Liu et al., 2024), have demonstrated success using diffusion-based decoders. However, such strategies tend to result in significant information loss. For instance, the semantic tokens GI. lack timbre information and require additional prompts to control timbre during decoding. Notably, the open-sourced GLM4- tokenizer uses a fixed timbre, meaning that any speech encoded by GLM4- Voice will lose original timbre. address this information loss in semantic tokens, SemantiCodec introduces acoustic streaming to enhance waveform reconstruction. key concern, however, is that both SemantiCodec and (Li4-Voice tokenizers demand significantly more computational resources during the inference stage. In the following, present comprehensive comparison between ALMTokenizer and SemantiCodec, focusing the following aspects: (1) reconstruction performance for speech, sound, and music; (2) semantic information performance for speech, sound, and music; and (3) computational resource requirements during inference, measured using RTF.
popular
Table 14 shows the speech reconstruction and semantic performance, where we observe that ALMTokenizer outperforms the alternatives in both aspects while using less bitrate. Table 15 presents experimental results for sound and music data, where ALMTokenizer again demonstrates superior performance across all metrics compared SemantiCodec. In Table 16. present the model size and RTF metrics, showing that ALMTokenizer has fewer model parameters and significantly surpasses SemantiCodee in inference speed (0.031 vs 0.92).
G. The details of ALMTokenizer structure and training
G.1. Model structure
Table 17 gives the details of ALMTokemizer configuration, which results in 174M parameters. all of experiments, for the MAF-transformer encoded and decoder, we adopt layer transformer layers.
19

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table 14. The performance comparison between AlMTokenezer and SemanticCodec VCTK dataset.
Reconstruction
Semantic
Models
FPS/IPS
CS/BR
UTMOS (1)
DNS-MOS (1)
VISQOL (1)
(1)
PESQ (1)
ASR (4)
EC (1)
SemantiCodec
50/50
16384/0.68kbb 68kbps
3.2
3.57
3.90
0.81
1.76
48.3
17.8
ALMTokenizer
12.5/37.5
2048/0.41kbps
3,76
3,64
3.78
0.81
2.0
18.3
29.0
Table 15. The performance comparison between ALMTokenizer and SemanticCodec Music (MusicCaps) and sound data (AudioCaps).
Reconstruction
Semantic
Models
FPS/TPS
CS/BR
Mel loss ())
STFT loss (1)
VISQOL (1)
Classification (1)
Sound data SemantiCodec ALMTokenizer
50/50
16384/0.68kbps
18.45
1.40
2.47
38.8%
12.5/37.5
2048/0.41kbps
15.0
1.24
2.99
44% %
Music data
SemantiCodec
50/50
16384/0.68kbps
47.9
1.58
2.49
48%
ALMTokenizer
12.5/37.5
2048/0.41kbps
34.4
1.32
3.96
59 %
Patchify and UnPatchify modules A single-chamnel audio signal æ E R XN (where N denotes the sampling points) processed through the Encodec-style Patchify and UnPatchify modules, which adopt the same structure as Encodec (Défossez et al., 2022), consisting of four convolutional blocks. Each convolutional block consists of residual unit followed by a down-sampling layer. These convolution blocks effectively encode the audio signal into audio frame representation E RTxd where T denotes the number of frames and denotes the dimension of cach vector. The convolution blocks are followed by two-layer LSTM for sequence modeling, followed by a final ID convolutional layer with kernel size of 7 and D output channels. The UnPatchify module mirrors the Patchify architecture by substituting stride convolutions with transposed convolutions and reversing the stride order.
For the SaableCodec-styiee Patchify and UnPatchify modules, we follow the approach StableCodec (Parker al., 2024) and use a reshape operation to transform æ E R*r into e E RTxd where T N/320 and d 320. We then apply a linear layer map the dimension to D. Finally, add four transformer layers for sequence modeling. Similarly, the UnPatchifyy module mirrors the Patchify architecture.
Discriminators For the discriminators, we follow prior work (Défossez et al., 2022). which combines mel-spectrogram and -ogeee-sppectoogram features and inputs them into a network consisting of several convolutional layers. Specifically, we use six discriminators with different configurations: the hidden dimensions are set as 64, 128, 256, 512, 512, 512, and the hop lengths are set as 32, 64, 128, 256, 512, 1024.
G.2. Reconstruction loss and adversarial loss for ALMTokenizer
Let the reconstructed signal be a. For the reconstruction loss, we design from two perspectives: the time domain and the frequency domain. We first compute the L1 loss between æ and & the time domain. Next, we compute the L₃ loss between the STFT spectrogram of and & in the frequency domain. Following (Wang et al., 2024b), we employ sub-band split strategy divide the spectrogram into several parts. The adversarial loss is employed to enhance the perceptual quality of the generated audio:
K max (0, 10,1 + max (0, Dx))))
Ca
(4)
where denotes the number of discriminators. During the training stage, the adversarial loss for the generator is computed hinge loss over the logits of these discriminators:
Lado
max (0, 1 (x))))
(5)
The feature loss Cjeat is computed by taking the average absolute difference between the discriminator's internal layer outputs for the generated audio and those for the corresponding real audio.
20

---

Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
Table 16. The model size and RTF comparison between SemantiCodec and M.oTkkenizer.
Model
Model size (M) (4)
RTF (1)
SemantiCodec ALMTokenizer (Ours)
507
174
0.031
ALMTokenizer
Input shape
(B, I) (B, T, d), T=N/320 [2, 4, 7, 8,9, 10)
Patchify module (output)
Token Interleaving and Retrieval Dimension of transformer encoder The number of transformer encoder
256
24
Dimension of transformer decoder
512
The number of transformer decoder Codebook size
24
2048
VQ layers
3
Number of Transformer heads
64
UnPatchify module (output)
(B, 1, N)
Table 17. ALMTokenizer model backbone configurations
G.3. Training details
The A optimizer is used in the training. We set the learn rate as le 4. We train the model with 200k steps. The final loss as following shows. We set λ = 0.5 and X2 0.1 during our experiments. We conduct all of the experiments with
NVIDIA A100-80G GPUs.
C Lado Lifeat + Lree + AI + A2LAR
H. Reproducibility Statement
To enhance reproducibility, we provide the pseudocode ALMTokenizer. In the future, we plan to improve both the model structure and training data obtain more robust models, especially for music and sound, and release the code for the research community.
Listing 1. Pseudocode of ALMTokenizer
class ALMrokenizer:
def _init_
self, transforeer_enoodr__e
transformer_deodde...nr
mae__ecoder_aags,
cepth_gpt_args,
patchity_args,
encoder_embed_diin
decoder_embeddddim,
semantic_prior_path,
Sindow_sizes = [2, 3, 4, 6,,,,,
wilf_window_sizes window window_sizes s_iees sclf.traanffommeoooncdo
Transformez (fcanoffrmee__cccoeee_.gggg)
self.traanfoomer_dccoomm = Transformer (ttp::/ettttttttteeoedddeoo.eee
sel self.mae_decoder = Transformer (naccmerrmmm_dddeddeerraagg))))
self.Patchify incodec_enccder (patcci fy_args)
self.UnPatchif" fy Encodec_ (patchi agg
21

---

A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
nn Parameter (torch. zeros (1, 1, encoder_embed__din
self mal.kaak__toee nn. Parameter (torch. zeros (1, 1, decoder_enbed_dee
checkpeint torch.load
map_location="<pe
self.vq = RVQ semantic
imput_ pruut_dim=encoder.cm embed dim,
semantic prriorrrrrrr checkpoint,
layers = 3)
self.depth..e.ee depth_gpt GPT rer_decoder (depth
self self.tmp_winddw_siie
self.mask_rate rate
def
x)
8, T, x. shape batch, length, dim
cls tokens self.cls self.clss.toennrreees repen. (B, (I//self. tmp wwndwwwwidweeeeeeeeeew unsqueze (2)
new T I (T // self. window size)
r_reshaeed x, reshape (B, I // self.tmp haafttmp_xindoxxxo window size, self.tmp selfdotm_xiidoww.... window_ size, D)
xwitthclss torch (Ix_reshaped, cls_tokens], dim-2)
new_x x_with cls reshape (B, -1, D) return new_x
def peecder_token. Bette__tttee_Retivaallllll (self, x)
B, new_T, , D = x. shape original_I mask indices [ii 1) (self. tmp_ wtp_:t/twwwwwweee.eec...tttttmtt window 1) for range (original_
new_I // (self. wttp:tttt_wiidw_eeeemm
// self. wttp:ttm__widooii_eeeeeee (iidow_size))
cls_tokens new_xl;, mask_ indices, :]
return cla_tokens
def Decoder token (selle,, en oo_to)en):
B, T, D en an_token.shape
self. hal...makkttoee.nne repeat (2, 1, 1)
new_T en_token. (ha.eeenshaplllll1ccc (sclfff.ll..l......... wttpttpp_wwwsseeee window_siieeeeee + en_token, shape [1]
x x.repeat (1, en_token, shape [1] ttg.e....____.oooooo........
1)
x = x.reshape (B, -1,
D)
x_with_maske = torch.cat. ([x, un_token.unsqqeee. (2) ] dim=2)
new_x E_with_masker reshape -1, D) new_x
def Decoder_ taken Retrieaat (self, new_x) B, new I, = new shape num_masks new_T // belf.interve. +
original_T new_T num_masks
mask mask_indices = [[1 1) (self. interval 1) for i in range (num_masks)
all all_indices = 1ist (range (new_T)
mask laskiindiees- = [i for in all indices if i not in bask_indices, frames new_x mask_indices, :]
return mask_frames
def forward self, x,
lel - [-sh
self tmp window size = choice windd. sizes)
emb franes self.Patchify (x)
if telf.nraining:
emb_frames_maak = self self.apply (om___araemmmmmmaeeeeee mask_rate self fal.f.ask_ratt)
interleving__rames ooo.i (emb com_ffranesmmake
predict_latent www..mac__cccoocooeoooo oi
mae lcas _l_loss (predict_lateent, emb frames)
latent_tckens self aat.aaaaassfferrrrrooooaeeaaeeea query_tcken = self. f Quantized_token, codes, aul_quantized self. (query_token) cat_quantized for emb all quantized::
(latent_tokees
22

---

Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling
__emb __mmb reshape (-1, e_mmb shape[-1]] unsqueeze (1)
cat_guuantized.appeeee (q_emb)
_tt_quantized torch. cat (cat_quant) _ua_tiiue dim-1)
gpt lot__oss = self. hwlff.dethhppge gpt.
loss (cat_ (caa__quatizedl
de =
(Quantized
de ve_latent_token wwwf.transformrr_ceeo sel decoder (de_oddrrddd__eeeeeeeeeeeeeeeeee interlevingg [fanees
mask_tckens =
x_ Sellf.unPPtchhff (mask_tekens)
return x, mae los_, gpt_
I. Limitation
In this study, we present ALMTokenizer, low-bitrate, semantic-rich audio codec tokenizer. We demonstrate that ALM- Tokenizer excels both reconstruction and semantic information retention under low-bitrate conditions. However, we acknowledge that there is still significant room for improvement in reconstruction performance, particularly for sound and music data. Building audio tokenizer for sound and music in the low-bitrate setting poses additional challenges. In terms of semantic information, ALMTokenizer still lags behind traditional SSL models. Although we propose several training losses to enhance semantic information in the codec model, the improvements are limited and, in some cases, negatively impact reconstruction quality. We recognize the need for a careful design and balance these semantic loss terms. Additionally, the multi-stage training strategy increases training complexity. These training strategy brings waste. Most of the components are eventually discarded, e.g. MAE-transformer encoder/decoder, MAE-decoder, and depth AR- ARtrransformer. These components would have made to still utilize them for some purpose, e.g. the AR decoder could have been used to initialize the depth transformer in the Language modeling task. These concerns left for future work.
23


Reviews:
[{"review_id": "l4MIMcSp32", "reviewer": "Reviewer_btwe", "summary": "The authors propose ALMTokenizer, an audio tokenizer designed to enhance compression efficiency and reconstruction quality at a low bitrate. Its key innovations include a query-based framework, semantic priors in vector quantization (VQ) codebooks by leveraging self-supervised learning (SSL) model feature clusters, MAE and LM losses, and a two-stage training approach. Experimental results show that ALMTokenizer meets or exceeds the performance of recent neural audio codecs in both compression and semantic information measures. Additionally, downstream generative modeling with ALMTokenizer’s audio tokens achieves stronger performance compared to competing tokenizers.\n\n\n## update after rebuttal\nAfter reviewing the authors’ rebuttal and their reply to the rebuttal comments, I find that the updated experimental results sufficiently address most of my concerns. I encourage the authors to incorporate the points discussed during this exchange into the revised manuscript. Given these improvements, I have updated my score from 3 to 4 and now recommend acceptance of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. We do appreciate the constructive comments the reviewer provided to us to further improve our paper. We are delighted to have the following discussion with the reviewer.\n\n**Q1** : Although the paper’s ablation study suggests that each proposed technique contributes, the simultaneous use of multiple methods (semantic prior VQ, MAE, LM loss, and two-stage training) complicates fair comparisons.\n\n**A** : We thank the reviewer for acknowledging the contributions of our proposed techniques. To systematically validate each component's effectiveness, we have conducted extensive ablation studies (Table 6 of manuscripts), including:\n\n**(1) Query-Based Framework** : We constructed a MimiCodec-style baseline [1] with identical convolutional patchify/unpatchify modules and transformer-based encoder/decoder. The key distinction lies in our proposed query-based framework, which demonstrably improves audio compression and semantic performance (Table 6, Row 3).\n\n**(2) MAE and LM (AR) Losses** : Ablation experiments confirm that both losses enhance semantic performance (Table 6, Rows 4) and Figure 3.\n\n**(3) Two-Stage Training** : Compared to one-stage training, our two-stage strategy consistently improves reconstruction and semantic metrics (Table 6, Row 6).\n\nThese experiments clearly clarify the impacts of our techniques.\n\n**Q2**:  For example, while semantic VQ priors provide only marginal improvements in semantic tasks, they slightly degrade audio reconstruction quality.\n\n**A:**  We agree with the reviewer’s observation regarding the trade-off between semantic performance and reconstruction quality. One of the potential reasons is that we fix the VQ codebooks during training, which is different from the traditional VQ training (as noted in Section 3.2). This aligns with our discussion on limitations part: while semantic VQ priors improve semantic performance, optimizing the semantic information and minimize reconstruction loss remains an open challenge. We highlight this trade-off to encourage future work on balanced solutions.\n\n**Q3:**  Additionally, MAE, LM loss, and two-stage training could likely be integrated independently into other neural audio codecs. \n\n**A:** We appreciate and agree with the reviewer that our proposed technique contributes, such as MAE loss, LM loss, two-stage training strategy can be integrated into other neural audio codec models, such as Encodec and MimiCodec. As we discussed in Q1, we have conducted ablation studies to validate the effectiveness of each part. We hope these contributions will inspire broader adoption in the audio codec community.\n\n**Q4:**  The paper does not sufficiently ..., analyzing these variations would help determine whether the approach is robust or ... under different configurations. \n\n**A:** We appreciate this suggestion and provide additional analyses:\n\n**(1) Mask Rate in MAE Loss.** \nInspired by MAE [2], we tested three group of mask rates ranges: (10–20\\%), (20–30\\%), and (30–40\\%). The experiments as following Table shows. Results indicate that higher rates (30–40\\%) benefit semantics but harm reconstruction, leading us to adopt an intermediate range (20–30\\%).\n\n| mask rate range | UTMOS  | DNSMOS  | VISQOL | PESQ |  STOI |  ASR  |  ER  |\n|:---------------:|:------:|:-------:|:------:|:----:|:-----:|:-----:|:----:|\n|      10-20%     |  3.77  |   3.62  |  3.78  |  2.0 |  0.81 |  18.7 | 27.7 |\n|      20-30%     |  3.76  |  3.64   |  3.78  | 2.0  | 0.81  | 18.3  | 29.0 |\n|      30-40%     |  3.36  |   3.06  |  3.31  | 1.58 |  0.77 |  18.1 | 29.6 |\n\n**(2) The hyperparameters of MAE loss and AR loss weighting.**   To better understanding the influence of hyperparameters of MAE loss and AR loss ($\\lambda_1$ and $\\lambda_2$), we design 3 group settings (1, 1), (0.5, 0.5), (0.5, 0.1). The experimental results as the following Table shows. We can see that (0.5, 0.1) obtains better performance, thus we empirically choose $\\lambda_1=0.5$ and $\\lambda_2=0.1$ as our default setting. \n\n| $\\lambda_1$  | $\\lambda_2$ | UTMOS   | DNSMOS     | VISQOL    | PESQ    | STOI    | ASR    |  ER  |\n|:------:|:------:|:-------:|:----------:|:---------:|:-------:|:-------:|:------:|:----:|\n|    1   |    1   |   3.69  |    3.55    |    3.70   |   1.8   |   0.78  |  18.4  | 29.7 |\n|   0.5  |   0.5  |   3.71  |    3.58    |    3.77   |   1.9   |   0.77  |  19.0  | 28.8 |\n|   0.5  |   0.1  |   3.76  |    3.64    |   3.78    |   2.0   |  0.81   |  18.3  | 29.0 |\n\n[1] Défossez A, et al. Moshi: a speech-text foundation model for real-time dialogue[J]. 2024. \\\\\n\n[2] He K, et al. Masked autoencoders are scalable vision learners CVPR 2022.", "claims_and_evidence": "The proposed methods and evaluation criteria are relevant.", "methods_and_evaluation_criteria": "Proposed methods and/or evaluation criteria are reasonable and aligned with the problem/application.", "theoretical_claims": "I reviewed proposed methods and found them sound.", "experimental_designs_or_analyses": "I reviewed the experimental design and analyses and found them sound.", "relation_to_broader_scientific_literature": "This work contributes to improving neural audio codecs at low bit rates and demonstrates enhanced performance on downstream generative tasks. These advancements are relevant to broader audio generative modeling, including applications such as speech-language modeling, text-to-speech, and voice conversion.", "essential_references_not_discussed": "Essential references are well discussed.", "other_strengths_and_weaknesses": "Strengths:\n* The proposed query-based compression framework is promising for easily adjusting compression rates, making it potentially useful across varied low-bitrate scenarios.\n* The paper is well-structured, with detailed explanations of the architecture and training process. The analysis of experimental outcomes is both comprehensive and straightforward, aiding reader understanding.\n*  Improved performance in downstream generative modeling including TTS indicates that the proposed approach can be integrated effectively into broader applications.\n\nWeaknesses:\n* Although the paper’s ablation study suggests that each proposed technique contributes, the simultaneous use of multiple methods (semantic prior VQ, MAE, LM loss, and two-stage training) complicates fair comparisons. For example, while semantic VQ priors provide only marginal improvements in semantic tasks, they slightly degrade audio reconstruction quality. Additionally, MAE, LM loss, and two-stage training could likely be integrated independently into other neural audio codecs. A more direct comparison using only the core techniques in the main experiments would strengthen the paper’s central claim and clarify the true necessity of the auxiliary techniques.\n* The paper does not sufficiently address how changes in specific hyperparameters or components (e.g., MAE or AR loss) might affect the final model’s behavior. If the authors assert that each technique is equally important, analyzing these variations would help determine whether the approach is robust or susceptible to performance degradation under different configurations.", "other_comments_or_suggestions": "I don't have any other comments or suggestions.", "questions_for_authors": "I don't have any other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ws63k49lnz", "reviewer": "Reviewer_st8J", "summary": "The paper presents a method to convert an audio signal to a sequence of discrete tokens, with an aim to maximize compression (low bit rate) while retaining maximum semantic information. To achieve this goal, it introduces the use of learnable query tokens, masked auto-encoders, semantic priors (to initialize VQ layer), and AR prediction loss, in the audio tokenization pipeline. The applications include audio generation, text-to-speech and multimodal LLMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's time and patience with our paper. We are delighted to solve the concerns of reviewer one by one.\n\n**Q1:**  Several essential ... query token is not clear to me. \n\n**A:**  We appreciate your feedback regarding the clarity of the \"query token\" concept. Below, we provide a detailed explanation.\n\nwe first revisit the workflow of previous tokenization methods, such as Encodec and SoundStream. As shown in the left part of Figure 2, the input audio data is first processed by the encoder module, transforming it into a series of audio frames, denoted as $\\boldsymbol{e} \\in \\mathcal{R}^{T \\times d}$, where $T$ represents the number of frames. The value of $T$ is determined by the down-sampling strides in the encoder module. Notably, previous works treat all audio frames equally. However, in such a setting, reducing a low frame rate (e.g., 12.5 Hz) requires increasing the down-sampling stride to 1920, which significantly degrades reconstruction due to: (1) ignoring the fact that different audio frames encode varying levels of information; and (2) failing to leverage contextual dependencies across frames. \n\nThus, we propose a query-based compression strategy. Instead of employing large down-sampling strides, we first use a Patchify module to segment the audio into frames. Our approach then introduces learnable query tokens, which dynamically extract information across multiple frames. **Notably, query tokens are learnable embedding vectors that are updated throughout the training process.** As described in Section 3.2, these learnable query tokens are combined with the audio frames and processed by a transformer encoder, where they adaptively aggregate important information. After that the original audio frames will be ignored, and these query tokens will be used for RVQ and decoder.\n\nIn summary, query tokens serve as learnable embedding vectors designed to capture holistic contextual information from the audio frames. Since the number of query tokens is smaller than the number of audio frames, which effectively reduces the frame rate while preserving essential information. The concept of query tokens is analogous to BERT [1], where a [CLS] token is placed at the beginning of a sentence to capture its semantic representation (In contrast, in our stdudy, we introduce multiple query tokens by Query token interleaving). To enhance clarity for readers, we denote this query token as [CLS] in our paper (line 205).\n\n**Q2:**   It seems the bit-rate is...\n\n**A:**  As discussed in Q1, the low bitrate is not achieved through hyperparameter tuning. Instead, we introduce query tokens that summarize contextual information across frames, where the number of query tokens directly determines the bitrate.\n\n**Q3:**  Second, the ... same is not clear to me.\n\n**A:**  We appreciate this comment. As discussed in Q1, we employ query tokens to capture holistic audio context information from audio frames. Similar to BERT, semantic information is effectively aggregated into the query tokens with the aid of a transformer encoder.\n\n**Q4:**  The use of transformers, MAE, .. seems novel ... audio conferences and journals. \n\n**A:** We thank the reviewer for recognizing our contributions. Although our approach involves audio processing techniques, its core contribution lies in machine learning methodologies—specifically, transformer-based compression and representation learning for audio modeling. Our method is not limited to audio; the query-based compression strategy can be extended to other sequential modalities. Given the increasing importance of efficient tokenization strategies in large-scale multimodal models, we believe our work is highly relevant to the ICML community. \n\n**Q4:**  Essential References Not Discussed...\n\n**A:** We appreciate and agree with the reviewer that tokenization can also be applied to audio retrieval, further highlighting the potential of our research for multimodal tasks. We are pleased to discuss the application of tokenization methods (such as DUSTED and wav2tok) in audio retrieval in our revised version.\n\n**Q5:**  Contributions: I am not convinced about contributions to ML.\n\n**A:**  Reviewer can refer to Q4. \n\n**Q6:** What are query tokens? ...for speech synthesis?\n\n**A:** The reviewer may refer to Q1 for further details. For the text-to-speech task, the corresponding query tokens can be predicted based on textual conditions. Because we use RVQ to quantize the query tokens into discrete IDs, enabling them to be modeled by an LM-based audio generation framework.\n\n**Q7:** The paper does an impressive number of experiments...\n\n**A:**  We thank the reviewer for recognizing our esperiments and contributions. The theoretical contributions to ML can refer to Q4. \n\n[1] Devlin J, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL. 2019.", "claims_and_evidence": "- Several essential concepts are not properly explained. For example, the concept of query tokens is introduced on page 3 line 157 right, without any definition. Then, line 205 left says that [CLS] is a learnable query token. Overall the concept of query token as used in this work is not clear to me.\n- I have concerns regarding the novelty of the work. \n  - It seems the bit-rate is reduced by hyperparameter tuning (12.5Hz, 25 Hz, 50Hz). Please correct me if I missed something essential. \n  - Second, the semantic richness of tokens is achieved by query tokens, but the concept of the same is not clear to me.\n  - The use of transformers, MAE, VQ layer initialization and AR prediction loss seems novel but they contribute more to the audio processing side and would be a very good contribution to audio conferences and journals.", "methods_and_evaluation_criteria": "The paper uses a wide range of experiments and tasks to evaluate the proposed method. They aim at evaluating both compression efficiency (bit rate) and the semantic richness of the tokens. They appear alright to me.", "theoretical_claims": "As I mentioned above, several things, such as the concept of query tokens, are not clearly explained.", "experimental_designs_or_analyses": "The experiments have been performed on a variety of tasks and look good.", "relation_to_broader_scientific_literature": "It relates to the broader literature very well. Many state of the art tokenizers and encoders have been discussed.", "essential_references_not_discussed": "Tokenization is also used for audio retrieval. One may refer to works such as \"Spoken-Term Discovery using Discrete Speech Units, Interspeech 2024\" and \"wav2tok: Deep Sequence Tokenizer for Audio Retrieval, ICLR 2023\".", "other_strengths_and_weaknesses": "- Clarity of writing: the proposed method is not clear to me, in particular, the concept of query tokens.\n- Contributions: I am not convinced about contributions to ML. They need to be highlighted by the authors.", "other_comments_or_suggestions": "None", "questions_for_authors": "- What are query tokens? One can learn them from the data, but how are they used during inference. E.g., for text-to-speech, how do we know them; do we derive them from the given text and use for speech synthesis?\n- The paper does an impressive number of experiments to show the utility of the proposed tokenizer. But the theoretical contributions to ML need to be highlighted.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M7NyKHjm0e", "reviewer": "Reviewer_THJp", "summary": "This paper introduces ALMTokenizer, a codec for speech, music and sound, which incorporates semantic and acoustic information into a single hierarchy of residual tokens with remarkable performance at a very low bitrate. The proposed improvement over previous codec include both architectural changes and training tricks (MAE, AR loss) that can be combined or used separately. The really remarkable scope of experiments and the many ideas of various importance introduced in the paper make it such that neural codec researchers will likely read it several times. Thus, I recommend an accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. We do appreciate the constructive comments the reviewer provided to us to further improve our paper. We are delighted to have the following discussion with the reviewer.\n\n**Q1:** One of the claims is that the \"semantic priors\" avoids distillation as done by previous work. The semantic prior involves training a k-means on self-supervised embeddings and using the centroids as the fixed codebook of the first VQ. This is a form of distillation, yet less costly as during training one does not need to pass audio through a teacher embedding...\n\n**A:** We appreciate and agree with the reviewer that semantic priors can be regarded as a form of distillation. Their advantages include: (1) reduced training cost, as they do not require audio to be passed through a teacher embedding during training; and (2) the flexibility to integrate multiple teacher models, such as Wav2Vec2 for speech semantic priors and BEATs for general sound semantic priors. In contrast, previous methods like SpeechTokenizer and MimiCodec rely on a single teacher model and primarily focus on speech semantic priors, although they can be extended to both speech and general sound.\n\nWe will revise our previous claim that 'semantic priors avoid distillation' to clarify that semantic priors are indeed a form of distillation. Additionally, we will highlight their advantages and application scenarios is different with previous methods.\n\n**Q2:** Presentation of results: Human evaluations of audio quality are put in Appendix...\n\n**A:** We appreciate this comment and agree that human evaluation performance should be presented as a primary result in the paper. Accordingly, we will update our manuscript to include the MUSHRA score results in the main text.\n\n**Q3:**  L230: what does \"continuous\" mean in that setting? that it's continuously trained? This requires more details in particular I guess a stop gradient is applied to avoid degenerate solutions? \n\n**A:** We appreciate this comment. The term 'continuous autoregressive (AR) transformer' is used to distinguish our approach from traditional discrete AR models, which operate on discrete token sequences and are optimized using cross-entropy loss. In our study, to facilitate gradient backpropagation, we apply the AR transformer directly to continuous features (i.e., the quantized features) and optimize using mean squared error (MSE) loss. We will put these details in our final version.\n\n**Q4:** Also, does \"predicting the third VQ from the first and second\" mean that this small model is only autoregressive along the VQ axis or is it also autoregressive along time?.\n\n**A:** Yes, the AR model is only autoregressive along the VQ axis. \n\n**Q5:** L357: 12.5kHz -> 12.5Hz.\n\n**A:** Thank you for your help to find this mistake. We will update it in the final version.\n\n**Q6:** Also, Mimi uses Transformers in the encoder and the decoder, is this included in this ablation? The baseline described in \"The Effectiveness of Query-based Audio Compression\" seems to be purely convolutional. Making sure those transformers are included would demonstrate the usefulness of the query-based proposal wrt a simple transformer.\n\n**A:** We appreciate this comment. In our ablation study, The Effectiveness of Query-based Audio Compression, we compare our approach against a reproduced version of MimiCodec, which incorporates convolutional and transformer layers. The details of our reproduced MimiCodec implementation are provided in Appendix B.4. Table 7 presents the performance of reproduced MimiCodec at three different frame rates: 50 Hz, 25 Hz, and 12.5 Hz.", "claims_and_evidence": "Pros:\n- Claims are convincingly supported, at the exception of the one below. Overall, the experimental design is remarkably ambitious and represents the most extensive codec evaluation I have read so far.\nCons:\n- One of the claims is that the \"semantic priors\" avoids distillation as done by previous work. The semantic prior involves training a k-means on self-supervised embeddings and using the centroids as the fixed codebook of the first VQ. This is a form of distillation, yet less costly as during training one does not need to pass audio through a teacher embedding. However, this is never compared to the distillation used by SpeechTokenizer or Mimi and the baseline in the corresponding ablation is instead a codec without distillation, which expectedly performs poorly on semantic tasks. A proper comparison should rather be done with a model using distillation.", "methods_and_evaluation_criteria": "See \"Experimental Designs and Analyses\"", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Pros:\n- Extensive experiments across speech, music and sounds, both in discriminative and generative settings.\n- Interesting ablations and overall compelling experimental design.\n\n\nNegative:\n- Presentation of results: Human evaluations of audio quality are put in Appendix, while they show a worse performance for ALMTokenizer than for baselines. These results contradict claims such as \"ALMTokenizer achieves better reconstruction performance at lower bitrate\". Objective proxies for audio quality assessment (UTMOS, DNSMOS, MOSNet, etc.) are notoriously limited and provide a much weaker signal than actual human judgments. The MUSHRA scores should thus appear as main results of the paper, even if they depict a less positive result for the proposed model (they actually are quite good since ALMTokenizer outperforms all previous models at matching bitrates).", "relation_to_broader_scientific_literature": "ALMTokenizer introduces new ideas wrt previous work along two main axes: first, a new Transformer architecture that improves over fully convolutional codecs, and is an alternative to the Tranformers used by Mimi. Second, a set of additional losses (MAE, AR loss) to force semantic information into the learned tokens without the need for semantic distillation. Both contributions are somehow independent, and properly evaluated as such in the ablations study. Overall, the proposed methods are not groundbreaking but are conceptually sound and simple, and properly supported, such that I expect the community of neural audio codecs to build on it in the future.", "essential_references_not_discussed": "None, the references are quite complete, and pretty much every open source baseline has been included in the experimental pipeline.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "1) - L230: what does \"continuous\" mean in that setting? that it's continuously trained? This requires more details in particular I guess a stop gradient is applied to avoid degenerate solutions? Also, does \"predicting the third VQ from the first and second\" mean that this small model is only autoregressive along the VQ axis or is it also autoregressive along time?.\n2) L357: 12.5kHz -> 12.5Hz. Also, Mimi uses Transformers in the encoder and the decoder, is this included in this ablation? The baseline described in \"The Effectiveness of Query-based Audio Compression\" seems to be purely convolutional. Making sure those transformers are included would demonstrate the usefulness of the query-based proposal wrt a simple transformer.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQEGnHokFN", "reviewer": "Reviewer_4P9V", "summary": "The paper introduces ALMTokenizer, a low-bitrate and semantically rich audio codec. \n\nIt incorporates a novel fixed-interval query interleaving mechanism which extracts contextual features from the acoustic features and quantizes (using RVQ) only the contextual features extracted by these queries, thus achieving low bitrates, empowered by transformer based encoder and decoder. The decoder receives the queries and the interval, and inserts mask tokens, which are then converted into acoustic features. Additionally the VQ codebook vectors are initialized from k-means clusters of wav2vec2 and BEATS and kept fixed during training to provide strong semantic priors to the quantization module.\n\nTraining involves two stages, first the Encodec style patchify/unpatchify modules are trained along with some dedicated stage one encoder/decoder transformers without a quantizer, with MAE loss. The goal of the first stage training is to enable the frontend patchify module to learn semantically rich features. Second stage intializes the patchify/unpatchify with the stage 1 parameters and patchify is kept frozen. Second stage is trained with a combination of several objective functions including MAE loss, AR loss and the standard codec reconstruction and GAN losses. \n\nTraining and evaluation is conducted on a mix of speech, music and general purpose audio datasets. Evaluation results indicate strong performance on semantic tasks while being competitive in terms of reconstruction quality.\n\n## Update after rebuttal\nAfter reading the rebuttal and discussions with the all reviewers, especially the additional experiments in response to reviewer btwe, I am confident that this work is valuable to the neural audio codec community. I keep my Accept recommendation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contributions. \n\n**Q1:**  latent space optimized for AR modeling...it seems that not using the LM loss is beneficial for several metrics.\n\n**A:** We appreciate this comment. We acknowledge that introducing the autoregressive (AR) loss may slightly impact reconstruction metrics. As discussed in the Limitations section, we emphasize this trade-off to encourage future research on achieving a better balance between reconstruction quality and modeling efficiency.\n\n **Q2:** Methods And Evaluation Criteria:  However, there are many training/evaluation datasets utilized for different tasks ... mention which evaluation set it used ... \n\n**A:** We appreciate this comment. We will update our paper to explicitly specify the evaluation set used for all tables and figures in the final version. Furthermore, both Table 1 and Table 6 report results on the VCTK dataset. In Appendix Table 12, we present a reconstruction performance comparison between our proposed method and previous works on the LibriTTS test set. Additionally, we have reported results with 95\\% confidence intervals for subjective evaluations. For objective reconstruction metrics, since they are deterministic, confidence intervals were not previously included. However, for generation experiments, we will incorporate confidence intervals by sampling multiple times in the final version.\n\n**Q3:** I have one concern, both Encodec and DAC are used at 1.5kbps...potential as well.\n\n**A:**  We appreciate this comment. In line with MimiCodec, we will include results for Encodec and DAC in two configurations: (1) using 3 RVQ layers, as reported in our paper, and (2) using 8 RVQ layers to demonstrate their full potential.\n\n**Q4:** The training strategy seems to be too wasteful... initialize the depth transformer...\n\n**A:** We appreciate this constructive comment, which provides valuable insights for improving our work. In particular, leveraging the AR decoder to initialize the deep transformer in language modeling is an interesting idea. We find this direction highly promising and plan to explore it in future work. Additionally, we will incorporate this discussion into the final version of our paper to inspire further research in this area.\n\n**Q5:** The variable bitrate strategy is useful for the model as a pure codec...\n\n**A:** We appreciate and agree with the reviewer that the applicability of the variable bitrate strategy for downstream language modeling tasks remains unexplored. Our proposed method primarily facilitates the selection of codec models with different frame rates, providing greater flexibility in bitrate allocation. We will add this discussion into the Limitation part.\n\n**Q6:** Table 7, codebook size and FPS columns seem to be interchanged.\n\n**A:** Thank you for your help to find this mistake. We will update it in the final version.\n\n**Q7:** Line 216: \"we found ...of this statement?\n\n**A:**  In our experiments, we observed that a high masking rate negatively impacts reconstruction performance. We evaluated three masking rate ranges: 10–20\\%, 20–30\\%, and 30–40\\%. As shown following, higher masking rates (30–40\\%) improve semantic representation but degrade reconstruction quality. Based on these findings, we adopt an intermediate masking range of 20–30\\% to balance semantic preservation and reconstruction fidelity.\n\n| mask rate range | UTMOS  | DNSMOS  | VISQOL | PESQ | STOI  | ASR   | ER   |\n|-----------------|--------|---------|--------|------|-------|-------|------|\n| 10-20%          | 3.77   | 3.62    | 3.80   | 2.0  | 0.81  | 18.7  | 27.7 |\n| 20-30%          | 3.76   | 3.64    | 3.78   | 2.0  | 0.81  | 18.3  | 29.0 |\n| 30-40%          | 3.36   | 3.06    | 3.31   | 1.58 | 0.77  | 18.1  | 29.6 |\n\n**Q8:** I think the subjective evaluation ... it could be one automatic and the MUSHRA score.\n\n**A:** We appreciate and agree with the reviewer that subjective evaluation performance should be presented as the primary result in the paper. Accordingly, we will update our manuscript to include the MUSHRA score results in the main text.\n\n**Q9:** It is better to include an explanation of the wastefulness...\n\n**A:** As we discussed in Q4, we will incorporate this constructive discussion into our final version, stating the existing wastefulness and listing the potential solutions.\n\n**Q10:** The paper relies a lot on the Appendix...several details might be missed.\n\n**A:** We appreciate this comment. Since the final version of ICML allows an additional page in the main text, we will move more experiments from the Appendix into the main text, such as (1) Table 10 (the subjective evaluation results) and (2) Table 11, the LM-based sound and music understanding and generation results.", "claims_and_evidence": "The paper makes the following claims with regards to the codec:\n\n- **low-bitrate**: experimental results show that it works well, competitive reconstruction performance.\n- **semantic-rich**: experimental results show that it achieves good results on speech and emotion recognition, and sound classification tasks.\n- **latent space optimized for AR modeling**: while the experimental result shows that the token prediction accuracy improves by using the AR loss, the actual effect of it does not seem to be too significant in terms of speech generation tasks (Fig. 3). In fact, from the ablation study in Table 6, it seems that not using the LM loss is beneficial for several metrics.", "methods_and_evaluation_criteria": "Proposed methods and evaluation criteria do make sense for the application. \n\nHowever, there are many training/evaluation datasets utilized for different tasks and it is sometimes difficult to follow. Would be better for all tables/figures to mention which evaluation set it used. \n\nAlso, Table 1 and Table 6, both utilize the same metrics but are performed on different evaluation datasets, so it is difficult to contextualize the ablation study with the main result.\n\nMost of the results are reported without confidence intervals, which makes it difficult to judge whether the difference in metrics is statistically significant or not.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "Experimental design is mostly fine. \n\nI have one concern, both Encodec and DAC are used at 1.5kbps, which are the lowest settings of the respective models. While this is fair in terms of the number of RVQ layers, it would have been nice to see a comparison with their full potential as well.", "relation_to_broader_scientific_literature": "There has been significant recent efforts in creating low bitrate codecs which are easy to use for downstream language models. \n\nWhile the overall structure of the proposed method takes inspiration from these related works (using Encodec style patchify/unpathify, transformers before and after RVQ), the key novelty lies in utilizing interleaved query vectors to encode contextual information and only quantizing these query vectors. \n\nThis method also enables an alternative approach to variable bitrate by changing the frequency of the query interleaving, unlike previous methods which typically use variable number of RVQ levels (SoundStream, Encodec, DAC).", "essential_references_not_discussed": "Most essential related works have been discussed.", "other_strengths_and_weaknesses": "**Strengths**\n- Query token interleaving is a very elegant idea and it has been demonstrated that it works well.\n\n**Weaknesses**\n1. The training strategy seems to be too wasteful. Most of the components are eventually discarded, MAE-transformer encoder/decoder in stage 1, MAE-decoder in stage 2, AR-transformer in stage 2, etc. While I understand the purpose of them, but simply discarding is very wasteful in terms of compute and energy efficiency. It would have made sense to still utilize them for some purpose, for example, stage1 MAE encoder could have been used to initialize the enocder in stage2, and stage1 MAE decoder could have been used to initialize stage 2 MAE decoder. Similarly, the AR decoder (depth GPT) could have been used to initialize the depth transformer in the Language modeling task, etc. \n2. The variable bitrate strategy is useful for the model as a pure codec, but its usefulness for downstream LM tasks is not explored. The LM is trained with a fixed bitrate and I think, it cannot generalize to a different bitrate, while in the case of variable RVQ based tokenizers, the downstream model can also take advantage of the variable bitrate for tradeoffs in quality and efficiency.", "other_comments_or_suggestions": "1.  Table 7, codebook size and FPS columns seem to be interchanged.\n2. Line 216: \"we found that using a large mask rate will significantly influence the reconstruction performance\", for the better or worse? Is there an experimental verification of this statement?\n3. I think the subjective evaluation result should be included in the main content and not relegated to the Appendix. Maybe integrating in Table 1, and instead of reporting two automatic quality metrics (UTMOS and DNSMOS), it could be one automatic and the MUSHRA score.\n4. It is better to include an explanation of the wastefulness and energy efficiency of the training pipeline in a Broader Impact Statement.\n5. The paper relies a lot on the Appendix, especially for the experimental sections, unless the reader carefully goes through the Appendix, several details might be missed.", "questions_for_authors": "Please see above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary:
The paper proposes ALMTokenizer, a novel audio codec tokenizer that advances audio language modeling by addressing two key challenges: achieving low bitrate compression while preserving semantic richness in audio representations. Existing approaches either lose semantic detail or sacrifice compression efficiency. ALMTokenizer introduces:
- A query-based compression strategy that models cross-frame audio context.
- A combination of MAE loss, VQ initialization, and AR prediction loss to improve semantic representation and latent space usability.
- A two-stage training procedure that separates semantic learning and quantization optimization.
- Evaluations that span speech, sound, and music tasks across reconstruction, understanding, and generation, with SoTA performance.

Strength: 
- Innovation: The query-based compression mechanism is novel, effectively combining transformer-based context modeling with discrete token generation at low bitrates.
- Experiments: Strong results across speech, sound, and music datasets on several benchmarks, for several tasks.
- Ablation Studies: Systematic evaluation of each component in the overall approach, offering insights into what drives overall performance.
- Versatility: Demonstrates success across modalities (speech, music, sound) and tasks (understanding + generation).

Weakness:
- Complexity: While modular, the need to implement multiple components may hinder adoption of this framework.
- Gap in Performance: ASR results lag behind SSL-based models, suggesting area for improvements of this approach.

Based on reviewer feedback, discussion and AC assessment, this paper is recommended for Accept.
Decision: Accept (poster)