Discrete Audio Tokens: More Than a Survey!
Authors: Pooneh Mousavi
Gallil Maimon
Adel Moumen
Darius Petermann
Jiatong Shi
Haibin Wu
Haici Yang
Anastasia Kuznetsova
Artem Ploujnikov
Ricard Marxer
Bhuvana Ramabhadran
Benjamin Elizalde
Loren Lugosch
Jinyu Li
Cem Subakan
Phil Woodland
Minje Kim
Hung-yi Lee
Shinji Watanabe
Yossi Adi
Mirco Ravanelli
Venue: Accepted by TMLR ()

Abstract: Discrete audio tokens are compact representations that aim to preserve perceptual quality, phonetic content, and speaker characteristics while enabling efficient storage and inference, as well as competitive performance across diverse downstream tasks. They provide a practical alternative to continuous features, enabling the integration of speech and audio into modern large language models (LLMs). As interest in token-based audio processing grows, various tokenization methods have emerged, and several surveys have reviewed the latest progress in the field. However, existing studies often focus on specific domains or tasks and lack a unified comparison across various benchmarks. This paper presents a systematic review and benchmark of discrete audio tokenizers, covering three domains: speech, music, and general audio. We propose a taxonomy of tokenization approaches based on encoder-decoder, quantization techniques, training paradigm, streamability, and application domains. We evaluate tokenizers on multiple benchmarks for reconstruction, downstream performance, and acoustic language modeling, and analyze trade-offs through controlled ablation studies. Our findings highlight key limitations, practical considerations, and open challenges, providing insight and guidance for future research in this rapidly evolving area. For more information, including our main results and tokenizer database, please refer to our website: https://poonehmousavi.github.io/dates-website/.

Discrete Audio Tokens: More Than Survey!
Pooneh Mousavi*çL.2, Gallil Maimon*', *3, Adel Moumen**, *4 Darius Petermann**; Jiatong Shi*o; Halbin Wu**, Haici Yang*, Anastasia Kuznetsova**, Artem Ploujnikov Ricard Marxer', Bhu- vana Ramabhadran "0 Benjamin Elizalde"), Loren Lugosch", Jinyu Li", Cem Subakan 12,2.1, Phil Woodland', Minje Kim Hung-yi Lee Shinji Watanabe", Yossi Adi Mirco Ravanelli*.2.
'Concordia University, "Mila-Quebec AT Institute, Thee Hebrew University of Jerusalem, 'University of Cambridge, IIniiana University, CCrnegie Mellon University, "Microsoft, Université de Montréal, Université de Toulon, Googge,, "Apple LLaval University, 13 "National Taiwan University, "University of Illinois at Urbana-Champaign
Abstract
Discrete audio tokens are compact representations that aim to preserve perceptual quality, phonetic content, and speaker characteristics while enabling efficient storage and inference, well competitive performance across diverse downstream tasks. They provide prac- lical alternative to continuous features, enabling the integration of speech and audio into modern large language models (LLMs). As interest in token-based audio processing grows, various tokenization methods have emerged, and several surveys have reviewed the latest progress in the field. However, existing studies often focus on specific domains or tasks and lack unified comparison across various benchmarks. This paper presents a system- atic review and benchmark of discrete audio tokenizers, covering three domains: speech, music, and general audio. We propose taxonomy of tokenization approaches based on encoder-decoder, quantization techniques, training paradigm, streamabilily, and application domains. We evaluate tokenizers on multiple benchmarks for reconstruction, downstream performance, and acoustic language modeling, and analyze trade-offs through controlled ablation studies. Our findings highlight key limitations, practical considerations, and open challenges, providing insight and guidance for future research in this rapidly evolving area. For more information, including our main results and tokenizer database, please refer our website: hntps://poonehmousavi/ wwt.wb...g.t........... httdddate-weebsite
1 Introduction
Audio compression has been a well-established research topic since the foundations of digital commu- nication (Shanmon 1948 Nyquist 1928). Traditional audio codecs, such as linear predictive coding (LPC) [Itaurra] 1968 Atal 1970), modified discrete cosine transform (MDCT) (Wang Vilermo 2003). and Code Excited Linear Prediction (CELP) (Schroeder & Atal 1985 Jage & Upadhya 2016), were de-
signed to reduce redundancy and remove perceptually irrelevant information. These models have been effective in compressing raw audio signals into compact bitstreams (encoding) and then restoring them to the original signal domain (decoding)- Codecs like USAC (Quackenbush) 2013), Opus (Valin et al., 2012). and EVS (Dictz et al., 2015) combine these techniques to support a range of content types, bitrates, and sampling rates while ensuring low latency for real-time communication. These approaches rely heavily on domain knowledge, combining signal processing pipelines with hand-crafted components achieve efficient but lossy compression.
Traditional codecs are elficient and optimized for perceptual quality, but their design requires substantial manual effort, including parameter tuning and subjective listening tests (Valin et al. 2012 Dietz et al. 2015). This has motivated a shift toward data-driven approaches with deep learning, known as neural codecs. Neural codecs consist of an encoder, decoder, and a quantization module, closely resembling standard
Prroject lead, Corresponding author (pcanel.moossvvi@mallloooooo 'Equal contribution, Core Team.

---

autoencoders. The key difference is that neural codecs produce discrete representations (audio tokens) instead of continuous ones. The discretization is performed by a differentiable quantizer, such as residual vector quantization (RVQ) (Zeghidour et al.) 2021), which enables end-to-end training by allowing gradients to propagate through the quantization step. Neural codces are often trained using a combination of losses. For example, reconstruction losses the time and frequency domains (Kankanahalij 2018), optionally combined with psychoacoustic calibration (Zhen et al., 2020), direct guide signal reconstruction. Adversarial losses (Zeghidour et al. 2021) and generative models (Kleijn et al.) 2018 Valin & Skoglund| 2019b Gārbacea
et al.
2019 indirectly improve the perceptual quality of the reconstructed signal. Finally, auxiliary losses are often introduced to improve the learning process and often act as regularizers or encode inductive bias (Zhang
et al.
2024a Défossez et al. 2024 Har- Tar-Tuu et al. 2025
Discrete tokens have several useful properties. As they are normally compact, audio tokens enable more efficient storage and transmission than continuous embeddings. They also simplify audio generation by converting tasks that involve modcling continuous distributions, such as regression, into discrete classification problems (Wu et. al. 2024f Mousavi et. al.
2024a). More importantly, they help bridge the gap between text and audio processing, making them natural choice for multimodal models and core component of many
recent multimodal LLMs (Pong et al.
2024 Cui et al. 2024 Ji et al. 2024a Latif et al. 2023 Liu et al.
2023 Wu et al.) 2024a Tian et al.
2025). ). Driven by these advantages, discrete audio tokens have already
been adopted as an alternative to contimuous features in a wide range of downstream tasks: automatic speech
recognition Chang et al.
2023
Du et al., 2023), speech-to-speech translation (Popuri et al.) 2022; Inaguma
et al.
2023 Wu et al.,
2023a
Chang et al.J 2004)) voice conversion (Maimon & Adil 2023 Wang et al.
2024d text-to-speech synthesis
(Ju et al. 2024; Chen et. al.; 2025a Hayashi & Watanabe 2020), speech
enhancement (Wang ct al.
2024c Yang et al. 2024f Xue et al. 2024), and source separation (Shi et al.,
2021c
Erdogan et al. 2023
Mousavi et al. 2024b Bie et al.] 2025
Yip et al. 2024 Discrete tokens are also
used in music and general audio tasks, including music generation
(Copet et al.) 2023 Chen et al. 2024a
environmental sound synthesis (Yang et al.) 2023b Kreuk et al.
2023). and multimodal generation Borsos
et al. 2023b Liu et al. 2023 Ziv et al. 2024 Rubenstein et al.
2023 Wang et al. 2024b
Recent studies have introduced a variety of tokenization methods, often grouped into two main categories:
acoustic and semantid" (Borsos al. 2023b Zhang ct al. 2024a Har-Tuv ct al. 2025 Guo et al. 2025b)
Acoustic tokens are typically learned through encoder-decoder architectures optimized for waveform recon-
struction ZZeghidour et al.] 2021: Défossez et al. 2023 Kumar et al., 2023 Yang et al. 2023a Semantic
tokens are derived from pretrained self-supervised learning (SSL) models (Lakhotia et al. 2021 Mousavi
ct al. 2024b), which are trained on raw audio without labels by solving proxy tasks (c.g., masked prediction)
to learn transferable representations to downstream lasks, or encoders trained in a supervised manner (Du ct al. 2024b) designed to capture phonetic or linguistic content for discriminative tasks such as speech recognition and translation. Some recent approaches aim to combine both types, introducing hybrid tok-
enizers [Zhang et al. 2024a Défossez et al. 2024) that balance acoustic and phonetic properties.
We argue the common division of discrete tokens into acoustic and semantic categorics has notable limita- tions.
Acoustic tokenizers can capture semantic information (Défossez et al., 2024 Du et al. 2023 Zhang
et al.
2024a Bai et al., 2024). while semantic tokenizers have been effectively used in generative tasks Polyak
et al.
2021 Wang et al. 2024e Nguyen et al. 2025 Lakhotia et al., 2021 Maimon et al.) 2025a Hassid et al.,
Mousavi et al. 2024b Wu et al., 2025). This overlap blurs the boundary between the two categories and suggests that the acoustic-semantic distinction alone is insufficient. Moreover, as tokenization hods continue to evolve, traditional classifications fail to capture key architectural differences and practical trade- offs. To address this limitation, we introduce refined taxonomy that captures key design choices, including mecoder-decoder quantization techniques, training paradigms, streamability, and application domains.
2023
Another notable gap in the literature is that existing surveys and benchmark papers have primarily focused on
speech applications (Cui et al. 2024 Kim & Skoglund 2024 Ji et al. 2024a
Anees 2024 Guo et al. 2025b
It is important to clarify that the term "semantic" in the speech context does not aliga with its conventional linguistic: meaning. In the speech context, these discrete tokens are more accurately described as phonctic units (Sicherman & Adi 2023
Choi ct al. 2024 and typically do not carry semantic content (Arora al. 2025). In this paper, maintain consistency across different domains (speech, audio, music) and with established terminology such "semantic distillation," we consistently use the term "semantic."
2

---

IC:
Intent
Tokenizer Training
Acoustic Language Models Evaluation
Classification
SE:
Speech
Sampling Rate
Training Data
Text Conditioned
Unconditioned
Enhancement
SS:
Q + 4
Source
Separation
ASR:
Automatic
ESPNet Codec
Zero Resource
SALMon
Speech
Recognition
SV:
Speaker
Verification
Downstream Evaluation
Reconstruction Evaluation
ER:
Emotion
Recognition
KS:
+J3
Signal-Level Metrics +
Application- Level Metric
Keyword Spotting
SEC:
IC
SE
SS
SEC
GC
4
Event-Audio
KS
Classification
ASR
ASS
MSS
ASS:
Audio Source
VERSA
Codec Superb
Separation
GC:
DASB
Genre
Classification
MSS:
Music Source
Encoder
Decoder
Separation
Audio Tokens
Figure Overview our empirical study, covering three domains: speech, music, and general audio, with four evaluation components: Downstream Evaluation (Section 3.2 using the DASB benchmark, Re- constructed Audio Evaluation (Section 3.1] using Codec-SUPERB and Versa, Acoustic LLM Evaluation (Section 3.3 using SALMon and the Zero-resource benchmark, Tokenizer Training Ablation Study (Section using ESP neteeCCodec.
Arora et al. 2025 Vashishth et al. 2024h. often overlooking tokenization methods for music and general audio. As result, the current literalure lacks unified study that covers multiple domains and diverse evaluation criteria. Moreover, rather than providing a holistic comparison, most existing works focus on single aspect, such as reconstruction quality Codee-SUPERB (Wu et al., 2024c b) downstream task performance in
DASB (Mousavi et al., 2024a), controlled evaluation settings in ESP DSPnee-Codec (Shi et al. 2024c), or audio
language modeling in SALMon MMaimon al. (025c).. These limitations persist even in the latest surveys.
For example, Guo al. (2025b) focuses reconstruction and voice conversion, while Cui al. (0024) Peng
et al. 22224) explores integration with LLMs. To help bridge this gap, present comprehensive benchmark of discrete audio tokenizers. Our benchmark covers three audio domains: speech, music, and general audio. It considers multiple evaluation criteria, including signal reconstruction, downstream task performance, and acoustic language modeling. Those aspects are analyzed jointly to provide a more robust and comprchensive assessment. An additional issue in current benchmarks is that tokenizers are often trained under inconsistent conditions, such as different datasets, domains, or sampling rates. These inconsistencies make direct and fair comparisons difficult. To ensure fair comparisons, we support our analysis with ablation studies that examine different quantization methods under controlled experimental settings.
Our contribution is organized into three core studies, as illustrated in Figures 1 and g
Study 1: Audio Tokenizer Taxonomy (Section 2). We propose a comprehensive taxonomy of discrete audio tokenization methods based key architectural and functional criteria.
3

---

Discriminator
LGAN, L Feats
Encoder
Decoder
100--111)))) Ii 11- pl
Quantization
x
zt
Q()
2t
LVQ
L Recon
Figure 2: Overall architecture a standard audio tokenizer. The input signal x is encoded into a latent representation 2t, which is then discretized by a quantizer Q(-). The decoder reconstructs the signal à from the quantized representations &. Training typically involves combination of roconstruction (LRecon): adversarial (LGAN, Leetss), and vector quantization losses (Lvg).
Study 2: Benchmark Evaluation (Section 3). We evaluate existing lokenizers using multiple benchmarks. Codec-SUPEREP and VERSA Shi al 2025 are used for reconstruction. DASE is used for downstream tasks. SALMor and the Zero-resource speech benchmarl (gguynn et al. 2020 are used for acoustic langnage modeling. All evaluations are conducted under consistent conditions.
Study 3: Ablation Studies (Section We perform controlled experiments to isolate the effects of specific design choices for training audio tokenizers, including sampling rate and single-domain versus multi-domain training using ESPnct-Codd Shh et al. 2024c).
This survey provides unified and practical perspective discrete audio tokenization and its role in speech, music, and general audio processing. We aim to clarify key design tradc-offs, highlight current limitations, and offer guidance for future research in this evolving field.
2 Literature Review and Proposed Taxonomy
2.1 Overall architecture
As shown in Figure 2 audio tokenizers typically comprise three components:
An encoder that converts the input waveform into sequence of frame-wise emboddings Z (x using encoder function Se, such that, Z fe(x), where each E R is a continuous embedding at time step t.
A quantization module that maps each embedding 2 to quantized vector & and set of discrete indices de = [q,,t.... (M,,] using a quantization function Q, i.e. (2,, = Q(++). Here, M denotes the number of codcbooks used in the quantizer. The full sequence of quantized embeddings is denoted as 2 = (-
httper//oodecuuerbbggtbbbbbbb
https: http:///ttt........a////////////
https:/ http:///ppmaahccaarr....paoooosa
htpp:///ttabb.ccc//ppptt///pppp

---

A decoder that reconstructs the waveform from the sequence of quantized embeddings using decoder function la, i.e., & = fa(2), where & is the reconstructed waveform.
To address the taxonomy issues outlined in the introduction, this section proposes a refined taxonomy based three core dimensions: the quantization method, the encoder-decoder architecture, and the training paradigm (e.g,, joint end-to-end training, and the use of auxiliary components). We also provide more fine-grained categories, including streamability and the target domain of cach tokenizer. The proposed laxonomy is illustrated in Figure 3 and the classification of existing audio tokenizers according to this taxonomy is shown in Table ! to the proposed categorization.
The following subsection summarizes the most popular methods according
Architecture
Representation
CNN RNN
CNN f
Audio Tokenizer
Encoder-
CNN
RNN
Icon Legend
Algorithi
Bitratt
Whff
Waveform Domein
Quantization
RVQ
GVQ
SVQ
RVQ
RVQ
PQ
FSQ
K- Means
T-F Domain
Disantenglement
Objectives
Auxiliary
Joint
Semantic Distillation
Training
GAN
Diff
MP
Feats
Recon
VQ
X
Supervised Semantic,
Speech
Music
Target Domain
General Audio
Fixed Bitrate
Adaptive Bitrate
Streamability
Figure 3: Taxonomy of audio tokenizers based on: encoder-decoder architecture (Section 2.3 quantization training paradigm (Section 2.4), and target domain and streamability (Section 2.5 CNN denotes Convolutional networks, represents Transformer models, and RNN refers to any recurrent neural network including LSTM and GRU. RVQ stands for Residual Vector Quantization, GVQ for Group Vector Quantization, SVQ for Single Vector Quantization, MSRVQ stands for Multi-Scale Residual Vector Quantization, CSRVQ stands for Cross-Scale Residual Vector Quantization, PQ stands for Product Quan- tization, FSQ for Finite Scalar Quantization. K-Means signifies that the tokenizer is trained independently of the encoder and the decoder pipeline. Objectives include adversarial learning (GAN), diffusion- based generation (Diff), and masked prediction (MP) as a generative training strategy, feature matching loss (Feats), and reconstruction loss (Recon). The interactive version of this figure can be accessed through https://dates tokessttoens..o github. 10/ httpa///aaaoooooyoitttrtrtttttee wii.ieraattieeetmm.
method (Section 2.2
2.2 Quantization Method
Quantization key component of the tokenization pipeline, transforming continuous frame-wise features (vectors) zt RD into discrete tokens &r [9..s..., and corresponding quantized vectors &, E RRP More formally, quantization maps data into smaller representation space with lower cardinality. It is defined two-step procedure involving encoding and decoding, which distinct from the encoder and decoder modules described in Figure 2 The encoder of the quantizer, denoted Eg RD + f1...., KYM, maps a continuous embedding z to a tuple of discrete indices (t, where each Qm.t E Im (1,2, K} corresponds to a quantization layer (codebook) m. These indices refer to entries in a set of M codebooks = {C) .c......cc CM. where each codebook Cm contains K learnable D dimensional continuous vectors. The decoder of the quantizer, Dy B.....K)M + R², reconstructs the quantized embedding &, by retrieving the selected codewords from the codebooks and combining them, typically via averaging or summation.
5

---

Table 1: Comprehensive overview audio tokenizers, organized alphabetically by tokenizer name. The table covers core design choices across five major dimensions: application domains (Speech, Music, Au- dio), encoder-decoder architecture (including encoder /ecoder type and feature representation), quantiza- tion (technique and bitrate strategy), training paradigms (objective, auxiliary loss, and joint optimization). and streaming capability.
Tokeniser
Domain
Frame
Encoder- Decoder
Quantisation
Training
Stream
M
Rate
Architecture
Tesh.
Dtt.
Objeetive(s)
Aux.
Join
Dosader
APOode
of al.
2024
150
CNN
CNN
T-F
RVQ
GAN, Rod, Rev, VQ GAN, Foat, Ree,
AedioDee
Wa
2724b
CNN
CNN
RVQ
Dest RQ
Chiu ct
2022
CNN+T
T.F
PQ
MP GAN, Fost, Rec, GAN, Post, Rec, VQ
BodCodec
Xin
2721
CNN+RNN CNN+HXN
SVQ
DAC
Kumar et Ab
2023
CNN
CNN
RVQ
Discrete 5SL
Mousari al.
20215
CNN+T
K cacons
GAN, Feot, Rec, MP
Diern:
TF-Coder
Jiang ot
2023
CNN+RNN
CNN
T.
GRVQ
GAN, Pest, Bec, (Pred) GAN, Pat, Ree, VQ
Di
dMel
Bai al
21224
CNN
T-F
SVQ
F
EnCadoe
150
RVQ
GAN, Fost, Roe, VQ
ESC
Cu Ar Dino
A024
150
T
T-F
CSRVQ
F
Ree, VQ
FACodee
2024
CNN+RNN CNNN
GRVQ
GAN, Fost, Ree, GAN, Feat, Ree, VQ
FunCodee
Du et al.
20125
1,25, 25, 50
CNN+RNN CNN+ T-F
RVQ
F
SD
Potermani
2021
FSQ
Hiki-Codee
Yong 06 al.
2023k
50, 75, 100
UNN+RNN CNN+RNN
GRVQ
GAN, Hod, Ree, VQ Rat,
HILCCdde
CNN
RVQ
Yang et all
30
CNN
ONX
RVQ
THI
Languago Coolon Ji
2034
ONN+RNN
ONK
T.T-P
RVQ
GAN, Rat,
LPSC
Chooenos ct al.
X725
21.5
CNN
ONX
T
FSQ
GAN, Fost, Roe
8D
LLMCadee Yang
3724b
CN
ONN
MS-RVQ
GAN, Roe
LSCodec
Grao et ml.
2025a
25, 30
CNN
CNN
SVQ
GAN, Fast, Rec
Dis.
MDCT Codes
Biang
2024
CNN
CNN
TF
RVQ
GAN, Fost, Rec,
Mini
Difossees et
2024
12.5
CNN+T
CNN+T
T
RVQ
GAN, Fot,, Rec, VQ
MMM
Shi ml.
2024b
CNN+T
CNN
K-rreons
GAN, Fest, Reg, MP
NAST
Mresica
Adi
2024
CNN+T
CNN+T
FSQ
Ros, VQ, MP
NDVQ
Niu al.
2024
75
RNN
CNNN
RVQ
F
GAN, Foat, Ree, VQ
RVQ
GAN, Fost,
PQ-VAE
Grao al.
2024h
73
CNN
CNN
T.F
PQ
F
Rec, VQ
Prompt
Pan
2024
CNN+RNN GRVQ
GAN, Fost, Rec, VQ. MP
RepCodes
Huang ct a3
2024
50
CNN CNN
T
SVQ
F
NeT
Jang
CNN+RNN
CSIVQ
GAN, Rec, VQ
89
Do et al
20246
T
T
SVQ
DHt
88T
SD-Codre B3e
ONN
CNK
RVQ
GAN, Rea, VQ
SedoatiCodoe Lt all
28244
CN
T
RVQ
Dif, VQ
SD
Single Codoe
3724
CNN RNN
CNN RRNN
T-F
SVQ
CAN, Roe, VQ
SingOM
Tong ot al. 3724b
CNN
CNX
K-Mosa
CAN, Fost, Rec, MP
SNAC
Stuadok
Vartable
CNN RNN
CNX
MS-RVQ
GAN, Foat,
SOCODEC
Goo ei al.
2024a
CNN
CNN
T-F
PQ
GAN, Roc, VQ
SoundStresm
Zeglidoor
2021
CNN
CNN
RVQ
GAN, Fost
Specinll Codecs
Langnon e6
2024
86.1
CNN
CNN
T-F
FSQ
GAN, Ficst, Roc
Tokenio Zhong ml
2023ka
UNN+RNN
CNN
RVQ
GAN, Nott VQ
SD
SQ-Codeee Yang mi
302441
CNN
FSQ
GAN, Rie
TAAR
Parker ml
2025
CNN+T
CNN+T
FSQ
GAN, Fost, Roe
SD
Taug et
2022h
GRVQ
Bec, VQ
Th Codex
Ren et. 2024b
75
CNN+RNN
CNN+ T
RVQ
F
GAN, Feat, Bec, V Bec,
D2s
Wia
20244
30
SVQ
USM
Zhang et
2025
25
CNN+T
T.F
PQ
F
MP
Standak
2024
RVQ
Fest,
Wiw Tokenises
Ji ot
X0246
40, 75
CNN
ONN FT
T
SVQ
GAN, Rost, Roe,
WiwZVo--EETT
Chunng
2021
CNN
PQ
MP
WMCodee
Zhoo et al
2014
76
C
CNN
T
RVQ
GAN, Fost, Roe
X- Codes
Ye
A025
CNN
CNN
RVQ
GAN, Rec, V
XEUS
ben al.
M224
CNN+T
T
K-mcans
MP

---

Depending on the design choices, quantization methods vary along two important axes: (1) the specific quantization algorithm used to convert continuous features into discrete tokens, such as k-means, product quantization, or residual vector quantization (RVQ), (2) whether the bitrate is fixed or adaptive. These aspects are described in the following subsections.
2.2.1 Quantization Algorithm
K-means. K-means clustering is frequently used for post-training quantization. While many recent codec- based acoustic tokenizers tend to adopt the joint-training quantization techniques, k-means is still prevalent in extracting tokens from SSL models (Mousavi et al., 2024b; Chang et al. 2023 Polyak et al. 2021 Wang et al. 2024e). Typically, a layer or multiple layers from a pretrained SSL model are selected, and representations ions are clustered using offline trained k-means to create discrete tokens. Such tokenizers natively lack built-in decoder, as they are primarily usod for discriminative tasks like ASR. Nevertheless, recent studies have investigated training separate decoders to reconstruct speech from discrete representations, such employing modified HiFi-GAN (Yang et al 2023a). Additionally, Mousavi et al. (2024b) introduced multi-layer training strategy with dropout mechanisms, enabling the decoder to flexibly handle varying bitrates during inference. The assignment of each embedding to its nearest centroid oj is performed using the standard K-means rule:
Q: arg min ke K.....KK
||ze ckll
(1)
Here, 2 RD denotes the continuous embedding at time step from a frozen SSL model, ck is the k-th cluster centroid, and qr [I....,KK the resulting discrete token index.
Residual Vector Quantization (RVQ). RVQ maps cach frame-wisc fcature to the closest entry in a codebook and then refines this process by computing the residual after quantization. The remaining residual is compressed by sequentially applying a series of quantizers, each refining the residuals left by the previous one. The first neural network-based RVQ method was first introduced SoundStream (Zeghidour et al. 2021) and has since been widely adopted in other models (Kumar et al. 2023 Délossez et al., 2023; 2024
Zhang et al. 2024a). Many approaches (Kumar et al., 2023; Délossez et al.] 2023 also incorporate bitrate scalability by performing variable bandwidth training, where the number of codebooks is randomly selected during training to support different bandwidths during inference. A variant of RVQ, called Residual Normal Distribution Vector Quantization (RNDVQ), is used in (Niu et al. 2024). Unlike standard RVQ, which selects the nearest neighbor deterministically, RNDVQ. formulates quantization as a probabilistic selection problem. This addresses issues such as low codebook utilization and sensitivity to noise, making the model more robust minor variations in input data. The procedure defined recursively as:
Algorithm Residual Vector Quantization (RVQ)
1: Input: Embedding 2t. Codebooks {C(m) M
Initialize residual: r + 2t
(1)
for m = to M do
4: (m) +- arg mink
(m) (m) rr
2
ck
5: (m)) + c (m)
9e
6: ( (m+1) +- (m))
{())
7: end for
8: Output: & --
2
(m) is the discrete index selected from the m-th codebook, and 5, is the final quantized vector produced by summing the quantized outputs from each residual stage.
Here, 2, is the input embedding at time t, qi
7

---

Single Vector Quantization (SVQ). SVQ uses single codebook for quantization, where each frame- wise embedding is mapped to single code, unlike RVQ, which uses multiple codes. SVQ can be viewed as a simplified form of RVQ with a single codebook (i.e., M=1), without iterative residual refinement, similar to VQ-VAE (Gârbacea et al.) 2019). It has gained popularity due to the architectural complexity introduced by multiple codebooks acoustic language models, such as managing multiple codebook streams. SVQ, by contrast, simpler and particularly useful for training acoustic language models. To compensate for the potential loss of information caused by using a single codebook, some SVQ-based codec models adopt larger codebook sizes. Examples of bysee codecs include BigCodec (Xin et al. 2024), TS3-Codec (Wu et al. 2024d), WayTokenizer
JJi et al. 2024c).
Group Vector Quantization (GVQ). One limitation of RVQ is that most of the information tends to be captured in the first-layer codebook, with later codebooks contributing minimally. To address this, GVQ YYagg et al.] 2023al) increases capacity at the first quantization stage by dividing the latent feature vector 21 E R into G non-overlapping groups:
2t
(1) = 24
|| [12((
(2)
where each 200) E D/G represents a segment the input feature, and || denotes concatenation. Each group is quantized independently using a separate RVQ module, producing a group-wise quantized embedding 2(9) The final quantized vector is formed by concatenating the quantized outputs from all groups:
(1) ( 1.. [
a =
(3)
This grouped structure improves performance while reducing the mumber of required codebooks.
Finite Scalar Quantization (FSQ). Unlike traditional vector quantization, FSQ maps each dimension of a feature vector to a fixed set of scalar values (Mentzer et al.) 2024 Specifically, each feature value 2; is first squashed into the range [-1, using a non-linear function such as tanh, and then quantized into a scalar latent space by computing round(2; S)/S, where S is a hyperparameter controlling the quantization resolution. This procedure results in 2S + 1 distinct scalar values per dimension, ensuring uniform coverage of the latent space. FSQ has been adopted in various recent models. SQ-Codec (Yang et al., 2024d) achieves this by creating a scalar latent space, while Spectral Codecs (Langman et al. 2024 use FSQ to encode mel-spectrogram features into flat codebook. FSQ is often used with diffusion models for high- quality audio generation. HARP-Net (Petermann et al., 2021 similarly applies FSQ but directly maps bottleneck features i.e,, single scalar values rather than vectors, to a set of learned scalar bins. Unlike other approaches, HARP-Net maintains the original input frame rate (44.1 kHz) by avoiding temporal decimation, instead expanding the feature dimension in intermediate layers before collapsing to scalar quantization. FocalCodoe (Della Libera et al.) 2025 instead uses variant of FSQ called Binary Spherical Quantization (BSQ), which relies on two scalar values.
Multi-Scale RVQ (MSRVQ). MSRVQ (Suzzdak et al. 2024 extends standard RVQ by applying quan- tizers at different temporal resolutions. This hicrarchical structure enables the model to efficiently capture both coarse and fine-grained details. The initial VQ layers operate at higher frame rates to encode fine details, while later layers work at lower temporal resolutions to refine the residuals using fewer tokens. At each stage i, the residual r/ is downsampled by a factor W,, quantized, and then upsampled back to length T:
( = Upsample (Q (Downsample(r!") (6) (W)))
(4)
This strategy reduces the number of tokens while preserving essential information in the representation.
Cross-Scale RVQ (CSRVQ). CSRVQ (Gu & Diao 2024 Jiang et al. 2022a extends RVQ by intograting multi-scale features that progressively encode coarse- to fine information. Unlike conventional RVQ, which applies residual quaniization only at single and lowest-resolution layer, CSRVQ encodes residuals bet
8

---

encoder and decoder features at multiple hierarchical levels. During decoding, each level is conditioned on quantized residuals from coarser scales, allowing the model to refine reconst in coarse-to- tine man- ner. This structure enables the preservation of low-level detail often lost in high-level-only representations. In practice, CSRVQ can include quantization modules across different decoder layers, with cach layer incor- poraling its own quantizer. ESC (Gu & Diao, 2024) adopts this design via hierarchical transformers and stepwise decoding, progressively improving reconstruction fidelity without requiring extra. fusion networks between encoder and decoder.
Product Quantization (PQ). Product quantization is commonly used in supervuee learning (SSL) models to discretize continuous speech reprosentations. PQ can be viewed as group of independent vector quantization modules (Chung et al., 2021 Guo et al. 2024a), partitioning embeddings into smaller sub- vectors and quantizing each separately. The quantized sub-vectors are then concatenated to form the final output. Other variations include Random- Prooetttoooooooonon Quantization, as seen in models like Best-RQ and USM (Chiu et al. 2022] Zhang et al.) 2023). This method maps speech signals into discrete labels using randomly initialized projection matrix. Unlike other quantization methods, most PQ-based approaches do not have built-in decoder, as they are primarily designed for SSL models rather than direct waveform reconstruction.
2.2.2 Fixed vs. Adaptive Bitrate
Depending on the system design, the bitrate of quantized representations can be either fized or adaptive. In fixed-allocation schemes, such as those based codebooks, the bitrate is determined by the number of bits required to represent each code index, irrespective of the actual loken distribution. In contrast, aduplive bitrate refers to entropy-based coding schemes that assign variable-length codes based on the statistical fre- quency of tokens (Agustsson et al. 2017 Kankanahalli 2018). More frequent tokens are encoded using fewer bits, while rarer tokens require more, leading to improved compression efficiency. Standard methods such as Huffman coding or arithmetic coding are commonly employed exploit this redundancy. Importantly, any quantization method, regardless of its original design, can benefit from post-hoc entropy coding to further reduce the effective bitrate. It is also important to distinguish between adoptive bitrate and scalable bitrate. Adaptive bitrate dynamically adjusts the number of bits per token according to the token distribution (e.g., via entropy coding (Jiang et al. 2023: Petermann et al. (2021) In contrast, scalable bitrate refers to systems capable of operaling at multiple fixed bitrates, typically achieved by varying the number of active codebooks. This bitrate level is selected manually or defined as a hyperparameter, but it remains fixed per run and does not adapt token-wise at runtime. For instance, Encodec (Défossez al. 2023) enables scalable bitrate by employing a codebook dropout strategy during training.
2.3
Encoder-Decoder
This section describes the main encoder and decoder architectures, along with the encoder input and decoder output representations used across different designs.
2.3.1
Architecture
Convolutional (CNN). CNN extracts and downsamples audio waveforms into lower framo-rate features using CNN layers. CNN is the most widely applied archilecture among early neural codecs (Kumar el al. Zeghidour al. 2021). CNN models are generally more compact, thus can be readily integrated with different system sizes and are especially useful in resource-constrained environments. However, CNN tokenizers cannot capture long-range dependencies.
2023
Convolutional + RNN (CNN+RNN). Some tokenizers Défosscz et al. 2023 Xin et al. 2024h combine CNN-based feature extraction with LSTMs or GRUs for sequential modeling. RNN provides mechanism for longer-range dependency than CNN, although it can still suffer from memory loss when the sequences reach a certain length. RNNs can easily add algorithm and system complexity to both training and inference. Therefore, the number of RNN layers used in tokenizers is usually small, and they are combined with CNN modules.
9

---

Transformer (T). This category refers to the fully transformer-based models without convolutional com- ponents (Wu et al. 2024db. This lype of tokenizer is less common than others. They may achieve impressive compression and reconstruction performance, but they generally demand a large amount of training data and much hcavier computational resourccs, which limit their practicality in tokenization.
Convolutional + Transformer (CNN+T). This group of tokenizers (Mousavi et al. 2024b Chiu et al. 2022 Yang et al. 2024b) uses CNN-based feature extraction followed by attention mechanisms to capture long-range dependencics. Transformers have recently started to appear in audio tokenizers, as replacement for RNN, due their effectiveness in capluring long-range dependencies.
2.3.2 Input and Output Representations
Encoders can process audio inputs in either the time domain or the frequeney domain. In the time domain approach, raw waveforms are directly passed to the encoder. In the frequency-domain approach, precomputed mel-spectrograms or other spectral features are used inputs. The output representation can follow two approaches: (1) time domain waveforms, where the decoder directly upsamples the discrete representation into waveforms (Défossez et al. 2023); or (2) Lime-frequenncy domain features, where the decoder outputs sime-freqeent domain features (Siuzdak 2024), and the Inverse Short-Time Fourier Transform (ISTFT) is applied for upsampling. In this case, the decoded features typically have a frame rate similar that of the codec tokens. SSuzdak (2024) argues that assigning the upsampling to the ISTFT reduces the burden on the decoder and leads to better performance.
2.4 Training Paradigm
In this section, we discuss three key dimensions of audio tokenizer training: the training strategy, the main training objectives used to optimize the model, and the auxiliary components that further enhance the learned reprosentations.
2.4.1 Training Strategies
Training strategies for audio tokenizers can be categorized into two broad approaches: separate (post-training) and joint (end-to-end training). These differ in how the encoder, quantizer, and decoder modules are optimized in relation to each other. In both cases, the encoder may be randomly initialized initialized using a pretrained model, such SSL model (e-g., wav2vec 2.0 (Baevski et al. 2020). HuBERT (Hsu et al.
2021 WavLM (Chen et al. 2022 or an ASR model (Radford et al. 2023)
In this approach, the encoder and decoder are optimized independently from the quantization module. This is common in semantic tokenizers and earlier neural codecs. The encoder is often initialized from pretrained SSL model such as wav2vee 2.0 (Bacvski et al. 2020), HuBERT (Hsu let al. 2021 , or WavLM (Chen el al. 2022), and typically kept frozen during quantizer training (Lakhotia Shi et al. 2024b). The quantizer is trained offline using methods
Separate PPost-Trainng
let al. 2021 Mousavi et al. 2024bb
such as k-means clustering on latent representations. Discrete SSI. (Lakhotia et al., 2021 Mousavi et al.
2024b Shi et al. 2024b), for example, uses k-means clustering quantize the latent semantic features after pretraining. T.PNNet-based @ (alin & Skoglund 2019a b Yang et al. 2023c) a combination of scalar quantization and multi-stage vector residual quantization the pre-extracted features (pitch and cepstra) with k-means, involving different levels of feature predictions. LAST (Turetzky & Adi 2024] uses VQ to quantize adapted SSL features, with the objective of improving SpeechLM, and then separately trains a HiFi-GAN based vocoder. p-llaw is also a popular quantization technique used earlier autoregressive vocoders (van den Oord et al.] 2016),, and some neural audio codecs (Kleijn et al. 2018). Decoders are then trained independently reconstruct waveforms or features from discrete tokens, often using HiFj-
GAN (Polyak al. 2021 Kong al. 2020 diffusion models DDu al. 2024b Zeng et al. 2024
10

---

Joint (End-to-End Training). In joint training, the encoder, quantizer, and decoder optimized simultaneously within a unified end-to-end framework. This approach is commonly adopted by acoustic tokenizers (Zeghidour et al. 2021 Défossez et al. 2023). The full model is optimized using a combination of reconstruction losses (c.g., MSE) and often adversarial losses (Goodfellow et al. 2020 to promote both signal fidelity and perceptual quality. To address the non-differentiabbiity of quantization, several gradient approximation techniques are used: (1) straight-through estimators (van den Oord et al.] (017); which
copy gradients across the quantizer; (2) soft-to-hard quantization with annealing (Agustsson al. 2017
Kankanahallij 2018); and (3) Gumbel-softmax relaxation (Jang al.] 2017
Maddison et al., 2017]. Joint
training also allows for incorporating auxiliary objectives
(see Section 2.4.3
to improve downstream task
utility, robustness, or bitrate flexibility (Niu et al. 2024
Kurnar et al. 2023
2.4.2 Main Training Objectives
Audio tokenizers are optimized using different main objectives, depending on the targeted application, as depicted in Figure
Reconstruction (Recon). The most common objective for training audio tokenizers is reconstruct the original audio imput from discrete tokens. This is achieved using regression loss, such the mcan squared error (MSE) or mean absolute error (MAE) between the input and the reconstructed out å (Délossez) 2023 Zeghidour et al. 2021
et al.
LHecon
1 2z 1?
Vector Quantization (VQ). In the trraight-througe estimator (van den Oord et al. 2017 used for vector quantization, gradients bypass the codebook, requiring additional lossos to align the embeddings with the encoder outputs. One example is the soft-to-hard scheme (Agustsson et al. 2017), where a quant ion loss is applied during training to encourage the boftmax-based quantization approximation to closely match the original continuous representation 2:
M
LvQ = 1|2 211, 2
Σ m=
Om * Crus
(6)
where M denotes the total number codebooks, Cm is the continnous representation that corresponds to the m'th codebook (as defined in Section 2.2 and am are their correspouding softmax weights.
Another example is the commitment loss, which encourages the encoder outputs align with the selected codebook embeddings. The loss computed between each residual 2(()) and its quantized counterpart a!m) from the m-th codebook, with gradients blocked from flowing through the quantized values:
T
M
(m) 2
LvQ
5g
where sg[:] denotes the stop-gradiens operator. This loss penalizes discrepancies between the encoder outputs and their corresponding quantized embeddings while ensuring that gradients do not update the codebook entries directly. Modern approaches often replace this loss with Exponential Moving Average (EMA) updates for the codebook, which improve training stability and mitigate codebook collapse. This collapse occurs when the model selects only few code vectors, leaving the rest inactive and unupdated (Dhariwal et al. 2020 Kumar et al. 2023). As a result, the effective codebook size decreases, lowering the target bitrate and degrading reconstruction quality. To address this issue, researchers have proposed several stralegies to improve codebook utilization. Some models (Dhariwal et al. 2020 Zeghidour et al. 2021) apply codebook expiration, periodically reinitializing inactive code vectors. Others Kummar et al. 20233 Yang et al. 2024d:
Défossez et al. (202)) use factorized quantization and L.2 normalization to encourage more balanced usage
across the codebook. Techniques like ESC (Gu & Diao
2024 and NDVQ (Niu et. al. 20t
11

---

Euclidean normalization represent codebooks as distributions, using margin-based probabilistic losses to prevent collapse and promote diverse activation. Additional solutions introduce auxiliary constraints, such as entropy penalties or code balancing losses. For example, Enhanced Residual Vector Quantization (ERVQ) Zhong et al. (2025] uses intra-codebook optimization via online clustering and a code balancing loss to reactivale unused vectors, while inter-codebook optimization minimizes redundancy between adjacent quantizers through a structural similarity loss, enhancing expressiveness and overall utilization.
Adversarial (GAN). Acoustic tokenizers often apply adversarial losses improve perceptual quality. A discriminator network D is trained to distinguish between real signals and reconstructed signals é, while the tokenizer (generator) is optimized to fool the discriminator. The adversarial loss is defined as hinge loss over the logits of the discriminator, averaged over multiple discriminators and over time.
The generator loss is:
K
1
LG =
max (1 (x))
(8)
K
#=1
The discriminator loss is:
K K
CD
max - Db(x), + max(1 + (+).
(9)
where D+(-) denotes the output of the k-th discriminator. Following VQGAN Esser et al. (021), the subso- quent audio neural vocoders commonly include multiple discriminators with a specific focus on the frequency- level rebuilding to enhance the perceptual quality (Kong et al. 2020 Défossez et al. 2023 Zeghidour et al. 2021). Spocifically, these multi-scale discriminators take the stack of multi-resolution or multi-scale complex- valued short-time Fourier transform (STFT) with the real and imaginary parts concatenated as input, c.g., different scales with STFT window lengths of [2048, 1024, 512, 256, 128), for capturing different, structures in audio signals.
Feature Matching (Fcat). To encourage the original and reconstructed signals to exhibit similar ab- stractions (or to align closely in the latent space), stabilize adversarial training, and encourage more nalural reconstructions, a feature-matching the discriminator for real and reconstructed signals. It is defined as:
matching loss is often applied. This loss compares intermediate activations from
(1((2) - D2 mcan (I))
Feats
(10)
KL
where is the number discriminators, is the number of layers in each discriminator, and (.(-) denotes the output of the 1-th layer of the k-th discriminator. Feature matching encourages the generator to match higher-level statistics of real signals, improving stability and perceptual quality.
Diffusion (Diff). Diffusion loss is used when the decoder is modeled as conditional denoising diffusion process. A diffusion model (Rombach et al. 2022) progressively adds noise to the latent representation during the forward process. A conditioned neural network, parameterized by 0, is trained to predict the noise at each timestep. The training objective minimizes the expected difference between the true noise e and the network prediction 6g (2,, t, 2g), conditioned on the discrete tokens aq:
Laitfusion Exo.t, [ller - EØ(2,, 1, 2z)
(11)
where 2, represents the discrete conditioning tokens provided during both training and generation. This approach enables the model recover acoustic features directly from discrete tokens (Yang et al., 2024c
San Roman al. 2023 Du al. 2024b).
12

---

Masked Prediction (MP). Masked prediction loss is commonly used in tokenizers where the encoder and decoder are trained separately. The encoder is trained to predict masked portions of the input sequence, typically capturing phonetic information rather than reconstructing the full waveform. Following the masked language modeling (MLM) paradigm (Devlin et al. 2019), a portion of the input is randomly masked, and the model is optimized to predict the masked frames from the surrounding context. Formally, given an input sequence of frame-level features X {I1, (21,,2,,.... 22,...,, Zr); ; a binary mask M E (0, 1) is applied, where M, 1 indicates masked position. The masked input is denoted as X where x aqmask [MASK] if M, = 1, and zt armaak 2z otherwise. The encoder processes this masked imput to produce latent representations
Z = Se (x(Xmaak), and the model is trained to minimize prediction loss.
T MM = Σ M, ((Z), 1=1
where fe is the encoder network and -(.-) is the cross-entropy loss. This approach widely used in speech pretraining models such HuBERT (Hsu et al.] 2021) and WavLM (Chen et al. 2022) and is adopted in
several semantic tokenizer designs LLkkhotia et al. 2021 Mousavi et al. 2024b)
2.4.3 Auxiliary Components
Beyond the main training objectives, neural audio tokenizers often combine auxiliary components to enhance generalization, improve represenlation learning, and refine specific features. These auxiliary components fall into three categories: disentanglement, semantic distillation, and supervised semantic tokenization
Disentanglement. Disentanglement methods separate different speech attributes into distinct represen- tations, reducing rodundancy while allowing independent control over acoustic propertics and simplifying downstream tasks. One type of disentanglement in the codec focuses on separating speech and background audio embedding space, enabling better bitrate, entropy control (Yang et al., (021) or speech enhance- ment (Omran et al. 2023). Those models normally aim to find latent space where two ideally orthogonal components 21 and Zg can be conveniently separated, F(x) = Z, @ Z2, with @ representing straightforward
operations such as splitting along the channel Yang et al. 2021
Another type separates the conceptual and fundamental components of speech, where each component is typically extracted by its specific encoder. 2& = Fx(x). Early attempls obtained efficient and low-bitrate speech coding through speaker and phoneme disentanglement, utilizing separate training (Polyak et all, 2021) or joint training (Jiang et al., 2023). More rocently, TiCodec (Ren et al., 2024b) minimizes token usage by separately quantizing lime-invariant global embeddings (e.g., limbre) and Lime-varying features (e.g., phonetic information). FACodec (Ju et al.) 2024) decomposes speech into subspaces such as content, prosody, timbre, and acoustic details through supervised techniques. The timbre extractor in FACodec is optimized with a speaker classification loss, while distinct RVQ modules process other components before supervised decoupling. T.SCodec (Guo et al. (2025a) introduces a low-bitrate, speaker-decoupled speech codec using a three-stage training framework with speaker perturbation. A VQ layer applied after VAE that disentangles speaker attributes in continuous space, followed by training token vocoder the quantized codes. Unlike most acoustic tokens that redundantly encode speaker timbre across time steps, LSCodec minimizes this inefficiency by isolating timbre from content and prosody. SoCodec (Guo et al.) 2024a) employs multi-stream phonetic sequences and ordered product quantization to encode speech into phonetic and vime-aariant token sequences using HuBERT as a pretrained SSL model. An ECAPA-TDNN based encoder (Desplanques et al., (020)) extracts an utterance-level global embedding to retain iive-invariant information, such as speaker identity, global speaking style, and acoustic environment. SD-Codec (Bie al., 2025) integrates audio coding with source separation by assigning diffeerntt audio domains (such speech, music, and sound effects) to distinct codebooks using multiple parallel RVQ modules.
We here inherit the original terminology from the referenced papers (i.c. semantic distillation, and supervised semantic tokenization). However, important note that both methads typically extract learn information from SSL features, which predominantly encode phonetic information.
13

---

Semantic Distillation. Semantic distillation enhances codec representations by incorporating phonetic information into specific codebooks. Various approaches have been explored to distill phonetic knowledge into tokenization while maintaining good reconstruction. Pretrained model guidance is a common approach, where models like SpkeccTokenizer (Zhang et al., 2024a), X-Codec (Ye et al., 2025), and Mimi (Défossez et al. 2024) use SSL features guide specific RVQ layers to learn information from such SSL. features. This distillation is implemented by applying regression or classification loss on the first RVQ output to align it with continuous SSL embeddings or discrete SSL tokens. In this way, the first RVQ layers are trained to learn more phonetic information, while later layers focus acoustic details. Another method injects semantic knowledge directly into the quantizer codebook. T.L.-Codec (Yang et al., 2024b) follows this approach by initializing codebooks with token embeddings from LLaMa2 (Touvron et al. 2023 and keeping them frozen during training. This strengthens the ability of the codec to encode meaningful linguistic representations. Some models integrate semantic features into the encoder-quantiier pipeline by combining pretrainod SSL representations with acoustic features through concatenation. ScmantiCodec (Liu et al., 2024a) and X-Codec Ye et al., 2025) adopt a dual--nnooeer-eccoder architecture to process SSL semantic tokens independently from acoustic features.
Supervised Semantic Tokenization. Some tokenizers explicitly capture phonetic detail through super- vised training, For example, Supervised Semantic Speech (S3) Du et al. 2024a b) employ single-codebook VQ layer and FSQ, positioned between two transformer encoder modules. Recently, Har-Tuv et al. (2025) proposed adding phonetic classification ion auxiliary loss over the first codebook of the RVQ. These models op- timize representations using an automatic speech recognition (ASR) loss. Additionally, they utilize optimal- transport conditional flow matching (OT-CFM) (Tong et al. 2024 to model and generate Mel spectrogram distributions conditioned on the produced discrete speech tokens. These supervised approaches produce discrete tokens that effectively preserve phonetic information, making them more aligned with content in- formation and suitable for understanding tasks in speech LMs (Zeng et al. 2024
2.5 Streamability and Domain Categorization
Beyond architecture and training paradigms, audio tokenizers also differ in their support for streaming and their domain of application.
Streamability. Streamability refers the ability of tokenizer to process and generate audio in real-time with minimal latency, using little or no future context. This property is critical for low-latency applications such real-time communication streaming. Latency can be analyzed from two main perspectives:
Algorithmic latency, determined by the look-ahead window i.e. how much future information needed to compute the current frame. CNN-based models (Délossez et al. 2023) support stream-
ability via causal convolutions, while Transformer-bar based (Wu al. 2024d) models require causal attention mechanisms.
Computational complexity, which becomes especially important when deploying models on resource-constrained systems like mobile or edge devices. Traditional and early neural audio codecs generally maintain low complexity for real-time feasibility. For instance, LPCNet (Valin & Skoglund 2019b) achieves real-time performance with fewer than 2M parameters at 1.6 kbps. In contrast, more recent models like Encodoe scale up to 144 paramcters to support 1.5 kbps, while BigCodee pushes further to 159M parameters at just 1.04 kbps to improve quality at low bitrates.
Many SST-based tokenizers rely on non-causal encoders, which limits their use in real-time set settings. Thus, achieving streamability with high-quality and efficient causal architectures remains an open research chal- lenge.
Target Domain. Some models (Xin et al.) 2024 Zhang et al. 2024a Mousavi et al.) 2024b Défossez et al. 2024) are specifically designed for speech tasks such ASR and TTS. Others are optimized for mu- sic (Petermann al. 2021 Tang et al. 2024b) generation and enhancement, capturing tonal and harmonic
14

---

Table 2: Audio tokenizers, their characteristics, and abbreviations used throughout the study, As abbrevia- tions, we denote tokenizers as [name] [domain (s)] [sample ratel.
Tokenizer
Abbreviations
Domain
SR
Frame
# #Codes
Params
MACs
Link
Speech
Music
Audio
(kHz)
Rate
(Mil)
(G)
Ene-SMA-24
1024
14.9
6.1
Link
EnCodec
Enc-M-32
32
50
2048
56.9
14.4
Link
Enc-A-16
16
50
2048
56.8
14.0
Link
DAC-SMA-44
V
v
44
86
1024
76.7
147.0
Link
DAC
DAC-SMA-24
V
24
1024
74.7
83.4
Link
DAC-SMA-16
v
1024
74.1
55.6
Link
SpeechTokenizeer Tokenizer
ST-S-16
1024
103.7
17.1
Link
Mimi
Mimi-8-24
/
24
12.5
2048
79.3
8.1
Link
Discrete- WavLM
DWavL-S-16
1000
331.9
Link
SQ-Codee
SQ-SMA-16
v
16
50
19683
23.5
14.7
Link
WT-SMA-24
24
75
4096
80.6
6.3
Link
Way Tokenizer
WT-S-24
24
4096
80.9
3.4
Link
structures. Some tokenizers Yangg et al. 2024b) designed for general audio, including environmental sounds and non-speech signals. A few models (Ji et al. 2024c Défossez et al. trained to handle multiple domains.
2023
Kumar et al. 2023 are
3 Benchmark Evaluation
Given the wide range of available tokenizers, researchers and practitioners may wonder which existing to- kenizers are best suited for a given use case. This depends not only on the expected performance for a given task but also computational efficiency and, in some cases, additional factors such as strcamability the ability to generalize across diverse domains. Several benchmarks have been proposed to evaluate audio tokenizers, offering some guidance on which tokenizers are best suited for different applications and tasks (Wu et al. 2024c Mousavi et al. 2024a Shi et al. 2024c Maimon et al. 2025c
Nevertheless, drawing solid insights from current benchmarks is challenging, as each focuses on a specific aspect or domain and a holistic comparison of audio tokenizers is missing. Furthermore, while each existing benchmark is internally consistent in its evaluation protocol, they differ significantly in the set of tokenizers acoustic models, while others evaluate semantic tokenizers even
they consider, some focus exclusively different configurations of the same model (e.g., EnCodec- 16k vs. FnCodec-24k). This lack of alignment makes it difficult to derive unified or comparable conclusions across studies. This section contributes to filling this gap by considering a diverse set of publicly available, pre-trained tokenizers across speech, music, and general audio tasks. Unlike previous benchmarks, we perform a joint evaluation across multiple dimensions:
1. Reconstruction Evaluation and Complerity Analysis. We assess the quality of resynthesized dio using the original decoder trained for each tokenizer, following protocols from CodecSUPERB and VERSA. We also evaluate the computational efficiency of each tokenizer based on model size (parameters), frame rate, token rate, and multiply-acommulate operations (MACs).
Downstream Evaluation. We assess the effectiveness of tokenized representations when used directly as input to lightweight models for both discriminative and generative tasks using DASB benchmark.
Acoustic Language Modeling. We analyze the effectiveness of each tokenizer in training acoustic language models, using the SALMon and Zero-resource benchmarks.
15

---

Table 3: Summary of evaluation metrics on resynthesized audio.
Metric
Functionality
Range
Domain
Speech
Music
Andio
Signal-level
SDR
Signal-to to disisorttion Ratio
(-inf, inf)
S1-SNR
Scale-invariant signal-to-noise ratio
(-inf, inf)
PESQ
Perceplual Evaluation of Speech Quality
[1. 5]
UTMOS
UTokyo-SaruLab System for VoiceMOS 2022
[1, 5]
DNSMOS P808
Deep Noise Suppression MOS Score of P.808
[1, 5]
DNSMOS P835
Deep Noise Suppression MOS Score of P.835
[1, 5]
PLCMOS
Packet Loss Concealment- MOS
[1, 5]
STOI
Short- Objective Intelligibility
[0,
VISQOL
Virtual Speech Quality Objective Listener
[1. 5]
SingMOS
Singing voice MOS
[1, 5]
Application-level
WER
Word Error Rate (beam=5)
[0, inf)
Spk Sim
Speaker Similarity
[-1, 1]
A summary of all tokenizers included in the benchmark evaluation is provided in Table 2 We select these tokenizers based on several factors: (1) We prioritize open-source models with accessible checkpoints and code to ensure reproducibility; (2) we include tokenizers representing diverse range of quantization strate- gies including RVQ (EnCodec, DAC), SVQ (WavTokenizerr) FSQ (SQ-Codec), KMeans (Discrete WavLM), and semantically distilled methods (SpeechTokenizer, Mimi); and (3) we aim to cover multiple domains such as speech, music, and general audio, favoring multi-domain tokenizers where available. The impact of single- versus multi-domain tokenizers further discussed Section
4 We utilize pre-trained checkpoints released by the original authors. An overview of our benchmark evaluation pipeline illustrated in Figure
3.1 Evaluation for Reconstructed Audio Quality and Complexity
Background. We evaluate audio reconstruction quality and examine key properties of audio tokenizers, such computational complexity, bitrate, and token rate. Reconstruction quality is particularly important for applications like transmission, where preserving signal fidelity is crucial. Moreover, high reconstruction quality might be a useful proxy for selecting effective tokenizers for downstream lasks, especially those that rely directly on the decoder for audio generation, such as speech enhancement and source separation. In such cases, the reconstruction performance of the tokenizer can impact the overall task performance.
Experimental Setup. The input audio first compressed using an audio tokenizer and then resynthe- sized through its corresponding decoder. The resynthesized audio is asscssod from both signal-level and application-level perspectives, providing insights into how well each tokenizer preserves information. To ensure comprehensive analysis, integrate the evaluation scripts from Codec-SUPERB and VERSA (Wul
ct al. 2024c Shi et al. 2024c )025). We extend the evaluation across three domains (music, general audio, and speech) to examine how different tokenizers perform in diverse acoustic scenarios.
Dataset. For speech evaluation, we use the LibriSpeech test-clean set (Panayotov et al. 2015). For music, we use the MUSDB dataset (Rafii et al., 2017]. which consists of approximately 10 hours of full-length and professionally-reoon musical tracks 44.1kHz. Lastly, for general audio we opt for the Audioset (Gem- meke et al. 2017) test-set, which accounts for approximately 55 hours of audio clips extracted from YouTube.
16

---

Table 4: Reconstruction performance audio tokenizers (speech).
Tokenizer
#Q
kbps
Token rate
SDR
PESQ
UTMOS
DNSMOS
DNSMOS
PLCMOS
STOI
WER
Spk
SNRI
P8081
PS351
Sim1
Ground truth
290.16
55.92
4.64
4.09
3.81
3.18
4.16
1.00
2,83
1.00
150
0.82
1.53
1.56
1.58
0.85
5.44
0.42
Eno-SMA-24
8
600
6,50
4.83
2.77
3,09
3.57
2.06
4.08
0.94
2.78
0.72
32
24
2400
9.75
7.90
3.71
3.74
3.74
3.19
4.29
0.97
2.77
0.78
150
40.57
8.40
1.68
0.83
9.59
0.45
DAC-SMA-24
8
6
600
1.79
-9.51
3.40
3,60
3.69
3.16
4.15
0.96
3,53
0.73
2400
2.20
99.47
4.40
0.99
2.72
0.80
100
-7.10
11446
1.21
2.32
3.37
2.78
2.96
0.77
4.20
0.33
ST-S-16
8
4010
3.01
0.33
2.62
3.81
3.77
3.17
4.00
0.92
2.41
0.86
100
3.43
1.19
3.60
0.90
3.72
0.70
Mini---24
32
4.4
400
9.32
7.45
3.38
3.92
3.74
3.18
4.40
0.96
2.96
0.85
100
-13.96
37.23
1.13
3.32
0.75
4.97
0.33
DW
6
3
300
-12.69
-33.43
1.19
33,32
3.72
3.13
4.05
0.75
4.31
0.33
SQ-SMA-16
4
3
200
1.91
-8661
3.31
3.90
3.83
3.28
4.13
0.96
2.37
0.87
WT-SMA-21
75
2.02
-0.79
1.88
3.77
4.41
0.87
8.10
0.00
WT--S-24
1
.62
40
0.17
-3.16
2.05
3.89
3.82
3.27
4.38
0.89
8.91
0.61
Evaluation Setup. Table reports the reconstruction metrics for resynthesized audio. Beyond quality, it is important to jointly consider factors often overlooked in the literature, such as computational efficiency and lokenizer properties. To address this, we also include these metrics as Table 2 (1). Model paramelers (Params): The total number of parameters in the audio tokenizer. (2). Computational complexity (MACs): Number of arithmetic operations performed by tokenizer. MACs are computed for one-second audio sample using PyFlops 10 For components incompatible with PyFlops, such as streaming self-attention, dot product, calculations balance between audio quality and compression efficiency. (4). Prame rate: The number of temporal frames used to encode one sccond of audio. (5). Token rate: Number of tokens required to encode one second of audio, an important factor for acoustic language modeling applications.
performed manually. (3). Bitrale: The number of bits per second, representing
Results and Discussion.
Speech. From Table 4 we make the following observations: (1) For both EnCodec and DAC, the reconstruction quality consistently degrades the bitrate decreases from 24k to 6k and 1.5k. This trend confirms that higher bitrates better preserve acoustic detail, resulting in improved reconstruction quality across all evaluated metrics. (2) For Speech Tokenize (4k vs. 1k) and Mini (4.4k vs. 1.1k), which both apply semantic distillation to the first codebook, all objective metrics decline at lower bitrates. However, the WER does not drop as drastically, indicating that somantic distillation effectively presorves linguistic content even when the overall reconstruction quality decreases. (3) Discrete WavLM exhibits significantly lower SDR, SI-SNR, PESQ, STOT, and Spk-Sim scores. Since these metrics rely on reference ground truth signals, the poor performance indicates these models are not optimized for precise waveform reconstruction. Metrics such as UTMOS, DNSMOS, and PLCMOS, however, remain reasonable, suggesting these tokenizers still preserve speech quality. This discrepancy indicates that discrete tokenizers focus more high-level representations than on exact waveform reconstruction. (4) SQ-SMA-16 performs comparable or even better than large bitrate codec models (e.g., Mimi-S-24 4.4kbps, and DAC-SMA-24 6kbps). (5) Finally, we find that SDR and SI-SNR are less reliable indicators. A possible reason is that the signal is over- neeecoopressed,, the generation of neural codec (especially in low-bitrate), usually have less consistency the local sample-level information. It is likely due to non-linear shifts or amplitude variations.
10 http://////tttiocooooo//rrrrrrre www.:o/tteeerooorrcreeeeeeattttt
17

---

Table 5: Reconstruction performance of audio tokenizers for both general audio and music experiments.
Audio
Music
Tokenizer
#Q
kbps
Tokkn
SDR
CI-
S1-
VISQOL
Sing MOS:
SDR
CI-
SI-
VISQOL
Sing
SNR1
1
SNR1
1
MOS
Ground truth
252.76
84.90
57.96
1.73
2.70
254.24
87.26
60.26
4.73
2.79
2
1.5
150
11229
-1.28
-1.31
3.91
2.59
2.16
2.13
0.46
4.03
2.67
Enc SMA.24 24
8
6
600
4.28
4.10
2.33
4.25
2.00
7.32
7.17
5.87
4.38
2.66
32
24
2400
7.72
7.33
5.64
4.36
2.60
11.04
10.75
9.19
4.50
2.66
2
1.5
150
22.00
2.55
-11.55
3.99
2.59
1.75
1.71
22.11
3.94
2.70
DAC-SMA-24
600
1.35
-10.28
4.35
4.82
-1.25
4.30
2.68
32
24
2400
2.45
2.22
-9.91
4.59
2.60
5.56
5.37
-1.16
4.56
2.66
SQ-SMA-16
4
3
200
2233
-2.33
10.50
4.32
2.62
3.44
3.39
-0.38
4.34
2.68
WT-SMA-24
1
.98
75
-4555
-1.45
9.78
3.96
2.56
14.30
- -11228
23.09
3.64
2.00
WT-S-24
1
.52
10
11.00
- 10.85
20.91
3.85
2.53
199991
- 19.89
-55.5
3.33
2.12
General Audio and Music. Table summarizes the reconstruction results for the general audio and music domains. As in the speech domain, reconstruction quality generally decreases with lower bitrates. The results also reveal notable trends related to both the training domain and optimization objectives for cach tokenizer. EnCodec achieves the best overall reconstruction performance across SDR, SI-SNR, and perceptual metrics (VISQOL and SingMOS), particularly at higher bitrates. In contrast, DAC shows surprisingly poor performance in time-domain metrics, with negative SI-SNR values in most settings. This suggests that DAC relics adversarial or perceptual optimization strategies that do not prioritize time-domain reconstruction loss. Nonetheless, despile poor Lime-domain fidelily, DAC maintains strong VISQOL and SingMOS scores, indicating its reconstructions remain perceptually plausible.
Similar to DAC, WayTokenizer is not explicitly optimized with a time-domain waveform reconstruction loss, resulting in poor SDR and ST-SNR scores. However, perceptual metrics (VISQOL. and SingMOS) remain relatively strong. This further highlights the limitations of time-domain metries in evaluating token-based representations, previously observed in the speech reconstruction results. Among the two WawTokknizerr variants, WT-S-24 and WT-SMA-24, only WT-S-24 is fully out-of-domain having been trained exclusively speech data. Both models exhibit poor performance across objective metrics such as SDR and SI-SNR, as well as perceptual metrics (VISQOL and SingMOS). This degradation stems from a combination of factors: the domain mismatch in WT-S-24, the use of the lowest bitrate among all models, and the absence of explicit waveform reconstruction losses during training.
Summary. Overall, these results underscore the importance of evaluating audio tokenizers beyond tradi- tional waveform fidelity measures. Models optimized for perceptual or downstream tasks may exhibit low signal reconstruction performance, yet still produce subjectively high-quality audio reconstructions.
3.2 Downstream Evaluation
Background. Evaluating token quality solely based on reconstruction performance raises an important question: how much task-releuant information is preserved in the tokens, independent of the decoder's ca- pacily? This distinction is critical in multimodal language modeling settings, where audio tokens are used directly as input to large language models. These models must perform both discriminative tasks (e.g., ASR, emotion recognition) that map audio text, and generative tasks (e.g., speech synthesis, speech-to-speech translation) that output audio.
Experimental Sctup. We evaluate discrete audio tokenizers using the DASB benchmark (Mousavi al. 2024a), built on the SpeechBrain toolkit (Ravanelli al., 2024), which isolates the representational quality of the tokens for rewmstream modeling. For each task, the encoder is frozen, and task-specific classification
18

---

Table Datasets, metrics, and downstream models for the DASB evaluation.
Task
Dataset
Architecture
Metric(s)
Data Link
Speech (Discrininateee)
ASR (Km)
LabriSpooch Korvas
Branchformer
ASR (Low-resouree)
CommonVoiiee
17.0
Ardila al.
2020
BiLSTM
WER
Link
Speaker ID/ / Verification
VoxCeleb1
Nagrani et al
2011
ECAPA TDNN
Accuracy / EER
Link
Emotion Reoognition
IEMOCAP
Busso et
2008
ECAPA-TDNN
Accuracy
Keyword Spotting
Speech
Commands Warden
2018
ECAPA-TDNN
Accuracy
Link
Intent, Classification
SLURP
Hastianelli et al.
2020
HiLSTTM+
Accuvary
Link
Spesch (Generative)
Speech Enhancement
VoireBank
Valentini- Botinhan et al.
2016
Conformer
DNSMOS / dWER
Link
Speech Separation
LibelMMxx
Cosentino al.
2020
Conformer
DNSMOS
dWER
SpkSim
Link
Music
Music Geure Classificatonn
GTZAN
Tzanetakis & Cook
2002
ECAPA-TDNN
Accuracy SDR / SIR / SAR
Link
Music Source Separation
MUSDB
Rafil et al 2017
Conformer
Link
General Audio
Sound Event Classffccation
ESC-50 PPizzak 2015
ECAPA-TDNN
Accuracy
Link
Audio Seporation
FUSS Wisdomm al.
2021
Conformer
SDR
Link
head trained. We use lightweight classification heads to avoid hiding weaknesses in the token representa- tions. Generative tasks additionally use the frozen decoder. All token embeddings are projected to fixed dimensionality of 1024 to ensure consistency across models. This value corresponds to the largest embedding size among the tokenizers in our benchmark. For tokenizers with multiple codebooks, weighted sum of codebook embeddings is computed, with weights learned jointly with the downstream head (Chen et al. 2022 Zaiem et al. 2023). For SQ-Codec, which uses scalar quantization and group vector quantization, we apply a ternary matrix-based embodding and concatenate four 256-dimensional group vectors to match the 1024-dimensional standard. Each tokenizer evaluated across multiple bitrate settings (low, high, and recommended). We tune the most relevant hyperparameters, such as learning rate and model capacity, using the Tree-structured Parzen Estimator (TPE) (Bouthillier et al. 2022) with 20 trials. To obtain more robust performance estimate, average the results of cach tokenizer over three downstream training runs with different random seeds. For ASR tasks, both character-level and byte pair encoding (BPF) segmentations are considered, and the perfor-performing configuration is reported. Table 6
summarizes the benchmark tasks and their corresponding downstream models. When multiple-domain tokenizer chockpoints are available, we use the multi-domain version for consistency. The impact of domain-specific vs. multi-domain training is further analyzod our ablation study (Soction 4h. Additional implementation details are provided in the DASB paper (Mousavi et al. 2024a).
Dataset. We evaluate audio tokenizers across diverse tasks and domains, including speech discriminative tasks such as ASR, low-resource ASR (L-R ASR), speaker identification and verification (SID, SV), emotion recognition (ER), intent classification (IC), and keyword spotting (KS). For generative tasks, we include speech enhancement (SE) and speech separation (SS). In the music and general audio domains, we evaluate music genre classification (MG), music source separation (MSS), general sound separation (ASS), and sound event classification (SEC). A full summary of datasets and tasks is provided in Table
Evaluation Sctup For continuous basclines, we follow Zaiem et al. (0223 by using weighted sum of WavLM-large layers as input across most tasks. To ensure a fair comparison between discrete and continuous representations, we adopt identical downstream architectures for both settings. While neither WavLM nor the chosen downstream architocture may represent the statc-of-tho-art for every task, using a consistent setup across all experiments allows isolate the effect of representation quality. For instance, in speech separation, well-established baselines such as Conv-TasNet. (Tuo & Mesgarani 2019) and Transformer-
19

---

models (Saijo et al., 2024) are excluded. Libri2Mix has become a saturated benchmark, with many ap- proaches reaching near-ceiling performance, and adding stronger backbones would not yield meaningful insights. Instead, our focus is on isolating the effects of discrete versus continuous representations under a shared architecture. There are two exceptions: for music and general audio separation, tasks that remain more challenging, we use stronger, task-specific architectures for the continuous baselines. Specifically, we adopt DEMUCS (Rouard et al., 2023) for music source separation and TDCN++ (Kavalerov et al., 2019) for general sound separation, as these models are better suited to the complexity of these domains. We evaluate each task using standard, task-specific metrics (Table 6): ASR is evaluated using Word Error Rate (WER); SV uses Equal Error Rate (EER); classification tasks including ER, STD, IC, MG, and SEC are evaluated using classification accuracy (ACC). For SE and SS, we report DNSMOS (Reddy et al. 2022) for perceptual audio quality, differential WER (dWER) using Whisper (Radford et al. 2023) for intelligibility, and speaker similarity (SpkSim) based on cosine similarity of WayLLMderrived embeddings. Music source separation is evaluated using signal-to-distores
ratio (SDR), signal-to-artifact ratio (SAR), and signal-to-interfernnee ratio (SIR) via the BSSEval toolkit (Vincent et al., 2006), while general sound separation is assessed using SDR on FUSS (Wisdom et al.) 2021 All results are averaged over three runs with different random seeds to ensure robustness.
Results and Discussion Tables 7 and 9 summarize performance across discriminative and generative tasks for speech, music, and general audio Below, we outline key findings from our experiments.
Speech Tasks, Discrete WavLM consistently performs best in discriminative tasks, likely due to its strong ability preserve phonetic content. SpeechTokenizer which uses semantic distillation, ranks second. In speaker recognition, however, DAC achieves the best results, suggesting that reconstruction- based objectives help preserve speaker identity. For speech separation and enhancement, WavLM performs well at low and medium bitrates but shows poor results in speaker similarity metries. This aligns with previous findings (van) Nickerk et al., 2022) that SSL-based tokenizers tend to losc speaker-related information. Another notable observation is that in many cases, the reconstructed DNSMOS score, representing the upper bound set by the codec alone without any separation, does not surpass the score obtained by using the raw mixture as the estimate (i.e., the lower bound), suggesting that limitations in reconstruction quality may constrain downstream performance, particularly for high figh-fideiity tasks like speech separation.
Audio and Music Tasks. For general audio and music tasks, EnCodec consistently outperforms other tokenizers across all bitrates and domains, while DAC lags behind. Although DAC is known for strong perceptual quality, its signal-level fidelity is generally lower, which likely impacts its separation performance. The difficulty of these tasks is evident from the S1-SDR of the unprocessed mixtures, for example, approxi- mately- -16 dB for general audio and -7.7 dB for music. Even the best-performing model (EnCodec at medium bitrate) only reaches about -7 dB SI-SDR for audio and -5.7 dB for music. We report the performance in terms of SI-SDR improvement ("SI-SDRi"), not absolute SI-SDR. Thus, the reason we also provide the per- formance "unprocessod" mixtures. For instance, for general audio, we report an improvement of 9.53 dB over the mixture (-16.5 dB). In absolute value, this means that the resulting predictions yield an average of -7-7 dB performance. While high-bitrate settings have proven to be challenging for downstream tasks, they perform particularly poorly in music separation, emphasizing that increasing bitrate alone does not improve separation quality and may even degrade performance. This may be due to the inherently polyphonic and less sparse nature of music (in contrast. to speech and general audio), which results in highly overlapping sources that are harder to disentangle from detailed but semantically entangled representations.
Impact of Codebook Size. Increasing the number of codebooks (e.g.. 2, 8, 32) improves signal recon- struction but often reduces downstream task performance. This trade-off suggests that while more codebooks enhance fidelity, they often degrade performance for both discriminative and generative tasks by incrcasing output dimensionality and modeling complexity. In RV Rase-based models, earlier codebooks capture more pho- netic information, while later ones often add redundancy, which may explain this trade-off. This highlights an important design principle for tokenizers: optimizing for reconstruction alone does not guarantee better performance on dounstream tasks. Modium bitrate settings typically provide the best balance between audio reconstruction quality and task performance.
20

---

Discrete vs Continuous. While discrete tokens show promise, they face notable limitations in complex scenarios such as polyphonic music separation or noisy environments. Continuous features consistently out- perform discrete tokens due to the information loss inherent in quantization, which affects critical attributes like phonetics, cmotion, and speaker identity. These limitations are further exacerbated in low-rcsource settings. For instance, although Discrete WavLM performs competitively at low and medium bitrales for low-resource ASR, it still lags behind the continuous bascline. RVQ-based tokenizers struggle even morc, especially on smaller datasets such as Welsh, ESC-50, and GTZAN, with high bitrate amplifying these issues. Performance improves with more data: Discrete WavLM, for example, achieves 6.0% WER on LibriSpeech (960h), 22.0% on Basque (116h), and 58.9% on Welsh (8h) at low bitrate using a BiLSTM head, illustrating a strong correlation between data scale and ASR accuracy. From the hyperparameter tuning experiments (not reported here for brevity), we noticed that larger downstream models help improve convergence and performance, particularly for acoustic tokenizers, which are more sensitive to both data scale and model capacity. Semantic tokenizers are gencrally more robust in low-rcsource scttings but still fall short of con- tinuous representations with extremely limited data. Overall, careful tuning and appropriate scaling of both data and model are essential for effective use of discrete representations, especially acoustic tokens.
Summary. Semantic tokenizers (e.g., Discrete WavLM) are generally more robust, especially in low- resource settings, but still fall short of continuous representations when data is limited. Training downstream models with semantic (Discrete WavLM) or semantically distilled tokenizers (Mimi and SpeechTokenizer Tokenizer) tends to be more stable and reliable compared to acoustic tokenizers (EnCodec, DAC, WavTokenizer, and SQ-Codec), which often require larger datasets and more careful model scaling. Overall, discrete tokenizers are more sensitive to architectural choices and hyperparameters of the downstream head, whereas continu- ous features typically yield more consistent performance across configurations. Therefore, careful tuning and appropriate scaling of both data and model architecture are crucial for effectively leveraging discrete repre- sentations. While discrete tokens offer ad advantages in efficiency and modularity, continuous representations still lead in overall performance. Bridging this gap is essential for the successful integration of audio tokens into future multimodal language models.
Table 7: DASB results for discriminative tasks (speech).
ASR-En
ASR-LR
ER
IC
KS
SI
SV
Tokenizer
#Q
WERI
WER.
ACC!
ACC
ACC1
ACC!
EER.
Clean
Other
Welsh
Basque
Continuous
4.07
6.81
41.77
14.32
63.10
86.10
99.00
99.70
2.10
2
12.70+00.37
20.09+0013
90.90:20.32
51.00::0.99
45.50+0.02
42:90:00116
77.13=3.12
80.81+5546
18. 18.10=0.26
Enc-SMA-24
8.43+00113
21.77= 0.36
84.50++1.0
45.36-#0.57
44.73+00.0
40.00=:0.29
74.30=1.69 1.69
94,26+33.98
13.54=0.57
32
9.17
23.24 1.22
97.00-11.19
58.21 #0092
42.06+0.02
33.064:2.65
60. 10.10-3.42
91.1241.92
10.12-6.60
2
14.8410.25
30.88:10200
96,2110.84
68.93.20.0
45.2010.01
20.8310.19
67. 67.27_1.56
97.88:10.79
21.80::1000
DAC-SMA-24
10.73.t 0.10
25.3911 0.20
97.20.10.1 14
62. 1.
44.7310.02
23.9710..11
65.27=2.62
87.93.110.95
15.86.55228
18.13.1.11
28.47.1011
98. 06.96.10.08
1.55
43.2011002
41.601:30.0
66.60 0.967
87.69.1-4.99
17.12 0.76
ST-S-16
2
9.1820.10
22.6820.10
71 71.86:00.32 322
42.17=0005
54.8610:01
56.8020008
94.11=0.63
73.16:20.55 37
24.23:0.29
9.06s: 0.45
21.7240.23
08.366:0.444
35.36:20:22
05.00+0.01
53.8320.05
91.11=0.07
96.78:20.85
10.45:0.45
9.7310.61
22.65110:11
91.59220.0
59.18:28.52
51.1310:02
50.8311.199
92.18_0.20
79.50.10.04 43
18 18.6520.35
Mimi S:24
32
10.8410.56
24.1011.36
96.89:10.07
58.15.10.90
46.7610.01
50.7310.50
91.31=0.19
63.93:113661
23.91=4.60
DWw-L-S-16
2
4.7820.22
10.58+0.17
58 08.98-80.11 15
22.022-00.
61.63+0022
76.33+0.117
96.82:+0.92
76.67#0035
22.41=0.19
5.07+0.17
9.57=0.20
08.94:2..88
19.66:20.33
63.20:00.01
78.73:20.12
05.8900.50
92.31:20.00
13:47=0022
SQ-SMA-16
91.3710.19
92.9010.11
91.0::0.888
91.21:1124
41.30:10006
58.031
92.71-0.12
97.38.10.03
9.69::0225
SQ-SMA-16*
4
11.63+0.08
30.91+0.17
WT-SMA-24
16.11:00188
35.4880.35
97:41:00.08
73.82=0:20
43.433-0002
15.25::0.16
69.13=2.10
85-90+2-48
19.38=0.35
3.3 Acoustic Language Models Evaluation
Following the rise of LLMs, researchers have extended the generative auto-regressive framework beyond text to discrete representations of speech (Lakhotia et. al.) 2021 audio (Borsos et. al. 2023b), and music (Copet
21

---

Table DASB results for generative tasks (speech).
SE
SS
Speech
Models' Tasks
#Q
DNSMOS
dW ER
Spk
DNSMOS
DNSMOS
dWER
Spk
Simt
Rect
Sept
Simt
Contimuous
3.49
4.92
0.93
3.68
9.97
0.94
3.151:001
34.9520.64
0.8610.00
3.19
3.13:0..0
80.33=1.77
0.88.0.00
Ene-SMA-24
3.08#0.01
22.70+1.84
0.8840.00
3.54
3.08:40.00
53.37=0.65
0.90#00.00
32
2.78100001
65.70-1.09
0.8010001
3.72
2.97:10.01
92.42-0.97
0.85:10.00
3.26:1.001
54.8511.82
0.8610000
3.16
3.01:1000
101.191::.99
0.8510000
DAC-SMA-24
3.51++0001
29.44+33.93
0.90++0001
3.67
3.30+0.00
12.77-2.48
0.93+00.0
32
2.93#0.001
30.66=0.97
0.88+0.00
3.76
2.67:+0.01
92.07=0.05
0.88#0.01
3.19::0.02
20.9840.58
0.8640.00
3.20
3.13:0.00
84.94=0.63
0.87+10.00
ST-S-16
3.49:10.01
21.65:0.57
0.87:10.00
3.72
3.43:10.01
60.90-0.77
0.91+10.00
3.25::0.01
2..5612.21
0.85+0.00
3.65
3.29::0.00
109.30+3.30
0.87=00.0
Mimi-S-24
32
3.18++0.01 102.61+2240
0.82+00.0
3.72
3.00++0.00
137.00++2.16
0.82+0.00
2
3.56::0.01
25.881:.11
0.88+0000
3.57
3.56::0.00
49.57=0.64
0.85+10.00
DWawL-S-16
3.57+00.01 9.43=0.33
0.89++0.00
3.75
3.75++0.01
30.39+0.445
0.91+00.00
SQ-SMA-16
4
3.28:+0001
122.33+8.74
0.83+00.00
3.77
3.19:+0.00
136.00+3.58
0.83+0000
WT-SMA-24
3.33+0.01
67.53+ 10.65
0.85++0.00
3.57
3.42++0.00
118.33+4.50
0.86++.00
Mixture
3.43
Table DASB results for generative and discriminative tasks (music and general audio).
SS - Audio
SS
Music
SEC
MGC
Tokenizer
#Q
SI-SDRi
SI-SDRi
SAR
SIR
ACC1
ACC1
Rec
Sep
Rec
Sep
Contimous
15.07
13.29
9.56
11.99
92.91
87.00
2
0.76
7.03+4049
3.36
1.49422.04
2.80+1.68
5.96++1.52
34.83:00447
70.33=11.70
Enc-SMA-24
8
3.87
0.33
7.99
1.98++0.38
-1.95-0.33
5.26+0.22
37-00+0.73
54.67+3.86 +3.86
32
5.76
-1.73.10.09
11.10
-11.72-0.35
-15.00.10.02
-0.42.10.01
35.43.11.45
39.67 11.25
0.12
3.84+20.48
2.37
1.01#0.17
-3.5920.09
5.9240.28
31.03:1.884 .44
50.00=0.82
DAC-SMA-24
8
3.33
3.62++0.21
6.66
-11.77+0.1
-10.62+2.35
-5.52+3.68
28.60+00.99
47. 17.67+3.09
32
4.73
-4.92.10.32
8.54
-11.32-10.12
-12.7010.17
10.05
06.67.10..99
50.00.1088
SQ-SMA-16
4
3.62
6.54++022
5.53
-3.6240.88
-5.8420.86
1.22:0.22
31.37+1.77
42.67=0.417
WT-SMA-24
-24.05
16..22+0.00
-2.66
-4.52+0.04
-8.32+0.07
2.65+0.11
34.50+0.82
48.00+
Mixture
-16.5
-7.71
50.01
-in
ct al.) 2023). Acoustic language models refer to models that learn to represent generate audio signals
(including speech, general audio, and music) using this generative autoregressive framework. This modeling approach has proven highly effective across domains, enabling powerful generation capabilities (Défossez ct al.] 2024). In this section, we begin by examining unconditional speech generation (Speech Language Models) and text-conditioned generation (text-to-speech (TTS)). We then explore audio generation and finally turn to the music modality.
22

---

3.3.1 Speech Language Modeling
Background. Speech Language Models (SLMs) have gained significant interest
AAoor et al. 2025 Wu
jt al. 2024a]
Peng al.
2024 Cui et al.) 2024 Ji et al.
2024a Latif et al.] 2023): demonstrating remark-
able performance in traditional speech tasks (Chen et al. 2025a Elmakies et al. 2025 diverse generative
applications Yang et al.
2024c b), and reasoning over speech and audio signals Chu et al. 2023
Wang
2025a Yosha et al. et al.
2025
SLMs can generally be classified into two main categories: (i) generative SLMs that are conditioned on
previous speech/text tokens and generate speech/text (Défossez et al. 2024 Cuervo et al.) 2025 Nguyen
et al.
2025), and (ii) speech-aware LMs that are conditioned on speech/ Text and generate text. (Chu et al.
2023
Tang et al. 2024a] Mousavi et al., 2025). This work focuses on the first category of SLMSs as there
is a growing interest from the research community to study generative SLMs (Nguyen et al.
2025
Maimon
et al.
2025b Rubenstein et al.
2023
Several SLMs operate over discrete speech representations derived from a pro-trained SSL model Ngguen
ett al.
2025 Lakhotia et al. 2021 Cuervo et al.) 2025). Others employ semantically distilled acoustic
tokenizers Défossez et al. 2024) or adopt a cascading, mixed-resolution strategy (Borsos et al.) 2023b), modeling speech hierarchically from coarse semantic content to fine acoustic details, using language models conditioned on previously generated streams. More recently, supervised semantic tokenizers have gained popularity. These methods typically quantize the output layer of pre-trained ASR system to produce discrete tokens (Zeng et al. 2024 2025). In this study, we analyze the impact of these choices by comparing SLMs trained under a controlled sctup with different audio tokenizers presented in Table 2
We focus on SLMs that model the joint probability of a sequence of speech tokens q as:
P(q qI...... ................................ 4n II P(g. I 9cs),
(12)
where yi E Vg, and Va denotes the vocabulary of speech tokens. These models are typically implemented as decoder-only transformers (Vaswani et al. 2017) and trained to minimize the negative log-likelihodd:
a LLM Σ P(all=)))
(13)
Each token is embedded via matrix E e R//l where d denotes the embedding dimension. The resulting sequence is processed by a stack of causal transformer layers, yiclding contextual representations e E Ra A final linear projection U E dx maps these to logits, which are converted to a probability distribution over the vocabulary via a softmax: p(4+1 c.).
Experimental Setup. Motivated by Maimon et al. (2025a), each SLM is built upon the Qwen-2.5 ar- chitecture (Yang et al. 2024a) (357M parameters in total, after removing the text embedding tables) and initialized using TWIST (Hassid et al. 2023). The textual embedding tables are replaced with new audio embedding tables corresponding to the audio codebooks. The models are trained using the standard cross- entropy loss reported in Eq. 13 To ensure all SLMs processed a comparable amount audio during training, we dynamically adjusted the tokens per batch according to their tokenizer's frame rate. To accommodate multiple codebooks, the delay pattern from MusicGen :Copet et al. 2023) is applied across all SLM models. The models trained for total of 50, 000 optimizer steps, with context length set to 1024. The audio target batch size is set to include about 2.9 hours of speech per backpropagation step. We used the Adam optimizer coupled with a lincar lcarning rate scheduler, applying a 1% warmup ratio (corresponding to 500 steps). All input samples are fed to the SLMs using a packing strategy by concatenating all samples together until having a sequence of the target length. To ensure fairer evaluations across different tokenizers, the mum- ber of codebooks is restricted to a maximum of 8, promoting better alignment between them. All code was developed using the SpeechBrain toolkit (Ravanelli et al. 2024), and Hugging Face Transformers (Wolf al., 2019).
23

---

Table 10: SLM results considering spoken content and acoustic elements using a subset of SALMon tasks.
Spoken Content
Acoustic Consistency
Sem.-Ac. Align.
Tokenizer
sBLIMP
sWUGGY1
sSCt
tSC
Gendert
Sent.t
Spkt
Sentiment
HUBERT 25Hx
60.89
70.51
53.23
71.46
69.50
62.50
69.00
53.00
Enc-SMA-24
51.14
51.29
50.18
48,20
70.50
56,50
65.00
50.00
DAC-SMA-16
51.51
50.73
48.95
51.52
81.00
60.00
77.00
50.00
ST-S-16
51.08
56.89
48.42
55.74
66.50
58.00
65.00
49.50
ST-S-16*
52.75
63.46
47.56
60.60
67.00
59.50
65.50
50.00
Mimi-S-24
52.25
62.21
51.52
54.30
77.50
71.50
78.00
52.00
Mimi-S-24*
60.17
67.57
51.68
68.51
76.50
77.00
76.00
52.00
DWavL-S-16
53.96
69.10
51.41
62.42
92.00
70.00
86.50
49.00
SQ-SMA-16
51.58
51.41
51.79
55.10
83.00
64.00
84.50
50.50
WT-SMA-21
51.22
54.60
52.00
52.75
81.50
78.50
69.00
50.50
Dataset. We use the publicly available dataset LibriHeavy (Kang et al. 2024 containing 56k hours of transcribed speech, and the official validation and test sets of LibriSpeech (Panayotov et al. 2015).
Evaluation Setup. To evaluate our SLMs, we use the ZeroSpeech (Dunbar et al. 2021 sBLIMP and sWUGGY evaluation. The sBLIMP task assesses model perplexity pairs of syntactically correct and incorrect sentences (e.g., the dog sleeps vs. the dogs sleeps). It evaluates the model's understanding of core grammatical phenomena in English. Similarly, sWUGGY mcasures whether the model assigns higher probability to a real word over a phonologically similar non-word (e.g. "brick" vs. "blick"), thus test- ing lexical discrimination. We further assess semantic understanding using Spoken Story-Cloze (sSC) and Topic Story-Cloze (tSC) tasks (Hassid al. 20023), derived from the spoken variant of the StoryCloze dataset (Mostalazadeh et al., 2016). In sSC, the model must choose between the correct continuation and randomly sampled, semantically incompatible adversarial one. This setup probes the model's ability to cap- ture fine-grained causal and temporal commonsense relations. Similarly, in tSC, the adversarial continuation is taken from a different topic, so success reflects the model's capacity to maintain topical coherence.
To evaluate acoustic modeling such as prosody, speaker identity, and sentiment we adopt the SALMon evaluation suite (Maimon et al., 2025c This includes several metrics that test whether the SLM retains and models key acoustic attributes. We focus on two aspects: (1) acoustic consistency, which evaluates the model's sensitivity to changes in speaker, gender, and sentiment; and (2) sentimenttaccuuticc alignment, which tosts whether the model assigns higher scorcs to utterances where the acoustic sentiment aligns with the spoken content. This comprehensive suite allows us to assess both the linguistic and paralinguistic modeling capacities of our SLMs.
Results and Discussions. The results of the SLM experiments are presented in Table 10 To establish clear baseline aligned with the current. literature, we train an SLM using the HuBERT 25 Hz tokenizer, originally introduced TWIST (Hassid al., 2023 using the same configuration as previously described. Following the methodology of Défossez al. (2024), all hybrid tokenizers marked with an asterisk (**1) correspond to SLMs trained with a semantic stream overweight factor of 100. This training setup prioritizes the semantic content over the acoustic content, and could be required for a better disentanglement of the semantic distillation. We maintain the same weighting strategy during evaluation.
The results reveal significant differences in performance across semantic and acoustic evaluation tasks. On semantically-orintte benchmarks (sBLIMP, SWUGGY, sSC, and tSC), most tokenizers exhibit limited se- mantic capacity. The tokenizer achieving the strongest semantic performance is HuBERT, followed by the semantically distilled weighted tokenizers that demonstrate the second-highess semantie performance. Specif-
24

---

ically, by overweighting the semantic stream, Mimi improves from 52.25% to 60.17% accuracy on SBLMMP. A similar trend is observed across all semantic evaluations, with an average performance gain of 6.91 accu- racy points. The same pattern holds when comparing ST-S-16 to ST-S-16*, where the overweighted version achioves superior semantic results. Since semantic information is not clearly localized in a specific stream, WavLM is evaluated without any weighting strategy. Despite this, it still ranks second among unweighted tokenizers, just after HuBERT.
This semantic trend is consistent with our expectations of HuBERT, Mimi, SpeechTokente and WavLM, since they primarily encode or have specific phonetic streams (via distillation self-supervised learning ob- jectives), thereby enhancing performance on linguistic understanding tasks. These results suggest that, when carefully tuned, semantic distillation approaches can rival SSL-based models like HuBERT, though they still fall slightly behind on sSC and tSC. Further investigation is needed to determine the optimal weighting be- tween semantic and acoustic streams. contrast, purely acoustic tokenizers such as Encodec and DAC show negligible semantic capability, rendering them unsuitable for this SLM configuration. Similarly, SQCodec and WavTokenizer offer limited semantic utility. We note that contrary to Encodec and DAC, SQCodec and Way'Tokenize obtained stronger semantic scores. One explanation could be due to the type of quantization (i.c. RVQ vs. SVQ / FSQ) or the limited number of strcams. Overall, the performance varies substantially across tokenizers, with only Mimi-S-24 24 approaching HuBERT's baseline on semantic tasks.
WavLM achieves the best acoustic performance on average, with accuracies of 92.00%, 70.00%, and 86.50% on gender, sentiment, and speaker consistency, respectively. This indicates that the model effectively captures and processes acoustic attributes in comparison to other tokenizers. Interestingly, Mimi also shows strong acoustic performance, outperforming HuBERT on each task. Purely acoustic tokenizers such as DAC and EnCodec, or SQ-codec and WavTokkniizrr also outperform HuBERT acoustic evaluations and achieve results comparable to the semantically distilled tokenizers. However, no method yields substantial results on semantic-acoustic alignment, highlighting a limitation of current approaches in jointly modeling and rcasoning over both modalitics.
Summary. Our study reveals that semantic and acoustic performance in SLMs varies significantly across tokenizer typos. HuBERT remains the strongest performer semantic tasks, while WavLM leads in acoustic consistency. Semantically distilled tokenizers, particularly those with semantic stream overweighting, showed promising results by narrowing the semantic gap with HuBERT. These gains, however, come with trade-offs, emphasizing the importance of carefully balancing semantic and acoustic objectives. Overall, our findings suggest that, for now, there is no single tokenizer that excels across all spoken and acoustic tasks.
3.3.2 Text-to-Speech
Background. Text-to-Speech (TTS) is one of the primary applications of audio tokens. Traditional TTS systems typically rely on neural networks that predict mel spectrograms from text (Shen et al.) 2018 Ren ct al. 2019), followed by neural vocoders (Morise et al.] 2016 Kong et al.) 2020 van den Oord et al.) 2016)
to synthesize waveforms. The introduction of discrete audio tokens offers several advantages. Notably, it reframes waveform generation as a classification task over fixed vocabulary, rather than regression over continuous values. This shift enables optimization via categorical distributions and negative log-likelihood loss, which is typically more stable and tractable than regression objectives. Second, off-the-shelf neural codec decoders can reconstruct waveforms directly from token sequences, removing the need to train separate vocoders. Third, discrete tokens reduce sequence length, improving inference efficiency compared to p-law quantization (van den Oord et al. 2016
Prior to the adoption of discrete representations, TTS was largely dominated by non-autoregressive (NAR) models (Ren et al. 2021 2019 Kim et al. 2021) due to their inference speed and stability relative to autoregressive (AR) models (Shen el al. 2018). However, the emergence of neural codecs has renewed interest in AR architectures (Wang et al. 2024c Chen et al. 2025a Yang et al., 2024c), which have demonstrated strong performance in expressive and zero-shot generation settings. Meanwhile, NAR models have also benefited from discrete token supervision, with recent advances incorporating diffusion- and flow-matching-
based methods (Ju et al. 2024 Yang et al. 2024d) to further enhance synthesis quality.
25

---

Experimental Setup. Our models are based customized adaptation of the ESPnet (Tian et al, 2025) implementation of VALT-E (Chen et al. 2025a). To facilitate convergence, we adopt staged training strategy that allows independent optimization of the AR and NAR components. We first perform 10 epochs of AR-only training, followed by 90 epochs of joint training with both AR and NAR layers to improve convergence for some tokenizers. All models use a 12-layer architecture for both AR and NAR decoders, with an attention dimension of 1024 and a dropout rate of 0.2. Following ESPnet conventions, we use looped nominal epochs, with 50,000 samples per epoch. The approach involves using fixed number of data samples per epoch taken sequentially from the dataset until the end of the dataset is reached, at which point training restarts from the beginning. The epoch achieving the lowest validation dWER is chosen for evaluation.
2019 using corresponding phoneme Dataset. We train our model on the LibriTTS dataset (Zen et al.
transcriptions obtained from LibriTTS with Forced Alignment dataset (McAuliffe et al., 2017). The model is trained to generate discrete audio tokens conditioned on phoneme prompts (representing content) and acous- tic code prompts (capturing target speaker characteristics), enabling it to synthesize speech that matches both the textual input and the target speaker's 's voice.
Evaluation Sctup. Wc evaluate all models on all samples in the test split that fall within the length limit. To evaluate the speech naturalness of synthesized speech, we use a pretrained UTMOS model (Saeki et al., 2022). For assessing promunciation accuracy, transcribe both the ground-truth and generated utterances using the Whisper Large model (Radford et al. 2023) with greedy decoding, We compute the degraded Word- Error-Rate (dWER) by treating the ASR prediction of the ground-truth audio as the reference instead of the original transcription. We report mean UTMOS scores and miero dWER values; that is, the Levenshtein edit distance computed over the entire dataset. To measure speaker fidelity, we compute the cosine similarity (SpkSim) between X-vectors extracted from the generated and reference audio using the base variant of WavLM Chen et al., 2022), fine-tuned for speaker verification.
Following ESPnet-Codee, to mitigate the variability of samples arising from using text-conditioned sampling-based generative model, we generale 10 samples sinultaneously for each prompt and choose the best one based on the WER calculated with the original label as the ground truth using Whisper Small, while final dWERs are calculated using the large model. To establish a clear bascline aligned with current literature, we train VALL-E using ESPNet's in-distribution retraining of EnCoded which is trained on LibriTTS data only instead of a mix of speech, audio, and music as in the original model.
We perform a grid search over sampling temperature values & E 0.7, 1.0, 1.2, 1.3 and top- values k e 10, 20, 30, where t controls the sampling temperature and k limits the number of highest considered during top-k sampling. The optimal hyperparameters are selected based on the lowest dWER on a filtered subset of the validation set (67 samples selected from an initial random pool of 100, based on sequence length). These optimal values are then used for evaluation on the test set.
tokens
Results and Discussion. Table presents the performance of various tokenizers on the TTS task. The highest audio quality is achieved by ESPNet EnCodec (Enc-S-24), which obtains UTMOS score of 3.77. This model is trained on specch-only data, likely enabling it to better capture fino-grained speaker characteristics. Notably, the original EnCodec model performs significantly worse, with a UTMOS of only 2.31. We further investigate the impact of training data in Section
The best adherence to the text achieved with WavLM (Ji et al. 2024c) with a dwER of 4.32, which is likely atrribtaabee to the preservation ion of higher-level semantic information. It is followed by Tokenizerr at 4.67, which employs single codebook with larger vocabulary. The model also achieves competitive audio quality at a UTMOS of 2.85. This result may be attributed to the expressive power of the larger vocabulary and the simplicity of single-stream generation.
Discrete WavLM6, achieves the second-best audio quality with a UTMOS of 3.42. It should be noted that during training, this tokenizer showed the most stable and robust results and early convergence, particularly in law-data regimes, and a reasonable speaker similarity at 0.90. These findings indicate that semantic rep-
encodec _4k
26

---

Table 11: Text-to-Speede synthesis results when using the VALL-E speech language model conditioned on
phoneme annotations.
Tokenizer
#Q
UTMOS1
dWER,
SpkSimt
Enc-SMA-22
8
2.31
4.77
0.91
Enc-S-24
8
3.77
5.74
0.91
DAC-SMA-24
8
2.47
11.71
0.88
ST-S-16
8
2.91
5.35
0.91
Mimi-S-24
8
2.60
7.93
0.91
DWavL-S-16
6
3.42
4.32
0.90
WT-SMA-24
2.85
4.67
0.88
resentations derived from self-supervised models are well-suited for TTS, supporting both natural-sounding and phonetically faithful speech synthesis.
In contrast, gencral-purpose acoustic tokenizers such EnCodec (Enc-SMA-24) and DAC (DAC-SMA-24) result. in decreased audio quality, and occasionally, as in the case of the latter, decreased pronunciation fidelity as well. These models likely require the TTS model to learn high-level speech abstractions from raw acoustic features, adding complexity to the generation process. Another possible contributing factor is that these tokenizers were trained on multi-domain audio data, whereas all other tokenizers evaluated were trained exclusively on speech. Finally, SQ-Codec, originally designed for diffusion-based generation with a large vocabulary (~ 20k), failed converge our AR/NAR sctup, likely due to the challenges of modeling such large token space in autoregressive setting. All models achieve comparable levels of speaker similarity (ranging from 0.88 to 0.91).
Summary. Overall, achieving strong TTS performance with discrete tokenizers remains challenging, es- pecially under constrained training conditions. Training with semantic tokenizers lcads to more robust and effective TTS performance compared lo acoustic or semantically distilled tokenizers; however, in high-data regimes with deep models, acoustic tokenizers, such EnCodec, can be competitive with even outperform semantic ones, particularly if they are trained on similar speech data, such shown with Enc-S-24 trained on the same LibriTTS dataset as the TTS itself.
3.3.3 Audio Generation
Generating realistic audio is long-standing goal in generative AI, with applications in media, accessibility, and human-computer interaction. Previous works have explored audio generation under various conditions, including text (Liu et al. 2024b Dong et al. | Saito et al. 2024 Kumar et al. 20244)
Background.
image (Sheffer & Adi 2023 Wang et al. 2024a), video (Luo et al. 2023 Pascual et al. 2024 Zhang et al.
2024b Wang et al.] 2025bb), and multimodal inputs (Jeong al. 2025 Chen et al.] 2025b), as well as unconditional scttings. In this study, we focus on both unconditional and text-conditioned generation, these represent the most common and well-benchumaredd paradigms.
be very broadly categorized into two main categories: diffusion diffusion-based methods Yang ct al. 2023b Huang et al. 2023 Liu et al.) 2023), and language model based approaches (Kreuk et al. 2023: Borsos el al. 2023b Ziv et al. 2024). In this study, we focus on the latter, as the use of discrete audio tokens is more prevalent in such generation approaches. While both autoregressive (Kreuk et al. 2023 Borsos et al. 2023b) and non-autoregressive (Ziv al.] (024) methods have been proposed for audio generation, we focus on AR models in this study. This choice motivated by their typically superior generation quality and their prevalence in recent work, despite the trade-off of slower inference.
Audio generation
27

---

Experimental Setup. We adopt the AudioCraft toolkit (Copet et al. 2023 which provides a training pipeline for text-to-audio synthesis based on MusicGen (Copet et al. 2023) and AudioGen (Kreuk et al. 2023). The framework uses a T5 model (Raffel et al. 2020) to encode the text prompt into a latent conditioning tensor CE T This tensor is passed to a causal decoder Transformer, where each block consists of a causal self-allention layer over previously generated audio tokens, followed by a cross-attention layer on the conditioning tensor. Following the original setup, we adopt delay pattern across streams and apply Classifier-Free Guidance (CFG). We use the base model configuration with approximately 300M parameters and T5-Base as the text encoder. This framework is well-established and supports both multi- stream and single-stream audio tokens. To enable unconditional generation with the same model, we apply CFG dropout during 10% of training steps, a strategy shown to also enhance robustness in both conditional and unconditional settings.
We use the same architecture across all tokenizers to ensure a fair comparison. To balance consistency and computational efficiency, each model trained for 100,000 steps using a batch size of 128 audio samples (cach 10 seconds long), with mix-up augmentation. This corresponds to half the batch size and training steps used in the original AudioGen, while still achieving competitive performance. Note that the effective number of tokens and training time may vary depending on the tokenizer's frame rate and number of codebooks. For generation, we use fixed sampling parameters across all tokenizers, specifically top-k sampling with k = 250, without tuning them for individual models. Additional experiments confirm that the observed trends remain consistent under different sampling hyperparameters, though detailed results are omitted for brevity.
Dataset. We use several audio datasets for training and evaluation, many of which are part of LAJON- AUDIO-630K (Wu et al. 2023c). Specifically, we include AudioCaps (Kim et al., 2019), AudioStock 12 BBC Sound If 13 EpidemicSound FreeSound Free Use Sounds 1o MACS (Morato & Mesaros 2021), and Odeon Sound E We follow Kreuk et al. (2023) and use the official splits of AudioCaps for validation and testing. All other datasets used for training. These datasets vary in sample rate and format. We resample all audio and convert it to mono to match the input requirements of each lokenizer. In total, the training data contains approximately 4,050 hours of audio.
Evaluation Setup. We evaluate both text-conditioned generation and audio continuation with a 2.5 sec- ond audio prompt (and no text condition). We use three objective metrics that provide complementary per- spectives for evaluation. First, we compute the Fréchet Audio Distance (FAD) using the FadTK toolkit (Gui et al. 2024) with the VGGish model. This metric is computed similarly to AudioGen (Kreuk et al. 2023
where FAD is calculated against the AudioCaps test set to mcasure the overall quality of synthesized audio.
Second, we assess semantic consistency using KL Divergence. Specifically, we follow Yang al. (2023b) and compare the output distribution of pre-trained audio classifier, specifically PASST (Koutini et al. 2022 real samples versus model-generate samples for the same conditions. Finally, we evaluate text-audio alignment using the CLAP score (Wu et al. 2023c; Huang et al. 2023). This measures how well the gen- erated audio malches the input prompt. All evaluation metrics used here are generative in nature and are therefore influenced not only by the language model's ability to predict tokens but also by the quality of the vocoder used to synthesize audio. As such, poor metric scores may reflect limitations in the vocoder rather than issues in the encoder or language model.
Results and Discussion. Table 12 summarizes the performance for both text-conditioned generation and audio continuation. Since all Audio LM evaluations rely on generative outputs, final performance is often inffuenced by vocoder quality. As a result, even language model with strong next-token prediction capabilities may underperform if paired with suboptimal vocoder. To better isolate the effect of vocoding, we report reconstruction quality metrics for each tokenizer without involving any language model training. These results highlight notable differences across tokenizers. The 44kHz variant of DAC performs particularly
12 httppp:// wtt.i//aatttooneeett.t..e net/sfx
bbcrewind.co co. uk
14 htap::///wwweep..eiicooommo com/ swtdd/ offectss/
attps://freesouudd
https::/uww. http:///www.ppaaamnnntstiooooooo odeon- effedtss
28

---

Table 12: Comparing performance of audio LMs over different tokenizers. We report FAD, KL divergence, and CLAP score on the AudioCaps test set. We also provide metrics for audio reconstruction. We note that ground truth audio in AudioCaps gets CLAP= 0.311, proving topline. For more information about the
tokenizers see Section 2
Text Cond. Generation
Uncond.
Generation
Reconstruction
Tokenizer
#Q
FAD4
KLD|
CLAPt
FAD
KLD1
CLAPt
EAD1
KLD
CLAP
Enc-SMA-24
8
3.771
1.555
.279
5.996
1.897
.202
3.806
0.456
.281
Enc-M-32
4
10.110
1.788
.295
13.400
2.840
.175
12.611
1.387
.251
Euc-A-16
1.955
1.576
.300
3.548
2.064
.205
1.816
0.419
.273
DAC-SMA-44
9
6.929
1.959
.267
6.732
2.041
.212
2.206
0.242
.299
DAC-SMA-24
9
7.708
1.966
.253
8.196
2.183
.199
4.124
0.446
.281
SQ-SMA-16
7.733
3.078
.151
5.977
2.301
.175
3.460
0.460
.268
WT-SMA-24
2.594
1.463
.291
4.441
2.224
.193
5.018
0.892
.253
well, reaching quality levels comparable the version of EnCodec trained specifically on audio. In contrast, the music-only EnCodec model shows poor reconstruction quality, as expected given its domain-specific training. Apart from the music-only EnCodec, which was not trained on general audio, WayTokenizer exhibits the weakest reconstruction performance among all evaluated tokenizers.
The Audio LM trained on the music-only tokenizer (ENC-M-32) shows weak performance, particularly in terms of FAD. This may be attributed to limitations in the vocoder or the sensitivily of distribution-based metrics. For example, the model may be effective at tokt-token prediction and produce acoustically coherent samples, yet still deviate from the reference waveform distribution. The relatively strong CLAP and KLD scores for text-conditioned generation support this possibility, even though current evaluation metries are insufficient to fully diagnose the cause of the performance drop. In contrast, the general-audio-trainee version of EnCodec achieves the best FAD scores and consistently strong performance across all evaluation settings, emphasizing the value of domain-specific training for audio tokenizers.
Tokenizze achieves strong performance in text-to-audio generation, despite its relatively poor recon- struction quality. One possible explanation is that its single-token stream format simplifies the modeling task for the language model, potentially enabling faster convergence. This performance gap may narrow with additional training compute. In contrast, both DAC variants and SQCodec exhibit weaker results in text-conditioned generation compared to their strong reconstruction performance. For SQCodec, the large and potentially redundant token vocabulary may make next-token prediction more difficult, reducing gen- eration quality. Similarly, while the DAC models offer excellent compression, their structure may be more challenging for autoregressive modeling, resulting in reduced generation performance. This gap may be due to its higher token rate leading to longer sequences, or modeling difficultics specific to DAC.
Summary. Our findings highlight the critical role of domain-specific training for audio tokenizers. Training the language model alone on in-domain data not sufficient: tokenizers must also be trained the same domain ensure strong performance. Our results also show that the best reconstruction performance does not correlate with the best modeling performance. In the future, we encourage the development of evaluation metrics that disentangle modeling ability from vocoder performance, as is common in the speech domain. We also emphasize the need for more robust modeling metrics (Chung et al. 2025
3.3.4 Music Generation
Background. Music generation is particularly challenging due to the structural complexity of musical compositions, which involve diverse instrumentation, long-term dependencies, and high expectations for
29

---

both acoustic quality and acsthetic coherence. Recent advances in generative models, especially diffusion- based approaches and LLM-based architect have shown strong potential for producing coherent and high-quality music, often conditioned on melodic (Borsos et al. 2023b or text (Huang et al. 2022) prompts.
A dominant paradigm in rext-to-musit generation involves latent diffusion models that operate over VAE- derived continuous representations (Evans et al. 2025 Chen et al.; 2024a Lam et al.] 2023). In contrast, the use of autoregressive language models for music generation over discrete tokens remains an evolving area. Early work such as Jukebox (Dhariwal et al. 2020) employed VQ-VAE (van den Oord et al. 2017) to obtain quantized features for autoregressive modeling. More recent developments in neural audio codecs have shown that their latent spaces can serve as compact and expressive discrete representations of music. Building on this, several studies (Borsos et al.] 2023b a Rouard et al.] 2024) have explored LM-based music generation using various codec tokenizers and decoding strategies. In this section, we evaluate the effectiveness of discrete tokenizers in LM-based music generation, considering both text-conditioned synthesis and unconditional generation (i.c., music continuation).
Experimental Sctup. Our experimental setup for text-to-music synthesis largely follows the same con- figuration described in the audio generation section. We use the AudioCraft toolkit (Copet et al. 2023) with decoder-only transformer and cross-attention over a T5-Base (Raffel et al. 2020) text encoder. The model has approximately 300M parameters and is trained using dlassffier-free guidance (CFG). Key differences from the audio sctup include the use of a higher CFG dropout rate of 30% (vs. 10% for audio), which allocates more emphasis to unconditioned (sell-conditioned) music generation. Additionally, no mix-up augmentations are applied, and models are trained for 200,000 steps using 4 A100 40GB GPUs, each with batch size of 32 samples (10 seconds each). This results in significantly less computing compared to the original MusicGen configuration (192 samples for 1M steps). As in the audio experiments, we use the same architecture across all tokenizers to ensure fair comparison. We adopt autoregressive modeling, which, while less efficient than masked non-autoregressive methods (Garcia et al. 2023 Ziv et al. 2024 is more widely used and known to produce better perceptual quality.
Datasets. For training, we use the genre-balancee Free Music Archive (FMA) dataset (Defferrard et al. 2017). following the setup of stable-audio-open (Evans et al.] 2025). All samples are 30 seconds long, and we follow the official split provided in the dataset repository The training set consists of 84,213 samples, totaling ~702 hours. We combine artist, album, keywords, genres, and titles from the metadata to build the text prompt of the model.
For evaluation, we use two datasets: (1)MusicCaps (Agostinelli et al. 2023 19 which contains 5,347 samples of 10 seconds cach, annotated with descriptive text; and (2) the FMA test split, originally containing 11,238 samples of 30 seconds. To reduce evaluation time while preserving genre coverage, we randomly select 10 clips from each of the 156 genres, resulting in a genre- balance subset of 1,560 samples.
Evaluation Setup. We evaluate music generation models on two lasks: lexl-conditioned generation and unconditioned generation, also referred to as continuation, where 2-second audio clip is used prompt to extend the content in a coherent manner. In addition, we assess the reconstruction performance of cach tokenizer on both test datasets, providing an upper bound on the potential quality of generated outputs. Our evaluation protocol follows the same setup as in the text-to-audio experiments. Specifically, we use three objective metrics: FAD et al.] 2024) computed with the VGGish model to assess audio quality, KLD between PASST classifier outputs outputs(Koutini (Kuuiini et al., 2022) for semantic consistency, and CLAP score (Wul et al.
2023c
Huang al. 2023) to measure alignment with textual prompts.
Results and Discussion.
The evaluation scores MusicCaps and FMA test set are presented in Table Surprisingly, the evaluation score on the MusicCaps is consistently better than those on the FMA- test, despite the theoretical similarity between the FMA-test and the FMA training data. Compared to MusicCaps, the FMA training set only provides very limited text prompts and compromised sound quality. However, thanks to its large dataset size and publicity, it is a natural choice for open-sourced model training
13
18 http:p/:///gituu..oo//////ae
19 htw.://wwww.....atttaaa///////aa
30

---

Table 13: Comparing the performance of text-to-music LMs over different tokenizers on MusicCaps and
FMA-test. The abbreviation of the tokenizer column are shown in Table 2 quantization layers used in the tokenizer.
#Q denotes the number of
Text Cond. Generation
Uncond. Generation
Reconstruction
Tokenizer
#Q
FAD4
KLD4
CLAPt
FAD
KLD4
CLAPt
FAD4
KLDD
CLAP1
MusicCaps
Enc-SMA-21
11.173
2.246
.108
4.632
0.904
.275
2.209
0.259
.358
Enc-M-32
4.264
2.006
.150
2.715
0.890
.282
1.995
0.356
.399
DAC-SMA-44
8.398
2.214
.19
3.724
0.784
.282
0.927
0.182
.340
DAC-SMA-21
9
9.403
2.127
.093
4.001
0.820
.277
1.335
0.209
.358
SQ-SMA-16
4
14.211
2.810
.064
5.163
0.979
.270
2.078
0.258
.338
WT-SMA-24
17.050
2.792
.056
5.550
1.105
.251
1.984
0.414
.336
FMA
Enc-SMA-21
15.380
2.161
.059
14478
1.827
.065
1.013
0.287
.141
Enc-M-32
8.871
1.299
.078
8.357
1.006
.079
0.784
0.344
.13
DAC-SMA-44
8.115
1.543
.062
6.398
1.100
.055
0.494
0.196
.158
DAC-SMA-21
9
8.789
1.746
.039
7.002
1.405
.043
0.708
0.222
.125
SQ-SMA-16
4
9.426
2,412
.048
4.690
1.592
.070
0.956
0.327
.133
WT-SMA-24
16.511
1.881
.030
6.890
1.414
.047
0.631
0.368
.129
We observe that models trained on FMA are to extent adaptable to other datasets. We believe the quality of the FMA dataset explains the lower evaluation scores the FMA-test. On this issue, we also want to call for efforts within this community to make prompt-rich text-to-music datasets publicly available.
Evans al. 2025
Regarding reconstruction scores, EnCodec-32k tokenizer (despile being trained exclusively on music) does not consistently produce the highest quality. WarTokenizer achieves the best FAD score for reconstructed audio. DAC-44k and DAC-24k achieve the lowest KLD scores, indicating strong preservation of acoustic content. For text consistency, EnCodec-24k and DAC-24k perform best based CLAP scores. Overall, DAC tokenizers at both sampling rates show the strongest reconstruction performance across metrics.
For text-conditioned generation, the music-specific tokenizer (EnCodec-32k) demonstrates clear advantages across all metrics and evaluation datasets. Among the multi-domain tokenizers, DAC-44k consistently out- performs DAC-24k, EnCodec-24k, and the single-stream WayTokenizer This performance gap may stem from DAC-44k's higher bitrate, which likely enables it produce richer and more expressive representation, essential factor for effective language modeling in music generation tasks.
For the unconditional generation task, EnCodec-32k and DAC-44k generally produce higher-quality out- puts. An exception is obscrved on the FMA datasot, where SQ-Codec achieves better generation quality, as indicated by the FAD seore. Across both datasets, unconditional generalion consistently outperforms text-conditioned generation. We attribute this to the higher CFG dropout rate used during training, which exposes the models to more unconditioned scenarios and improves their ability to generate coherent con- tinuations. Additionally, the poor quality of text prompts the FMA dataset, also evident from its lower reconstruction scores, likely hinders the models' performance text-conditioned tasks.
31

---

Eacodoc
DAC
SOCodac
WirTokenizer
Disczete WevlM
Mimi
Speech Tokemior
Discrtminatiive
Dicctiinaative (Acoustic)
Reconstruction (AppBcctioo))
Reconstruction
Generative
Speech M (Phonstic)
Text
Speech
Speech
(a) Speech Ranking
Recenstrurzion
Recomstruttio
Uncone
Uscond
Brat to
Test to
(b) Audio Ranking
Music Ranking
Figure 4: Average ranking of audio tokenizers across three domains: speech, general audio, and music.
Summary. For music LM, we observe that tokenizers with higher sample rates and multi-codebook,, asso- ciated with higher bitrates, tend perform better. This contrasts with audio and speech generation, where higher bitrate lokenizers temporal structure, benefits more from detailed representations, whereas such granularity may be excessive or less critical for general audio tasks. Additionally, unconditional generation consistently outperforms text- conditioned generation, emphasizing the benefits of providing melody prompts in music generation tasks.
harder to model. We hypothesize that music, with its complex harmonic and
3.4 General Trend
Figure 4 summmarizes the overall ranking of audio tokenizers in three domains: speech, general audio, and music. These rankings are computed by first sorting the performance each tokenizer per metric within each task (with rank as worst and rank N as best for N tokenizers), then averaging the ranks across all tasks the respective category. For tasks with multiple metries, rankings are computed separately for each metric and then averaged. The resulting radar charts illustrate the average rank of each tokenizer across the following high-level categories, with higher values indicating better performance. For speech, distinguish between two types of reconstruction: Reconstruction (Signal), which captures low-level fidelity (c.g., UTMOS), and Reconstruction (Application). which reflects the performance of resynthesized audio in downstrremm tasks (e.g., WER for ASR, speaker similarity for SV). Downstream tasks are categorized as either Discriminative or Generative. speech, Discriminative tasks are further split into Content-level, which require higher-level semantic understanding (e.g., ASR, intent classification, keyword spotting), and Acoustic-leved which depend more fine-grained acoustic cues (e.g., emotion recognition, speaker ID,, speaker verification). Language
32

---

modeling tasks are grouped into two types: Text-conditioned and Unconditional (i.e., contimattion generation). For speech, we include SpeechLM with separate subcategories for phonetic and acoustic metrics.
The radar charts highlight general trends in tokenizer strengths and weaknesses across domains. No tokenizer consistently outperforms others on all axes. The performance is strongly task- and Nomain-dependent Some models excel at reconstruction but fall short semantic modeling, while others achieve strong downstream results despite poorer signal fidelity. These plots are not meant to give strict recommendations, but rather to provide a high-level overview of performance trends. For real-world applications, we encourage referring to the full benchmark tables and task-specific analyses to identify the most appropriate tokenizer for the target use case.
4 Ablation Studies
While the above discussion has extensively evaluated publicly available discrete audio tokens across various applications, conducting fair comparisons between these codecs remains challenging. The development of audio tokenizers inherently involves numerous hyperparameters, including codebook setups, quantization algorithms, and training data composition, all of which significantly influence performance. This variability in model design and implementation creates substantial obstacles for researchers attempting to make mean- ingful comparisons, ultimately hindering both a comprehensive understanding of cxisting approaches and a systematic exploration of new audio tokenizer designs.
In this section, we aim to mitigate this issue by conducting experiments in carefully controlled setup, Specifically, we use ESPnet-Codee (Shi et al. 2024c) as the base training framework to evaluate the effects of training data, codebook setups, quantization methods, and pre-trained model distillation.
4.1 Experimental Setups
To ensure reproducibility and isolate key variables in our experimental invesligation, we establish a method- ical framework that controls for potential confounding factors. First, we present our training data in various domains and sampling rates. Next, we establish uniform implementation protocols for model architecture and hyperparameter configuration, enabling direct performance comparisons across model variants. Building this controlled foundation, we present a comprehensive set of experimental models, each systematically varying only the target parameters under investigation. Finally, we detail our evaluation methodology, in- cluding metrics and testing environments, to provide consistent basis for assessing model performance and drawing meaningful conclusions about codec effectiveness.
Training Dataset. We prepare the dataset in three major domains, including speech, general audio, and music. All data in the three domains sourced from the AMUSE dataset discussed in ESPnet-Codec (Shi et al. 2024c). The AMUSE dataset is combination of high-quality datasets for codec training purposes. For speech, it contains DAPS, DNS Challenge 4, Commonvoice, VCTK, AISHELL3, Googlci18n- TTS corpora,
and Mexican endangered languages (Dubey et al., 2024 Kuhn et al. 2014 Yamagishi et al. 2019 Shi
let al. 2021a by Amith et al. 2021 Amith & López Francisco 2022 Amith & Castillo Castillo 2021 Ardila et al. 2020 Shi et al. (2021d). For general audio, it contains all data from the AudioSet unbalanced training set (Gemmeke et al. 2017). For music, it contains MusDB, Jamendo, OpenSinger, StyleSing111,
M4Singer, Kiritan-singing, Oniku Kurumi Utagoe database, Natsume Singing database, Opencpop. ACF-
KiSing (excluding original voices), PJS, and JSUT singing (Rafi et al. 2017 Bogdanov al. 2019: Huang
et al. 2021 Dai al. 2023 Zhang al. 2022 Ogawa & Morise 2021 Wang al.] 2022, Shi et al.
2024a
Koguchi et al. 2020 Takamichi et al. 2020) To ensure equal consideration of the three domains and reduce the effect of unbalanced data distributions, we randomly sample 1k hours of data from each domain over the AMUSE dataset. The following experiments are conducted in either one of the domain-specific datasets or a combination of all three subsets. For each set of training data, we provide both 16 kHx version and 44.1 kHz version to consider the effect of different sampling rates.
Model Implementation. For our experiments, we utilize the Descript Audio Codec (DAC) framework implemented in ESPnet-Codec (Shi et al. 2024c Kumar et al. 2023) as the foundation for neural-codec
33

---

Table 14: Summary of models used in the ablation study across 16kHz and 44.1kHz setups. Models are grouped by quantization method, RVQ, SVQ, FSQ, and Unit-basod, with all models using the DAC backbone except D, which adopts Uni-HiGGGAN S, A, and M denote training on speech, general audio, and music, respectively. The "+" symbol indicates the use of distillation from SSI-based reprosentations. Checkmarks (V) indicate domain-specific training data used for each model variant.
Data Domains
Model
Base
Quantization
Distillation
Speech
Audio
Music
RVQ-S
RVQ-S+
RVQ-A
DAC
RVQ
RVQ-M
RVQ-3
SVQ-S
SVQ-S+
SVQ-A
DAC
SVQ
SVQ-M
SVQ-3
FSQ-S
FSQ-A
DAC
FSQ
FSQ-M
FSQ-3
K-means-S
Unit-HifGAN
K-means
training. To evaluate discrete SSL approaches, employ the discrete unit-based HiFiGAN vocoder implemented by Yan et al. (2023).
Our ablation studies focus on two key aspects: quantization techniques and semantic distillation. For quantization, we compare three distinct approaches: RVQ, single-layer VQ, and FSQ. Additionally, following recent research demonstrating the efficacy of self-supervised learning representations as distillation targets for quantizer training (see Soction 2 for more discussion), we incorporate semantic distillation variants for both our RVQ and VQ-based models systematically evaluate its impact.
Summary of Models. Candidate models summarized in Table 14 For each model listed in the table, we conduct the training at 16 kHz and 44.1 kHz with the corresponding dataset. Here, we mostly follow the previous literature discussed in Section 2 where we ignore model setups with no or limited related work, such as the scenarios of using SSL-based distillation in andio or music domains or the use of Uni-HifiGAN for higher sampling rates. While we standardize core training parameters across all experimental conditions ensure fair comparisons, we implement targeted customization for specific model variants to integrate different ablation factors. Complete documentation of both the standardized parameters and model-specific adjustments is available in our released model checkpoint 20
Evaluation Setup. We follow the reconstruction evaluation protocol outlined in Section 3.1 adapting our methodology to accommodate the specific requirements of different data domains. For evaluation data, wo utilize the LibriSpecch test-clean set (speech), the AudioSet test set (gencral andio), and the MUSDB test set (music).
34

---

Table 15: Ablation experiments on audio tokenizer reconstruction performance (speech domain).
SDR1
SI-SNR
PESQU
UTMOS
DNSMOS P8351
WER,
Spk
Simt
Model),SR((Ha))
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
Ground truth
4.09
4.09
3.18
3.18
2.83
2.83
RVQ-S
4.08
8.24
115
6.38
2.59
3.24
3.35
3.62
3.16
3.16
2.04
2.63
0.67
0.90
RVQ-S+
1.63
8.40
-1.77
5.07
2.22
3.30
3.12
3.65
3.12
3.14
2.12
3.17
0.69
0.89
RVQ-A
0.59
6.92
3.15
2.58
2.92
0.51
0.73
RVQ-M
2.80
6.74
-0.68
4.61
2.00
2.41
1.64
2.33
2.64
2.68
3.20
2.85
0.44
0.56
RVQ-3
2.46
7.50
-1.12
4.03
2.43
3.06
2.71
3.33
2.96
3.08
2.22
2.66
0.61
0.87
SVQ-S
4.90
0.92
-13.59
-2.04
1.43
1.69
2.19
2.61
3.09
3.10
13.16
8.28
0.35
0.53
SVQ-S+
-145
-0.64
-11.74
-3,61
1.42
1.63
2.14
2.40
3.00
3.07
13.71
7.89
0.36
0.52
SVQ-A
11.80
-5.04
-33.46
-9.56
1.19
1.18
1.25
1.26
1.86
1.97
31.40
34.68
0.19
0.14
SVQ-M
-5.19
-4.70
-11.17
-7.89
1.20
1.14
1.29
1.24
2.19
1.36
14.88
33.87
0.15
0.11
SVQ-3
-5.90
-3.09
-15.13
-7.12
1.29
1.79
1.44
2.37
3.10
3.01
22.24
6.71
0.26
0.49
FSQ-S
3.89
4.58
1.75
2.47
2.08
2.10
3.29
3.0G
3.21
3.12
3.57
4.02
0.48
0.66
FSQ A
1.14
1.00
1.94
3.15
5.12
7.59
0.43
0.39
FSQ-M
-1.08
-2.09
-4.39
-4.61
1.39
1.22
1.57
1.26
2.66
2.04
24.46
20.26
0.17
0.16
FSQ-3
2.41
1.29
0.01
-1.28
1.97
1.79
3.06
2.37
3.20
3.01
4.35
6.71
0.44
0.49
K-means-S
-18.21
-42.98
1.05
2.28
2.46
6.78
0.13
Table 16: Ablation experiments on audio tokenizer reconstruction performance (audio and music domain).
Audio
Music
SDR
CI-SDR:
SI-SNR1
VISQOLI
SingMOS;
SDR
CI-SDR
SI-SNR:
VISQOL:
SingMO8
(odee))T((HHte)
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
16
44.1
RVGGS
-1.05
2.85
-1.02
2.51
-10.07
-0.00
4.18
3.81
2.50
2.68
0.45
6.80
0.44
6.75
-2.20
4.85
4.21
4.12
2.67
2.60
RVQ-S+
-3.90
2.52
-8.6 83
2.19
-17.57
-1.14
4.13
3.79
2.57
2.63
-6.73
6.60
-6.70
6.55
-11.10
4.46
4.07
4.24
2.64
2.70
RVQ-A
-0.00
3.5
-0.61
3.21
-5.56
0.48
4.22
3.78
2.63
2.65
5.78
8.24
5.70
8.17
2.87
5.97
4.21
4.12
2.71
2.72
RVQ-M
0.65
3.76
0.60
3.35
-4.07
1.06
4.13
3.62
2.63
2.63
6.84
8.33
6.25
8.26
3.82
6.48
4.18
3.59
2.70
2.68
RVQ-3
0.14
3.99
0.11
3.50
-1.47
0.70
4.17
3.83
2.61
2.65
5.75
8.54
5.88
R.46
3.32
0.34
4.12
4.07
2.68
2.72
SVQ-
-14.80
-8.43
-14.72
-8.07
-31.86
-14.78
3.83
3.28
2.50
2.00
-14.25
4.78
-14.22
-4.76
-27.16
-7.77
3.68
3.64
2.69
2.70
SVQ S+
14.74
9.39
14.66
9.00
30.69
16.37
3.84
3.23
2.54
2.59
15.10
5.24
15.06
5.23
27.89
8.80
3.69
3.60
2.58
2.88
S
13.08
6.50
-13.00
6.45
300.00
12.30
3.56
3.32
2.56
2.59
68.54
0.49
66.52
00.40
-14.76
-3.21
3.08
3.52
2.65
2.69
SVQ-M
4.53
-5.00
-6:47
-5.19
131.43
-10.02
3.94
3.30
2.50
2.588
-0.35
0,512
-0.35
0.52
-2.94
-1.80
3.87
3.31
2.70
2.63
SVQ3
-9.26
-7.47
-9.11
-7.12
110.50
-14.32
3,88
3.34
2.52
2.62
-2.75
-1.35
-2.73
-1.35
-6.75
-1.63
3.72
3.54
2.63
2.73
T8QUS
-7.30
-4.25
-7.96
-1.04
-14.22
-8.1
4.06
3.32
2.60
2.63
-5.04
-0.37
-5.02
-0.37
-8.42
-2.84
3.80
3.65
2.69
3.71
TSQ-A
-2.79
-2.00
-2.76
-1.00
-7.05
-5.37
4.00
3.53
2.63
2.65
0.90
2.62
0.30
2.60
-1.14
0.69
4.00
3.02
2.70
2.73
TSQ M
-3.77
-3.25
-3.33
53.05
-8.23
6.85
4.02
3.41
2.62
2.60
1.54
2.70
1.52
2.77
-0.67
0.68
3.90
3.54
2.71
2.68
3
3.22
2.36
3.19
2.24
7.56
6.01
4.06
3.53
2.61
2.62
0.89
2.75
0.88
2.73
1.38
0.19
3.96
3.74
2.69
2.88
K-mocre8
-21.01
-20.80
-47.37
3.14
2.78
-10.58
-10.60
-66.03
2.82
2.87
Sampling rate considerations necessitated domain-specific evaluation approaches. For speech reconstruction, all evaluations are conducted at 16 kHz, even when testing 44.1 kHz neural codecs, to maintain consistency with the source LibriSpeech dataset. For music evaluation, we use different sampling rates based model capabilities: 16 kHz for models designed at that native rate, and 24 kHz for 44.1 kHz neural codecs, reflecting the upper-frequency limitations in the MUSDB test set. For general audio evaluation, we align the evaluation with the codec models. These adjustments ensure fair comparisons while respecting both technical constraints and the inherent charactoristics of each datasct.
4.2 Results and Discussion
The results of our ablation study are shown in Table and Table 16
We summarize our findings follows:
35

---

Data Domains. Domain alignment between training and testing data has emerged as critical determi- nant of performance in discrete audio representation modeling. Our experiments confirm that reconstruction quality consistently peaks when models are evaluated on domains matching their training data. More sig- nificantly, we observe that even with carcfully balanced multi-domain training datasets, models still exhibit notable performance degradation when assessed on individual domains compared to domain-specific training. These challenges have become increasingly relevant in light of recent andio foundation models, which aim for broad generalization across diverse audio types. Our findings highlight the need for two crucial research di- rections: developing more effective methodologies for balancing domain-speciffe optimization, and addressing the fundamental challenges of cross-domain generulization in discrete andio representation learning.
While prior research has rarely examined sampling rate effects on discrete audio represen- tation, this gap is largely due to methodological challenges in creating controlled comparisons across different rate conditions. Our systematic ablation study addressos this limitation and reveals sampling rate as a sig- nificant factor in model performance. As shown in Table 15 and Table [6 RVQ-based models consistently demonstrate performance improvements across multiple evaluation metrics when trained al 44.1 kHz, even when the reconstructed audio is downsampled to 16 kHz for evaluation. These benefits, however, are not universal across all quantization approaches. Models utilizing Finite Scalar Quantization (FSQ) actually ex- hibited performance degradation on several metrics when trained at higher sampling rates. This contrasting behavior indicates that the relationship between sampling rate and model effectiveness is contingent on the specific quantization methodology employed. Based on these findings, we recommend that future research on discrete audio representation should incorporate sampling rate as critical design parameter, with careful optimization based on the selected quantization approach and target application domain.
Sampling Rate.
Distillation Effect. Prior studies involving distillation from pre-trained models have frequently demon- strated that such distillation supports comparable or improved signal reconstruction, while providing sub- stantial performance benefits for downstream tasks (Du et al., 2023 Défossez et al.) 2024] Zhang et al., 2024a). In our controlled ablation analyses, we similarly obscrved that incorporating distillation from pre- trained speech representations can enhance model performance on cerlain metrics for signal reconstruction, as demonstrated by our comparison between models trained with and without distillation (e.g.. A-S vs. A-S+). However, it should be noted that the domain-specific nature of the pre-trained model may limit generalization, especially when applied to broader domains such as general audio or music, as evidenced in Table 16 This highlights a potential trade-off between achieving high performance in specialized tasks and maintaining broader generalization capabilities.
Quantization Methods. Our experiments demonstrate that different quantization methods significantly impact codec performance. The RVQ modeling consistently outperforms other quantization approaches across most reconstruction metrics. Conversely, SVQ models typically yield the poorest results. An inter- esting exception emerges with FSQ at 16 kHz, which surpasses RVQ in speech quality metrics measured by UTMOS and DNSMOS. Generally, RVQ demonstrates a higher potential for audio quality due to its high- fidelity reconstruction capabilities. However, we caution readers that the performance alignment between audio reconstruction and downstream applications is not guaranteed. The metrics observed in this st study may not directly translate to broader application performance, and further research needed to establish definitive correlations.
5 Conclusion and Future Directions
Discrete units offer several advantages in audio representation. They provide compact, modular, and scalable abstractions that are particularly well-suited for generative tasks such speech synthesis, music generation, and general audio modeling, By converting regression- based waveform modeling into classification tasks, discrete tokenization simplifies both training and inference. These reprosentations are also more efficient in terms of storage and transmission, making them advantageous for resourte-consttriined deployment and streaming applications. Additionally, discrete tokens align naturally with large language model architectures, facilitating integration into multimodal systems. While discrete representations may currently underperform continuous features on certain discriminative tasks, particularly in low-resource or semantically fino- graigra scenarios, recent vances show that they can match or even outperform continuous representations in some
36

---

applications, such as text-to-speech, especially when trained on large-scale data. These trends highlight the growing promise of discrete audio tokenization and motivate continued research to improve their robustness, expressiveness, and generalization across diverse downstream tasks.
Based on our comprehensive analysis, we identify several key observations and open challenges in the design, evaluation, and application of discrete audio tokenizers. These point to promising future research directions:
Scaling Limitations and Generalizability: While our experiments used moderately sized models and datasets, this choice was intentional to ensure fair and controlled comparisons across tokenizers. These settings reflect realistic constraints for many academic and open-source efforts. Importantly, our key findings, such as the superior performance of semantic over acoustic tokenizers for semantic tasks, are aligned with trends observed in larger-scale systems, suggesting that these insights are likely to hold as models and datasets scale up.
Correlation between Reconstruction and Downstream Performance: We observed a clear trade-off between high-fidelity signal reconstruction and downstream task performance. Optimizing tokenizers purely for reconstruction often fails preserve task-relevant features such as phonetics semantic content. This especially evident in tasks where the decoder is not involved (e.g., ASR or SLU). Future research should aim to jointly optimize for both signal fidelity and semantic utility, possibly using multi-task or adversarial training.
Fair and Consistent Evaluation: Tokenizers vary significantly in training data, sampling rate, and domain scope (e.g., speech-only vs. multi-domain). These discrepancies complicate bench- marking. Our study highlights the importance of standardizing evaluation pipelines to allow fair comparison across lokenizers. Establishing unified benchmarks with consistent experimental settings remains an urgent need.
Benchmark vs. Reported Performance Gap: Results reproduced under controlled benchmark settings often fall short of the originally reported numbers. This indicates that some improvements reported in prior works may rely on favorable hyperparameter luning or large-scale resources. There is need for reprocuciiblity--oooudd evaluations and scaling studies to better understand real-world tokenizer performance.
Semantic Distillation Beyond Speech: Most existing distillation- tokenizers focus exclu- sively on speech. Extending semantic distillation techniques music and general audio domains an underexplored direction that could substantially improve discrete token quality across diverse audio tasks.
Discrete Continuous Representations: While discrete tokenizers have shown promising progress, continuous representations often remain superior for speech-language understanding tasks that rely on preserving fine-grained acoustic cues such as prosody, emotion, and speaker traits. However, this performance gap may not be universal. For instance, discrete tokens can be more suitable in settings involving autoregressive or masked generative modeling, where classification- based objectives are advantagcous. Conversely, models based on score matching or diffusion may still benefit from continuous conditioning inputs. Bridging these differences remains an open challenge for integrating audio Lokenizers into multimodal LLMs that require semantic richness.
Toward Unified Tokenizers: Future systems may benefit from unified tokenizers that can support both generative and discriminative tasks across multiple audio domains. Achieving this will likely require architectures that balance streamability, semantic alignment, reconstruction quality, and domain generalization, possibly via modular or hierarchical designs.
Trustworthy concerns such as bias (Ren et al. 2024a and deepfakes (Wu et al, 2024c| Du al. 2025) are required to be considered. Modern discrete-unit- unit-based speoch generation can mimic voices with human-like realism (Chen et al. (2025)), raising risks of misuse by malicious actors e.g- generating fake news using public figure's voice.
Trustworthiness:
37

---

In summary, while some questions about audio tokens remain open, our evaluation provides a comprehensive perspective that highlights general trends, key challenges, and guidelines for selecting a tokenizer. We hope our study offers valuable insights to the research community and helps pave the way for future advancements in this rapidly progressing field.
Author Contributions
All core members participatod equally in the project. While this scction outlines the primary responsibilities of each core contributor, many also contributed to other aspects of the project. Pooneh Mousavi led the overall survey and benchmarking effort. Haibin Wu, Pooneh Mousavi, and Anastasia Kuznetsova co-led the design of the tokenizer taxonomy, with collaboration with Haici Yang and Gallil Maimon. Haibin Wu, Jiatong Shi, and Darius Petermann conducted the reconstruction experiments. Poonch Monsavi led the development of the downstream section, with contributions from Darius Petermann, and Anastasia Kuznetsova. Adel Moumen led the development of the SpeechLM evaluations. Artem Ploujuikov was responsible for the TTS evaluation experiments. Gallil Maimon conducted the AudioLM experiments. Haici Yang led the MusicLM evaluation. Jiatong Shi performed the ablation study in controlled setup. All authors participated in writing, editing, and refining the paper. The advisors provided guidance and critical feedback throughout the project.
Acknowledgments
We thank Jinchuan Tian for valuable discussions with TTS. We also greatly thank Dongchao Yang for sharing his in-depth, hands-on experience with SQ-Codec development and thoughts about the tokenizer taxonomy design, which helped clarify important aspects of our survey. We are grateful to Huizhong Lu for computing support. We thank the NVIDIA Academic Grant. Program for donating GPU hours used for this project. This work was supported by the Cambridge Commonwealth, European & International Trust (scholarship to Adel Mounen); the Natural Sciences and Engineering Research Council of Canada (NSERC); the Digital Research Alliance of Canada (alliancecan.ca): The Israel Science Foundaton, grant 2049/22 (scholarship to Gallil Maimon); an Amazon Rescarch Award (ARA); Electronics and Telccommunictioos Research Institute (ETRI) grant funded by the Korean government [24ZC1100, The research of the basic media/ contents technologies]. We also thank Jean Zay GENCI-IDRIS for their support in computing (Grant 8024-AD01101530 and Grant 2024 A0161015099). Cem Subakan is supported by Discovery Grant RGPIN 2023-05759.
Part of the experiments used the Bridges2 at PSC and Delta/DeltaAT NCSA computing systems through allocation C1S210014 from the Advanced Cyberinfrastruuture Coordination Ecosystem: Services & Support (ACCESS) program, supported by National Science Foundation grants 2138259, 2138286, 2138307, 2137603. and 2138296.
References
Andrea Agostinelli al. MusicLM: Generating music from text. arXiv preprint arXio:2301.1139) 2023.
Eirikur Agustsson, Fabian Mentzer, Michael Tschannen, Lukas Cavigelli, Radu Timofte, Luca Benini, and Lue V Gool. Soft-to-hard vector quantization for end-to-end learning compressible representations. In Proc. NewrIPS, 2017.
Sung Hwan Ahn, Beom Jun Woo, Mingrui Han, Chan Yeong Moon, and Nam Soo Kim. Hilcodec: High- fidelity and lightweight neural audio codec. IEEE Journal of Selected Topics in Signal Processing, 18: 1517-1530, 2024.
Yang Ai, Xiao-Hang Jiang, Ye-Xin Lu, Hui-Peng Du, and Zhen-Hua Ling. Apcodec: A neural audio codec with parallel amplitude and phase spectrum encoding and decoding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Jonathan D. Amith and Rey Castillo Castillo. Audio corpus of yoloxóchitl mixtec with accompanying time- coded transcriptions in elan, 2021. URL htt.btts///w..oppsslll.. www.:opesssrrrr.........
38

---

Jonathan D. Amith and Osbel López Francisco. Audio corpus of totonac recordings from northern puebla and adjacent areas of veracrus, 2022. URL https://www. httpps///www.oeesssr.rrr////////
Jonathan D. Amith, Amelia Domínguez Alcántara, Hermelindo Salazar Osollo, Ceferino Salgado Castañeda, and Eleuterio Gorostiza Salazar. Audio corpus of sierra nororiental and sierra norte de puebla nahuat(I) with accompanying time-code transcriptions in elan, 2021. URL httpss
ooppe/slr.or/990 o2g/92/
Mohamed Anees. Speech coding techniques and challenges: A comprehensive literature survey. Mullimedia Tools and Applications, 83(10):29859 29879, 2024.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common Voice: A massively-mullilnggaal speech corpus. In Proc. LREC, 2020.
Siddhant Arora, Kai- Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yi Yi Lee, Karen Livescu, and Shinji Watanabe. On the landscape of spoken language models: A comprchensive survey. arXic preprint arXiv:2504. 08528, 2025.
Bishnu S Atal. Speech analysis and synthesis by lincar prediction of the speech wave. The journal of the acoustical sociely of America, 47(1A Supplement):65 65, 1970.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. In Proc. NewrIPS, 2020.
He Bai, Tatiana Likhomanenko, Ruixiang Zhang, Zijin Gu, Zakaria Aldench, and Navdeep Jaitly. Dmel: Speech tokenization made simple. arXiv preprint X5x:24007.1583, 2024.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swictojanski, and Verena Rieser. SLURP: A spoken language understanding resource package. In Proc. EMNLP, 2020.
Xiaoyu Bie, Xubo Liu, and Gaël Richard. Learning source disentanglement neural audio codec. In Proc. ICASSP, 2025.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The MTG-Jamendo dataset for automatic music tagging. In Proc. ICML, 2019.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. SoundStorm: Efficient parallel audio generation. arXiv preprint ar Xii:2090 2i::20050,663, 2023a.
Zalán Borsos, Raphaäl Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Do- minik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. AudioLM: A language modeling approach to audio generation. JEEE/ACK ACM Transactions on Audio, Speech, and Lan- guage Processing, 31:2523 2533,, 2023b.
Xavier Bouthillier, Christos Tsiigotiis, François Corneau-Teenne Tremblay, Thomas Schweizer, Lin Dong, Pierre Delaunay, Fabrice Normandin, Mirko Bronzi, Dendi Suhubdy, Reyhane Askari, Michael Noukhovitch, Chao Xue, Satya Ortiz-Gagné, Olivier Breuleux, Arnaud Bergeron, Olexa Bilaniuk, Steven Bocco, Hadrien Bertrand, Guillaume Alain, Dmitriy Serdyuk, Peter Henderson, Pascal Lamblin, and Christo- pher Beckham. Epiittmmoo/ /oioon: Asynchronous Distributed Hyperparameter Optimization, Aug 2022.
URL https: org/ 10 www.//eeoo....... 3478592
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazenzadeh, Emily Mower, Samuel Kim, Jeannetle N Chang, Sungbok Lee, and Shrikanth S Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-350 2008.
Edresson Casanova, Ryan Langman, Paarth Nockhara, Shehzeen Hussain, Jason Li, Subhankar Ghosh, Ante Jukié, and Sing-gil Lee. Low framo-rate speech codec: codec designed for fast high-quality speech LLM training and inference. In Proc. ICASSP, 2025.
39

---

Kai-Wei Chang, Haibin Wu, Yu-Kai Wang, Yuan-Kuei Wu, Hua Shen, Wei-Cheng Tseng, lu-thing Kang, Shang-War Wen Li, and Hung-yi Tee. Speechprompt: Prompting speech language models for speech processing tasks. IEEE/ACM Transactions on Audio, Speech, and Languoge Processing, 2024.
Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Maekaku, and Shinji Watanabe. Exploration of efficient end-to-end ASR using discretized input from self-supervised learning. In Proc. Interspeech, 2023.
Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Berg-Kirkpatrick and Shlomo Dubnov. Musicldm: Enhancing novelty in text-lo-music generation using beat-synchronous mixup strategies. In Proc. ICASSP, 2024a.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale supervsed pre-training for full stack speech processing. IEFE Journal of Selected Topics in Signal Processing, 16(6):1505-1515, 2022.
Sanyuan Chen, Chengyi Wang, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei Hc, Sheng Zhao, and Furu Wei. Neural codee language models are zero-shot text to speech synthesizers. IEEE Trasaccions on Audio, Speech and Language Processing, 33:705 718, 2025a.
William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, and Shinji Watanabe. Towards robust speech representation learning for thousands of languages. In Proc. EMNLP, 2024b.
Ziyang Chen, Prem Seetharaman, Bryan Russell, Oriol Nieto, David Bourgin, Andrew Owens, and Justin Salamon. Video-guided foley sound generation with multimodal controls. In Proc. CVPR, 2025b.
Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. Self-supervised learning with random-projection quantizer for speech recognition. In Proc. ICML, 2022.
Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, and Shinji Watan- abe. Self-supervised speech representations are more phonetic than semantic. In Proc. Interspeech, 2024.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXit:2311.07919 2023.
Yoonjin Chung, Pilsun Eu, Junwon Lee, Keunwoo Choi, Juhan Nam, and Ben Sangbae Chon. Kad: No more fad! an effective and efficient evaluation metric for audio generation. arXiv preprint orXiv:2502.155 15602, 2025.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In Proc. ASRU, 2021.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In Proc. NeurIPS, 2023.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. LibriMix: An open-source dataset for generalizable speech scparation. arXio preprint [xXi::2005.12660 2020.
Santiago Cuervo, Adel Moumen, Yanis Labrak, Sameer Khurana, Antoine Laurent, Mickael Rouvier, and Ricard Marxer. Text-speech language models with improved cross-modal transfer by aligning abstraction levels. arxio preprinl or Yiw:2503.06211 2025.
Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Mong, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. Recent advances in speech language models: A survey. arXie preprinl ar 2rXiu:2410.0377 2024.
Shuqi Dai, Siqi Chen, Yuxuan Wu, Ruxin Diao, Roy Huang, and Roger B Dannenberg. Singstylel11: A multilingual singing dataset with style transfer. In Proc. ISMIR, 2023.
40

---

Michaöl Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. FMA: A dataset for music analysis. In Proc. ISMIR, 2017.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. Transactions on Machine Learning Research, 2023.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410. 10.00037, 2024.
Luca Della Libera, Francesco Paissan, Cem Subakan, and Mirco Ravanelli. Focalcodec: Low-bitrate speech coding via focal modulation networks. arXiv preprint or arXiv:2502.00 044550 2025.
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN; Emphasized channel atten- tion, propagation and aggregation in TDNN based speaker verification. In Proc. Interspeech, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and llya Sutskever. Juke- box: A generative model for music. arXie preprint orXiv:2005.003: 00341, 2020.
Martin Dietz, Markus Multrus, Vaclav Eksler, Vladimir Malenovsky, Erik Norvell, Harald Pobloth, Lei Miao, Zhe Wang, Lasse Laaksonen, Adriana Vasilache, et al. Overview of the EVS codec architecture. In Proc. ICASSP, 2015.
Hao-Wes Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrå, Taylor Berg- Kirkpatrick, and Julian McAuley. Clipsonic: Text-to-audio synthesis with unlabeled videos and pretrained language-viss vision models. In Proc. WASPAA, 2023.
Jiawei Du, Xuanjun Chen, Haibin Wu, Lin Zhang, I Lin, I Chiu, Wenze Ren, Yuan Tseng, Yu Tsao, Jyh- Shing Roger Jang, et al. Coderfake-omni: A large-scale codec- based deepfake speech dataset. arxiv preprint wxXi:2251.082553 2025.
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng, FunCodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. arXiv preprint arXiv:2309.00 07405, 2023.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. ar preprint 3rXiv:2407.05400 2024a.
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al. Cosyvoice 2: Scalable streaming speech synthesis with large language models. arXiv preprint ar Xiv:2244 12. 10117, 17, 2024b.
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mchdi Zohourian, Min Tang, Mehrsa Golestanch, et al. ICASSP 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024.
Ewan Dunbar, Mathieu Bernard, Nicolas Hamilakis, Tu Anh Nguyen, Maurcon de Scyssel, Patricia Rozé, Morgane Rivière, Eugene Kharitonov, and Emmanuel Dupoux. The zero resource speech challenge 2021: Spoken language modelling. In Proc. Interspeech, 2021.
Avishai Elmakies, Omri Abend, and Yossi Adi. Unsupervised speech segmentation: A general approach using speech language models. arXiv preprint XXix:5001003111 2025.
Hakan Erdogan, Scott Wisdom, Xuankai Chang, Zalán Borsos, Marco Tagliasacchi, Neil Zeghidour, and John Hershey. TokenSplit: Using discrete speech representations for direct, refined, and transcript- conditioned speech separation and recognition. In Proc. Interspeech, 2023.
41

---

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resohttion image synthesis. In Proc. CVPR, 2021.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. Proc. ICASSP, 2025.
Cristina Gårbacea, Aäron van den Oord, Yazhe Li, Felicia SC Lim, Alejandro Luebs, Oriol Vinyals, and Thomas C Walters. Low bit-rate speech coding with vq-vae and wavenet decoder. In Proc. ICASSP, 2019.
Hugo F Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. In Proc. ISMIR, 2023.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In Proc. ICASSP, 2017.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nelworks. Communicaaiinns of the ACM, 63(1)): 139-144, 2020.
Yuzhe Gu and Enmao Diao. FSc: Elfficient speech coding with cross-scale residual vector quantized trans- formers. In Proc. EMNLP, 2024.
Azalea Gui, Hannes Gamper, Scbastian Braun, and Dimitra Emmanouilidon. Adapting frechet audio distance for generative music evaluation. Proc. ICASSP, 2024.
Haohan Guo, Fenglong Xie, Kun Xie, Dongchao Yang, Dake Guo, Xixin Wu, and Helen Meng. Socodec: A semantic-ordered multi-stream speech codec for efficient language model based text-to-speech synthesis. In Proc. SLT, 2024a.
Haohan Guo, Fenglong Xie, Dongchao Yang, Hui Lu, Xixin Wu, and Helen M. Meng. Addressing index col- lapse of large-codebook speech tokenizer with dual-decoding product-quantized variational auto-encoder. In Proc. SLT, 2024b.
Yiwei Guo, Zhihan Li, Chenpeng Du, Hankun Wang, Xie Chen, and Kai Yu. Lscodec: Low-bitrate and speakcr-dccoupled discrete speech codec. In Proc. Interspeech, 2025a.
Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review. arXiv preprint arXiv:2502.064 2025b.
Nadav Har-Tuv, Or Tal, and Yossi Adi. Past: Phonetic-acoustic speech tokenizer. arXiv preprint arXiv:2505.14479 2025.
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copel, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, al. Textually pretrained speech language models. In Proc. NewrIPS, 2023.
Tomoki Hayashi and Shinji Watanabe. Discretalk: Text-to-speech machine translation problem. ar:iiv preprint ar Xiv:20005. 05525, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov,, and Al-
delrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451 3460, 2021.
Qingqing Huang, Aren Jansen, Joonscok Lee, Ravi Ganti, Judith Yue Li, and Daniel PW Ellis. Mulan: A joint embedding of music audio and natural language. In Proc. TSMIR, 2022.
42

---

Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Multi-singer: Fast multi- singer singing voice vocoder with large-scale corpus. In Proc. ACMMM, 2021.
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In Proc. ICML, 2023.
Zhichao Huang, Chutong Meng, and Tom Ko. RepCodec: A speech representation codec for speech tok- enization. In Proc. ACL, 2024.
Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. Unity: Two-pass direct speech-to-speech translation with discrete units. In Proc. ACL, 2023.
Fumitada Itakura. Analysis synthesis telephony based on the maximum likelihood method. Reports of the 6th Int. Cong. Acoust., 1968.
Rhuluja Jage and Savitha Upadhya. Celp and melp speech coding techniques. In Proc. WiSPNET, 2016. E. Jang, S. Gu, and B. Poole. Categorical reparameterization with gumbel-softmax. In Proc. ICLR, 2017.
Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound generation from text and video. In Proc. AAAT, 2025.
Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujic Liu, Xize Cheng, et al. Wavchat: A survey of spoken dialogue models. arXiv preprint 11.157741..18577, 2024a.
Shengpeng Ji, Minghui Fang, Ziyue Jiang, Rongjie Huang, Jialung Zuo, Shulei Wang, and Zhou Zhao Language-codec: Reducing the gaps between discrete codec representation and speech language models. arXiv preprint 2002 1 12208, 2024b.
Shengpeng Ji, Ziyue Jiang, Xize Cheng, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, et al. Wavtokenizer: efficient acoustic discrete codec tokenizer for audio language modeling. In Proc. ICLR, 2024c.
Xiao-Hang Jiang, Yang Ai, Ruixin Zheng, Hui-Peng Du, Ye-Xin Lu, and Zhenhua Ling. Mdetcodec: A lightweight mact-based neural audio codec towards high sampling rate and low bitrate sccnarios. In Proc. SLT, 2024.
Xue Jiang, Xiulian Peng, Huaying Xue, Yuan Zhang, and Yan Lu. Cross-scale vector quantization for scalable neural speech coding. In Proc. Interspeech, 2022a.
Xue Jiang, Xiulian Peng, Chengyu Zheng, Huaying Xue, Yuan Zhang, and Yan Lu. End-to-end neural speech coding for real-time communications, In Proc. ICASSP, 2022b.
Xue Jiang, Xiulian Peng, Yuan Zhang, and Yan Lu. Disentangled feature learning for real-time neural speech coding. In Proc. ICASSP, 2023.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Eric Liu, Yichong Leng, Kaitao Song, Siliang Tang, al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In Proc. 1CML, 2024.
Wei Kang, Xiaoyu Yang. Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: A 50,000 hours asr corpus with punctuation casing and context. In Proc. ICASSP, 2024.
Srihari Kankanahalli. End-to-end optimized speech coding with deep neural networks. In Proc. ICASSP, 2018.
43

---

Ilya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin W. Wilson, Jonathan Le Roux, and John R. Hershey. Universal sound separation. In Proc. WASPAA, 2019.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generaling captions for audios in the wild. In Proc. NAACL, 2019.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-ond text-to-spooch. In Proc. ICML, 2021.
Minje Kim and Jan Skoglund. Neural speech and audio coding. arXiv preprint (XX0:2208.06654 2024.
W. Bastiaan Kleijn, Felicia S. C. Lim, Alejandro Luebs, Jan Skoglund, Florian Stimberg, Quan Wang, and Thomas C. Walters. Wavenet based low rate speoch coding. In Proc. ICASSP, 2018.
Junya Koguchi, Shinnosuke Takamichi, and Masanori Morise. PJS: Phoneme-balaneed japanese singing-voice corpus. In Proc. APSIPA ASC, 2020.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. In Proc. NewrIPS, 2020.
Matěj Korvas, Ondřej Plátek, Ondřej Dušek, Lukáš Zilka, and Filip Jurčiček. Free English and Czech telephone speech corpus shared under the CC-BY-SA 3.0 license. In Proc. LREC, 2014.
Khaled Koutini, Jan Schlüter, Hamid Eghhal-zadeh, and Gerhard Widmer. Efficient training of audio transformers with patchout. In Proc. Interspeech, 2022.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: Textually guided audio generation. In Proc. ICLR, 2023.
Nicolas Kuhn, Emmanuel Lochin, Ahlem Mifdaoui, Golam Sarwar, Olivier Mehani, and Roksana Boreli. DAPS: Intelligent delay-aware packet scheduling for multipath transport. In Proc. ICC, 2014.
Rithesh Kumar, Prem Seotharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN. In Proc. NewrIPS, 2023.
Sonal Kumar, Prem Seetharaman, Justin Salamon, Dinesh Manocha, and Oriol Nieto. Sila: Signal-to- language augmentation for enhanced control in text-to-audio generation. arXiv preprint 0rX.022412009799, 2024.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexci Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Compulational Linguistics, 9:1336 1354, 2021.
Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, et al. Efficient neural music generation. In Proc. NewrIPS, 2023.
Ryan Langman, Ante Jukié, Kunal Dhawan, Nithin Rao Koluguri, and Boris Ginsburg, Spectral codecs: Spectrogram- audio codecs for high quality speech synthesis. arXiv preprint arXiv:2406. 05298, 2024.
Siddique Latif, Moazzam Shoukat, Fahad Shamshad, Muhammad Usama, Yi Ren, Heriberto Cuayáhuitl, Wenwu Wang, Xulong Zhang, Roberto Togneri, Erik Cambria, et al. Sparks of large audio models: A survey and outlook. arXiv preprint arXiv:2308.1177 12792, 2023.
Hanzhao Li, Liumeng Xue, Haohan Guo, Xinfa Zhu, Yuanjun Lv, Lei Xie, Yunlin Chen, Hao Yin, and Zhifei Li. Singlc-codec: Single-codebook specch codec towards high-performance speech generation. In Proc. Interspeech, 2024.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In Proc. ICML, 2023.
44

---

Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound. JEEP Journal of Selected Topics in Signal Processing, 2024a.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiugiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yux- uan Wang, and Mark D Plumbley. AudioLDM 2: Learning holistic audio generation with sclf-supervised pretraining, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024b.
Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized vidco-to-audio synthesis
with latent diffusion models. In Proc. NewrIPS, 2023.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time frequency magnitude masking for speech separation. IEEE/ACM Transactions on audio, speech, and language processing, (1(8)::256-1266) 2019.
Chris J Maddison, Andriy Muih, and Yee Whye Teh. The concrete distribution: A contimuous relaxation of discrete random variables. In Proc. ICLR, 2017.
Gallil Maimon and Yossi Adi. Speaking style conversion in the waveform domain using discrete self-supervised units. In Proc. EMNLP, 2023.
Gallil Maimon, Avishai Elmakies, and Yossi Adi. Slamming: Training a speech language model on one gpu in day. arXiv preprint Xrx:22502. 15814, 2025a.
Gallil Maimon, Michael Hassid, Amit Roth, and Yossi Adi. Scaling analysis of interleaved speech-text language models. arXiv preprint arXiv:2504.02335 2025b.
Gallil Maimon, Amit Roth, and Yossi Adi. Salmon: A suite for acoustic language model evaluation. In Proc. ICASSP, 2025c.
Michael McAuliffe, Michaela Socolof, Sarah Mihue, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speoch alignment using kaldi. In Proc. Interspecch, 2017.
Fabian Mentzer, David Minnen, Firikur Agustsson, and Michael Tschannen. Finite scalar quantization:
Vq-vae made simple. In Proc. ICLR, 2024.
Shoval Messica and Yossi Adi. Nast: Noise aware speech tokenization for speech language models. In Proc. Interspeech, 2024.
Irene Martin Morato and Annamaria Mesaros. Diversity and bias in audio captioning datasets. In Proc. DCASE, 2021.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: vocoder-based high-quality speech synthosis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 99(7):1877 1884, 2016.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proc. NAACL, 2016.
Pooneh Mousavi, Luca Della Libera, Jarod Duret, Artem Ploujnikov, Cem Subakan, and Mirco Ravanelli. Dasb-discrete audio and speech benchmark. arXiv preprint ar 14t22406.1124,, 2024a.
Pooneh Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujníkov, Cem Subakan, and Mirco Ravanelli. Semantic token tuning: How should we extract discrete audio tokens from self-supervised models? In Proc. Interspeech, 2024b.
Poonch Mousavi, Shubham Gupta, Cem Subakan, and Mirco Ravanelli. Listen: Learning soft token embed- dings for neural audio llms. arXiv preprint XrXi:22005.185 18517, 2025.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: largo-scale speaker identification dataset. In Proc. Interspeech, 2017.
45

---

Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. The wero resource speech benchmark 2021: Metries and baselines for unsupervised spoken language modeling. In Proc. NeurIPS, 2020.
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. SpiRit-T.M: Interleaved spoken and written language model. Transactions of the Association for Computational Linguistics, 13: 30-52, 2025.
Zhikang Niu, Sanyuan Chen, Long Zhou, Ziyang Ma, Xie Chen, and Shujic Liu. NDVQ: Robust neural audio codec with normal distribution-baseed vector quantization. In Proc. SLT, 2024.
Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47:617-644, 1928.
Ttsuki Ogawa and Masanori Morise. Tohoku Kiritan singing database: singing database for statistical parametric singing synthesis using japanese pop songs. AST, 42(3):140 145, 2021.
Ahmed Omran, Neil Zeghidour, Zalán Borsos, Félix de Chaumont Quitry, Malcolm Slancy, and Marco Tagliasacchi. Disentangling speech from surroundings with neural embeddings. In Proc. ICASSP, 2023.
Yu Pan, Xiang Zhang, Yuguang Yang, Jixun Yao, Yanni Hu, Jianhao Ye, Hongbin Zhou, Lei Ma, and Jianjun Zhao. Pscodee: series of high-fidelity www-bitrate neural speech codecs leveraging prompt encoders. arXiv preprint 2rXiv:2444.022 02702, 2024.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based public domain audio books. In Proc. ICASSP, 2015.
Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, and Xubo Liu. Scaling transformers for low-bitrate high-quality speech coding. In Proc. ICML, 2025.
Santiago Pascual, Chunghsin Yeh, Toannis Tsiamas, and Joan Serrå. Masked generative video-to-audio transformers with enhanced synchronicity. In Proc. ECCV, 2024.
Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhno Zhang, and Kai Yu. A survey on speech large language models. arXiv preprint arXiv:2410. 18908, 2024.
Darius Petermann, Seungkwon Beack, and Minje Kim. Harp-net: Hyper-auboencoded reconstruction prop- agation for scalable neural audio coding. In Proc. WASPAA, 2021.
Karol Piczak. Esc: Dataset for environmental sound classification. In Proc. ACM, 2015.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised repre- sentations. In Proc. Interspeech, 2021.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speed translation using self-supervised pre-training and data augmenta- tion. In Proc. Interspeech, 2022.
Schuyler Quackenbush. Mpeg unified speech and audio coding. IEEE MultiMedia, 20(2):72-78) 78, 2013.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Tya. Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. ICML, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1 1-70 2020.
46

---

Zafar Rafii Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The MUSDB18 corpus for music separation, December 2017. URL https://doi.orr: org/ 10. 5281/zenodo. 1117872
Mirco Ravanelli, Titouan Parcollet, Adel Moumen, Sylvain de Langen, Cem Subakan, Peter Plantinga, Yingzhi Wang. Poonch Mousavi, Luca Della Libera, Artem Ploujnikov, al. Open-source conversational ai with speechbrain 1.0. Journal of Machine Learning Research, 25( 25(33)::1 2024.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. Proc. ICASSP, 2022.
Wenze Ren, Yi-Cheng Lin, Shang-Cheng Chou, Haibin Wu, Yi-Chiao Wu, Chi-Chun Lec, Hung-yi Lee, Hsin-Min Wang, and Yu Tsso. Emo-codec: An in-depth look emotion preservation capacity of legacy and neural codec models with subjective and objective evaluations. In Proc. APSIPA ASC, 2024a.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Proc. NeurIPS, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In Proc. ICLR, 2021.
Yong Ren, Tao Wang, Jiangyan Yi, Le Xu, Jianhua Tao, Chu Yuan Zhang, and Junzuo Zhou. Fewer-token neural speech codec with (ime-invariant codes. In Proc. ICASSP, 2024b.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. Tn Proc. CVPR, 2022.
Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source separation. In Proc. ICASSP, 2023.
Simon Rouard, Yossi Adi, Jade Copet, Axel Roebel, and Alexandre Défossez. Audio conditioning for music generation via discrete bottleneck features. In Proc. ISMIR, 2024.
Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, et al. AudioPaLM: A large language model that can speak and listen. arXiv preprint arxi::2306. 12:006.2200 12925, 2023.
Takaaki Saeki et al. UTMOS: U'rukyo-Srrllab System for VoiceMOS Challenge 2022. In Proc. Interspeech, 2022.
Kohei Saijo, Gordon Wichern, François G. Germain, Zexu Pan, and Jonathan Le Roux. Tf-lcoformmr: Trans- former with local modeling by convolution for speech separation and enhancement. In Proc. IWAENC, 2024.
Koichi Saito, Dongjun Kim, Takashi Shibuya, Chich-Hsin Lai, Zhi Zhong, Yuhta Takida, and Yuki Mitsufuji. SoundCTM: Uniting score-based and consistency models for text-to-sound generation. In Proc. NeurIPS Workshops, 2024.
Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez. From discrete tokens to high-fidelity audio using multi-band diffusion. In Proc. NewrIPS, 2023.
Manfred Schroeder and B Atal. Code-excited lincar prediction (celp): High-quality speech very low bit rales. In Proc. ICASSP, 1985.
Claude E Shannon. A mathematical theory of commuication. The Bell system technical journal, 27(3): 379-423, 1948.
Roy Sheffer and Yossi Adi. I hear your true colors: Image guided audio generation. In Proc. ICASSP, 2023.
47

---

Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, Rif A. Saurous, Yannis Agionvrgiannakis, and Yonghui Wu. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions. In Proc. ICASSP, 2018.
Jiatong Shi, Jonathan D Amith, Xuankai Chang, Siddharth Dalmia, Brian Yan, and Shinji Watanabe. Highland Puebla. Nahuatl speech translation corpus for endangered language documentation. In Proc. AmericasNLP, 2021a.
Jiatong Shi, Jonathan D Amith, Rey Castillo García, Esteban Guadalupe Sierra, Kevin Duh, and Shinji Watanabe. Leveraging end-to-end ASR for endangered language documentation: An empirical study on Yolóxochitl Mixtec. In Proc. EACL, 2021b.
Jiatong Shi, Yueqian Lin, Xinyi Bai, Keyi Zhang, Yuning Wu, Yuxun Tang, Yifeng Yu, Qin Jin, and Shinji Watanabe. Singing voice data scaling-up: An introduction to ACE-Opencpop and ACE-KiSing, In Proc. Interspecch, 2024a.
Jiatong Shi, Xutai Ma, Hirofumi Inaguma, Anna Sun, and Shinji Watanabe. MMM: Multi-layer multi- residual multi-stream discrete speech representation from sele-supervised learning model. In Proc. Inter- speech, 2024b.
Jiatong Shi, Jinchuan Tian, Yihan Wu, Jee-weon Jung, Jia Qi Yip, Yoshiki Masuyama, William Chen, Yuning Wu, Yuxun Tang, Massa Baali, et al. ESPnet-Coder Comprehensive training and evaluation of neural codecs for audio, music, and speech. In Proc. SLT, 2024c.
Jiatong Shi, Hye-jin Shim, Jinchuan Tian, Siddhant Arora, Haibin Wu, Darius Petermann, Jia Qi Yip, You Zhang, Yuxun Tang, Wangyou Zhang, et al. VERSA: A versatile evaluation toolkit for speech, audio, and music. In Proc. NAACL, 2025.
Jing Shi, Xuankai Chang, Tomoki Hayashi, Yen-Ju Lu, Shinji Watanabe, and Bo Xu. Discretization and re-synthesis: an alternative method to solve the cocktail party problem. ar preprint arXiv:21 12.0212200000 12.00988, 2021c
Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. AISHELL-3: A multi-speaker mandarin TTS corpus. In Proc. Interspeech, 2021d.
Amitay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken lan- guage modeling. In Proc. ICASSP, 2023.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high- quality andio synthesis. In Proc. ICLR, 2024.
Hubert Siuzdak, Florian Grötschla, and Luca A Lanzendörfer. SNAC: Multi-scale neural audio codec. arxiv preprint Xrv:24440 14411, 2024.
Shinnosuke Takamichi, Ryosuke Sonobe, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko Tanji, and Hiroshi Saruwatari. JSUT and JVS: Free japanese voice corpora for accelerating speech synthesis research. AST, 41(5):761-7688 2020.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. Proc. ICLR, 2024a.
Yuxum Tang, Yuning Wu, Jiatong Shi, and Qin Jin. SingOMD: Singing oriented multi-resolution discrete representation construction from speech models. In Proc. Interspeech, 2024b.
Jinchuan Tian, Jiatong Shi, William Chen, Siddhant Arora, Yoshiki Masuyama, Takashi Mackaku, Yihan Wu, Junyi Peng, Shikhar Bharadwaj, Yiwen Zhao, et al. ESPnet-SpeechLM:: An open speech language model toolkit. In Proc. NAACL, 2025.
48

---

Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian FATRAS, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09955 2023.
Arnon Tureizky and Yossi Adi. Last:
Language model aware speech tokenization. arviv preprinl
arXiv:2409.0327 03701, 2024.
George Tzanetakis and Perry Cook. Musical genre classification of audio signals. IEEE Transactions speech and audio processing, 10(5):293 302, 2002.
Cassia Valentini-Botinhan Xin Wang, Shinji Takaki, and Junichi Yamagishi. Investigating RNN-basod speech enhancement methods for noise-robust text-to-speech. In Speech Synthesis Workshop, pp. 146 152, 2016.
Jean-Mare Valin and Jan Skoglund. LPCNet: Improving neural speech synthesis through linear prediction. In Proc. ICASSP, 2019a.
Jean-Mare Valin and Jan Skoglund. real-time wideband neural vocoder 1.6 kb/s using Ipenet. In Proc. Interspeech, 2019b.
Jean-Mare Valin, Koen Vos, and T Terriberry. Rfc 6716: Definition of the opus audio codec, 2012.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXio:1609.03/23 2016.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. In Proc. NewrIPS, 2017.
Benjamin van Niekerk, Marc-André Carbonneau, Julian Zaädi, Matthew Baas, Hugo Seuté, and Herman Kamper. A comparison of discrete and soft speech units for improved voice conversion. In Proc. ICASSP, 2022.
Shikhar Vashishth, Harman Singh, Shikhar Bharadwaj, Sriram Ganapathy, Chulayuth Asawaroengchai, Kar- tik Audhkhasi, Andrew Rosenberg, Ankur Bapna, and Bhuvana Ramabhadran. STAB: Speech tokenizer assessment benchmark. arXiv preprint arxi:22409.00 02904, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, 2017.
E. Vincent, R. Gribonval, and C. Fevotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462-144) 1469, 2006.
Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proc. AAAI, 2024a.
Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. Viola: Conditional language models for speech recognition, synthesis, and translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 3709-3716, 2024b.
Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujic Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as versatile speech transformer. TEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:3355-388 2024c.
49

---

Ye Wang and Mikka Vilermo. Modified discrete cosine transform: Its implications for audio coding and error concealment. Journal of the Audio Engineering Society, 51 61, 2003.
Yingzhi Wang, Poonch Mousavi, Artem Ploujuikov, and Mirco Ravanelli. What are they doing? joint audio-speech co-reasoning. In Proc. ICASSP, 2025a.
Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient vidco-to-audio generation network with rectified flow matching. In Proc. NeurIPS, 2025b.
Yu Wang, Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. In Proc. Interspeech, 2022.
Zhichao Wang, Ynanzhe Chen, Xinsheng Wang, Lei Xie, and Yuping Wang. Streamvoice: Strcamable context-aware language modeling for real-time zero-shot voice conversion. In Proc. ACL, 2024d.
Ziqian Wang, Xinfa Zhu, Zihan Zhang, YuanJun Lv, Ning Jiang, Guoqing Zhao, and Lei Xic. SELM: Speech enhancement using discrete tokens and language models. In Proc. ICASSP, 2024e.
Pete Warden. Speech Commands: A dataset for linited- speech recognition. arXiv preprint arXiv:18040030 2018.
Scott Wisdom, Hakan Erdogan, Daniel PW Ellis, Romain Serizel, Nicolas Turpault, Eduardo Fonseca, Justin Salamon, Prem Scotharaman, and John R Hershey. What's all the fuss about free universal sound separation data? In Proc. ICASSP, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Pln, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2019.
Haibin Wu, Kai-Wei Chang, Yuan-Kuei Wu, and Hung-yi Lee. Speechgen: Unlocking the generative power of speech language models with prompts. arXiv preprint -XXi:2306.02207, 2023a.
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kai-Wei Chang, Ho-Lam Chung, Alexander H. Liu, and Hung-yi Lee. Towards audio language modeling-an overview. arXiv preprint Xrx::22402. 13236, 2024a.
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kai-wei Chang, Jiawei Du, Ko-Han Lu, Alexander H Liu, Ho-Lam Chung, Yuan-Kuei Wu, Dongchao Yang, et al. Codee-SUPERBB SLT 2024: A lightweight benchmark for neural audio codec models. In Proc. SLT, 2024b.
Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuan-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai-Wei Chang, Alexander H Liu, and Hung-yi Lee. Codec-SUPERI: An in-depth analysis of sound codec models. In Proc. ACL, 2024c.
Haibin WI, Naoyuki Kanda, Sefik Emre Eskimez, and Jinyu Li. Ts3-codec: Transformer-based simple streaming single codec. arXiv preprint Xrx: 11.12111.88803 18803, 2024d.
Haibin Wu, Yuan Tseng, and Hung-yi Lee. Codecfake: Enhancing anti-spoofing models against deepfake audios from codec-based speech synthesis systems. In Proc. Interspecch, 2024c.
Haibin Wi,, Yuxuan Hu, Ruchao Fan, Xiaofei Wang, Kenichi Kumatani, Bo Ren, Jianwei Yu, Heng Tu, Lijuan Wang, Yao Qian, et al. Towards efficient specch-text jointly decoding within one speech language model. Xixiv preprint arXiv:2506. 04518, 2025.
Yi-Chiao Wu, Israel D. Gebru, Dejan Marković, and Alexander Richard, Audiodec: An opunesource stream- ing high-fidelity neural audio codec. In Proc. ICASSP, 2023b.
50

---

Yuning Wu, Chunlei Zhang, Jiatong Shi, Yuxun Tang, Shan Yang, and Qin Jin Toksing: Singing voice synthesis based on discrete tokens. In Proc. Interspeech, 2024f.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- win-KKikppatrick, and Shlomo Dubnov. Targe- scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In Proc. ICASSP, 2023c.
Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodec: Pushing the limits of low- bitrate neural speech codec. arXiv preprint (cXic:2409.05377 2024.
Huaying Xue, Xiulian Peng, and Yan Lu. Low-latency speech enhancement via speech token generation. In Proc. ICASSP, 2024.
Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. CSTR VCTK corpus: Euglish multi- speaker corpus for CSTR voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR), pp. 271 350, 2019.
Brian Yan, Jiatong Shi, Yun Tang, Hirofumi Inaguma, Yifan Peng, Siddharth Dalmia, Peter Polak, Patrick Fernandes, Dan Berrebbi, Tomoki Hayashi, et al. ESPnct-ST-v2. Multipurpose spoken language transla- tion loolkit. In Proc. ACL, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint ar Xrxiv:2 12. 15115, 2024a.
Dongehao Yang, Songxiang Liu, Rongjic Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. HiFi-Codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint or urXiv:2305.027 027655 2023a.
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Specch, and Language Processing, 31:1720 1733, 2023b.
Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjic Huang, Xiang Li, Xu Tan, Xixin Wu, and Helen M. Meng. UniAudio 1.5: Large language model-driven audio codec is a few-shot audio task learner. In Proc. NewrIPS, 2024b.
Dongehao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Haohan Guo, Xuankai Chang, Jiatong Shi, Jiang Bian, Zhou Zhao, et al. UniAudio: Towards universal audio generation with large language models. In Proc. ICML, 2024c.
Dongchao Yang, Dingdong Wang, Haohan Guo, Xueyuan Chen, Xixin Wu, and Helen Meng. Simplespeech: Towards simple and efficient text-to-speech with scalar latent transformer diffiusion models. In Proc. Interspeech, 2024d.
Haici Yang, Kai Zhen, Scungkwon Beack, and Minje Kim. Sourco-aware neural speech coding for noisy speech compression. In Proc. ICASSP, 2021.
Haici Yang, Wootaek Lim, and Minje Kim. Neural feature predictor and discriminative residual coding for low-bitrate speech coding. In Proc. ICASSP, 2023c.
Haici Yang, Inscon Jang, and Minje Kim. Generative de-quantization for neural speech codoc via latent diffusion. In Proc. ICASSP, 2024e.
Haici Yang, Jiaqi Su, Minje Kim, and Zeyu Jin. Genhancer: High-fidelity speech enhancement via generative modeling discrete codec tokens. In Proc. Interspeech, 2024f.
51

---

Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Tiu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proc. AAAI, 2025.
Jia Qi Yip, Shengkui Zhao, Dianwen Ng, Eng Siong Chng, and Bin Ma. Towards andio codec-basod speech separation. In Proc. Interspeech, 2024.
Iddo Yosha, Gallil Maimon, and Yossi Adi StressTest: Can your speech LM handle the stress? arXiv preprint (XXx:2505.22766 2025.
Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, and Mirco Ravanelli. Speech self-supervised representation benchmarking: Are we doing it right? In Proc. Interspeech, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. SoundStream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 495-507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from librispeech for text-to-speech. In Proc. Interspeech, 2019.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 4-i-e-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint 2xXiv:241.026612 2024.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Yuxiao Dong, Jie Tang, et al. Scaling speech-text pre-training with synthetic interleaved data. In Proc. ICLR, 2025.
Lichao Zhang, Ruiqi Li, Shoutong Wang, Liqun Deng, Jinglin Liu, Yi Ren, Jinzheng He, Rongjie Huang, Jieming Zhu, Xiao Chen, et al. M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus. In Proc. NeurIPS, 2022.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Tokenize.::, Unified speech tokenizer for speech large language models. In Proc. ICLR, 2024a.
Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Folcycrafter: Bring silent vidcos to life with lifelike and synchronized sounds. arxi preprint arXiv:2407.01494, 2024b.
Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. Google USM: Scaling automatic speech recognition beyond 100 languages. arXiv preprint Xic::23003.0 010037, 2023.
Kai Zhen, Mi Suk Lee, Jongmo Sung, Scungkwon Beack, and Minje Kim. Psychoacoustic calibration of loss functions for efficient end-to-end neural audio coding. IEEE Signal Processing Letters, 27:2159 2163, 2020. doi: 10.1109 /LS./ LLSP.2020.0399666
Rui-Chen Zheng, Hui-Peng Du, Xiao-Hang Jiang, Yang Ai, and Zhen-Hua Ling. Ervq: Enhanced residual vector quantization with sairr--nddinter-codeeee optimization for neural audio codecs. IEEE Transac- tions on Audio, Speech and Language Processing, 2025.
Junzuo Zhou, Jiangyan Yi, Yong Ren, Jianhua Tao, Tao Wang, and Chu Yuan Zhang. Wmeodee: End-to-end neural speech codec with deep watermarking for authenticity verification. arXiv preprint or XXxv:2200.19122 2024.
Alon Ziv, Itai Gat, Gaël Le Lan, Tal Remez, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-aullorerressvve transformer. In Proc. ICLR, 2024.
52

---

A
Additional Evaluation for Reconstructed Audio Quality
Table I7 shows the UTMOS V2 scores for different audio tokenizers.
Table 17: UTMOS V2 scores of audio tokenizers across different bitrates.
UTMOS
(kbps)
V2
1.5
1.40
Eue-SMA-24
6.0
2.13
24.0
2.71
1.5
1.40
DAC-SMA-21
6.0
2.46
24.0
3.06
ST-S-16
4.0
1.0
2.92
1.52
Mimi-S-24
4.1
1.1
2.82
2.12
DWav1-8-16
3.0
1.0
2.33
2.16
SQ-SMA-16
WT-SMA-21
0.98
2.39
WT-8-24
0.52
3.12
B Computational Setup
Table 18 shows computaional setting for our experiments. Approximate runtimes are provided per run and may vary slightly between tokenizers depending token rate, number of token streams, and other factors. For nownstreaa lasks time is reported per run, runtimes range from 2 hours (e.g., Keyword Spotting) to 48 hours (e.g., ASR), depending on the task complexity.
Table 18: Computational settings for our experiments.
Experimental Name
Computational Resource
Approx. Time
Downstream Evaluation
1xA100 (80GB)
2-48 hrs
Reconstructed Audio Quality
1xA6000 (48G)
hrs
Speech Language Modeling
2xA100 (40GB)
48 hrs
Texl-Lo-Speech
AxA100 (80GB)
96 hrs
Audio Generation
2xA100 (80GB)
48 hrs
Music Generation
4xA100 (40GB)
48 hrs
A Ablation Studies
2xGH200 (100GB)
48 hrs
C
Dataset
For cach experiment, we report the datasot, source, and approximate duration of the corresponding dataset.. A detailed summary provided in Table 19 For more information, please refer to the 'Dataset" subsection in each benchmark section.
53

---

Table 19: Summary datasets, their specifications, and approximate number of hours used for experi- ments this survey.
Task
Dataset
Hours
Data Link
Reconstruction
Speech
LibriSpeech (oovva et al.
2014
Link
Music
MUSDB (Rafii et al. 2017
Link
Audio
Audioset Gemmeeke et al. 2017
56
Link
Downstream
ASR (En)
LibriSpeech [orvvs et al.
2014
1,000
Link
ASR (Welsh)
CommonVoice 17.0 Ardila al.
2020
Link
ASR (Basque)
CommonVoice 17.0
Ardila al.
2020
116
Link
Speaker ID / Verification
VoxCeleb1 Nagrani et al.
2017
350
Link
Emotion Recognition
IEMOCAP Busso et al.
2008
Link
Keyword Spotting
Speech Commands Warden
2018
18
Link
Intent Classification
SLURP Bastianelli al.
2020
10
Link
Speech Enhancement
VoiceBank (Valentini-ooiinn Botinhao al.
2016
10
Link
Speech Separation
Libri2Mix (Cosentino al.
2020
400
Link
Music Genre Classification
GTZAN Tzanetaais & Cook
2002
Link
Music Source Separation
MUSDB (Rafii et al.
2017
10
Link
Sound Event Classification
ESC-50 Piczak 2015
2
Link
Audio Separation
FUSS
Wisdom et al.
2021
23
Link
Acoustic LM
Speech Language Modeling
LibriHeavy
Kang et al.
2024
56,000
Link
Text-to-Speech
LibriTTS
Zen et al. 2019
960
Link
Audio Generation
Data Mix (see Sec. 3.3.3 for details)
4050
Music Generation
FMA
Defferrard et al.
2017
702
Link
Ablation
Speech
Data Mix (see Sec.
for details)
1,000
Music
Data Mix (see Sec. 4 for details)
1,000
Audio
Data Mix (see Sec.
4 for details)
1,000
54


Reviews:
[{"review_id": "sVwJqEUrY8", "reviewer": "Reviewer_YDdA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work presents a clear and well-defined taxonomy for classifying state-of-the-art discrete audio tokenizers, offering the community a structured framework for comparing and analyzing their different components.\n- This work thoroughly benchmarks a range of tokenizers through extensive experiments. The benchmarking tasks are organized into three categories, each reflecting a distinct aspect of model capacity and performance. Task descriptions, evaluation datasets, setups, and metrics are clearly detailed, and the results yield valuable insights into the strengths and limitations of different approaches.\n- The investigation covers a wide range of application domains—including speech, music, and general audio—and also illustrates the domain adaptation and generalization capabilities of different tokenizers.\n- The paper is clearly written and well-organized, with figures and tables effectively illustrating the content.\n\nOverall, this is a high-quality piece of work that presents a thorough investigation of current discrete audio tokenizers, offering a comprehensive empirical understanding of state-of-the-art approaches. I am confident that this work will contribute significantly to advancing research and development in the field.\n\nWeakness:\n- This work lack a discussion of the training datasets and data volume, which are likely key factors influencing the model’s performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive review and benchmark of discrete audio tokenizers.\n\n- It introduces a fine-grained taxonomy to categorize tokenizers across multiple dimensions, including encoder-decoder architecture, quantization method, training paradigm, streamability, and application domains.\n\n- It conducts thorough benchmarking from three key perspectives: reconstruction quality, downstream task performance, and acoustic modeling/generation capabilities.\n\n- This study further investigates the impact of factors such as training data domain, sampling rate, semantic distillation, and quantization techniques on tokenizer performance through ablation studies by isolating relevant components.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Below are some of my detailed comments and questions.\n- In Section 2.4.2, for the Vector Quantization section, when presenting the training objective of vector quantization, the authors could add reference for the VQ-VAE work [1].\n- In Section 3.1, General Audio and Music section, it writes “SQ-Codec, the only fully out-of-domain model (trained only on speech data), gives the poorest performance in SDR and SI-SNR metrics.” However, as illustrated in Table 5, it seems that the WT-S-24 model is the out-of-domain model and achieves the poorest performance in SDR and SI-SNR metrics.\n- In Section 3.2, Results and Discussion for Audio and Music Tasks, the paper states: 'Even the best-performing model (EnCodec at medium bitrate) only reaches about -7 dB SI-SDR for audio and -5.7 dB for music.' However, Table 9 reports that EnCodec at medium bitrate (#Q=8) achieves 9.53 dB SI-SDR for audio and 1.98 dB for music. Could the authors clarify whether I have misunderstood this, or if there is a discrepancy between the statement and the table?\n- In Section 3.2, Summary, the authors discuss the performance of semantic and acoustic tokenizers. However, the paper does not clearly indicate which benchmarked methods fall under each category. It would be helpful if the authors could clarify this distinction.\n- In Section 5, the authors discuss semantic distillation beyond speech. However, it is unclear to me how 'semantic' is defined in the context of music and general audio. While this may be an open question, I would appreciate hearing the authors' perspective on this.\n\n[1]Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" In Advances in neural information processing systems, 2017.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6nKXdsKhcv", "reviewer": "Reviewer_ktZN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Proposes a systematic and detailed taxonomy, clearly categorizing audio tokenizers by key dimensions, improving conceptual clarity.\n- Provides extensive benchmarking across speech, music, and general audio, including reconstruction quality, downstream tasks, and acoustic language modeling.\n- Conducts rigorous ablation studies, effectively highlighting trade-offs regarding quantization methods, sampling rates, domain-specific training, and semantic distillation.\n- Presents structured visualizations (tables, radar charts), enabling effective comparison of performance.\n- Identifies practical trade-offs between reconstruction accuracy and downstream task performance, providing insights valuable for tokenizer design.\n- Covers comprehensive downstream applications (discriminative and generative tasks) showcasing broad applicability.\n- Evaluates acoustic language modeling tasks, extending analysis beyond traditional benchmarks.\n- Highlights practical challenges related to cross-domain generalization, offering useful insights for universal audio model development.\n- Provides meaningful future research directions based on comprehensive analyses of current tokenizer limitations and trade-offs.\n\n**Weaknesses**\n\n- Although addressing limitations of the traditional semantic-acoustic dichotomy, the newly proposed taxonomy does not fully replace it; semantic and acoustic token distinctions remain practically significant.\n- Relies predominantly on objective metrics, lacking subjective perceptual assessments (e.g., MOS for text-to-speech tasks), limiting comprehensive evaluation of perceptual audio quality.\n- Some benchmarking depends on publicly available pretrained checkpoints rather than uniformly trained models, potentially introducing biases or inconsistencies.\n- Relatively insufficient discussion on computational complexity, inference speed, and latency, important aspects for practical applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a comprehensive analysis and benchmark for discrete audio tokenizers, which convert audio signals into discrete, compact representations useful for efficient audio processing and multimodal integration. The contributions of the paper include a taxonomy that classifies audio tokenizers by encoder-decoder architecture, quantization techniques, training methods, streamability, and application domains; an extensive benchmark evaluation across multiple audio domains assessing reconstruction quality, downstream task effectiveness, and acoustic language modeling performance; and ablation studies systematically examining how different factors such as quantization methods, sampling rates, and domain-specific training affect performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Instead of fully replacing the semantic vs. acoustic distinction, consider incorporating it into a broader, hierarchical taxonomy to ensure comprehensive coverage and practical utility.\n- Include subjective perceptual evaluation (e.g., MOS for text-to-speech tasks) or clearly justify its omission.\n- Clarify explicitly the sources and nature of checkpoints and evaluation tools used in benchmarking.\n- Explore and suggest additional evaluation metrics beyond FAD and CLAP for a more robust quality assessment.", "broader_impact_concerns": "As a reviewer, I have no significant ethical concerns regarding the broader impact of this work. The paper focuses on the taxonomy, evaluation, and analysis of discrete audio tokenizers, which are foundational tools in audio processing. The authors do not propose new generative models or applications that could directly lead to misuse, such as deepfakes or surveillance.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3lAPu5gEmJ", "reviewer": "Reviewer_ZrAE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- The proposed taxonomy for audio tokenization techniques is more intuitive and understandable than the existing \"semantic vs. acoustic\" classification.\n- Thorough experimental investigation through a series of various benchmarks and ablation studies provides insights for further development of audio language modeling techniques.\n\n[Weaknesses]\n- This paper requires the reader to have advanced knowledge of recent techniques proposed in audio/speech processing, such as self-supervised learning (SSL) for speech/audio signals.\n- Experimental setups in benchmarks and ablation studies are not fully justified.\n- Although the quantization techniques are important factors that affect the performances of the neural audio codec models, their mathematical formulations are not always provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contributions by this paper are threefold.\n\n1. This paper presented a systematic review and benchmark of audio tokenizers for speech, music, and general audio signals.\n\n2. This paper proposed a taxonomy of tokenization approaches based on the DNN architectures, quantization techniques, training paradigm, streamability, and application domains.\n\n3. This paper evaluated audio tokenizers on multiple benchmarks for reconstruction, downstream performance, and audio language modeling. It also analyzed trade-offs through ablation studies.\n\nIn summary, the findings by this paper highlight key limitations, practical considerations, and open challenges that provide insight and guidance for future research in audio tokenizers and their applications in audio language modeling. I really appreciate the authors' effort to conduct such large-scale experimental investigations. I believe that this paper should be valuable for research community in not only speech/audio but also machine learning research, with some revisions to improve the presentation and reproducibility.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "[Major comments]\n\n[C1] As this is the journal named as Transactions on \"Machine Learning Research,\" it would be better to include discussion of compared codec models and evaluation results from the viewpoints of machine learning. For example:\n- [C1-1] The codebook collapse is well-known issue in VQVAE-based generative models. Does this issue also occur in the context of neural audio codecs and affect the quality of generated samples?\n- [C1-2] Please add explanation with mathematical formulations in some parts, such as \"Masked Prediction (MP)\" in Section 2.4.2 and \"Disentanglement\" in Section 2.4.3.\n- [C1-3] In Section 2.4.2 \"Adversarial (GAN),\" please explain what the \"perceptual losses\" are, with proper mathematical formulation.\n\n[C2] The following terms should be explained with proper definition, as not all readers are familiar with these technologies:\n- \"acoustic language model(s)\"\n- \"self-supervised learning (SSL) model(s)\" for speech/audio data\n\n[C3] Please clarify the following points regarding the experimental setups and evaluation results:\n- [C3-1] Why did the authors choose the tokenizers listed in Tables 2 and 14 for their experiments from various ones listed in Table 1?\n- [C3-2] Why did the authors employ multiple models to estimate the perceptual quality of generated samples?\n  - As the UTMOS was developed in 2022, it might not be suitable for evaluating the quality of synthetic speech produced by recent neural speech synthesis models like VALL-E. How about using a newer MOS prediction model, such as UTMOSv2[a]?\n- [C3-3] For all experiments please describe the computational resources, i.e., the number of GPUs and their device names.\n\n[a] K. Baba et al., \"The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech,\" in Proc. SLT 2024.\n\n[C4] The following claims should be explained in more detail:\n- [C4-1] In Section 3.4: \"For real-world applications, we encourage referring to the full benchmark tables and task-specific analyses to identify the most appropriate tokenizer for the target use case.\" -> I understand that, but actual speech/audio processing tasks, such as zero-shot TTS, prompt-instructed TTS, and audio style transfers, are really complicated. Therefore, I doubt that the benchmark results presented in this paper are really useful in selecting a suitable neural speech/audio codec model for such complicated tasks.\n- [C4-2] In Section 5: \"While discrete representations may currently underperform continuous features on certain discriminative tasks, particularly in low-resource or semantically fine-grained scenarios, recent advances show that they can match or even outperform continuous representations in some applications, such as text-to-speech, especially when trained on large-scale data. \" -> How much data can be considered “large-scale”?\n\n[Minor comments]\n\nPlease correct the following typos and errors:\n- In Section 2.2.1 (SVQ): \"t has gained popularity due to ...\" -> should be \"It has gained ...\" ?\n- In Section 2.4.1: \"... such a SSL model ...\" -> \"... such an SSL model ...\"\n- In Section 3.3.2 (Evaluation Setup): \"To evaluate the vocal quality of synthesized speech, we use a pretrained UTMOS model (Takaaki et al., 2022).\" \n  - The term \"vocal quality\" sounds unclear. Please rewrite it as \"speech naturalness.\"\n  - Takaaki is the author's first name. Please modify it as \"Saeki et al., 2022.\"\n- Please use \"$k$-means\" throughout the paper.\n\nIt would be helpful for readers if the authors provide samples of generated speech, audio, and music in their personal web page or supplementary material.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)