DASB - Discrete Audio and Speech Benchmark
Authors: Pooneh Mousavi
Jarod Duret
Darius Petermann
Artem Ploujnikov
Luca Della Libera
Anastasia Kuznetsova
Cem Subakan
Mirco Ravanelli
Venue: Accepted by TMLR ()

Abstract: Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling multimodal language models that can both generate and understand audio.  However, preserving key information such as phonetic content, speaker identity, and paralinguistic cues remains a major challenge.  
Identifying the optimal tokenizer and configuration is further complicated by inconsistent evaluation settings across existing studies.  To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on a range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, learning rate, and capacity.   Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evaluation setup, and leaderboards are publicly available at https://poonehmousavi.github.io/DASB-website/.

Published in Transactions on Machine Learning Research (04/2026)
DASB Discrete Audio and Speech Benchmark
Pooneh Mousavi Jarod Duret", Darius Petermann", Artem Ploujnikov4., Luca Della Libera? Anastasia Kuznetsova", Cem Subakan Mirco Ravanelli
'Concordia University, "Mila-Queber AT Institute, "Avignon Université, *Université de Montréal, Universitéé Laval, "Carnegie Mellon University, "Microsoft, $Indiana University
Reviewed on OpenReview: http:://ppppeerreeea//////////tp
Abstract
Discrete audio tokens have recently gained considerable attention for their potential to bridge audio and language processing, enabling multimodal language models that can both generate and understand audio. However, preserving key information such as phonetic content, speaker identily, and paralinguistic cues remains a major challenge. Identifying the optimal tokenizer and configuration further complicated by inconsistent evaluation scttings across existing studics. To address this, we introduce the Discrete Audio and Speech Benchmark (DASB), a comprehensive framework for benchmarking discrete audio tokens across speech, general audio, and music domains on range of discriminative and generative tasks. Our results show that discrete representations are less robust than continuous ones and require careful tuning of factors such as model architecture, data size, lcarning rate, and capacity. Semantic tokens generally outperform acoustic tokens, but a gap remains between discrete tokens and continuous features, highlighting the need for further research. DASB codes, evalustion setup, and leaderboards are publicly available at https hrg.aeeenoussii.ggggtaa..aaaaaaa
1 Introduction
Traditional speech and audio processing systems have long relied on handcrafted low-level features such as Mel-Frequency Cepstral Coefficients and Filterbanks (Rabiner & Juang, 1993). Recently, self-supervised learning (SSL) led to outstanding performance improvements by learning more complex, robust, and general speech features through deep neural networks. Notable models include Wav2Vec2 (Bacvski et al., 2020), WavLM (Chen al. 2022). and HuBERT (Hsu el al. (022). In all these cases, the rich information speech and audio signals encoded into sequence of continuous vectors. Even though continuous vectors have proven effective in capturing the complex details embedded in speech and audio, there is growing interest in discrete representations. Discrete audio representations, known audio tokens, transform the original waveform into finite set of vectors. These tokens are derived using methods such as quantization of self-supervised learning (SSL) models (Polyak et al. 2021 Mousavi et al.) 20224 Wells et al. 2022: Chung let al., 2021 Shi et al. 2024a). neural acoustic techniques
(oodecs)KKuaal et al.) 2023 Défossez et al. 2023
Xin et al. 2024 Wu et al. |l.]||22200 2024c; Ji al.
2024 Yang et al. 2024a). or hybrid approaches (Zhang el al.
2024 Du al. 2023: Défossc2 al.] 2024 Ye al. 2025 which enhance codoe representations by distilling
phonetic information into specific codebooks.
What is driving the interest in audio tokens? The growing interest in discrete audio tokens is inspired by the success of autoregressive Large Language Models (LLMs) [Tovvron et al. 2023 Chowdhery et al. 2024 Devlin et al. 2019), which operate over discrete text. Motivated by this, researchers have explored representing audio as sequences of discrete tokens (Mousavi et al. 2025 van den Oord et al. 2017 Wang et al. 2023b
Kreuk et al.] (023),, enabling the development of audio language models and multimodal LLMs (Gemini Team, Google 2023 Zhan al.) 2024).. Discrete tokens simplify generation tasks by reframing them classification problems (Goodfellow al.] 2016) and enable efficient compression and storage. However,

---

Published in Transactions on Machine Learning Research (04/2026)
00000000
#codebook
#oodebook
Hybrid
Discriminative
Generative
Semantic
Acoustic
Semantic
Acoustic
Scalable HiFi-GAN
Decoder
only for
Quantizer
K-Means
Layer Dropout
Quantizer
Lincar
ReLU
Encoder
Encoder
Linear
Atlention
Attention
Discrete Audio Encoder
Downstream Model
Discrete Audio Deconlr
Figure 1: The workflow of DASB consists of three steps. First, discrete audio encoder converts the audio signal into discrete tokens (left). Then, the tokens are combined using attention and fed to a neural model for the final prediction (middle). For generative tasks, the predicted tokens are passed to a discrete decoder, which converts them back into an audio waveform (right). Both the encoder and decoder are pretrained and frozen during downstream model training.
discretization inevitably introduces information loss. As result, continuous features remain preferred for speech understanding tasks (Tang et al., 2024 Chu et al. 2023 while discrete tokens are increasingly favored for generation (Maimon et al. 2025 Arora et al. 2025 Ideally, audio tokens should preserve crucial information from the original waveform, including phonetic and linguistic content, speaker identity, emotion. and other paralinguistic cucs. However, despite the growing interest in audio tokens (Wu et al. 2024b:
Puyvada et al.j 2024 Wang et al.) 2024b Zhang et al. 2023 Chang et al. 2024b Shi et al. 2024b), there is still limited understanding of how much information lost during tokenization across different tasks, an essential factor for building multimodal models that both understand and generate audio. To address this gap, introduce the Discrete Audio and Speech Benchmark (DASB). DASB systematically assesses various audio tokens across several common audio processing tasks. Tn particular, our contribution is the following:
Direct evaluation of discrete tokens. Unlike prior benchmarks that rely on decoding tokens back waveforms, DASB directly evaluates tokenizers without decoder, isolating the information loss introduced by quantization. This allows more faithful analysis of how discrete reprosentations impact downstream tasks and multimodal modeling. Moreover, we explicitly quantify the gap between the performance obtained with discrete representations and continous representations on wide range of discrete and generative downstream tasks.
Diverse tokenizers. We benchmark a diverse set of discrete audio encoders from all three categories:
semantic (Discrete HuBERT, Discrete WavLM, Discrete Wav2Vec2) (Mousavi et al.
2024). acoustic
(EnCodoc (Défossoz et al. 2023 DAC (Kumar et al. 2023). WayTokenizer (Ji et al. 2024 SQCodec Yang (Zhang et al. 2024), and Mini (Défossez et al. 2024
et al. 2024a]), and hybrid
Comprehensive tasks and domains. We consider a wide range of tasks, including speech recognition, speaker verification, emotion recognition, keyword spotting, intent classification, sound event classification, music genre classification, speech enhancement, speech scparation, text-to-specch, and music and general sound source separation. For reliable ovaluation, we apply extensive hyperparameter tuning, test two downstream architectures per task, and average results over multiple seeds.
2

---

Published in Transactions on Machine Learning Research (04/2026)
Ablation studies and practical guidance. We systematically study the effects of mumber of codebooks, downstream head design, and initialization strategies. These ablations provide practitioners with actionable insights into which tokenizers are best suited for specific tasks and conditions, and clarify trade-offs between reconstruction fidelity and downstream utility.
Public leaderboard and reproducible results. DASB is released modular, reproducible benchmark built on SpeechBrain (Ravanelli et al. 2021). A public leaderboard is provided for community submissions, with code and documentation available at the official website and GitHub repository"
2 Related Work
Recent studies have explored the use of discrete audio tokens as an alternative to continuous features
across range of speech and audio tasks. Discrete representations have been applied to automatic speech
rocognition (Chang ct al.) 2023 Du et al. 2023
Mousavi ct al.; 2024). spoech-to-speech translation (Popuri
et al. 2022
Inaguma et al.] 2023 Wu et al.
2023a Chang et al.
2024a). voice conversion (Maimon &
Adi
2023 Wang et al. 2024a), text-fo-speech synt (Ju et al.)
2024 Wang et al.)
2023a Hayashi &
Watanabe 2020), speech enhancement (Wang et al.
2024b Yang et al.
2024b] Xue et al.
2024 and source
separation (Shi et al.) 2021 Erdogan et al.] [023] Mousavi et al.) 2024 Bie al. 2024
Yip et al.] 2024).
Beyond speech, discrete tokens have also been used for music generation (Copet et al.
2023: Chen et al.
2024) and environmental sound synthesis (Yang et al.) 2023b Kreuk et al.
2023). Discrete audio tokens can (Mousavi et al. 2025). Several benchmarks have been proposed to
be evaluated from multiple perspectives
evaluate discrete tokenization (Shi et al.
2024a; Wu et al. 2024a Wang & Székely) 2024 Mousavi et al.)
2024 Codec-SUPERB (Wu et al. 2024a) assesses audio reconstruction quality using subjective scores and pre-trained models, while ESPnet- Codec (Shi et al. 2024a and VERSA (Shi et al.) 2025) offer a toolkit for codec training and evaluation. However, these aforementioned prior work primarily relies on reconstructed audio and focuses on a single category of tokenizers, semanti or acoustic tokens.
To address the limitations of existing work, we introduce DASB, the first benchmark specifically designed to directly evaluate discrete audio tokens. Unlike prior benchmarks such as Codec-SUPER or VERSA, which decode tokens back into waveforms before evaluation, DASB evaluates tokenizers without relying on reconstruction. By removing the decoder, DASB isolates the information loss introduced by tokenization itself, providing a clearer picture of how discrete representations affect downstream tasks and multimodal language model training.Howe However, we would like to clarify that while gencrative tasks require waveform reconstruction, all modeling and evaluation are performed in the discrete token space. A pre-trained frozen decoder is used only at the final stage to convert tokens into waveforms and is not part of the downstream modeling.- This distinction is critical. Reconstruution--asse benchmarks can overestimate representation quality because a strong decoder may mask deficiencies in the tokens, especially since most training losses are applied to the decoder output. In contrast, DASB quantifies the true information loss from quantization and highlights trade-offs between reconstruction fidelity and downstream utility (Table 13 Appendix F). For example, tokenizers optimized for waveform fidclity often fail to preserve phonotic or semantic cues, which are essential for lasks where no decoder is involved (e.g,, ASR or SLU).
Beyond isolating information loss DASB provides a systematic framework to compare discrete and continuous reprosentations. This is particularly relevant for multimodal models, where the choice of representation has practical consequences. Prior work shows that models such as SALMON (Tang et al., 2024) and Qwen (Chu et al. 2023) often prefer contimuous features for tasks requiring deep speech understanding, while generative models tend to favor discrete tokens because classification- based training is easier than regression. DASB makes these trade-off's explicit by quantifying how much information discrete tokens retain and how this impacts task-level performance.
DASB also distinguishes itself through broad coverage. We evaluate a diverse set of tokenizers including semantic, acoustic, and hybrid models, across both discriminative and generalive tasks in speech, general
We will release the code and evaluation pipcline with the camera-ready version. II speech processing, the term "semantic" docs not align with its standard linguistic meaning. Semantic tokens in this context better described as phonetic units (Sicherman Adi 2023g, as they generally lack semantic content (Arora al. 2025
For consistency across speech, audio, and music domains, we adopt the term "semantic" throughout this paper.
3

---

Published in Transactions on Machine Learning Research (04/2026)
Table 1: Key Features the Discrete Audio Encoders.
Domain
Model
Abr.
SR
FR
# vocab
Quantization Method
Speech
Music
Sound
EnCodee Défosses et. al.
2023
Enc SMA-24
24KHz
75
1024
acoustic IVQ
DAC Kumar e6 a.. 2123
DAC-SMA-24
24KH*
75
1024
aconstic RVQ
Sporth Iooeriier
bangg
2124
ST-S-16
16KHz
50
hybrid RVQ
Mini Défoeww
21p4
Mimi-8-24
24KHx
12.5
2048
hybrid IVQ
Discrese WayLM
Mousavi et al
2024
DWavT-S- 16
16KH*
50
1000
semantic kmoans
Discrete HuBERT
Mousavi el 01
DHaBERT-S-10
16KHz
50
1000
semantic kmaans
Discrete Way2Ver2
Kintsay] el
DWav2Vor2-S.
16KHx
kmeans
SQ-
Yang
2021a
SQ- 10-1
16KHz
30
20000
acoustic - FSQ
Way TuwTuukeue
Ji cl ml.
2124
WT-SMA-24-2
24KHz
75
4096
acoustic SVQ
audio, and music. Prior benchmarks typically focus on narrow domain or a single type of tokenizer. In contrast, DASB adopts the design philosophy of continuous benchmarks such SUPERB (wen Yang et al. 2021h, HEAR (Turian et al. 2022,, and SpeechBench (Zaiem et al. 2023), but adapts it to the unique challenges of discrete representations, which are far more sensitive to hyperparameters and architectural design choices. To ensure reliable comparisons, we perform extensive hyperparameter tuning, multi-sced evaluations, and analyze factors such as codebook size and downstream head design.
Finally, DASB provides practical guidance for researchers and practitioners. Our analyses highlight which tokenizers are most. effective for different tasks, under what conditions they succeed fail, and how design choices such as the number of codebooks or initialization strategies influence performance. Therefore, DASB complements prior works and provides practical guidance on tokenizer design and task suitability, filling crucial gaps in evaluation methodology and broadening our understanding of discrete tokenization (Mousavi et al. 2025
3
Benchmark Design
The pipeline of DASB, illustrated in Figure [ consists of three components: Audio Encoder, Downstream Model, and Audio Decoder. The following subsections describe each module detail.
3.1
Tokenizer Selection Criteria
The main features of the considered tokenizers are summarized in Table [ while Figure reports their time and memory requirements for both encoders and decoders. Following the terminology from (Borsos et al.) 2023 Zhang et al. 2024 Guo et al.) 2025), we categorize audio tokens into three classes: semantic, acoustic, and brid. To ensure broad coverage, we select tokenizers that reflect diverse desigu choices within these categories. When available, we prioritize models trained on multiple domains (speech, music, and general audio) to maintain consistency across tasks.
generated by clustering quantizing layers from SST. models (Baevski et al. 2020 Chen et al. 2022 Hsu et al. 2021). The tokenization process typically involves selecting specific layers from pretrained SSL model and applying the k-means algorithm to group their representations. Semantic tokens primarily capture high-level information, such as phonetic, semantic, and syntactic information. They are not optimized for waveform reconstruction, making them potentially better suited for discriminative tasks like speech recognition. Recent studios, however, have shown that semantic tokens can also be effective in generative tasks (Wang al. 2024b Yang et al., 2024cc Mousavi et al. 2024). We adopt the tokenization algorithm proposed in (Mousavi et al. 2024h. In particular, we consider three widely-used open-source SSL models: Wav2Vec2-large, WavLM-large, and HuBERT-large, each composed of 24 layers. Then, we cluster six of these layers using the k-means algorithm and select two layers from the lower part (1, 3) to capture low-level information, two from the middle layers (7, 12), and two from the higher layers (18, 23) to encode content and meaning as well.
Semantic tokens (Polyak et al. 2021 Wells et al. 2022 Chung et al. 2021
Acoustic tokens [Zeghdoorr al. 2021 Défossez al. 2023 Kumar et al. 2023 are primarily designed for audio compression and are trained to accurately reconstruuc the original waveform, making them suitable
4

---

Published in Transactions on Machine Learning Research (04/2026)
for generation tasks. Our benchmark includes four acoustic tokenizers reflecting different design choices. EnCodec (Défossez et al. 2023) uses a 1D convolutional encoder, a two-layer LSTM, and decoder, with Residual Vector Quantization (RVQ) (Zeghidour et al. 2021) applied to compress the latent space. It is trained end-to-end with a combination of reconstruction and perceptual losses and supports multiple bitrates from 1.5 to 24 kbps. DAC (Kumar al. 2023 improves upon EnCodec by adopting advanced vector quantization techniques from the image domain, enhanced adversarial training, and quantizer dropout to enable flexible bitrate control. Beyond RVQ-based models, WarTokenizer replaces RVQ with single vector quantization (SVQ), expanding the vocabulary size to simplify the model architecture and avoid the complexity of managing multiple codebooks. Finally, SQ-Codec (Mentzer et al. 2023 Yang et al. 2024a) adopts finite scalar quantization (FSQ), mapping each feature dimension independently to a fixed set of scalar values. This approach improves codebook utilization,, prevents codebook collapse, and creates a scalar latent space rather than relying on traditional vector quantization.
Hybrid tokenizers (Zhang et al., 2024 Du al., 2023) combine semantic and acoustic information by disentangling different aspects of speech hierarchically across quantization layers. SpeechTokenizer (Zhang) et al. 2024) and Mimi (Défossez et al. 2024) adopt this approach by introducing semantic distillation into the residual vector quantization (RVQ) process. Both models build on RVQ-GAN architectures, where a semantic teacher network supervises the first RVQ quantizer to encourage early tokens to capture phonetic and linguistic content. Subsequent RVQ layers capture residual paralinguistic features such as speaker identity and prosody, while preserving low-level acoustic details necessary for reconstruction. In SpeechTokenizer HuBERT is used as the semantic teacher, whereas Mimi uses WavLM. The training objective includes maximizing the cosine similarity between the outputs of the first RVQ layer and the semantic teacher representations.
3.2 Discrete Audio Encoder
The audio encoder converts the audio signal into a sequence of discrete tokens. is pretrained on large amounts of unlabeled data and remains frozen during the training of downstream tasks. Different encoders may compress the information in the original waveform at different rates. The compression level is measured by the bitrate, defined as:
(1) where C' is the number of codebooks, V is the number of vectors in each codebook (vocabulary), and the frame rate of codes per second. Tt is worth mentioning that a single sequence of takens might. be insufficient. to capture the rich and complex information embedded in speech signals. The encoders thus often output multiple discrete sequences, with cach sequence corresponding to a different codebook C. The encoders can operate at different. bitrates simply by adjusting the number of codebooks C. However, this introduces a trade-off between efficiency and performance. To ensure a fair comparison, we evaluate tokenizer performance across multiple bitrate settings: the lowest, the highest, and the configuration recommended by the original authors. If no alternatives are provided, we use the default configuration. We consider this approach prevent the trivial conclusion that some audio tokens perform better than others simply due to a higher bitrate.
bitrate logg VCC. -C.R,
3.3 Downstream Model
In this step, we employ neural networks to solve supervised tasks of common interest. We first map discrete tokens to embeddings and then dynamically combine embeddings from different codebooks using an attention mechanism. A simple multi-layer perceptron (MLP) computes attention scores across codebooks at each time step, allowing the model to adaptively weigh information based on task requirements. The MLP generates a score for each codebook, which is normalized by a softmax function, as shown below:
exp(zo.t)
Re, (emb(de.:)), de,t Dhe exp(2x.))
h = Σ an,tZe,to
(2)
where dz is the discrete token from codebook e at time t, Z is the MLP score, and emb(-) refers to the lookup table mapping tokens to embeddings. The attention weight ae determines the contribution of each codebook, and hy is the final combined representation passed to the downstream model. All embeddings are
5

---

Published in Transactions on Machine Learning Research (04/2026)
Coooc
DAC
Speechllobenize
Discrete WAVLM
Discrete HUBERT
Discrete Wer2Wec?
SQ-Cadec
Wwvibkerizerr
Encoder
Encoder
Decoder
Decoder
0.225
0.11
1.4
0.200
1.2
0.12
Time
0.323
Memoryy
1ime
0.08
Memory
0.8
0.100
0.075
0.050
0.4
0.025
16
16
16
Bitrate (kbps)
Bitrate (kbps)
Bitrate (kbp2)
Bitrate (kbps)
Figure 2: Time and memory required to process a 16-second utterance for the encoders and decoders of the considered audio tokenizers, measured on an NVIDIA Quadro RTX 8000 GPU with 128 GB RAM. The number of points per tokenizer depends on the available bitrate configurations.
randomly initialized with dimension of 1024 to ensure consistency. We study the impact of embedding initialization in Appendix D For SQ-Codec, which uses scalar quantization and Group VQ, we replace learnable embeddings with a
approsch. The final 1024-dimensional embedding is constructed by concatenating four separate 256-dimensional group emboddings to efficiently handle its large vocabulary (-20k tokens). The combined representations h2 are fed into task-specific neural models, trained end-to-end along with their attention and embedding layers. For discriminative tasks, the model outputs either single or sequential prediction, while for generative tasks, it predicts token sequences per codebook, which are later decoded into audio.
3.4 Discrete Audio Decoder
The decoder, used for generative (asks only, converts the predicted lokens into audio signals. The decoder is frozen during training. The choice of decoder depends the encoder used in the first step. For acoustic and hybrid tokenizers, we use their built-in decoder. For semantic tokens, we use the scalable vocoder proposed in (Mousavi et al. 2024), which is a modified HiFi-GAN (Yang et al. 2023a) pretrained with LibriSpeech-960n (Krvvs et al. 2014). The scalable vocoder accepts a variable number of multi-layer semantic tokens as input. and can handle different bitrates using a layer dropout mechanism.
4 Experiments
We evaluate each tokenizer across multiple bitrate settings and tune two key hyperparameters (number of layers and learning rate) using the TPE algorithum (Watanabe) 2022), implemented through the Orion framework (Bouthillier et al., (0221.. Each configuration tuned over 20 trials and evaluated using three random seeds. To ensure fair comparison, we adopt simple downstream architectures that highlight the quality of the tokens rather than compensating with model complexity. Our objective is not to achieve state-of-the-art results on individual tasks but to provide a fair and consistent. evaluation across different. tokenizers. Each task is evaluated with both shallow and deep models to study the effect of model capacity and improve robustncss. For contimnous basclines, we use the same downstream sctup as for discrete tokens, with WayLM weighted-suum features unless otherwise specified. This design isolates the effects of discrete versus continuous representations under shared architecture. Additional experimental details are provided in Appendix G
4.1 Discriminative Tasks
Automatic Speech Recognition (ASR): We evaluate ASR on two settings: English ASR using LibriSpeech (Korvas et al.) 2014) and low-rcsource ASR on Welsh and Basque from CommonVoice 17.0 (Ardila al. (0200.. For English ASR, training uses the train-clean-961 subset, validation is performed dev-clean, and testing is conducted test-clean and hest-other subsets. For CommonVoice datasets,

---

Published in Transactions on Machine Learning Research (04/2026)
we follow the official train/valldatiinntess splits. We use two downstream architectures: (i) multi-layer BiLSTM with linear layer, and (ii) a Branchformer &Peng et al., 2022) with sequence-level cross-entropy and CTC loss. Word Error Rate (WER) is used for evaluation.
Speaker Identification/Verii /Verificatinnn (SID, SV): Speaker Identification involves classifying cach utterance by its speaker identity as multi-class classification, with the same predefined set of speakers for both training and testing. The evaluation metric is the accuracy. Automatic Speaker Verification (ASV), instead, involves training a binary classifier to determine whether the speakers in pair of utterances are the same. The evaluation metric adopted in this case is the equal error rate (EER). We use the widely-used VoxCelebl (Nagrani et al.) 2017) train and lest splits for both tasks. First, we test. the ECAPA- TDNN (Desplanques et al. 2020) architecture with AM-Softmax (Wang et al. 2018) loss for training the speaker embeddings. For verification, we use the cosine similarity between speaker representations. As a second architecture, we replace the ECAPA-TDNN with BiLSTM (followed by linear classifier).
Emotion Recognition (ER): The task involves predicting one of the four classes: happy, sad, angry, and neatral We use the popular IEMOCAP (Busso et al. 2008) dataset, which contains about 10k samples from 10 speakers. As a first architecture, we use ECAPA-TDNN, For the second downstream architecture, we directly input the representations into a linear classification layer after averaging them along the time axis. The evaluation metric is the accuracy.
Intent Classification (IC): This lask aims to determine the intention or purpose given utterance a speech recording. In particular, we here aim to classify each utterance into one of 18 scenarios, including calendar, email, and alarm. For this task, we utilize the SLURP datasct (Bastianclli et al. 2020), which comprises around 72k audio recordings of single-turn user interactions with a home assistant. We employ ECAPA-TDNN and a BiLSTM (followed by a linear classifier) as downstream architectures. We evaluate the performance using the accuracy.
Keyword Spotting (KS): Keyword Spotting involves detecting predefined keywords by classifying utterances into a set of specified words. We use the Speech Commands dataset v1.0 (Warden) 2018) for this task. The dataset includes ten classes of keywords, a class for silence, and an unknown class to account for false positives. We employ both the X-vector and ECAPA-TDNN architectures. The evaluation metric is the accuracy.
Event Sound Detection (ES): This task involves classifying audio clips into 50 different sound event categories, using the ESC-50 dataset (Piczak 2015), which contains 2,000 labeled 5-second recordings across 50 sound classes (40 samples per class). We employed two downstream architectures: linear and ECAPA- TDNN, with accuracy as the evaluation metric. For a continuous baseline, we utilized the pre-trained Beats model
al. 2023).
Music Genre Classification (MG): This task focuses on classifying music clips into 10 genre categories using the GTZZAN dataset (Tzanetakis & Cook 2002), which contains 100 audio files per genre, each 30 seconds long. Similar to event sound detection, we used two downstream architectures: lincar and ECAPA-TDNN, with accuracy as the evaluation metric. We used the MERT (Li et al. 2023) model, which has similar architecture to wav2vec2 as a continuous baseline.
Urban Sound Classification (USC): This lask involves classifying audio clips into 10 urban sound categories using the UrbanSound8K dataset (Ooi et al., 2021). which contains 8732 labeled audio clips (4 seconds) across 10 classcs. We employed two downstream architectures: Xvector and ECAPA-TDNN, with accuracy as the evaluation metric.
Pitch Classification (PC): This task involves classifying instrumental sounds into 88 pitch classes using the NSynth dataset (Engel al. 2017), which contains a large collection of annotated musical notes. The goal is to predict the pitch of each audio sample, and performance is measured using pitch accuracy. We employed two downstream architectures: Xvcetor and ECAPA-TDNN.
7

---

Published in Transactions on Machine Learning Research (04/2026)
Music Speech Classification (MSC): This task focuses on distinguishing between music and speech using the GTZAN Music-Speech subset ( (Tzanetakis) 1999), which contains 120 audio tracks (60 per class), each 30 seconds long. We employed two downstream architectures: Xveetor and ECAPA-TDNN, with accuracy as the evaluation metric.
4.2 Generative Tasks
Speech Enhancement (SE): Speech enhancement aims to improve audio quality by cleaning up noisy imput recordings. For this task, we utilize the popular VoiceBank dataset (Valentini- We employ two downstream architectures: a non-autoregresssive Conformer encoder (Gulati et al. 2020
al. 2016
and a convolutional recurrent deep neural nerk (CRDNN). The input Lokens are extracted from the noisy signal, while target tokens from the clean one. Training is performed using the cross-entropy loss. The speech quality is assessed using the deep noise suppression mean opinion score (DNSMOS) (Reddy et al. 2022). The intelligibility is evaluated through the differential word error rate (dWER) (Wang et al. 2021). which measures the WER between the transcribed enhanced signal and the transcribed target signal. The transcriptions are obtained using the small version of Whisper (Radford et al. 2022p. Additionally, measure speaker fidelity, we use the cosine similarity (SpkSim) between X-vectors extracted from the enhanced signal and the target signal using the base variant of WavLM (Chen et al. speaker verification.
2022) fine-tuned for
Speech Separation (SS): SS aim at isolating individual voices in multi-speaker signals using Libri2Mix (Cosentino et al., 2020). We employ a Conformer and CRDNN, trained using permutation- invariant loss (Kolback et al. 2017). The same evaluation metrics as SE are used. We adopt the same architecture as in the discrete experiments for the continuous baseline, using a weighted sum of WavLM as input. While models such as Conv-TasNet (Luo & Mesgarani 2019) or Transformer-baa let al. 2024) are well-established baselines, we do not include them here as Libri2Mix has become a largely saturated benchmark, with many approaches already achieving near-ceiling performance. In this context, adding stronger separation backbones would not yield meaningful insighls. Instead, our focus is on isolating the effects of discrete versus continuous representations under a shared architecture. For more challenging domains like general audio and music, where scparation remains an open problem, we do include SOTA architectures to establish stronger baselines.
ones (Saijo
Music Separation: Music source separation (MSS) aims at isolating individual instrumental components from a musical mixture. Typically, sources are categorized as "Bass", "Drums", "Vocals" and "Others" with the latter encompassing any remaining elements. Similar to our speoch separation task, we employ a Conformer-based and CRDNN model trained the MUSDB dataset (Rafi et al. 2017a). We evaluate its performance using the BSSEval toolkit (Vincent et al. 2006), which provides metrics such signal- to-distortion ratio (SDR), signal-to-artiface ratio (SAR), and signal-to-interfernnee ratio (SIR). As a
continuous baseline, we use the music source separation model DEMUCS (Rouard et al. 2023 which achieves state-of-the-art results in music source separation.
Sound Separation: For general audio source scparation, we employ a Conformer-based and CRDNN model trained on the FUSS dataset (Wisdom et al.] 2021). Unlike speech and music separation, where sources are well-defined (e.g., vocals, instruments, or speakers), FUSS consists of arbitrary sound mixtures, making the separation task more challenging. We evaluate performance using SDR as well. We employ the TDCN+ masking network as our continuous baseline system (Kavalerov al. 2019).
Text-to-Speech (TTS): We evaluate both single-speaker and multi-speaker TTS tasks. For single- speaker TTS, we use simple autoregressive encoder-decoder Transformer (Vaswani et al. 2017 trained on L.Speecch [Tto] 2017) with raw text inputs. For multi-speaker TTS, we adopt a simplified version of ESPNet's VALL-E (Wang el al., 2023a) trained on LibriTTS-1000 (Zen et al. 2019)) with phoneme inputs provided by LibriTTS alignments (Minixhofer) 2023 To ensure a fair comparison, we keep all training sottings, including the number of epochs and sampling strategics, consistent across tokenizers. Speech quality assessed using the pretrained UTokyo-SaruLab System (UTMOS) (Takaaki et al., 2002)). Pronunciation accuracy is measured using dWER, and speaker similarity is also reported following the
8

---

Published in Transactions on Machine Learning Research (04/2026)
Table 2: DASB results for discriminative tasks (speech) with the first downstream architecture. #Q refers to number of codebook.
ASR-En
ASR-LR
F.R.
TC
KS
S1
SV
Maakss
#Q
WER
WER
ACC1
ACC1
ACC1
ACC1
EER,
Clean
Other
Welsh
Basque
Contimmous
2.26
5.09
41.77
14.32
63.10
86.10
99.00
99.70
2.10
2
39.16::1.5 .8G
63 1.55
90. 90 32
51 000=0.98
45 15.50:44440
12 12.90000110
17.73=3.12
89. 89.81=25.46
18.33+00
Enc-SMA-24
8
16.8311 1.22
28.9842.16
64 1.950
15.36.0.57
41.7322..2
10 10.08
71.30:1169
94.26:1:399
13.54110.57
32
50.84:1:3453
26.04
07.30= 1.19
58.21=0.92
42 1.79
33 23.66=2265
89:10:33:42
91.12:1.92
10.12466.6
2
24.574.0.16
50.60:10.25 28
95.21 05.21=0.550
66.90
45. 1.520
20 00.80.00.19
67 6..27
97.8
21.80:11.00
DAC-SMA-24
16,29 +6.29
39.17+0.29
97. 00.0001
62.45 1.40
44.73+1..0
23.97-0.44
65-27+2822
87.33+ 10.98
15.86+55.20
32
98, 00.44000..34 34
10.-12:0.00 32
98 96 18
73.57=
43 20:20022.00
44 39.00:399.00 19
08 08.67=29.1
87 07.009949999
17.12 0.76
14.92411.00
1..7
T1.36 52
42.17=0.08 05
54.80.00.85
56.80 08
94.1110066
73. 13.16.10.37
24.23.10:22 29
ST-S-16
12.69::0:12
30.99 00.69
68. 36.366:0114 14
35 15.36=0..2 22
56.00:10.7
53 18.83300.00
91.11=00007
96, 18.78800.14
10.15:20044
Mimi-S-24
19.9640.16
31. 01:-0.2-.26
91. 50:00.001
50.18=8.58
51.13:1.57
53.80=0.15
92.18:20.20
79. 19.50:40.443
18.68-40.35
32
13.52+00.1
30.77+0.17
96 89-0000707
08.15-6.30
46.76+00.6
50.73-0.54 0.50
91.311+0.19
63.93+13..00 13.64
23.01 +-4.60
D-11L-S-16
6.00:20.40 10
14.3820.58
58. 9..08820.1 15
22.02=0.17
61 1.76
76 10.33:0.11 0.7
96.82:10.99 92
76.5720033 33
22.11:10.11
4.3210.23
10.73:10.42
48.94:10.36
19.06:10.33
63.20::0.85
78.73:10.11
95.89=0.50
92:31:00.0
13.47::0.22
DInDERTT-S16
5.89.10.19
14.71
50.51
22.16:20.26
55.97.10.63
17.101
62.91:0.09
86.30:1 64
23.15::0.07
4.377+00311
10:92+00.66
54.74=0.45
20.80=0.07
63:00+1.77
71 11.20=00100
18.10=1..70
06.13++0.24
22.64++0.29
DWar2Vec2-9:1 16
2
4.47=0088 31
32.87:0011
61. (1.86=00000 0900
28 68800.00 16
56, (6./3320.39
67, 97.90 0.35
98. 18.15=:1.80
76. 76.86288.4 44
21/19::0..
6.5711021
15.91100331
54.35.00.25
23.19=0.40
60.63110.90
11:10=0.10
96.2120081
89.70.10.30
14.
SQ-SMA-16
4
11.63+0.008
30:11+0.177
84.80-0.88
94.24-1.22 1.24
41 41.30+55.48
08.13-0.25
92.74+0.42
07.38-+0003
9.69-0025
WT-SMA-242
28.69+0.96
52.90=1.112
87.41-0.08
75.82-0.20
48.40+-241
15.25-0.15
50.13+2.10
85.90+-2.48
19.38++0.66
Table 3: DASB results for generative lasks (speech) with the first rownstream architecture. #Q refers to
number of codebook.
Modele) Tosks
#Q
SE dWER
55 Speech
TTS
DNSMOS
Spk Sim!
DNSSMS
DNSMOS
dWER
Spk
UTMOS
dW EWER
Spk
Reet
Sept
Simt
Sim!
Contimonns
3.40
4.02
0.93
3.68
9.97
0.94
3.10
7.14
D.83
3.15
0.01
34.55
0.64
0.56 0.00
3.19
3.13=0.00
10.32:1777
0.58000
18220.00
4.6400.2 28
0.88200.00
Rno-SMA-24
3.08
0.01
22.70
0.88
0.00
3.54
3.08-0.00
53.37+0.65
0.000.000
2
5.22+0122
0.91+0.00
32
2.78
+ 0.01
66.70 +: 6.09
0.01 0.50
3.72
2.97=0.01
92.42=0.97
0.85=00
199:0.15
23.29:0:00
0.87=0.000
0.01
0.00
3.00:100
10:119
0.85:0000
2.37:0005
8.3410.39
0.88:0.00
DAC-SMA-24
8
3.51
0.01
29.44 + 3.93
0.90 0.01
3.67
3.3+0.00
52.7742.48
0.93++.0
3.10-0.01
12.59-0448
0.88+0.00
32
2.93
0.01
30.66
0.97
0,88 0.00
3.76
2.67:00.01
92.0710000
0.8810.01
1.6010.02
21.5510.11
0.86.10.01
ST-8-16
2
8.19 0.02
29.98 1: 0.58
0.86
0.00
3:20
3.13:10.0
81.0110.63 63
0:87:00000
1.65:10.0
4,31
0.80:10:00
8
3.49 à: 0.01
21.65 à: 0.57
0.57 à: 0.00
3.72
3. 3.4320001
00.90.10.77
0:91:0.00
3.3720024
10.3911.00
0.9110.00
3.25
0.01
67.56
2.21
0.00
3.65
3:29+00.
100 30-33
0.87-0.00
2.62-0.04
9.62+0.38
0.91+0.00
Mimii-824
32
3.18 + 0.01
102.61 + 2.40
0.82 0.00
3.72
3.00=0.00
137 33.000
0.82-000
1.53=0.00
65.31=0.00
0.93+0000
DWiiL-8-16
2
3.56 0.01
25.88
1. 2.15
0,88 0.00
3.57
3.56110.0
49.5710.61
0.85..0
2.9010.011
5.3210.03
0. 0.66000000
3.57
0.01
9.48
0.33
0.00
.377
3.75-0001
30.30+0.45
0.91 0.01-0.00
3.70-0.01
3.98+0.44
0.80+6.00
DHoBERT-S-16
2
3.45 1: 0.01
59:00 d: 3.92
0.55 1: 0.00
3.68
3. 0.63=20.01
80 $0.921
0.8420000
3 3.36110.00
3.2811029
0.85:10.00
0.01
0.15
0.00
3.72
3..8-00.0
65.8+1.46 1.46
0.80=0.00
3.94=0.00
3.70++0.07
0.90+00.00
DWar2Vec2-5 10
2
3.55 + 0.0
30.14 + 0.44
0.88 + 0.00
3.61
3. 3.58==0.01
15+10.
0.82=0.00
2 2.82=0002
5.58+0.06
0.82 + 0.00
6
3.57 +: 0.01
12.93
0.56
0.80 0.00
3.72
3 3.80-00.77
70 10.00:10:27
0.87=0.04
3 3.28:0003
12.21+1.06
0.88:00.0
SQ-SMA-18
0.01
122.30
8.74
0.00
3.19::0.00
136.00:11588
0.80.00000
WT-SMA-24
1
1.33 à: 0.01
07.53 à: 10.05
0.85 0.00
3.57
3.42::0.0
118.33::4.50
0.50=000
2.86120004
4.96.10.00
0.89.10.00
Mixtare
3.43
same setup as SE. For the multi-speaker continuous baseline, we use a Tacotron2 variant. pretrained on LibriTTg3
(hen el al. 2018
5 Results
Tables 3 and Appendix D summarize the performance on discriminative and generative tasks across the two downstream architectures evaluated (see Appendix C for the ordering of the first and second heads). We observe notable variation in tokenizer performance across different tasks and domains, suggesting that the

---

Published in Transactions on Machine Learning Research (04/2026)
Table 4: DASB results for generative and discriminative tasks (music audio) with the first downstream architecture. #Q refers to number of codebook.
Models) Tasks
#Q
Audio
Music
MSC
MGC
SI-SDRit
SI-SDRII
SAR
SIR:
ACC:
ACC1
ACC1
ACO1
ACC1
Rac
Sep
Rec
Sep
Contonsonts
15.07
13.29
9.56
11.59
02.91
85.70
100.00
90.60
87.00
0.76
7.08
0.49
3.36
1.49 #: 2.04
-2.80 +
5.96 :: 1.52
34.83 0.47
00.08:22.24
100.00:0.00
69.80=1.85
70.58 # 1.70
Eac-SMA-24
0.33
1.98
-1.95
0.33
5.26 0.22
37.00
0.77
19:4811.95
97.22:3350
65.4011611
54.87
3.86
1.73
11:10
11.72
0.35
15.00
0.02
35.43
1.45
33,70112.55
94.4513.50
00.2710:02
39.67
0.48
2.37
0:17
5.52 0.98
31.03
46.53+3.02
17. 17.22-3.08
50.33+1.75
50.00
DAC-SM-221
9.33
5.62 +: 0.21
6.96
-11.77
0.1
-10.62 + 2.53
-5.52 3:68
28.60 + 0.79
46.65+3.50
91. 11.67700000
68.47=1.51
47.67
5.09
82
4.73
-4.92 0.82
8.04
-11.32
+ 0.32
-12.70 4: 0.17
-2.05
0.41
36:67 +: 6.92
40.001222226
94, 84.45.5
60.27:00.62
50,000
0.82
SQ-SMA-16
3,62
6.54 0.22
0.03
-3.62
0.57
-5.84 0.96
1.42 0.82
31.57
1,37
35. 15.47:11.06
94, 8.5-0
43, 48.57 13
42.67
0.47
WT SMA 24-2
25.05
18.72 0.08
2.65
4.52 0.34
8.32 0.07
2:65 0.11
34.50 0.82
45.87.3.17
000.001..000
46.9012.41
48,00
1.41
Mistumn
-7.71
60.01
best tokenizer depends on the specific task. However, several consistent patterns emerge, which we highlight below.
5.1 Comparison of Discrete Audio Models
Speech Tasks. Semantic tokens outperform acoustic tokens in most discriminative tasks, reflocting their ability to preserve phonetic content, as also mentioned in (Borsos al.) 2023). Hybrid tokenizers based semantic distillation rank second. Speaker recognition tasks are an exception, where DAC achieves the best results. This suggests that acoustic tokens, trained with reconstruction objectives, better capture speaker identity. This finding is consistent with (van Niekerk et al.) 2022), which shows that quantizing SST representations tends to remove speaker-specific information. A similar trend appears in generative tasks, where semantic tokens achieve the highest MOS and dWER scores, followed by hybrid tokenizers. For speaker similarity, acoustic tokenizers perform better. We also observe that, in separation lasks, reconstructed DNSMOS scores often fail to surpass the 'Unprocessed' baseline, indicating that reconstruction quality can limit downstream performance, especially for tasks requiring high-fidelity signals.
Audio and Music Tasks. For general audio and music lasks, EnCodec consistently oulperforms other tokenizers across all bitrates and domains. DAC performs worse, likely due to its emphasis on perceptual quality at the expense of signal fidelity. The difficulty of these tasks is evident from the low SI-SDR of the unprocessed mixtures, approximately -16 dB for general audio and -7.7 dB for music. Even the best-performing model (EnCodec at medium bitrate) achieves only around -7 dB SI-SDR for audio and -.77 dB for music.
Continuous vs. Discrete Representations. Across all domains and tasks, continuous representations outperform discrete tokens. Tokenization causes information loss, affecting crucial aspects such as phonetics, speaker identity, and emotion. Our findings point to a critical research direction: addressing information loss to enable the next generation of audio tokenizers suitable for training multimodal models capable of both audio understanding and generation tasks.
5.2 Impact of Tokenizer Design and Evaluation Setting
Impact of Codebook Size and Bitrate. We study the effect of bitrate and codebook size on tokenizer performance. Tables 2 g and 4 show that medium bitrate settings achieve the best results across both discriminative and generative tasks. Higher bitrates, while preserving more information, often degrade performance by increasing output dimensionality and modeling complexity. Similarly, increasing the number of codebooks improves signal reconstruction but tends to reduce downstream task performance. In RVQ-based models, earlier codebooks capture more phonetic information, while later ones often add redundancy, which may explain this trade-off. This observation is also supported by prior work. Recent studies show that higher bitrates do not always lead to belter rownstream performance. For example, Moshi (Défossez et. al., 2024) reports that lower bitrates can sometimes yield better results. Similarly, VoxtLM MMatti et al. 2024 shows that increasing the number of centroids from 200 to 1000 worsens CER (3.5 - 6.1) TTS. These findings suggest that higher bitrate can introduce redundancy and complexity that harm semantic learning, even if reconstruction improves. This highlights an important design principle for tokenizers: optimizing for
10

---

Published in Transactions on Machine Learning Research (04/2026)
reconstruction olone does not guarantee better performance on downstream tasks, Additional reconstruction results for each tokenizer are provided in Appendix
Efficiency. We also evaluate the computational efficiency of encoders and decoders, as this can impact real-world applications. Figure 2 shows the time and memory usage for each encoder-decoder pair across all bitrate settings. Semantic tokenizers typically use large, computationally heavy encoders, while their decoders, based on compact HiFi-GAN models, are much more efficient. For streaming applications (Wul et al., 2023b) where time and memory constraints are critical, Mimi and Encodee stand out as better candidate due to the efficiency of both its encoder and decoder.
Impact of Scaling. We observe that continuous representations, semantic tokenizers, and, to some extent, hybrid tokenizers maintain stable performance across different model architectures and data scales, whereas acoustic tokenizers are much more sensitive to these factors. We organize our findings into two parts: the effect of data scaling, and model scaling.
Scaling Dala. Performance consistently improves with larger datasets, particularly for acoustic vokenizers. For example, Discrete WavLM achieves 6.0% WER LibriSpeech (960h), 22.0% on Basque (116h), and 58.9% on Welsh (8h) at low bitrate with BILSTM head, illustrating a clear correlation between training data size and WER. Acoustic models such as EnCodec and DAC are particularly sensitive in low-resource settings, while continuous representations maintain strong performance even with limited data. Similar trends are observed in tasks like music genre classification and sound event classification, where discrete tokenizers degrade more sharply as training data shrinks.
Scaling Model Architecture. Increasing model capacity stabilizes training and improves performance, especially for acoustic tokenizers. In ASR tasks, switching from a BiLSTM to a Branchformer significantly reduces WER, with larger relative gains observed for acoustic models (e.g., EnCodec achieves 8% WER with Branchformer versus 16% with BiLSTM). Similarly, in intent classification, moving from a shallow linear-statistical pooling model to a deeper BiLSTM, and in TTS, replacing Tokotron with VALL-E, both lead to improvements, particularly for acoustic tokenizers that otherwise struggle with shallow architectures. When training data is limited, deeper models often fail to converge with discrete tokens, particularly for acoustic tokenizers, making it difficult to benefit from increased model capacity.
Hyperparameter Sensitivity and Stability. High-bitrate acoustic models exhibit greater training instability, with larger variance across random seeds, particularly when using shallow architectures. This instability likely stems from the optimization challenges associated with managing multiple codebooks and the informational redundancy in high-bitrate codes, which small datasets cannot effectively leverage. Deeper models, such as Branchformer for ASR and VALL-F for TTS, substantially improve stability, although large-vocabulary models like SQ-Codec (around 20k tokens) can still struggle to converge due increased prediction complexity.
Acoustic tokenizers are highly sensitive to data scale and model capacity, often failing to converge in low- resource settings. Semantic tokenizers offer better stability with limited data but still lag behind continuous representations when data is extremely searce. Careful luning and appropriale scaling of both dala and model architecture are crucial when using acoustic tokens, especially in low-resource or high-bitrate scenarios.
5.3 Ranking
We present rankings that summarize the overall performance of audio tokenizers across high-level task categorics. To compute these rankings, tokenizers are first ordered by their performance on cach metric within task (with rank N as best. and rank 1 as worst for N tokenizers), and the ranks are then averaged across all relevant tasks in each category. For tasks with multiple metrics, rankings are computed per metric and averaged to obtain the final score. Speech downstream tasks are divided into Discriminative and Generative. Discriminative tasks are further split into Content-level, which require higher-level semantic understanding (e.g,, ASR, intent classification, keyword spotting), and Acoustic-level, which rely on fine-grained acoustic cues (e.g., emotion recoguition, speaker identification, speaker verification).
The figure 3 highlights overall trends and the relative strengths and weaknesses of different tokenizers across domains. No tokenizer consistently dominates all categories, performance varies by task and domain. In
11

---

Published in Transactions on Machine Learning Research (04/2026)
Enc-SMA-24
DAC-SAA24
SQ-SMA-16
W1-SMA2422
ST-5-16
Min---24
DitexL-S-16
DHA.SERT-S-16
DWwc2Wc2---166
(AcospAk))
56 Nusic
USC
Audio
MGC
Figure 3: Radar plots of average rankings of audio tokenizers across task categories. Higher values indicate better overall performance. The right plot shows results for speech tasks, where Discrete WavLM achieves the strongest performance. The left plot shows results for music and audio tasks, where EnCodec performs best.
speech, Discrete WavLM achieves the highest overall ranking, while in music and general audio, EnCodec performs best. These rankings are intended as a high-level overview performance trends rather than strict recommendations. For practical applications, we encourage consulting the full benchmark tables and task-specific analyses to select the most suitable tokenizer. A key factor lies in the type of information preserved during quantization. For example, in tasks such ASR, where linguistic content is critical, we observe that semantic tokenizers derived from sele-supe models (e.g., HuBERT, WavLM, wav2vec 2.0) achieve the best performance, followed by approaches that incorporate SSL distillation into the codebook (e.g,, Speech Tokenizer, Mimi). In contrast, for tasks such as speaker identification, where fine-grained acoustic cues are essential, tokenizers designed to retain detailed signal information for reconstruction (e.g.. EnCodec, DAC) perform better, For general audio and music tasks, EnCodec consistently outperforms other tokenizers across bitrates and domains, while DAC tends underperform, likely due to emphasis on perceptual quality over signal fidelity. We also note that additional factors such sampling rale, training daía, and model scale influence performance. These aspects have been studied in prior work (Mousavi et al., 2025 and are beyond the scope of our study, as we focus on comparing off-the-shelf tokenizers under unified evaluation framework.
5.4 Tokenizer Category Trade-offs
We provide qualitative comparison of acoustic, semantic, and hybrid tokenizers across key evaluation dimensions, including reconstruction quality, downstream performance, and robustness in low-rosource settings (Table 5 This comparison highlights the strengths and limitations of each category, offering guidance for selecting appropriate tokenizers based on task requirements and resource constraints.
Table Qualitative comparison of tokenizer categories across key evaluation dimensions.
Tokeniser Type
Reconstruction
Downstream
Low- Resource Robustness
Model Scalability
Convergence Speed
efficiency
Acoustic
Hlgh
Low
Requires large models
Slow
High
Somantic
Low
Can work with smaller models Depends on data
Fost
Low
Hybrid
Moderatle
Modecate
Moderate
Modecate
12

---

Published in Transactions on Machine Learning Research (04/2026)
6 Conclusion
We introduced DASB, a benchmark for evaluating discrete audio tokens across range of tasks and do- mains. Our results show that semantic tokens generally outperform acoustic tokens in both generative and discriminative tasks. Scaling data and model capacity helps discrete representations narrow the gap with contimuous self-supervised features. However, in low-rcsource scttings, scaling is not feasible, and discrete lokens, especially acoustic ones, struggle to match the robustness of continuous representations. This highlights the need for future work on improving tokenization methods better preserve phonetic, semantic, and paralinguistic information for integration into multimodal language models.
7
Acknowleegements
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Digital Research Alliance of Canada (alliancecan.ca). We thank NVIDIA for donating part of the GPUs needed for this work through the NVIDIA Academic Grant Program.
References
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common Voice: A muassvely-multilingeee speech corpus. In Language Resources and Evaluation Conference (LREC), pp. 4218 4222, 2020.
Siddhant Arora, Kai- Wei Chang, Chung- Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yt Yi Lee, Karen Livescu, and Shinji Watanabe. On the landscape of spoken language models: A comprehensive survey. arXie preprinl arXiv:2504 Xii::504008528,, 08528, 2025.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In International Conference on Neural Information Processing Systems (NewrIPS), volume 33, pp. 12449 12460, 2020.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: spoken language
understanding resource package. In Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7252 2262, 2020.
Xiaoyu Bie, Xubo Liu, and Gaël Richard. Learning source disentanglement in neural audio codec. arXiv preprint Xrx:::2409. 1228, 2024.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The MaG-Jamendo dataset for automatic music tagging. In International Conference on Machine Learning (ICML), 2019.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. AudioLM: A language modeling approach to audio generation. IEEE/ACM ACM Transactions on Audio, Speech, and Language Processing, 31:2523 2533, 2023.
X Bouthillier, C Tsirigotis, F Corneau- D Suhubdy, Askari, al. Epistimio/oriin Asynchronous distributed hyperparameter optimization, 2021.
Corneau-Tremblay, Tremblay, Schweizer, I Dong, P Delaunay, F Normandin, M Bronzi,
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation,
2008
Kai-Wei Chang, Haibin Wu, Yu-Kai Wang, Yuan-Kuci Wu, Hua Shen, Wei-Cheng Tseng, Tu-thing Kang, Shang-Wer Wen Li, and Hung-yi Lee. Speechprompt: Prompting speech language models for speech processing tasks. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2024a.
13

---

Published in Transactions on Machine Learning Research (04/2026)
Xuankai Chang, Brian Yan, Yuya Fujita, Takashi Mackaku, and Shinji Watanabe. Exploration of efficient end-to-end ASR using discretized input from self-supervised learning. In Interspeech, pp. 1399-14944 14000,113,,-- 2023.
Xuankai Chang, Brian Yan, Kwanghee Choi, Jee- Weon Jung, Yichen Lu, Soumi Maiti, Roshan Sharma, Jiatong Shi, Jinchuan Tian, Shinji Watanabe, Yuya Fujita, et al. Exploring speech recognition, translation, and understanding with discrete speech units: A comparative study. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11481-11485, 2024b.
Ke Chen, Yusong Wu, Haohe Liu, Marianna Nezhurina, Taylor Kirk-tirkp,,rick, and Shlomo Dubnov. Musicldm: Enhancing novelty in text-to-music generation using beat-synchronous mixup strategies. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1206-1210. TEFE, 2024.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujic Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEFE Journal of Selected Topics in Signal Processing, 16(6):1505-1D5 2022.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pro-training with acoustic tokenizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 5178 5193. PMLR, 23 29 Jul 2023. URL htttps:/pproceeddngss mlr. www..presssvee Prress/v202/ooo 2rreass/202/chen23ag chen23ag html
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: scaling language modeling with pathways. Journal of Machine Learning Research, 24, 2024.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijic Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXic preprint arXte:2311.07911 2023.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Chirn Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. w2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training, In IEEE Aulomatic Speech Recognition and Understanding Workshop (ASRU), pp. 244 250, 2021.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. LibriMix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005.1 11005.1110000 11262, 2020.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. Transactions on Machine Learning Research (TMLR), 2023. ISSN 2835 8856.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dialogue. arXio preprint 2:Xi::241000005 2024.
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification. In Interspeech, pp. 1830-3834 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (ods.), Conference of the North Americon Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies, Volume 1 (Long and Short Papers), pp.
2019.
14

---

Published in Transactions on Machine Learning Research (04/2026)
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. FunCodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. arXiv preprint (xXi::230000405,, 2023.
Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mehdi Zohourian, Min Tang, Mehrsa Golestaneh, al. ICASSP 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In International conference on machine learning, pp. 1068-1077 PMTR, 2017.
Hakan Erdogan, Scott Wisdom, Xuankai Chang, Zalán Borsos, Marco Tagliasacchi, Neil Zeghidour, and John R Hershey. TokenSplit: Using discrete speech representations for direct, refined, and transcript- conditioned speech separation and recognition. In Interspeech, pp. 3462 3466, 2023.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSD50K: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 329-852, 2021.
Gemini Team, Google. Gemini: A family of highly capable multimodal models. ar preprint arXiv:2312.11800 2023.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, 2017.
lan J. Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Anmol Gulati, James Qin, Chung-Cheme Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: involutioonangmeeneen transformer for speech recognition. In Interspeech, pp. 5036-5040, 2020.
Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review. arXiv preprint 2XXi::502.06499, 2025.
Tomoki Hayashi and Shinji Watanabe. Discretalk: Text-lo-speech as machine translation problem. arXiv preprint Xrx::20005. 05525, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451 3460, 2021.
Hirofumi Inaguma, Sravya Popuri, Ilia Kulikov, Peng-Jen Chen, Changhan Wang, Yu-An Chung, Yun Tang, Ann Lee, Shinji Watanabe, and Juan Pino. UnitY: Two-pass direct speech-to-speech translation with discrete units. In Anna Rogers, Jordan Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15655-15680, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653 v1/2022.222/////// acl- acl--.ng..Ss. long.872. URL
and Naoaki Okazaki (eds.), Proceedings of the 61st
org. acl- 872/
Keith Ito. The I.J speech dataset.
wwwmm.Jooaao Www--SSecc---aassseeeee 2017.
Shengpeng Ji, Zi Jiang, Wen Wang, Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: efficient acoustic discrete codec tokenizer for audio language modeling. Xiv preprint arXiv:2408. 16532, 2024.
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. Xiv preprint arXim:2403.0311 0310000 2024.
15

---

Published in Transactions on Machine Learning Research (04/2026)
Nya Kavalerov, Scott Wisdom, Hakan Erdogan, Brian Patton, Kevin W. Wilson, Jonathan Le Roux, and John R. Hershey. Universal sound separation. 2019 TEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 175 109, 2019. URL http//ppp..emmaiiiscchooooarr org/Corpusiin
148571507
M. Kolback, D. Yu, Z.-H. Tan, and J. Jensen. Multitalker speech scparation with utteranco-level permutation invariant training of deep recurrent neural networks. IEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 25:1901-1913, 2017. URL
Matěj Korvas, Ondřej Plátek, Ondřej Dušek, Lukáš Zilka, and Filip Jurěíček. Free English and Czech telephone speech corpus shared under the CC-BY-SA 3.0 license. In International Conference Language Resources and Evaluation (LREC), pp. 4423 4428, 2014.
Felix Krcuk, Gabricl Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: Textually guided audio generation. In International Conference on Learning Representations (TCLR); 2023.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved RVQGAN. In International Conference on Neural Information Processing Systems (NeurIPS), 2023.
Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, et al. Mert: Acoustic music understanding model with large-scale self-supervised training. aix preprint arXin:2306. 00107, 2023.
Yi Luo and Nima Mesgarani. Conv-lasnet: Surpassing ideal time freeuenrqucc magnilude masking for speech separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8):1256 1266, 2019.
Gallil Maimon and Yossi Adi. Speaking style conversion in the waveform domain using discrete self-supervised units. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9048-8061, 2023.
Gallil Maimon, Avishai Elmakies, and Yossi Adi. Slamming: Training a speech language model on one gpu in a day. arXiv preprint Xxix:2502.18844 2025.
Soumi Maiti, Yifan Peng, Shukjac Choi, Jee-weon Jung, Xuankai Chang, and Shinji Watanabe. Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 13326-13330. IEFE, 2024.
Fabian Mentzer, David Minnen, Firikur Agustsson, and Michael Tschannen. Finite scalar quantization:
Vq-vae made simple. arXiv preprint arXio:2309.15500 2023.
Christoph Minixhofer.
Libritts phones-and-nneeee hamiinnii/llbrittt--pooosssss med 2023.
http:///nggingfacc.co/aaaaattt https: http::///uuguugggo..co////aattta
Libritts-phones-andd
Poonch Mousavi, Jarod Duret, Salah Zaiem, Luca Della Libera, Artem Ploujníkov, Cem Subakan, and Mirco Ravanelli. How should we extract discrete audio tokens from self-supervised models? In Proceedings of Interspeech, 2024.
Pooneh Mousavi, Gallil Maimon, Adel Mounen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, Bhuvana Ramabhadran, Benjamin Elizalde, Loren Lugosch, Jinyu Li, Cem Subakan, Phil Woodland, Minje Kim, Hung yi Lee, Shinji Watanabe, Yossi Adi, and Mirco Ravanelli. Discrete audio tokens: More than a survey! Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL http:://ppenneevieennee///eeeeee http://ww...eeeooo//////////aaan
Gautham Mysore. Can we automatically transform speech recorded on common consumer devices real-world environments into professional production quality speech?-4 datasct, insights, and challenges. IEFE Signal Processing Letters, 1206):1006 2014.
16

---

Published in Transactions on Machine Learning Research (04/2026)
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: A large-scale speaker identification dataset. In Interspeech, pp. 2616-26668 2017.
Kenneth Ooi, Karn N Watcharasupat, Santi Peksi, Furi Andi Karnapi, Zhen-Ting Ong, Danny Chua, Hui-Wen Leow, Li-Long Kwok, Xin-Lei Ng, Zhen-Ann Loh, et al. A strongly-labelled polyphonic dataset of urban sounds with spatiotemporal context. In 2021 Asia-Pacifia Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 982 988. IEEE, 2021.
Yifan Peng, Siddharth Dalmia, lan Lane, and Shinji Watanabe. Branchformer: Parallel mlp-attention architectures to caplure local and global context for speech recognition and understanding. In International Conference on Machine Learning, pp. (16627-16643, PMLR, 2022.
Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrah- man Mohamed, and Emmanuel Dupoux. Speech resyuthesis from discrete disentangled self-supervised reprosentations. In Interspeech, pp. 3615-3619, 2021.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speed translation using self-supervised pre-training and data augmentation. Xrviv preprial ar Xix:22004 02967, 2022.
Krishna C. Puvvada, Nithin Rao Koluguri, Kunal Dhawan, Jagadeesh Balam, and Boris Ginsburg. Discrete audio representation International Conference on Acoustics, Speech and Signal Processing (TCASSP), pp. 12111-12115, 2024.
alternative to Melsspectrograss for speaker and speech recognition. In IEEE
Lawrence Rabiner and Biing-Hwang Juang. Pundamentals of Speech Recognétion. Prentice-Hall Signal Processing Series, 1993.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:221.04356, 2022.
Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Toannis Mimilakis, and Rachel Bittner. The MUSDB18 corpus for music separation, December 2017a. URL http:://lo..orr//0.522////edd
1113122
Zafar Rafii, Antoine Liutkus, Rabian-Robert Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The MUSDB18 corpus for music separation.
1117372 2017b.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samucle Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, et al. SpeechBrain: A general-purpose speech toolkit. arXiv preprint S.K::2106.06622, 2021.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. DNSMOS P.835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
Simon Rouard, Francisco Massa, and Alexandre Défossez. Hybrid transformers for music source soparation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
Kohei Saijo, Gordon Wichern, François G. Germain, Zexu Pan, and Jonathan Le Roux. Té-locoformer: Trans- former with local modeling by convolution for speech separation and enhancement. In Proc. International Workshop on Acoustic Signal Enhancement (IWAENC), September 2024.
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-yaan, Rif A. Saurous, Yannis Agionvrgiannakis, and Yonghui Wu. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions. In IEEE International Conference on Acoustics, Speech and Signal Processing (TCASSP), pp. 1779-4783) 2018.
17

---

Published in Transactions on Machine Learning Research (04/2026)
Jiatong Shi, Xutai Ma, Hirofumi Inaguma, Anna Sun, and Shinji Watanabe. MMM: Multi-layer multi-residual multi-strean discrete speech representation from self-supervised learning model. In Proc. Interspeech 2024, pp. 2569 2573, 2024a.
Jiatong Shi, Jinchuan Tian, Yihan Wu, weon Jung, Jia Qi Yip, Yoshiki Masuyama, William Chen, Yuning Wu, Yuxun Tang, Massa Baali, et al. ESPnet-Codec: Comprehensive training and evaluation of neural codecs for audio, music, and speech. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 562-569. TEEE, 2024b.
Jiatong Shi, Hye jin Shim, Jinchuan Tian, Siddhant Arora, Haibin Wu, Darius Petermann, Jia Qi Yip, You Zhang, Yuxum Tang, Wangyou Zhang, Dareen Safar Alharthi, Yichen Huang, Koichi Saito, Jionghao Han, Yiwen Zhao, Chris Donahuc, and Shinji Watanabe. VERSA: A versatile evaluation toolkit for speech, audio, and music. In 2025 Annual Conference of the North American Chapter of the Association for Computational Linguisties - System Demonstration Track, 2025. URL httpps://openreevaww net
http://iwwweee/oorrueeeeee nett/Eoruum/dd-zUUnnbbyyyy
Jing Shi, Xuankai Chang, Tomoki Hayashi, Yen-Ju Lu, Shinji Watanabe, and Bo Xu. Discretization and re-synthesis: an alternative method to solve the cocktail party problem. arXiv preprint arXiv:2112.0938: 2021.
Amilay Sicherman and Yossi Adi. Analysing discrete self supervised speech representation for spoken language modeling. 2023.
Saeki Takaaki et al. UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022. In Interspeech, pp. 4521-4525, 2022.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Represenlations, 2024. URL http:: htttpa://openreviev. http:///ppeneeeeeeeeeeeettrrrrre
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xixi preprint or wxxx:2307.09255 2023.
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanctakis, Gissel Velarde, Kirk McNally, et al. Hcar: Holistic evaluation of audio representations. ions. In NewrIPS 2021 Compelitions and Demonstrations Truck, pp. 125 145. PMLR, 2022.
George Tranetakis. Gizan music/speech collection, 1999. URL http://marsyas.
George Tzanetakis and Perry Cook. Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293-33) 000,,02,2202 2002.
Cassia Valentini-Botinhaa. Xin Wang, Shinji Takaki, and Junichi Yamagishi. Investigating RNN-based speech enhancement methods for noise-robust text-to-speech. In Speech Synthesis Workshop, pp. 146 152,,,2,5551 2016.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, 2017.
Benjamin van Nickerk, Marc-André Carbonneau, Julian Zaïdi, Matthew Baas, Hugo Scuté, and Herman Kamper. A comparison of discrete and soft speech units for improved voice conversion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6562-6566, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In International Conference on Neural Information Processing Systems (NeurIPS), pp. 6000-6010, 2017.
E. Vincent, R. Gribonval, and C. Fevotte. Performance measurement in blind audio source scparation. JEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462- 1469, 2006. doi: 858005.
18

---

Published in Transactions on Machine Learning Research (04/2026)
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXic:2301.021
2023a.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verification. IEEE Signal Processing Letters, 25(7):926 9300, 2018.
Jiamnnng Wang, Zhihao Du, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, Wen Wang, Siqi Zheng, Chang Zhou, Zhijie Yan, and Shiliang Zhang. LauraGPT: Listen, 2023b.
attend, understand, and regenerate audio with GPT. arXiv preprint urXiv:2310.0400
Siyang Wang and Êva Székely. Evaluating text-to-speech synthesis from a large discrete token-based speech language model. In Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pp. 6464 6474, 2024.
Zhichao Wang, Yuanzhe Chen, Xinsheng Wang, Lei Xie, and Yuping Wang. Streamvoice: Streamable context-aware language modeling for real-time zero-shot voice conversion. In Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers), pp. 7328-17777 7338, 2024a.
Zhong-QUi Qiu Wang et al. Sequential multi-frame neural beamforming for speech separation and enhancement. In IEEE Spoken Language Technology (SLT) Workshop, pp. 905 911, 2021.
Ziqian Wang, Xinfa Zhu, Zihan Zhang, YuanJun Lv, Ning Jiang, Guoqing Zhao, and Lei Xie. SELM: Speech enhancement using discrete tokens and language models. In JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 11511111665, 2024b.
Pete Warden. Speech Commands: A dataset for limited- speech recognition. or preprint arXiv: 00888844..0200 2018.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Interspeech 2018, pp. 2207-2211, 2018. doi: 00.21437/Interspecc/ 201--/ 1416..
Shuhei Watanabe. Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance. arXiv preprint arXiv:2304.11192 2023.
Dan Wells, Hao Tang, and Korin Richmond. Phonetic analysis of self-supervised represontations of English speech. In Interspeech, pp. 3583 3587, 2022.
Shu wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. SUPERB: Speech Processing Universal PERformance Benchmark. In Interspeech, pp. 1194 1198, 2021.
Scott Wisdom, Hakan Erdogan, Daniel PW Ellis, Romain Serizel, Nicolas Turpault, Eduardo Fonseca, Justin Salamon, Prem Soctharaman, and John R Hershoy. What's all the fuss about free universal sound scparation data? In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 186-190. IEFE, 2021.
Haibin Wu, Kai-Wei Chang, Yuan-Kuei Wu, and Hung-yi Lee. Speechgen: Unlocking the generative power of speech language models with prompts. arXiv preprint arXiv:2306. 02207, 2023a.
Haibin Wu, Xuanjun Chen, Yi-Cheng Lin, Kai-wei Chang, Ho-Lam Chung, Alexander H. Liu, and Hung-yi Lec. Towards audio language modeling-an overview. arXiv preprint arXin:2402. 13236, 2024a.
Haibin Wu, Ho-Lam Chung, Yi-Cheng Lin, Yuen-Kuei Wu, Xuanjun Chen, Yu-Chi Pai, Hsiu-Hsuan Wang, Kai- Wei Chang, Alexander Liu, and Hung-yi Lee. Codec-SUPERB: An in-depth analysis of sound codec models. Xiv preprint or Xrx::22002 13071, 2024b.
19

---

Published in Transactions on Machine Learning Research (04/2026)
Haibin Wu, Naoyuki Kanda, Sefik Emre Eskimez, and Jinyu Li. Ts3-codec: Transformer-basee simple streaming single codec. arXiv preprint ar 18803, 2024c.
Yi-Chiao Wu, Israel D. Gebru, Dejan Marković, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-,, 2023b.
Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigeodec: Pushing the limits of low-bitrate neural speech codec. arXiv preprint (Xx::2409005377 2024.
Huaying Xue, Xiulian Peng, and Yan Lu. Low-latency speech enhancement via speech token generation. ICASSP 2024- 2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 661-665, 2024. doi: A1109//CASSSP4445552222
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019.
Dongchao Yang, Songxiang Liu, Rongiie Huang. Jinchuan Tian, Chao Weng, and Yuexian Zou. HiFi-Codee Group-residual vector quantization for high fidelity andio codec. arXiv preprint ar Xix:29055002666 2023a.
Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, ond Language Processing, 31:1720-175 2023b.
Dongchao Yang, Dingdong Wang, Haohan Guo, Xueyuan Chen, Xixin Wu, and Helen Meng. Simplespeech: Towards simple and efficient text-to-she text-to-speech with scalar latent transformer diffusion models, 2024a. URL https://arxinn org/ ab/a//aa/// abapaaaa...2/////////////// 2406.02328
Haici Yang, Jiaqi Su, Minje Kim, and Zeyu Jin. Genhancer: High-fidelity speech enhancement via generative modcling on discrete codec tokens. In Interspeech 2024, pp. 1170-1174, 2024b. doi: 10.21437 IIteerspctpp 2024-590.
Yifan Yang, Feiyu Shen, Chenpeng Du, Ziyang Ma, Kai Yu, Daniel Povey, and Xie Chen. Towards universal speech discrote tokens: A case study for ASR and TTS. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10401 10405, 2024c.
Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 25697 25705, 2025.
Jia Qi Yip, Shengkui Zhao, Dianwen Ng, Eng Siong Chng, and Bin Ma. Towards audio codec-based speech separation. arXiv preprinl ur Xin:24406. 12434, 2024.
Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, and Mirco Ravanelli. Speech self-supervised representation benchmarking: Are we doing it right? In Interspeech, pp. 2873-2877 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. SoundStream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 493-507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. In Proc. Interspeech 2019, pp. 1526-1530, 2019.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, al. Anygpt: Unified multimodal 11m with discrete sequence modeling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1667-9662, 2024.
20

---

Published in Transactions on Machine Learning Research (04/2026)
Dong Zhang, Rong Ye, Tom Ko, Mingxuan Wang. and Yaqian Zhou. DUB: Discrete unit back-translation for speech translation. In Findings of the Associalion for Computational Linguisties: ACT, pp- 7147-7164, 2023.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speech Unified speech tokenizer for speech large language models. In International Conference on Learning Representations (ICLR), 2024.
21

---

Published in Transactions on Machine Learning Research (04/2026)
A General Information
A.1
Computational Resources
We designed our benchmark to be computationally accessible. Every task runs on GPUs with 32 GB or more of VRAM. Tasks like keyword spotting takes only 8 hours, while TTS might require about 48 hours on a single NVIDIA A100 GPU.
A.2 Impact
We believe DASB can have a positive impact on the research community. We do not foresee a direct negative socictal impact or misuse of our benchmark. However, we acknowlodge that DASB can potentially accelerate progress multi-modal large language models, which, in turn, have a wide range of polential positive and negative uses that society is still working to assess.
A.3 Hosting and Maintenance Plan
DASB a community-driven and open-source initiative. We plan to extend by running additional experiments and including new audio tokenizers and tasks. We welcome external contributors.
A.4 Licensing
Our work licensed under Apache 2.0 htttss://ww..paceee http:///www.pppcce.ro//eeee/e/// orgg//icemeee/LLCCSS5222...
Table 6: Licenses for the models used in our benchmark.
Model
License
HuBERT-large
Apache 2.0
WavLM-large
ATTRIBUTION-SHAREALLKK 2.0
Wav2vec2-large
Apache 2.0
EnCodec
MIT license
DAC
MIT license
SpeechTokenize
Apache 2.0
SQCodec
MIT license
Mimi
CC-BY-4.0
Wokeezenieer
MIT license
A.5 Author Statement
We, the authors, will bear all responsibility in case of violation of rights.
22

---

Published in Transactions on Machine Learning Research (04/2026)
B
Discrete Audio Models Details
Additional information about the tokenizers used in our benchmark is provided in Table 7
Table 7: Features of the Considered Discrete Audio Encoders.
Model
Dataset
Repo
Dicroto Hubert
MMusevii et al.
X024
LibriSpwec006) Korvas et al.
2014
buggingffoc eo, ww./poomehmmpppoo 1851, Quactirationn
Disrrete WarLM
Mousavi
LibrSppecclta Korvas
2014
bugeiugfacee.co poooocumooasss//////// /SL, Quastizationn
Diecete War2Vor2
Monsevi
2024
LibrSSweehdii Korvas
2014
brggingffice en www.ppoohmmssssii...p.oooo (SSI, Quaetiration
EnCodee
Délorez et al. 2125
DNS Daaeey et al. 2024 CommontValce
Ardila
github. com facehookzeesearch /encodee
AndlosSet (Gamneee
FSDMIK Fossecca al. [1021] 60d
Jamendl
(Hoplanov el. xl. 2015
DAC Kumar 2025
Mysore 2014
(Dubey
gislub coeu/ dddesrippticc dewpciiiiiieeeeiipeeeeeeeeeepeee
VormmonVoce
Arvila al
2020)
VCTK Yonaegiola 2119)
USD5
Rata
el ab. 2017 J and Jamendo [Boglaune et
2019
SpencToolenzee Tokonizes Zhang et al.
2024
LibriSpocchdiii Korvas et al. M114
gitbab.com, XInFn D/Speshhhhhhennneeee
Minii Déforses al.
Unmpervised English speech
Irugging cu/ /kyuta/mmiiii
SQCodee
(Yang et al.
20248
Maltilinuua Lihrippeehh (MLS) (Korvas et al.
2014
huggingface coy Dowgo/ooo/////o UssAnioo
War Tokenisee Ji
2024
LibiT'SS Zem al 2010 CommonNoice
Andik
Ingginggfoe hww.ssuurrrruuuu Waw www.:meee--emninttt---eeeee 75tolom
VCTK Yammggllii et al. 200
AndioSet Gemaneo
and Jameudo Degianov
C Dataset and Downstream Models
Table
S provides a summary of the datasets and the two downstream architectures used for each task.
Table 8: Dataset and Downstream Models
Dusmeet
Tink
lat Architecture
2nd Archilecture
Dataast Lisk
Speash Reseptic
BIESTM
MaC
Me
speilee we werlforinn/dee aasis////eliiiiii
ECAPA, TONX
BILSTM + Tiver
Emsotiea Beosaaticz
ECAPA-TENS
Tims-Pholle Lisse
Spenth
Eoyand Spocting
TCAPA- TTANN
Vestons
Dewcrrio yo Aorler 47 12. (1..0/ commard
Vak:Buuo
Esheseecacal
Canferner
CRDSK
Sporth Speeaalion
Conforncr
ORDSS
1/ei 10
15 2M1T
Tiikk Speeet
VALL-X
SwaTow Thondoriee
LTS opendr (e00 (1 siar purire coreer Sheeh Pabart
F1SS8
[ Keikk/z11 X1
Auiio Soute Sepentiku
Confernar
CRINX
las (an00 www.gnnnnaeeeo....... 20 109 dine 1h2 (anee Calllollll
Musi Souree Sepaniikss
Confeence
CRINN
692AN
Teae mir 1iii 3101
Mtudc Graze Chedfraation
LUAAPA-EINS
Lavez
Fvan Sond C Chesiivation
ECAPA. TONN
Xweise
632AN
2313
Midi w Spoch Castiration
ECARA- 1INNN
Xiettee
betpec/ ptrpel/ggkch.cc..oo www.hh.eeeee.............oo
38yath
(upi e
Plda Clusdaatan
ECAPA-EINN
Xwettee
23

---

Published in Transactions on Machine Learning Research (04/2026)
D Second Probing Head Result
Tables 9 10 and 11 show the results obtained with second downstream architectures. Note that table g
indicates the first and second architectures explored for each task.
For English ASR, the Branchformer consistently outperforms the BiLSTM across most tokenizers, with the exception of SQ-Codcc. This deviation likely due two main factors: first, the use of cross-cntropy loss in Branchformer versus CTC loss in BiLSTM; and second, the exceptionallyy large vocabulary size of SQ-Codec (~20k tokens), which makes optimization more challenging under cross-entropy. Additionally, SQ-Codec was originally developed for diffusson- based models rather than standard autoregressive transformer architectures, which may further limit its compatibility with peneence-to-oqqueee training objectives. We observe a similar trend for SQQCodeec in other tasks as well, including SE and TTS.
The second TTS architecture we use is a simple autoregressive encoder-decoder transformer with a single linear prediction head that jointly predicts all codebooks. While this setup is effective for single-layer continuous features and tokenizers that retain high-level representations, does not scale well to low-level representations that require many codcbooks for high-fidelity reconstruction (c.g., acoustic tokenizers). This has led to non-convergence in some configurations. Architectures with built-in residual prediction, such as VALL-E, are better suited for handling such complex tokenizations.
Table 9: DASB results for discriminative tasks (speech) with the second downstream architecture.
ASR-En
ASR-LR
KS
Models) Models|Tasks Tasks
#Q
WERI
WERI
ACC1
ACC1
ACC*
ACC1
EER
Clean
Olher
Welsh
Basque
Contimoons
4.07
6.81
34.98
13.51
68.60
75:20
98.70
99.02
5.817447
12.70:0.37
20.0000.13
100.00=1.45
11.56-00666
42.10=1
19.47+0.05
68.53-5.22
84.42412.45
21.11=00.21
Eac SMA-1
8.4310:13
21,772 0.36
99.33:30.00
56 26.288226.00 16
417721.50
18.50:0.36
68 68.58:1.92
78. 111-111144
15.76:20009
0.95+1.17
23.24
80.86-00211
63.13-031
43.4741.27
17.30+0.22
67.23-22335
80:1247-98
21.27=1.7
14.84++0.25
+3.88+0:20
98.04+0.26
48.37+44.07
45-20+0.54
17.10+0.42
68 08.00+6.30 6.30
78-98+110
23.00+00.0
DAC-SMA-24
10.73+ 0.10
25.00= 0.20
97. 97.20-00.66
44.50=3.22
44.00=3342
16.50=0.10
61.80=2288
02.662+3322 23
16.38-20.68 6.68
13.13:0.116
28.47:10.19
98.0410..7
47.174 40. 034433578
35. 25..3993.9
15.30.10.00
57 57.20
73.00112.58 50
21.0410.0 45
9.4810.10
12.6810.10
99.31110.06
36.95.00.31
56. 1.11
49.00.0.14
57.30113.08
77 7.51
21,18:0. 13
ST-S-16
9.06s: 0.45
21.22+0.23
99.83:00.84
33.28=00.0
56.37+
47.07=0.26
52.6 52.60-#0.41
$1.20++0.22 22
19.24 44
9.73
22.65::0.41
98.60:20.46
39.02=:1.38
46.90 1.22
44.80
92.15:0067
78.036h00111
17.27=0006
Minni S-24
10.8010.58 56
24.1010.36
97.4710.07
43.9012.45
42 12.7342..7
35.90.10.45 45
50.0110.00
02.33.10.96
21.15
DWavL-S-116
1.7810225
10.5811.117
$1.18111.9
22 22.5320.31
00:67:0.70
61.6710.05
96.7910.79
$8.711-1.90
21,29.0.77
6.0710117
9.67:0020
81.01:2..00
21.69:1.34
61.20:1..00
70.27:10412
97.21
02 12:13144778
12.95:20.37
DHuBERT-S-R
65
10.38+0.00
74.46:11.50
23. 03.36-50.57
58. 53:-3300000
57.23::0.17
63 63.53:2218
82 02..001-.000 20
17:84=20.6
4.02+0.31
8.96+ 0.23
$2.62+1.34
32.07+13.07
50.40+1.70
61.90+0.08
83.90+1.355
$9.064-4.04
12.00-+0.12
DWav2Vo228-16
8.44+0.17
19.57+00.2
88.60+-3.44 3,46
29.05-00.64
50. 10.10++1.90 1.90
51.93+00122
92.28-0.22
74.53+00.000
21.04=00.52
5.52: 0.12
12.52:0015
92 12.86:11180 1.88
25. 37:3 0.20
59. 7.3.733
61.10:4036
95.41=1.022
80 0.065
11.63:10.19
SQ-SMA-16
01. 01135400400
92 W2.50+00:41
97. 07.0000030
98 107
40.913+1.11 18
16:27+40.05
85.47+3.24
US 18.22411.22
13.93-H0.57
WT-SMA-24-2
16-11+0.18
35.48+0035
99.45+00.04
62.63+00776
56. 07-0010
16,23+0000
71.27+1.64
94.58-6.225
17.08++048
24

---

Published in Transactions on Machine Learning Research (04/2026)
Table 10: DASB results for generative tasks (speech) with the second downstream architecture.
Modele) Maske
SE
SS
Speech
TTS
DNSMOS
dWER
Spk Stust
DNSMOS Rect
DNSMOS
dWER
Spk
UTMOS
dWER
Sept
Simn
Contimnons
3.49
4.92
0.93
3.43
24.42
0.873
4.08
4.76
3:17
0:01
36.78
1.15
0.85 0.00
3.19
3.11=20:01
86 86.95:00774
0.87:00000
1.34 0.00
53 63.93=0.99
Enc-SM SMA-21
g
2.91
2: 0.01
50.32
2.81
0.81 + 0.01
3.54
3.06:20.00 0000
67. 1T.66.22.22
0.88.2
32
3,00
0.02
108.06
0.76
3.72
3.00=0.00
98 84.8222.68
0.83+0.00
2
3.31
4: 0.01
61.72 4: 2.54
0.56 4: 0.00
3.16
2.97:0.011 01
11..5.10.50
0.539110.00
1.7940.00
36. 16.93=6.05
DAC-SMA-24
2.91
0.10
131.33
0.68 0.03
3.67
3.16++001
82:31+11442
0.892++0.00
1.61+00.5
102.08+42955
32
2.84
0.02
103 88
8.79
0.78 0.01
3.76
2.06+00.0
178.0+25.0
0.80+0.01
ST-S-16
3.15 0.01
31.76
0.74
0.85 4: 0.00
3,20
3.12:0.01
89.9 89.91.0.94
0.56210.00
2.18:10.01
16. 5.32
3.30
0.01
33.95 3.29
0.85 0.00
3.72
3.29:10:00
70.43:0.75
0.8740.01
1.76:00.02
80. 89.06=3.16
Mimi-S-21
8
3.31 +: 0.01
107:27 #: 14.68
0.81 + 0.01
3.37
3.30 3.30::0000 00
109.00 019.00=22110 1.
0.806
1.89 03
25.26 = 4.40
2.98
0.14
159.0 =: 35.59
0.75
3.65
3.20:20.00
115. 115.662:1225
0.8120.00
1.21:0001
112.22:8875
DWarT,S-16
2
3.57 +: 0.01
27.64
4.13
0.58 + 0.00
3.57
2.97=#0.01
$0.36+:1.05
0.839+0000
3.01=00.2
4.90:#44.1
3.58
0.00
9.39
0.28
0.89 0.00
3.75
3.77=0.00
33.55 -0.5
01806+0000
3.87=0.02
13.36-22:20
DHuBERT-S-16
2
3.46
0.03
09:97
3.33
0.85 + 0.00
3.88
3.65 3.65++0.00
83.95+0.18
0.826+0000
-..11+0.01
1.79+11.79
0.01 3.50
0.09 13.31
0.58 + 0.00
3.72
3.09::0.00 00
76. 76.99#1.226
88:0.00
3.80: 0.00
5. 6.76:00499
D8
3.55
0.01
30,91
0.32
0.88 0.00
3.61
3.64:10.01
94.88111.68
0,505.10.00
2,26:10.02
26, 16.:26=1.22
6
0.01 3.57
13:06
0.40
0.89 + 0.00
3.77
3.74+0001
74.50++1.40
0.88+0.00
3.07+0001
13:90-0.32
SQ-SMA-16
3.09
0.01
+ 35.00
0.76
3.77
3.58::0.00
72.39::0.6
0.90:10.00
WT-SMA-24
1
3.36 0.01
62.15 2.52
0.85 +: 0.00
3.57
3.43::0.01
128.333:2.49
0.85:10.00
2.44
14.00=0.84
Mixture
3-43
Table 11: DASB results for generative and discriminative tasks (music audio) with the second downstream architecture.
Modells Taske
SS Audio
SS Music
ESC ACC1
USC ACC1
MSC
PC
MGC
SI-SDRi
81-SDRI
ACC1
ACC!
Rec
Sep
Rec
Sep
Contionous
15.07
18:20
9.58
31.99
94.50
$7.60
100.00
90.80
87.00
0.76
5.57.10.49
3.35
-0.03
1-46
-3.78
1 1.19
4.76 1. 1.13
31 51.8010.47
25.8014.47
07 1350
71.30.0.59
10.33.11.70
Emc-5MA-24
3.87
0.01 1: 0.07
7.00
-11.96
1: 0.38
-15.55
0.52
-0.16 0.1
07.00:10773
30,301-4005
100.00:0.000
72.1510.74
51.673380
5.76
-3.25
11.10
0.09
-15.75
-0:35
35-13+145
67.27-1.46
30-67+1775
0.12
2.58 0.00
2.37
-1.88
0.08
-0.31 0.1
3.55 + 0.05
31
60.08-2.24
100.00-0.00
56. 08.73-0038
50:00+0.0
DAC-SMA-21
3.38
-0.06
0.58
6.66
-8.19
+ 0.76
-11.61
+ 0.75
0.84 + 0.59
28
21.50e22.25
51 91,67+0000
58.40 18.40=0.300 50
47.67 37.67+3300
32
4.78
-7.8
+ 0.15
8.54
-31.00
+: 0.18
-11.78
0.53
-3.29 +: 0.86
25.7780822
15,07=1..5
100.00:00000
48 18.87:0.00
39. 67
SQ SMA-IR
3.62
5.18 1 0.26
5.53
-1.78
0.08
..71
1 0.19
3.77 1 0.04
31.3711.57
13,2010086
01.67.0000
49. 10.20-22.16
42:67 1.0.7
24-2
21.6
0.57
-7.99 0.88
3.05 1 0.79
34.50.10.22
18.05.2.766
07:22
18.87
48.00.1141
Mixture
-16.5
-7.71
50.01
-inf
E Effect of Embedding initialization
We study different configurations for initializing the embedding layers of audio tokens (Table 12 Two options are considered: (1) random initialization of the embedding layers, and (2) initialization using the encoder's embedding layer, without freesing it during training. We evaluate both setups on two tasks: ASR LibriSpeech (In-domain, as the dataset is included in all the tokenizer's 's training data) and keyword spotting on the Google Speech Commands dataset (out-of-distributtinn) With regard to in-domain vs. out-of-domain performance, we hypothesized that encoder-basee initialization would be more beneficial in the in-domain setting- Our observations partially support this: semantic and hybrid tokenizers appear more sensitive to domain shift, and initialization can sometimes harm generalization on out-of-distribution tasks. In contrast, acoustic tokenizers tend to benefit more consistently from initialization. Consistent with findings in (Mousavi et al. 2024), semantic tokenizers tend to perform better with random initialization, while acoustic tokenizers improve with encoder-based initialization. However, this trend may also reflect architectural differences. For example, encoder embeddings are often lower-dinensional (e.g., 128), requiring an additional projection layer to match the 1024-dimensional downstream embeddings. This added capacity may itself contribute to improved performance, independent of initialization strategy. To ensure consistency and fairness, we adopt random initialization for all embeddings when reporting final results. In the case of SQ-Codec the learnable embedding configuration failed to converge, likely due to its large vocabulary size (~20k tokens). As
25

---

Published in Transactions on Machine Learning Research (04/2026)
workaround, use ternary matrix-based dequantization scheme to represent its token embeddings. Finally, to eliminate optimization bias, we perform separate learning rate tuning for all embedding initialization experiments.
Table 12: Comparison of random initialization vs. encoder-based initialization (pretrained + fine-tuned) on ASR
BiLSTM) and keyword spotting (ECAPATDNN).
ASR (WER)
KS (ACC)
Models) Tasks
#Q
Random
Initialized
Random
Initialized
Euc-SMA-24
8
16.83
38.98
10.69
28 64
68.53
76.50
DAC-SMA-24
8
16.29
39.57
15.52
37.54
61.83
77.60
ST-S-16
12.09 /30.99
8.56
24.44
92.69
87.60
Mimi-S-24
8
13.96 31.53
11.86
28.18
92.14
91.17
DWavL-S-16
4.32
10.73
3.77
9.79
97.20
87.50
WT-SMA-24
28.69
52.90
20.7
43.99
71.26
75.50
F Reconstruction Analysis
We use the CodecSUPERB benchmark and VRESA results to evaluate reconstruction quality. Table 13 summarizes the result on the Librispeech test-clean set. UTMOS is used as the primary perceptual metric. For application-level evaluation, the rosynthesized audio is passed through pre-trained models for ASR and speaker identification (SPK), with WER and accuracy the evaluation metrics respectively. Overall, DAC and SQ-Codec achieve the highest average performance. Interestingly, these same models rank among the lowest in our downstream evaluations and show high variability. This contrast is particularly noteworthy. While semantic tokenizers typically outperform hybrid and acoustic ones in downstream tasks, the trend is reversed in reconstruction-bassd evaluations. Additionally, we observe that increasing the number of codebooks improves reconstruction quality. In contrast, in downstream tasks, more codebooks often reduce performance due to added redundancy and complexity. These findings suggest that strong reconstruction quality does not necessarily indicate better preservation of semantic or task-relevant information. One reason for this is the role of the decoder. Since most reconstruction losses are applicd at the decoder output during training, the decoder plays significant role in shaping the perceptual quality of the resynthesized audio. As result, reconstruction performance may reflect decoder strength more than the quality of the token representations. In general, reconstruction metrics remain essential for evaluating tokenizers in generative settings or transmission scenarios, where the output is audio and fidelity and intelligibility are critical. These findings highlight the importance of evaluating discrete tokens from multiple perspectives. While discrete tokenizers offer clear advantages in modularity and efficiency, there still a performance gap compared to continuous features. Bridging this gap is crucial for establishing discrete tokens as viable alternative in future multimodal language models.
G Hyperparameters
For English ASR, we compare character-level and 500-unit byte pair encoding (BPE) tokenizations, selecting the better-performing option. For multilingual datasets, BPE segmentations with 100, 200, 500 units are chosen based on dataset size and tokenizer characteristics. The best-performing setting is reported.
General TTS Scaling Observations. Similar to the approach used in ESPNet's (Watanabe et al., 2018 implementation, during inference, we generate 10 independent samples using top-K sampling and then, using Whisper Large (Radford et al.) (2022), select the one that best matches the text label. While this technique significantly reduces variability and brings most tokenizers into dWER range comparable to that of mainstream generally available systems, notably, it does not sufficiently improve performance for high-band widdd configuration, indicating fundamental difficulty with predicting a large number of codebooks in each step rather than a simple issue of high sampling variability. Without multi-sampling, we observe that a much wider gap between semantic and acoustic tokens in word error rates.
26

---

Published in Transactions on Machine Learning Research (04/2026)
Table 13: CodecSUPERB benchmark performance on the LibriSpeech test-clean set.
Models
#Q
UTMOS
WER. (beam=5)
Spk
Sim
Ground truth
4.09
2.83
100.00
2
1.58
5.44
41.70
Ene-SMA-24
8
3.09
2.78
71.74
32
3.74
2.77
78.19
1.67
9.59
DAC-SMA-24
3.83
2.41
85.65
32
4.05
2.72
79.89
2
2.32
4.20
34.76
ST-S-16
8
3.60
3.53
72.66
3.60
3.72
69.72
Mini-S-24
32
3.91
2.96
85.1
2
3.31
4.97
32.55
DWavL-S-16
6
3.31
4.34
34.86
3.43
7.18
28.64
DHaBERT-S-16
3.71
5.52
31.35
2
2.75
14.12
18.29
DWav2Ver2-8-18
6
3.69
3.99
31.79
SQ-SMA-16
3.90
2.37
86.66
WT-SMA-22 24-2
3.77
8.10
59.89
H TTS Inference Hyperparameter Search
The TTS task, in particular, proved rather sensitive to the selection of certain hyperparameters in preliminary experiments; furthermore, given VALI-E is a generative model conditioned on text tokens with top-K sampling used during inference, the selection hyperparameters involves a trade-off betweon natrualness and fidelity the text conditioning.
We perform grid search of combinations of the the k hyperparameter in top- K and the sampling temperature using a random sample of 100 speech samples.
Figure 4: TTS Inference Hyperparameter Search UTMOS
UTMOS, k=10
UTMOS, k=20
UTMOS, k=30
4.5
DAC SMA-24 (32)
4.0
4.0
4.0
Lnc-5MA-24
DHUBERT:S166 (6)
Mimi-5-24 (8)
DWavL-S-16 (6)
DHUBERT-S-16 (2)
3.0
3.0
3.0
Enc SMA-24 (8)
DWww2Ver2-5- (6)
DWav2Ve:2-S-26
2.0
2.0
2.0
DAC SMM-24
Drfavt-S-18 (2)
1.5
Enc-SMA-24 (2)
S (32)
1.
0.8
Tamperature
1.0
1.2
0.8
Temperature
1.0
1.2
1.0
0.8
Temperature
1.0
1.2
WT 11--MM-44 (1)
ST.5 16 (2)
Figuces and show the relationship between k, sampling temperatures and the UTMOS and dWER observed (simplified sampling, no post-selection). We observe empirically that for most tokenizers, there appears to be an optimal setting for both hyperparameters, and performance can be affected significantly by the selection. The selection of optimal parameters is not universal and, rather, is, tokenizer-dependent. The effect of the sampling temperature and the chosen on UTMOS appears to be rather minor for most tokenizers except the extremes, whereas the effect dWER can be very significant, which is consistent.
27

---

Published in Transactions on Machine Learning Research (04/2026)
Figure 5: TTS Inference Hyperparameter Search dWER
OWER, km10
dWER, K-20
dWER. km30
100
100
100
daclarge
encodec-large
hubert-mid
mini-mid
wavlm-mid
hubert
60
60
60
dWCR
OWER
SWER
dsc
war2ver2-mid
40
dacmid
speechtokenize-nn
encodoc
wartokenizer
3.0
1.2
0.0
1.0
1.2
1.0
1.2
speechtokanizer
Temperature
Temperature
with the intuitive expectation that the parameters control primarily whether the extent to which the tokens from tail ends of the distribution will be explored. The lower the k and the temperature, the closer the sample is to selecting only the most likely tokens given the conditioning, the higher the parameters, the more diversity is allowed. UTMOS being measure of vocal quality is more related the audio quality of the tokenizer's reconstruction and will be affected minimally if the model mispronounces word emits non-word sequence of sounds. We also observe that for many tokenizers the relationship between the temperature and the dWER is approximately convex, especially at the medium selection (k = 20). One likely explanation for the observed behaviour is that insufficient sampling diversity may cause instability in antoregressive infiuence, i.e. once a prediction error is made, it more difficult for recover; however, too much tail-end sampling may lead to sequences with low overall likelihood given the conditioning (i.e. utterances loosely inspired by the text rather than reproducing it with fidelity).
28


Reviews:
[{"review_id": "XnOkV2FmYg", "reviewer": "Reviewer_GNDm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DASB (Discrete Audio and Speech Benchmark), a comprehensive framework for evaluating discrete audio tokens across speech, general audio, and music domains. The benchmark systematically compares semantic, acoustic, and hybrid tokenizers on both discriminative tasks (ASR, speaker verification, emotion recognition, etc.) and generative tasks (speech enhancement, separation, TTS). The key contribution is the direct evaluation of discrete tokens without relying on decoder reconstruction, isolating the information loss introduced by quantization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I found that a majority of claims made in this paper are well supported with empirical results. However, I am not entirely certain on the following perspectives:\n\n- Regarding the performance difference between continuous and discrete representations, the authors claim that it comes from the information loss due to quantization in the discrete approaches. This should be justified more thoroughly. I wonder whether this hypothesis is true, as it would imply that simply increasing the bitrate could address the gap in the discrete approach, while this does not appear to be valid in practical cases.\n\n- The authors claim that DASB directly evaluates tokenizers without a decoder. This claim confuses me because the generative tasks in Table 3 require a pre-trained decoder or vocoder.\n\n- While the authors carefully design the experimental protocol for fair and rigorous benchmarking, the tokenizers differ in training data, sample rates, and architectures simultaneously, making causal attribution difficult.", "audience_yn": "Yes", "audience_explanation": "Discrete audio tokens are increasingly central to multimodal LLMs. Understanding their limitations is crucial for researchers building such systems. The finding that discrete tokens lose significant information compared to continuous features has direct implications for system design. The set of experiments to evaluate the understanding and generation tasks of different types of speech tokens is exhaustive, and the resulting take-away messages appear insightful.", "requested_changes": "It would be appreciated if you could address my comments above. Additionally, I have a couple of requests as listed below.\n\n- It would be appreciated by readers who are not familiar with this field if the tables showing experimental results indicated whether models are equipped with acoustic or semantic tokenizers.\n- The main tables are too dense, making it difficult for readers to grasp the global tendency. I suggest creating radar plots similar to Figure 3 but with more detail.\n- While I understand that you adopt simple architectures for the downstream tasks for fair evaluations, providing some numbers with state-of-the-art level architectures would still be helpful for readers to know whether the scores are reasonable.\n- What is #Q? I am fairly certain that this number is related to the bitrate (the number of codebooks?), but it is not defined in the paper. Additionally, the performance with respect to #Q differs depending on the methods. Could you elaborate more on this phenomenon?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GzsInm3JS3", "reviewer": "Reviewer_ygXm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new benchmark to evaluate the quality of audio representations over a variety of discriminative and generative tasks. Compared to prior works that mainly evaluates the quality based on reconstruction quality, this paper proposes to directly evaluate on various downstream tasks without the reconstruction step, as reconstruction may not accurately reflect the performance for the task of interest. \n\nThe authors show that semantic tokens generally outperform acoustic ones, and proposes other suggestions for training\n\nStrengths:\n1. The idea to evaluate the quality of tokenization for downstream tasks is interesting and very relevant in practice as the performance on downstream tasks is what people care about.\n2. The experiments are extensive performed on a wide variety of tasks. The results provide clear overall picture about the strengths and disadvantages of existing tokenizers on different tasks.\n\nWeakness:\n1. The benchmark datasets are mainly from existing datasets in previous works and do not introduce new data sources. The tasks are also somewhat standard classification and generative tasks for speech and sound. Perhaps new and more challenging tasks may be needed to fully evaluate the quality of existing discrete or continuous representations and suggest more interesting future directions for the community.\n2. The paper lacks qualitative evaluation of the tokenization strategies (e.g. cluster plots of representation) or case study, which are both very helpful to understand the quality of tokenization/representation.\n3. A minor weakness is that the benchmark seems to be more focused on speech than sound and music: there much more tasks for speech than sound and music.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by extensive experiments. Different tokenization strategies are tested on a wide variety of discriminative and generative tasks, providing strong empirical support for the claims.", "audience_yn": "Yes", "audience_explanation": "This paper should be interesting to researchers working on speech and audio processing, as well as those interested in incorporating audio inputs to multi-model large language models.", "requested_changes": "1. The reviewer suggests that either design one or two new tasks and/or datasets that could better test the power of existing audio representations or provide a justification why the current choice of tasks and datasets is sufficient. This is critical for my recommendation.\n2. The authors discover that the performance is not promising for sound and music tasks, and that discrete tokens are generally worse than continuous representations. It would be helpful for the community to provide some qualitative illustration/plots of the representations or case studies, which could provide more specific insights on how to improve these representations. This is less critical than the first point, but having this would strengthen the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RXox51QXg0", "reviewer": "Reviewer_Rjak", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces DASB (Discrete Audio and Speech Benchmark) — a large-scale, comprehensive benchmark designed to systematically evaluate discrete audio tokenizers across speech, general audio, and music domains. Its goal is to provide a unified and reproducible framework for assessing the trade-offs between discrete and continuous audio representations across both discriminative (e.g., ASR, emotion recognition) and generative (e.g., TTS, separation, enhancement) tasks.\n\nKey contributions include:\n- Direct evaluation of discrete tokens: Unlike prior work that relies on decoding to waveforms, DASB isolates the information loss from tokenization itself, enabling a more faithful assessment of discrete representations.\n- Comprehensive benchmark coverage: DASB spans multiple domains and tasks with standardized evaluation protocols, supporting semantic, acoustic, and hybrid tokenizers.\n- Extensive ablations and hyperparameter tuning: The paper studies how bitrate, codebook size, and model architecture affect downstream performance, offering practical guidance for future tokenizer design.\n- Quantitative comparison with continuous features: The benchmark establishes a clear performance gap between discrete and continuous representations, identifying where discrete tokens remain insufficient.\n- Open and reproducible framework: The authors release code, documentation, and leaderboards, encouraging community participation and standardization.\n\n**Strengths**  \n- The paper unifies a fragmented research area, covering a broad range of models and tasks with strong experimental rigor (multi-seed, multi-architecture, controlled settings).\n- The decision to evaluate tokens without decoding is methodologically sound and clarifies how much representational power is lost through quantization.\n- The paper is well-structured, clearly explaining benchmark design, tasks, and implementation. Public release of code and leaderboard further enhances impact.\n- Empirical findings (e.g., semantic tokens’ superiority for discriminative tasks, acoustic tokens’ speaker retention) offer practical guidance for both researchers and industry practitioners.\n- The benchmark addresses a pressing need for standardization in the development of multimodel audio-language models.\n\n**Weaknesses**\n- The study is primarily empirical; it does not provide deeper explanations for why certain tokenization strategies succeed or fail.\n- While the benchmark itself is new, much of the underlying setup (datasets, architectures) builds on existing benchmarks (e.g., SUPERB, SpeechBench) limiting the novelty of the contribution.\n- Although encoder/decoder runtime is discussed, the benchmark does not deeply analyze latency or compression trade-offs under deployment constraints.\n- The work evaluates known models rather than proposing new architectures or training objectives to improve tokenization, but this is okay since the primary contribution is a benchmark.\n- The paper highlights that discrete tokens lag behind continuous representations but does not offer concrete strategies to bridge the gap.\n\nOverall, this paper provides a valuable, timely, and well-executed benchmark that fills a crucial gap in evaluating discrete audio representations. Its primary contribution is infrastructural and diagnostic rather than algorithmic, but it is likely to have broad community impact by standardizing comparisons and informing the design of future multimodal audio-language systems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors make claims about (a) the comparative performance of discrete versus continuous representations, (b) the differing strengths of semantic, acoustic, and hybrid tokenizers, and (c) the impact of design factors such as bitrate and codebook size. These claims are convincingly supported by extensive quantitative experiments across a wide variety of datasets (speech, audio, music) and both discriminative and generative tasks. Tables 2–4 and multiple ablation studies back up each assertion with well-controlled comparisons. The authors use consistent evaluation settings and multiple random seeds, reducing variance-driven artifacts — a significant strength compared to prior benchmarks.\n\nThe benchmark pipeline is clearly explained, including tokenizer configurations, downstream architectures, hyperparameter tuning procedures, and evaluation metrics. The design choices (e.g., using the same downstream architecture for continuous and discrete baselines) make the comparisons methodologically fair and isolate the effect of discretization. Figures and tables are generally clear, though dense; the summary of patterns in Section 5 effectively synthesizes the evidence to support the main claims. The authors are careful not to overstate findings — they frame conclusions in terms of observed tendencies and practical implications rather than universal superiority.\n\nWhile the empirical evidence is broad, it is mostly descriptive rather than explanatory. The evidence could be strengthened by deeper causal or statistical analysis.", "audience_yn": "Yes", "audience_explanation": "TMLR’s readership includes researchers working on multimodal learning, representation learning, and self-supervised models. The paper’s focus on discrete audio tokens directly connects to current efforts to unify speech and text modalities (e.g., in models like Gemini, Qwen, and AudioLM). Understanding how tokenization choices affect performance is crucial for building scalable multimodal LLMs — a topic of high current relevance.\n\nThe community increasingly values standardized, reproducible benchmarks (e.g., SUPERB, HELM, BIG-Bench). This work’s DASB benchmark provides a rigorous, domain-spanning evaluation framework for discrete audio representations — a clear contribution to methodological best practices, which is directly aligned with TMLR’s scope.", "requested_changes": "The following changes would strengthen the work.\n\nIf I understand correctly, DASB measures representational utility rather than end-to-end model quality, and that findings may not directly transfer to models trained jointly with audio decoders. The authors should state that if so.\n\nDiscuss more dataset coverage and potential biases. Clarify that handling non-English, low-resource, or noisy-domain data would need further extensions.\n\nWriting issues:\n- Incorrect bracketing of citations. Please use \\citep{} where appropriate.\n- For references, please cite peer-reviewed versions wherever possible. E.g. for “Unity: Two-pass direct speech-to-speech translation with discrete units” by Inaguma et al. please use the ACL 2023 citation (and similarly for other references to arXiv versions).\n- Appendix numbering is off. Appendix A is empty.", "broader_impact_concerns": "The work does not include a broader impact statement. Some reflection on potential bias (coverage of linguistic, gender and cultural diversity) in the data and discussion on potential misuse of multimodal generative models would be useful.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)