DASB-Discrete Audio and Speech Benchmark

## Abstract

Abstract Discrete audio tokens have recently gained considerable attention for their potential to connect audio and language processing, enabling the creation of modern multimodal large language models.
Ideal audio tokens must effectively preserve phonetic and semantic content along with paralinguistic information, speaker identity, and other details.
While several types of audio tokens have been recently proposed, identifying the optimal tokenizer for various tasks is challenging due to the inconsistent evaluation settings in existing studies.
To address this gap, we release the Discrete Audio and Speech Benchmark (DASB), a comprehensive leaderboard for benchmarking discrete audio tokens across a wide range of discriminative tasks, including speech recognition, speaker identification and verification, emotion recognition, keyword spotting, and intent classification, as well as generative tasks such as speech enhancement, separation, and text-to-speech. Our results show that, on average, semantic tokens outperform compression tokens across most discriminative and generative tasks. However, the performance gap between semantic tokens and standard continuous representations remains substantial, highlighting the need for further research in this field.

## 1 Introduction

Traditional speech and audio processing systems have long relied on handcrafted low-level features such as Mel-Frequency Cepstral Coefficients and Filterbanks []. Recently, self-supervised learning (SSL) led to outstanding performance improvements by learning more complex, robust, and general speech features through deep neural networks. Notable models include Wav2Vec2 [], WavLM [], and HuBERT []. In all these cases, the rich information in speech and audio signals is encoded into a sequence of continuous vectors.
Even though continuous vectors have proven effective in capturing the complex details embedded in speech and audio, there is a growing interest in discrete representations. Discrete audio representations, known as audio tokens, transform the original waveform into a finite set of vectors. These tokens are derived using methods such as quantization of self-supervised learning (SSL) models [, , ], neural compression techniques (codecs)[, ], or hybrid approaches [, ] that combine both methods.

What is driving the interest in audio tokens?
Arguably, this trend is linked to the remarkable success of autoregressive Large Language Models (LLMs) such as LLama [], PALM [], BERT [], and GPT []. Unlike audio, these models operate on text, which is inherently discrete.
Inspired by their effectiveness, researchers are exploring audio language models [, , , , , , , ] by representing the audio as a sequence of discrete tokens.
Moreover, audio and text tokens can be naturally combined, paving the way for the development of modern multi-modal LLMs [] capable of processing audio, text, and visual data.
Discrete tokens also simplify audio generation tasks like speech enhancement and synthesis by turning them into classification problems instead of regression models  []. Finally, they also enable efficient data compression for better transmission and storage.
The main drawback of audio tokens is the inevitable loss of information introduced by the discretization process. We ideally aim for audio tokens that preserves crucial information of the original waveform, including phonetic and linguistic content, speaker identities, emotions, and other paralinguistic cues. However, despite the growing trend toward audio tokens, there is still a lack of standardized evaluation benchmarks, with different studies employing varied experimental settings [, , , , ]. Without a consistent framework for measuring and comparing performance, it becomes challenging to determine which audio tokens perform optimally across various tasks.

Figure: Figure 1: The workflow of DASB consists of three steps. First, a discrete audio encoder converts the audio signal into discrete tokens (left). Then, the tokens are combined using attention and fed to a neural model for the final prediction (middle). For generative tasks, the predicted tokens are passed to a discrete decoder, which converts them back into an audio waveform (right). Both the encoder and decoder are pretrained and frozen during downstream model training.
Refer to caption: x1.png

To address this gap, we introduce the Discrete Audio and Speech Benchmark (DASB). DASB systematically assesses various audio tokens across several common speech processing tasks. In particular, our contribution is the following:

- •
We benchmark a diverse set of discrete audio encoders from all three categories: semantic (Discrete HuBERT, Discrete WavLM, Discrete Wav2Vec2), compression (EnCodec [], DAC  []), and hybrid (SpeechTokenizer []).
- •
We consider a wide range of discriminative tasks, including speech, speaker, emotion recognition, keyword spotting, and intent classification. We also tackle generative tasks, such as speech enhancement, separation, and text-to-speech.
For a more reliable assessment, we consider different downstream architectures for each task, following the insights in [].
To the best of our knowledge, this is the first comprehensive benchmark of audio tokens that covers both discriminative and generative tasks.
- •
We publicly release DASB(^1^11[https://github.com/speechbrain/benchmarks/tree/DASB](https://github.com/speechbrain/benchmarks/tree/DASB)) as a modular code repository built on the popular SpeechBrain [] toolkit and licensed under Apache 2.0.

## 2 Related Work

Several research efforts have recently explored using discrete audio tokens as an alternative to continuous features.
Some studies focused on using discrete features for speech recognition and speech translation  [, , ], specifically evaluating the tokens obtained from the quantized versions of the HuBERT model. Similarly, Yang et al. [] examined discrete features for speech recognition and text-to-speech.
Audio tokens have been proposed for speech enhancement as well. For example, Wang et al.  [] investigated the application of semantic tokens to speech enhancement, whereas Erdogan et al. [] proposed a hybrid tokenizer called TokenSplit for both speech enhancement and separation.

While previous studies investigated the use of compression or hybrid tokens [, , ], these efforts were often limited to specific applications and a few audio tokenizers. In particular, previous benchmarking attempts focused on one category of tokenizers, either semantic or compression-based, and mostly on discriminative or generative tasks. For instance, Puvvada et al.  [] compared the performance of EnCodec and DAC [] for speaker verification, speaker diarization, and speech recognition. Mousavi et al. [] benchmarked various discriminative and generative tasks with semantic tokens. Wu et al.  [] provided a comprehensive study of the quality of resynthesized sound with compression and hybrid tokenizers. The latter attempt used pretrained models for speech, speaker, and emotion recognition, and assessed how much information is preserved by feeding them resynthesized audio.
However, it did not address the direct use of tokenized input for training downstream tasks, nor did it deeply analyze the role of semantic tokens. Our analyses, instead, suggest that semantic tokens outperform other tokenizers.

To the best of our knowledge, the proposed DASB benchmark is the first to compare several audio tokenizers from three categories (semantic, compression, and hybrid) across many discriminative and generative speech tasks of broad practical interests. Moreover, unlike previous works on discrete audio tokens, we draw inspiration from the findings presented in [] for reliably benchmarking continuous SSL representations and we consider different downstream architectures for each task.
Similar to the approach taken by SUPERB [] for continuous representation, we offer a standardized evaluation benchmark where researchers can easily evaluate novel audio tokens.

## 3 Benchmark Design

The pipeline of DASB, illustrated in Fig. 1, consists of three components: Audio Encoder, Downstream Model, and Audio Decoder.
The main features of the considered tokenizers are summarized in Table 1, while Figure 2 reports the time and memory resources required by both encoders and decoders. The following subsections describe each module in detail.

### 3.1 Discrete Audio Encoder

The audio encoder converts the audio signal into a sequence of discrete tokens. It is pretrained on large amounts of unlabeled data and remains frozen during the training of downstream tasks.
Different encoders may compress the information in the original waveform at different rates. The compression level is measured by the bitrate, defined as:

$$ $\text{bitrate}=\log_{2}{V}\cdot C\cdot R,$ (1) $$

where $C$ is the number of codebooks, $V$ is the number of vectors in each codebook (vocabulary), and $R$ is the rate of codes per second. It is worth mentioning that a single sequence of tokens might be insufficient to capture the rich and complex information embedded in speech signals. The encoders thus often output multiple discrete sequences, with each sequence corresponding to a different codebook $C$. The encoders can operate at different bitrates simply by adjusting the number of codebooks $C$.
For a fairer comparison, we define three different distinct bitrate ranges that we have identified from the literature [, , ]: low (0-1.5 kbps), medium (2.9-6 kbps), and high (24 kbps). We consider this approach to prevent the trivial conclusion that some audio tokens perform better than others simply due to a higher bitrate.

The design of DASB is flexible, allowing for easy integration and benchmarking of various tokenizers. Using the terminology from [, ], we categorize audio tokens into three classes: semantic, compression, and hybrid tokenizers.

**Table 1: Key Features of the Discrete Audio Encoders. #Params is computed for medium bitrate.**
| Model | #Params | Sampling Rate | Bitrate (kbps) | #Codebooks |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| low | medium | high | low | medium | high |  |  |  |
| Discrete HuBERT | 309.0M | 16KHz | 0.98 | 2.9 | - | 2 | 6 | - |
| Discrete WavLM | 309.0M | 16KHz | 0.98 | 2.9 | - | 2 | 6 | - |
| Discrete Wav2Vec2 | 309.0M | 16KHz | 0.98 | 2.9 | - | 2 | 6 | - |
| EnCodec [] | 17.9M | 24KHz | 1.5 | 6.0 | 24.0 | 2 | 8 | 32 |
| DAC [] | 22.4M | 24KHz | 1.5 | 6.0 | 24.0 | 2 | 8 | 32 |
| SpeechTokenizer [] | 85.3M | 16KHz | 1.0 | 4.0 | - | 2 | 8 | - |

Semantic tokens [, , ] are generated by clustering or quantizing layers from SSL models [, , ]. The tokenization process typically involves selecting specific layers from a pretrained SSL model and applying the k-means algorithm to group their representations. Semantic tokens primarily capture high-level information, such as phonetic, semantic, and syntactic information.
They are not optimized for waveform reconstruction, making them potentially better suited for discriminative tasks like speech recognition. Recent studies, however, have shown that semantic tokens can also be effective in generative tasks [, , ].
We adopt the tokenization algorithm proposed in []. In particular, we consider three widely-used open-source SSL models: Wav2Vec2-large, WavLM-large, and HuBERT-large, each composed of 24 layers.
Then, we cluster six of these layers using the k-means algorithm and select two layers from the lower part (1, 3) to capture low-level information, two from the middle layers (7, 12), and two from the higher layers (18, 23) to encode content and meaning as well.

Compression tokens [, , ] are mainly used for audio compression. They are trained to accurately reconstruct the original audio, making them potentially suitable for audio generation tasks. We integrated two publicly available compression-based tokenizers in our baseline. EnCodec [] has three main components: (i) an encoder network $E$ consisting of a 1D convolution followed by a two-layer LSTM that processes the audio input and produces a latent representation $z$; (ii) a quantization layer $Q$ that compresses $z$ into $z_{q}$ using Residual Vector Quantization (RVQ) [], where distinct codebooks quantizes residuals in multiple steps; and (iii) a decoder network $G$ that mirrors the encoder and reconstructs the time-domain signal $\hat{x}$ from $z_{q}$. The system is trained end-to-end to minimize reconstruction loss over time and frequency domains. It also adopts a perceptual loss using discriminators at different resolutions. EnCodec offers multiple models at low to medium bitrates (1.5 to 24 kbps). DAC [] is an improved version of EnCodec. It combines advances in high-fidelity audio generation with better vector quantization techniques from the image domain, along with improved adversarial and reconstruction losses. DAC also supports quantizer dropout, allowing a single model to support variable bitrates.

Hybrid tokenizers [, ] unify semantic and acoustic tokens by disentangling different aspects of speech information hierarchically. SpeechTokenizer [] is a unified speech tokenizer for large language models. It combines semantic and acoustic tokens, separating different speech information across RVQ layers. The model is based on RVQ-GANs, similar to EnCodec, and uses a convolutional encoder-decoder network from EnCodec. A two-layer BiLSTM replaces the original two-layer LSTM to improve semantic modeling. A semantic teacher guides the first RVQ quantizer, allowing the first layer tokens to capture content information effectively. With a residual structure, the subsequent quantizers capture the remaining paralinguistic information.
SpeechTokenizer employs HuBERT as the semantic teacher to capture content information. The training procedure maximizes the cosine similarity between the RVQ layer outputs and the semantic teacher representations. HuBERT Layer 9 units represent semantic tokens, while EnCodec codes represent acoustic tokens.

Figure: Figure 2: Time and memory required to process an utterance of 16 seconds for encoders and decoders of the considered audio tokenizers on an NVIDIA GeForce RTX 3070 GPU @ 8 GB.
Refer to caption: x2.png

### 3.2 Downstream Model

In this step, we employ neural networks to solve supervised tasks of common interest. To achieve this, we first assign each discrete token to a corresponding embedding vector through a lookup table. Subsequently, we dynamically combine the embeddings from different codebooks using attention, enabling the model to adjust the importance of codebooks for the specific task of interest. The attention mechanism consists of a simple multi-layer perceptron (MLP) that takes the embeddings of the audio tokens as input. The MLP generates a score for each selected codebook, which is normalized by a softmax function as shown in the following equations:

$$ $\displaystyle z_{c,t}=f\big{(}\text{emb}(d_{c,t})\big{)},\;a_{c,t}=\frac{\exp( z_{c,t})}{\sum_{k=1}^{C}\exp(z_{k,t})},\;h_{t}=\sum_{c}a_{c,t}z_{c,t},$ (2) $$

where $d_{c,t}$ is the discrete token obtained from codebook $c$, at time $t$ and, $z_{c,t}$ represents the score assigned to codebook $c$
at time $t$ by the MLP function $f$. The function $\text{emb}(\cdot)$ refers to the lookup table that assigns embeddings to each discrete token. The variable $a_{c,t}$ denotes the attention assigned to the codebook $c$ at time $t$, and lastly $h_{t}$ is the representation that is fed to the downstream MLP model.
We would like to note that the MLP learns different codebook combinations $h_{t}$ at each time step $t$ depending on the attention weights $a_{c,t}$, enabling the model to extract the necessary information when required. We also considered summing all embeddings, as has been done in previous literature [, ], but found that attention weights performed slightly better.

The combined representations $h_{t}$ are fed into neural models designed for different tasks.
The downstream models are jointly trained with their attention and embedding layers in an end-to-end fashion.
For discriminative tasks, the model outputs either a single prediction (e.g., for emotion recognition) or a sequence of predictions (e.g., for speech recognition).
For generative tasks (e.g., speech enhancement), the neural network outputs the targeted tokens per codebook, with the output shape being $C\times L$, where $L$ is the sequence length. The predicted audio tokens are converted into an audio waveform via an audio decoder, as explained in the next section.

### 3.3 Discrete Audio Decoder

The decoder, used for generative tasks only, converts the predicted tokens into audio signals. The decoder is frozen during training.
The choice of decoder depends on the encoder used in the first step. For compression and hybrid tokenizers, we use their built-in decoder. For semantic tokens, we use the scalable vocoder proposed in [], which is a modified HiFi-GAN [] pretrained with LibriSpeech-960h []. The scalable vocoder accepts a variable number of multi-layer semantic tokens as input and can handle different bitrates using a layer dropout mechanism.

## 4 Experiments

In the following sections, we describe the discriminative and generative tasks considered in our experiments. For a more reliable evaluation, we tested two downstream architectures for each task.
For detailed information about the hyperparameters used in each experiment, refer to Appendix
F.

### 4.1 Discriminative Tasks

For the downstream architectures and training procedures, we follow the best-performing approaches for classic continuous self-supervised representations proposed in  [].

- •
Automatic Speech Recognition (ASR): The goal of ASR is converting speech signals into written text. We address two ASR tasks. The first task involves English ASR using the popular LibriSpeech dataset []. Training and validation are performed on the train-clean-100 and dev-clean subsets, respectively, while testing is conducted on the test-clean and test-other subsets.
The downstream architecture for this task consists of two layers of Bidirectional Long Short-Term Memory (BiLSTM) followed by a linear layer for mapping audio to characters. The second architecture utilizes ContextNet [] with unitary strides to maintain the frame rate of the encoder models.
Additionally, we explore low-resource languages, specifically Welsh (Cymraeg) and Basque (Euskera) datasets extracted from CommonVoice 17.0 []. Here, we evaluate the performance using both the BiLSTM architecture and a two-layer dense neural network mapping frame representations to character probabilities. We use the Word Error Rate (WER) as the error metric for all ASR tasks.
- •
Speaker Identification/Verification (SID, SV): Speaker Identification involves classifying each utterance by its speaker identity as a multi-class classification, with the same predefined set of speakers for both training and testing. The evaluation metric is the accuracy. Automatic Speaker Verification (ASV), instead, involves training a binary classifier to determine whether the speakers in a pair of utterances are the same. The evaluation metric adopted in this case is the equal error rate (EER). We use the widely-used VoxCeleb1 [] train and test splits for both tasks. First, we test the X-vector [] architecture with AM-Softmax [] loss for training the speaker embeddings. For verification, we use the cosine similarity between speaker representations. As a second architecture, we replace the X-vectors with an ECAPA-TDNN neural network [].
- •
Emotion Recognition (ER): The task involves predicting one of the four classes: happy, sad, angry, and neutral. We use the popular IEMOCAP [] dataset, which contains about 10k samples from 10 speakers. As a first architecture, we directly input the representations into a linear classification layer after averaging them along the time axis. For the second downstream architecture, we use ECAPA-TDNN. The evaluation metric is the accuracy.
- •
Intent Classification (IC): This task aims to determine the intention or purpose given utterance a speech recording. In particular, we here aim to classify each utterance into one of 18 scenarios, including calendar, email, and alarm. For this task, we utilize the SLURP dataset [], which comprises around 72k audio recordings of single-turn user interactions with a home assistant.
We employ ECAPA-TDNN and a two-layer BiLSTM (followed by a linear classifier) as downstream architectures. We evaluate the performance using the accuracy.
- •
Keyword Spotting (KS): Keyword Spotting involves detecting predefined keywords by classifying utterances into a set of specified words. We use the Speech Commands dataset v1.0 [] for this task, as done in SUPERB. The dataset includes ten classes of keywords, a class for silence, and an unknown class to account for false positives. We employ both the X-vector and ECAPA-TDNN architectures. The evaluation metric is the accuracy.

### 4.2 Generative Tasks

- •
Speech Enhancement (SE):
Speech enhancement aims to improve audio quality by cleaning up noisy input recordings.
For this task, we utilize the popular VoiceBank dataset [].
We employ two downstream architectures: a non-autoregressive Conformer encoder [], and a convolutional recurrent deep neural network (CRDNN).
The input tokens are extracted from the noisy signal, while target tokens from the clean one. Training is performed using the cross-entropy loss.
The speech quality is assessed using the deep noise suppression mean opinion score (DNSMOS) []. The intelligibility is evaluated through the differential word error rate (dWER) [], which measures the WER between the transcribed enhanced signal and the transcribed target signal. The transcriptions are obtained using the small version of Whisper []. Additionally, to measure speaker fidelity, we use the cosine similarity (SpkSim) between X-vectors extracted from the enhanced signal and the target signal using the base variant of WavLM [] fine-tuned for speaker verification.
- •
Speech Separation (SS): Speech separation aims to isolate individual voices from an audio recording containing multiple speakers. For this task, we use the Libri2Mix dataset [], which contains mixtures of two overlapping speakers. We employ two downstream architectures: a non-autoregressive Conformer encoder [], and a convolutional recurrent deep neural network (CRDNN).
The input tokens are extracted from the mixture, while target tokens from each of the two sources. Training is performed using the cross-entropy loss with permutation invariant training [].
To measure performance, we employ the same metrics as speech enhancement.
- •
Text-to-Speech (TTS): The task involves generating raw speech audio from a given text input. For downstream architectures, we consider both a small and a large autoregressive Transformer []. We train all models on the LJSpeech dataset []. To assess the speech quality, we use a pretrained UTokyo-SaruLab System (UTMOS) [], which is specifically designed for TTS and trained to predict human Mean Opinion scores. To measure pronunciation accuracy, we use the dWER metric. This involves comparing the transcriptions provided by a speech recognizer for the synthesized speech sample with the transcriptions from the ground truth. As for enhancement and separation, we considered the small version of Whisper [].

**Table 2: Benchmarking results for discriminative tasks.**
| Models/Tasks | ASR-En | ASR-multiling | ER | IC | KS | SI | SV |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WER $\downarrow$ | WER $\downarrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | EER $\downarrow$ |  |  |  |
| Clean | Other | Welsh | Basque |  |  |  |  |  |  |
| Low Bitrate |  |  |  |  |  |  |  |  |  |
| Discrete Hubert | 8.99 | 21.14 | 58.50 | 26.83 | 57.20 | 68.70 | 90.54 | 0.90 | 24.99 |
| Discrete WavLM | 11.72 | 27.56 | 60.37 | 28.63 | 59.80 | 73.40 | 97.94 | 0.70 | 26.02 |
| Discrete Wav2Vec2 | 12.14 | 28.65 | 66.30 | 32.25 | 57.80 | 74.10 | 96.16 | 0.40 | 33.53 |
| EnCodec | 52.37 | 77.04 | 92.01 | 58.20 | 44.70 | 31.50 | 86.00 | 58.30 | 17.40 |
| DAC | 63.96 | 83.61 | 94.86 | 66.29 | 49.20 | 22.10 | 81.00 | 45.10 | 20.62 |
| SpeechTokenizer | 19.77 | 43.12 | 76.67 | 47.92 | 49.10 | 57.90 | 95.09 | 47.40 | 20.41 |
| Medium Bitrate |  |  |  |  |  |  |  |  |  |
| Discrete Hubert | 7.91 | 18.95 | 54.77 | 23.63 | 62.10 | 70.50 | 94.69 | 67.40 | 15.71 |
| Discrete WavLM | 8.52 | 20.35 | 54.22 | 22.06 | 57.60 | 78.00 | 98.09 | 80.80 | 8.00 |
| Discrete Wav2Vec2 | 8.76 | 21.32 | 60.39 | 26.64 | 59.10 | 75.10 | 96.64 | 65.47 | 17.64 |
| EnCodec | 46.80 | 74.24 | 91.23 | 47.95 | 51.30 | 31.40 | 88.70 | 91.90 | 7.81 |
| DAC | 59.54 | 81.48 | 97.43 | 56.16 | 45.80 | 18.90 | 76.60 | 83.80 | 11.78 |
| SpeechTokenizer | 18.32 | 41.21 | 75.17 | 38.94 | 52.10 | 57.80 | 94.86 | 91.40 | 7.88 |
| High Bitrate |  |  |  |  |  |  |  |  |  |
| EnCodec | 45.18 | 72.56 | 93.40 | 87.65 | 46.40 | 19.60 | 83.60 | 92.81 | 7.18 |
| DAC | 99.53 | 99.38 | 99.40 | 99.68 | 46.00 | 15.70 | 75.20 | 85.61 | 10.89 |
| Continuous Baseline |  |  |  |  |  |  |  |  |  |
| SSL | 3.370 | 7.04 | 41.77 | 14.32 | 63.10 | 86.10 | 99.00 | 99.70 | 2.10 |

**Table 3: Benchmarking results for generative tasks. N.C. indicates “Not Converged".**
| Models/Tasks | SE | SS | TTS |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$ | DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$ | UTMOS $\uparrow$ | dWER $\downarrow$ |  |
| Low Bitrate |  |  |  |  |  |  |  |  |
| Discrete HuBERT | 3.33 | 15.47 | 0.824 | 3.52 | 80.86 | 0.840 | 3.24 | 2.55 |
| Discrete WavLM | 3.26 | 16.52 | 0.830 | 3.43 | 62.34 | 0.847 | 3.84 | 3.01 |
| Discrete Wav2Vec2 | 3.55 | 18.86 | 0.779 | 3.75 | 96.70 | 0.787 | 3.32 | 3.45 |
| EnCodec | 3.15 | 34.35 | 0.852 | 3.11 | 83.55 | 0.877 | 1.46 | 8.85 |
| DAC | 3.30 | 57.41 | 0.853 | 3.01 | 102.00 | 0.854 | 1.97 | 10.68 |
| SpeechTokenizer | 3.18 | 30.13 | 0.858 | 3.13 | 85.25 | 0.874 | 2.51 | 3.69 |
| Medium Bitrate |  |  |  |  |  |  |  |  |
| Discrete HuBERT | 3.48 | 12.62 | 0.875 | 3.70 | 66.29 | 0.891 | 3.80 | 3.40 |
| Discrete WavLM | 3.48 | 10.18 | 0.889 | 3.68 | 34.03 | 0.912 | 3.82 | 2.45 |
| Discrete Wav2Vec2 | 3.54 | 17.60 | 0.858 | 3.75 | 78.42 | 0.866 | 3.68 | 2.89 |
| EnCodec | 3.10 | 19.07 | 0.885 | 3.09 | 48.57 | 0.906 | 1.50 | 94.6 |
| DAC | 3.49 | 31.14 | 0.906 | 3.26 | 55.43 | 0.924 | 1.71 | 71.26 |
| SpeechTokenizer | 3.49 | 23.44 | 0.876 | 3.42 | 60.75 | 0.906 | 1.96 | 53.26 |
| High Bitrate |  |  |  |  |  |  |  |  |
| EnCodec | 2.87 | 68.22 | 0.814 | 2.95 | 97.73 | 0.839 | N.C | N.C |
| DAC | 2.95 | 46.07 | 0.860 | 2.53 | 208 | 0.784 | N.C | N.C |
| Continuous Baseline |  |  |  |  |  |  |  |  |
| SSL | 3.49 | 4.92 | 0.928 | 3.68 | 9.97 | 0.939 | 3.71 | 2.94 |

## 5 Results

### 5.1 Comparison of Discrete Audio Models

Tables 2 and 3 show the performance of discriminative and generative tasks among the two downstream architectures explored. For each value in the table, we report the best performance obtained with the two downstream architectures.
Detailed results for each architecture, along with the settings for each experiment using continuous SSL models, are provided in Appendix E.

We observe a significant variation in the tokenizer performance across different tasks. This result suggests that the optimal choice of tokenizer depends on the specific task at hand. However, some interesting patterns emerge. For instance, semantic tokens significantly outperform compression tokens for most discriminative tasks. This trend is due to the ability of semantic tokens to capture high-level information from the audio signal as also observed in existing findings in the literature []. The only exceptions are speaker recognition tasks, where EnCodec achieves the best results. This suggests that compression tokens better encode speaker information. It is consistent with a previous study [] that shows discrete tokens obtained from quantization of SSL layers remove speaker information.

Semantic tokens show the best performance for generative tasks as well, achieving the highest MOS and dWER scores, indicating better overall quality and intelligibility in the generated outputs. However, for preserving speaker identity, compression tokens are more effective, as shown by superior speaker similarity (SpkSim) metrics. We think our findings for generative tasks are particularly interesting. While prior research efforts [, ] explored the use of semantic tokens for generation, they did not include a comparison with the performance of compression tokens. It is important to remark that the success observed with both semantic tokens relies heavily on the effectiveness of the decoder architecture used in our benchmark. Our scalable decoder minimizes distortions and artifacts in the generated speech, leading to better performance on various generative tasks.

In Table  4 (right), we present the ranking aggregation for the considered tokenizers (medium bitrate). Each model is individually ranked for every task, and we compute the average position across all ranks. This analysis shows that discrete WavLM generates the top-performing audio tokens. While the continuous version of WavLM ranks highest in the SUPERB benchmark, our findings demonstrate for the first time that this model maintains strong performance even after tokenization.

Our comparison between discrete tokens and the best continuous baseline reveals a significant performance gap favoring continuous representations. This suggests that tokenization loses valuable information, such as phonetics, speaker identity, and emotion. Addressing this information loss is a key challenge for future generations of audio tokens.

### 5.2 Impact of Bitrate

We also study the impact of different bitrates on the performance of the tokenizers. Tables 2 and 3 show that a medium bitrate achieves the best results for both discriminative and generative tasks. Interestingly, higher bitrates, when available (e.g., for EnCodec and DAC), tend to degrade performance. While higher bitrates can potentially preserve more information, they also increase the output dimensionality of the model, making the task more challenging to solve.
In some cases, we found the task so challenging with high bitrates that the model did not converge, as observed in the case of TTS.
It is worth noting that semantic tokens have a lower bitrate than compression tokens, as shown in Table 1. For example, in the medium bitrate range, discrete WavLM has a bitrate of 2.9 kbps, while EnCodec has 6.0 kbps. This difference is due to the varying number of codebooks (6 vs. 8) and sampling rates (16 kHz vs. 24 kHz). Despite their lower bitrate, semantic tokens provide better performance.

Another aspect we investigate is the efficiency of the encoders and decoders, as this could impact some applications. Figure 2 shows the time and memory usage for each encoder-decoder pair across all bitrate ranges.
For semantic tokens, the encoder is a large neural network and is computationally demanding. In contrast, the decoders are based on a compact HiFi-GAN model and are very efficient. For streaming tasks [] where time and memory are critical, EnCodec turned out to be a better candidate due to the efficiency of both its encoder and decoder models.

**Table 4: (left) Evaluating various discrete decoders on the speech re-synthesis task (medium bitrate). (right) Ranking aggregation for models (medium bitrate).**
| Models/Metrics | DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$ |
| --- | --- | --- | --- |
| Discrete HuBERT | 3.68 | 6.60 | 0.92 |
| Discrete WavLM | 3.64 | 5.19 | 0.94 |
| Discrete Wav2Vec2 | 3.71 | 8.72 | 0.91 |
| EnCodec | 3.54 | 2.16 | 0.98 |
| DAC | 3.74 | 2.36 | 0.99 |
| SpeechTokenizer | 3.58 | 5.12 | 0.94 |
| Continuous SSL | 3.73 | 2.33 | 0.98 |

### 5.3 Analysis of Discrete Audio Decoder

Finally, we present a comparative evaluation of the decoders in Table 4 (left).
The decoder evaluation is conducted on the LibriSpeech test-clean subset using a speech re-synthesis task,
where we extract the tokens from each discrete audio encoder and reconstruct the speech using the associated decoders. Then, we evaluate the reconstructed speech based on speaker similarity, Mean Opinion Score (MOS), and differential Word Error Rate (dWER). The goal of this experiment is to establish if a given system is able to provide a high-fidelity reconstruction of the input audio after encoding it in the discrete space. This is especially important to establish for generative tasks.

The results show that the built-in decoders of compression tokens outperform other models in preserving speaker similarity, further confirming that current semantic tokens do not adequately preserve speaker information. Compression-based tokens also achieve better dWER scores. However, in terms of speech quality (assessed with DNSMOS), there are no significant differences between semantic and compression-based tokens.
This trend indicates that while semantic tokens produce good-quality audio, they may be slightly more prone to semantic degradation (e.g., mispronunciations of words or phonemes). As expected, continuous baselines perform better than their discrete counterparts. Additional analysis on low and high settings can be found in Appendix D.

## 6 Conclusion

This paper introduces DASB, a comprehensive benchmark designed to evaluate the performance of discrete audio tokens across diverse tasks of broad interest. We employ various evaluation metrics, downstream architectures, and bitrates for more robust comparisons. Interestingly, our findings reveal that semantic tokens outperform, on average, compression tokens in both generative and discriminative tasks.
In particular, discrete WavLM emerged as the top-performing model, making it a natural candidate for adoption in multi-modal text+audio LLMs.
A significant performance gap, however, persists when compared to traditional self-supervised continuous representations.
This highlights the need for further research, which we believe is essential for better incorporating audio tokens into large multimodal language models.

One limitation we encountered is the proprietary nature of some audio tokenizers, such as Soundstream zeghidour2021soundstream [], which are not publicly accessible. Additionally, the benchmark is currently limited to speech tasks, but we plan to broaden it including music and sound processing. Our goal is to help the research community establish a shared benchmark and evaluation protocol for discrete audio representations. We will thus keep expanding DASB by continuously incorporating novel tokenizers and tasks.

## 7 Acknowledgements

We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Digital Research Alliance of Canada (alliancecan.ca). We thank OVHCloud for donating part of the GPU computing resources needed for this work.

## Appendix A General Information

### A.1 Computational Resources

We designed our benchmark to be computationally accessible. Every task runs on GPUs with 32 GB or more of VRAM. Tasks like keyword spotting takes only 8 hours, while speech recognition (e.g, ASR-Basque) might require about 30 hours on a single NVIDIA V100 GPU.

### A.2 Impact

We believe DASB can have a positive impact on the research community. We do not foresee a direct negative societal impact or misuse of our benchmark. However, we acknowledge that DASB can potentially accelerate progress in multi-modal large language models, which, in turn, have a wide range of potential positive and negative uses that society is still working to assess.

### A.3 Hosting and Maintenance Plan

DASB platform is hosted and version-tracked via GitHub.
It is available at [https://github.com/speechbrain/benchmarks](https://github.com/speechbrain/benchmarks).
DASB is a community-driven and open-source initiative.
We plan to extend it by running additional experiments and including new audio tokenizers and tasks. We welcome external contributors.

### A.4 Licensing

Our work is licensed under Apache 2.0 ([https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)).

**Table 5: Licenses for the models used in our benchmark.**
| Model | License |
| --- | --- |
| HuBERT-large | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| WavLM-large | [ATTRIBUTION-SHAREALIKE 2.0](https://creativecommons.org/licenses/by-sa/2.0/) |
| Wav2vec2-large | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |
| EnCodec | [MIT license](https://opensource.org/license/mit) |
| DAC | [MIT license](https://opensource.org/license/mit) |
| SpeechTokenizer | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) |

### A.5 Author Statement

We, the authors, will bear all responsibility in case of violation of rights.

## Appendix B Discrete Audio Models Details

**Table 6: Features of the Considered Discrete Audio Encoders.**
| Model | Dataset | Repo |
| --- | --- | --- |
| Discrete Hubert [] | LibriSpeech960[] | [huggingface.co/poonehmousavi/SSL_Quantization](https://huggingface.co/poonehmousavi/SSL_Quantization) |
| Discrete WavLM [] | LibriSpeech960[] | [huggingface.co/poonehmousavi/SSL_Quantization](https://huggingface.co/poonehmousavi/SSL_Quantization) |
| Discrete Wav2Vec2 [] | LibriSpeech960[] | [huggingface.co/poonehmousavi/SSL_Quantization](https://huggingface.co/poonehmousavi/SSL_Quantization) |
| EnCodec [] | DNS [], CommonVoice [], AudioSet [], |  |
|  | [github.com/facebookresearch/encodec](https://github.com/facebookresearch/encodec) |  |
| DAC [] | DAPS[], DNS [], CommonVoice [], |  |
|  | [github.com/descriptinc/descript-audio-codec](https://github.com/descriptinc/descript-audio-codec) |  |
| SpeechTokenizer [] | LibriSpeech960[] | [github.com/ZhangXInFD/SpeechTokenizer](https://github.com/ZhangXInFD/SpeechTokenizer) |

## Appendix C Dataset and Downstream Models

Table 7 provides a summary of the datasets and the two downstream architectures used for each task.

**Table 7: Dataset and Downstream Models**
| Dataset | Task | 1st Architecture | 2nd Architecture | Dataset Link |
| --- | --- | --- | --- | --- |
| LibriSpeech [] | Speech Recognition | BiLSTM | ContextNet | [openslr.org/12](openslr.org/12) |
| CommonVoice 17.0 [] | Speech Recognition | BiLSTM | Linear | [commonvoice.mozilla.org/en/datasets](https://commonvoice.mozilla.org/en/datasets) |
| VoxCeleb1 [] | speaker verification/identification | ECAPA-TDNN | X-Vectors | [robots.ox.ac.uk/ vgg/data/voxceleb/vox1.html](https://www.robots.ox.ac.uk/%C2%A0vgg/data/voxceleb/vox1.html) |
| IEMOCAP [] | Emotion Recognition | ECAPA-TDNN | Time-Pooling + Linear | [sail.usc.edu/iemocap/](https://sail.usc.edu/iemocap/) |
| Speech Commands [] | Keyword Spotting | X -Vectors | ECAPA-TDNN | [tensorflow.org/datasets/catalog/speech_commands](https://www.tensorflow.org/datasets/catalog/speech_commands) |
| SLURP [] | Intent Classification | BiLSTM + Linear | Time-Pooling + Linear | [zenodo.org/record/4274930](https://zenodo.org/record/4274930) |
| VoiceBank [] | Speech Enhancement | Conformer | CRDNN | [datashare.ed.ac.uk/handle/10283/2791](https://datashare.ed.ac.uk/handle/10283/2791) |
| Libri2Mix [] | Speech Separation | Conformer | CRDNN | [github.com/JorisCos/LibriMix](https://github.com/JorisCos/LibriMix) |
| LJSpeech [] | Text-to-Speech | Shallow Transformer | Deep Transformer | [keithito.com/LJ-Speech-Dataset/](https://keithito.com/LJ-Speech-Dataset/) |

## Appendix D Additional Analysis of Discrete Audio Decoders

In this section, we expand on the results from Section 5.3 by including low and high bitrates in addition to the medium bitrate. Additionally, for speaker similarity, we measure the cosine similarity between X-vectors extracted from the reconstructed and target signals using two different models: WavLM (SpkSim WavLM) and ECAPA-TDNN (SpkSim ECAPA), both fine-tuned for speaker verification. When analyzing low and high bitrate settings, distinct trends emerge compared to the medium bitrate. At low bitrates, the models perform worse in terms of speaker similarity, MOS, and dWER, as expected. This is particularly pronounced for compression-based decoders, where the degradation is more significant in terms of dWER. In contrast, at high bitrate, there is an overall small improvement in all metrics. In particular, the DAC model consistently outperforms EnCodec across all evaluated metrics, even for high bitrate settings.

The analysis shows that bitrate significantly impacts the performance of discrete decoders. Higher bitrates better preserve speech characteristics and result in lower error rates, while lower bitrates degrade these aspects. This highlights the trade-off between bitrate and speech synthesis quality.

**Table 8: Evaluation of various discrete decoders for the speech re-synthesis task at low, medium, and high bitrates.**
| Models/Metrics | SpkSim ECAPA $\uparrow$ | SpkSim WavLM $\uparrow$ | DNSMOS $\uparrow$ | dWER $\downarrow$ |
| --- | --- | --- | --- | --- |
| Low Bitrate |  |  |  |  |
| Discrete Hubert | 0.34 | 0.87 | 3.51 | 8.25 |
| Discrete WavLM | 0.34 | 0.88 | 3.45 | 7.01 |
| Discrete Wav2Vec2 | 0.28 | 0.82 | 3.73 | 10.58 |
| EnCodec | 0.52 | 0.92 | 3.20 | 11.71 |
| DAC | 0.54 | 0.91 | 3.42 | 14.02 |
| SpeechTokenizer | 0.22 | 0.72 | 3.21 | 45.7 |
| Medium Bitrate |  |  |  |  |
| Discrete Hubert | 0.47 | 0.92 | 3.68 | 6.60 |
| Discrete WavLM | 0.53 | 0.94 | 3.64 | 5.19 |
| Discrete Wav2Vec2 | 0.43 | 0.91 | 3.71 | 8.72 |
| EnCodec | 0.87 | 0.98 | 3.54 | 2.16 |
| DAC | 0.87 | 0.99 | 3.74 | 2.36 |
| SpeechTokenizer | 0.65 | 0.94 | 3.58 | 5.12 |
| High Bitrate |  |  |  |  |
| EnCodec | 0.94 | 0.99 | 3.69 | 1.47 |
| DAC | 0.98 | 100 | 3.79 | 0.73 |

## Appendix E Additional Results

Tables 9 - 12 show the results obtained with 2 different downstream architectures. Note that table 7 indicates the first and second architectures explored for each task. For the continuous baseline, we follow the same architecture as the discrete experiments except for TTS. For the TTS continuous baseline, we use a modified Tacotron2 [] architecture enhanced with guided attention [] that predicts SSL representations instead of Mel spectrograms.

Varying the architecture of the downstream decoder leads to significant variations in task performance. For ASR tasks, BiLSTM performs better. For classification tasks, ECAPA-TDNN shows the best performance, except for keyword spotting where X-vector is slightly better. For speech enhancement and separation, Conformer shows the best performance. For TTS, a notable pattern is observed: semantic tokens yield the best results with shallow models, while acoustic and hybrid tokens perform better with deeper models but still underperform compared to semantic tokens. One reason might be that discrete SSL models retain higher-level features closer to phonetic transcriptions, requiring lower-capacity models to capture the relationship between raw text and such representations. Higher-capacity models can lead to slower training and potential overfitting. In contrast, shallow models appear to underfit acoustic tokens, resulting in high dWERs and speech-like sounds with only surface resemblance to the original sentence, rather than intelligible speech.

**Table 9: Results for discriminative tasks with the first downstream architecture.**
| Models/Tasks | ASR-En | ASR-multiling | ER | IC | KS | SI | SV |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WER $\downarrow$ | WER $\downarrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | EER $\downarrow$ |  |  |  |
| Clean | Other | Welsh | Basque |  |  |  |  |  |  |
| Low Bitrate |  |  |  |  |  |  |  |  |  |
| Discrete Hubert | 8.99 | 21.14 | 58.50 | 26.83 | 57.20 | 68.70 | 90.54 | 0.90 | 24.99 |
| Discrete WavLM | 11.72 | 27.56 | 60.37 | 28.63 | 59.80 | 73.40 | 97.94 | 0.70 | 26.02 |
| Discrete Wav2Vec2 | 12.14 | 28.65 | 66.30 | 32.25 | 57.80 | 74.10 | 96.16 | 0.40 | 33.53 |
| EnCodec | 52.37 | 77.04 | 92.01 | 58.20 | 44.70 | 31.50 | 86.00 | 58.30 | 17.40 |
| DAC | 63.96 | 83.61 | 94.86 | 66.29 | 49.20 | 22.10 | 81.00 | 1.10 | 29.99 |
| SpeechTokenizer | 19.77 | 43.12 | 76.67 | 47.92 | 49.10 | 57.90 | 95.09 | 47.40 | 20.41 |
| Medium Bitrate |  |  |  |  |  |  |  |  |  |
| Discrete Hubert | 7.91 | 18.95 | 54.77 | 23.63 | 62.10 | 70.50 | 94.69 | 67.40 | 15.71 |
| Discrete WavLM | 8.52 | 20.35 | 54.22 | 22.06 | 57.60 | 78.00 | 98.09 | 80.80 | 8.00 |
| Discrete Wav2Vec2 | 8.76 | 21.32 | 60.39 | 26.64 | 59.10 | 75.10 | 96.64 | 65.47 | 17.64 |
| EnCodec | 46.80 | 74.24 | 91.23 | 47.95 | 51.30 | 31.40 | 88.70 | 91.90 | 7.81 |
| DAC | 59.54 | 81.48 | 97.43 | 56.16 | 45.80 | 18.90 | 76.60 | 83.80 | 11.78 |
| SpeechTokenizer | 18.32 | 41.21 | 75.17 | 38.94 | 52.10 | 57.80 | 94.86 | 91.40 | 7.88 |
| High Bitrate |  |  |  |  |  |  |  |  |  |
| EnCodec | 45.18 | 72.56 | 93.40 | 87.65 | 46.40 | 19.60 | 83.60 | 92.81 | 7.18 |
| DAC | 99.53 | 99.38 | 99.40 | 99.74 | 46.00 | 15.70 | 75.20 | 85.61 | 10.89 |
| Continuous Baseline |  |  |  |  |  |  |  |  |  |
| SSL | 3.370 | 7.04 | 41.77 | 14.32 | 63.10 | 86.10 | 99.00 | 99.70 | 2.10 |

**Table 10: Results for discriminative tasks with the second downstream architecture.**
| Models/Tasks | ASR-En | ASR-multiling | ER | IC | KS | SI | SV |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| WER $\downarrow$ | WER $\downarrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | ACC $\uparrow$ | EER $\downarrow$ |  |  |  |
| Clean | Other | Welsh | Basque |  |  |  |  |  |  |
| Low Bitrate |  |  |  |  |  |  |  |  |  |
| Discrete Hubert | 11.99 | 23.45 | 97.30 | 99.20 | 61.40 | 57.80 | 90.54 | 15.60 | 18.34 |
| Discrete WavLM | 14.98 | 30.32 | 100.00 | 99.29 | 61.20 | 57.70 | 97.80 | 14.00 | 18.45 |
| Discrete Wav2Vec2 | 15.45 | 35.30 | 99.89 | 98.39 | 59.50 | 60.20 | 96.52 | 5.80 | 23.5830 |
| EnCodec | 90.84 | 94.97 | 99.88 | 100.00 | 40.20 | 18.40 | 88.50 | 34.70 | 23.08 |
| DAC | 125.00 | 119.00 | 100.00 | 100.00 | 47.20 | 17.70 | 83.90 | 21.50 | 27.92 |
| SpeechTokenizer | 26.50 | 47.91 | 100.00 | 99.11 | 50.40 | 49.10 | 94.55 | 24.50 | 25.92 |
| Medium Bitrate |  |  |  |  |  |  |  |  |  |
| Discrete Hubert | 10.91 | 21.65 | 95.81 | 97.20 | 64.80 | 59.10 | 94.38 | 14.80 | 17.67 |
| Discrete WavLM | 11.12 | 22.63 | 96.51 | 97.38 | 60.60 | 63.80 | 97.85 | 25.60 | 15.74 |
| Discrete Wav2Vec2 | 12.48 | 25.40 | 94.34 | 95.76 | 61.90 | 63.00 | 96.92 | 11.70 | 19.07 |
| EnCodec | 124.00 | 119.00 | 99.82 | 99.98 | 41.50 | 18.20 | 89.10 | 30.10 | 21.73 |
| DAC | 124.00 | 122.00 | 99.96 | 100.00 | 46.80 | 16.40 | 80.30 | 15.90 | 29.75 |
| SpeechTokenizer | 118.00 | 117.00 | 97.08 | 96.49 | 56.60 | 47.60 | 89.10 | 32.80 | 20.15 |
| High Bitrate |  |  |  |  |  |  |  |  |  |
| EnCodec | 124.00 | 122.00 | 100.00 | 99.93 | 43.40 | 17.10 | 80.40 | 23.40 | 25.52 |
| DAC | 122.00 | 122.00 | 99.74 | 100.00 | 47.70 | 15.40 | 76.00 | 16.40 | 29.94 |
| Continuous Baseline |  |  |  |  |  |  |  |  |  |
| SSL | 10.05 | 13.80 | 68.72 | 48.60 | 68.60 | 75.20 | 98.70 | 88.40 | 4.31 |

**Table 11: Results for generative tasks with the first downstream architecture. N.C. indicates “Not Converged".**
| Models/Tasks | SE | SS | TTS |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$ | DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$ | UTMOS $\uparrow$ | dWER $\downarrow$ |  |
| Low Bitrate |  |  |  |  |  |  |  |  |
| Discrete Hubert | 3.33 | 15.47 | 0.824 | 3.52 | 80.86 | 0.840 | 3.25 | 2.93 |
| Discrete WavLM | 3.26 | 16.52 | 0.830 | 3.43 | 62.34 | 0.847 | 2.74 | 12.69 |
| Discrete Wav2Vec2 | 3.55 | 18.86 | 0.779 | 3.75 | 96.70 | 0.787 | 3.32 | 3.45 |
| EnCodec | 3.15 | 34.35 | 0.852 | 3.11 | 83.55 | 0.877 | 1.46 | 8.84 |
| DAC | 3.30 | 57.41 | 0.853 | 3.01 | 102.00 | 0.854 | 1.97 | 10.66 |
| SpeechTokenizer | 3.18 | 30.13 | 0.858 | 3.13 | 85.25 | 0.874 | 2.55 | 16.42 |
| Medium Bitrate |  |  |  |  |  |  |  |  |
| Discrete HuBERT | 3.48 | 12.62 | 0.875 | 3.70 | 66.29 | 0.891 | 3.78 | 16.09 |
| Discrete WavLM | 3.48 | 10.18 | 0.889 | 3.68 | 34.03 | 0.912 | 3.77 | 14.76 |
| Discrete Wav2Vec2 | 3.54 | 17.60 | 0.858 | 3.75 | 78.42 | 0.866 | 3.68 | 5.98 |
| EnCodec | 3.10 | 19.07 | 0.885 | 3.09 | 48.57 | 0.906 | 1.43 | 92.4 |
| DAC | 3.49 | 31.14 | 0.906 | 3.26 | 55.43 | 0.924 | 1.71 | 71.25 |
| SpeechTokenizer | 3.49 | 23.44 | 0.876 | 3.42 | 60.75 | 0.906 | 1.97 | 53.26 |
| High Bitrate |  |  |  |  |  |  |  |  |
| EnCodec | 2.87 | 68.22 | 0.814 | 2.95 | 97.73 | 0.839 | N.C | N.C |
| DAC | 2.95 | 46.07 | 0.860 | 2.53 | 208 | 0.784 | N.C | N.C |
| Continuous Baseline |  |  |  |  |  |  |  |  |
| SSL | 3.49 | 4.92 | 0.928 | 3.68 | 9.97 | 0.939 | 2.86 | 4.687 |

**Table 12: Results for generative tasks with the second downstream architecture. N.C. indicates “Not Converged".**
| Models/Tasks | SE | SS | TTS |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$ | DNSMOS $\uparrow$ | dWER $\downarrow$ | SpkSim $\uparrow$ | UTMOS $\uparrow$ | dWER $\downarrow$ |  |
| Low Bitrate |  |  |  |  |  |  |  |  |
| Discrete HuBERT | 3.31 | 13.98 | 0.821 | 3.52 | 97.58 | 0.817 | 3.24 | 2.55 |
| Discrete WavLM | 3.27 | 16.50 | 0.825 | 3.44 | 60.12 | 0.834 | 3.84 | 3.01 |
| Discrete Wav2Vec2 | 3.55 | 17.17 | 0.777 | 3.74 | 95.20 | 0.785 | 3.32 | 3.45 |
| EnCodec | 3.16 | 45.07 | 0.851 | 2.32 | 99.17 | 0.495 | 1.40 | 53.50 |
| DAC | 3.31 | 59.28 | 0.863 | 2.88 | 138 | 0.804 | 1.81 | 19.77 |
| SpeechTokenizer | 3.32 | 27.54 | 0.860 | 3.14 | 90.68 | 0.846 | 2.51 | 3.69 |
| Medium Bitrate |  |  |  |  |  |  |  |  |
| Discrete HuBERT | 3.48 | 13.71 | 0.857 | 3.72 | 91.63 | 0.843 | 3.80 | 3.40 |
| Discrete WavLM | 3.47 | 9.63 | 0.878 | 3.68 | 37.53 | 0.902 | 3.82 | 2.45 |
| Discrete Wav2Vec2 | 3.53 | 16.58 | 0.853 | 3.74 | 83.86 | 0.831 | 3.68 | 2.89 |
| EnCodec | 3.04 | 84.78 | 0.796 | 3.06 | 70.25 | 0.882 | 1.50 | 94.62 |
| DAC | 3.35 | 71.07 | 0.831 | 3.12 | 95.11 | 0.872 | 1.47 | 77.00 |
| SpeechTokenizer | 2.12 | 99.62 | 0.549 | 3.10 | 89.78 | 0.862 | 1.85 | 64.26 |
| High Bitrate |  |  |  |  |  |  |  |  |
| EnCodec | 2.93 | 95.06 | 0.780 | 2.96 | 157 | 0.768 | N.C | N.C |
| DAC | 2.22 | 99.62 | 0.571 | 2.31 | 234 | 0.787 | N.C | N.C |
| Continuous Baseline |  |  |  |  |  |  |  |  |
| SSL | 3.42 | 6.05 | 0.861 | 3.43 | 24.92 | 0.873 | 2.86 | 4.687 |

## Appendix F Hyperparameters

We maintain the embedding dimension at 1024 for consistency across all experiments. Training continues until convergence or until the validation loss stops improving. For detailed settings for each experiment, please visit the [GitHub repository](https://github.com/speechbrain/benchmarks/tree/DASB/benchmarks/DASB).

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - In addition to the GitHub repository, do the authors plan to provide a leaderboard for tracking and comparing results?
- Would it be possible to include a comparison of computational costs as well?
- Do the authors have plans to add support for higher sample rates in future comparisons?
- Since a codec might perform well on one downstream task but poorly on another, what approach would you recommend to achieve balanced results for selecting a universally effective codec?
soundness: 3
strengths: - This is an open-source project dedicated to enhancing the quality of neural audio codecs, which are becoming increasingly valuable to the community as their importance grows.
- It provides an exciting opportunity to benchmark performance across a variety of downstream tasks—not only for speech but also for broader audio and music applications
summary: - The authors offer a benchmark for evaluating the quality of neural audio codecs and assessing their impact on various downstream tasks.
- These downstream tasks encompass several standard audio processing applications, including speech, sound, and music.
weaknesses: - I've noticed a surge in similar open-source projects released recently—some of which were published after the ICLR submission deadline, highlighting the relevance of this work. However, I have some concerns regarding its future adoption. To encourage broader usage, it might be beneficial to emphasize the unique advantages of this repository and clarify why researchers would prefer it over others. For instance, a comparison table could be added to showcase its strengths in contrast to similar projects.

Examples of recent related works include: [1] Wu, Haibin, et al. "Codec-SUPERB@ SLT 2024: A lightweight benchmark for neural audio codec models." arXiv preprint arXiv:2409.14085 (2024). [2] Shi, Jiatong, et al. "ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech." arXiv preprint arXiv:2409.15897 (2024).

- Inference time may be quite lengthy, potentially taking up to 8 hours for tasks like ASR. Providing a smaller development set for quick experiments could be a valuable addition.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. What is the continuous baseline? it doesn't seem clear. 
2. Why are the SI and SV performance of SSL baseline best? It feels a little weird because they are semantic features.
soundness: 2
strengths: 1. The paper conducted plenty of experiments to evaluate the understanding and generation tasks of different types of speech tokens. 
2. The presentation is clear and the results are easy to follow.
summary: This article provides a benchmark of discrete tokens of speech and audio on comprehension and generation tasks. This paper divides tokens into three categories, namely semantic, compression, and hybrid. Through a large number of experiments, the paper gives some conclusions.
weaknesses: 1. Training data of different types of tokens is different, in addition, the decoders(Vocoders) of semantic tokens are only trained on Librispeech 960. As a result, the horizontal comparison is not particularly fair, especially for music and sound. 
2. The amount of experiments in the paper is large, but there is no obvious insight. Most of the conclusions are already common sense, such as semantic tokens are good at content and compression tokens are good at paralinguistics. 
3. The experimental design and data of the paper are basically the same as SUPERB, Codec-SUPERB, and SUPERB-SG, with limited novelty.
4. I don't understand why in the author's experiment, the high bit rate is not as good as the medium one. I suspect there is some kind of implementation or hyperparameter setting problem.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In the abstract, the authors use the terms "semantic" and "compression" tokens, whose definitions do not appear until Section 3.1. Therefore, I was a bit confused when reading the abstract (though I could still guess to some extent). It would be helpful if the authors could elaborate more on this.

2. I believe there have been several studies on hybrid tokens this year, but only SpeechTokenizer is included. This type of token is a very active area of research. Is there a chance of providing more results in this category?

3. At the beginning of Section 4, there is an error in the Appendix index. Please fix it.

4. Which continuous SSL models are used exactly in Tables 2, 3, and 4?

5. In ASR, the authors include two low-resource languages. I wonder if there are any specific reasons for choosing them. The evaluation results on these two languages are very poor across the models, and I think that instead of directly comparing them at this level, using phoneme recognition could lower the difficulty of the task and provide more comparable results.

6. For speech generation tasks, the authors use automatic MOS prediction models to evaluate the quality of generated speech. I understand that this is mainly because MOS involves human effort, and it is infeasible to use only the original MOS in a benchmark. However, as far as I know, the correlation between automatic MOS models and human ratings is still limited. Could you provide the correlation of these results with human ratings?

7. In Tables 5 and 6, the discrete SSL parts are all replaced with CNN14 and MERT. Is there a chance to add results on the discrete HuBERT and WavLM? This could help test the generalizability from speech to audio and music.
soundness: 3
strengths: This paper presents a benchmark for evaluating the direct use of discrete speech and audio units. The authors have constructed a unified pipeline along with open-source implementations, making it convenient for researchers to conduct various experiments using the benchmark. I anticipate that this benchmark will enable fair comparisons in future research on discrete speech and audio units. The evaluation results appear mostly reasonable. The presentation is clear and easy to read.
summary: This paper presents DASB, a benchmark for evaluating discrete speech and audio units. DASB consists of several discriminative and generative tasks, including speech recognition, emotion recognition, speech enhancement, text-to-speech synthesis, etc. It provides a unified pipeline for leveraging different discrete units to perform these tasks. Preliminary evaluation results indicate that units derived from self-supervised learning models generally outperform codec units and that continuous representations significantly outperform discrete units.
weaknesses: Although the authors present DASB as an audio and speech benchmark that includes both speech and audio tasks for evaluation, the number of audio tasks is rather limited, and there are fewer relevant descriptions regarding the development of audio and music units. Additionally, the models used in the experiments are limited. For example, I believe there have been several works on hybrid tokenizers, but in the experiment section, only SpeechTokenizer is included.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. How do you ensure that the benchmark isolates token quality when multiple trainable components are included?
2. Why were layers 1, 3, 7, 12, 18, and 23 selected for WavLM and HuBERT token extraction? This choice may appear arbitrary without reference to literature or specific validation.
3. The results lack practical takeaway. How do you plan the community to benefit from the insights obtained using DASB? There seems to be contrasting results (e.g., DAC vs Encodec) compared to recent papers [1] (e.g., on ESC-50); what could be the cause of these differences?
4. How DASB compares with existing benchmarks in the literature?
5. Do you notice any differences across multiple runs of the same model on specific DASB tasks? Can those differences influence the evaluation?
6. Are the experimental settings (in terms of hyperparameters) the same for all evaluated models?

[1] Ji, Shengpeng, et al. "Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling." arXiv preprint arXiv:2408.16532 (2024).


Minor remarks:
- BERT is cited as an autoregressive model, how is it classified in this category?
- Mentioning Speech Enhancement with discrete tokens, the authors cited Deep Learning, the book from Goodfellow et al. (2016), the citation seems not pertinent.
soundness: 2
strengths: 1. The paper addresses an important issue in emerging areas of Audio Representation Learning: evaluating the quality of representations, in this specific case given by discrete tokens. 
2. The benchmark actually covers a broad set of speech-related tasks (e.g., ASR, classification with keyword spotting and intent classification, speaker identification and so on). It also include TTS and Speech Enhancement tasks as generative tasks.
3. The paper propose a straightforward explanation of each task and the architecture setup personalized by the paper for the task at hand. The paper clearly mention the dataset used for the evaluation.
summary: This paper introduces DASB, a benchmark for testing how well different architectures that generate "discrete" audio tokens perform in speech- and audio-related tasks. The benchmark include both discriminative tasks (e.g., ASR or classification) and generative ones (e.g., speech enhancement and TTS). 

The paper clarify that the goal is to test different types of tokens, semantic-based, compression-based and hybrid. It tests several models on the benchmark in those three categories. For compression-based tokens it investigates how the bitrates affect token quality. In the results, however, there is no clear evidence of better models given by higher bitrates.

The main goal of the paper can be summarized as the definition of a standard ways to evaluate tokens generated by different SSL models that have been pretrained for different tasks and using different approaches (i.e., codec, wav2vec2-style mainly).
weaknesses: 1. The main goal of a benchmark framework should be to fairly assess the quality of the tested models. However, the current version of the evaluation design, while broad in terms of tasks, include multiple trainable components for each task making very hard to isolate the evaluation of the token quality. This issue is even more puzzling because the results are reported as a single run of a model instead of averaging multiple runs and reporting standard deviations. The results may be influenced by several factors that may hinder the real comparison.
2. The benchmark design involved selecting specific metrics for each task. While the selection for most of them is standard (e.g., accuracy for classification), in other cases it may need discussion (e.g., why not PESQ instead of DNSMOS for speech enhancement?).
3. There are several experimental details that are missing. For example, the extraction of discrete tokens from WavLM and HuBERT involves a K-Means step, what is the value of K used in the paper? It may significantly influence all the experiments together with all set of other parameters (e.g., batch size, learning rate...). Are those hyperparameters consistent for all models? In addition, it is not clear what is the hardware used to train the downstream models, Is it consistent for all models? The paper make reference to the RTX 3070 GPU (8GB of VRAM) for computing time and memory requirements but then states in Appendix A1 "Every task runs on GPUs with 32 GB or more of VRAM".
4. The benchmark fails to reference existing benchmark that evaluate audio representations. It makes reference to SUPERB, however, the benchmark position itself as a general one (Discrete **Audio** and Speech Benchmark). There are several examples of general (including speech) audio representation frameworks [1][2][3]. The paper fails to compare with them leaving an unclear picture of DASB's specific contributions. In most of the downstream tasks, the model under testing is obtaining tokens -- that are then mapped back to "vectors" using codebooks, is thus only a matter of how those models are pre-trained (or discretized in the case of WavLM and HuBERT) that changes? Why evaluating them in an existing benchmark is not sufficient? DASB includes ASR and generative tasks (e.g., speech enhancement and TTS), is this the only difference in addition to the evaluated models?
5. Concerning audio events, the paper states "For sound classification, we utilized the pre-trained CNN14 model...", this is quite a difference with respect to other transformer-based models used all along the paper. Transformer-based SSL models exist pre-trained on general audio datasets (e.g., AudioSet)[2][4] and they are available open-source.

[1] Turian, Joseph, et al. "Hear: Holistic evaluation of audio representations." NeurIPS 2021.
[2] M. La Quatra et al., "Benchmarking Representations for Speech, Music, and Acoustic Events", ICASSPW 2024.
[3] Wang, Luyu, et al. "Towards learning universal audio representations." ICASSP 2022.
[4] Gong, Y. et al "AST: Audio Spectrogram Transformer.", Proc. Interspeech 2021.

Meta-review: In this work, the authors presents DASB, a benchmark for evaluating how well architectures generating "discrete" audio tokens perform on various speech and audio tasks, including discriminative tasks (e.g., ASR, classification) and generative ones (e.g., TTS, speech enhancement). The benchmark focuses on three token types: semantic-based, compression-based, and hybrid. It examines several models across these categories, analyzing how bitrates impact compression-based token quality.

Four independent reviewers assessed the work and despite the different small nuances among the overall the final reccomandations, the work seems not ready for publication. In fact, even after a good discussion phase between authors and reviewers, two of the reviewers kept their negative opinion suggesting to reject the work. Futhermore, although the remaning two reviewers were for a weak acceptance, one of them has however stressed out that "including only SpeechTokenizer limits the scope of the experiments."

Despite the interesting issue on Audio Representation Learning addressed in the proposed work, namely evaluating the quality of representations, in this specific case given by discrete tokens. Several major concerns were not resolved after the discussion period, namely: (i) The incremental nature of the benchmark design (especially in comparison to the SUPERB series) does not offer a significant advancement in the field. (ii) The evaluation is  incomplete within the audio domain,  which raises concerns regarding the thoroughness of the study, (iii) Some of the authors' claim not fully supported by facts, e.g., referencing to [1] to justify using 1000 clusters.

Finally, the following three questions have apparently been left unswered: (a) What unique insights does DASB provide beyond existing frameworks? (b) Why couldn't these tokens be evaluated using existing benchmarks? (c) How the DASB methodological choices compare to evaluation approaches in representation learnig (e.g. linear probing)?

Decision: Reject