dMel: Speech Tokenization Made Simple

## Abstract

Abstract Large language models have revolutionized natural language processing by leveraging self-supervised pretraining on vast textual data.
Inspired by this success, researchers have investigated complicated speech tokenization methods to discretize continuous speech signals so that language modeling techniques can be applied to speech data.
However, existing approaches either model semantic (content) tokens, potentially losing acoustic information, or model acoustic tokens, risking the loss of semantic (content) information.
Having multiple token types also complicates the architecture and requires additional pretraining.
Here we show that discretizing mel-filterbank channels into discrete intensity bins produces a simple representation ( dMel ), that performs better than other existing speech tokenization methods.
Using an LM-style transformer architecture for speech-text modeling, we comprehensively evaluate different speech tokenization methods on speech recognition (ASR) and speech synthesis (TTS).
Our results demonstrate the effectiveness of dMel in achieving high performance on both tasks within a unified framework, paving the way for efficient and effective joint modeling of speech and text.
The code will be open sourced.

## 1 Introduction

Large language models (LLMs) have achieved remarkable success in various natural language processing tasks by leveraging self-supervised pretraining on massive amounts of textual data [].
Inspired by this success, numerous works [, , , ] have sought to extend the language modeling approach to speech processing, aiming to build unified models capable of both speech understanding and generation tasks.
However, a key challenge lies in the continuous nature of speech signals, necessitating effective tokenization methods to discretize the input for language model-based processing.

Current speech tokenization approaches can be broadly categorized into two types: semantic (content) tokens and acoustic tokens(^1^11We use a word ‘semantic’ with the meaning of ‘content’ to keep prior work notation [].).
Semantic tokens, extracted from self-supervised (SSL) pretrained speech models [, ], where the speech signal is first encoded into speech representations and then clustered into semantic tokens with $k$-means method.
However, such SSL pretrained models are not useful for high fidelity speech synthesis as speaker identity and other details of raw speech are lost in training [].
Conversely, acoustic tokens can be obtained from audio compression models that are trained to compress the speech signal into codebook indices with residual vector quantization (RVQ) and reconstruction objectives [, ].
These tokens prioritize acoustic reconstruction but lose semantic information which can lead to poorer results in generating audio [].

To combine the advantages of both semantic and acoustic tokens, AudioLM [] proposed to model both semantic tokens and acoustic tokens with 3 stages: semantic modeling, coarse acoustic modeling, and fine acoustic modeling.
The coarse-to-fine modeling strategy is designed to match the residual structure of RVQ based acoustic tokens.
This solution addresses both content and speech quality, but its multi-stage hierarchical structure complicates the model and can lead to slower training and inference.
Another solution is to combine the semantic and acoustic features together. Zhang et al. [] proposed to distill the semantic tokens into the acoustic token’s first residual channel during the training of the RVQ model in a teacher-student manner.
In this way,
the new feature can preserve the semantic information better and also reconstruct high quality speech signals.

In this paper, we raise the following fundamental question – do we really need to separate speech into semantic and acoustic tokens first, and process them with idiosyncratic architectures?
We propose a simple alternative called dMel (see Figure 1) that discretizes log mel-filterbanks (Mel) energies directly into ordinal bins. Intriguingly, we find that discretizing Mel has little impact on the ability of off-the-shelf Mel vocoders to reconstruct waveforms(^2^22We used vocoders from [https://github.com/kan-bayashi/ParallelWaveGAN.](https://github.com/kan-bayashi/ParallelWaveGAN.)).
In Table 1 we show different vocoders to reconstruct waveforms from Mel and discretized Mel (dMel) computed on them, as well as ASR models trained on Mel and dMel.

**Table 1: Impact of discretization.**
|  | Reconstruction WER (%) | Recognition WER (%) |  |  |
| --- | --- | --- | --- | --- |
|  | P-WaveGAN^1 | HifiGAN^2 | Seq2seq^3 | CTC^4 |
| Ground-truth | 2.02 | - |  |  |
| Mel | 2.13 | 2.08 | 2.4 | 2.1 |
| dMel | 2.23 | 2.11 | 2.5 | 2.1 |

We find that the word error rate (WER) of an ASR system run on the reconstructed waveforms, is quite similar to the WER of the same system run on the ground-truth audio, showing that dMel captures the acoustic information needed to reconstruct good waveforms. Similarly, we find that the WER of ASR models trained on Mel and dMel are similar, indicating that dMel are good at preserving semantic content that can easily be found by ASR models. This shows that discretizing Mel has limited impact on information content.

By operating on the log mel-filterbanks and preserving the frequency and intensity information (with some loss of resolution from discretization), dMel inherently preserves both semantic and acoustic information in a unified representation, without the need for separate tokenization or additional pretraining of a tokenization model. There are many advantages to discretizing log mel-filterbanks:

- •
Log mel-filterbanks is an interpretable representation of speech signal, where both the semantic and acoustic information is preserved. As discretization has little impact, dMel inherits the properties.
- •
dMel is a model-free representation grounded in raw acoustic space. As a result it can be converted to waveforms by any mel-filterbank vocoder, unlike other tokenization schemes that have feature representations that are intricately coupled to both the encoder and the decoder.
- •
Different channels of dMel do not have the complex hierarchical dependencies on each other that is typical of coarse-to-fine acoustic tokens; we find that they can be modeled independently in each frame using a simple decoder-only (LM-style) transformer architecture.

Through comprehensive evaluations, we show that using dMel allows us to use a single decoder-only model, and achieve high performance on both automatic speech recognition (ASR) and text-to-speech (TTS) tasks.
The ASR task validates that dMel preserves semantic information, while the TTS task shows that dMel are useful for high-fidelity acoustic reconstruction of speech.
We also compare dMel to other tokenization methods and find that dMel achieves the best WER for ASR task, which indicates that the semantic information is well preserved.
Also, dMel achieves the lower WER score for TTS task when using WhisperX [] for automatic evaluation and we find model trained with dMel can generate long and natural speech samples (see Supplementary Material).

Figure: Figure 1: Prior works on speech tokenization use either heavy self-supervised pretrained encoders [, ] to extract semantic tokens (and train a separate decoder for it []) or learn compression encoder-decoder models with residual vector quantizations [, ] to obtain acoustic tokens. By contrast we eliminate the encoder and simply discretize mel-filerbanks (dMel) to encode audio, and use a simple mel-filterbank vocoder [] to reconstruct speech signals.
Refer to caption: x1.png

## 2 Method

In this section, we first introduce our proposed dMel speech tokenization method, which discretizes log mel-filterbanks energies directly into bins. We then describe our unified LM-style transformer model for ASR and TTS tasks, which leverages dMel for speech tokenization. The model architecture is illustrated in Figure 2.

### 2.1 dMel Speech Tokenizer

Different from existing VQ-VAE [, , , ] based speech tokenizers, we propose a discretized log mel-filterbanks based speech tokenizer.
The outline of the discretization method is shown in Figure 1.
Later in the paper, we show that this tokenizer allows the model to process the input speech signal efficiently and capture the relevant acoustic features for both ASR and TTS tasks.

We denote tensors as $\mathbf{X}$ while $\mathbf{X}_{i,...}$ denote the $(i,...)$-th component of tensor $\mathbf{X}$. First, the speech tokenizer takes the input speech signal $\mathbf{x}$ and computes the log mel-filterbanks representation $\mathbf{M}$:

$$ $\mathbf{M}=\text{Mel}(\mathbf{x}),$ (1) $$

where $\text{Mel}(\cdot)$ represents the function that computes the log mel-filterbanks, $\mathbf{M}\in\mathbb{R}^{T\times N}$, $N$ is the number of log mel-filterbanks and $T$ is the number of frames in the spectrogram.

##### Tokenization

To discretize the log mel-filterbanks representation $\mathbf{M}$ into speech tokens, we adopt a codebook $\mathbf{C}$.
In this paper, we apply a simple linear discretization, so that the codebook $\mathbf{C}\in\mathbb{R}^{2^{K}}$ and its values are evenly spaced in the range of the log mel-filterbanks values:

$$ $m=\min_{t,i}(\mathbf{M}_{t,i}),\qquad M=\max_{t,i}(\mathbf{M}_{t,i})\qquad \delta=\frac{M-m}{2^{K}},$ (2) $$

$$ $\mathbf{C}=\left[m,\,m+\delta,\,m+2\delta,\,\dots,\,m+(2^{K}-1)\delta\right].$ (3) $$

In practice, we compute the minimum $m$ and maximum $M$ values of log mel-filterbanks across the entire dataset to define the codebook $\mathbf{C}$.
Then we map a magnitude $\mathbf{M}_{t,i}$ of every frequency channel $i=1\dots N$ for the time frame $t=1\dots T$ into a bin index of the codebook $\mathbf{C}$ in the following way:

$$ $\mathbf{S}_{t,i}=\text{Discretize}(\mathbf{M}_{t,i})=\text{argmin}_{j}|\mathbf {M}_{t,i}-\mathbf{C}_{j}|$ (4) $$

where $\mathbf{S}\in\mathbf{B}^{T\times N}$ represents the discretized log mel-filterbanks (dMel) with $\mathbf{B}=\{j|j=1,2,3,\dots 2^{K}\}$ and $\mathbf{S}_{t}\in\mathbf{B}^{N}$ being the $t$-th speech token.
As the codebook $\mathbf{C}$ has $2^{K}$ distinct values and thus number of bins $|\mathbf{B}|=2^{K}$, each speech token is represented by $N\cdot K$ bits where every $K$ bits are used to represent one of $N$ frequency channels.

##### Detokenization

To reconstruct the speech signal $\mathbf{x}$ from the speech tokens $\mathbf{S}$, we first transform bin indices back to the log mel-filterbanks representation via the codebook $\mathbf{C}$:

$$ $\hat{\mathbf{M}}_{t,i}=\mathbf{C}_{\mathbf{S}_{t,i}}.$ (5) $$

Then, we apply a vocoder [] to transform reconstructed log mel-filterbanks $\hat{\mathbf{M}}_{t,i}$ back into the time domain signal $\mathbf{x}$.
The vocoder is trained independently and is not part of the transformer decoder-based model.

### 2.2 Unified Speech-Text Transformer Decoder

Figure: Figure 2: Unified Speech-Text Transformer Decoder with speech tokens as dMel.
Refer to caption: x2.png

Modeling speech and text sequences jointly is essential for a model to understand and generate both modalities.
However, it is challenging to design a unified model that can handle both speech-to-text and text-to-speech effectively.
In this work, we apply a unified LM-style transformer model that takes speech and text tokens as input and generates the output tokens in the target sequence.
The model is trained in end-to-end on a combined dataset of speech and text pairs, enabling it to learn the joint representations for ASR and TTS tasks.
As we show in the rest of the paper, the crucial part for the joint model training is the proper speech tokenization which dMel provides.

##### Token Representation

For text data, we apply a character-level tokenizer to convert the input text into a sequence of text tokens.
The text tokens are passed through an embedding layer, $\text{Embed}(\cdot):\{j|j=1,2,3\dots L\}\to\mathbb{R}^{D}$, where $D$ is the embedding dimension and $L$ is the vocabulary size.
The dimension of the speech token embedding is set to be the same as the text token embedding $D$ and no further mapping is required.
The motivation for using a character-level tokenizer is to reduce the vocabulary size $L$ and improve the model’s generalization ability.
Also, character tokens can capture the fine-grained linguistic features that are essential for both ASR and TTS tasks.

For speech signal, we apply the dMel speech tokenizer to convert the input speech signal into a sequence of speech tokens. Then, the speech tokens $\mathbf{S}\in\mathbf{B}^{T\times N}$ are passed through a learnable embedding layer, $\text{Embed}(\cdot):\mathbf{B}\to\mathbb{R}^{d}$, and a learnable linear layer, $\text{Linear}(\cdot):\mathbb{R}^{N\times d}\to\mathbb{R}^{D}$, to obtain the speech token representation $\mathbf{E}\in\mathbb{R}^{T\times D}$:

$$ $\displaystyle\mathbf{E}_{t}=\text{Linear}(\mathbf{E^{\prime}}_{t}),\,\text{and }\,\,\,\mathbf{E^{\prime}}_{t}=\text{Concatenate}([\text{Embed}(\mathbf{S}_{t, 1}),\text{Embed}(\mathbf{S}_{t,2}),\ldots,\text{Embed}(\mathbf{S}_{t,N})]),$ (6) $$

where $\mathbf{E}_{t}\in\mathbb{R}^{D}$ is the speech token representation.
Here, for every time frame $t$, a speech token $\mathbf{S}_{t}$ is processed in parallel and independently for every frequency channel $i$ by $\text{Embed}(\mathbf{S}_{t,i})$ mapping, and then embeddings of all frequency channels are stacked together to form one vector representation $\mathbf{E^{\prime}}_{t}$ for the frame $t$.
Finally, the speech token embeddings $\mathbf{E}_{t}$ are fed into the LM-style transformer models for further processing.

We also implemented other popular speech tokenizers including HuBERT-KM [] and SpeechTokenizer [] for comparison.
The main difference among these speech tokenizers is the codebook size and codes dimension, shown in Table 2.
For both HuBERT-KM and SpeechTokenizer the speech tokens are mapped via a learnable linear layer from their dimension to the text embedding dimension $D$ before feeding into the LM-style transformer model.

**Table 2: Comparison between different speech tokenizers: dMel (ours), HuBERT-LM and SpeechTokenizer. For dMel we use $N=80$ log mel-filterbanks and $2^{K}=16$ values of the codebook $\mathbf{C}$.**
|  | dMel | HuBERT-KM | SpeechTokenizer |
| --- | --- | --- | --- |
| Codebook Size | 16 | 200 | 1024 |
| Code Dimension | 80 | 1 | 8 |
| Training-free? | ✓ | ✗ | ✗ |

##### Speaker Representation

To properly model multi-speaker data, we also include speaker embeddings as input to the transformer decoder.
The speaker embeddings are extracted from an independent dvector [] model(^3^33We use a pretrained model “Speaker Encoder” from the YourTTS [] repository [https://github.com/Edresson/YourTTS](https://github.com/Edresson/YourTTS).).
We use a learnable linear layer to map the speaker embeddings to the same dimension as the speech and text token embeddings $D$.
The speaker representation is optional for ASR task, but required for TTS task.
Hence, during the training, it is applied for text-to-speech and ignored for speech-to-text.

##### Transformer Decoder

The transformer decoder is trained end-to-end on a combined dataset of speech and text pairs.
For TTS training, the input sequence is constructed by concatenating the speaker embedding (extracted from a random audio for the same speaker of the current sample), text tokens, and speech tokens.
For ASR training, the input sequence is constructed by concatenating the speech tokens and text tokens.
Both tasks are trained with causal masking, where the model is trained to predict the next token based on the previous tokens.
The loss is calculated using the cross-entropy loss between the predicted tokens and the ground-truth tokens. Loss calculation is skipped on the speech tokens for ASR task and on the text tokens for TTS task. Note, that all frequency channels at time frame $t$ for dMel tokenizer are predicted independently and in parallel.

To capture the relative distances between tokens in the input sequence, we apply multiplicative relative positional embedding RoPE [].
This allows the model to learn the positional relationships between speech tokens, text tokens, and speaker embeddings, enhancing its ability to generate coherent output sequences.
For positional embeddings we do not distinguish between text, speech and speaker tokens and thus having global positions notation across all of them, see Figure 2.

##### Robust Training

Compared to LMs, audio frames are highly redundant with strong local correlations.
This makes longform generation difficult for models due to exposure bias [].
To mitigate exposure bias during training, we apply span-masking [] to the speech token context, masking out multiple random spans of speech frames.
The model is trained to predict the next token based on the masked context.
This context-masking strategy helps the model learn to generate accurate speech tokens in the presence of missing information, improving its robustness and generalization. It forces the model to attend to the text rather than copying previously inferred speech tokens due to learnt correlations. We also find that span-masking text tokens improves the ASR task.

## 3 Experiments

In this section, we begin by evaluating different speech tokenizers through a common practice in the literature: tokenizing speech into discrete units and then reconstructing the speech to assess the quality of the reconstruction.
This approach helps gauge the effectiveness of various tokenization techniques.
Following this, we present both TTS and ASR results using an LM-style (decoder-only) model with different speech tokens.
While most related work focuses solely on speech synthesis, our study encompasses both speech generation and recognition, providing a more comprehensive evaluation of the tokenization methods.
We evaluate the performance of our model mainly on the LibriSpeech dataset and compare it with state-of-the-art speech tokenizers, ASR and TTS models.

### 3.1 Training Data

We use several open-sourced datasets with paired speech and text transcription to conduct experiments: i) LibriSpeech [] dataset (CC BY 4.0) consists of English speech recordings (960h, 16kHz) from various speakers ($\sim$2k) and conditions; ii) LibriTTS [] (CC BY 4.0) dataset (500h) derived from LibriSpeech improves on it with the proper sentence split, text normalization and keeping samples 24kHz; iii) VCTK [] contains 44h of English speech (108 speakers); iv) LJSpeech [] (public domain in US) is a single speaker English audio recordings of 16kHz with read speech from LibriVox(^4^44[https://librivox.org/pages/public-domain/](https://librivox.org/pages/public-domain/).).
While LibriSpeech is used to train ASR and TTS models, LibriTTS, VCTK and LJSpeech are only used to train the TTS.

### 3.2 Training Configuration

We train the LM-style transformers in three different sizes: Small, Base, and Large (see Appendix Table 12).
Unless stated otherwise, the Base model is used in all experiments if not stated otherwise.
All models use pre-LayerNorm with dropout set to 0.1 for residual, attention and embedding layers and 0.3 for positional embedding.
dMel uses 16 discrete bins for each channel while text is tokenized with a character vocabulary; the speaker embedding dvector has 512 dimensions (see Appendx E for details). In all experiments, training data are sampled to 16kHz.

### 3.3 Main Results

#### 3.3.1 Speech Reconstruction

Following [], we randomly sample 300 speech utterances and their ground truth transcriptions from the LibriSpeech test-clean dataset.
We use the speech2unit and unit2speech modules to convert the speech signal to speech tokens and then reconstruct the speech signal from the speech tokens.
We compute the WER between the ASR outputs from HuBERT-Large [](^5^55We use checkpoint [https://huggingface.co/facebook/hubert-large-ls960-ft](https://huggingface.co/facebook/hubert-large-ls960-ft).) on the audio samples and their ground truth transcripts.
We also report MOS-LQO (Mean Opinion Score – Listening Quality Objective) score to measure the reconstruction quality using ViSQOL [].
Finally, we use human evaluation to measure the naturalness of the reconstructed speech using a MOS score with 95% confidence interval.
We instruct the human evaluators to rate the naturalness of the reconstructed speech on a scale of 1 to 5, where 1 is the worst and 5 is the best.
The results are shown in Table 3.

**Table 3: Speech reconstruction results on 300 random samples from LibriSpeech test-clean set.**
| Tokenizer | Speech2Unit<br>(M params) | Unit2Speech<br>(M params) | Frame<br>Rate | WER↓ | MOS-LQO↑ | MOS↑<br>(95% CI) |
| --- | --- | --- | --- | --- | --- | --- |
| GroundTruth | - | - | - | 2.02 | - | 3.91$\pm$0.12 |
| HuBERT-KM | 95 | 111 | 50Hz | 8.71 | 2.06 | 2.74$\pm$0.14 |
| EnCodec | 7 | 7 | 75Hz | 2.03 | 4.03 | 3.69$\pm$0.13 |
| SpeechTokenizer | 65 | 34 | 50Hz | 2.41 | 4.19 | 3.77$\pm$0.13 |
| Mel-HifiGAN | n/a | 12 | 80Hz | 2.08 | 4.52 | 3.80$\pm$0.12 |
| dMel-HifiGAN | n/a | 12 | 80Hz | 2.11 | 4.47 | 3.68$\pm$0.13 |
| Mel-PWG | n/a | 1 | 80Hz | 2.13 | 4.40 | 3.27$\pm$0.14 |
| dMel-PWG | n/a | 1 | 80Hz | 2.23 | 4.37 | 3.23$\pm$0.14 |
| Mel-PWG | n/a | 1 | 40Hz | 2.36 | 4.34 | 2.99$\pm$0.15 |
| dMel-PWG | n/a | 1 | 40Hz | 2.51 | 4.29 | 2.97$\pm$0.15 |

From Table 3, we can see that semantic tokenization (HuBERT-KM) is not good for speech reconstruction.
Meanwhile, acoustic tokenizers that are optimized to reconstruct the signal directly (EnCodec and SpeechTokenizer) do well.

We apply different vocoders to reconstruct the speech signal from log mel-filterbanks, and find that the WER of the reconstructed speech signal is comparable to the acoustic tokenization methods with a fraction of the parameters.
Also, log mel-filterbanks achieve a better MOS-LQO score, which indicates that the reconstructed audio is more similar to the original audio.
By comparing Mel and dMel, we can see that discretization has little impact on WER and MOS-LQO scores.
We also find that the exact vocoder matters much less than the frame rate of tokenization: the WER goes from 2.08 to 2.13 when switching from HifiGAN to ParallelWaveGAN, but it falls from 2.13 to 2.36 when the frame rate is changed from 80Hz to 40Hz.
However, even a 1M parameter vocoder operating at a 40Hz frame rate is comparable to the much larger SpeechTokenizer on WER and MOS-LQO metrics.

Considering the efficiency and performance, we choose the dMel speech tokenizer in 40Hz with ParallelWaveGAN vocoder for the following experiments.

#### 3.3.2 LM-Style Text-to-Speech

Here we compare the accuracy and naturalness of speech synthesized by LM-style text-to-speech (TTS) models trained on different tokenization methods.
For TTS evaluation, we utilize WhisperX [] (“base.en” from []) to transcribe our generated speech into text and calculate the WER and the character error rate (CER).
We report both WER and CER to facilitate comparisons to prior works which have reported only one or the other.

We trained the TTS model using the same architecture but with three different tokenization methods: HuBERT+KM (with 200 clusters), SpeechTokenizer, and dMel.
Additionally, we present the results from VOXTLM [] and USLM [] for comparison.
VOXTLM is a larger model trained on more data that is initialized from a pretrained LLM (OPT) using HuBERT-KM as the speech tokenizer.
USLM comprises an autoregressive (AR) model and a non-autoregressive (NAR) model, both trained with the SpeechTokenizer.

As shown in Table 4 for training on LibriSpeech dataset, our LM-style model with dMel tokenization achieves a WER of 4.3 and a CER of 1.8, significantly outperforming the baseline methods. This indicates that our model can generate more accurate speech with less hallucination and distortion.
Furthermore, we observed that the AR model trained on SpeechTokenizer tokens exhibits a much higher WER compared to the idiosyncratic coarse to fine models (labeled AR+NAR) developed for these residual tokenizers – indicating that dMel lies on a simpler data manifold.

Given the success of our LM-style dMel TTS model, dubbed RichTTS, we further evaluate it on various datasets, including LJSpeech, VCTK, and LibriTTS, and compare it with popular open-sourced TTS models, including Tacotron2 [], FastSpeech2 [], and VITS [].
We conduct human evaluation to measure the naturalness of 50 randomly sampled synthesized speech from VCTK test set.
RichTTS achieves competitive performance on the TTS task in terms of both MOS and WER demonstrating its effectiveness in generating high-quality synthesized speech, see Table 5.
Interestingly, we find that VITS performs poorly on the VCTK WER.
We suspect this is because VITS tends to make more mistakes at the beginning of each sequence, and since VCTK comprises short sequences, even one or two word errors can lead to a high WER.

**Table 4: Text-to-speech results for different tokenizers. RichTTS is trained on LibriSpeech 960h. WER (%) and CER (%) are evaluated with WhisperX ASR (“base.en”) and reported on test-clean.**
| Model | WER↓ (%) | CER↓ (%) | Params |
| --- | --- | --- | --- |
| VOXTLM (HuBERT+KM)†, [] | - | 3.5 | 350M |
| USLM (SpeechTokenizer)†, AR+NAR, [] | 6.5 | - | 356M |
| RichTTS (HuBERT+KM) | 9.5 | 4.3 | 258M |
| RichTTS (SpeechTokenizer), AR | 11.4 | 5.9 | 258M |
| RichTTS (dMel) | 4.3 | 1.8 | 258M |

**Table 5: WER (%) (evaluated with WhisperX ASR “base.en”) and MOS of different TTS models’ generations using transcriptions from each evaluation set that correponds to data used for training.**
|  | WER↓ (%) | MOS↑ (95% CI) |  |  |
| --- | --- | --- | --- | --- |
| Model | LJSpeech | LibriTTS | VCTK | VCTK |
| GroundTruth | 2.6 | 3.8 | 3.4 | 4.18$\pm$0.10 |
| Tacotron2, [] | 4.4 | 7.3 | 4.2 | 2.91$\pm$0.15 |
| FastSpeech2, [] | 6.1 | 10.2 | 3.8 | 3.03$\pm$0.14 |
| VITS, [] | 6.4 | 8.3 | 11.1 | 3.56$\pm$0.12 |
| RichTTS (dMel) | 4.0 | 4.5 | 2.2 | 3.34$\pm$0.14 |

**Table 6: results for TTS models trained on LibriSpeech 960h and evaluated on LJSpeech test set.**
| Sequence Length | Tacotron2 | FastSpeech2 | VITS | RichTTS |
| --- | --- | --- | --- | --- |
| Total WER↓ (%) | 4.4 | 6.1 | 6.4 | 4.0 |
| 10-20 words | 5.5 | 3.1 | 7.4 | 3.5 |
| 20+ words | 3.3 | 9.1 | 5.3 | 3.0 |

Furthermore, we observed that our model with dMel tokenization can generate long audio sequences with high quality. Here, we evaluate the performance of our model on different lengths of text sequences using the LJSpeech test set.
Table 6 shows the WER results for our model on text sequences with 10-20 words and more than 20 words.
We ignore text sequences with fewer than 10 words, as they are too short and not robust for WER evaluation.
From Table 6, we observe that our model achieves competitive performance across different text lengths, demonstrating its robustness and generalization ability in generating synthesized speech for varying text inputs lengths.
Additionally, we find that the non-autoregressive (NAR) model FastSpeech2 achieves the lowest WER on shorter sequences but the highest WER on longer sequences. This suggests that NAR models may not be well-suited for generating long audio sequences.

#### 3.3.3 LM-Style Speech-to-Text

**Table 7: Speech recognition results for different tokenizers measured with WER (%). All models are trained on LibriSpeech 960h.**
| Model | dev-clean↓ | dev-other↓ | test-clean↓ | test-other↓ | Params |
| --- | --- | --- | --- | --- | --- |
| RichASR (SpeechTokenizer) | 6.5$\pm$0.3 | 16.9$\pm$0.7 | 6.9$\pm$0.4 | 17.5$\pm$0.5 | 258M |
| RichASR (HuBERT+KM) | 5.3$\pm$0.1 | 13.7$\pm$0.2 | 5.8$\pm$0.1 | 13.8$\pm$0.1 | 258M |
| RichASR (dMel) | 3.8$\pm$0.1 | 10.3$\pm$0.1 | 4.2$\pm$0.2 | 10.4$\pm$0.1 | 258M |

**Table 8: Comparison of WER (%) for best RichASR trained with dMel tokenization and prior work with LM-style ASR models and HuBERT+KM with subword modeling on top as tokenization.**
| Model | Data (h) | dev-clean↓ | dev-other↓ | test-clean↓ | test-other↓ | Params |
| --- | --- | --- | --- | --- | --- | --- |
| VOXTLM | 280k | - | - | 6.5 | 17.6 | 350M |
| VOXTLM | 280k | - | - | 4.6 | 12.1 | 1.3B |
| [] | 960 | 3.6 | 7.8 | 3.8 | 8.3 | 355M |
| RichASR (dMel) | 960 | 3.1 | 8.4 | 3.4 | 8.6 | 355M |

Training an LM-style speech-to-text (ASR) model can test if the speech tokens can preserve the semantic information in the speech signal and support the speech content-based task.
Table 7 shows results of our model dubbed RichASR, trained with different tokenizations including dMel for the ASR task. Our LM-style model with dMel speech tokenization achieves 4.2% WER on the test-clean and 10.4% WER on the test-other sets outperforming both HuBERT-KM and SpeechTokenizer.
We also observe that our model with HuBERT-KM [] outperforms the SpeechTokenizer [] for ASR, which is reasonable as semantic tokens are more suitable for the ASR task.

In Table 8, we further compare RichASR with dMel speech tokenizer trained with GPT-2-meduim architecture [] on LibriSpeech 960h with prior work: VOXTLM [] that uses larger model trained with more data and initialized from a pretrained LLM (OPT []), and HuBERT-KM with additional subword modeling on top as the speech tokenizer; [] that also uses GPT-2 architecture trained on LibriSpeech 960h and HuBERT-KM with additional subword modeling on top as the speech tokenizer(^6^66We use official codebase to train this model w/o text pretraining as [] report results only with text pretraining.).
RichASR with dMel outperforms VOXTLM; it also outperforms [] on clean sets and a bit behind it on other sets.

The ASR results clearly demonstrate the benefit of using our dMel speech tokenizer for the content-related tasks in speech, as it better preserves the semantic information in the speech signal. Further details and ablations can be found in Appendix E and F.

#### 3.3.4 Ablations

We first investigate the impact of the codebook sizes, shown in Table 9.
The 16-bin configuration used in the paper demonstrates the best overall performance across tasks.
While the 32-bin setup slightly outperforms on the ASR test-other set, it shows degraded
performance in TTS.
This trade-off likely stems from the increased speech vocabulary size, which may pose challenges for accurate prediction.
The results may get better with increased data and
model size.
And 8-bin configuration looses too much information with discretization.

We then ablate the ASR results to understand why ASR LM-style model is behind the state-of-the-art ASR results on LibriSpeech.
We take two existing transformer ASR baselines, Seq2Seq and CTC, that use 80 log mel-filterbanks and characters as targets.
We then modify these baselines by using dMel instead (the discretization, embedding layer and linear layer) while keeping all other hyper-parameters the same (we adjust only the SpecAugment time masking max width accordingly to keep total masking in ms the same).
Our results (Table 10) suggest: i) dMel brings only small degradation compared to Mel; ii) additional discrepancy is coming from different hop distance in featurization; iii) the main and significant performance degradation is coming from switching to LM-style model.
The latter is in line with [] and [], though was not discussed in detail by any prior work. We hypothesise this gap is due to observed overfitting of the LM-style models.

**Table 9: ASR and TTS results (WER, %) with dMel speech tokenizer and different number of bins (codebook size) for discretization in dMel. All models are trained on LibriSpeech 960h.**
| N-Bins | ASR test-clean↓ | ASR test-other↓ | TTS WER↓ (%) |
| --- | --- | --- | --- |
| 8 | 6.6 | 16.5 | 7.3 |
| 16 | 4.4 | 10.7 | 4.8 |
| 32 | 4.7 | 10.2 | 5.7 |

**Table 10: WER (%) comparison for CTC, Seq2Seq, and LM-style ASR models ($\sim$260M) trained on LibriSpeech 960h with dMel and Mel features. We compute 80 log mel-filterbanks with 25ms (50ms) window and 10ms (25ms) hop distance, denoted as ‘10ms’ (‘25ms’).**
| Model | Features | dev-clean↓ | dev-other↓ |
| --- | --- | --- | --- |
| [] (RNN-T – Conformer) | Mel-10ms | 1.9 | 4.1 |
| [] (CTC – Squeezeformer) | Mel-10ms | 2.3 | 5.8 |
| Seq2Seq [] | Mel-10ms | 2.4 | 5.4 |
| dMel-10ms | 2.5 | 5.9 |  |
| Mel-25ms | 2.8 | 6.5 |  |
| dMel-25ms | 2.7 | 6.2 |  |
| CTC [] | Mel-10ms | 2.1 | 5.4 |
| dMel-10ms | 2.1 | 5.6 |  |
| Mel-25ms | 2.1 | 5.4 |  |
| dMel-25ms | 2.3 | 6.1 |  |
| LM-style | dMel-25ms | 3.4 | 9.5 |

#### 3.3.5 Unlocking Joint Speech-Text Modeling

Our model design allows us to train a single model for both ASR and TTS tasks leading to a simpler setup.
We train a single model with the same architecture and tokenization as RichTTS, by constructing the training data with <text, speech> and <speech, text> pairs for ASR and TTS tasks, respectively.
By mixing these two types of data, we can train a single model for both tasks.

Table 11 shows that the joint model is worse on both tasks, but ASR is affected more than TTS.
Comparing our results to VOXTLM, which initializes its model from pretrained LLM (OPT) and finetunes it with multiple tasks and datasets, we speculate that our joint model needs text-only training to learn a good LM for better ASR performance. Our model structure trivially allows for this text-only training, but we leave those experiments for future work (for further discussion see Appendix F.3).

## 4 Related Work

##### Speech Tokenization

Recent advancements in speech tokenization have primarily focused on two approaches: semantic tokens and acoustic tokens. This section examines these methods, their combinations, and their limitations, highlighting the need for more efficient and generalizable solutions.
Semantic tokens, extracted from self-supervised pretrained speech models, have shown promise in capturing high-level content information. Methods like wav2vec [] and HuBERT [] employ $k$-means clustering on speech representations to generate these tokens.
While effective in capturing semantic content, these approaches often struggle with preserving fine-grained acoustic details crucial for high-quality speech synthesis.
In contrast, acoustic tokens, derived from pretrained audio compression models, excel at preserving low-level acoustic information.
Techniques such as SoundStream [] and EnCodec [] utilize residual vector quantization (RVQ) with reconstruction objectives. These methods achieve high-quality audio compression but may not capture higher-level semantic structures effectively.

Recognizing the complementary nature of semantic and acoustic tokens, recent works have attempted to combine these approaches.
AudioLM [] introduced a three-stage model: semantic modeling, coarse acoustic modeling, and fine acoustic modeling. While comprehensive, this approach introduces complexity and computational overhead.
AudioPalm [] further demonstrated the critical importance of large-scale training data and model parameters for effective multi-stage modeling, highlighting potential generalization issues in low-resource scenarios.
An alternative hybrid approach, proposed by Zhang et al. [], attempts to distill semantic information into acoustic tokens during RVQ model training.
However, this method still requires additional pretraining and does not fully achieve a single-stage model architecture.

Despite these advancements, several challenges persist in the field of speech tokenization: i) balancing semantic and acoustic information in a unified representation; ii) reducing model complexity and computational requirements; iii) improving generalization to low-resource / out-of-domain data e.g. with mixed speech from multiple speakers, or multiple languages, or changing characteristics of recording equipment/sampling rate etc.; iv) developing truly single-stage tokenizers.
Our proposed method, dMel, addresses these challenges by offering a training-free speech tokenization approach.
By directly discretizing log mel-filterbanks into bins, it inherently preserves both semantic and acoustic information in a unified representation, while significantly reducing computational complexity.

**Table 11: Results of ASR and TTS jointly trained model on LibriSpeech 960h.**
|  | ASR, WER↓ (%) | TTS |  |  |
| --- | --- | --- | --- | --- |
|  | test-clean | test-other | WER↓ (%) | CER↓ (%) |
| VOXTLM+OPT, 350M | 3.5 | 8.7 | - | 3.5 |
| RichASR-RichTTS, single models, 258M | 4.2 | 10.4 | 4.3 | 1.8 |
| RichASR-RichTTS, joint model, 258M | 7.6 | 20.0 | 4.4 | 1.9 |

##### Speech-Text Modeling

Modeling speech and text jointly is a challenging task, as speech signals are continuous and while text is discrete.
Existing works have explored various approaches to address this challenge, including usage of separate encoders for different modalities [, ].
Bai et al. [] proposed an encoder-only model A3T for speech-text modeling, by introducing alignment embedding to encourage cross-modal transfer between text and speech.
Although A3T achieved good performance on speech synthesis and editing tasks, it cannot generate text and cannot generalize to longform generation because of its encoder-only architecture and mask-reconstruction training strategy.
VioLA [] also targets a unified speech-text model which can generate speech and text with a single model, but it is specifically designed for the Encodec [] style feature, and compelled to model speech tokens in a multi-stage hierarchical manner.
Maiti et al. [] proposed a LM-style model VOXTLM, to model speech and text jointly.
However, VOXTLM is only models the HuBERT semantic tokens, and relies on an external generation model to transform semantic tokens into waveform, but the speaker and acoustic information are lost.
In comparison, the model architecture in this paper is a simple, single stage LM-style transformer model, and can handle both the speech generation and text generation tasks.

## 5 Conclusion

In this work, we proposed dMel, a novel train-free speech tokenization method that discretizes log mel-filterbank energies directly into bins. By operating on the authentic log mel-filterbank representation, dMel inherently preserves both semantic and acoustic information in a unified tokenized representation. Our key contribution is the evaluation of dMel within a unified LM-style transformer architecture for speech recognition (ASR) and speech synthesis (TTS) tasks.
Our dMel-based ASR model, RichASR, achieved the lowest word error rate among tokenization methods, robustly preserving semantic content. For TTS, dMel’s generation yielded the lowest WER, accurately reconstructing speech waveforms. Our dMel-based TTS model, RichTTS, achieved competitive naturalness, lowest error rates, and long audio generation capabilities.

dMel’s simplicity circumvents separate tokenizers or multi-stage modeling, reducing computational overhead and dependence on pretrained models. By unifying semantic and acoustic modeling, dMel enables efficient speech-text modeling frameworks.
While initial joint TTS-ASR training showed promise, further work is needed. Our primary contribution demonstrates dMel’s effectiveness for high-performing separate TTS and ASR models within a unified LM-style architecture.

## Appendix A Ethics Statement

The development and deployment of speech technologies carry important ethical considerations. While our proposed dMel method aims to advance the state-of-the-art in speech-text modeling, it is crucial to highlight potential ethical risks and raise the awareness so that new methods may be developed to mitigate these risks.

Our first main concern is the potential dual-use of speech synthesis technologies for nefarious purposes such as impersonation, misleading audio-visual content generation, or voice spoofing attacks.
Proactive measures, including watermarking techniques and robust speaker verification methods, should be explored to counter such risks.
The former attempts to build markers into the generated speech that make it easy to detect, while the latter focusses on distinguishing synthetic from real data.
Prior work [] has shown that neural networks can be trained to distinguish speech synthesized from their model from real speech, probably because of artifacts from the use of mel spectral vocoders.
While we did not train a network to do so in our work yet (we will create one before code release), the vocoders we use are similar to their work – going from mel spectrogram to raw waveforms.
Our model also does not use prosody, phoneme duration and other predictions that more sophisticated TTS systems use to allow the model to perform very well on imitating speaker styles in zero-shot settings.
However our model can probably mimic the styles of training speakers very well.
It is our hope that releasing our methods will facilitate more research on fake speech verification and watermarking techniques – even if current classifiers are able to perform this detection, the quality of the generative models is improving.
It is also our hope that future works will attempt to perform more credit assignment – by providing metrics that show which real data samples a synthetic speech example copies its style and substance from.

Another concern is the perpetuation of societal biases encoded within training data. Speech datasets may exhibit biases along dimensions such as gender, race, age, or socioeconomic status, which could be propagated or amplified by trained models. Rigorous debiasing techniques and careful curation of representative training data are essential to mitigate these risks. On the mitigating side of this equation, we also hope that with better, more controllable TTS systems, ASR systems can improve because more data can be generated for underrepresented segments of the distribution from the TTS models.

Furthermore, the development and deployment of speech technologies should prioritize accessibility and inclusivity. Models should be evaluated for performance across diverse demographics, accents, and language varieties to ensure equitable access and quality of service.

Finally, it is important to foster transparency and accountability in the research and development process. Clear documentation of model capabilities, limitations, and potential failure modes should be provided to enable informed decision-making and responsible usage.

Addressing these ethical considerations requires a multistakeholder approach involving researchers, developers, policymakers, and end-users. By prioritizing ethical principles such as fairness, privacy, and accountability, we can work towards realizing the benefits of speech technologies while mitigating potential risks and adverse societal impacts.

## Appendix B Limitations

Because TTS work is tremendously fragmented and clear protocols are not often available for training and evaluation, we reimplemented other tokenizers within our code base using publicly available, official implementations where available: e.g. we used Hubert-KM and speech tokenizer features extraction from the public codebases and pluged them into our LM-style model training.
While we made the best effort to tune the tokenization methods and the models, there is always a possibility we missed some details.
However, our results seem to tell a consistent story when viewed from multiple angles, and when viewed on multiple datasets.
We also did not train on larger model sizes (>1B parameters), larger datasets (>1k hours), or using pretrained models.

The real challenge for modern multimodal LLMs is complex semantic understanding tasks.
While our current experiments focus on text-to-speech and speech-to-text tasks, these encompass critical aspects of speech processing. dMel’s effective performance within a decoder-only architecture for both tasks suggests potential for broader applications.
We recognize the importance of more sophisticated speech understanding tasks and view our work as a foundation for future research leaving other tasks out of scope of the paper.
Scaling up pretraining and exploring complex semantic understanding tasks could further validate our approach’s versatility across a wider range of multimodal language processing challenges.

We acknowledge that our current scope targets only speech on purpose, as indicated in our title.
While dMel may potentially support non-speech tasks, our current exploration and verification focus solely on speech, not general audio.
Regarding the “speaker variations” – mel-spectrogram is used for speaker recognition widely, thus it preserves necessary speaker information on which we thus rely in dMel too.

## Appendix C Data, Code, Reproducibility

We made the best effort to use publicly available data and official implementations of prior works where it is possible. All data we used are under permissive license for research. We provided as much as detail as is possible without code such as details on our model training and hyperparameters throughout the paper and in the Appendix. We plan to open-source our code upon paper acceptance.

We do not plan to open-source any pre-trained models for sake of privacy, safety and misuse.

## Appendix D Subjective Evaluation for TTS

We use crowd-sourcing to collect subjective ratings to compare the naturalness of the reconstructed speech from the different tokenizers. We evaluate the quality of the same (randomly sampled) 50 utterances for each model by collecting around seven ratings per sample. Overall, we collect 3500 ratings from 65 raters. The raters were English-speaking and were paid at least the minimum wage.

We present the raters with a generated speech sample and instruct them to rate how natural it sounds on a five-point Likert scale, where 1 corresponds to very unnatural and 5 corresponds to very natural. Figure 3 shows a screenshot of our subjective test as seen by the rater.

Figure: Figure 3: A screenshot of the assessment task, as the crowd-sourced rater sees it.
Refer to caption: extracted/5896998/figs/crowdsource_screenshot.png

We noticed human annotators have bias over audio volume so we do volume normalization on top of all reconstructed or generated audio before giving them to human annotators.

We report Mean Opinion Score (MOS) results throughout the paper with confidence intervals calculated using bootstrap resampling with 1000 iterations, providing a reliable estimate of the variability MOS results.

## Appendix E Training Details

### E.1 Baselines

For reproducibility, we provide the HuggingFace model cards used in our experiments in Table 5:

- •
Tacotron2 [], [https://huggingface.co/espnet/espnet/kan-bayashi_vctk_tts_train_xvector_tacotron2_raw_phn_tacotron_g2p_en_no_space_train.loss.ave](https://huggingface.co/espnet/espnet/kan-bayashi_vctk_tts_train_xvector_tacotron2_raw_phn_tacotron_g2p_en_no_space_train.loss.ave)
- •
FastSpeech2 [], [https://huggingface.co/espnet/kan-bayashi_vctk_gst_fastspeech2](https://huggingface.co/espnet/kan-bayashi_vctk_gst_fastspeech2)
- •
VITS [], [https://huggingface.co/espnet/kan-bayashi_vctk_multi_spk_vits](https://huggingface.co/espnet/kan-bayashi_vctk_multi_spk_vits)

### E.2 RichASR and RichTTS

For our LM-style model we stack together speaker embedding, speech tokens and text tokens. Both speech and text tokens have prepended begin of sentence token (<bos>) and appended end of sentence token (<eos>).

We train all models using the Adam optimizer with a learning rate of 1e-3, learning rate warmup of 4k steps for ASR and 5k for TTS, cosine learning rate schedule and gradient clipping of 1.0 for TTS and 0.1 for ASR and joint models.
We use dynamic batching to optimize the data packing with total batch size of 1.4h/1.4h/0.7h for ASR training and 1h/2h/2h for TTS training for Small/Base/Large models.
We train TTS models for 100k steps and ASR models 80k steps with mixed precision training and BF16 on A100 and H100 GPUs with 80GB.
Both ASR models and TTS models are trained with 8GPUs for less than a day and for 2-4 days for ASR and TTS respectively.

**Table 12: LM-style transformer model configurations for ASR, TTS and joint models training.**
|  | Small | Base | Large |
| --- | --- | --- | --- |
| # of layers | 18 | 36 | 48 |
| # of attention heads | 2 | 4 | 8 |
| # of hidden units ($D$) | 512 | 768 | 1536 |
| # of parameters | 59M | 258M | 1.3B |

### E.3 LM-Style Speech-to-Text

For ASR training as an augmentation we apply SpecAugment [] with 2 frequency masks with max width 30 and 10 time masks with max width 50 and ratio 0.1.
With ablations we found that SpecAugment masking with average value instead of zero is slightly better.
Without applying SpecAugment performance of ASR is 7.3% WER on dev-clean and 20.3% WER on dev-other, which is further can be improved with usage of frequency masking only to 6.4% WER on dev-clean and 16.6% WER on dev-other. Usage of both frequency masking and time masking results in the best performance of Table 7.

We found that span masking is key part of model training to enforce slef-attention to attend to speech part as well as to reduce exposure bias. The masking strategy is similar to the one used for TTS training: for every training step with probability $p$ the sample in the minibatch is masked with the mean span of 3 tokens with masking ration of 0.5. We found that the mean span of 1 token or 5 tokens gives the same results; while the mask probability $p$ is the most important hyper-parameter. The optimimal value for ASR is found to be 0.8, which is used in all final models.

As we found one best model configuration for the Base model with dMel we then change only i) model size ii) speech tokenization iii) training data (here we increase model dropout to 0.3 for training on train-clean-360 and to 0.5 for training on train-clean-100 as otherwise models drastically overfit); the rest of hyper-parameters stay the same.

## Appendix F Ablations

### F.1 LM-Style Text-to-Speech

Scaling results for RichTTS are shown in Table 13.

**Table 13: Text-to-speech results for different model sizes with dMel. All models are trained on LibriSpeech 960h dataset. Evaluation is done via speech generation on the full test-clean transcriptions and speakers, and then evaluated WER with WhisperX base.en.**
|  | WER↓ (%) |
| --- | --- |
| RichTTS (dMel), Small | 8.1 |
| RichTTS (dMel), Base | 4.3 |
| RichTTS (dMel), Large | 5.4 |

### F.2 LM-Style Speech-to-Text

ASR ablations for different model sizes, data sizes, and tokenizers are shown in Table 14.

We noticd the results in [] seems to be the SOTA for LM-style ASR model to the best of our knowledge.
However, as many ablations are missed in [], we took their open-sourced code and run ablations ourselves to have proper comparison with it. The final results, including ablation with dMel are shown in Table 15:

- •
We successfully reproduced [] results (row 1 and 2).
- •
Without pretraining (rows 3, 4, 5):
dMel outperforms HuBERT-KM on both clean and other datasets;
dMel surpasses BPE on top of HuBERT-KM on clean data, while BPE on HuBERT-KM performs better on other.
- •
Without pretraining and without speed perturbation (rows 6, 7, 8):
BPE on HuBERT-KM performance decreases significantly after diabling speed perturbation (compare rows 3 and 6), raising questions about its generalizability to other domains, given that BPE tokens are trained on speed-perturbed LibriSpeech data.
Our dMel (row 8) achieves substantially better results than both HuBERT-KM and BPE on HuBERT-KM (rows 7 and 6), demonstrating robust performance even without speed augmentation.

Note that in dMel, we use SpecAugment (masking across time and channels) and [] also use SpecAugment. According to their code, the time masking is 30%, while channel masking is impossible as there is only 1 channel).

We believe these results demonstrate the effectiveness, simplicity in use, and robustness of our dMel tokenization method, particularly in scenarios where extensive pretraining or domain-specific augmentations may not be feasible.

Note that [] did not show applicability of BPE on HuBERT-KM or HuBERT-KM to TTS task, while in VOXTLM (also uses BPE on HuBERT-KM) it is shown that this tokenization is not suited for TTS (the performance is poor). dMel in contrary is shown to perform well on TTS task too in addition to ASR.

**Table 14: Our ASR models trained on different subsets (train-clean-100 LS-100, train-clean-360 LS-360, full LibriSpeech LS-960) of LibriSpeech, with different model sizes and different speech tokenizations (greedy decoding is reported). Results are shown across 2 runs with mean WER and standard deviation.**
| Tokenization | Model Size | Data | dev-clean↓ | dev-other↓ | test-clean↓ | test-other↓ |
| --- | --- | --- | --- | --- | --- | --- |
| dMel | Base | LS-100 | 18.1$\pm$1.0 | 39.4$\pm$1.2 | 19.0 $\pm$1.0 | 41.3$\pm$1.1 |
| dMel | LS-360 | 6.4$\pm$0.4 | 20.1$\pm$1.1 | 6.9 $\pm$0.6 | 20.5$\pm$0.9 |  |
| SpeechTokenizer [] | Small | LS-960 | 6.2$\pm$0.2 | 16.8$\pm$0.3 | 6.5$\pm$0.2 | 17.4$\pm$0.3 |
| HuBERT+KM [] | 5.8$\pm$0.2 | 14.6$\pm$0.1 | 6.0$\pm$0.1 | 14.9$\pm$0.1 |  |  |
| dMel | 6.0$\pm$0.4 | 15.2$\pm$0.8 | 6.1$\pm$0.4 | 15.7$\pm$0.7 |  |  |
| SpeechTokenizer [] | Base | LS-960 | 6.5$\pm$0.3 | 16.9$\pm$0.7 | 6.9$\pm$0.4 | 17.5$\pm$0.5 |
| HuBERT+KM [] | 5.3$\pm$0.1 | 13.7$\pm$0.2 | 5.8$\pm$0.1 | 13.8$\pm$0.1 |  |  |
| dMel | 3.8$\pm$0.1 | 10.3$\pm$0.1 | 4.2$\pm$0.2 | 10.4$\pm$0.1 |  |  |

**Table 15: Ablations on the LM-style ASR model with GPT-2 architecture using the setup from []: we ablate pretraining with text, speed perturbation and speech tokenization methods. All models are trained on LibriSpeech 960h. We report WER (%) on LibriSpeech validation and test sets.**
| Tokenization | Text Pretraining | Speed Perturbation | dev-clean↓ | dev-other↓ | test-clean↓ | test-other↓ |
| --- | --- | --- | --- | --- | --- | --- |
| BPE on HuBERT+KM, [] | ✓ | ✓ | 2.9 | 6.2 | 3.0 | 6.6 |
| BPE on HuBERT+KM, reproduction | ✓ | ✓ | 2.9 | 6.3 | 3.2 | 6.7 |
| BPE on HuBERT+KM | ✗ | ✓ | 3.6 | 7.8 | 3.8 | 8.3 |
| HuBERT+KM | ✗ | ✓ | 5.1 | 8.9 | 5.5 | 9.3 |
| dMel | ✗ | ✓ | 3.1 | 8.4 | 3.4 | 8.6 |
| BPE on HuBERT+KM | ✗ | ✗ | 5.4 | 9.7 | 5.3 | 10.0 |
| HuBERT+KM | ✗ | ✗ | 6.0 | 10.5 | 6.4 | 10.9 |
| dMel | ✗ | ✗ | 3.7 | 9.7 | 3.9 | 9.5 |

### F.3 Joint Speech-Text Modeling Discussion

We found it to be challenging to train joint model for ASR and TTS, similar to observations as in [] and e.g. [, ] for joint audio-visual speech recognition. Also, there is a very recent research work [], that also shows training TTS and ASR jointly is challenging, and needs carefully designed model architecture and training loss fusion technique.

One of the reasons is the different pace of learning. Careful consideration of training strategies can mitigate some of the challenges in joint modeling of TTS and ASR tasks, highlighting the complexities inherent in combining these distinct but related tasks within a single model.

Another reason we suspect is the mismatch between train and test time, which is more pronounced for the joint modeling: if we compare individual validation losses per task in joint model to their one-task training counterparts we see they match each other (so training is fine), however the generation (test time which mismatches how the train loss is defined) for both tasks is broken: longer sequences has hallucination and high repetition issues. This could be due to different length of sequences between text and audio and thus learnt attention pattern could be different which creates longer sequences generation issue for the joint model.

Last but not the least, the two tasks have opposite modalities in the input and output, making it rather difficult to model. Most previously researched multi-task work have the same modality in the output. The combination of ASR and TTS is a rather recent phenomenon, such as Viola and VOXTLM.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: refer to weaknesses
soundness: 3
strengths: - The proposed method is efficient, as it avoids hierarchical dependencies among mel-spectrogram channels, allowing for independent modeling of each channel within each frame using a straightforward, decoder-only (LM-style) transformer architecture.
- The approach is robust, simple yet innovative, with comprehensive evaluations that support the design choices.
- The encoder operates independently of the decoder, unlike many other tokenizers, making it compatible with any vocoder that accepts mel-spectrogram inputs.
- A detailed analysis of the setup is provided to enhance reproducibility.
- The paper is well-written and easy to follow, with a comprehensive analysis included.
summary: The paper introduces a novel approach to speech tokenization by discretizing mel-filterbank channels. This method effectively preserves both semantic and acoustic information, offering an interpretable, model-free representation grounded in the raw acoustic space. The authors train a transformer-based language model for speech-text modeling and evaluate their proposed tokenization approach on speech recognition (ASR) and speech synthesis (TTS) tasks.
weaknesses: - The evaluation could be more thorough by incorporating existing benchmarks such as Codec-Superb and DASB, allowing for a more comprehensive comparison of the proposed method against existing models under standardized settings.
 
- The related works section could be expanded to include methods that use frequency domain inputs, such as those discussed in the following papers:
    -  https://arxiv.org/pdf/2406.05298
    -  https://arxiv.org/pdf/2201.09429
    -  https://arxiv.org/pdf/2405.00233
    -  https://arxiv.org/pdf/2402.10533
    -  https://www.arxiv.org/pdf/2406.07422
- While Hubert-KM, Encodec, and Speech Tokenizer are reasonable baselines, it would be beneficial to include additional baselines with more similar setups, such as SPECTRAL CODECS (https://arxiv.org/pdf/2406.05298) or SemantiCodec (https://arxiv.org/pdf/2405.00233), for a fuller assessment.
- The proposed model is only evaluated on speech data, leaving other domains, such as general audio and music, unexplored.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Is there a fundamental difference between finite scalar quantization (FSQ; [1]) and dMel's quantization?
If not, I think the FSQ paper should be acknowledged.

[1] https://arxiv.org/pdf/2309.15505
soundness: 3
strengths: - The idea of quantizing mel spectrogram as tokenization is interesting and simple (in a good way).
- Results on TTS and ASR show dMel quantization has a small impact on models trained on continuous representation, training downstream models on top of dMel also provided similar results to their continuous counterparts.  These observations are interesting, showcasing the generalizability of dMel.
- Overall, I believe dMel is much more efficient in terms of model size and inference speed comparing to existing speech tokenizers (but this part is not well evaluated in the experiment section, see weaknesses).
summary: This paper presents dMel, an encoder-free speech tokenizer that simplifies speech tokenization by discretizing log mel-filterbank outputs into discrete intensity bins, eliminating the need for complex encoding architectures. Unlike previous tokenization methods that separate semantic and acoustic information, dMel maintains both in a single, unified representation. The tokenization process reduces precision of each filter output per frame while retaining the essential information needed for high-quality speech resynthesis, achieved by leveraging pre-trained vocoders. Additionally, the paper explores the application of dMel in language model (LM)-style training for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks. The results demonstrate that dMel performs comparably or better than existing methods in preserving semantic content and reconstructing natural-sounding audio. This efficient, unified approach to speech tokenization facilitates streamlined ASR and TTS training, advancing joint modeling of speech and text.
weaknesses: - As a speech tokenization paper, this work lacks a discussion on the overall bit rate for compression besides frame rate. Especially in the comparison with the prior works (e.g., Table 3).  dMel is over 12.8kbps~5kbps (assuming 40 fps $\times$ 32 mel filters $\times$ 4 bit-per-filter)~, which is higher than Hubert-KM and Speech Tokenizer. 

- This paper spent most of the space discussing ASR & TTS systems based on dMel. While the numbers are good, it is still not as good as a normal mel spectrogram (which is expected). This makes the content of the paper somewhat sparse, which is the biggest weakness in my opinion. The current paper seems to only suggest dMel is a spectrogram quantization approach, as it is essentially lowering numerical precision and showing the distortion is minimal on vocoder, ASR, and TTS. It would be more interesting to involve some other studies, for example:
  - Efficiency-related studies, such as how the encoder-free and lightweight-decoder design of dMel can speed up or lower memory usage downstream applications.
  - Applications where speech tokenization matters more, e.g., spoken LM [1,2], would better justify whether dMel can be viewed as a good speech tokenization approach. 

These scenarios/experiments would all be more suitable (than just plain ASR/TTS WER/MOS) for assessing the value of dMel. (I would like to note that these are not concerns that are expected to be addressed during the ICLR rebuttal period, they can be viewed as suggestions for future version of the paper)


[1] https://arxiv.org/pdf/2102.01192

[2] https://arxiv.org/abs/2410.00037

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1) What are the hyperparameters for extracting log Mel spectrograms? Window size? Stride?
2) Why are the model names "RichASR" and "RichTTS?" Any specific reasons?
3) What is the codebook utilization rate or distribution of dMel? The proposed quantization approach divides the intensity into equally-spaced bins. However, a potentially better way is to assign bin sizes according to the data distribution for a uniform codebook utilization.
4) Does pre-training the LM with speech-only data help downstream performance? In spoken LM applications, it is common to pre-train the LM on speech tokens with large unlabeled data.
5) Are there any decoding techniques involved in RichTTS and RichASR? E.g., beam search.
soundness: 2
strengths: The proposed dMel mitigates the issues in existing speech tokenizers. First, prior works like self-supervised learning (SSL) based tokenizers require extensive pre-training and sometimes not being able to preserve acoustic details for speech generation and synthesis. Second, neural codecs preserve fine-grained acoustic representations but might not be able to perform ASR and TTS because of the weak correlations between codebooks and frames. The authors propose a parameter- and training-free approach to achieve similar ASR and TTS performance.
summary: This paper proposes dMel, a simple method for quantizing Mel spectrograms into discrete units for LM-style decoder-only ASR and TTS. Unlike self-supervised semantic tokens and neural codecs, dMel is parameter- and optimization-free. Experimental results indicate superior ASR and TTS performance compared to prior methods like HuBERT + K-means and SpeechTokenizer.
weaknesses: Despite the success of the dMel method presented in the experiment results, the following issues question its novelty and effectiveness.

1) **Bitrate:**  
Bitrate is a crucial metric for comparing different tokenizers in prior studies but is not included in this paper. According to the provided information, dMel@40Hz, HuBERT-KM, and SpeechTokenizer, respectively, have bitrates of 12.8, 0.4, and 4kbps. The huge difference in bitrates might lead to an **unfair comparison**. Moreover, the number of centroids of K-means clustering in HuBERT-KM could be increased since 200 is considered a small codebook size (Table 2), while 500 and larger values are more commonly used in past literature.

2) **Baselines:**  
Advances in speech tokenization techniques have improved many downstream applications, including ASR and TTS. However, this paper only compares dMel with HuBERT + K-means and SpeechTokenizer, where the K-means method was proposed in 2021 [1]. Also, speech tokenization papers usually consider spoken language modeling a standard evaluation task [2,3,4].

3) **Writing:**  
Writing could be improved with the assistance of writing tools, including LLMs. For instance, from lines 299 to 301, the original text is "From Table 3, we can see that semantic tokenization (HuBERT-KM) is not good for speech reconstruction. Meanwhile, acoustic tokenizers that are optimized to reconstruct the signal directly (EnCodec and SpeechTokenizer) do well." The sentence is generally clear, but in academic writing, it's often better to use more precise language and avoid subjective terms like "not good" or "do well." A revised version is "Table 3 shows that semantic tokenization (HuBERT-KM) performs poorly in speech reconstruction, while acoustic tokenizers optimized for direct signal reconstruction (EnCodec and SpeechTokenizer) demonstrate superior performance."

[1] Lakhotia, Kushal, et al. "On generative spoken language modeling from raw audio." Transactions of the Association for Computational Linguistics 9 (2021): 1336-1354.  
[2] Gat, Itai, et al. "Augmentation invariant discrete representation for generative spoken language modeling." arXiv preprint arXiv:2209.15483 (2022).  
[3] Borsos, Zalán, et al. "Audiolm: a language modeling approach to audio generation." arXiv preprint arXiv:2209.03143 (2022).  
[4] Défossez, Alexandre, et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv preprint arXiv:2410.00037 (2024).

Note: dMel@40Hz bitrate = $40 \times 80 \times \log_2 16 = 12800 = 12.8$kbps

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: - For RichTTS and RichASR, what about the implementation details like architecture/training data (compare to speechgpt?)
- Is there any open-source plan to support the community?
soundness: 3
strengths: - New idea of using mel spectrum, which is continuous signal, for language modeling. This is different from recently popular codec based TTS.
summary: This work propose to discretize mel-spectrum into a special kind of intensity bins, which is proved to be a simple representation but more effective than commonly used speech tokenizers (i.e., codec). The authors claim that the newly proposed dMel well carry both acoustic and semantic information within speech signal, without losing information during quantization like codec. Experimental results have proved the effectiveness in tts and asr tasks.
weaknesses: - For TTS and ASR evaluation, there are only limited baselines for comparison, more powerful models like vall-e (TTS) and whisper (ASR) should also be included.
- The results of dMel are only reported on top of RichTTS and RichASR, experiments on more backbones are expected for better evaluation.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: see above
soundness: 3
strengths: see above
summary: This work propose to solve the problem of codec: it is hard for one codebook to cover both semantic and acoustic information, but multiple codebook will complicate the architecture and require additional pretraining. Therefore, this work proposes to discretize mel-filterbank channels into discrete intensity bins, which produces a simple representation that outperforms existing speech
tokenization methods.

I believe this is a pioneering work that may open a potentially new track of TTS research --> use continous mel to replace discrete codec in lm base TTS.

One question:
- How is it compared to another similar work MELL-E (https://arxiv.org/abs/2407.08551) that also use continous mel tokens for lm based TTS?
weaknesses: see above

Meta-review: This paper proposes dMel, a simple method for quantizing Mel spectrograms into discrete units. Unlike self-supervised semantic tokens and neural codecs, dMel is model-free. The authors train a transformer-based language model for speech-text modelling and evaluate their proposed tokenization approach on ASR and TTS. Experimental results indicate superior ASR and TTS performance compared to prior methods like HuBERT + K-means and SpeechTokenizer. Reviewer qtqg and wcjv requested an evaluation under the framework of spoken LM. The authors noted that some spoken tokenization papers also use similar downstream tasks. Both reviewers qtqg and wcjv suggested that the comparison is unfair due to dMel's higher bit rate. The authors argued that a comparison under the same bit rate is not required, but the reviewers remained not fully convinced. The meta-reviewer supports that a comparison under a similar bit rate is crucial (please refer to Additional Comments on Reviewer Discussion).

Decision: Reject