DM-Codec: Distilling Multimodal Representations for Speech Tokenization

## Abstract

Abstract Recent advancements in speech-language models have yielded significant improvements in speech tokenization and synthesis. However, effectively mapping the complex, multidimensional attributes of speech into discrete tokens remains challenging. This process demands acoustic, semantic, and contextual information for precise speech representations. Existing speech representations generally fall into two categories: acoustic tokens from audio codecs and semantic tokens from speech self-supervised learning models. Although recent efforts have unified acoustic and semantic tokens for improved performance, they overlook the crucial role of contextual representation in comprehensive speech modeling. Our empirical investigations reveal that the absence of contextual representations results in elevated Word Error Rate (WER) and Word Information Lost (WIL) scores in speech transcriptions. To address these limitations, we propose two novel distillation approaches: (1) a language model (LM)-guided distillation method that incorporates contextual information, and (2) a combined LM and self-supervised speech model (SM)-guided distillation technique that effectively distills multimodal representations (acoustic, semantic, and contextual) into a comprehensive speech tokenizer, termed DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM during the training process. Experiments show DM-Codec significantly outperforms state-of-the-art speech tokenization models, reducing WER by up to 13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility by 1.85% on the LibriSpeech benchmark dataset. The code, samples, and model checkpoints are available at https://github.com/mubtasimahasan/DM-Codec .

## 1 Introduction

In recent years, the advent of Large Language Models (LLMs) has revolutionized various domains, offering unprecedented advancements across a wide array of tasks (OpenAI, ). A critical component of this success has been the tokenization of input data, enabling vast amounts of information processing (Du et al., ; Rust et al., ). Inspired by these breakthroughs, significant attention has shifted towards replicating similar successes in the realm of speech understanding and generation (Défossez et al., ; Hsu et al., ). However, tokenizing speech into discrete units presents unique challenges compared to text, as speech is inherently continuous and multidimensional, requiring various speech attributes such as acoustic properties, semantic meaning, and contextual clues (Ju et al., ). Traditional approaches using feature representations such as Mel-Spectrograms (Sheng et al., ), Mel-frequency cepstral coefficients (MFCCs) (Juvela et al., ), and Waveforms (Kim et al., ) have proven inadequate in capturing this full spectrum of information, resulting in suboptimal performance in downstream tasks such as speech synthesis (Ju et al., ).

These limitations led researchers to explore various approaches, and one prominent direction leading to audio codecs (Borsos et al., ). Notable examples include SoundStream (Zeghidour et al., ) and EnCodec (Défossez et al., ), which utilize Residual Vector Quantizers (RVQ) within a neural codec framework, iteratively refining quantized vectors to discretize speech into acoustic tokens. Concurrently, self-supervised speech representation learning models such as HuBERT (Hsu et al., ) and wav2vec 2.0 (Baevski et al., ) facilitated extracting speech representations as semantic tokens (Borsos et al., ). Efforts to unify acoustic and semantic representations have led to two notable approaches: SpeechTokenizer (Zhang et al., ), which utilizes semantic distillation from HuBERT, and FACodec (Ju et al., ), which proposes a factorized vector quantizer to disentangle speech representation into different subspaces using separate RVQs with supervision.

Figure: Figure 1: An overview of speech tokenization approaches using discrete acoustic, semantic, and contextual tokens. DM-Codec integrates these multimodal representations for robust speech tokenization, learning comprehensive speech representations.
Refer to caption: extracted/5939010/figures/intro_diag_v2.png

While these approaches have shown promising results, they often overlook a crucial aspect of speech representation: the integration of contextual language information. Language models (LMs) have demonstrated a remarkable ability to learn contextual representations that capture the meaning of tokens based on their broader linguistic context (Devlin et al., ). These contextual representations can provide essential insights into speech representation, allowing for a more nuanced understanding of words in varying linguistic contexts. Our empirical investigations also reveal that existing discrete speech representation models struggle to align reconstructed speech with accurate textual form, resulting in elevated Word Error Rates (WER) and Word Information Lost (WIL) scores in speech transcription tasks. This observation underscores the need for a more comprehensive approach to speech tokenization that incorporates contextual language information.

To address these challenges, we propose DM-Codec, a novel speech tokenizer that unifies multimodal language and speech representations within a comprehensive tokenizer for speech. Our approach builds on a neural codec architecture incorporating RVQ with encoder, decoder, and discriminator components. Central to our innovation is the introduction of an LM-guided distillation method that effectively incorporates contextual representations into the speech tokenization process. This technique allows DM-Codec capturing the nuances of linguistic context often missed by existing models. Building upon the LM-guided approach, we further propose a hybrid distillation method combining both LM and speech model (SM)-guided techniques. The distillation method only utilizes the LM and SM during training, without increasing model complexity and parameters, and eliminates the need for them during inference. To the best of our knowledge, we are the first to attempt to integrate all three essential aspects of speech representation—acoustic, semantic, and contextual—within a single codec. See Figure 1 for a depiction.

Through extensive experimentation on the LibriSpeech benchmark dataset (Panayotov et al., ), we demonstrate the superiority of DM-Codec, which achieves significantly lower WER and WIL compared to state-of-the-art baseline speech tokenizers. Specifically, DM-Codec achieves a WER of $4.05$ and a WIL of 6.61, outperforming SpeechTokenizer ($4.49$, $7.10$), FACodec ($4.68$, $7.33$), and EnCodec ($4.53$, $7.17$). Furthermore, DM-Codec exhibits improved speech quality, as evidenced by its Virtual Speech Quality Objective Listener (ViSQOL) score of $3.26$, surpassing the performance of baseline models (EnCodec: $3.08$; SpeechTokenizer: $3.09$; FACodec: $3.13$).

Our research makes the following key contributions:

- •
We introduce DM-Codec, a novel speech tokenizer that incorporates contextual representations via an LM-guided distillation method.
- •
We present a novel combined LM and SM-guided representation distillation approach, uniting acoustic, semantic, and contextual representations into a unified framework.
- •
Through comprehensive experiments and ablation studies, we demonstrate the effectiveness of DM-Codec in preserving increased contextual information and enhancing the retention of acoustic and speech information in reconstructed speech.

## 2 Proposed Method

In this section, we present DM-Codec, a novel speech tokenizer designed to encapsulate a comprehensive fusion of multimodal (acoustic, semantic, and contextual) representations. As illustrated in Figure 2, we propose two distinct training approaches to incorporate these representations: (i) a language model (LM)-guided distillation method, and (ii) a combined LM and self-supervised speech model (SM)-guided distillation method. The first approach distills contextual representations from the LM and integrates them with learned acoustic representations. The second approach combines SM and LM to further incorporate semantic representations with contextual and acoustic representations. It ensures that DM-Codec captures the essential elements of speech by harmonizing the acoustic features with contextual and semantic information. The following subsections detail our proposed distillation methods (§2.1), model details (§2.2), and components (§2.3).

### 2.1 Speech and Language model Guided Distillation

Our approach first transcribes the raw speech $\mathbf{x}$ into its corresponding text $\mathbf{x^{\prime}}$ using a Speech-to-Text (STT) model $M_{STT}$, such that $\mathbf{x^{\prime}}\,=\,M_{STT}(\mathbf{x})$. For simplicity, we omit any post-processing techniques on the $\mathbf{x^{\prime}}$. Subsequently, we pass the text $\mathbf{x^{\prime}}$ through a pretrained language model $M_{LM}$ to obtain contextual representations of $\mathbf{x^{\prime}}$, tokenized into a set of tokens, $\mathcal{T}=\{t_{i}\}_{i=1}^{n}$. For each token $t_{i}$, we extract its corresponding layer-wise hidden representations $\{\mathbf{h}_{i}^{l}\}_{l=1}^{L}$, where $L$ denotes the total number of layers in $M_{LM}$. We utilize all layer representations to derive the representations for each token, as each layer of a pre-trained language model captures hierarchical and contextually distinct information (Niu et al., ; Kovaleva et al., ; Hao et al., ). To obtain the contextual representation $\mathbf{S}_{i}$ for token $t_{i}$, we average the hidden representations across all layers, yielding $\mathbf{S}_{i}=\frac{1}{L}\sum_{l=1}^{L}\mathbf{h}_{i}^{l}$, where $\mathbf{S}_{i}\in\mathbb{R}^{D}$ where $D$ is hidden dimension. Consequently, we obtain the contextual representations $\mathbf{S}=\{\mathbf{S}_{i}\}_{i=1}^{n}$ for the speech input $\mathbf{x}$, which captures the contextually diverse information from $M_{LM}$.

Simultaneously, we process the raw speech $\mathbf{x}$ through an Encoder $\mathbf{E}(\mathbf{x})$ to obtain the latent feature $\mathbf{v}$. We then pass $\mathbf{v}$ through a Residual Vector Quantizer (RVQ) to obtain quantized features $\mathbf{Q}=\{\mathbf{Q}_{k}\}_{k=1}^{K}$, where $K$ represents the number of quantization layers in the RVQ, and $\mathbf{Q}_{k}\in\mathbb{R}^{D^{\prime}}$ where $D^{\prime}$ is hidden dimension of $k^{th}$ RVQ layer. These quantized features are subsequently used to reconstruct the audio $\mathbf{\hat{x}}$ via a decoder. To align the quantized feature $\mathbf{Q}_{k}$ with the LM distilled features $\mathbf{S}_{i}$, we apply a linear transformation $\mathbf{Q}_{k}^{\prime}=\mathbf{W}\mathbf{Q}_{k}$, where $\mathbf{W}\in\mathbb{R}^{D^{\prime}\times D}$, ensuring the dimensional consistency for the distillation process.

Figure: Figure 2: DM-Codec framework consists of an encoder that extracts latent representations from the input speech signal. These latent vectors are subsequently quantized using a Residual Vector Quantizer (RVQ). We designed two distinct distillation approaches: (i) distillation from a language model, and (ii) a combined distillation from both a language model (LM) and a speech model (SM). These approaches integrate acoustic, semantic, and contextual representations into the quantized vectors to improve speech representation for downstream tasks.
Refer to caption: extracted/5939010/figures/tokenizer_v4.png

LM Guided Distillation: In this approach, we distil the LM representations $\mathbf{S}$. To calculate the LM-guided distillation loss, we adopt a continuous representation distillation technique, similar to the one employed by SpeechTokenizer (Zhang et al., ), which maximizes the cosine similarity at the dimension level across all time steps. In our case, we calculate the continuous representation distillation of the transformed quantized features $\mathbf{Q}_{k}^{\prime}$ and the LM representation features $\mathbf{S}$ as follows:

$$ $\mathcal{L}_{L}=-\frac{1}{D}\sum_{d=1}^{D}\log\left(\sigma\left(\frac{\mathbf{ Q}_{k}^{\prime(:,d)}\cdot\mathbf{S}^{(:,d)}}{\|\mathbf{Q}_{k}^{\prime(:,d)}\| \|\mathbf{S}^{(:,d)}\|}\right)\right)$ (1) $$

Here, the notation $(:,d)$ indicates a vector that includes values from all time steps at the $d^{th}$ dimension. The function $\sigma(\cdot)$ represents the sigmoid activation function, commonly used to squash input values into a range between 0 and 1.

Combined LM and SM Guided Distillation:
To further enhance the capabilities of DM-Codec, we propose a hybrid approach that utilizes both audio and text modalities. To derive semantic representations from the speech model (SM), we adopt a similar distillation strategy as we used for the LM. We first pass the raw speech $\mathbf{x}$ through the pretrained speech model $M_{SM}$, which generates its own set of layer-wise hidden representations $\{\mathbf{h}_{j}^{l}\}_{l=1}^{L}$. The semantic features are derived by averaging the hidden states across all layers, yielding $\mathbf{A}_{j}=\frac{1}{L}\sum_{l=1}^{L}\mathbf{h}_{j}^{l}$, where $\mathbf{A}_{j}\in\mathbb{R}^{D}$. This process results in the semantic representations $\mathbf{A}=\{\mathbf{A}_{j}\}_{j=1}^{n}$ for the speech input $\mathbf{x}$. The distillation loss in this case considers both the LM and SM representations, jointly optimizing for the alignment of the quantized features $\mathbf{Q}_{k}^{\prime}$ with the representations $\mathbf{A}$ and $\mathbf{S}$ derived from $M_{SM}$ and $M_{LM}$, respectively. Finally, the distillation loss for the SM, $\mathcal{L}_{SM}$, is first computed, followed by averaging with the LM distillation loss, $\mathcal{L}_{L}$, to ensure a balanced contribution from both losses. The combined distillation loss is computed as:

$$ $\displaystyle\mathcal{L}_{SM}$ $\displaystyle=$ $\displaystyle-\frac{1}{D}\sum_{d=1}^{D}\log\left(\sigma\left(\frac{\mathbf{Q}_ {k}^{\prime(:,d)}\cdot\mathbf{A}^{(:,d)}}{\|\mathbf{Q}_{k}^{\prime(:,d)}\|\| \mathbf{A}^{(:,d)}\|}\right)\right)$ (2) $\displaystyle\mathcal{L}_{LS}$ $\displaystyle=$ $\displaystyle\frac{1}{2}\left(\mathcal{L}_{SM}+\mathcal{L}_{L}\right)$ (3) $$

This formulation ensures that DM-Codec effectively integrates both acoustic and semantic knowledge from SM, along with the contextual information provided by LM, resulting in a more robust and comprehensive set of features for speech discretization.

### 2.2 Model Details

Our framework builds upon the Residual Vector Quantizer with Generative Adversarial Networks (RVQ-GAN) architecture, incorporating state-of-the-art components and novel distillation techniques. The core of our model consists of an Encoder $\mathbf{E}$ and Decoder $\mathbf{D}$ with an RVQ architecture, inspired by Encodec (Défossez et al., ) and SpeechTokenizer (Zhang et al., ). Moreover, we employ a multi-discriminator framework, comprising: Multi-Scale Discriminator (MSD), Multi-Period Discriminator (MPD), and Multi-Scale Short-Time Fourier Transform (MS-STFT) Discriminator, adopted from HiFi-Codec (Yang et al., ) and HiFi-GAN (Kong et al., ).
This foundation provides a robust basis for speech quantization. To further enhance the quantizer with distilled multimodal representations, we use wav2vec 2.0 (wav2vec2-base-960h) as $M_{STT}$ (Baevski et al., ), BERT (bert-base-uncased) as $M_{LM}$ (Devlin et al., ), and HuBERT (hubert-base-ls960) as $M_{SM}$ (Hsu et al., ). We extract the quantized output from the first layer of the RVQ (RVQ-1) for LM-guided distillation and the average of the quantized features across all eight layers (RVQ-1:8) for SM-guided distillation to calculate the distillation loss.

### 2.3 Model Components

Encoder Decoder.
The encoder-decoder architecture in DM-Codec is based on SEANet (Tagliasacchi et al., ), leveraging the successful design employed in recent speech tokenization models (Zhang et al., ; Défossez et al., ; Zeghidour et al., ). The architecture is designed to efficiently process and reconstruct speech signals while maintaining high fidelity. The Encoder $\mathbf{E}$ consists of a 1D convolution layer with $\mathit{C}$ channels and a kernel size of 7, followed by $\mathit{B}$ residual convolutional blocks. Each block contains a strided convolutional downsampling layer with kernel size $\mathit{K}$ (where $\mathit{K}=2\mathit{S}$ , and $\mathit{S}$ represents the stride), paired with a residual unit. The residual unit comprises two convolutional layers with a kernel size of 3 and a skip connection, while the number of channels is doubled at each downsampling stage. This is followed by a two-layer BiLSTM and a final 1D convolutional layer with $\mathit{D}$ output channels and a kernel size of 7. The Decoder $\mathbf{D}$ mirrors the encoder’s structure but replaces BiLSTM with LSTM, strided convolutions with transposed convolutions, and employs reversed strides for up-sampling. The final audio output is reconstructed from $\mathbf{D}$. For the experiments, we use the following configuration: $\mathit{C}$ = 32, $\mathit{B}$ = 4, and $\mathit{S}$ = (2, 4, 5, 8).

Residual Vector Quantizers.
The Residual Vector Quantizer (RVQ) plays a central role in our tokenization process, quantizing the encoder’s outputs. Our implementation is inspired by the training procedures described in Encodec (Défossez et al., ) and SpeechTokenizer (Zhang et al., ). The RVQ projects input vectors to the most similar entry in a codebook, and the residual is calculated and processed in subsequent quantization steps, each utilizing a different codebook. The codebook entries are updated using an exponential moving average (EMA) with a decay rate of 0.99 for the matched item, while unmatched entries are replaced by candidates from the current batch. To ensure proper gradient flow during training, we employ a straight-through estimator. A commitment loss is also computed and added to the total training loss to promote stability. In our experiments, we utilize a codebook size of 1024 and 8 quantization levels.

Discriminators.
We incorporate a trio of discriminators to enhance the quality and realism of the generated speech: the Multi-Scale Discriminator (MSD), the Multi-Period Discriminator (MPD), and the Multi-Scale Short-Time Fourier Transform (MS-STFT) discriminator. The MS-STFT discriminator follows the implementation outlined in (Défossez et al., ), operating on the real and imaginary components of multi-scale complex-valued STFTs. It begins with a 2D convolutional layer, followed by 2D convolutions with increasing dilation rates in the time dimension (1, 2, and 4) and a stride of 2 across the frequency axis in each sub-network. A final 2D convolution with a kernel size of 3 × 3 and a stride of (1, 1) is applied to produce the prediction. The MSD and MPD discriminators follow the architectures introduced in (Kong et al., ), with adjustments to the channel numbers to align the parameter count more closely with the MS-STFT discriminator. This ensemble of discriminators works in concert to provide comprehensive feedback on various aspects of the generated speech, contributing to the overall quality and naturalness of the output.

### 2.4 Training Objective

Our training strategy employs a GAN-guided framework, following methodologies established in recent work (Zhang et al., ; Yang et al., ). In addition to the distillation loss described in Section 2.1, we utilize reconstruction losses, adversarial and feature matching losses, and a commitment loss to guide the learning process. For the original speech $\mathbf{x}$ and the reconstructed speech $\hat{\mathbf{x}}$, we calculate the losses as described below.

Reconstruction Loss. To ensure that the model preserves the key attributes of speech, we employ both time-domain and frequency-domain reconstruction losses. The time-domain loss $\mathcal{L}_{t}$ is computed as the L1 distance between $\mathbf{x}$ and $\hat{\mathbf{x}}$. For the frequency-domain loss $\mathcal{L}_{f}$, we combine L1 and L2 losses over 64-bin Mel-spectrograms $\text{Mel}_{i}$, with varying window sizes of $2^{i}$, hop lengths of $2^{i}/4$, and scales $e=\{5,\ldots,11\}$.

$$ $\displaystyle\mathcal{L}_{\text{t}}$ $\displaystyle=$ $\displaystyle\|\mathbf{x}-\hat{\mathbf{x}}\|_{1}$ (4) $\displaystyle\mathcal{L}_{f}$ $\displaystyle=$ $\displaystyle\sum_{i\in e}\left(\|\text{Mel}_{i}(\mathbf{x})-\text{Mel}_{i}( \hat{\mathbf{x}})\|_{1}+\|\text{Mel}_{i}(\mathbf{x})-\text{Mel}_{i}(\hat{ \mathbf{x}})\|_{2}\right)$ (5) $$

Adversarial Loss. The adversarial loss promotes the generator to produce realistic and indistinguishable speech. We apply a hinge loss formulation to compute the adversarial loss for both the generator $\mathcal{L}_{g}$ and the discriminator $\mathcal{L}_{d}$. These losses are computed across all three discriminators: the multi-scale discriminator (MSD), multi-period discriminator (MPD), and the multi-scale STFT guided (MS-STFT) discriminator.

$$ $\displaystyle\mathcal{L}_{g}$ $\displaystyle=$ $\displaystyle\frac{1}{N}\sum_{n=1}^{N}\max(1-R_{n}(\hat{\mathbf{x}}),0)$ (6) $\displaystyle\mathcal{L}_{d}$ $\displaystyle=$ $\displaystyle\frac{1}{N}\sum_{n=1}^{N}\left(\max(1-R_{n}(\mathbf{x}),0)+\max(1 +R_{n}(\hat{\mathbf{x}}),0)\right)$ (7) $$

where $N$ is the number of discriminators and $R_{n}$ represents the $n^{th}$ discriminator.

Feature Matching Loss. To prevent the generator from overfitting to the discriminator’s decisions, we apply a feature matching loss $\mathcal{L}_{fm}$. This loss compares features from each discriminator $R_{n}$’s internal layers $M$ across all dimensions, promoting stability and better generalization.

$$ $\mathcal{L}_{fm}=\frac{1}{NM}\sum_{n=1}^{N}\sum_{m=1}^{M}\frac{\|R_{n}^{m}( \mathbf{x})-R_{n}^{m}(\hat{\mathbf{x}})\|_{1}}{\text{mean}(\|R_{n}^{m}(\mathbf {x})\|_{1})}$ (8) $$

RVQ Commitment Loss. To guide the encoder to produce outputs that closely match their corresponding quantized values in the residual vector quantization (RVQ) process, we introduce a commitment loss $\mathcal{L}_{w}$. For $N_{q}$ quantization vectors, where $\mathbf{q}_{i}$ represents the current residual and $\mathbf{q}_{c_{i}}$ is the closest entry in the corresponding codebook for the $i^{th}$ entry, the $\mathcal{L}_{w}$ is computed as:

$$ $\mathcal{L}_{w}=\sum_{i=1}^{N_{q}}\|\mathbf{q}_{i}-\mathbf{q}_{c_{i}}\|_{2}^{2}$ (9) $$

Overall Generator Loss. The total generator loss $\mathcal{L}_{G}$ is a weighted sum of the individual loss components, including the distillation loss $\mathcal{L}_{L/LS}$ (which is either $\mathcal{L}_{L}$ or $\mathcal{L}_{LS}$ depending on the chosen distillation method). We use the corresponding weighting factors $\lambda_{L/LS},\lambda_{t},\lambda_{f},\lambda_{g},\lambda_{fm}$, and $\lambda_{w}$ to control the influence of each loss component on the overall training objective as:

$$ $\mathcal{L}_{G}=\lambda_{L/LS}\mathcal{L}_{L/LS}+\lambda_{t}\mathcal{L}_{t}+ \lambda_{f}\mathcal{L}_{f}+\lambda_{g}\mathcal{L}_{g}+\lambda_{fm}\mathcal{L}_ {fm}+\lambda_{w}\mathcal{L}_{w}$ (10) $$

This comprehensive training objective ensures DM-Codec learns acoustic speech representations while incorporating semantic and contextual representation through novel distillation approaches.

## 3 Experimental Setup

Dataset.
We trained DM-Codec using the LibriSpeech training set of 100 hours of clean speech (Panayotov et al., ). This dataset was selected primarily because of its successful use for training and evaluation in various speech tokenizer and modeling tasks (Zhang et al., ; Ju et al., ; Hsu et al., ). Before training, we made the data uniform by randomly cropping each sample to three seconds and ensuring a consistent sample rate of 16 Hz.

Training.
We trained DM-Codec utilizing 2 to 4 A100 GPUs until the model converged within 100 epochs. The batch size ranged from 6 to 20, depending on GPU resource availability. We applied a learning rate of $1\times 10^{-4}$ using the Adam optimizer with a 0.98 learning rate decay. The embedding size was set to 1024 for RVQ and 768 for the LM and SM. For all experiments, we used a random seed of 42 to ensure reproducibility. We also share our training code with the entire configuration file and a docker file to reproduce the training environment, which can be accessed through the GitHub link referenced in the abstract.

Baselines.
We compared DM-Codec with the baseline speech tokenizers: EnCodec (Défossez et al., ), SpeechTokenizer (Zhang et al., ), and FACodec (NaturalSpeech3) (Ju et al., ). We reproduced SpeechTokenizer using the official training code and used official model checkpoints of EnCodec (EnCodec_24khz_6kpbs) and FACodec as the baselines.

Evaluation Dataset.
To evaluate DM-Codec, we randomly selected 300 audio samples from the LibriSpeech test subset, following a similar practice of sampling test data used in our baselines (Zhang et al., ; Zeghidour et al., ) and to align the experimental setup with that of SpeechTokenizer. In our experiments, we sampled the test subset of LibriSpeech using a random seed of 42. We also evaluated the baseline models with the same sampled test dataset for a fair comparison.

Evaluation Metrics.
To evaluate DM-Codec, we employed different metrics suited to get insights into various aspects of information and quality preservation in the reconstructed speech. First, we used the Word Error Rate (WER) and Word Information Lost (WIL) metrics to evaluate context preservation by calculating the amount of word-level transcription errors and key information missing in transcription, respectively. For these metrics, we used the Whisper (whisper-medium) (Radford et al., ) model to extract the transcription from the reconstructed speech. To provide a fairer comparison and indicate the level of transcription error by the Whisper model, we also included the Groundtruth WER and WIL scores for the Whisper’s transcribed text from the original speech versus the true text. Next, we assessed the acoustic and semantic information preservation using the ViSQOL (Virtual Speech Quality Objective Listener) (Hines et al., ) and Short-Time Objective Intelligibility (STOI) metrics, respectively. The ViSQOL metric measures the similarity between a reference and a test speech sample using a spectro-temporal measure and produces a MOS-LQO (Mean Opinion Score - Listening Quality Objective) score ranging from 1 (worst) to 5 (best). For this metric, we used the wideband model suited for speech evaluation. Lastly, the STOI metric evaluates the perceived intelligibility of speech by analyzing short-time correlations between original and reconstructed speech, with scores ranging from 0 to 1.

**Table 1: Evaluation of speech reconstruction quality of DM-Codec and comparison with baselines. DM-Codec^$\spadesuit$ achieves the best performance in WER, WIL, and ViSQOL, highlighting its enhanced content preservation and speech quality, with competitive intelligibility results. ^$\heartsuit$ means the results were reproduced using the official training code. ^$\diamondsuit$ means the results were obtained using official model checkpoints. ^$\clubsuit$ indicates LM-guided Distillation method. ^$\spadesuit$ indicates combined LM and SM-guided Distillation method. Bold highlights the best result and underline the second-best result.**
| Tokenizer | WER $\downarrow$ | WIL $\downarrow$ | ViSQOL $\uparrow$ | STOI $\uparrow$ |
| --- | --- | --- | --- | --- |
| Groundtruth | 3.78 | 6.03 | - | - |
| EnCodec^$\diamondsuit$ | 4.53 | 7.17 | 3.08 | 0.920 |
| SpeechTokenizer^$\heartsuit$ | 4.49 | 7.10 | 3.09 | 0.923 |
| FACodec^$\diamondsuit$ | 4.68 | 7.33 | 3.13 | 0.949 |
| DM-Codec ^$\clubsuit$ | 4.36 | 7.06 | 3.18 | 0.935 |
| DM-Codec ^$\spadesuit$ | 4.05 | 6.61 | 3.26 | 0.937 |

## 4 Experimental Results and Discussion

We conducted extensive experiments to evaluate DM-Codec’s reconstructed speech using WER and WIL for contextual information retention, and ViSQOL and STOI for semantic-acoustic information preservation. To demonstrate the effectiveness of our two distillation approaches, we present results for DM-Codec^$\clubsuit$ (LM-guided Distillation) and DM-Codec^$\spadesuit$ (LM and SM-guided Distillation).

### 4.1 Comparison of Speech Tokenization models

We compared the quality of DM-Codec’s discrete speech representations by reconstructing speech from quantized vector features and comparing it with state-of-the-art (SOTA) speech tokenization models: EnCodec, SpeechTokenizer, and FACodec. For LM-guided distillation, we utilize quantized features from the first Residual Vector Quantizer layer (RVQ-1), and for the combined LM and SM-guided distillation, we average all layers (RVQ-1:8).

Results: The results in Table 1 show that DM-Codec outperformed all evaluated SOTA speech tokenization models across most metrics. Specifically, DM-Codec with only LM-guided distillation exceeds the SOTA models, achieving improved scores: WER 4.36, WIL 7.06, and ViSQOL 3.18. Furthermore, DM-Codec’s with combined LM and SM-guided distillation outscore LM-guided distillation and all previous scores with 4.05 WER, 6.61 WIL, 3.26 ViSQOL, and achieved highly compatible 0.937 STOI scores compared to SOTA models.

Discussion: The observed performance gains stem from the proposed LM-guided distillation, which enhances the quantized features by leveraging LM’s contextual representations. This process aligns the speech with its overall context and word relation, resulting in more accurate reconstructions, as reflected in the reduced WER and WIL scores. By embedding contextual cues, the method effectively grounds isolated phonetic units within their overall context, reconstructing speech that aligns with human expectations, as demonstrated by the higher ViSQOL and STOI scores.

Moreover, the integration of LM and SM-based distillation further amplifies these improvements. The addition of SM distillation contributes to enhanced semantic-acoustic fidelity, as SM models capture phonetic nuances alongside prosodic and tonal characteristics. This dual representation—context from LM and phonetic detail from SM—produces a more coherent and natural speech reconstruction, yielding superior results across all metrics.

**Table 2: Significance Analysis of DM-Codec (D) compared to baselines EnCodec (E), SpeechTokenizer (S), and FACodec (F). Results reveal DM-Codec consistently achieves significantly better scores in key metrics across all individual samples. ✓ indicates that DM-Codec is significantly better, a ★ denotes dominance, and a ✗ means no significant improvement over the baseline. Avg and Std mean the average and standard deviation of each score.**
| WER $\downarrow$ | WIL $\downarrow$ | ViSQOL $\uparrow$ | STOI $\uparrow$ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DM-Codec |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Avg | Std | E | S | F | Avg | Std | E | S | F | Avg | Std | E | S | F | Avg | Std | E | S | F |
| 0.053 | 0.113 | ✓ | ✓ | ✓ | 0.082 | 0.157 | ✓ | ✓ | ✓ | 3.258 | 0.184 | ★ | ✓ | ✓ | 0.937 | 0.019 | ✓ | ✓ | ✗ |
| EnCodec |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Avg | Std | D | S | F | Avg | Std | D | S | F | Avg | Std | D | S | F | Avg | Std | D | S | F |
| 0.061 | 0.131 | ✗ | ✗ | ✗ | 0.090 | 0.158 | ✗ | ✗ | ✗ | 3.078 | 0.201 | ✗ | ✗ | ✗ | 0.920 | 0.017 | ✗ | ✗ | ✗ |
| SpeechTokenizer |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Avg | Std | E | D | F | Avg | Std | E | D | F | Avg | Std | E | D | F | Avg | Std | E | D | F |
| 0.060 | 0.139 | ✓ | ✗ | ✗ | 0.089 | 0.166 | ✓ | ✗ | ✗ | 3.087 | 0.190 | ✓ | ✗ | ✗ | 0.923 | 0.021 | ✓ | ✗ | ✗ |
| FACodec |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Avg | Std | E | S | D | Avg | Std | E | S | D | Avg | Std | E | S | D | Avg | Std | E | S | D |
| 0.057 | 0.123 | ✓ | ✓ | ✗ | 0.086 | 0.163 | ✓ | ✓ | ✗ | 3.129 | 0.250 | ✓ | ✓ | ✗ | 0.949 | 0.923 | ✓ | ✓ | ✓ |

### 4.2 Significance Analysis of Speech Tokenizer performance

We conducted a significance analysis at $\alpha=0.05$, following the approach of Dror et al. (), to measure the stochastic dominance of DM-Codec over the baselines: EnCodec, SpeechTokenizer, and FACodec. Specifically, we computed inverse cumulative distribution functions (CDFs) for all reconstructed speech samples’ individual WER, WIL, ViSQOL, and STOI scores. Notably, the average WER and WIL are calculated from each sentence individually, while the Table 1 scores are calculated by concatenating all sentences into one. Significance was evaluated using the $\epsilon$ value and categorized as: significantly better when $0.0<\epsilon\leq 0.5$, significantly dominant when $\epsilon=0.0$, and not significantly better when $\epsilon>0.5$. For this analysis, we selected DM-Codec^$\spadesuit$, trained with combined LM and SM-guided distillation. To the best of our knowledge, we are the first to conduct significance analysis to measure the effectiveness of different speech tokenizers.

Results and Discussion: The results in Table 2 show that DM-Codec significantly outperforms the baselines in WER, WIL, ViSQOL, and STOI scores. The improved average values (0.053 WER, 0.082 WIL, 3.258 ViSQOL, 0.937 STOI) and consistent standard deviations (0.113 WER, 0.157 WIL, 0.193 ViSQOL, 0.019 STOI) further demonstrate the statistical significance. Notably, DM-Codec’s performance in WER and WIL underscores the importance of contextual representation distillation for enhanced speech reconstruction. Additionally, its dominance in ViSQOL and STOI, especially over EnCodec, highlights the benefits of combining LM and SM distillation for retaining semantic-acoustic fidelity. While DM-Codec does not achieve significant dominance over FACodec in terms of STOI, it significantly outperforms the baselines across all other metrics. Among the baseline, however, FACodec achieves improved results over EnCodec and SpeechTokenizer, whereas SpeechTokenizer surpasses EnCodec in performance.

### 4.3 Ablation Studies

We conducted a thorough analysis of DM-Codec’s performance and the impact of each methodological choice in LM-guided and combined LM and SM-guided distillation. Unless otherwise stated, we use distillation for both LM and SM from the first Residual Vector Quantizer layer (RVQ-1) for comparison consistency and simplicity.

#### 4.3.1 Ablation Study: Impact of Combined Semantic Distillation

**Table 3: Effects of weights on combined representation distillation: Higher LM weight enhances content preservation, leading to lower WER. $\lambda_{SM}$ is the SM weight, $\lambda_{LM}$ is the LM weight.**
| $\lambda_{SM}$ | $\lambda_{LM}$ | WER $\downarrow$ |
| --- | --- | --- |
| 1.0 | 0.0 | 4.83 |
| 0.9 | 0.1 | 4.63 |
| 0.8 | 0.2 | 4.44 |
| 0.7 | 0.3 | 4.23 |
| 0.6 | 0.4 | 4.76 |
| 0.5 | 0.5 | 4.18 |
| 0.4 | 0.6 | 4.54 |
| 0.3 | 0.7 | 4.34 |
| 0.2 | 0.8 | 4.07 |
| 0.1 | 0.9 | 4.33 |
| 0.0 | 1.0 | 4.36 |

We conducted experiments with different weighted combinations of LM and SM distillation loss to evaluate their impact on reducing WER. The combined distillation loss from Equation 3 was updated using SM and LM weights ($\lambda_{SM}$ and $\lambda_{LM}$), ranging from $0.0$ to $1.0$, with the constraint $\lambda_{SM}+\lambda_{LM}=1$.

$$ $\mathcal{L}_{LS}=\frac{1}{2}\left(\lambda_{SM}\cdot\mathcal{L}_{SM}+\lambda_{ LM}\cdot\mathcal{L}_{L}\right)$ (11) $$

Results and Discussion: The experimental results are presented in Table 3, showing the speech reconstruction results with WER scores for different weighted combinations. From the values, we notice a trend showing that incorporating LM representations significantly improves WER, especially when LM distillation is dominant. The lowest WER score of 4.07 occurs with a weight of $\lambda_{LM}=0.8$ for LM, while $\lambda_{SM}=0.2$ for SM, highlighting the strong influence of LM distillation on capturing contextual information. A balanced weighting of $\lambda_{SM}=0.5$ and $\lambda_{LM}=0.5$ produces a WER of 4.18, confirming that distillation from both LM and SM is beneficial. However, as the weighting shifts more in favor of SM ($\lambda_{SM}>0.7$), WER deteriorates, reaching 4.83 when relying entirely on SM. This underscores that over-reliance on SM distillation compromises contextual accuracy in favor of raw speech features. Thus, an LM-dominant approach yields optimal results, while using SM alone is less effective in preserving content.

#### 4.3.2 Ablation Study: Impact of Distillation on Different RVQ Layers

We evaluated the effect of applying distillation at various Residual Vector Quantizer (RVQ) layers, including the first layer (RVQ-1), the average of eight layers (RVQ-1:8), and the last layer (RVQ-8). Table 4 shows the full results.

Results and Discussion: In LM-guided distillation, RVQ-1:8 achieves the best WER and WIL scores (4.23 and 6.94), though with lower ViSQOL and STOI scores (3.12 and 0.929) compared to RVQ-8 (3.28 and 0.935). The RVQ-1 layer provides the best overall balance between content preservation and perceptual quality, with WER, WIL, ViSQOL, and STOI scores of 4.36, 7.06, 3.18, and 0.935. This demonstrates RVQ-1:8 prioritizes contextual integrity, while RVQ-8 favors perceptual quality. Thus, we select RVQ-1 for LM-guided distillation due to its balanced performance.

For LM and SM-based distillation, the RVQ-1 and RVQ-1:8 combination achieves the best WER and WIL scores (4.05 and 6.61), with RVQ-1 and RVQ-1 as the second-best (4.18 and 6.84). In contrast, the RVQ-1 and RVQ-8 combination yields the highest ViSQOL and STOI scores (3.33 and 0.939), followed by RVQ-8 and RVQ-1 (3.30 and 0.938). RVQ-1 captures contextual representation more effectively due to its simpler quantized vector, while RVQ-1:8 incorporates more nuanced semantic and acoustic aspects. Overall, this ablation shows that selecting RVQ layers for LM and SM-based distillation greatly affects the balance between contextual accuracy and semantic-acoustic fidelity, allowing layer combinations to be tailored to task requirements.

**Table 4: Analysis of different RVQ layers effect on speech reconstruction. LM-guided distillation on RVQ-1 layer ensures greater content preservation, while SM-guided distillation on RVQ-1:8 layer is more effective at preserving semantic representation. LM-layer and SM-layer indicate the RVQ layer used for respective distillation. ^$\clubsuit$ indicates LM-guided Distillation. ^$\spadesuit$ indicates combined LM and SM-guided Distillation. Bold highlights the best result and underline the second-best result.**
| Tokenizer | LM-Layer | SM-Layer | WER $\downarrow$ | WIL $\downarrow$ | ViSQOL $\uparrow$ | STOI $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| DM-Codec ^$\clubsuit$ | RVQ-1 | - | 4.36 | 7.06 | 3.18 | 0.935 |
| DM-Codec ^$\clubsuit$ | RVQ-1:8 | - | 4.23 | 6.94 | 3.12 | 0.929 |
| DM-Codec ^$\clubsuit$ | RVQ-8 | - | 4.44 | 7.22 | 3.28 | 0.935 |
| DM-Codec ^$\spadesuit$ | RVQ-1 | RVQ-1 | 4.18 | 6.84 | 3.13 | 0.933 |
| DM-Codec ^$\spadesuit$ | RVQ-1:8 | RVQ-1 | 4.59 | 7.34 | 3.21 | 0.937 |
| DM-Codec ^$\spadesuit$ | RVQ-8 | RVQ-1 | 4.49 | 7.24 | 3.30 | 0.938 |
| DM-Codec ^$\spadesuit$ | RVQ-1 | RVQ-1:8 | 4.05 | 6.61 | 3.26 | 0.937 |
| DM-Codec ^$\spadesuit$ | RVQ-1 | RVQ-8 | 4.39 | 7.08 | 3.33 | 0.939 |

#### 4.3.3 Ablation Study: Impact of Different Models on Distillation

We experimented with different LM and SM distillations to analyze performance variations based on different model selections. In addition to our selected BERT (Devlin et al., ) and HuBERT (Hsu et al., ), we experiment with ELECTRA (electra-base-discriminator) (Clark et al., ) as the LM and wav2vec 2.0 (wav2vec2-base-960h) (Baevski et al., ) as the SM. Table 5 shows the full results.

Results and Discussion: In LM-guided distillation, the ELECTRA model significantly enhances performance, achieving WER and WIL scores of 4.12 and 6.63, respectively, compared to BERT’s scores of 4.36 and 7.06. This indicates the architecture of ELECTRA’s effectiveness for the proposed LM-guided distillation, demonstrating its superior contextual representation. These results are consistent with ELECTRA’s better performance in general natural language processing tasks. However, we select BERT for its simplicity and established performance.

In LM and SM-guided distillation, the combination of BERT and wav2vec 2.0 achieves the highest overall performance, with scores of WER 4.13, WIL 6.77, ViSQOL 3.15, and STOI 0.942. However, the combination of BERT and HuBERT closely follows with second-best scores of WER 4.18, WIL 6.84, and ViSQOL 0.933. These findings demonstrate that different speech models can be effectively integrated with the BERT model.

**Table 5: Analysis of representation distillation from different models. BERT can be effectively combined with HuBERT or wav2vec 2.0, however, ELECTRA in LM-guided distillation outperforms BERT. ^$\clubsuit$ indicates LM-guided Distillation. ^$\spadesuit$ indicates combined LM and SM-guided Distillation. Bold highlights the best result and underline the second-best result.**
| Tokenizer | LM | SM | WER $\downarrow$ | WIL $\downarrow$ | ViSQOL $\uparrow$ | STOI $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| DM-Codec ^$\clubsuit$ | BERT | - | 4.36 | 7.06 | 3.18 | 0.935 |
| DM-Codec ^$\clubsuit$ | ELECTRA | - | 4.12 | 6.63 | 3.10 | 0.936 |
| DM-Codec ^$\spadesuit$ | BERT | HuBERT | 4.18 | 6.84 | 3.13 | 0.933 |
| DM-Codec ^$\spadesuit$ | BERT | wav2vec 2.0 | 4.13 | 6.77 | 3.15 | 0.942 |
| DM-Codec ^$\spadesuit$ | ELECTRA | wav2vec 2.0 | 4.70 | 7.51 | 3.14 | 0.933 |
| DM-Codec ^$\spadesuit$ | ELECTRA | HuBERT | 4.67 | 7.58 | 2.94 | 0.932 |

#### 4.3.4 Ablation Study: Impact of Different Distillation Layer(s)

We evaluated speech reconstruction using different distillation layers of the LM and SM, examining which combination of layers yields the most relevant representations of semantic and contextual information. For this ablation, we considered the average of all layer representations, the $9^{\text{th}}$ layer representations, and the last layer representations. Table 6 shows the full results.

Results and Discussion: In LM-guided distillation, the use of the average layer achieves superior overall performance, with a WER of 4.36, WIL of 7.06, ViSQOL of 3.18, and STOI of 0.935, compared to the variants utilizing the last and 9th layers. Similarly, in LM and SM-guided distillation, the average layer yields superior results compared to the last and $9^{\text{th}}$ layer variants.

The results indicate that averaging all layers leads to more comprehensive representations of semantic or contextual information. In the case of LM, the averaging process provides greater contextual representation and synergizes syntactic information from earlier layers and abstract word relations from higher layers. In combined LM and SM-guided distillation, averaging all SM layers provides a more nuanced understanding of the earlier layer’s phonetic information and the higher layers’ richer semantic information. Conversely, relying solely on the last layer or the $9^{\text{th}}$ layer fails to capture the overall context and semantic information, yielding less relevant representation distillation.

**Table 6: Analysis of different distillation layers representation on speech reconstruction. Average layer provides more comprehensive representations. ^$\clubsuit$ indicates LM-guided Distillation. ^$\spadesuit$ indicates combined LM and SM-guided Distillation. Bold highlights the best result and underline the second-best result.**
| Tokenizer | Distillation Layer(s) | WER $\downarrow$ | WIL $\downarrow$ | ViSQOL $\uparrow$ | STOI $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| DM-Codec ^$\clubsuit$ | Average | 4.36 | 7.06 | 3.18 | 0.935 |
| DM-Codec ^$\clubsuit$ | Last | 4.62 | 7.56 | 2.95 | 0.926 |
| DM-Codec ^$\clubsuit$ | $9^{\text{th}}$ | 4.75 | 7.80 | 2.88 | 0.925 |
| DM-Codec ^$\spadesuit$ | Average | 4.18 | 6.84 | 3.13 | 0.933 |
| DM-Codec ^$\spadesuit$ | Last | 4.68 | 7.55 | 3.03 | 0.933 |
| DM-Codec ^$\spadesuit$ | $9^{\text{th}}$ | 4.52 | 7.43 | 3.00 | 0.933 |

## 5 Related Work

Tokenization Techniques in Speech. Tokenization in speech processing can be broadly categorized into two main approaches: (i) speech encoder-based and (ii) language-based. In the speech encoder-based tokenization approach, a pre-trained speech encoder provides audio representations. These representations are then used to guide the training model, either through an alignment network (Messica & Adi, ) or by optimizing specific losses (Zhang et al., ; Liu et al., ). The language-based tokenization approach involves processing audio through a speech encoder to obtain discrete representations, or using the corresponding text to feed into a language model. This has been explored by (Hassid et al., ), (Wang et al., ),(Zhang et al., ), and (Zhang et al., ). Recently, LAST (Turetzky & Adi, ), explored a language model to tokenize speech toward improved sequential modeling, using the LLM to perform the next token prediction of quantized vectors. However, these approaches significantly differ from our language representations distillation method and do not focus on combining multimodal representations.

Discrete Speech Representation. There are two well-known methods for discrete speech representation: semantic tokens and acoustic tokens. Semantic tokens are derived through self-supervised learning (SSL) techniques for speech (Baevski et al., ; Hsu et al., ; Chung et al., ) and capture abstract, high-level features that relate to general, symbolic aspects of speech, while omitting details related to speaker identity and acoustic characteristics. In contrast, acoustic tokens are obtained using neural audio codecs (Zeghidour et al., ; Défossez et al., ; Yang et al., ) and focus on delivering precise reconstructions of acoustic features. However, recent models (Turetzky & Adi, ; Liu et al., ; Shi et al., ) have shown that speech models based on self-supervised learning (SSL) are effective at extracting acoustic representations where LMs be employed to refine these models further, enhancing their ability to extract more nuanced semantic representations.

Textual Language Models in Speech. Research on speech models, including works by (Nguyen et al., ), (Borsos et al., ), and (Kharitonov et al., ), has focused on utilizing raw audio to extract prosodic features, identify speaker characteristics, and generate audio without depending on textual features or supervision from textual LMs. In contrast, many newer methods have started using audio encoders to transform audio signals into discrete tokens, which can be processed by textual LMs. TWIST method introduced by (Hassid et al., ) initializes the weights of the SpeechLM using a pre-trained text LM, showing that this combination significantly improves performance. Similarly, the SELM model developed by (Wang et al., ) leverages GPT (Radford, ; Radford et al., ) as its foundation due to its enhanced parallel processing capabilities and capacity. However, text-based LLMs such as GPT-3 (Brown, ) and Llama (Touvron et al., ) are essential for speech modeling. Once discrete audio representations are obtained, these large text models are trained to align with or enhance the original text embedding space, as explored in studies by (Zhang et al., ), (Fathullah et al., ), (Shu et al., ), and (Rubenstein et al., ). This trend of integrating textual LMs into speech modeling has become increasingly popular in recent research.

## 6 Limitations and Broader Impact

Limitations. In this work, we present the effectiveness of our proposed method, DM-Codec, based on the LibriSpeech dataset. Future research could investigate its performance across a variety of datasets and domains. Additionally, exploring the capabilities of DM-Codec in multilingual contexts would be valuable. Another limitation of our work is the absence of experiments with emerging LLMs. Currently, we focus solely on masked language models to derive representations. Further investigation into these decoder-based LLMs’ impact on DM-Codec can be studied and addressed.

Broader Impact. The integration of language models in speech processing has traditionally focused on model-specific implementations or specific training objectives. In this work, we propose a novel approach by leveraging a language model during the tokenization phase through our model, DM-Codec. By incorporating language-specific representations from the corresponding text, DM-Codec enhances the quality of discrete speech representations. This method bridges the gap between language and speech models, offering a more unified approach to multimodal representation learning. DM-Codec provides a robust framework for generating high-quality audio representations, with potential applications in various domains, including multilingual speech processing, low-resource languages, and other audio-related tasks. Our findings pave the way for more effective and contextually aware speech processing models, contributing to advancements in the broader field of speech and language technologies.

## 7 Conclusion

In this work, we introduced a speech tokenizer DM-Codec, with two novel distillation methods to leverage multimodal (acoustic, semantic, and contextual) representations from a language model and speech self-supervised learning model. Our extensive experimental results and ablation studies suggest that distilling multimodal representations enables DM-Codec to introduce salient speech information in discrete speech tokens. Our significance analysis further revealed that DM-Codec with comprehensive multimodal representations consistently outperforms existing speech tokenizers. This approach highlights the potential of multimodal representations to enhance speech tokenization in various domains, including multilingual and code-switched speech processing.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Most of my questions are covered in the weaknesses, please see above.

In addition, here are some minor questions/suggestions:
- The term speech-to-text (STT) used in the paper, although understandable, is not very common in the field. Is there a reason to not use the more common term automatic speech recognition (ASR)? 
- It would be a good improvement if human evaluation can be involved in the next version.
soundness: 2
strengths: The only strength of this paper is the idea of leveraging a text-based language model to improve audio codec, which is reasonable and interesting. That being said, the idea is not properly executed in this work (see weaknesses).
summary: This paper introduces DM-Codec, a speech tokenizer based on SpeechTokenizer (that distills speech self-supervised model, SM, to audio codec) that additionally distills representations from text LM to audio codec. The DM-Codec architecture uses a streamlined encoder-decoder structure with a Residual Vector Quantizer (RVQ), incorporating LM and SM representations only during training. Evaluation is done by comparing it against existing codec models on the quality of decoded speech.
weaknesses: - **Novelty**: DM-codec is essentially SpeechTokenizer with additional LM distillation, which is somewhat incremental.
- **Technical correctness**: While the extra text-based target embedding is the only novel component, it's unclear how the text representation and the acoustic representation are aligned. Text embeddings and acoustic embeddings are clearly **not** sharing the same length, but this work did not provide any detail on this. Instead, it uses a misleading notation $n$ to indicate the sequence length of both text and speech representation.
- **Depth of the study**: The content of the experiment is quite monotonous. It's basically repeating a single set of training and evaluation with different hyper-parameters, loss weights, and backbones. There is no in-depth analysis. For example, how LM distillation can affect bit rate (which is a very common metric for audio codecs), how well the first VQ layer aligns to text after training, does DM-codec possess speaker disentangle ability due to the text distillation (similar to SpeechTokenizer), etc.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: •	In equation 1, for a given speech x, the length of S(:,d) is the token length of text x’ output from LM (n in figure 2). The length of Q(:,d) should be the time length (T’ in figure 2). How to make sure they are the same?  
•	There are several weighting factors in equation 10, how are they decided except LM/SM weights?
•	In the paper of SpeechTokenizer , the speaker similarity also been evaluated. Why isn’t this evaluated in this paper with the proposed method? Do the author think adding LM would increase or decrease the speaker similarity?
soundness: 3
strengths: The proposed DM-Codec method improved the existing speech tokenization system which only incorporates acoustic and semantic information (e.g SpeechTokenizer)  by adding textual representations via an LM-guided distillation method.  The contextual information is learned with continuous representation distillation technique, and it’s then combined with the speech self-supervised learning model  (SM) guided distillation. The proposed method is compared with several existing methods, including EnCodec, SpeechTokenizer  and FACodec. The experimental results prove its advantage over these baseline systems. And the ablation studies provide a thorough analysis of the performance of the proposed method.
summary: In this work, a speech tokenizer method called DM-Codec is introduced. In this method, based on the encoder-decoder framework with a Residual Vector Quantizer (RVQ), two novel distillation methods are used to leverage multimodal (acoustic, semantic, and contextual) representations from a language model and speech self-supervised learning model. The experimental results showed the proposed method significantly outperforms state-of-the-art speech tokenization models, reducing WER by up to 13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility by 1.85% on the LibriSpeech benchmark dataset.
weaknesses: •	The testing set is small. Only 300 audio samples are randomly selected from the LibriSpeech test subset as the evaluation set. Although the author explained that this is to make it consistent with the baseline, but 300 audios are not big enough to get a statistically meaningful conclusion. The results in table 3 maybe a proof of this: We couldn’t get obvious relation between WER and SM/LM loss weights based on these results.  E.g. the author said LM information is more helpful for lower WER but we could see LM weight = 0.9 get higher WER than LM weight=0.3/0.5. 
•	A lot of metrics results are given including the ablation study, some of them are confusing: 
o	The lowest WER in table 3 is 4.07, but it’s 4.05 in table 1. It’s not clear what’s the difference between them. 
o	The best results of WER, WIL, ViSQOL and STOI for system using distillation for both LM and SM in table 1 is 4.05/6.61/3.26/0.937. These results should be obtained by distilling semantic representation with RVQ layers 1 to 8 according to table 4. But in  table 5 and 6, the results for combining LM/SM-guided distillation are only with RVQ layer 1. Why didn’t the author use the best option in table 5 and 6? 
o	In table 5, the results of two LM, BERT and ELECTRA, have similar accuracy for LM-guided distillation only system. But for combined LM and SM-guided Distillation systems, the results are more different, the author didn’t give analysis about the reason for this.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Small suggestion: Table 3 can be replaced by a line chart
soundness: 2
strengths: This work first incorporates contextual representations via an LM-guided distillation method, and it enhancs the retention of acoustic and speech information in reconstructed speech.
summary: This paper introduce a new speech tokenization model that combines acoustic, semantic, and contextual information into a comprehensive multimodal speech representation. The primary contribution lies in DM-Codec's ability to integrate contextual representations from language models and self-supervised speech models to improve speech tokenization accuracy and quality. DM-Codec’s approach highlights the value of integrating multimodal data for speech tokenization, promising advances in speech quality, intelligibility, and contextual preservation in applications.
weaknesses: I do not fully agree that contextual representation should hold equal importance to acoustic and semantic representations. The improved intelligibility of DM-Codec is mainly due to an additional teacher LM serving a distillation role within the RVQ. However, this LM relies on an ASR system (M_STT in Fig2) for transcription. Given this setup, it is difficult to ascertain whether the improvements are driven by 
the Bert LM or by the M_STT ASR system. More importantly, the reason audio codecs are being actively explored today is that they serve as a discretization tool designed for audio LLMs or TTS systems. Unfortunately, I did not see any extension related to this aspect in the paper. In other words, if DM-Codec is based solely on the concept of contextual representation, I believe it would be challenging to consider it a preferred choice when designing TTS or audio LLM systems.
Additionally, from the technical contribution perspective, this work essentially adds an extra LM distillation on top of the SpeechTokenizer. I think this may struggle to meet the acceptance threshold for ICLR.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: As mentioned in the Weakness Section, I would like to ask the authors to explain and justify why the LM-guided distillation method is useful even though it seems like the speech and LM representations are not aligned in time. All the issues listed in the Weakness Section must be addressed or answered in order to get a higher rating.
soundness: 1
strengths: The authors conducted significance tests, an important analysis that demonstrated the performance gap between methods but was not included in most prior literature. Moreover, the ablation studies are comprehensive and cover most aspects of the method design.
summary: This paper proposes DM-Codec, a neural codec to tokenize speech via knowledge distillation from language models for better contextualized representations. The experimental results indicate superior speech reconstruction performance on several metrics compared with prior methods.
weaknesses: 1) **LM Guided Distillation (learning target):**  
According to the text, the authors did not explicitly mention how they aligned the codec encoder features with the LM hidden representations. It is a commonly known fact that transcribed speech has a significantly shorter utterance length, especially when the transcription is tokenized as words or subword units. Hence, the LM representations of the transcriptions must be shorter than the codec encoder output, leading to a length mismatch when computing the distillation loss. By looking through the code provided in the supplementary materials, line 107 of `extract_rep.py` pads the transcriptions to the length of the logits produced by the wav2vec2-CTC model, but the BERT-based LM only processes the unpadded part of the text sequence, making the actual LM representations consist of a small part of the encoded sequence. Thus, without further speech and text alignment techniques, it is difficult to justify the effectiveness of the LM-guided distillation targets. Also, the distillation loss in Eq. (1) was first proposed by Chang et al. [1], as indicated in the SpeechTokenizer paper.
2) **LM Guided Distillation (necessity):**  
In addition to the unaligned learning targets for this distillation approach, the use of a supervised ASR model in this method (wav2vec2-CTC) raises another question: can the transcription (either ground truth or the ASR) directly be used as a supervised target? For instance, CTC ASR can be performed with the RVQ representations as an additional loss. Furthermore, it is still unclear how the contextualized representations relate to speech reconstruction and other applications.
3) **Evaluation tasks and metrics:**  
For the speech reconstruction quality evaluation, some commonly used metrics are not reported like mean opinion scores (MOS). Simply reporting automatic assessment results like UTMOS [2] is also acceptable, as they appear in many prior works [3]. When listening to the audio provided in the supplementary materials, I found SpeechTokenizer has a better reconstruction quality, which implies the necessity of including MOS. Besides speech reconstruction, other downstream applications of the proposed speech tokenizer are not considered in this paper. For instance, ASR, TTS, and spoken language modeling could be used.
4) **Baseline comparison:**  
The authors missed a very simple baseline, which is DM-Codec without any distillation objectives. Because the implementation and training data of previous tokenizers like EnCodec and SpeechTokenizer might differ from this paper's, the authors must include this baseline to demonstrate the effectiveness of the proposed distillation methods.

[1] Chang, Heng-Jui, Shu-wen Yang, and Hung-yi Lee. "Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.  
[2] Saeki, Takaaki, et al. "Utmos: Utokyo-sarulab system for voicemos challenge 2022." arXiv preprint arXiv:2204.02152 (2022).  
[3] Chang, Xuankai, et al. "The Interspeech 2024 Challenge on Speech Processing Using Discrete Units." arXiv preprint arXiv:2406.07725 (2024).