DM-Codec: Distilling Multimodal Representations for Speech Tokenization

Authors: Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, AKMMAHBUBUR RAHMAN, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: speech tokenizer, speech codec, multimodal representation learning, representation learning, language model

Abstract:
Recent advancements in speech-language models have yielded significant improvements in speech tokenization and synthesis. However, effectively mapping the complex, multidimensional attributes of speech into discrete tokens remains challenging. This process demands acoustic, semantic, and contextual information for precise speech representations. Existing speech representations generally fall into two categories: acoustic tokens from audio codecs and semantic tokens from speech self-supervised learning models. Although recent efforts have unified acoustic and semantic tokens for improved performance, they overlook the crucial role of contextual representation in comprehensive speech modeling. Our empirical investigations reveal that the absence of contextual representations results in elevated Word Error Rate (WER) and Word Information Lost (WIL) scores in speech transcriptions. To address these limitations, we propose two novel distillation approaches: (1) a language model (LM)-guided distillation method that incorporates contextual information, and (2) a combined LM and self-supervised speech model (SM)-guided distillation technique that effectively distills multimodal representations (acoustic, semantic, and contextual) into a comprehensive speech tokenizer, termed DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM during the training process. Experiments show DM-Codec significantly outperforms state-of-the-art speech tokenization models, reducing WER by up to 13.46\%, WIL by 9.82\%, and improving speech quality by 5.84\% and intelligibility by 1.85\% on the LibriSpeech benchmark dataset.

Decision: reject
Reviewer scores: 1, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729544528597

thread_type:
official_review

<thread_object>
creator: reviewer_qrxh

<content>
summary:
This paper proposes DM-Codec, a neural codec to tokenize speech via knowledge distillation from language models for better contextualized representations. The experimental results indicate superior speech reconstruction performance on several metrics compared with prior methods.

strengths:
The authors conducted significance tests, an important analysis that demonstrated the performance gap between methods but was not included in most prior literature. Moreover, the ablation studies are comprehensive and cover most aspects of the method design.

weaknesses:
1) **LM Guided Distillation (learning target):**  
According to the text, the authors did not explicitly mention how they aligned the codec encoder features with the LM hidden representations. It is a commonly known fact that transcribed speech has a significantly shorter utterance length, especially when the transcription is tokenized as words or subword units. Hence, the LM representations of the transcriptions must be shorter than the codec encoder output, leading to a length mismatch when computing the distillation loss. By looking through the code provided in the supplementary materials, line 107 of `extract_rep.py` pads the transcriptions to the length of the logits produced by the wav2vec2-CTC model, but the BERT-based LM only processes the unpadded part of the text sequence, making the actual LM representations consist of a small part of the encoded sequence. Thus, without further speech and text alignment techniques, it is difficult to justify the effectiveness of the LM-guided distillation targets. Also, the distillation loss in Eq. (1) was first proposed by Chang et al. [1], as indicated in the SpeechTokenizer paper.
2) **LM Guided Distillation (necessity):**  
In addition to the unaligned learning targets for this distillation approach, the use of a supervised ASR model in this method (wav2vec2-CTC) raises another question: can the transcription (either ground truth or the ASR) directly be used as a supervised target? For instance, CTC ASR can be performed with the RVQ representations as an additional loss. Furthermore, it is still unclear how the contextualized representations relate to speech reconstruction and other applications.
3) **Evaluation tasks and metrics:**  
For the speech reconstruction quality evaluation, some commonly used metrics are not reported like mean opinion scores (MOS). Simply reporting automatic assessment results like UTMOS [2] is also acceptable, as they appear in many prior works [3]. When listening to the audio provided in the supplementary materials, I found SpeechTokenizer has a better reconstruction quality, which implies the necessity of including MOS. Besides speech reconstruction, other downstream applications of the proposed speech tokenizer are not considered in this paper. For instance, ASR, TTS, and spoken language modeling could be used.
4) **Baseline comparison:**  
The authors missed a very simple baseline, which is DM-Codec without any distillation objectives. Because the implementation and training data of previous tokenizers like EnCodec and SpeechTokenizer might differ from this paper's, the authors must include this baseline to demonstrate the effectiveness of the proposed distillation methods.

[1] Chang, Heng-Jui, Shu-wen Yang, and Hung-yi Lee. "Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert." ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.  
[2] Saeki, Takaaki, et al. "Utmos: Utokyo-sarulab system for voicemos challenge 2022." arXiv preprint arXiv:2204.02152 (2022).  
[3] Chang, Xuankai, et al. "The Interspeech 2024 Challenge on Speech Processing Using Discrete Units." arXiv preprint arXiv:2406.07725 (2024).

questions:
As mentioned in the Weakness Section, I would like to ask the authors to explain and justify why the LM-guided distillation method is useful even though it seems like the speech and LM representations are not aligned in time. All the issues listed in the Weakness Section must be addressed or answered in order to get a higher rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729855765459

thread_type:
official_review

<thread_object>
creator: reviewer_ohpz

<content>
summary:
This paper introduce a new speech tokenization model that combines acoustic, semantic, and contextual information into a comprehensive multimodal speech representation. The primary contribution lies in DM-Codec's ability to integrate contextual representations from language models and self-supervised speech models to improve speech tokenization accuracy and quality. DM-Codec’s approach highlights the value of integrating multimodal data for speech tokenization, promising advances in speech quality, intelligibility, and contextual preservation in applications.

strengths:
This work first incorporates contextual representations via an LM-guided distillation method, and it enhancs the retention of acoustic and speech information in reconstructed speech.

weaknesses:
I do not fully agree that contextual representation should hold equal importance to acoustic and semantic representations. The improved intelligibility of DM-Codec is mainly due to an additional teacher LM serving a distillation role within the RVQ. However, this LM relies on an ASR system (M_STT in Fig2) for transcription. Given this setup, it is difficult to ascertain whether the improvements are driven by 
the Bert LM or by the M_STT ASR system. More importantly, the reason audio codecs are being actively explored today is that they serve as a discretization tool designed for audio LLMs or TTS systems. Unfortunately, I did not see any extension related to this aspect in the paper. In other words, if DM-Codec is based solely on the concept of contextual representation, I believe it would be challenging to consider it a preferred choice when designing TTS or audio LLM systems.
Additionally, from the technical contribution perspective, this work essentially adds an extra LM distillation on top of the SpeechTokenizer. I think this may struggle to meet the acceptance threshold for ICLR.

questions:
Small suggestion: Table 3 can be replaced by a line chart
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604998302

thread_type:
official_review

<thread_object>
creator: reviewer_l3he

<content>
summary:
In this work, a speech tokenizer method called DM-Codec is introduced. In this method, based on the encoder-decoder framework with a Residual Vector Quantizer (RVQ), two novel distillation methods are used to leverage multimodal (acoustic, semantic, and contextual) representations from a language model and speech self-supervised learning model. The experimental results showed the proposed method significantly outperforms state-of-the-art speech tokenization models, reducing WER by up to 13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility by 1.85% on the LibriSpeech benchmark dataset.

strengths:
The proposed DM-Codec method improved the existing speech tokenization system which only incorporates acoustic and semantic information (e.g SpeechTokenizer)  by adding textual representations via an LM-guided distillation method.  The contextual information is learned with continuous representation distillation technique, and it’s then combined with the speech self-supervised learning model  (SM) guided distillation. The proposed method is compared with several existing methods, including EnCodec, SpeechTokenizer  and FACodec. The experimental results prove its advantage over these baseline systems. And the ablation studies provide a thorough analysis of the performance of the proposed method.

weaknesses:
•	The testing set is small. Only 300 audio samples are randomly selected from the LibriSpeech test subset as the evaluation set. Although the author explained that this is to make it consistent with the baseline, but 300 audios are not big enough to get a statistically meaningful conclusion. The results in table 3 maybe a proof of this: We couldn’t get obvious relation between WER and SM/LM loss weights based on these results.  E.g. the author said LM information is more helpful for lower WER but we could see LM weight = 0.9 get higher WER than LM weight=0.3/0.5. 
•	A lot of metrics results are given including the ablation study, some of them are confusing: 
o	The lowest WER in table 3 is 4.07, but it’s 4.05 in table 1. It’s not clear what’s the difference between them. 
o	The best results of WER, WIL, ViSQOL and STOI for system using distillation for both LM and SM in table 1 is 4.05/6.61/3.26/0.937. These results should be obtained by distilling semantic representation with RVQ layers 1 to 8 according to table 4. But in  table 5 and 6, the results for combining LM/SM-guided distillation are only with RVQ layer 1. Why didn’t the author use the best option in table 5 and 6? 
o	In table 5, the results of two LM, BERT and ELECTRA, have similar accuracy for LM-guided distillation only system. But for combined LM and SM-guided Distillation systems, the results are more different, the author didn’t give analysis about the reason for this.

questions:
•	In equation 1, for a given speech x, the length of S(:,d) is the token length of text x’ output from LM (n in figure 2). The length of Q(:,d) should be the time length (T’ in figure 2). How to make sure they are the same?  
•	There are several weighting factors in equation 10, how are they decided except LM/SM weights?
•	In the paper of SpeechTokenizer , the speaker similarity also been evaluated. Why isn’t this evaluated in this paper with the proposed method? Do the author think adding LM would increase or decrease the speaker similarity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686068094

thread_type:
official_review

<thread_object>
creator: reviewer_nusp

<content>
summary:
This paper introduces DM-Codec, a speech tokenizer based on SpeechTokenizer (that distills speech self-supervised model, SM, to audio codec) that additionally distills representations from text LM to audio codec. The DM-Codec architecture uses a streamlined encoder-decoder structure with a Residual Vector Quantizer (RVQ), incorporating LM and SM representations only during training. Evaluation is done by comparing it against existing codec models on the quality of decoded speech.

strengths:
The only strength of this paper is the idea of leveraging a text-based language model to improve audio codec, which is reasonable and interesting. That being said, the idea is not properly executed in this work (see weaknesses).

weaknesses:
- **Novelty**: DM-codec is essentially SpeechTokenizer with additional LM distillation, which is somewhat incremental.
- **Technical correctness**: While the extra text-based target embedding is the only novel component, it's unclear how the text representation and the acoustic representation are aligned. Text embeddings and acoustic embeddings are clearly **not** sharing the same length, but this work did not provide any detail on this. Instead, it uses a misleading notation $n$ to indicate the sequence length of both text and speech representation.
- **Depth of the study**: The content of the experiment is quite monotonous. It's basically repeating a single set of training and evaluation with different hyper-parameters, loss weights, and backbones. There is no in-depth analysis. For example, how LM distillation can affect bit rate (which is a very common metric for audio codecs), how well the first VQ layer aligns to text after training, does DM-codec possess speaker disentangle ability due to the text distillation (similar to SpeechTokenizer), etc.

questions:
Most of my questions are covered in the weaknesses, please see above.

In addition, here are some minor questions/suggestions:
- The term speech-to-text (STT) used in the paper, although understandable, is not very common in the field. Is there a reason to not use the more common term automatic speech recognition (ASR)? 
- It would be a good improvement if human evaluation can be involved in the next version.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732567277368

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers (Part 1/4)

comment:
We sincerely thank the reviewers for their comprehensive feedback and thoughtful evaluation of our work. We greatly appreciate the recognition of the several critical contributions of our work. Specifically, the reviewers acknowledged the novel approach of integrating a text-based language model (LM) to enhance speech tokenization (*Reviewer nuSP*) and our method's advancement over existing systems by introducing contextual representations via LM-guided distillation (*Reviewer L3HE, ohpz*). We are particularly grateful for the recognition of our rigorous experimental design, including the comprehensive ablation studies, which thoroughly analyze the contributions of the proposed components (*Reviewer L3HE, qRxh*), and the inclusion of statistical significance tests, which emphasize the performance differences among the compared methods (*Reviewer qRxh*). Furthermore, we appreciate the acknowledgment of DM-Codec's ability to effectively retain both acoustic and speech information in reconstructed speech (*Reviewer ohpz*), a critical factor in improving the overall quality and intelligibility of speech tokenization systems.

Nevertheless, the reviewers expressed several concerns regarding our approach. We value the reviewers' constructive criticism and welcome the feedback. We will systematically address the specific concerns and constructive critiques to further strengthen our manuscript.

In this thread, we address the general concerns of the respected reviewers on our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732569841588

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New Experiments’ Results (Part 1/2)

comment:
## Table A: Speech Reconstruction Evaluation
DM-Codec♠ achieves the best performance in WER, WIL, ViSQOL, and MOS, highlighting its enhanced content preservation and speech quality, with competitive intelligibility results. ♡ means the results were reproduced using the official training code. ♢ means the results were obtained using official model checkpoints. ♣ indicates LM-guided Distillation method. ♠ indicates combined LM and SM-guided Distillation method. CLS indicates LM’s [CLS] token-based distillation method. Baseline indicates DM-Codec without any distillation method.

| Model                       	|   WER ↓ |   WIL ↓ | ViSQOL ↑ |  STOI ↑ |  MOS ↑  |
|---------------------------------|-------|-------|--------|-------|-------|
| DM-Codec (Baseline)         	|  4.97 |  8.02 |   2.95 | 0.935 |  3.13 |
| DM-Codec ♠                  	|  **4.05** |  **6.61** |   **3.26** | 0.937 |  **3.72** |
| DM-Codec ♣                  	|  4.36 |  7.06 |   3.18 | 0.935 |  3.69 |
| DM-Codec (CLS)      	|  4.47 |  7.08 |   3.12 | 0.926 |  3.65 |
| EnCodec ♦                   	|  4.53 |  7.17 |   3.08 | 0.920 |  3.09 |
| FACodec ♦                   	|  4.68 |  7.33 |   3.13 | **0.949** |  3.70 |
| SpeechTokenizer ♡           	|  4.49 |  7.10 |   3.09 | 0.923 |  3.67 |


&nbsp;
&nbsp;
&nbsp;


## Table B: Significance Analysis on Full Test Set
Significance analysis is conducted at α = 0.05 between LM and SM-guided DM-Codec (D), EnCodec (E), SpeechTokenizer (S), and FACodec (F). E/D, S/D, and F/D indicate comparisons with the baseline (e.g., D vs E, E vs S) or with DM-Codec when the row corresponds to the same baseline (e.g., E vs D, not E vs E). Results reveal DM-Codec consistently achieves significantly better scores in key metrics across all individual samples. A ✓ indicates significantly better, a ★ denotes dominance, and a ✗ means no significant improvement. Avg and Std mean the average and standard deviation of each score.

| Model       	| WER ↓ |   	| 	|  	| 	| WIL ↓ |   	| 	| 	| 	| ViSQOL ↑ |   	| 	| 	| 	| STOI ↑ |   	| 	| 	| 	|
|-----------------|-------|-------|-----|------|-----|-------|-------|-----|-----|-----|---------|-------|-----|-----|-----|--------|-------|-----|-----|-----|
|             	| Avg   | Std   | E/D | S/D  | F/D | Avg   | Std   | E/D | S/D | F/D | Avg 	| Std   | E/D | S/D | F/D | Avg	| Std   | E/D | S/D | F/D |
| DM-Codec ♠  	| 4.774 | 0.100 | ✓   | ✓	| ✓   | 7.510 | 0.139 | ✓   | ✓   | ✓   | 3.197   | 0.184 | ★   | ✓   | ✓   | 0.937  | 0.021 | ✓   | ✓   | ✗   |
| EnCodec     	| 4.828 | 0.100 | ✗   | ✓	| ✓   | 7.593 | 0.137 | ✗   | ✓   | ✓   | 3.064   | 0.201 | ✗   | ✗   | ✗   | 0.917  | 0.021 | ✗   | ✗   | ✗   |
| SpeechTokenizer | 4.942 | 0.101 | ✗   | ✗	| ✗   | 7.725 | 0.138 | ✗   | ✗   | ✗   | 3.080   | 0.190 | ✓   | ✗   | ✗   | 0.920  | 0.025 | ✓   | ✗   | ✗   |
| FACodec     	| 4.914 | 0.103 | ✗   | ✓	| ✗   | 7.643 | 0.141 | ✗   | ✓   | ✗   | 3.113   | 0.250 | ✓   | ✓   | ✗   | 0.946  | 0.023 | ✓   | ✓   | ✓   |

&nbsp;
&nbsp;
&nbsp;


## Table C: Neural Codec Based Text-to-Speech Model Evaluation
Dataset indicates the training dataset of the model (e.g., MLS (Eng) refers to the English subset of Multilingual LibriSpeech). ♢ denotes results obtained using officially shared model checkpoints. † means the results directly obtained
from the paper (i.e., USLM from SpeechTokenizer). The results demonstrate that DM-Codec-TTS achieves the best overall performance, preserving content information (WER, WIL, MOS) and achieving comparable or better speaker similarity (Similarity, SMOS). Notably, DM-Codec-TTS-small outperforms USLM_libri, trained on the same smaller dataset (LibriTTS).

&nbsp;
&nbsp;

### LibriSpeech Evaluation  
| Model                 	| Dataset   	|   WER ↓ |   WIL ↓ | Similarity  ↑ |   MOS  ↑|   SMOS  ↑ |
|---------------------------|---------------|-------|-------|------------|-------|--------|
| DM-Codec-TTS          	| Libriheavy	|  **5.08** |  **7.32** |   	**0.82** |  **3.70** |   3.89 |
| Vall-E †              	| Librilight	|  5.90 |   -   |   	0.58 |  - |   **4.38** |
| DM-Codec-TTS-small    	| LibriTTS  	| 10.26 | 13.79 |   	0.82 |  3.24 |   3.20 |
| USLM_libri ♢| LibriTTS  	| 16.72 | 25.65 |   	0.80 |  3.11 |   2.83 |

&nbsp;
&nbsp;

### VCTK Evaluation  
| Model                 	| Dataset   	|   WER ↓ |   WIL ↓ | Similarity  ↑ |   MOS  ↑|   SMOS  ↑ |
|---------------------------|---------------|-------|-------|------------|-------|--------|
| DM-Codec-TTS          	| Libriheavy	|  **3.58** |  **5.65** |   	0.82 |  **3.78** |   **3.85** |
| Vall-E †              	| Librilight	|   -   |   -   |   	0.38 |  - |   3.81 |
| USLM †                	| MLS (eng) 	|  6.50 |   -   |   	**0.84** |  3.63 |   3.45 |
| DM-Codec-TTS-small    	| LibriTTS  	|  5.02 |  8.21 |   	0.79 |  3.39 |   3.28 |
| USLM_libri ♢| LibriTTS  	| 14.79 | 23.24 |   	0.78 |  2.94 |   2.63 |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732800543126

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Uploaded

comment:
We thank all reviewers for their constructive and insightful efforts in evaluating this work. We have uploaded the revised manuscript with several modifications:

1. Revised INTRODUCTION with highlights of novel contributions;

2. Added [CLS] Token Guided Distillation Method and DM-Codec-TTS details in the PROPOSED METHOD;

3. Updated EXPERIMENTAL SETUP and EXPERIMENTAL RESULTS AND DISCUSSION sections based on the new suggested experiments by the reviewers;

4. Included human evaluation results in EXPERIMENTAL RESULTS and added details in the EXPERIMENTAL SETUP and Appendix;

5. Added a detailed discussion of the representation alignment issue and the irrelevance of temporal correspondence in our approach to the Appendix;

6. Updated the ablation studies with the newly suggested Impact of Low Bit Rate ablation.

**The above points are marked in blue (in both the main paper and Appendix).**

In addition to these points, we have also revised typos, references, figure captions, formulations, and all other minor points mentioned during the review process.

Thanks again for the constructive efforts in the comments and reviews.  
Authors
</content>
</thread_object>

</thread>


Metareview: