DMDSpeech: Distilled Diffusion Model Surpassing The Teacher in Zero-shot Speech Synthesis via Direct Metric Optimization

Authors: Yinghao Aaron Li, Rithesh Kumar, Zeyu Jin
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text-to-speech, zero-shot speech synthesis, diffusion model, diffusion distillation, metric optimization

Abstract:
Diffusion models have demonstrated significant potential in speech synthesis tasks, including text-to-speech (TTS) and voice cloning. However, their iterative denoising processes are inefficient and hinder the application of end-to-end optimization with perceptual metrics. In this paper, we propose a novel method of distilling TTS diffusion models with direct end-to-end evaluation metric optimization, achieving state-of-the-art performance.  By incorporating Connectionist Temporal Classification (CTC) loss and Speaker Verification (SV) loss, our approach optimizes perceptual evaluation metrics, leading to notable improvements in word error rate and speaker similarity. Our experiments show that DMDSpeech consistently surpasses prior state-of-the-art models in both naturalness and speaker similarity while being significantly faster. Moreover, our synthetic speech has a higher level of voice similarity to the prompt than the ground truth in both human evaluation and objective speaker similarity metric. This work highlights the potential of direct metric optimization in speech synthesis, allowing models to better align with human auditory preferences. The audio samples are available at https://dmdspeech.github.io/demo/.

Decision: reject
Reviewer scores: 8, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730176304879

thread_type:
official_review

<thread_object>
creator: reviewer_qwfw

<content>
summary:
This paper proposes a distilled TTS diffusion model using CTC and SV loss. While the paper is mostly well-written, it lacks novelty and includes some unfair comparisons. Firstly, distilled diffusion models are already well established in computer vision, with a substantial body of research. Even in audio and speech processing, the authors are not among the first to apply this approach, as seen in Nvidia’s blog [1] and other studies. Using SV loss for TTS is also not novel, as it has been previously applied in [3]. Thus, the primary novelty appears to lie in combining these techniques.
The claim of “DIRECT METRIC OPTIMIZATION” is misleading. In speech synthesis, there is no truly direct metric for evaluating generated speech quality. At best, we rely on MOS, while WER and speaker similarity measures serve only as indirect proxies. Generated speech may exhibit machine-like articulation yet achieve low WER. Additionally, speaker similarity measures can be highly domain-dependent and often perform poorly on out-of-domain speakers.
The comparisons made are not entirely fair, given that training with CTC and SV loss naturally results in improved WER and speaker similarity. For RTF evaluation, it seems the authors mainly compare their model with autoregressive models such as XTTS and VoiceCraft, which are not known for their speed. Moreover, the RTF values for DiTTO-TTS and CLaM-TTS are drawn from their respective papers, which is inaccurate since RTF must be measured on the same hardware to be comparable.
NaturalSpeech 3 is not open-source, making it difficult to assess whether its RTF in this paper reflects the original results. A more appropriate comparison would be with StyleTTS2, an open-source, state-of-the-art non-autoregressive model.


[1]https://developer.nvidia.com/blog/speeding-up-text-to-speech-diffusion-models-by-distillation/
[2]Bai, Yatong, et al. "Consistencytta: Accelerating diffusion-based text-to-audio generation with consistency distillation." arXiv preprint arXiv:2309.10740 (2024).
[3]E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. Golge, and ¨ M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” in International Conference on Machine Learning. P

strengths:
The paper is well written. The experiments are comprehensive although some are not fair comparison.

weaknesses:
The novelty is limited and the claim of directly optimization is simply not true. Some comparisons in experiments section is not fair.

questions:
Have the authors try other objective than CTC?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730236084942

thread_type:
official_review

<thread_object>
creator: reviewer_9zdq

<content>
summary:
This paper introduces DMDSpeech, a distilled diffusion model designed for efficient, high-quality zero-shot speech synthesis. It optimizes perceptual metrics by incorporating Connectionist Temporal Classification (CTC) and Speaker Verification (SV) losses, targeting improvements in word error rate (WER) and speaker similarity. The model outperforms previous state-of-the-art approaches in naturalness and speaker similarity, while achieving faster synthesis. This approach highlights the benefits of direct metric optimization in TTS and demonstrates the effectiveness of DMDSpeech in aligning generated speech with human auditory preferences.

strengths:
**Integration of CTC and SV losses** The paper optimize diffusion TTS model by using end-to-end metric optimization, applying CTC and SV losses to improve word error rate (WER) and speaker similarity.

**Model results** DMDSpeech achieves better performance compared to state-of-the-art baselines with significantly reduced inference time.

weaknesses:
**Limited Novelty of the Proposed Approach** DMDSpeech utilizes the existing DMD 2 method to distill the teacher model for faster sampling and direct metric optimization, except that DMD 2 simulates a four-step inference process during training, while DMDSpeech simulates a single step. Overall, this paper feels more like an application of DMD 2 in text-to-speech rather than an original methodological advancement.

**Need for Additional Experimental Validation** In DMDSpeech, the student generator is trained to match the teacher model's distribution via distribution matching distillation, but the results in Table 4 indicate that DMD 2 alone decreases MOS-N, SMOS, SMOS-S, and SIM scores compared to the teacher model, with MOS-Q remaining similar. This suggests DMD 2 may offer limited improvement. Why not simply fine-tune the pretrained teacher model with multi-modal adversarial learning and direct metric optimization? Could you conduct additional experiments comparing the teacher model with L_CTC and L_SV​ using GAN against DMDSpeech? Also, please compare the teacher model with L_CTC and L_SV without GAN against DMDSpeech. These would help clarify the impacts of DMD, DMD 2, and GAN in DMDSpeech.

questions:
1. In line 412, could you elaborate on the statement, "Table 5 shows that our model achieved the highest speaker similarity score (SIM) to the prompt, even surpassing the ground truth"? It is unclear how Table 5 demonstrates that DMDSpeech exceeds the ground truth.

2. Does your training dataset exclude the official samples from the baseline models which are used for evaluation?

3. Could you clarify why you chose to fine-tune the CTC-based ASR model to derive the latent SV model? From your explanation in Appendix C.4, you use a distillation approach to align the CTC-based ASR model's embeddings with the concatenated embeddings from the WeSpeaker's ResNet-based SV model and EPACA-TDNN with a fine-tuned WavLM Large model. Wouldn't it be more efficient to utilize the WeSpeaker's ResNet-based SV model and EPACA-TDNN with a fine-tuned WavLM Large model to directly extract the concatenated embeddings of the prompt and the concatenated embeddings of the generated speech, then calculate the SV loss in Equation 14 using these two embeddings? Can you provide experiment to evaluate that the latent SV model fine-tuned from CTC-based ASR model performs better in speaker similarity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730422738496

thread_type:
official_review

<thread_object>
creator: reviewer_4zge

<content>
summary:
In this work, the authors introduce DMDSpeech, a distilled diffusion-based TTS model. In this framework, a student model distills the distribution learned by a diffusion-based teacher model, enabling the generation of high-quality speech in just four steps. By integrating the distillation loss with speaker verification (SV) and connectionist temporal classification (CTC) losses, DMDSpeech achieves excellent speaker similarity and a low word error rate while maintaining high speech quality. The experimental results demonstrate that DMDSpeech outperforms several state-of-the-art baselines as well as the teacher diffusion model.

strengths:
- This paper is technically sound and well-presented.
- It provides ample implementation details, allowing readers to reproduce the results effectively.
- The authors conduct sufficient experiments to showcase the advantages of diffusion matching distillation, such as faster generation times, and highlight the effectiveness of combining speaker verification (SV) and connectionist temporal classification (CTC) perceptual losses.
- Regarding the mode shrinkage phenomenon resulting from the distillation process, the authors offer a detailed analysis, explaining that this effect may not be undesirable in the context of unconditional TTS.

weaknesses:
The two major themes of this paper—(1) distillation of diffusion-based TTS models and (2) joint optimization of TTS and perceptual losses—have both been extensively explored in the field of speech generation. While the paper is well-presented and technically sound, the proposed method is relatively straightforward and lacks novelty.

questions:
The paper is clear enough.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693357701

thread_type:
official_review

<thread_object>
creator: reviewer_i2sd

<content>
summary:
The paper presents DMDSpeech, a distilled diffusion model for zero-shot speech synthesis that achieves state-of-the-art performance while significantly reducing inference time. By employing distribution matching distillation, the model generates high-quality speech in just four steps and facilitates direct metric optimization through the use of speaker verification (SV) and connectionist temporal classification (CTC) losses. The authors demonstrate that optimizing these metrics leads to improvements in speaker similarity and word error rate, enhancing the overall intelligibility and quality of synthesized speech. The research highlights the potential of direct metric optimization in bridging the gap between generative modeling and human auditory preferences.

strengths:
**Originality**
- The authors borrow the concept of distribution matching distillation (DMD) from image synthesis. Yet, the use of DMD in speech synthesis using SV and CTC losses appears to be inspiring and original.
- This should be the first successful application of DMD in speech domain that illustrates promising results.

**Quality**
- As far as I checked, the mathematical derivations are technically sound, and I believe some equations are taken from DMD 2. 
- The training framework designed for zero-shot TTS is also sensible. Expectedly, the CTC loss improves the text alignment and WER, while the SV loss improves the speaker similarity.

**Clarity**
- Section 3 provides comprehensive details about the model architecture, training procedures, and loss functions used, which can aid readers in reproducing the proposed model.
- The results are presented both subjectively and objectively with comparisons to baseline models, which aids in understanding the effectiveness of the proposed model.
- The use of pitch comparison in figure 2 to confirm the improvement of student model is persuasive. This suggests a decent way to analyze the effects of DMD and direct metric optimization.

**Significance**
- DMDSpeech significantly reduces inference time while achieving competitive performance. This could yield a substantial impact to the industry and relative researches.

weaknesses:
There are some minor issues in this paper. 
- In figure 1, the "subtitle" of the four blocks should be consistently placed (e.g. upper leftmost), otherwise the readers may be difficult to spot them. (I just neglected "Inference", which is inconsistently located at the lower part)
- The choice of four steps in DMDSpeech is not discussed well. The readers may wonder why not 3 or 5 or 20 steps, which both can exhibit significant speedups when compared to 128 steps.

questions:
Would the quality be better if we use more steps for the student model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731776842166

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We thank all reviewers for highlighting the key strengths of our work, including its technical soundness, comprehensive experiments, and strong implementation details that ensure reproducibility. Reviewer 4Zge appreciated our effective analysis of mode shrinkage and robust experimental results, while Reviewer 9zdq acknowledged the significant improvements in naturalness, speaker similarity, and inference speed. Reviewer i2SD recognized the originality and significance of our work, specifically highlighting DMDSpeech as the first successful application of distribution matching distillation (DMD) in the speech synthesis domain with SV and CTC loss, achieving state-of-the-art results while reducing inference time. These consistent acknowledgments underscore the rigor, innovation, and practical impact of our contributions to the TTS field.

Regarding the novelty concerns raised, we would like to clarify the distinct contributions of our work: 

- **Beyond DMD in Speech Synthesis:**
While distillation has been applied to diffusion models, prior works in speech synthesis (e.g., [1,2,3]) have consistently shown that student models underperform or at best perform on par with teacher models in key metrics such as MOS and SIM. Our work is the first to demonstrate that, through direct metric optimization, the distilled student can consistently outperform the teacher across all standard metrics (MOS, SMOS, WER, SIM) while maintaining faster inference.

-  **Challenges in Direct Metric Optimization:**
Direct metric optimization in TTS is non-trivial in all previous approaches:

   1. **Traditional Models:** Models relying on components like monotonic alignment search (MAS) and duration predictors (such as YourTTS and NaturalSpeech 2&3) cannot propagate gradients to all components such as text encoder and duration predictor, preventing true end-to-end optimization. Although YourTTS uses a speaker similarity loss, this architectural limitation prevents optimization of metrics like WER, which depend heavily on optimizing text representations (text encoder) and phoneme durations (duration predictor). 

   2. **Iterative Sampling Models:** Modern transformer-based models (autoregressive or diffusion) require iterative sampling (e.g., 120 steps for autoregressive to generate 3-second audio or 16 steps for most advanced flow-matching models [4]), making backpropagation computationally prohibitive because of memory constraints and unstable due to vanishing gradients. 

      Furthermore, it is not possible to directly optimize through the teacher model because reasonably intelligible speech can only be obtained at low noise levels. At higher noise levels, the generated samples are unintelligible and unsuitable for optimization using SV and CTC loss, as the gradients are not instructive enough. We have newly added samples to our demo page to illustrate this limitation, showing that direct optimization through the teacher model is not trivial. These examples highlight why the only viable way to perform direct metric optimization in diffusion models is through distillation. You can find these samples on our demo page: https://dmdspeech.github.io/demo#rebuttal.

- **Our Solution:**
By distilling the diffusion model into a single-step generator, we enable efficient and stable backpropagation for optimizing differentiable metrics. Moreover, our framework generalizes beyond speaker similarity (SIM) and can be applied to word error rate (WER), showcasing it can be extended to any differentiable metrics, offering adaptability as new metrics emerge that can better align with human preference. We believe this represents a meaningful advancement in TTS technology.

**References:**

[1] Ye, Z., Xue, W., Tan, X., Chen, J., Liu, Q., & Guo, Y. (2023, October). Comospeech: One-step speech and singing voice synthesis via consistency model. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 1831-1839).

[2] Ye, Z., Ju, Z., Liu, H., Tan, X., Chen, J., Lu, Y., ... & Guo, Y. (2024, October). Flashspeech: Efficient zero-shot speech synthesis. In Proceedings of the 32nd ACM International Conference on Multimedia (pp. 6998-7007).

[3] Liu, Z., Wang, S., Inoue, S., Bai, Q., & Li, H. (2024). Autoregressive Diffusion Transformer for Text-to-Speech Synthesis. arXiv preprint arXiv:2406.05551.

[4] Chen, Y., Niu, Z., Ma, Z., Deng, K., Wang, C., Zhao, J., ... & Chen, X. (2024). F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching. arXiv preprint arXiv:2410.06885.
</content>
</thread_object>

</thread>


Metareview: