DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis
Authors: Yinghao Aaron Li
Rithesh Kumar
Zeyu Jin
Venue: ICML 2025 poster (2025)
Keywords: text-to-speech
zero-shot speech synthesis
diffusion model
diffusion distillation
metric optimization
Primary area: applications->language_speech_and_dialog
TL;DR: DMOSpeech achieves highly efficient and accurate zero-shot speech synthesis by directly optimizing a distilled diffusion model based on objective quality metrics.

Abstract: Diffusion models have demonstrated significant potential in speech synthesis tasks, including text-to-speech (TTS) and voice cloning. However, their iterative denoising processes are computationally intensive, and previous distillation attempts have shown consistent quality degradation. Moreover, existing TTS approaches are limited by non-differentiable components or iterative sampling that prevent true end-to-end optimization with perceptual metrics. We introduce DMOSpeech, a distilled diffusion-based TTS model that uniquely achieves both faster inference and superior performance compared to its teacher model. By enabling direct gradient pathways to all model components, we demonstrate the first successful end-to-end optimization of differentiable metrics in TTS, incorporating Connectionist Temporal Classification (CTC) loss and Speaker Verification (SV) loss. Our comprehensive experiments, validated through extensive human evaluation, show significant improvements in naturalness, intelligibility, and speaker similarity while reducing inference time by orders of magnitude. This work establishes a new framework for aligning speech synthesis with human auditory preferences through direct metric optimization. The audio samples are available at https://dmospeech.github.io/demo

DMOSpeech: Direct Metric Optimization via Distilled Diffusion Model in Zero-Shot Speech Synthesis
Yinghao Aaron Li
Rithesh Kumar? 2 Zeyu Jin
Abstract
ing law (Kaplan et al., 2020), the focus of TTS research has shifted from small-scale datasets to large-scale models trained on tens to hundreds of thousands of hours of data en- compassing a wide variety of speakers (Wang et al., 2023a;c; Shen et al., 2024; Peng et al., 2024; Lajszczak et al., 2024; Li al., 2024b). Two primary methodologies have emerged for training these large-scale models: diffusion-based ap- proaches and autoregressive language modeling (LM)-based methods. Both frameworks enable end-to-end speech gener- ation without the need for sand-engineered features such as prosody and duration modeling as seen in works before the LLM era (Ren et al., 2020; Kim et al., 2021), simplifying the TTS pipeline and improving scalability.
Diffusion models have demonstrated significant potential in speech synthesis tasks, including text- to-speech (TTS) and voice cloning. However, their iterative denoising processes are computa- tionally intensive, and previous distillation at- tempts have shown consistent quality degradation. Moreover, existing TTS approaches are limited by non-differentiable components or iterative sam- pling that prevent true end-to-end optimization with perceptual metrics. We introduce DMO- Speech, a distilled diffusion- based TTS model that uniquely achieves both faster inference and superior performance compared to its teacher model. By enabling direct gradient pathways to all model components, we demonstrate the first successful end-to-end optimization of dif- ferentiable metrics in TTS, incorporating Connec- tionist Temporal Classification (CTC) loss and Speaker Verification (SV) loss. Our comprehen- sive experiments, validated through extensive hu- man evaluation, show significant improvements in naturalness, intelligibility, and speaker simi- larity while reducing inference time by orders of magnitude. This work establishes new frame- work for aligning speech synthesis with human auditory preferences through direct metric op- timization. The audio samples are available at https:/ /dmospeech. github. io.
As these models scale to handle increasingly diverse speak- ers and scenarios, ensuring consistent quality and speaker similarity becomes paramount. While directly optimizing relevant perceptual metrics would be a natural approach, it remains a main challenge across all TTS approaches. Tradi- tional models that rely on monotonic alignment and duration predictors (Shen et al., 2024) cannot propagate gradients through these non-differentiable components, preventing true end-to-end optimization of critical elements such as text encoders and duration predictors. While some approaches like YouTTTS (Casanova et al., 2022) have attempted to incorporate speaker similarity loss, these architectural limi- tations resulted in minimal improvements and prevented op- timization of other key metrics like word error rate (WER). Modern end-to-end models face different challenges with direct optimization due to their reliance iterative sam- pling. Autoregressive models require sampling steps that scale linearly with the length of the generated speech; dif- fusion models, while more efficient, still require iterative sampling: even the most advanced approaches need at least 16 steps (Chen et al., 2024c). This iterative nature not only makes backpropagation computationally prohibitive but also leads to gradient instability. Furthermore, direct optimiza- tion through the diffusion process is impossible because intelligible speech can only be generated at low noise levels, making the gradients from perceptual metrics uninstructive at higher noise levels. This suggests that achieving direct metric optimization requires first addressing the fundamen- tal limitations of iterative sampling. While previous work has explored distillation (Salimans & Ho, 2022: Song et al.,
1. Introduction
Text-to-speech (TTS) technology has witnessed remarkable progress over the past few years, achieving near-human or even superhuman performance on various benchmark datasets (Tan et al., 2024; Li et al., 2024a; Ju et al., 2024). With the rise of large language models (LLMMs) and scal-
Equal contribution 'Columbia University. Work done during an internship at Adobe 'Adobe Research. Correspondence to: Yinghao Aaron Li cy44579@columbia.cdd
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
2023; Sauer et al., 2023) as a way reduce sampling steps, these approaches have focused solety on inference speed, consistently showing performance degradation through the distillation process from teacher to student models (Bai et al., 2023; Ye et al., 2023). These challenges highlight the need for a fundamentally new approach that not only reduces sampling steps but also enables true end-to-end op- timization while maintaining or improving speech quality.
2023; Liu et al., 2024). However, these models face fun- damental limitations in optimizing perceptual metrics due to their reliance on iterative sampling. DMOSpeech ad- dresses these limitations by enabling true end-to end-to-end metric optimization while maintaining efficient inference.
Diffusion Distillation Previous approaches to acceler- ating diffusion models have explored various distillation techniques, each with distinct trade-off'. Progressive dis- tillation (Huang et al., 2022) and consistency distillation (Ye et al., 2023; 2024) attempt to match intermediate states of the teacher's 's sampling trajectory, while rectified flow methods (Guo et al., 2024; Guan et al., 2024) focus on straightening these trajectories. However, these approaches often compromise quality by constraining the student to fol- low the exact path of the teacher, which may be suboptimal for models with reduced capacity. Distribution matching approaches, whether adversarial (Sauer et al., 2023; 2024) or via score function matching (Yin et al., 2024b), offer an alternative by aligning the student with the teacher in distribution rather than trajectory. While these methods typically require computationally expensive noise-data pair generation (Sauer et al., 2024; Yin et al., 2024b; Liu et al., 2023), DMD2 (Yin al., 2024a) overcomes this limitation by prioritizing efficiency over diversity. Prior studies have typically viewed this tendency to reduce output diversity as a limitation. However, we demonstrate in this work that in strongly conditional generation tasks like TTS, where strict adherence to input text and speaker prompts is re- quired, this diversity reduction can enhance output quality while maintaining sufficient variation across different inputs. This insight makes DMD2 particularly suitable for zero-shot TTS, offering natural way to balance quality and diversity.
In this work, we introduce Direct Metric Optimization Speech, distilled diffusion-based based speech synthesis model that achieves both superior performance and faster inference compared to existing approaches. Our key innovation is to enable, for the first time, true and-to-end (E2E) optimiza- tion of differentiable metries in TTS, with two technical advances: (1) reducing sampling steps from 128 to 4 via dis- tribution matching distillation (Yin et al., 2024b;a), and (2) providing direct gradient pathway from the noise input to speech output without non-differentiable components. This allows us to directly optimize speaker similarity and word error rate through speaker verification (SV) and CTC losses respectively, capability not achievable in previous TTS approaches due to either non-differentiable components like duration predictors (Casanova et al., 2022) or prohibitively expensive backpropagation through hundreds of sampling steps. Our comprehensive experiments demonstrate that this E2E optimization leads to significant improvements across all metrics, outperforming both the teacher model and other recent baselines in both subjective and objective evaluations. Importantly, we discover that our optimized metrics strongly correlate with human perception, and the distillation process induces beneficial mode shrinkage that improves quality in strongly conditional generation by fo- cusing on high-probability regions without compromising output diversity across different prompts and text inputs.
Direct Metric Optimization While optimizing percep- tual metrics has shown promise in speech enhancement through approaches like MetrieGAN (Fu et al., 2019) for PESQ and STOI, and recent attempts have explored RLHF for improving naturalness (Zhang et al., 2024; Chen et al., 2024a), implementing these approaches in modern TTS sys- tems has remained challenging. This difficulty stems from architectural limitations such as non-differentiable duration upsamplers (Li et al., 2024b; Ye et al., 2024) or computation- ally intensive iterative sampling (Lee et al., 2024; Peng et al., 2024). Previous attempts like YouTTTS (Casanova et al., 2022) reported minimal improvements from speaker similar- ity optimization due to their inability to propagate gradients through all model components. DMOSpeech overcomes these limitations by enabling comprehensive optimization of all model elements through direct gradient pathways, mark- ing the first successful demonstration of true end-to-end metric optimization in speech synthesis while maintaining fast inference and high quality.
2. Related Works
Zero-Shot Text-to-Speech Synthesis Zero-shot TTS has evolved significantly in its approach to quality optimiza- tion. Early methods relied on speaker embeddings from pre-trained encoders (Casanova et al., 2022; 2021; Wu et al., 2022; Lee et al., 2022) or end-to-end speaker encoders (Li et al., 2024a; Min et al., 2021; Li et al., 2022; Choi et al., 2022), but struggled with generalization and quality opti- mization due their reliance on extensive feature engi- neering and non-differentiable components. More recent prompt-based methods have demonstrated improved scala- bility using both autoregressive (Shen et al., 2024; Le et al., 2024; Ju et al., 2024; Lee et al., 2024; Yang et al., 2024; Eskimez et al., 2024; Liu et al., 2024) and diffusion frame- works (Jiang et al., 2023b; Wang et al., 2023ace; Jiang et al., 2023a; Peng al., 2024; Kim al., 2024; Chen et al.,
2024b; Meng et al., 2024; Yang et al., 2024; Lovelace et al.,
2

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
3. Methods
for efficient speech synthesis, including the formulations corresponding our implementation.
3.1. Preliminary: End-to-End Latent Speech Diffusion
Background on Distribution Matching Distillation
Our model starts with a pre-trained teacher model based on an end-to-end latent speech diffusion framework such as SiMpleTTS (Lovelace et al., 2023) and DiTTo-TTS (Lee al., 2024). This section outlines the formulation of the diffusion process and objective function.
DMD aims to train a student generator Gg to produce sam- ples whose distribution matches the data distribution Pdaa after a forward diffusion process, The objective is to min- imize the Kullback- Lieblerr (KL) divergence between the distributions of the diffused real data pdata, and the diffused student generator outputs po,, across all time t [0, 1]:
We begin by encoding raw audio waveforms y E R RIXT where T is the audio length, into latent representations Xo = Ey)) using a latent autoencoder E. The latent au- toencoder follows DAC (Kumar et al., 2024) with residual vector quantization replaced by the variational autoencoder loss (see Appendix C.1 for more information). We denote the ground truth latent distribution as Pdata- The diffusion process involves adding noise to X Pdata over continuous time E [0, 1] through a noise schedule. Our noise schedule follows Lovelace et al. (2023), which is a shifted cosine noise schedule formulated with (₁ and a that control the amount of signal and noise (see Appendix C.2.1).
Pdatu, (x) }
D (.o.ppe.lllaaaaa Exeppos log
= -Exx-po.. [log (Pdata.t (x)) - log (po, (x)]]x
(2)
The DMD loss is LDMD = 1 [D (po,e [IPaaa,))] accordingly. Since DMD trains Go through gradient descent, the formulation DMD only requires the gradient of the DMD loss with respect to the generator parameters 8, which is derived in Yin et al. (2024b) as:
dG dB
During training, the model learns to remove noise added to the latent representations. Given latent variable Xo and noise ~ (O, I). the noisy latent X₁ at time step t is generated as X, = 0,X0 + o,C. We use a binaray prompt mask m to selectively preserve the original values in regions corresponding to the prompt. The noisy latent x, is adjusted as x, + x, + (1 - m) + Xg O m, where C denotes element-wise multiplication. The binary mask m is randomly sampled to mask between 0% to 50% of the length of Xu- We define a reparameterized velocity = 0 Xn. which serves as the training objective as in Huang et al. (2022). We train our diffusion transformer (Peebles& & Xie, 2023) model fa. parameterized by Φ, to predict v given the noisy latent xt, conditioned on text embeddings c, prompt mask m, and the time step t:
CDLD
wron (Sreal(X,, t) so (x(, 6)) E f,
where x, is the diffused version x. = Gz(z), the distilled generator output for z ~ NV I), Steal (X,, and so (xxx t) are neural network approximation of score functions of the diffused data distribution and student output distribution, and Wi is a weighting factor defined in eq. 25.
In our speech synthesis task, the generator Go produces latent speech representations Xn conditioned on input text c and a speaker prompt. The teacher diffusion model fo serves as the score function Sreal for the real data distribu- tion. We train another diffusion model Pø to approximate the score of the distilled generator's is output distribution po following eq. 1. The scores are estimated as:
[fi Pdinaa = E x xor-
[[lv~ - (x( ; m, 1)]l2)
X Cx
(0111,1)
s s(x,, to)
(4)
(0.)
of
(1)
During inference, the model takes noise z E (0, 1) with fixed size [d, I.] where I, is the total duration of the tar- get speech. L is estimated by multiplying the number of phonemes in the target text with the speaking rate of the prompt speech (see Appendix C.2.3 for more details).
where Xy are estimation of xo from the diffusion models h:
Xo X, - oxh((x : C, m,
(5)
O't
where h = f for xpaa and h = Iw for xfolee Accordingly,
Sreal (Xe, () = s sx,,, 1, xgaa) and su (x 1) = =((x, L, &fole). ).
3.2. Improved Distribution Matching Distillation
The parameters of Go and ge are both initialized from the teacher diffusion model's parameters sb
We employ improved Distribution Matching Distillation (Yin et al., 2024a), or DMD 2, to distill our teacher model for fast sampling and direct metric optimization. DMD 2 improves upon DMD (Yin et al., 2024b) by incorporating adversarial training on the real data, eliminating the need for noise-data pair generation and significantly reducing the training cost. This section details how we adapt DMD
DMD 2 for Speech Synthesis We notice that the one-step student model results in noticeable artifacts, as the student model lacks the computational capacity to capture all the acoustic details that the teacher model generates through multiple iterative steps. To address this issue, we adopt
3

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Distribution Matching Distiliation
Decoder
Oulput Latent %
Student Generator
Adi Noise
Teacher Score Model
Syeal
Go
Diflused Output
VoCIMM
(0, 1)
X
Sracc
Inference
Prompt
Student Score Model
She
Direct Metric Optimization
Layer Stack
Multi-Modal Adversarial Training
ASR Model
Muitimodal Discriminator
Text
Noise
DAC Encoder
Cay
SV Model
Core
D
Promot
Diflused Latent
Real Latent
X,
Cude
Figure 1. Overview of the DMOSpeech framework, consisting of inference and three training components: (1) Inference: A one-step
distilled generator synthesizes speech directly from noise (red arrow). The three training components are: (2) Distribution Matching Distillation: The student score model matches the teacher align their distributions in terms of score functions (purple arrow). (3) Mudti-Modal Adversarial Training: A discriminator distinguishes between real and synthesized noisy latents (yellow arrows), and (4)
Direct Metrie Optimization: End-to-end optimization of word error rate (pink) and speaker similarity (blue arrows).
framework from Yin et al. (2024a) by condi- tioning the student generator Gg on the noise level I. This conditioning allows the model to estimate the clean latent speech representation Xy from its noisy counterpart X2 for sequence of predefined time steps E { (N). This multi-step sampling (Algorithm 1) is similar to the con- sistency model proposed by Song et al. (2023). It goes as follows: for each time step tz. the student model pro- duces an estimate X = Go(x,, ;, m, (n), which is then re-noised to obtain Xt,+1 as input for the next time step:
the DMD 2
the adversarial discriminator used in Yin et al. (2024a) to a conditional multimodal discriminator, inspired by Janiczek et al. (2024). Following Li et al. (2024b), our discriminator D is conformer that takes as input the stacked features from all transformer layers of the student score network Iv with noisy input, along with the text embeddings c, prompt mask m, and noise level t (denoted as C). The discriminator is trained with the LSGAN loss (Mao et al., 2017):
Ladv (Ga; D) = Er mmm (D(2 -1)
(7)
Xt, Cr,, 8D + 1t, 6 6~ (O, I).
(6)
L (D; Go) R,
Exp (D(x:
This process generates progressively less noisy versions of Xy at decreasing noise levels t.1 or.
Eg [Ex.mpmm...m
-
(8)
We use the schedule {1.0, 0.75, 0.50, 0.25) mapped from teacher's full range t E [0. for our four-step model. We simulate one-step inference during training to mini- mize the training/inference mismatch. Instead of using the noisy version of ground truth Xr, = Qt,, Xo + or,, e as input, we use the noisy version of student prediction or,, Gg (x,,,1; m, 4n-1) + or,, e from the noisy ground truth xt,,-; at the noise level 0n-1 nn Different from Yin et al. (2024a), which simulates all four steps, we found that simulating just one step is sufficient for producing high- quality speech while saving GPU memory during training.
where C = {c, m, is the conditional input and &; = or Gz; + OrE is the noisy version of the student- generated speech (z( at time step t with z ~ NO,
3.3. Direct Metric Optimization
We directly optimize two metrics, speaker embedding co- sine similarity (SIM) and word error rate (WER), which are commonly used for evaluating zero-shot speech syn- thesis models and are both shown to correlate with human perception for speaker similarity (Thoidis et al., 2023) and naturalness (Alharthi et al., 2023). To improve WER, we incorporate a Connectionist Temporal Classification (CTC) loss (Graves et al., 2006). The CTC loss aligns the syn- thesized speech with the input text the character level, reducing word error rates and enhancing robustness:
To further improve the performance of the student model, we incorporate adversarial training following the approach of Yin et al. (2024a) that allows the students learn from the real data. However, unlike in text-to-imate tomee synthesis, where text acts as a weak condition for the generated image, text-to-speech synthesis requires strong conditioning on both text and speaker prompt. The generated speech must strictly adhere to the semantic content of the text and the prompt speaker's voice and style. To this end, we modify
Lerc Exparpp |- log C(x
where Xtake is the student- -eneerteeeerated speech, c is the text transcript, and C(-) a tre-trained CTC-based ASR model
4

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
on speech latent (see Appendix C.3 for details). We also employ a Speaker Verification (SV) loss to ensure the syn- thesized speech matches the target speaker's identity. We use a pre-trained speaker verification model S on latent (see Appendix C.4 for details) for the SV loss:
generate intelligible speech from single step. To address this, we set Acrc and Asv 0 for the first 5,000 and 10,000 iterations, respectively. This allows Go to stabilize under the influence of LIMD before integrating these additional losses. After that, both AcrO and Asy are set to 1.
Creal Ctake
Lsv = E. Keca Papan Misee PPo
(10)
4. Experiments
||Creal|
| |I
4.1. Model Training
where Greal =
((xca) and efake = (xtake) ) are the speaker embeddings of the prompt and student-generated speech.
We conducted our experiments on the LibriLight dataset (Kahn et al., 2020), which consists of 57,706.4 hours of audio from 7,439 speakers. The data and transcripts were obtained using Python scripts provided by the LibriLight authors' ! All audio files were resampled to 48 kHz to match the configuration of our DAC autoencoder, and the text was converted into phonemes using Phonemizer (Bernard & Ti- teux, 2021). To manage memory constraints, we segmented the audio into 30-second chunks using WhisperX (Bain et al., 2023). The teacher model fø was trained for 400,000 steps with a batch size of 384, using the AdamW optimizer (Loshchilov & Hutter, 2018) with 3 = 0.9, B2 = 0.999, weight decay of 10-?, and an initial learning rate of 10-4. The learning rate followed a cosine decay schedule with a 4, 4,000-step warmup, gradually decreasing to 10-5, Model weights were updated using an exponential moving average (EMA) with decay factor of 0.99 every 100 steps. The teacher model consists of 450M parameters in total. For student training, we initialized both the student generator Ge and the student score model ge with the EMA-weighted teacher parameters. The initial learning rate was set to match the final learning rate of the teacher model (A = 10-5), while the batch size was reduced to 96 due to memory con- straints. Lowering the batch size further negatively impacted performance, as sufficiently large batch size is required for accurate score estimation (see Section 4.4 for further discussion). The student generator Gg and the discrimina- tor D were trained for an additional 40,000 steps, and the student score model Ie for 200,000 steps accordingly using the same optimization settings as the teacher. All models were trained on 24 NVIDIA A100 40GB GPUs.
3.4. Training Objectives and Stability
The overall training objective for Ga combines DMD and adversarial losses with SV and CTC losses:
min LDMD + MudyCaas (Go; D) + AsvCsv + dcreCere, 9
and the training objectives for g and D are:
min Lai (o): + We employ an alternating training strategy where the student generator Gø. the student score estimator Ie- and the discriminator D are updated at different rates to maintain stability: for every update of Gø, we perform five updates of Fv.. This ensures that the score estimator ge can adapt quickly the dynamic changes in the generator distribution Pg- Unlike Yin et al. (2024a), where D) are updated five times for every single update of Go, we update D and Go at the same rate. This prevents the discriminator from becoming too powerful and destabilizing training.
min Laty (D)
D
The learning rates for GG and ge play a critical role in maintaining training stability since both models are initial- ized from the teacher's parameters, Φ. Treating this as fine-tuning process, we set their learning rates close to the teacher model's final learning rate to prevent catastrophic forgetting and training collapse. The teacher model was trained using a cosine annealing warmup scheduler, which gradually reduced the learning rate over time. Thus, starting with high learning rate for Gg and By can cause them to deviate significantly from the pre-trained knowledge, lead- ing to training failure. Conversely, the learning rate for D is less sensitive and does not require such precise tuning.
4.2. Evaluation Metries
We performed both subjective and objective evaluations to assess the performance of our model and several state-of- the-art baselines. For subjective evaluation, we employed four metrics rated on a scale from I to 5. The Mean Opin- ion Score for Naturalness (MOS-N) assessed the human- likeness of the synthesized speech, where indicates fully synthesized audio and 5 indicates completely human speech. The Mean Opinion Score for Sound Quality (MOS-Q) evalu-
Balancing different terms in the overall objective function is crucial for successful training. The primary loss, LDMD, responsible for transferring knowledge from the teacher model, aligning the synthesized speech with the text. Other losses, such as Lave, Lsv, and Lere. need to be scaled properly to match the gradient of LDMD. We set Anav = 10 to ensure the gradient norm of Ladv is comparable to that of LDMD- During early training stage, we observed that the gradient norms of Lsv and Lere were significantly higher than LOMD. likely because Go was still learning to
'The code is available at https: / /ttp://///tth.//// com/ Bacebookresearcc/1ibbr -ighht/
5

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Table 1. Comparison between our models and non-E2E baselines on four subjective metrics: naturalness (MOS-N), sound quality (MOS-Q), voice similarity (SMOS-V), and speaking style similarity (SMOS-S). Scoes are presented means (+ standard error). One asterisk (*) indicates a statistically significant difference (p < 0.05) and double asterisk (*t) indicates p < 0.01 compared to DMOSpeech. The best models and those within one standard error of the best are highlighted.
Model
MOS-N
MOS-Q
SMOS-V
SMOS-S
Ground Truth
4.47 (= 0.03)
4.61 (+ 0.03)
3.86 (+ 0.05)*
3.81 (+ 0.05)**
Ours (DMOSpeech, N=4)
4.42 (= 0.03)
4.59 (+ 0.03)
4.49 (# 0.03)
4.30 (+ 0.03)
Ours (Teacher, N=128)
4.32 (+: 0.04)*
4.55 (# 0.03)
4.17 (#: 0.04)*
4.00 (#: 0.04)"*
NaturalSpecch 3 (Ju et al., 2024)
4.24 (+ 0.04)**
4.55 (# 0.03)
4.44 (# 0.03)
4.25 (+ 0.04) 4.20 (+ 0.03)*
StyleTTS-ZS (Li et al., 2024b)
4,40 (= 0.03)
4.54 (# 0.03)
4.34 (# 0.04)
Table 2. Comparison between our models and end-to-end baseline models.
Model
MOS-N
MOS-Q
SMOS-V
SMOS-S
Ground Truth
4.37 (= 0.03)*
4.49 (# 0.03)
3.51 (# 0.05)**
3.39 (#: 0.05)*
Ours (DMOSpeech, N=4)
4.27 (1 0.03)
4,45 (1: 0.03)
4.35 (1 0.03)
4.16 (1: 0.03)
Ours (Teacher, N=128)
4.22 (= 0.04)
4,40 (# 0.03)
4.03 (# 0.04)**
3.87 (#: 0.04)*
DiTTo-TTS (Lee et al., 2024)
4.28 (#: 0.04)
4,41 (#: 0.03)
4.16 (4: 0.04)**
4.07 (#: 0.03)"
VoiceCraft (Peng et al., 2024)
3.76 (#: 0.05)"
3,88 (#: 0.04)
3.41 (+: 0.05)"*
3.37 (1: 0.05)*
CLaM-TTS (Kim et al., 2024)
3.77 (#: 0.05)**
3,87 (# 0.04)
3.67 (# 0.05)"*
3.43 (# 0.05)*
XTTS (Casanova al., 2024)
3.63 (# 0.05y**
3.89 (+ 0.04)
3.25 (# 0.05)**
3.22 (# 0.05)**
Table Objective evaluation results between our models and other baseline models. The real-time factor (RTF) was computed on a NVIDIA V100 GPU except TTTT and CTS, whose RTF is obtained from their papers using the inference time needed to synthesize 10s of speech divided by 10 on unknown devices. Additional evaluation results on emotion are in Table 7.
prompt speech served as an anchor that is supposed to score 5 on all metrics; we also included intentionally mismatched speakers serving as low anchor for similarity, which should have a rating lower than 3. The participants who fails to correctly rate the anchors hidden in the test are disquali- fied and their answers removed (details in Appendix E.2), For objective evaluation, previous works (Wang et al., 2023a; Lee et al., 2024) and measured speaker similarity using the cosine similarity be- tween speaker embeddings of the generated speech and the promot (SIM), using the WavLM-TDCNN speaker em- bedding model?. We also calculated the Word Error Rate (WER) with CTC-based HuBERT ASR model following (Ju et al., 2024; Shen et al., 2024).
followed the approach from
Model
# Params.
WER
SIM 1
RTF +
Ground Truth
2.19
0.67
DMOSpeech (N=4)
450M
1.94
0.69
0.07
Teacher (N=128)
450M
9.51
0.55
0.96
NaturalSpeech 3
500M
1.81
0.67
0.30
VoiceCraft
830M
6.32
0.61
1.12
DITTO-ITS
740M
2.56
0.62
0.16
CLaM-TTS
584M
5.11
0.49
0.42
XTTS
482M
4.93
0.49
0.37
4.3. Comparison to Other Models
ated audio quality degradation relative to the prompt, with representing severe degradation and 5 indicating degrada- tion. The Similarity Mean Opinion Score for Voice (SMOS- V) measured the similarity of the synthesized voice to the prompt speaker's 's voice, where means completely different and 5 means identical. Lastly, the Similarity Mean Opin- ion Score for Style (SMOS-S) S) assessed the speaking style similarity the prompt speaker with the same scale. These subjective evaluations were conducted through listening test survey on the crowdsourcing platform Prolific, with 1,000 tests (30 samples each) taken by native English speak- ers with no hearing impairments who had experience in content creation or audio/video editing, ensuring they could better differentiate synthesized audio from real human. The
We conducted two evaluation experiments to compare our models against two categories of baselines: recent non non-end- to-end models that include explicit duration and prosody modeling, and end-to-end (E2E) models without such ex- plicit modeling. For both experiments, the samples were downsampled to 16 kHz for faimess and prompts were tran- scribed using WhisperX for synthesis.
In the first experiment, compared our model Natu- ralSpeech 3 and StyleTTS-ZS, both with explicit duration
htcres//ggthaa.ccoo/////eeeeeees tree thte//aiii//doussttteem/ggeeeeee hubert- Larer---argee1L9600-E
6

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
0,0200
Teacher Model
0.030
Teacher Model
0.0175
Student Model
Student Model
0.0150
Prompt FO mean
0.025
Prompt mean
Density
0.0125
Density
0.020
0.0100
0.015
0.0075
0.010
0,0050
0.005
0,0025
0.0000
0.000
100
200
300
400
50
100
150
200
250
300
FO Value
FO Value
Figure 2. Illustration of mode shrinkage in terms of pitch. Speech with the same text and prompt were synthesized 50 times, and their frame-level FO values are shown as histograms and kernel density estimates. The red dashed line represents the mean FO value of the prompt. In both examples, the student's distribution shifts toward the most likely region, centering around the prompt's mean value.
All baselines, except for NaturalSpeech 3, were evaluated using the 3,711 samples as per Lee et al. (2024). Since lacked sufficient samples for a direct evaluation of Natu- ralSpeech 3, its results are taken from their original paper. Table shows that our model achieved the highest speaker similarity score (SIM) to the prompt, even surpassing the ground truth. The Real-Time Factor (RTF) of the distilled model is 13.7 times lower than the teacher model, which is lower than all baseline methods by large margin. Although our model had slightly higher WER (1.94) compared to NaturalSpeech 3 (1.81) is important to note that our model is entirely end-to-end without explicit duration modeling, unlike NaturalSpeech 3. Both DMOSpeech and Natural- Speech 3 also exhibited lower WER than the ground truth. One point consider is the high WER of our teacher model, which is mainly due to cutoff the end of sentences in the training set caused by faulty segmentation with WhisperX. It affects about 10% of the utterances. After distillation, this issue was resolved due to mode shrinkage (discussed in Sec- tion 4.4). Morcover, our model demonstrates significantly faster inference speed compared to all baseline models, as it only requires four sampling steps.
Table 4. Objective and subjective evaluation results on Seed-TTS- en and Seed-7TS-ch evaluation sets trained with Emilia datasct.
Seed-TTS-en
Seed-T7S
Model
RTF/
SIM
WER|
SIM1
CER
MaskGCT
0,717
2.62
0.752
2.27
1.21
F5-TTS (N=32)
0.647
1.83
0.741
1.56
0.32
DMOSpeech (N=4)
0.687
1.78
0.757
1.43
0.06
and prosody modeling and trained on the large-scale Lib- riLight dataset. Since neither model has public or official checkpoints available, we used 47 official samples from the authors and other sources (details in Appendix I. from the LibriSpeech rest-clean subset, covering all 40 speakers. As shown in Table 1, our distilled model significantly outper- formed NaturalSpeech 3 in naturalness and StyleTTS -ZS in similarity metries. It also outperformed the teacher model in terms of naturalness, voice similarity, and style similarity.
In the second experiment, we evaluated E2E speech synthe- sis models, including three popular autoregressive models, XTTS, CLaM-TTS, VoiceCraft, and one diffusion-basse model, DiTTo-TTS. Since official code and checkpoints for CLaM-TTS and DiTTo-TTS were unavailable, we ob- tained 3 samples from the authors from the LibriSpeech test-clean subset and synthesized the corresponding sam- ples using XTTS, VoiceCraft, and our models. For subjec- tive evaluation, we selected 80 samples, ensuring that each speaker from the test-clean subset was represented by two samples. As shown in Table 2, our model significantly out- performed all recent E2E speech synthesis baselines except DITTO-TTS in MOS, with which it achieved comparable performance in naturalness and sound quality. This indi- cates that our model is consistently preferred across both naturalness and similarity by human listeners.
To further demonstrate the general applicability of framework, we also conducted experiments training DMO- Speech model using F5-TTS (Chen et al., 2024c) as the teacher model on Emilia dataset (He et al., 2024) and com- pared it against other recent state-of-the- the models, includ- ing F5-TTS itself and MaskGCT (Wang et al., 2024). We followed the setup in (Chen al., 2024c) train the teacher model and used the same hyperparameters as detailed in section 1 to train the student model for 200k steps. Our model and baseline models were evaluated on the Seed-TTS test set (Anastassiou et al., 2024). As shown in Table 5, our model performs comparably or better than other of the-art models but operates in a significantly faster speed, showcasing our model's strength in both efficiency and ef- fectiveness.
'Prompts and samples were generated according to instruc- tions provided in https: //tps::///tttt// www..////eeeeeeeeeeo0 eva wualeueeeeee zww.eeerrottttteee
7

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Table 5. Ablation study comparing our proposed model with different conditions. MOS-N, MOS-Q. SMOS- V. and SMOS-S reported as mean (#t standard error). Models with statistically significant differences (p < 0.05) compared to DMOSpeech are marked with one
asterisk (*). Additional evaluation results emotion reflection presented in Table 8.
Model
MOS-N
MOS-Q
SMOS-V
SMOS-S
WER
SIM
CV
Teacher (N=128)
4.22 (+ 0.04)
4= 0.03)
4.03 (# 0.04)*
3.87 (+ 0.04)*
9.51
0.55
0.70
DMD 2 only (N=1)
3.11 (#: 0.05)
2.99 (# 0.05)*
2.57 0.05)
2.74 0.05))
5,93
0.42
0.68
DMD 2 only (N=4)
4.19 (+: 0.03)*
4.43 0.04)
3.69 0.05)
3.62 0.05)
5.67
0.53
0.61
+Crc only
4.25 (#: 0.04)
4.42 (# 0.03)
3.73 0.05)*
3.62 0.05)*
1.79
0.55
0.57
+sy only
4.07 (+ 0.04)*
4.33 (+ 0.03)*
4.35 (+ 0.04)
4.15 (+: 0.04)
6.62
0.70
0.61
DMOSpeech (N=4)
4.27 (#: 0.03)
4,45 (# 0.03)
4.35 (#: 0.03)
4.16 (#: 0.03)
1.94
0.69
0.58
B. S. 96 + 16
4.20 (+ 0.04)
4,30 (+ 0.03)*
4.27 (+ 0.04)*
4.11 (+ 0.04)
3.38
0.67
0.60
4.4. Ablation Study
We conducted ablation studies to assess the contribution of each proposed component, with results summarized in Table 5. We evaluated models trained solely with DMD 2 using one sampling step (DMD only, N=1) and four sampling steps (DMD only, N=4), as well as models trained with only CTC loss or SV loss on top of four-step DMD 2 model. Additionally, we examined the impact of reducing the batch size from 96 to 16 (B. S. 96 + 16). The ablation study used the same 80 samples for subjective evaluation as in the second experiment and 3,711 samples for objective evalu- ation. To measure the trade-off between speech diversity and model capacity, we included the coefficient of variation of pitch (CV,,). This metric was calculated by synthesiz- ing speech with the same text and prompt 50 times and computing the coefficient of variation of the frame-level FO values averaged over the speech frames. The final results re- ported were averaged over 40 prompts from the LibriSpeech test- cest-clean subset, covering all 40 speakers.
0.8
0.6
SIM Score
0.4
0.2
0.0
0=0.5
2
3
4
5
SMOS-V
Figure 3. Scatter plot of human-rated voice similarity (SMOS-V) versus speaker embedding cosine similarity (SIM) at the utterance level. The correlation coefficient is 0.55.
when synthesizing speech from the same prompt and text. Given that zero-shot TTS is highly conditional, requiring strict adherence to the input text and speaker prompt, this reduction in diversity is not necessarily undesirable. As we found out in the subjective test, MOS-N increases even when diversity decreases. The distilled model achieves suffi- cient mode coverage across varying prompts and texts while benefiting from direct metric optimization and faster infer- ence. Notably, mode shrinkage also corrected a cut-off issue in the teacher model, which mimicked the cutoff patterns in the training data. Since these cutoff samples represent a small portion of the dataset, they were significantly reduced by the student models during distillation, leading to a much lower word error rate. This observation prompted us to in- clude CTC loss, further enhancing the model's intelligibility and robustness (see Appendix A for more discussion).
Effects of Distribution Matching Distillation Using a single sampling step resulted in significantly degraded per- formance compared to the full DMOSpeech model. While using four steps improved naturalness and sound quality to approach the teacher model's level, speaker similarity remained significantly lower. Interestingly, the speaker veri- fication model's SIM score showed only a slight decrease, suggesting a phenomenon we term mode shrinkage (Figure 2). where distillation emphasizes high siobbbbbbbbblty regions of the data distribution. This focus result in a more generic speaker profile, reducing perceived uniqueness in the prompt speaker's voice, while maintaining global speaker features as reflected in the SIM score. To address this, we introduced speaker verification loss in this work to better capture the distinct characteristics of the prompt speaker.
Lastly, since DMD training involves estimating the score functions from training data through Monte Carlo simulation in mini-batch, the batch size plays a critical role in the accuracy of distribution matching. Reducing the batch size from 96 to 16 significantly decreases sound quality and speaker similarity. Maintaining sufficiently large batch size is crucial for stable DMD training.
Mode shrinkage also led reduced diversity, as indicated by lower CV fo across student models compared to the teacher. There is also a trade-off between diversity and sam- ple quality, as one-step student obtained close-to-teacher diversity despite its lowest sample quality. However, as shown in Figure 5, this reduction in diversity applies only
Effects of Direct Metric Optimization
The metries we
8

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
directly optimize are significantly correlated with human subjective ratings at the utterance level. Figure 3 shows the scatter plot between human-rated similarity SMOS-V V and SIM, one of the optimized metrics, with correlation coef- ficient = 0.55. Another metric, word error rate (WER), is significantly correlated with naturalness (MOS-N) even at the utterance level, with a correlation p = -0.15 (see Figure 5). These correlations suggest notable impact of these metrics on their associated subjective ratings. When using only the CTC loss, we observe substantial reduction in WER (from 5.67 to 1.79), but no improvement in speaker similarity, alongside a slight reduction in diversity and a minor improvement in naturalness. This aligns with the cor- relation between WER and human-rated naturalness with P p= 00.5 (p < 0.01). In contrast, with only the SV loss, we see significant improvements in all speaker similarity metrics (SMOS-V. SMOS-S, SIM), but these gains come with a decrease in naturalness and sound quality, as well as an increase in WER. This suggests that while SV loss can enhance speaker similarity, it negatively impacts intel- ligibility and naturalness. Therefore, combining both CTC and SV losses achieves a balance between these metrics, yielding the best overall performance, with improvements across speaker similarity, intelligibility, and naturalness.
speaker verification techniques specifically designed to iden- tify synthetic speech is crucial. Second, implementing ro- bust audio watermarking systems could help track the origin and authenticity of synthesized content. Third, establishing clear ethical guidelines and legal frameworks for the deploy- ment of TTS technology is essential to prevent misuse while preserving beneficial applications.
The technology also has significant potential for positive impact. It could enable more accessible communication tools for individuals with speech impairments, improve edu- cational resources through personalized audio content, and enhance human-computer interaction across languages and cultures. However, these benefits must be balanced against the need for responsible development and deployment.
We encourage the research community to prioritize these eth- ical considerations in future work, particularly in developing better detection methods and establishing best practices for responsible TTS deployment. As this technology continues to evolve, maintaining a balance between innovation and ethical responsibility will be crucial for ensuring its positive contribution to society.
A Acknowledgements
5. Conclusions
We extend our gratitude to Tianwei Yin, the author of DMD, for valuable discussions and insights regarding DMD. We also thank Jiaqi Su for their support and assistance to Y.A. Li during the internship.
We presented DMOSpeech, a text-to-speech model enabling true end-to-end optimization of perceptual metrics while achieving fast inference through distribution matching dis- tillation. Our experiments demonstrate significant improve- ments in synthesis quality while revealing insights about controlled diversity reduction in conditional generation. The current approach faces challenges in balancing sampling speed and speech diversity, particularly when scaling to larger datasets. Future work could explore larger-scale mul- tilingual training data and develop new differentiable met- rics for human preference alignment through RLHF.
References
Alharthi, D., Sharma, R., Dhamyal, H., Maiti, S., Raj, B., and Singh, R. Evaluating speech synthesis by train- ing recognizers on synthetic speech. arXiv preprint arXivv2310.00706, 2023.
Anastassiou, P., Chen, J., Chen, J., Chen, Y., Chen, Z., Chen, Z.., Cong, J., Deng, L.., Ding, C., Gao, I.., et al. Seed- tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.024330 2024.
Impact Statement
This work advances the capabilities of text-to-speech syn- thesis in ways that warrant careful consideration of socictal implications. Our findings that DMOSpeech can generate speech with higher perceived similarity the prompt than real utterances from the same speaker raises important con- cerns about potential misuse, particularly in the creation of unauthorized synthetic speech or deepfakes. This capability highlights current limitations in speaker verification systems and emphasizes the need for robust detection methods to distinguish between synthetic and authentic speech.
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., and Weber, G. Common voice: A massively-multiinguall speech corpus. arXiv preprint
2019.
Bai, Y., Dang, T., Tran, D., Koishida, K., and Sojoudi, S. Ac- celerating diffusion- bbaed text-to-audio generation with consistency distillation. arXiv preprint arXiv:2309.107400 2023.
To address these concerns, we recommend several mitiga- tion strategies. First, the development of more sophisticated
Bain, M., Huh, J., Han, T., and Zisserman, A. Whisperx: Time- speech transcription of long-form audio. 2023.
INTERSPEECH 2023.
9

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Bernard, M. and Titeux, H. Phonemizer: Text Phones Transcription for Multiple Languages in Python. Jour- nal of Open Source Software, 6(68):3958, 2021. doi: 00.21105/joss.033588 URL https: /t///ioooooo org/o.00... 21 10/05//2 joss. 03958.
Eskimez, S. E., Wang. X., Thakker, M., Li, C., Tsai, C.-H.,
Xiao, Z., Yang, H., Zhu, Z.., Tang, M., Tan, X., al. E2 Us: Embarrassingly easy fully non-autoregressive zero- shot tts. arXiv preprint arXiv:2406. 18009, 2024.
Fu, S.- Liao, C.-F., Tsao, Y., and Lin, S.-D. Metriegan: Generative adversarial networks based black-box metric scores optimization for speech enhancement. In Interna- tional Conference Machine Learning, pp. 2031-2044 PmLR, 2019.
Cai, D. and Li, M. Leveraging asr pretrained conform- ers for speaker verification through transfer learning and knowledge distillation. IEEE/ACM Transactions Au- dio, Speech, and Language Processing, 2024.
Casanova, E., Shulby, C., Gölge, E., Müller, N. M., De Oliveira, F. S., Junior, A. C., Soares, A. d. S., Aluisio, S. M., and Ponti, M. A. Se-glowtts: An efficient zero- shot multi-speaker text-to-speech model. arXiv preprint vrxiv:2104.05555 2021.
Graves, A., Fernández, S., Gomez, F., and Schmidhuber, J. Connectionist temporal classification: labelling un- segmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., and Ponti, M. A. Yourtis: Towards zero-shot multi- speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pp- 2709 2720. PMLR, 2022.
Guan, W., Su, Q. Zhou, H., Miao, S., Xie, X., Li, L., and Hong, Q. Reflow-tts: A rectified flow model for high- fidelity text-to-speech. In ICASSP 2024-2024 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 10501-10505. IEEE, 2024.
Casanova, E., Davis, K., Gölge, E., Göknar, G., Gulea, I,, Hart, L., Aljafari, A., Meyer, J., Morais, R., Olayemi, S., et al. Xits: a massively multilingual zero-shot text-to- speech model. arXiv preprint 2.Xi::4406.04944 2024.
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. Con- former: Convolution- Convolution-angmented augmented transformer for speech recognition. arXiv preprint ar arXiv:2005.08/
2020.
Chen, C., Hu, Y., Wu, W., Wang, H., Chng, E. S., and Zhang, C. Enhancing zero-shot text-to-speech synthesis with human feedback. arXiv preprint arXiv:2406.0065.. 2024a.
Guo, Y., Du, C., Ma, Z., Chen, X., and Yu, K. Voiceflow: Efficient text-to-speech with rectified flow matching. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 11121-11125. IEEE, 2024.
Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint 10Xi::2106.0699, 2021.
He, H., Shang. Z., Wang. C., X., Gu, Y., Hua, H., Liu, L., Yang. C., Li, J., Shi, P., et al. Emilia: An extensive, multi- lingual, and diverse speech dataset for large-scale speech generation. In 2024 IEEE Spoken Language Technology Workshop (SLT), pp. 885-890. IEEF, 2024.
Chen, S. Liu, S., Zhou, L., Liu, Y., Tan, X., Li, J., Zhao, S., Qian, Y., and Wei, F. Vall-e 2: Neural codec lan- guage models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.085 05370, 2024b.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840 68511,668111111 2020.
Chen, Y., Niu, Z., Ma, Z., Deng, K., Wang, C., Zhao, J., Yu, K., and Chen, X. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching, arXiv preprint arXiv:2410.06885, 2024c.
Hoogeboom, E., Heek, J., and Salimans, T. simple diffu- sion: End-to-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232 PMLR, 2023.
Choi, H.-S., Yang, J., Lee, J., and Kim, H. Nansy++: Unified voice synthesis with neural analysis and synthesis. arXiv preprint ar arXiv:2211.094000 2022.
Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., and Ren, Y. Prodiff: Progressive fast diffusion model for high- quality text-to-speech. In Proceedings of the 30th ACM International Conference Multimedia, pp. 2595-2605, 2022.
Desplanques, B., Thienpondt, J., and Demuynck, K. Ecapa- tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification. arXiv preprint arXiv:2005.074 1414,, 2020.
Ito, K. and Johnson, L. The Ij speech dataset. https:// // reithito com L.J Sappeech-Dataseeee 2017.
10

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Janiczek, J., Chong, D., Dai, D., Faria, A., Wang, C., Wang, T., and Liu, Y. Multi-modal adversarial training for zero- shot voice cloning, arXiv preprint arXiv:2408.159) 15916, 2024.
Lajszezak, M., Cámbara, G., Li, Y., Beyhan, F. van Korlaar,
A., Yang. F., Joly, A., Martín-Cortinas, Á., Abbas, A., Michalski, A., et al. Base uss: Lessons from building billion-paramette parameter text-to-speech model on 100k hours of data, arXiv preprint 2rXiv:2402.08000 2024.
Jiang, Z., Liu, J., Ren, Y., He, J., Zhang, C., Ye, Z., Wei, P., Wang, C., Yin, X., Ma. Z., et al. Mega-tts 2: Zero- shot text-to-speech with arbitrary length speech prompts. arXiv preprint (rXXx:2307.07//15 2023a.
Le, M., Vyas, A., Shi, B.. Karrer, B., Sari, L., Moritz, R., Williamssn, M., Manohar, V., Adi, Y., Mahadeokar, J., et al. Voicebox: Text-guided multilingual universal speech generation scale. Advances in neural informa- tion processing systems, 36, 2024.
Jiang, Z., Ren, Y., Ye. Z., Liu. J., Zhang, C., Yang, Q., S., Huang, R., Wang, C., Yin, X., et al. Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias, arXiv preprint ar Xiv:223006.000 03509, 2023b.
Lee, K., Kim, D. W., Kim, J., and Cho, J. Ditto- tts: Effi- cient and scalable zero-shot text-to-speech with diffusion transformer. arXiv preprint arXiv:2406. 1 1427,, 2024.
Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang. D., Liu, Y., Leng, Y., Song, K., Tang, S., et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and dif- fusion models. arXiv preprint arXiv:2403.03/1 2024.
Lee, S.-H., Kim, S.-B., Lee, J.-H., Song, E., Hwang. M.-J., and Lee, S.-W. Hierspeech: Bridging the gap between text and speech by hierarchical variational inference us- ing self-supervised representations for speech synthesis. Advances in Neural Information Processing Systems, 35: 16624-16636, 2022.
Kahn, J., Riviere, M., Zheng. W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. (1699-6633 IEEE, 2020.
Li, A. Han, C., and Mesgarani, N. Styletts: A style-based generative model for natural and diverse text-to-speech synthesis. arXiv preprint arXiv:2205. 15439, 2022.
Li, Y. A., Han, C., Raghavan, V., Mischler, G., and Mes- garani, N. Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems, 36, 2024a.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint 200.:2000..836,,, 2020.
Kim, J., Kong, J., and Son, Conditional variational au- toencoder with adversarial learning for end-to-end text-to- speech. In International Conference Machine Learn- ing. pp. 5530-5540. PMLR, 2021.
Li, Y. A., Jiang, X., Han, C., and Mesgarani, N. Styletts-zs: Efficient high-quality zero-shot text-to-speech synthesis with distilled time-varying style diffusion. arXiv preprint arXiv:2409. 240.08800,0008, 2024b.
Kim, J., Lee, K., Chung, S., and Cho, J. Clam-tts: Improving neural codec language model for zero-shot text-to-speech. arXiv preprint arXiv:2404.000 0,0711,,12222.. 2024.
Liu, X., Zhang, X., Ma, J., Peng, J., et al. Instaflow: One step is enough for high-quality diffusion based text-to- image generation. In The Twelfth International Confer- ence on Learning Representations, 2023.
Kingma, D., Salimans, T., Poole, B., and Ho, J. Varia- tional diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021.
Liu, Z.,, Wang, S., Inoue, S., Bai, Q., and Li, H. Autoregres- sive diffusion transformer for text-to-speech synthesis. arXiv preprint
Stl.:2240005555.. 2024.
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv: 13X.v:31122.11111 14, 2013.
Loshchilov, I. and Hutter, F. Fixing Weight Decay Regular- ization in Adam, 2018. URL https: /tttt:/e//ppeeevien. net / formm?ormm? 1it.k
Koizumi, Y., Zen, H., Karita, S., Ding, Y., Yatabe, K., Morioka, N., Bacchiani, M., Zhang, Y., Han, W., and Bapna, A. Libritts-r: A restored multi-speaker text-to- speech corpus. arXiv preprint 2rXiv:2305.18802 2023.
Lovelace, J., Ray. S., Kim, K., Weinberger, K. Q. and Wu, F. Simple-tts: End-to-end text-to-speech synthesis with latent diffusion. 2023.
Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., and Ku- mar, K. Blighfidelity audio compression with improved rvggan. Advances in Neural Information Processing Sys- tems, 36, 2024.
Mao, X., Li, Q. Xie, H., Lau, R. Y., Wang, Z., and Paul Smolley, S. Least squares generative adversarial networks. In Proceedings of the IEEE international con- ference on computer vision, pp. 2794-2802, 2017.

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Meng, L.,, Zhou, L.,, Liu, S., Chen, S., Han, B., Hu, S., Liu, Y., J., Zhao, S., Wu, X., et al. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.08551, 2024.
End-to-end text-to-speech synthesis with human-level quality. IEEF Transactions on Pattern Analysis and Ma- chine Intelligence, 2024.
Thoidis, I., Gaultier, C., and Goehring, T. Perceptual analysis of speaker embeddings for voice discrimina- tion between machine and human listening. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Min, D., Lee, D. B., Yang, E., and Hwang, S. Meta- stylespeech: Multi-speaker adaptive text-to-speech gener- ation. In International Conference on Machine Learning, pp. 7748-7759. PMLR, 2021.
Nguyen, T. A., Hsu, W.-N., d' d'Avirro, A., Shi, B., Gat, I., Fazel-Zarani, M., Remez, T., Copet, J., Synnaeve, G., Hassid, M., et al. Expresso: benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.0577 05725, 2023.
Wang, C.,, Chen, S., Wu, Y., Zhang, Z., Zhou, I.., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint WrXiv22301021111 2023a.
Wang, H., Liang, C., Wang, S., Chen, Z., Zhang, B., Xiang, X., Deng, Y., and Qian, Y. Wespeaker: A research and production oriented speaker embedding learning toolkit. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023b.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195-4205, 2023.
Peng, P. Huang, P.-Y,, Li, D., Mohamed, A., and Har- wath, D. Voicecraft: Zero-shot speech editing and text- to-speech in the wild, arXiv preprint arXiv:2403. 16973, 2024.
Wang. X., Thakker, M., Chen, Z., Kanda, N., Eskimez, S. E.. Chen, S., Tang, M., Liu, S., Li, J., and Yoshioka, T. Speechx: Neural codec language model as a versatile speech transformer. arXiv preprint arXiv:23085 068733, 2023c.
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411 2020.
Wang. Y., Zhan, H.,, Liu, L., Zeng, R., Guo, H., Zheng, J., Zhang, Q., Zhang, X., Zhang, S., and Wu, Z. Maskget: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.007555 2024.
Ren, Y., Hu, C., Tan, X.. Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint 20Xi:2000.00458, 2020.
Wu, Y., Tan, X., Li, B., He, L.. Zhao. S., Song, R., Qin, T., and Liu, T.-Y. Adaspeech 4: Adaptive text to speech in zero-shot scenarios. arXiv preprint 22X1:2204.004336, 2022.
Salimans, and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint 20Xi:22022.00/22 12, 2022.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042. 2023.
Yamagishi, J., Veaux, C., MacDonald, K., et al. Cstr vetk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). 2019.
Sauer, A., Boesel, F., Dockhorn, T., Blattmann, A., Esser, P.. and Rombach, R. Fast high-resolution image synthe- sis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403. 12015, 2024.
Yang, D., Huang, R., Wang. Y., Guo, H., Chong, D., Liu, S., Wu, X., and Meng, H. Simplespeech 2: To- wards simple and efficient text-to-speech with flow-based scalar latent transformer diffusion models. arXiv preprint arXiv: 13893, 2024.
Shen, K., Ju, Z., Tan, X., Liu, E., Leng, Y., He, L., Qin, T., Bian, J., et al. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, 2024.
Ye, Z., Xue, W., Tan, X., Chen, J., Liu, Q., and Guo, Y. Co- mospeech: One-step speech and singing voice synthesis via consistency model. In Proceedings of the 31 ACM International Conference Multimedia, pp. 1831-1839 2023.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. arXiv preprini ar arXiv:2303.01466 2023.
Ye, Z., Ju, Z., Liu, H., Tan, X., Chen, J., Lu, Y., Sun, P., Pan, J., Bian, W., He, S., et al. Flashspeech: Efficient zero- shot speech synthesis. arXiv preprint arXiv:2404.14700, 2024.
Tan, X., Chen, J., Liu, H., Cong, J., Zhang. C., Liu, Y., Wang, X., Leng, Y., Yi, Y. He, I.., al. Naturalspeech:
12

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Yin, T., Gharbi, M., Park, T., Zhang, R., Shechtman, E., Du- rand, F., and Freeman, W. T. Improved distribution match- ing distillation for fast image synthesis. arXiv preprint 1rX1 2024a.
Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F,, Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 661 6613-6623, 2024b.
Zhang, D., Li, Z., Li, S., Zhang, X., Wang, P., Zhou, Y., and Qiu, X. Speechalign: Aligning speech generation to human preferences. arXiv preprint wxxv:2404.05600 2024.
13

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
A. Mode Shrinkage
To further explore the effects mode shrinkage, we conducted experiments on unconditional diversity and mode coverage. Specifically, we used a continuation task where the model was asked to generate speech following truncated prompt with its full text transcription, allowing us compare the generated speech to its corresponding ground truth from real speakers. We evaluated two key aspects of speech: pitch (FO) and energy. As shown in Figure 5, the student model closely matches the teacher's distribution in both FO and energy, demonstrating minimal mode shrinkage in contrast to the results shown in Figure 2, where mode shrinkage was evident.
Distribution (Speaker 61)
Log Energy Distribution (Speaker 61)
0.030
Ground Truth
0.35
Ground Truth
Teacher Model
Teacher Model
0.025
Student Model
0.30
Student Model
0.020
Density
Density
0.20
0.015
0.15
0.010
0.10
0.005
0.05
0.000
150
200
250
400
450
12
FO Value
Log Energy Value
FO Distribution (Speaker 1580)
Log Energy Distribution (Speaker 1580)
Ground Truth
Ground Truth
Teacher Model
Teacher Model
0,010
Student Model
0.20
Student Model
0.008
0.15
Density
Density
0.006
0.10
0.004
0.05
0.002
0,000
0.00
100
200
300
400
500
-500
-2.5
0.0
2.5
5.0
7.5
10.0
12.5
FO Value
Log Energy Value
Figure 4. Two examples for mode coverage with continuation task from LibriSpeech rest-clean subset. The model continues from prompt with the exact same text as the ground truth. This task synthesizes speech with varying prompts and texts but from the same speaker, allowing to compare the mode coverage without the same text and prompt. The student exhibits very similar behavior to the teacher and shows minimal mode shrinkage. The misaligament in energy between ground truth and our models is caused by normalization during data pre-processing where the audio is normalized between -1 I in amplitude, causing the generated samples to have different amplitude range.
We further assessed the model's 's mode coverage quantitatively by calculating the Wasserstein distance between the student and teacher models, as well as the ground truth, in terms of pitch (FO) and energy. The Wasserstein distances Wi, fo (for pitch) and Wy (for energy) were computed across all 40 speakers in the LibriSpeech test-clean subset. Additionally, we compared the Wasserstein distance between the student and teacher W (Pa. P&) in both conditional and unconditional settings. The conditional case involved synthesizing speech 50 times with the same text and prompt, while the unconditional case used varying texts and prompts from the same speaker as a speech continuation task.
14

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Table Wasserstein distance between student distribution (ps), teacher distribution (p²) and real data distribution (Prea) when samples are generated with the same text and prompt and varying texts and prompts in terms of pitch (FO) and log energy.
Sample Conditons
Aspect
W (po, Paums)
W W(p: Paam))
W W(pe.
Varying text-prompt pairs (unconditional)
Pitch (Wro fo
3.35
2.25
2.55
Same text-prompi pairs (conditional)
Pitch (W fo
16.53
Varying text-prompt pains (unconditional)
Energy (Ww)
5.47
4.88
1.34
Same text-prompt pairs (conditional)
Energy (Wx)
12.49
As shown in Table 6, the difference between the student and teacher in terms of Wasserstein distance to the ground truth is relatively small in the unconditional case, and the distance between the student and teacher is much smaller compared to the conditional case (2.55 vs. 16.53). This suggests that the reduction in diversity, or mode shrinkage, primarily occurs in the conditional setting (i.e., when synthesizing with the same text and prompt). In the unconditional setting, the student model still spans the entire support of the teacher's distribution and closely matches the ground truth distribution.
Given that zero-shot TTS highly conditional, where the output must closely match the prompt in both voice and style, this reduction in conditional diversity is not necessarily drawback. In fact, this narrowing of diversity is often preferred by human listeners, it leads to outputs that are more aligned with the prompt, as demonstrated in Figure 2 and Table 5,
B. Additional Evaluation Results
conducted additional evaluations acoustic features that capture emotional nuances in speech, following Li et al. (2022), focusing on pitch (mean and standard deviation), energy (mean and standard deviation), Harmonics-to-Noon Ratio (HNR), jitter, and shimmer.
Table 7 compares our model with several baselines. Our model consistently outperforms others across all metrics, except for energy mean, likely due to data normalization during preprocessing, which scales audio between -1 and 1, misaligning the energy with the prompt. Nevertheless, our model's higher scores across other features demonstrate its capability to reproduce the emotional content of the prompt speech effectively.
Table 7. Correlation of acoustic features related to speech emotions between synthesized speech and prompt compared to other baseilne models.
Pitch standard deviation
Energy standard deviation
Pitch mean
Energy mean
Model
HNR
Jitter
Shimmer
DMOSpoech (N=4)
0.93
0.52
0.40
0.52
0.86
0.77
0.69
Teacher (N=128)
0.86
0.37
0.30
0.34
0.79
0.65
0.56
DITTo-TTS
0.89
0.41
0.76
0.17
0.82
0.71
0.65
VoiceCraft
0.84
0.38
0.74
0.23
0.78
0.61
0.60
CLaM-TTS
0.85
0.39
0.61
0.31
0.79
0.66
0.61
XTTS
0.91
0.42
0.38
0.01
0.85
0.70
0.64
In the ablation study presented in Tables in 8, we compare the impact of different training strategies on preserving emotional content in synthesized speech. The teacher model shows strong correlations for most acoustic features, while DMD only models demonstrate performance improvements with additional sampling steps, similar SIM results in Table Adding CTC loss improves word error rate (WER) but does significantly enhance speaker-related features. However, including SV loss significantly improves speaker-related features, with the model trained with SV loss only achieving the highest scores in multiple metrics, such pitch mean (0.94), HNR (0.87), and shimmer (0.65). This highlights the importance of SV loss in capturing speaker identity and emotional content.
Finally, reducing the batch size from 96 to 16 resulted in slight performance drop across most metrics, demonstrating the importance of maintaining a larger batch size for optimal performance in distribution matching distillation.
15

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
Table Correlation of acoustic features related speech emotions hetween synthesized speech and prompt for the ablation study. The best-performing model is highlighted while the second best model is underlined.
Pitch standard deviation
Energy standard deviation
Pitch mean
Energy mean
Model
HNR
Jitter
Shimmer
Teacher (N=128)
0.86
0.37
0.30
0.34
0.79
0.65
0.56
DMD 2 only (N=1)
0.84
0.32
0.15
0.43
0.65
0.60
0.10
DMD only (N=4)
0.87
0.36
0.38
0.36
0.76
0.64
0.44
+ Lere only
0.91
0.40
0,34
0.40
0.77
0.63
0.46
+ Lsv only
0.94
0.54
0,41
0.52
0.87
0.77
0.65
DMeSpeech (N=4)
0.93
0.52
0.40
0.52
0.86
0.77
0.69
B. S. 96 16
0.92
0.48
0.39
0.51
0.85
0.74
0.60
C. Implementation Details
C.1. DAC Variational Autoencoder
We utilize latent audio autoencoder to compress raw waveforms into compact latent representations for diffusion modeling. Our architecture follows the DAC model proposed by Kumar et al. (2024), with a key modification to use a variational autoencoder (VAE) bottleneck instead of residual vector quantization, enabling continuous latent spaces and end-to-end differentiable training.
The DAC consists of encoder &, VAE bottleneck, and decoder DD The encoder maps the input waveform y E R into latent representation E _CXL, where C and L denote channels and downsampled temporal resolution. The VAE bottleneck introduces stochasticity by modeling x distribution, and the decoder reconstructs the waveform by minimizing the reconstruction loss.
The encoder applies an initial convolution followed by residual units with dilated convolutions at scales 1,3, 9 to capture multi-scale temporal features. After each block, strided convolutions reduce the temporal resolution by a factor of 1200. For 48 kHz audio, the encoded latent is 40 Hz, making it ideal for efficient speech synthesis tasks. The latent channel dimension of our autoencoder is C 64.
The encoder's output is split into mean and scale o parameters:
z == C~.().
(11)
where z sampled using the reparameterization trick (Kingma, 2013). The decoder mirrors the encoder with transposed convolutions and residual units upsample latent representations back the original waveform = Dzz), where is the reconstructed waveform. The encoder and decoder architectures are the same as DAC (Kumar al., 2024).
The KL divergence between the approximate posterior q(lly) and prior p(z) is computed as:
LKI. By
[n of log of m;
(12)
where N' is the number of channels, and m; is the channel mask. The autoencoder is trained to minimize combination of reconstruction loss and KL divergence:
LAE = Ey llly 5111] CKL
(13)
where AXL = 0.1 to balance the KL loss. In addition to the KL loss, we also employ adversarial training following Kumar al. (2024) with the complex STFT discriminator.
16

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
C.2. DMOSpeech
In this section, we present the implementation details of our DMOSpeech model, including the noise schedule, gradient calculation of DMD loss, detailed architecture, and sampling algorithm.
C.2.1. SHIFTED COSINE NOISE SCHEDULE
We follow Lovelace et al. (2023); Hoogeboom et al. (2023) and use the shifted cosine noise schedule with a and oe denoting the amount of signal and noise at time The noise-to-signal ratio (SNR) X = 04/0; of the noise schedule shifted by a factor s, from which the shifted SNR X1, and noise schedule ag Ct, are defined:
(14)
X
Ct
A
8
at
8²,
(15)
tte
Using the fact o₂ sigmoid (log():)) stated in Kingma al. (2021), the shifted noise schedule can then computed in the log space for numerical stability:
at = sigmoid (Ar) + 2 log(s))
(16)
v 1Vl-aai. ai.s
(17)
Lower emphasizes the higher noise levels and can potentially improve the model's performance. We set = 0.5 following Lovelace et al. (2023) as it is shown to produce the most robust results.
C.2.2. GRADIENT CALCULATION DMD LOss
The gradient of the DMD loss with respect to the generator parameters 0 is given by eq. 3. The actual implementation of gradient calculation follows the following steps.
We first sample latent variables x, generated via forward diffusion process
X Xx + orE,
(18)
where is the clean latent representation, and (0,
The clean latents xg and stfike then are estimated using the predicted noise by both of the teacher fo and student I diffusion models following eq. 5.
From there, we calculate the numerical gradient of LDVD. define the following quantity as the difference between the ground truth clean latent and estimated latents:
Preal Xo
x
(19)
Plake Xo - fake
(20)
Then the difference in score (numerical gradient) can be calculated as:
A
WiCk (Sreal sp)
(21)
(x+ - 0x - (xi ax
wi
of
(22)
2
(- (xtpaa x(fak))
(23)
0f
(24)
where the weighting factor wi is defined as:
o/
of 0t
W1
(25)
ai
[x0
Preal 1
Hence, eq. 21 can be written as:
A (Preal Prake)
(26)
Preal I
which is back back-propag Go via gradient descent algorithm.
17

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
C.2.3. DETAILED ARCHITECTURE
In this section, we present the architecture of our Diffusion Transformer (DiT) model (Peebles & Xie, 2023). The DiT model integrates diffusion processes with transformer architectures to generate high-quality speech representations conditioned on textual input.
Our DiT model consists of the following key components:
Embedding Layers: Transform input IPA tokens, binary prompt masks, and speech latents into continuous embeddings.
Transformer Encoder: Encodes the textual input (IPA tokens) into contextual representations.
Transformer Decoder: Decodes the latent representations conditioned on the encoder outputs and additional embeddings.
The model parameters are summarized Table
Table 9. DMOSpeech DiT model parameters.
Parameter
Value
Latent dimension
64
Model dimension
1024
Fecd-forward dimension
3072
Number of attention heads
Number of encoder layers
Number of decoder layers
ferwwrd activation function
SwiGLU
Text conditioning dropout
0.1
Noise schedule shitting scale (s)
0.5
The embedding layer maps input tokens and latent variables into continuous embeddings. Specifically, IPA tokens are embedded into vectors of size 1024 using an embedding matrix, and speech latents are projected from dimension 64 to 1024 using linear layer. A binary mask prompt indicating prompt positions in the latent sequence is encoded into mask embedding, and sinusoidal time embedding represents the diffusion timestep t. Positional embeddings are added to both IPA and latent embeddings to encode positional information.
The encoder processes the embedded IPA tokens through 8 layers, each containing multi-head self-attention and feed-forward sublayers with layer normalization and residual connections. The -orwwrd sublayers use a hidden dimension of 3072 and the SwiGLU activation function. The encoder outputs the text condition
The decoder generates latent representations conditioned on the encoder outputs and additional embeddings over 16 layers. Each layer includes self-attention, normalization (AdaI.N), conditioned on the timestep embedding, is applied within the decoder. The output layer projects the decoder outputs back to the latent space dimension of 64 using a linear layer.
with the encoder outputs, and weew-fo sublayers. Adaptive layer
Classifier- free guidance (CFG) is employed by randomly dropping the textual conditioning during training with a probability of 0.1 and is the guidance scale. The modified Sycal with CFG becomes:
Sreal [x;; w) = S (x ; c, m. () + w (So(x, ;c, m, () - fx( 0. m, ()),
(27)
where 0 denotes the null condition of c which a fixed embedding. We set w both for inference the teacher model and DMD training.
The teacher model generates samples through DDPM sampler (Ho et al., 2020) with discrete time steps ( where N is the total sampling steps:
[0, 1]
of (x ;c, f :,, + Oen-ie
Xn-1
Xn
(28)
at
an
~ n 1, and 6 nf
18

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
C.2.4. DMD SAMPLING
Our sampling algorithm of the student (DMOSpeech) is similar that of the consistency model (Song et al., 2023). The sampling procedure is outlined in Algorithm 1.
Algorithm 1 DMD Multi-Step Sampling Procedure Require:
the text embeddings
Xprompe: the prompt latent
L: total length of the target speech
{t noise level schedule with N steps
1: Initialize noisy latent x; ~ (O. of shape (L, dlatent)
2: for = 1 to N do
3: Xr X; ② (1 m) + Xprompt m
b. Re-apply prompt > Run student network Predict x from Re-apply prompt to X
4: + Go(xi5 c, m,
5:
Xo
at, ot.-
6:
Xn ① (1 m) xrompt ② m
7:
i < N then
8:
(O.
9:
X = attt xo + 0t:t1
b Re-noise x to get new x at t,41
end if
11: end for
12:
return Xo
C.3. Latent CTC-based ASR Model
To directly optimize word error rate (WER) within our speech synthesis framework, we implement a Connectionist Temporal Classification (CTC)-based ASR model that operates on latent speech representations. Traditional ASR models work on raw audio or mel-spectrograms, adding computational overhead and potential mismatches when integrated with latent-based synthesis since need decode the latent back into waveforms before computing the ASR output. Our latent ASR model processes these representations directly, enabling efficient, end-to-end computation of the CTC loss and direct WER optimization.
The ASR model based on the Conformer architecture (Gulati et al., 2020), which effectively captures local and global dependencies using convolution and self-attention. Input latent representations z RTxd are processed through a 6-layer conformer stack and the model outputs a logit for each latent token over IPA phonemes.
The ASR model is trained using the CTC loss, allowing alignment-fiee free training of sequence-to to sequence models. The CTC loss is defined using softmax function:
Lere
log p (y o),
(29)
where y is the target IPA sequence, represents the logits over the IPA symbols, and p (y |o) is computed by summing over all valid alignments between the input and target sequences. The probabilities are calculated as:
exp (O
(x.(t)
(30)
ΣV k= exp (or,k)
We trained our ASR model on Common Voice (Ardila et al., 2019) and LibriLight (Kahn et al., 2020) datasets for 200k steps with the AdamW (Loshchilov & Hutter, 2018) optimizer. The optimizer configuration is the same as teacher training described in Section 4.1.
C.4. Latent Speaker Verification Model
We develop a latent speaker verification (SV) model that operates directly on latent speech representations in order to optimize speaker similarity within our speech synthesis framework. Unlike traditional SV models, which process raw
19

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
audio waveforms, our latent SV model integrates seamlessly with our latent-based synthesis, enabling efficient, end-to-end computation of speaker verification loss for direct speaker similarity optimization.
Our latent SV model fine-tunes our CTC-based ASR model for feature extraction following (Cai & Li, 2024) and integrates it with an SCAPA-TDNN architecture (Desplanques et al., 2020) for speaker embedding extraction. We train the latent SV model using a distillation approach, transferring knowledge from two pre-trained teacher models: a ResNet-based SV model 5 from the WeSpeaker (Wang et al., 2023b) and EPACA-TDNN with a fine-tuned WavLM Large model as the feature extractor. The training objective minimizes the cosine similarity loss between embeddings from the latent SV model and the concatenated embeddings from the teacher models:
Lsv E2,y
@seacher Glatent
(31)
|reacter|
||
|latent
where elatent and Oreacher are the embeddings from the latent SV and teacher models, respectively.
Our latent SV model was trained on Common Voice (Ardila et al., 2019) and LibriLight (Kahn et al., 2020) datasets for 400k steps with the AdamW optimizer. Since we did not use VoxCeleb dataset that was used originally to train the teacher SV models, we used data augmentation to shift the pitch of the speakers to create new speaker identity to prevent overfitting during training.
D. Human Rating Correlations
We generated scatter plots to visualize the relationships between the four subjective metries: MOS-N (naturalness), MOS-Q (sound quality). SMOS-V V (voice similarity), and SMOS-S (style similarity). and two objective evaluation metrics: word error rate (WER) and speaker embedding cosine similarity (SIM). The scatter plots are displayed in Figure 5, and they cover all subjective evaluation experiments conducted in this work at the utterance level.
Despite the noise and variance in the utterance-level subjective ratings, the plots reveal important trends. A strong correlation exists between human-rates speaker similarity (SMOS-V V and SMOS-S) and the SIM score from the speaker verification model, with correlation coefficients of 0.55 and 0.50, respectively. This highlights the alignment between subjective human judgments and the objective speaker embedding similarity. On the other hand, there is a weaker but still significant negative correlation between WER and both naturalness (MOS-N) and sound quality (MOS-QU Q). with coefficients of -016 for both. These findings validate our approach to directly optimize these metrics. Future research could explore other differentiable metrics or reward models that align even more closely with human auditory preferences.
E. Evaluation Details
E.1. Baseline Models
This section briefly introduces the baseline models used in our evaluations and the methods employed to obtain the necessary samples.
CLaM-TTS: CLaM-TTS (Kim et al., 2024) is a strong autoregressive baseline for zero-shot speech synthesis, trained various datasets including Multilingual LibriSpeech (MLS) (Pratap et al., 2020), GigaSpeech (Chen et al., 2021). LibriTTS-R (Koizumi et al., 2023), VCTK (Yamagishi et al., 2019), and LJSpeech (Ito & Johnson, 2017). Since this model is not publicly available, we obtained 3,711 samples from the authors using instructions provided by the authors at httpp:: / thub. cow/keonle ee 92099422////// uate- zer-- hhtt-tss.
DiTTo-TTS: DiTTo-TTH -TS (Lee et al., 2024) is a previous of (SOTA) end-to-end model for zero-shot speech synthesis, trained on the same datasets as CLaM-TTS, with the addition of Expresso (Nguyen et al., 2023). Like CLaM-TTS, this model is also not publicly available, so we acquired the same set of 3,711 samples from the authors.
NaturalSpeech NaturalSpeech 3 (Ju et al., 2024) is previous SOTA model in zero-shot speech synthesis, trained on LibriLight (Kahn al., 2020). Using factorized codec and discrete diffusion models, it achieves near-human
$ 'Available at https: https/:////ggggga/ageee http://www......oooeeeee//eeeeee resnet..3
Shttps:// thub. cww/mi www.//icooooot//ooooooooooo /uni http://t/tnnnnnnnsseeeeeeeeee htppecch/tree//aeee Tttpps: //gi thub. htt.://///ccoooeeeeeeeeeeeeeeeee wwwpchh//awwuueett..tt
httpe///eeeeaieeeoeeeeaea/////// speatttaerrrsssessssreeeeee f fication
20

---

tested on DMOSpeech: Direct Metric Optimization Speech Synthesis
SIM vs. Voice Similarity
SIM vs. Style Similarity
0.8
0.55
< 0.011)
0.8
0.50 (p x 0.01)
0.6
0.6
SIM
SIM
0.4
0.4
0.2
0.2
SMOS-V
SMOS-S
WER vs. Naturalness
WER vs. Sound Quality
0.6
0.6
0.16 (p < 0.01)
0.16 (p < 0.01)
0.5
0.5
0.4
0.4
WER
WER
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
1.5
2.0
2.5
3.0
4.0
4.5
2.5
3.0
3.5
4.0
4.5
5.0
MOS-N
MOS-Q
Figure 5. Top: Scatter plots showing the relationship between human-rated naturalness (MOS-N) and sound quality (MOS-Q) versus word error rate (WER). The correlation coefficients are -0.16 for both, indicating weak negative correlation (p < 0.01). Bottom: Scatter plots of human-rated voice similarity (SMOS-V) and style similarity (SMOS-S)
speaker embedding cosine similarity (SIM). The correlation coefficients are 0.55 and 0.50, reflecting a strong positive correlation (p < 0.01). These plots demonstrate how objective evaluations (WER and SIM) align with subjective human ratings.
performance in prompt speaker similarity. Since it is not publicly available, collected 40 samples from the authors, along with text transcriptions and 3-second prompt speeches, to synthesize speech for comparison. We also sourced 7 official samples from https: // /www.www mi c crosoft. www.ccom//ennee http://enntssseeeeerrrcrppppoe/e the LibriSpeech test-clean subset, totally 47 samples.
StyleTTs-ZS, StyleTTS-ZE ZS (Li et al., 2024b) is another previous SOTA model for zero-shot speech synthesis, known for its fast inference speed and high naturalness and speaker similarity. As the model is not publicly available, we requested 47 samples from the authors to match those provided by Ju et al. (2024).
VoiceCraft (Peng et al., 2024) is strong autoregressive baseline model trained GigaSpeech (Chen et al., 2021) and LibriLight (Kahn et al., 2020), performing well in speaker similarity and can be used for speech editing, This model is publicly available at hs / / g thub. com/ asooppy /oo ceCra ft. and we synthesized 3,711 samples using the same text and 3-second speech prompts provided for CLaM-TTS and DiTTo-TTS with the 830M TTS-enhanced model.
VoiceCraft:
"VoiceCra
XTTS: XTTS (Casanova et al., 2024) is another strong zero-shot speech synthesis baseline, trained on various public and proprictary datasets totaling around 17k hours. The model is publicly available at https: h/hhggingfacc..... co coqui XxTS-v2.T and we synthesized the same 3,711 samples as above.
21

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
E.2. Subjective Evaluation
Progress: 2/31
Listen reference recording and sample recording, which be real voice synthetic voice attempting to mimic the voice of the reference the
different language. Cast ruting from 1. Very Poor 5. Excellent the following axes:
Naturalness: Does the voice sound like real buman? 5 Real, 1 Synthetic and unnatural, 2-4 Somewhat synthetic but okay for creating content.
Voice Similarity: Does the voice sound like the person in the reference? Identical, Entirely different, 2-4
Somewhat similar but not identical.
Quality: Is the audio quality maintained? 5 Same better than the reference, Unintelligible, 2-4 Worse than the original.
Style Similarity: Do the voice's speaking style and emotion match the reference 7 Almost identical, 1 = Entirely different, 2- 4 Somewhat different.
If the audio sample entirely unintelligible, please mark "yes" in the last question. Otherwise, please mark "no".
Reference
0:00 / 0:03
b
Sample
0:00 0:00/0:00 00:07
bD
Naturalness:
Voice similarity:
2
3
N/A
1
4
5
N/A
Quality:
Style similarity:
2
3
4
N/A
1
3
4
5
N/A
Is content broken?
Submit
Yes
No N/A
Figure 6. Screenshot of the subjective evaluation survey used for the perceptual quality assessment of speech synthesis models. Participants are presented with reference (prompt) and sample to evaluated and are asked to rate various attrihutes such as naturalness, voice similarity, style similarity, and quality scale from 1 to 5. If the sample is unintelligible, participants must mark it as "Yes" under the "Is content broken?" section. The survey prevents submission if any slider remains at the default "N/A" position, ensuring that each aspect is rated.
conducted two subjective evaluations using the Prolific crowdsourcing platform to assess the perceptual quality the generated speech samples. These evaluations measured key attributes including naturalness, voice similarity, style similarity, and audio quality based reference speech sample provided to the raters.
Because some workers may "game" the systems by answering randomly, or skipping the reference sample, we used two forms of validation tests. The first uses mismatched speaker where the test presents the workers with different voices for the reference and test sample, both being real speakers. a participant rated these mismatched samples with a speaker similarity score above 3, all their ratings were excluded from the analysis. The second validation test involved identical sample pairs, where participants were asked to rate identical reference and sample pairs. If any of the subjective attributes, including naturalness, similarity, style, or quality, were rated below for these identical pairs, all responses from that participant were excluded.
The first subjective evaluation experiment, referred to the "bigger" experiment, involved 501 unique workers. There are total of 80 parallel utterances for each method, which include all end-to-end (E2E) baselines and models in the ablation study were rated. The results were present in Table 2 and 5. Each worker was assigned provide ratings for 30 samples. There are 4 validation tests in this experiment. Approximately 30% the responses were invalidated due to participants failing the validation test at least once. The second, "smaller" experiment that compared non-E2E baselines over utterances per method. There are 290 unique workers, with each worker completing 28 ratings. The validation test doubled per test.
hhtpps: /tttpss//wwprrrlfffccc
22

---

DMOSpeech: Direct Metric Optimization Speech Synthesis
In this smaller study, 40% of the ratings were invalidated because the stricter validation process led more failures. The number of invalid samples consistent with prior work carried similar platforms.
The survey (Figure 6) interface presented participants with reference (prompt) and a corresponding sample recording. Participants rated each sample on a scale from to across several categories:
1. Naturalness, evaluating how real or synthetic the voice sounded;
2. Quality, determining whether the audio quality was maintained or degraded compared to the prompt;
3. Voice similarity, assessing how closely the sample matched the reference speaker;
4. Style similarity, considering the alignment of the speaking style and emotion;
5. Intelligibility, for which raters were asked to mark such to flag broken samples during the analysis if the audio sample was entirely unintelligible.
The last rating category "is the content broken;" helps us to identify if any samples are unintelligible which would indicate completely failed generation or corrupted files. In the end, we do not have any samples that are rated "broken" by the majority.
Compensation for both experiments to rate of $15 per hour, higher than Prolific's recommendation of $12 per hour with target average time of 12 minutes per test.
23


Reviews:
[{"review_id": "EXFomwRX3H", "reviewer": "Reviewer_yWEm", "summary": "Diffusion models have shown strong potential in speech synthesis tasks such as text-to-speech (TTS) and voice cloning. However, their iterative denoising process is computationally expensive, and previous distillation methods have led to quality degradation. Existing TTS approaches also suffer from non-differentiable components or iterative sampling, preventing true end-to-end optimization with perceptual metrics.\n\nTo address these issues, the authors propose DMOSpeech, a distilled diffusion-based TTS model that achieves both faster inference and superior performance compared to its teacher model. A key innovation of DMOSpeech is its ability to enable direct gradient pathways to all model components, allowing for the first successful end-to-end optimization of differentiable perceptual metrics in TTS. The model incorporates Connectionist Temporal Classification (CTC) loss and Speaker Verification (SV) loss, aligning speech synthesis quality with human auditory preferences.\n\nExtensive experiments, including human evaluations, demonstrate significant improvements in naturalness, intelligibility, and speaker similarity, while also reducing inference time by orders of magnitude. This work introduces a new framework for optimizing speech synthesis directly with perceptual metrics, setting a new standard for high-quality and efficient TTS models.\n\n## update after rebuttal\nI have read rebuttal from authors and comments from other reviewers, I think this is a good paper that I vote for Accept", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their positive recommendation for our paper. We appreciate the recognition of our model's ability to enable direct gradient pathways for end-to-end optimization and the acknowledgment of our efficiency improvements. While we agree with many points raised in the review, we would like to address two specific concerns:\n\n**On Theoretical Claims and Scientific Contribution**:\n\nWe respectfully disagree with the assessment that our theoretical claims are \"not significant, novel\" and that there is \"no significant contribution to the broader scientific literature.\" Our work makes several important theoretical and scientific contributions:\n\n- **Novel Distribution Matching Framework**: We present the first successful application of distribution matching distillation in speech synthesis that achieves superior quality to the teacher model. This counters the prevailing view in the field that distillation necessarily leads to quality degradation.\n\n- **Mode Shrinkage Insight**: Our analysis of mode shrinkage during distillation reveals a fundamental insight about conditional generation tasks: in strongly conditional generation, diversity reduction can be beneficial when it emphasizes high-probability regions without compromising output variation across different prompts and text inputs.\n\n- **Unified Optimization Framework**: By enabling direct metric optimization in a diffusion framework, we bridge the gap between two previously separate research areas: perceptual metric optimization and diffusion models, establishing a foundation for future research on optimizing generative models with human preferences.\n\nThese contributions extend beyond speech synthesis and offer valuable insights for any conditional generative modeling task, especially those requiring both quality and efficiency.\n\nWe believe our work advances both the theoretical understanding of generative model distillation and provides a practical framework applicable to numerous domains requiring conditional generation with perceptual quality constraints.\n\nThank you again for your overall positive assessment. We hope this clarification addresses your concerns regarding the broader impact and theoretical novelty of our work.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Methods are clear and the evaluation criteria are suitable.", "theoretical_claims": "Theoretical claims are not significant, novel, but not wrong for this specific application.", "experimental_designs_or_analyses": "Experiment design is clear and analysis supported the claim.", "relation_to_broader_scientific_literature": "No significant contribution to the broader scientific literature.", "essential_references_not_discussed": "No further reference missing in discussion.", "other_strengths_and_weaknesses": "## Strength \n- The paper introduces a novel approach to optimizing perceptual metrics in TTS by enabling direct gradient pathways, which has not been successfully achieved in previous models.\n- By reducing sampling steps from 128 to 4 while maintaining or improving quality, DMOSpeech addresses a major efficiency bottleneck in diffusion-based TTS.\n- The paper is well-structured, with clear explanations of the model architecture and loss functions. The inclusion of human evaluation results strengthens the validity of claims.", "other_comments_or_suggestions": "no further comments", "questions_for_authors": "no further questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d1XehTSy2M", "reviewer": "Reviewer_V6zd", "summary": "This paper presents DMOSpeech, a distilled diffusion-based speech synthesis model that achieves true end-to-end optimization of perceptual metrics, specifically through CTC loss for intelligibility and SV loss for voice similarity. The authors integrate these loss functions into a distilled student model trained via DMD2, enabling efficient inference without sacrificing synthesis quality.\n\n## update after rebuttal\n\n**I keep my initial assessment as the rebuttal addressed my concerns.**", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thorough evaluation and constructive feedback. Below, we address each point raised:\n\n**StyleTTS-ZS Comparison**\n\nThe performance of StyleTTS-ZS in our evaluation aligns with what was reported in their original paper. The fundamental architectural difference is that StyleTTS-ZS employs a specialized decomposition approach, modeling specific prosody components (F0, energy, duration) separately, while DMOSpeech adopts a more holistic end-to-end generation framework.\n\nOur approach offers several advantages:\n\n1. Greater generalizability across diverse speech conditions (such as non-speech vocalizations and multiple speakers in the same utterance). \n2. Support for end-to-end optimization with perceptual metrics\n3. Robustness to challenging audio conditions (e.g., background noise, processing artifacts, as we presented on our demo page)\n\nStyleTTS-ZS achieves high efficiency through its decomposition strategy (enabling one-step generation, since generating prosodic features only is easier than generating the whole speech), but this same design introduces limitations when faced with complex or noisy audio conditions. DMOSpeech maintains comparable efficiency while providing superior generalizability and performance.\n\n**Value of Super-Efficient Generation**\n\nWhile the teacher model achieves real-time generation (RTF < 1) on a high-end GPU (V100), there are compelling reasons to pursue even greater efficiency:\n\n- **Device Compatibility**: Enabling deployment on resource-constrained environments (CPUs, mobile devices, edge computing)\n\n- **Service Scalability**: A 13.7× reduction in inference time translates to substantially higher throughput for cloud services supporting millions of users\n\n- **Energy Efficiency**: Reduced computation requirements lead to lower power consumption and carbon footprint\n\nFor industrial applications, these efficiency gains are critical for accessibility, scalability, and sustainability.\n\nRegarding model scaling, while extremely large models (10B-100B parameters) are indeed being explored in production environments (e.g., by ByteDance's Seed-TTS and Amazon's Base-TTS), analyzing scaling behaviors was outside our paper's scope. Our distillation technique remains relevant regardless of teacher model size, as the efficiency benefits become even more pronounced with larger models.\n\n**Ablation of Individual Losses**\n\nAs shown in Table 4, we conducted comprehensive ablation studies applying CTC and SV losses individually:\n\n- **CTC Loss Only**: Achieved superior word error rate (1.79 vs. 1.94) but significantly lower speaker similarity\n\n- **SV Loss Only**: Produced slightly higher speaker similarity (0.70 vs. 0.69) but substantially worse WER (6.62 vs. 1.94)\n\nThese results demonstrate that combining both losses achieves the optimal balance between intelligibility and speaker similarity, which aligns with human preference as shown in our subjective evaluations.\n\n**Loss Optimization and Potential Trade-offs**\n\nThe reviewer raises an important point about potential conflicts between optimization objectives. In theory, overly aggressive optimization of auxiliary losses (CTC, SV) could indeed harm performance by causing distribution mismatches with the training data, especially when the loss is optimized below that achieved in ground truth data.\n\nTo mitigate this risk, we carefully balanced the gradient contributions from each loss component, ensuring the auxiliary loss gradients are comparable in magnitude to the primary DMD loss. This calibrated approach prevents any single objective from dominating and maintains distributional alignment throughout training.\n\nRegarding CLAP regularization, while it's an intriguing direction, we focused specifically on speech synthesis rather than general audio generation in this work. We appreciate the suggestion and will mention this as a potential future direction in our revised manuscript.\n\n**Theoretical Contributions**\n\nWhile our paper emphasizes empirical results, we provide some theoretical contributions, especially our **Analysis of Mode Shrinkage**. Our detailed examination of distributional changes during distillation (**Figure 2** and **Appendix A**) offers novel insights into how distillation affects output diversity in conditional generation tasks. Our analysis shows that although distillation causes a loss in diversity for a fixed prompt and text input, this reduction in diversity is not necessarily negative or could even be beneficial to the performance of conditional generation, and the mode coverage is not compromised when the model processed different prompt and text inputs. \n\nThese insights contribute to the theoretical understanding of both diffusion model distillation and perceptual metric optimization in generative models.\n\nWe thank the reviewer again for their thoughtful comments and positive assessment. Hope our responses address their concerns satisfactorily.", "claims_and_evidence": "- The claims are very clear that their newly introduced plausible loss functions work in practice. I cannot believe these loss functions have not yet been applied to this diffusion distillation so far, and support this paper acceptance.\n- The paper backs its claims with comprehensive experiments. Especially, exceling its teacher with 4-step synthesis with such auxiliary loss functions are impressive, but also surpassing strong baselines (e.g., NaturalSpeech 3). However, in all Tables, the authors should write the number of inference steps. For example, StyleTTS-ZS is 1-step generation, and the performance seems to be different from the original paper. Can you explain this? What makes StyleTTS-ZS and DMPSpeech perform differently?\n- The teacher model is already achieving the real-time generation (RTF < 1). Why should we bother generating that fast algorithm? Say, the teacher is 10B model and clearly not a real-time. Maybe a usecase of this algorithm is to distill that teacher into few-step model that fits to real-time. Can the authors give a good intuition why the model sizes are not scaling to 10B - 100B? Is there an insight of the model behavior once we really scale up to this region?\n- Is there any ablation that applies CTC and SV losses individually? Will there be any discrepancy between the expected outcome and the real outcome?\n- Will the code released?", "methods_and_evaluation_criteria": "Yes. They all make sense to me.", "theoretical_claims": "There is no theory on this paper. One question. When the loss function is fully optimized, will both CTC loss and SV loss not hurt the optimality? It is true in the image domain that using CLIP regularization with strong weight hurts the performance. How about CTC and SV? Also, will CLAP regularization work?", "experimental_designs_or_analyses": "Not carefully checked the validity of all details. Typically, in this GAN-based experiments, there are many hidden (or appendixed) materials that a reviewer can easily miss.", "relation_to_broader_scientific_literature": "Very related to the broad community", "essential_references_not_discussed": "Key references are discussed", "other_strengths_and_weaknesses": "One minor issue is about the contribution of this paper. It's all about the empirical study, and I was wonder if there's any good theoretic catch in this paper. Well, I understand it's very hard to do so, but is there any interesting analysis?", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oWTJwPtdoN", "reviewer": "Reviewer_yvih", "summary": "This paper introduces DMOSpeech, a distilled diffusion-based text-to-speech (TTS) model that achieves faster inference and superior performance compared to its teacher model. It has two advantages: (1) reducing sampling steps from 128 to 4 via distribution matching distillation, and (2) providing direct gradient pathways from noise input to speech output. This allows direct optimization of speaker similarity and word error rate through speaker verification (SV) and Connectionist Temporal Classification (CTC) losses. The comprehensive experiments demonstrate significant improvements across all metrics, outperforming the teacher model and other recent baselines in subjective and objective evaluations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback, but we believe there is a fundamental misunderstanding about our paper's contribution. Our work presents several key innovations that have not been explored in prior research, including FlashSpeech:\n\n**Clarification on Direct Metric Optimization vs. Adversarial Training**\n\nThe reviewer conflates direct metric optimization with adversarial (GAN) training, which are fundamentally different approaches:\nFlashSpeech does not implement direct metric optimization. After careful examination of the FlashSpeech (Ye et. al 2024) paper, we confirm it uses adversarial consistency training but makes no mention of direct metric optimization of perceptual metrics such as speaker similarity or word error rate. Their adversarial training is solely for improving general speech quality.\n\nOur direct metric optimization is novel. DMOSpeech enables true end-to-end optimization of specific perceptual metrics (SV loss for speaker similarity, CTC loss for word error rate) through differentiable pathways - not simply adversarial training. This is a significant advancement for TTS systems.\n\n**State of Direct Metric Optimization in TTS**\n\nThe reviewer suggests that \"both diffusion distillation and direct metric optimization in TTS are well studied.\" This is incorrect. As we explicitly state in our paper:\n\n>While optimizing perceptual metrics has shown promise in speech enhancement through approaches like MetricGAN for PESQ and STOI, and recent attempts have explored RLHF for improving naturalness, implementing these approaches in modern TTS systems has remained challenging. Previous attempts (e.g., YourTTS) reported minimal improvements from speaker similarity optimization due to their inability to propagate gradients through all model components. The field has struggled with this problem due to architectural limitations such as non-differentiable components or computationally prohibitive backpropagation through iterative sampling steps.\n\n**Relationship to DIFFUSION-GAN**\n\nOur work bears limited similarity to DIFFUSION-GAN. Our focus is not on GAN-based techniques but on enabling direct optimization of perceptually relevant metrics through a differentiable pathway created by one-step generation through distillation. This is a fundamentally different approach with different goals.\n\n**Novel Contributions**\n\nOur paper makes several novel contributions:\n\n1. We present the first distilled TTS model that consistently outperforms its teacher model (not merely matching it), while reducing inference time by over 13×.\n\n2. We introduce a framework enabling true end-to-end optimization of differentiable metrics in TTS, demonstrating substantial improvements in speaker similarity and word error rate.\n\n3. We provide comprehensive analyses establishing correlations between objective metrics and human perceptions, revealing new insights into sampling speed and diversity trade-offs.\n\nThese contributions represent significant advancements in the field of speech synthesis that are well-aligned with ICML's focus on machine learning innovations.\n\n**Regarding F5TTS and MASKGCT Comparisons**\n\nWe thank the reviewer for their suggestions for more baseline comparisons. Per the reviewer's suggestion, we have trained a new DMOSpeech model using F5-TTS as the teacher on the Emilia dataset and conducted comprehensive objective evaluations comparing our new model against both F5TTS and MASKGCT models on the SeedTTS-Eval benchmark dataset:\n\n| Model | SIM (en) $\\uparrow$ | WER (en) $\\downarrow$ | SIM (zh) $\\uparrow$ | CER (zh) $\\downarrow$ | RTF $\\downarrow$ |   \n|-----------|-----------|-----------|-----------|-----------|-----------|\n| MaskGCT | **0.717** | 2.62 | 0.752 | 2.27 | 1.21 |\n| F5-TTS (teacher, N=32) | 0.647 | 1.83 | 0.741 | 1.56 | 0.32 |\n| DMOSpeech (N=4) | 0.687 | **1.78** | **0.757** | **1.43** |  **0.06** |\n\nOur model has achieved similar or better performance than both MaskGCT and F5-TTS on SeedTTS-eval in both Chinese and English test sets for both intelligibility and similarity. Moreover, our model is significantly faster than both MaskGCT and F5-TTS.\n\nWe will include the complete evaluation results in the appendix of our revised manuscript. This additional analysis provides a more comprehensive understanding of how our approach compares to current state-of-the-art methods.\n\n**Regarding FlashSpeech Comparison**\n\nWe appreciate the suggestion to compare with FlashSpeech. However, despite our best efforts, we have been unable to conduct direct experimental comparisons due to the unavailability of publicly accessible pre-trained checkpoints for this model, as documented in https://github.com/zhenye234/FlashSpeech/issues/3.\n\nIn our revised manuscript, we will include a thorough discussion of FlashSpeech, addressing its approach and how it relates to our work. While we cannot provide direct experimental comparisons, this discussion will help contextualize our contributions among other efficient zero-shot TTS systems.", "claims_and_evidence": "Overall, the claims are clearly and well supported. \n\n1. The biggest problem of this paper is: This paper combines diffusion distillation and metric optimization (by GAN loss). Since traditional diffusion/flow matching can not generate intelligible speech at high noise levels, this paper bypasses it by diffusion distillation to skip this stage and apply the GAN optimization. Firstly, both diffusion distillation and direct metric optimization in TTS are well studied. Secondly, FlashTTS[2], which is also a consistency model (bypassing the same challenge of unintelligible speech under high noise levels) with direct metric optimization, is trained without distillation and is much easier. So, it lacks novelty for the scope of ICML. \n\n2. This paper should discuss more with some closely related papers, such as DIFFUSION-GAN[1], FlashTTS[2].\n\n[1] DIFFUSION-GAN: TRAINING GANS WITH DIFFUSION\n\n[2] FlashSpeech: Efficient Zero-Shot Speech Synthesis", "methods_and_evaluation_criteria": "Overall, the methods are written clearly. The evaluation criteria is sufficient.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Some questions:\n1. For experiments comparing with end-to-end systems, I recommend to compare with more baselines: F5TTS[1], MASKGCT[2].\n2. It should be compared with FlashTTS[3], which is also a strong baseline of an efficient zero-shot TTS system with few iterative steps and direct metric optimization.\n\n[1] F5-TTS: Diffusion Transformer with ConvNeXt V2, faster trained and inference.\n\n[2] Maskgct: Zero-shot text-to-speech with masked generative codec transformer\n\n[3] FlashSpeech: Efficient Zero-Shot Speech Synthesis", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The strengths and weaknesses are discussed in previous sections.", "other_comments_or_suggestions": "## Update after Rebuttal\n\nThanks for the authors' replies.\n\nRegarding the main concern about Direct Metric Optimization vs GAN, I agree that FlashSpeech does not involve direct metric optimization in adversarial training. I misunderstood the concept of adversarial training and direct metric optimization. My concerns still exist: 1. The method to distill the teacher model for fast inference via the adversarial training is not novel: works such as DMD 2 [1] and FlashSpeech have studied it. 2. The direct metric optimization is not novel either. The early work, such as StyleTTS 2 [2], also uses pretrained WavLM (a proxy for speaker similarity metric) as the direct optimization objective and achieved good results (see Table 5 w/o SLM adversarial training in the ablation study of StyleTTS 2).\n\nThe comparisons of F5TTS and MackGCT are good.\n\nFinally, I agree with the author's claim: enabling direct optimization of perceptually relevant metrics through a differentiable pathway created by one-step generation through diffusion distillation, especially in the area of TTS. I will update the score to 3 by combining all the proposed methods and considering the good results.\n\n[1] Improved Distribution Matching Distillation for Fast Image Synthesis [2] StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models", "questions_for_authors": "The questions are asked in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ow9nEdvct", "reviewer": "Reviewer_rYkV", "summary": "This paper proposes a DMOSpeech speech synthesis method. It utilizes Connectionist Temporal Classification (CTC) loss and Speaker Verification (SV) loss to realize the direct optimization of diffusion-based models. It was evaluated using subjective and objective tests. It outperforms the previous methods in most metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper. We appreciate your feedback and address your concerns below:\n\n**Correlation Between Subjective and Objective Metrics**\n\nWe have indeed analyzed the correlation between subjective and objective metrics in our paper. As shown in Figure 3 and Figure 5 (appendix), the Pearson correlation between speaker embedding similarity (SIM) and human-rated voice similarity is 0.55, while the correlation with style similarity is 0.50. Similarly, word error rate (WER) correlates with naturalness and sound quality at -0.16 for both metrics. All correlations are statistically significant ($p << 0.01$), demonstrating that our optimized objective metrics strongly align with human perception even at the individual utterance level.\n\n**Applicability Beyond Speech Synthesis**\n\nWhile our implementation focuses on speech synthesis, the core framework of enabling direct metric optimization through distillation can be applied to other generative domains. For example:\n\n- In music generation, differentiable models like instrument detection models or melody extraction models could optimize text-to-music alignment or ensure generated music matches specified instruments or melodic (MIDI input) constraints.\n\n- In image generation, differentiable models could maxizmize CLIP scores between the prompt text and generated image, or verify the presence of all text-described objects using image segmentation models\n\n- In video generation, similar principles could ensure temporal consistency\n\nThe key innovation is creating a direct gradient pathway that enables end-to-end optimization with any differentiable metric within the diffusion model frameworks, which has broad applications beyond speech synthesis.\n\n**Comparison with StyleTTS-ZS**\n\nWe have conducted objective evaluations of StyleTTS-ZS that will be included in the revised manuscript. DMOSpeech significantly outperforms StyleTTS-ZS in speaker similarity (0.69 vs. 0.56) with comparable real-time factor (0.07 vs. 0.04). While StyleTTS-ZS achieves lower WER (1.17 vs. our 1.94), our model delivers better overall performance as confirmed by subjective evaluations in Table 1. Additionally, our training pipeline is more straightforward without requiring aligner training, making it easier to scale across languages and larger datasets.\n\n**Parameter Selection Process**\n\nWe have detailed our parameter selection approach in the paper. The process is intuitive rather than difficult—we observe gradient norms for each loss term and balance them accordingly. As described in Section 3.4:\n\n>\" We set $\\lambda_{\\text{adv}} = 10^{-3}$ to ensure the gradient norm of adversarial loss is comparable to that of DMD loss. During early training stage, we observed that the gradient norms of SV loss and CTC loss were significantly higher than DMD loss, likely because $G_\\theta$ was still learning to generate intelligible speech from single step. To address this, we set $\\lambda_{\\text{CTC}} = 0$ for the first 5,000 iterations and $\\lambda_{\\text{SV}} = 0$ for the first 10,000 iterations. This allows $G_\\theta$ to stabilize under the influence of DMD loss before integrating these additional losses. After that, both $\\lambda_{\\text{CTC}}$ and $\\lambda_{\\text{SV}}$ are set to 1.\"\n\nThis approach follows established practices in the literature (Yin et. al, 2024) and doesn't require extensive hyperparameter tuning.\n\n**Regarding Other Suggestions**\n\n1. >The description in Section 3.2 should be improved. Most of the contents are not new, so it is unclear which parts are novel.\n\nSection 3.2 provides necessary background on Distribution Matching Distillation, establishing context for our improvements. We acknowledge this is primarily background material and will clarify which aspects represent our specific contributions in the revised manuscript.\n\n2. >The MOS values of various methods in Table 1 are almost the same as the ground truth. I am not sure whether the improvement obtained by the proposed method is meaningful enough.\n\nWhile MOS values are similar to ground truth, our key contribution is achieving comprehensive improvements across multiple metrics simultaneously. Previous models like StyleTTS-ZS achieve high naturalness but lower similarity, while NaturalSpeech 3 achieves high similarity but lower naturalness. DMOSpeech uniquely excels in both dimensions while maintaining significantly faster inference speed (13.7x faster than the teacher model). This balanced performance across all metrics represents a meaningful advancement in the field.\nWe appreciate your constructive feedback and will incorporate these clarifications in our revised manuscript.", "claims_and_evidence": "The authors claim that the proposed method provides a direct pathway to realize the end-to-end optimization of diffusion-based synthesis model. Its performance has become better than without it. The subjective and objective evaluation supports this claim.", "methods_and_evaluation_criteria": "The evaluation methods are sound. The authors conducted many subjective tests using human annotators, and the results were informative.", "theoretical_claims": "The proposed method's theoretical claim is that it has direct gradient pathways to all the model components. I believe this is sound enough.", "experimental_designs_or_analyses": "The authors perform subjective and objective tests. They are fine, but it is not clear how the results of these two are correlated. Their consistent results, and costly subjective tests are meaningful only for ablation studies.", "relation_to_broader_scientific_literature": "The novel point is the use of CTC loss and SV loss to realize direct optimization of diffusion based speech synthesis model. This point is somewhat limited to the domain of speech synthesis.", "essential_references_not_discussed": "Most of the references are OK. However, Table 3 does not report the result of the objective test of the previous study StyleTTS-ZS. Therefore, it is not clear how much improvement the proposed method achieves.\n\n## update after rebuttal ##\nThe comparison result with StyleTTS-ZS will be added. Then, there will be no problem at this point.", "other_strengths_and_weaknesses": "It seems the control parameters of the model (such as \\lambda) and the learning parameters are difficult to choose. The ablation studies in this regard should be needed.\n\n## update after rebuttal ##\nThank you for the rebuttal comments. Now I believe there is no problem.", "other_comments_or_suggestions": "1) The description in Section 3.2 should be improved. Most of the contents are not new, so it is unclear which parts are novel. \n2) The MOS values of various methods in Table 1 are almost the same as the ground truth. I am not sure whether the improvement obtained by the proposed method is meaningful enough.\n\n## update after rebuttal ##\n\nAbout 2), I agree with the authors' comments in the rebuttal. They should insist that improving MOS is not the only contribution in the manuscript.\n\nOverall, I raised my score from WR to WA.", "questions_for_authors": "What are the differences of the proposed method and the previous method StyleTTS-ZS in the objective test in Table 3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: DMOSpeech combines DMD2 and direct metric optimization (DMO) to give a distilled model matching or exceeding recent diffusion and AR baselines. The system is well-motivated (e.g. DMD vs. progressive distillation), though the choice and weighting (over time) of direct metrics is empirical, requiring tuning (R4; authors could cite clarify where e.g. in Yin et al. their heuristics come from) and extra memory (R2). 

DMD (R1, R3) is not theoretically novel. I do think DMO during distillation is novel (R3 points to StyleTTS 2, but it’s not distillation, it’s adversarial, and the ablation seems to drop it entirely). “Mode shrinkage” is also not novel: even the original distillation paper (https://arxiv.org/abs/1503.02531) talks about intentionally sharpening the student via T < 1, and this effect is viewed as why self-distillation helps (https://arxiv.org/abs/1805.04770). Perhaps it’s not studied for DMD, but authors should not oversell this in a camera-ready. Would be interesting to see how well other diffusion distillations sharpen their distributions.

Regardless, their analysis of distributions and effects of each DMO loss in NAR TTS is informative. All reviewers agree the experiments (subjective and objective) are comprehensive and ultimately support acceptance.
Decision: Accept (poster)