SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In equation 5, the generator loss is independent of ground truth  $\mathbf{x}$  and relies only on input text  $\mathbf{t}$ . This enables training on out-of-distribution (OOD) texts, which we show in section 5.3 can improve the performance on OOD texts. In practice, to prevent  $D_{SLM}$  from over-fitting on the content of the speech, we sample in-distribution and OOD texts with equal probability during training.
3.2.4 Differentiable Duration Modeling
The duration predictor produces phoneme durations  $d_{\mathrm{pred}}$ , but the upsampling method described in section 3.1 to obtain  $a_{\mathrm{pred}}$  is not differentiable, blocking gradient flow for E2E training. NaturalSpeech [5] employs an attention-based upsampler [42] for human-level TTS. However, we find this approach unstable during adversarial training because we train our model using differentiable upsampling with only the adversarial objective described in eq. 5 and without extra loss terms due to the length mismatch caused by deviations of  $d_{\mathrm{pred}}$  from  $d_{\mathrm{gt}}$ . Although this mismatch can be mitigated with soft dynamic time warping as used in [42, 5], we find this approach both computationally expensive and unstable with mel-reconstruction and adversarial objectives. To achieve human-level performance with adversarial training, a non-parametric upsampling method is preferred for stable training.
Gaussian upsampling [41] is non-parametric and converts the predicted duration  $\pmb{d}_{\mathrm{pred}}$  into  $\pmb{a}_{\mathrm{pred}}[n,i]$  using a Gaussian kernel  $\mathcal{N}_{c_i}(n;\sigma)$  centered at  $c_{i} := \ell_{i} - \frac{1}{2}\pmb{d}_{\mathrm{pred}}[i]$  with the hyperparameter  $\sigma$ :
$$
\mathcal {N} _ {c _ {i}} (n; \sigma) := \exp \left(- \frac {\left(n - c _ {i}\right) ^ {2}}{2 \sigma^ {2}}\right), \tag {8}
$$
where  $\ell_{i}$  is the end position and  $c_{i}$  is the center position of the  $i^{\mathrm{th}}$  phoneme  $t_i$ . However, Gaussian upsampling has limitations due to its fixed width of Gaussian kernels determined by  $\sigma$ . This constraint prevents it from accurately modeling alignments with varying lengths depending on  $d_{\mathrm{pred}}$ . Non-attentive Tacotron [57] extends this by making  $\sigma_{i}$  trainable, but the trained parameters introduce instability for E2E training with adversarial loss, similar to issues of attention-based upsamplers.
We propose a new non-parametric differentiable upsampler without additional training while taking into account the varying length of the alignment. For each phoneme  $\pmb{t}_i$ , we model the alignment as a random variable  $a_i \in \mathbb{N}$ , indicating the index of the speech frame the phoneme  $\pmb{t}_i$  is aligned with. We define the duration of the  $i^{\text{th}}$  phoneme as another random variable  $d_i \in \{1, \dots, L\}$ , where  $L = 50$  is the maximum phoneme duration hyperparameter, equivalent to 1.25 seconds in our setting. We observe that  $a_i = \sum_{k=1}^{i} d_k$ , but each  $d_k$  is dependent on each other, making the sum difficult to model. Instead, we approximate  $a_i = d_i + \ell_{i-1}$ . The approximated probability mass function (PMF) of  $a_i$  is
$$
f _ {a _ {i}} [ n ] = f _ {d _ {i} + \ell_ {i - 1}} [ n ] = f _ {d _ {i}} [ n ] * f _ {\ell_ {i - 1}} [ n ] = \sum_ {k} f _ {d _ {i}} [ k ] \cdot \delta_ {\ell_ {i - 1}} [ n - k ], \tag {9}
$$
where  $\delta_{\ell_{i-1}}$  is the PMF of  $\ell_{i-1}$ , a constant defined in eq. 8. This delta function is not differentiable, so we replace it with  $\mathcal{N}_{\ell_{i-1}}$  defined in eq. 7 with  $\sigma = 1.5$  (see Appendix C.2 for more discussions).
To model  $f_{d_i}$ , we modify the duration predictor to output  $q[k, i]$ , the probability of the  $i^{\text{th}}$  phoneme having a duration of at least  $k$  for  $k \in \{1, \dots, L\}$ , optimized to be 1 when  $d_{\text{gt}} \geq k$  with the cross-entropy loss (see Appendix G). Under this new scheme, we can approximate  $d_{\text{pred}}[i] := \sum_{k=1}^{L} q[k, i]$ , trained to match  $d_{\text{gt}}$  with  $\mathcal{L}_{\text{dur}}$  as in section 3.1. The vector  $q[:, i]$  can be viewed as the unnormalized version of  $f_{d_i}$ , though it is trained to be uniformly distributed across the interval  $[1, d_i]$ . Since the number of speech frames  $M$  is usually larger than the number of input phonemes  $N$ , this uniform distribution
aligns single phonemes to multiple speech frames as desired. Finally, we normalize the differentiable approximation  $\tilde{f}_{a_i}[n]$  across the phoneme axis, as in [41], to obtain  $a_{\mathrm{pred}}$  using the softmax function:
$$
\boldsymbol {a} _ {\text {p r e d}} [ n, i ] := \frac {e ^ {\left(\tilde {f} _ {a _ {i}} [ n ]\right)})}{\sum_ {i = 1} ^ {N} e ^ {\left(\tilde {f} _ {a _ {i}} [ n ]\right)}}, \tag {10}
$$
where  $\hat{M} \coloneqq \lceil \ell_N \rceil$  is the predicted total duration of the speech and  $n \in \{1, \dots, \hat{M}\}$ . An illustration of our proposed differentiable duration modeling is given in Figure 4 (Appendix C).
4.1 Model Training
We performed experiments on three datasets: LJSpeech, VCTK, and LibriTTS. Our single-speaker model was trained on the LJSpeech dataset, consisting of 13,100 short audio clips totaling roughly 24 hours. This dataset was divided into training (12,500 samples), validation (100 samples), and testing (500 samples) sets, with the same split as [3, 5, 6]. The multispeaker model was trained on VCTK, comprising nearly 44,000 short clips from 109 native speakers with various accents. The data split was the same as [3], with 43,470 samples for training, 100 for validation, and 500 for testing. Lastly, we trained our model on the combined LibriTTS train-clean-460 subset [14] for zero-shot adaptation. This dataset contains about 245 hours of audio from 1,151 speakers. Utterances longer than 30 seconds or shorter than one second were excluded. We distributed this dataset into training (98%), validation (1%), and testing (1%) sets, in line with [6]. The test-clean subset was used for zero-shot adaptation evaluation with 3-second reference clips. All datasets were resampled to  $24\mathrm{kHz}$  to match LibriTTS, and the texts were converted into phonemes using phonemizer [58].
We used texts in the training split of LibriTTS as the out-of-distribution (OOD) texts for SLM adversarial training. We used iSTFTNet decoder for LJSpeech due to its speed and sufficient performance on this dataset, while the HifiGAN decoder was used for the VCTK and LibriTTS models. Acoustic modules were pre-trained for 100, 50, and 30 epochs on the LJSpeech, VCTK, and LibriTTS datasets, and joint training followed for 60, 40, and 25 epochs, respectively. We employed the AdamW optimizer [59] with  $\beta_{1} = 0$ ,  $\beta_{2} = 0.99$ , weight decay  $\lambda = 10^{-4}$ , learning rate  $\gamma = 10^{-4}$  and a batch size of 16 samples for both pre-training and joint training. The loss weights were adopted from [6] to balance all loss terms (see Appendix G for details). Waveforms were randomly segmented with a max length of 3 seconds. For SLM adversarial training, both the ground truth and generated samples were ensured to be 3 to 6 seconds in duration, the same as in fine-tuning of WavLM models for various downstream tasks [11]. Style diffusion steps were randomly sampled from 3 to 5 during training for speed and set to 5 during inference for quality. The training was conducted on four NVIDIA A40 GPUs.
We employed two metrics in our experiments: Mean Opinion Score of Naturalness (MOS-N) for human-likeness, and Mean Opinion Score of Similarity (MOS-S) for similarity to the reference for multi-speaker models. These evaluations were conducted by native English speakers from the U.S. on Amazon Mechanical Turk. All evaluators reported normal hearing and provided informed consent as monitored by the local institutional review board and in accordance with the ethical standards of the Declaration of Helsinki  ${}^{2}$  . In each test,80 random text samples from the test set were selected and converted into speech using our model and the baseline models, along with ground truth for comparison. Because [7] reported that many TTS models perform poorly for OOD texts, our LJSpeech experiments also included 40 utterances from Librivox spoken by the narrator of LJSpeech but from audiobooks not in the original dataset as the ground truth for OOD texts. To compare the difference between in-distribution and OOD performance, we asked the same raters to evaluate samples on both in-distribution and OOD texts.
Our baseline models consisted of the three highest-performing public models: VITS [3], StyleTTS [6], and JETS [32] for LJSpeech; and VITS, YourTTS [60], and StyleTTS for LibriTTS. Each synthesized
2We obtained approval for our protocol (number IRB-AAAR8655) from the Institutional Review Board.
Table 1: Comparative mean opinion scores of naturalness and similarity for StyleTTS 2 with p-values from Wilcoxon test relative to other models. Positive scores indicate StyleTTS 2 is better.
Model	Dataset	CMOS-N (p-value)	CMOS-S (p-value)
Ground Truth	LJSpeech	+0.28 (p=0.021)	—
NaturalSpeech	LJSpeech	+1.07 (p<10-6)	—
Ground Truth	VCTK	-0.02 (p=0.628)	+0.30 (p=0.081)
VITS	VCTK	+0.45 (p=0.009)	+0.43 (p=0.032)
Vall-E	LibriSpeech (zero-shot)	+0.67 (p<10-3)	-0.47 (p<10-3)
speech set was rated by 5 to 10 evaluators on a 1-5 scale, with increments of 0.5. We randomized the model order and kept their labels hidden, similar to the MUSHRA approach [61, 62]. We also conducted Comparative MOS (CMOS) tests to determine statistical significance, as raters can ignore subtle differences in MOS experiments [3, 7, 5]. Raters were asked to listen to two samples and rate whether the second was better or worse than the first on a -6 to 6 scale with increments of 1. We compared our model to the ground truth and NaturalSpeech [5] for LJSpeech, and the ground truth and VITS for VCTK. For the zero-shot experiment, we compared our LibriTTS model to Vall-E [8].
All baseline models, except for the publicly available VITS model on LibriTTS from ESPNet Toolkit [63], were official checkpoints released by the authors, including vocoders used in StyleTTS. As NaturalSpeech and Vall-E are not publicly available, we obtained samples from the authors and the official Vall-E demo page, respectively. For fairness, we resampled all audio to  $22.5\mathrm{kHz}$  for LJSpeech and VCTK, and  $16\mathrm{kHz}$  for LibriTTS, to match the baseline models. We conducted ablation studies using CMOS-N on LJSpeech on OOD texts from LibriTTS test-clean subset for more pronounced results as in [7]. For more details of our evaluation procedures, please see Appendix E.
5.1 Model Performance
The results outlined in Table 1 establish that StyleTTS 2 outperforms NaturalSpeech by achieving a CMOS of  $+1.07$  ( $p \ll 0.01$ ), setting a new standard for this dataset. Interestingly, StyleTTS 2 was favored over the ground truth with a CMOS of  $+0.28$  ( $p < 0.05$ ). This preference may stem from dataset artifacts such as fragmented audiobook passages in the LJSpeech dataset that disrupt narrative continuity, thus rendering the ground truth narration seemingly unnatural. This hypothesis is corroborated by the performance of StyleTTS 2 on the VCTK dataset, which lacks narrative context, where it performs comparably to the ground truth (CMOS = -0.02,  $p \gg 0.05$ ). Samples from our model were more similar to the reference audio speaker than the human recording, suggesting our model's effective use of the reference audio for style diffusion. Moreover, StyleTTS 2 scored higher than the previous state-of-the-art model VITS on VCTK, as evidenced by CMOS-N and CMOS-S.
Consistent with the CMOS results, our model achieved a MOS of 3.83, surpassing all previous models on LJSpeech (Table 2). In addition, all models, except ours, exhibited some degrees of quality degradation for out-of-distribution (OOD) texts. This corroborates the gap reported in [7], with our results providing additional ground truth references. On the other hand, our model did not show any degradation and significantly outperformed other models in MOS for OOD texts (Table 2), demonstrating its strong generalization ability and robustness towards OOD texts.
In zero-shot tests, StyleTTS 2 surpasses Vall-E in naturalness with a CMOS of  $+0.67(p\ll 0.01)$  although it falls slightly short in similarity (Table 1). Importantly, StyleTTS 2 achieved these results with only 245 hours of training data, compared to Vall-E's 60k hours, a  $250x$  difference. This makes StyleTTS 2 a data-efficient alternative to large pretraining methods like Vall-E. The MOS results in Table 3 support these findings,
Table 2: Comparison of MOS with  $95\%$  confidence intervals (CI) on LJSpeech.  $\mathrm{MOS_{ID}}$  represents MOS-N for in-distribution texts, while  $\mathrm{MOS_{OOD}}$  is that for OOD texts.
Model	MOSID (CI)	MOSOOD (CI)
Ground Truth	3.81 (± 0.09)	3.70 (± 0.11)
StyleTTS 2	3.83 (± 0.08)	3.87 (± 0.08)
JETS	3.57 (± 0.09)	3.21 (± 0.12)
VITS	3.34 (± 0.10)	3.21 (± 0.11)
StyleTTS + HifiGAN	3.35 (± 0.10)	3.32 (± 0.12)
as our model exceeds all baseline models in both MOS-N and MOS-S. However, the difference in
(a) LJSpeech model.
(b) Unseen speakers on LibriTTS. (c) Zoomed-in unseen speaker.
Figure 2: t-SNE visualization of style vectors sampled via style diffusion from texts in five emotions, showing that emotions are properly separated for seen and unseen speakers. (a) Clusters of emotion from styles sampled by the LJSpeech model. (b) Distinct clusters of styles sampled from 5 unseen speakers by the LibriTTS model. (c) Loose clusters of emotions from Speaker 1 in (b).
MOS-S between StyleTTS and StyleTTS 2 was not statistically significant, hinting at future directions for improvement in speaker similarity.
5.2 Style Diffusion
Figure 2 shows t-SNE visualizations of style vectors created using our style diffusion process. Due to the lack of emotion-labeled audiobook text datasets, we used GPT-4 to generate 500 utterances across five emotions for this task [64]. In Figure 2a, style vectors from the LJSpeech model illustrate distinct emo
Table 3: Comparison of MOS with  $95\%$  confidence intervals (CI) on test-clean subset of LibriTTS for zero-shot speaker adaptation.
Model	MOS-N (CI)	MOS-S (CI)
Ground Truth	4.60 (± 0.09)	4.35 (± 0.10)
StyleTTS 2	4.15 (± 0.11)	4.03 (± 0.11)
YourTTS	2.35 (± 0.07)	2.42 (± 0.09)
VITS	3.69 (± 0.12)	3.54 (± 0.13)
StyleTTS + HiFi-GAN	3.91 (± 0.11)	4.01 (± 0.10)
tional styles in response to input text sentiment, demonstrating the model's capability to synthesize expressive speech in varied emotions without explicit emotion labels during training. This process was repeated with the LibriTTS model on five unseen speakers, each from a 3-second reference audio. As depicted in Figure 2b, distinct clusters form for each speaker, showcasing a wide stylistic diversity derived from a single reference audio. Figure 2c provides a more nuanced view of the first speaker, revealing visible emotion-based clusters despite some overlaps, indicating that we can manipulate the emotional tone of an unseen speaker regardless of the tones in the reference audio. These overlaps, however, can partly explain why the LibriTTS model does not perform as well as the LJSpeech model, as it is harder to disentangle texts from speakers in the zero-shot setting (see Appendix A.2 for more results). Table 4 displays our synthesized speech diversity against several baseline models with the coefficient of variation of duration  $(\mathrm{CV}_{\mathrm{dur}})$  and pitch curve  $(\mathrm{CV}_{\mathrm{f0}})$  from the same input text [3]. Our model yields the highest variation, indicating superior potential for generating diverse speech. Despite being diffusion-based, our model is faster than VITS, FastDiff [28], and ProDiff [18], two of the fastest diffusion-based TTS models, with 5 iterations of diffusion and iSTFTNet decoder.
Table 4: Speech diversity metrics and real-time factor (RTF). ProDiff and FastDiff were tested with 4 steps of diffusion.
Model	CVdur↑	CVf0↑	RTF↓
StyleTTS 2	0.0321	0.6962	0.0185
VITS	0.0214	0.5976	0.0599
FastDiff	0.0295	0.6490	0.0769
ProDiff	2e-16	0.5898	0.1454
Table 5: CMOS-N relative to the StyleTTS 2 baseline on OOD texts in the ablation study.
Model	CMOS-N
w/o style diffusion	-0.46
w/o differentiable upsampler	-0.21
w/o SLM adversarial training	-0.32
w/o prosodic style encoder	-0.35
w/o OOD texts	-0.15
Table 5 details the ablation study, underlying the importance of our proposed components. When style vectors from style diffusion are substituted with randomly encoded ones as in [6], the CMOS is  $-0.46$ , highlighting the contribution of text-dependent style diffusion to achieving human-level TTS. Training without our differentiable upsampler and without the SLM discriminator results in a CMOS of  $-0.21$  and  $-0.32$ , validating their key roles in natural speech synthesis. Removing the prosodic style encoder also yields a  $-0.35$  CMOS. Last, excluding OOD texts from adversarial training leads to a CMOS of  $-0.15$ , proving its efficacy for improving OOD speech synthesis. Table 6 in Appendix A.3 shows similar outcomes with objective evaluations, further affirming the effectiveness of various components we proposed in this work. Figure 7 in Appendix D details a layer-wise analysis of input weights of the SLM discriminator, providing a different view of the efficacy of SLM discriminators.
In this study, we present StyleTTS 2, a novel text-to-speech (TTS) model with human-level performance via style diffusion and speech language model discriminators. In particular, it exceeds the ground truth on LJSpeech and performs on par with it on the VCTK dataset. StyleTTS 2 also shows potential for zero-shot speaker adaption, with remarkable performance even on limited training data compared to large-scale models like Vall-E. With our innovative style diffusion method, StyleTTS 2 generates expressive and diverse speech of superior quality while ensuring fast inference time. While StyleTTS 2 excels in several areas, our results indicate room for improvement in handling large-scale datasets such as LibriTTS, which contain thousands of or more speakers, acoustic environments, accents, and other various aspects of speaking styles. The speaker similarity in the aforementioned zero-shot adaptation speaker task could also benefit from further improvements.
However, zero-shot speaker adaptation has the potential for misuse and deception by mimicking the voices of individuals as a potential source of misinformation or disinformation. This could lead to harmful, deceptive interactions such as theft, fraud, harassment, or impersonations of public figures that may influence political processes or trust in institutions. In order to manage the potential for harm, we will require users of our model to adhere to a code of conduct that will be clearly displayed as conditions for using the publicly available code and models. In particular, we will require users to inform those listening to samples synthesized by StyleTTS 2 that they are listening to synthesized speech or to obtain informed consent regarding the use of samples synthesized by StyleTTS 2 in experiments. Users will also be required to use reference speakers who have given consent to have their voice adapted, either directly or by license. Finally, we will make the source code publicly available for further research in speaker fraud and impersonation detection.
In addition, while human evaluators have favored StyleTTS 2 over ground truth with statistical significance on the LJSpeech dataset, this preference may be context-dependent. Original audio segments from larger contexts like audiobooks could inherently differ in naturalness when isolated, potentially skewing the evaluations in favor of synthesized speech. Additionally, the inherent variability in human speech, which is context-independent, might lead to lower ratings when compared to the more uniform output from StyleTTS 2. Future research should aim to improve evaluation methods to address these limitations and develop more natural and human-like speech synthesis models with longer context dependencies.
We thank Menoua Keshishian, Vishal Choudhari, and Xilin Jiang for helpful discussions and feedback on the paper. We also thank Grace Wilsey, Elden Griggs, Jacob Edwards, Rebecca Saez, Moises Rivera, Rawan Zayter, and D.M. for assessing the quality of synthesized samples and providing feedback on the quality of models during the development stage of this work. This work was funded by the national institute of health (NIHNIDCD) and a grant from Marie-Josee and Henry R. Kravis.
[1] Yishuang Ning, Sheng He, Zhiyong Wu, Chunxiao Xing, and Liang-Jie Zhang. A review of deep learning based speech synthesis. Applied Sciences, 9(19):4050, 2019.
[53] Ashish Vavwani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[54] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. arXiv preprint arXiv:2206.00927, 2022.
[55] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. Layer-wise analysis of a self-supervised speech representation model. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 914-921. IEEE, 2021.
[56] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected GANs Converge Faster. Advances in Neural Information Processing Systems, 34:17480-17492, 2021.
[57] Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu. Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling. arXiv preprint arXiv:2010.04301, 2020.
[58] Mathieu Bernard and Hadrien Titeux. Phonemizer: Text to Phones Transcription for Multiple Languages in Python. Journal of Open Source Software, 6(68):3958, 2021.
[59] Ilya Loshchilov and Frank Hutter. Fixing Weight Decay Regularization in Adam, 2018.
[60] Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölgge, and Moacir A Ponti. YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone. In International Conference on Machine Learning, pages 2709-2720. PMLR, 2022.
[61] Yinghao Aaron Li, Ali Zare, and Nima Mesgarani. StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion. arXiv preprint arXiv:2107.10394, 2021.
[62] Yinghao Aaron Li, Cong Han, and Nima Mesgarani. Styletts-vc: One-shot voice conversion by knowledge transfer from style-based tts models. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 920-927. IEEE, 2023.
[63] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, John Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of Interspeech, pages 2207-2211, 2018.
[64] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models. arXiv preprint arXiv:2304.01852, 2023.
Appendix A Additional Evaluation Results
A.1 Feedback Analysis from Survey Participants
On the last page of our survey, we encouraged the raters to provide their feedback on the study. While the majority opted not to comment, those who did provided valuable insights into our survey design and results interpretation. The main point of feedback revolved around the difficulty in discerning differences between different samples, specifically for the LJSpeech CMOS experiments. Below are a few representative comments from our CMOS experiment participants: