SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yinghao Aaron Li Cong Han Vinay S. Raghavan
Gavin Mischler Nima Mesgarani
{yl4579,ch3212,vsr2119,gm2944,nm2764}@columbia.edu
In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github.io/.
Text-to-speech (TTS) synthesis has seen significant advancements in recent years, with numerous applications such as virtual assistants, audiobooks, and voice-over narration benefiting from increasingly natural and expressive synthetic speech [1, 2]. Some previous works have made significant progress towards human-level performance [3, 4, 5]. However, the quest for robust and accessible human-level TTS synthesis remains an ongoing challenge because there is still room for improvement in terms of diverse and expressive speech [5, 6], robustness for out-of-distribution (OOD) texts [7], and the requirements of massive datasets for high-performing zero-shot TTS systems [8].
In this paper, we introduce StyleTTS 2, an innovative TTS model that builds upon the style-based generative model StyleTTS [6] to present the next step towards human-level TTS systems. We model speech styles as a latent random variable and sample them with a probabilistic diffusion model, allowing the model to efficiently synthesize highly realistic speech without the need for reference audio. Since it only needs to sample a style vector instead of the entire speech as a latent variable, StyleTTS 2 is faster than other diffusion TTS models while still benefiting from the diverse speech synthesis enabled by diffusion models. One of the key contributions of StyleTTS 2 is the use of large pre-trained speech language models (SLMs) like Wav2Vec 2.0 [9], HuBERT [10], and WavLM [11] as discriminators, in conjunction with a novel differentiable duration modeling approach. This end-to-end (E2E) training setup leverages SLM representations to enhance the naturalness of the synthesized speech, transferring knowledge from large SLMs for speech generation tasks.
Our evaluations suggest that speech generated by StyleTTS 2 surpasses human recordings as judged by native English speakers on the benchmark LJSpeech [12] dataset with statistically significant comparative mean opinion scores (CMOS) of  $+0.28$  ( $p < 0.05$ ). Additionally, StyleTTS 2 advances the state-of-the-art by achieving a CMOS of  $+1.07$  ( $p \ll 0.01$ ) compared to NaturalSpeech [5]. Furthermore, it attains human-level performance on the multispeaker VCTK dataset [13] in terms of naturalness ( $\mathrm{CMOS} = -0.02$ ,  $p \gg 0.05$ ) and similarity ( $\mathrm{CMOS} = +0.30$ ,  $p < 0.1$ ) to the reference speaker. When trained on a large number of speakers like the LibriTTS dataset [14], StyleTTS 2 demonstrates potential for speaker adaptation. It surpasses previous publicly available models in this task and outperforms Vall-E [8] in naturalness. Moreover, it achieves slightly worse similarity to the target speaker with only a 3-second reference speech, despite using around 250 times less data compared to Vall-E, making it a data-efficient alternative for large pre-training in the zero-shot speaker adaptation task. As the first model to achieve human-level performance on publicly available single and multispeaker datasets, StyleTTS 2 sets a new benchmark for TTS synthesis, highlighting the potential of style diffusion and adversarial training with SLMs for human-level TTS synthesis.
Diffusion Models for Speech Synthesis. Diffusion models have gained traction in speech synthesis due to their potential for diverse speech sampling and fine-grained speech control [15]. They have been applied to mel-based text-to-speech [16, 17, 18, 19, 20], mel-to-waveform vocoder [21, 22, 23, 24, 25, 26], and end-to-end speech generation [27, 28, 29]. However, their efficiency is limited compared to non-iterative methods, like GAN-based models [30, 31, 32], due to the need to iteratively sample mel-spectrograms, waveforms, or other latent representations proportional to the target speech duration [15]. Furthermore, recent works suggest that state-of-the-art GAN-based models still perform better than diffusion models in speech synthesis [26, 33]. To address these limitations, we introduce style diffusion, where a fixed-length style vector is sampled by a diffusion model conditioned on the input text. This approach significantly improves model speed and enables end-to-end training. Notably, StyleTTS 2 synthesizes speech using GAN-based models, with only the style vector dictating the diversity of speech sampled. This unique combination allows StyleTTS 2 to achieve high-quality synthesis with fast inference speed while maintaining the benefits of diverse speech generation, further advancing the capabilities of diffusion models in speech synthesis.
Text-to-Speech with Large Speech Language Models. Recent advancements have proven the effectiveness of large-scale self-supervised speech language models (SLMs) in enhancing text-to-speech (TTS) quality [34, 35, 36, 37] and speaker adaptation [8, 38, 29, 39]. These works typically convert text input into either continuous or quantized representations derived from pre-trained SLMs for speech reconstruction. However, SLM features are not directly optimized for speech synthesis, while tuning SLMs as a neural codec [34, 35, 8, 29] involves two-stage training. In contrast, our model benefits from the knowledge of large SLMs via adversarial training using SLM features without latent space mapping, thus directly learning a latent space optimized for speech synthesis like other end-to-end (E2E) TTS models. This innovative approach signifies a new direction in TTS with SLMs.
Human-Level Text-to-Speech. Several recent works have advanced towards human-level TTS [3, 4, 5] using techniques like BERT pre-training [4, 40, 7] and E2E training [32, 5] with differentiable duration modeling [41, 42]. VITS [3] demonstrates MOS comparable to human recordings on the LJSpeech and VCTK datasets, while PnG-BERT [4] obtains human-level results on a proprietary dataset. NaturalSpeech [5], in particular, achieves both MOS and CMOS on LJSpeech statistically indistinguishable from human recordings. However, we find that there is still room for improvement in speech quality beyond these state-of-the-art models, as we attain higher performance and set a new standard for human-level TTS synthesis. Furthermore, recent work shows the necessity for disclosing the details of evaluation procedures for TTS research [43]. Our evaluation procedures are detailed in Appendix E, which can be used for reproducible future research toward human-level TTS.
3.1 StyleTTS Overview
StyleTTS [6] is a non-autoregressive TTS framework using a style encoder to derive a style vector from reference audio, enabling natural and expressive speech generation. The style vector is incorpo-
rated into the decoder and duration and prosody predictors using adaptive instance normalization (AdaIN) [44], allowing the model to generate speech with varying duration, prosody, and emotions.
The model comprises eight modules, organized into three categories: (1) a speech generation system (acoustic modules) with a text encoder, style encoder, and speech decoder; (2) a TTS prediction system with duration and prosody predictors; and (3) a utility system for training, including a discriminator, text aligner, and pitch extractor. It undergoes a two-stage training process: the first stage trains the acoustic modules for mel-spectrogram reconstruction, and the second trains TTS prediction modules using the fixed acoustic modules trained in the first stage.
In the first stage, the text encoder  $T$  encodes input phonemes  $t$  into phoneme representations  $\pmb{h}_{\mathrm{text}} = T(\pmb {t})$ , while the text aligner  $A$  extracts speech-phoneme alignment  $\pmb{a}_{\mathrm{algn}} = A(\pmb {x},t)$  from input speech  $\mathbf{x}$  and phonemes  $t$  to produce aligned phoneme representations  $\pmb{h}_{\mathrm{algn}} = \pmb{h}_{\mathrm{text}}\cdot \pmb{a}_{\mathrm{algn}}$  via dot product. Concurrently, the style encoder  $E$  obtains the style vector  $\boldsymbol {s} = E(\mathbf{x})$ , and the pitch extractor  $F$  extracts the pitch curve  $p_{x} = F(x)$  along with its energy  $n_x = \| x\|$ . Lastly, the speech decoder  $G$  reconstructs  $\hat{\pmb{x}} = G(h_{\mathrm{algn}},s,p_x,n_x)$ , which is trained to match input  $\pmb{x}$  using a  $L_{1}$  reconstruction loss  $\mathcal{L}_{\mathrm{mel}}$  and adversarial objectives  $\mathcal{L}_{\mathrm{adv}}$ ,  $\mathcal{L}_{\mathrm{fm}}$  with a discriminator  $D$ . Transferable monotonic aligner (TMA) objectives are also applied to learn optimal alignments (see Appendix G for details).
In the second stage, all components except the discriminator  $D$  are fixed, with only the duration and prosody predictors being trained. The duration predictor  $S$  predicts the phoneme duration with  $d_{\mathrm{pred}} = S(h_{\mathrm{text}}, \mathbf{s})$ , whereas the prosody predictor  $P$  predicts pitch and energy as  $\hat{p}_{x}, \hat{n}_{x} = P(h_{\mathrm{text}}, \mathbf{s})$ . The predicted duration is trained to match the ground truth duration  $\bar{d}_{\mathrm{gt}}$  derived from the summed monotonic version of the alignment  $a_{\mathrm{algn}}$  along the time axis with an  $L_{1}$  loss  $\mathcal{L}_{\mathrm{dur}}$ . The predicted pitch  $\hat{p}_{x}$  and energy  $\hat{n}_{x}$  are trained to match the ground truth pitch  $p_{x}$  and energy  $n_{x}$  derived from pitch extractor  $F$  with  $L_{1}$  loss  $\mathcal{L}_{f0}$  and  $\mathcal{L}_n$ . During inference,  $d_{\mathrm{pred}}$  is used to upsample  $h_{\mathrm{text}}$  through  $a_{\mathrm{pred}}$ , the predicted alignment, obtained by repeating the value 1 for  $d_{\mathrm{pred}[i]}$  times at  $\ell_{i-1}$ , where  $\ell_i$  is the end position of the  $i^{\mathrm{th}}$  phoneme  $t_i$  calculated by summing  $d_{\mathrm{pred}[k]}$  for  $k \in \{1, \ldots, i\}$ , and  $d_{\mathrm{pred}[i]}$  are the predicted duration of  $t_i$ . The mel-spectrogram is synthesized by  $x_{\mathrm{pred}} = G(h_{\mathrm{text}} \cdot a_{\mathrm{pred}}, E(\tilde{x}), \hat{p}_{\tilde{x}}, \hat{n}_{\tilde{x}})$  with  $\tilde{x}$  an arbitrary reference audio that influences the style of  $x_{\mathrm{pred}}$ , which is then converted into a waveform using a pre-trained vocoder.
Despite its state-of-the-art performance in synthesizing diverse and controllable speech, StyleTTS has several drawbacks, such as a two-stage training process with an additional vocoding stage that degrades sample quality, limited expressiveness due to deterministic generation, and reliance on reference speech hindering real-time applications.
3.2 StyleTTS 2
StyleTTS 2 improves upon the StyleTTS framework, resulting in a more expressive text-to-speech (TTS) synthesis model with human-level quality and improved out-of-distribution performance. We introduce an end-to-end (E2E) training process that jointly optimizes all components, along with direct waveform synthesis and adversarial training with large speech language models (SLMs) enabled by our innovative differentiable duration modeling. The speech style is modeled as a latent variable sampled through diffusion models, allowing diverse speech generation without reference audio. We outline these important changes in the following sections with an overview in Figure 1.
3.2.1 End-to-End Training
E2E training optimizes all TTS system components for inference without relying on any fixed components like pre-trained vocoders that convert mel-spectrograms into waveforms [3, 32]. To achieve this, we modify the decoder  $G$  to directly generate the waveform from the style vector, aligned phoneme representations, and pitch and energy curves. We remove the last projection layer for mel-spectrograms of the decoder and append a waveform decoder after it. We propose two types of decoders: HifiGAN-based and iSTFTNet-based. The first is based on Hifi-GAN [30], which directly generates the waveform. In contrast, the iSTFTNet-based decoder [45] produces magnitude and phase, which are converted into waveforms using inverse short-time Fourier transform for faster training and inference. We employ the snake activation function [46], proven effective for waveform generation in [31]. An AdaIN module [44] is added after each activation function to model the style dependence of the speech, similar to the original StyleTTS decoder. We replace the mel-discriminator in [6] with the multi-period discriminator (MPD) [30] and multi-resolution discriminator (MRD)
(a) Acoustic modules pre-training and joint training. To accelerate training, the pre-training first optimize modules inside the blue box; the joint training then follows to optimize all components except the pitch extractor, which is used to provide the ground truth label for pitch curves. The duration predictor is trained with only  $\mathcal{L}_{\mathrm{dur}}$ .
(b) SLM adversarial training and inference. WavLM is pre-trained and not tuned. Unlike (a), the duration predictor is trained E2E with all components using  $\mathcal{L}_{slm}$  (eq. 5) via differentiable upsampling. This process is separate from (a) during training as the input texts can be different, but the gradients are accumulated for both processes in each batch to update the parameters.
Figure 1: Training and inference scheme of StyleTTS 2 for the single-speaker case. For the multi-speaker case, the acoustic and prosodic style encoders (denoted as  $\pmb{E}$ ) first take reference audio  $\pmb{x}_{\mathrm{ref}}$  of the target speaker and produce a reference style vector  $\pmb{c} = \pmb{E}(\pmb{x}_{\mathrm{ref}})$ . The style diffusion model then takes  $\pmb{c}$  as a reference to sample  $\pmb{s}_p$  and  $\pmb{s}_a$  that correspond to the speaker in  $\pmb{x}_{\mathrm{ref}}$ .
[47] along with the LSGAN loss functions [48] for decoder training, and incorporate the truncated pointwise relativistic loss function [49] to enhance sound quality (see Appendix F and G for details).
We found that well-trained acoustic modules, especially the style encoder, can accelerate the training process for TTS prediction modules. Therefore, before jointly optimizing all components, we first pre-train the acoustic modules along with the pitch extractor and text aligner via  $\mathcal{L}_{\mathrm{mel}}$ ,  $\mathcal{L}_{\mathrm{adv}}$ ,  $\mathcal{L}_{\mathrm{fm}}$  and TMA objectives for  $N$  epochs where  $N$  depends on the size of the training set, in the same way as the first training stage of [6]. However, we note that this pre-training is not an absolute necessity: despite being slower, starting joint training directly from scratch also leads to model convergence,
After acoustic module pre-training, we jointly optimize  $\mathcal{L}_{\mathrm{mel}}$ ,  $\mathcal{L}_{\mathrm{adv}}$ ,  $\mathcal{L}_{\mathrm{fm}}$ ,  $\mathcal{L}_{\mathrm{dur}}$ ,  $\mathcal{L}_{\mathrm{f0}}$  and  $\mathcal{L}_{\mathrm{n}}$ , where  $\mathcal{L}_{\mathrm{mel}}$  is modified to match the mel-spectrograms of waveforms reconstructed from predicted pitch  $\hat{p}_x$  and energy  $\hat{n}_x$  (Fig 1a). During joint training, stability issues emerge from diverging gradients, as the style encoder must encode both acoustic and prosodic information. To address this inconsistency, we introduce a prosodic style encoder  $E_p$  alongside the original acoustic style encoder  $E_a$ , previously denoted as  $E$  in section 3.1. Instead of using  $s_a = E_a(x)$ , predictors  $S$  and  $P$  take  $s_p = E_p(x)$  as the input style vector. The style diffusion model generates the augmented style vector  $s = [s_p, s_a]$ . This modification effectively improves sample quality (see section 5.3). To further decouple the acoustic modules and predictors, we replace the phoneme representations  $h_{\mathrm{text}}$  from  $T$ , now referred to as acoustic text encoder, with  $h_{\mathrm{bert}}$  from another text encoder  $B$  based on BERT transformers, denoted as prosodic text encoder. Specifically, we employ a phoneme-level BERT [7] pre-trained on extensive corpora of Wikipedia articles as the prosodic text encoder. This approach has been shown to enhance the naturalness of StyleTTS in the second stage [7], similar to our proposed usage here.
With differentiable upsampling and fast style diffusion, we can generate speech samples during training in a fully differentiable manner, just as during inference. These samples are used to optimize  $\mathcal{L}_{slm}$  (eq. 5) during joint training to update the parameters of all components for inference (Fig 1b).
3.2.2 Style Diffusion
In StyleTTS 2, we model the speech  $\pmb{x}$  as a conditional distribution  $p(\pmb{x}|\pmb{t}) = \int p(\pmb{x}|\pmb{t},\pmb{s})p(\pmb{s}|\pmb{t})ds$  through a latent variable  $s$  that follows the distribution  $p(\pmb{s}|\pmb{t})$ . We call this variable the generalized speech style, representing any characteristic in speech beyond phonetic content  $t$ , including but not limited to prosody, lexical stress, formant transitions, and speaking rate [6]. We sample  $s$  by EDM [50] that follows the combined probability flow [51] and time-varying Langevin dynamics [52]:
$$
\boldsymbol {s} = \int - \sigma (\tau) [ \beta (\tau) \sigma (\tau) + \dot {\sigma} (\tau) ] \nabla_ {\boldsymbol {s}} \log p _ {\tau} (\boldsymbol {s} | \boldsymbol {t}) d \tau + \int \sqrt {2 \beta (\tau)} \sigma (\tau) d \tilde {W} _ {\tau}, \tag {1}
$$
where  $\sigma (\tau)$  is the noise level schedule and  $\dot{\sigma} (\tau)$  is its time derivative,  $\beta (\tau)$  is the stochasticity term,  $\tilde{W}_{\tau}$  is the backward Wiener process for  $\tau \in [T,0]$  and  $\nabla_{s}\log p_{\tau}(\boldsymbol {s}|\boldsymbol {t})$  is the score function at time  $\tau$ . We follow the EDM [50] formulation with the denoiser  $K(s;t,\sigma)$  preconditioned as:
$$
K (\boldsymbol {s}; \boldsymbol {t}, \sigma) := \left(\frac {\sigma_ {\text {d a t a}}}{\sigma^ {*}}\right) ^ {2} \boldsymbol {s} + \frac {\sigma \cdot \sigma_ {\text {d a t a}}}{\sigma^ {*}} \cdot V \left(\frac {\boldsymbol {s}}{\sigma^ {*}}; \boldsymbol {t}, \frac {1}{4} \ln \sigma\right), \tag {2}
$$
where  $\sigma$  follows a log-normal distribution  $\ln \sigma \sim \mathcal{N}(P_{\mathrm{mean}}, P_{\mathrm{std}}^2)$  with  $P_{\mathrm{mean}} = -1.2$ ,  $P_{\mathrm{std}} = 1.2$ ,  $\sigma^* := \sqrt{\sigma^2 + \sigma_{\mathrm{data}}^2}$  is the scaling term,  $\sigma_{\mathrm{data}} = 0.2$  is the empirical estimate of the standard deviation of style vectors, and  $V$  is a 3-layer transformer [53] conditioned on  $\pmb{t}$  and  $\sigma$  which is trained with:
$$
\mathcal {L} _ {\mathrm {e d m}} = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {t}, \sigma , \boldsymbol {\xi} \sim \mathcal {N} (0, I)} \left[ \lambda (\sigma) \| K (\boldsymbol {E} (\boldsymbol {x}) + \sigma \boldsymbol {\xi}; \boldsymbol {t}, \sigma) - \boldsymbol {E} (\boldsymbol {x}) \| _ {2} ^ {2} \right], \tag {3}
$$
where  $\mathbf{E}(\mathbf{x}) \coloneqq [E_a(\mathbf{x}), E_p(\mathbf{x})]$ , and  $\lambda(\sigma) \coloneqq (\sigma^* / (\sigma \cdot \sigma_{\mathrm{data}}))^2$  is the weighting factor. Under this framework, equation 16 becomes an ODE where the score function depends on  $\sigma$  instead of  $\tau$ :
$$
\frac {d \boldsymbol {s}}{d \sigma} = - \sigma \nabla_ {\boldsymbol {s}} \log p _ {\sigma} (\boldsymbol {s} | \boldsymbol {t}) = \frac {\boldsymbol {s} - K (\boldsymbol {s} ; \boldsymbol {t} , \sigma)}{\sigma}, \quad \boldsymbol {s} (\sigma (T)) \sim \mathcal {N} (0, \sigma (T) ^ {2} I). \tag {4}
$$
Unlike [50] that uses 2nd-order Heun, we solve eq. 4 with the ancestral DPM-2 solver [54] for fast and diverse sampling as we demand speed more than accuracy. On the other hand, we use the same scheduler as in [50] with  $\sigma_{\mathrm{min}} = 0.0001$ ,  $\sigma_{\mathrm{max}} = 3$  and  $\rho = 9$ . This combination allows us to sample a style vector for high-quality speech synthesis with only three steps, equivalent to running a 9-layer transformer model, minimally impacting the inference speed (see Appendix B for more discussions).
$V$  conditions on  $\pmb{t}$  through  $h_{\mathrm{bert}}$  concatenated with the noisy input  $\pmb {E}(\pmb {x}) + \sigma \xi$ , and  $\sigma$  is conditioned via sinusoidal positional embeddings [53]. In the multispeaker setting, we model  $p(s|t,c)$  by  $K(s;t,c,\sigma)$  with an additional speaker embedding  $\pmb {c} = \pmb {E}(\pmb{x}_{\mathrm{ref}})$  where  $\pmb{x}_{\mathrm{ref}}$  is the reference audio of the target speaker. The speaker embedding  $\pmb{c}$  is injected into  $V$  by adaptive layer normalization [6].
3.2.3 SLM Discriminators
Speech language models (SLMs) encode valuable information ranging from acoustic to semantic aspects [55], and SLM representations are shown to mimic human perception for evaluating synthesized speech quality [45]. We uniquely transfer knowledge from SLM encoders to generative tasks via adversarial training by employing a 12-layer WavLM [11] pre-trained on 94k hours of data $^{1}$  as the discriminator. As the number of parameters of WavLM is greater than StyleTTS 2, to avoid discriminator overpowering, we fix the pre-trained WavLM model  $W$  and append a convolutional neural network (CNN)  $C$  as the discriminative head. We denote the SLM discriminator  $D_{SLM} = C \circ W$ . The input audiors are downsampled to  $16\mathrm{kHz}$  before being fed into  $D_{SLM}$  to match that of WavLM.  $C$  pools features  $h_{\mathrm{SLM}} = W(\pmb{x})$  from all layers with a linear map from  $13 \times 768$  to 256 channels. We train the generator components  $(T,B,G,S,P,V)$ , denoted as  $G$  and  $D_{SLM}$  to optimize:
$$
\mathcal {L} _ {s l m} = \min  _ {\boldsymbol {G}} \max  _ {D _ {S L M}} \left(\mathbb {E} _ {\boldsymbol {x}} \left[ \log D _ {S L M} (\boldsymbol {x}) \right] + \mathbb {E} _ {\boldsymbol {t}} \left[ \log \left(1 - D _ {S L M} (\boldsymbol {G} (\boldsymbol {t}))\right) \right]\right), \tag {5}
$$
Available at https://huggingface.co/microsoft/wavlm-base-plus
where  $G(t)$  is the generated speech with text  $t$ , and  $x$  is the human recording. [56] shows that:
$$
D _ {S L M} ^ {*} (\boldsymbol {x}) = \frac {\mathbb {P} _ {W \circ \mathcal {T}} (\boldsymbol {x})}{\mathbb {P} _ {W \circ \mathcal {T}} (\boldsymbol {x}) + \mathbb {P} _ {W \circ \mathcal {G}} (\boldsymbol {x})}, \tag {6}
$$
where  $D_{SLM}^{*}(\pmb{x})$  is the optimal discriminator,  $\mathcal{T}$  and  $\mathcal{G}$  represent true and generated data distributions, while  $\mathbb{P}_{\mathcal{T}}$  and  $\mathbb{P}_{\mathcal{G}}$  are their respective densities. The optimal  $\pmb{G}^{*}$  is achieved if  $\mathbb{P}_{W\circ \mathcal{T}} = \mathbb{P}_{W\circ \mathcal{G}}$ , meaning that when converged,  $\pmb{G}^{*}$  matches the generated and true distributions in the SLM representation space, effectively mimicking human perception to achieve human-like speech synthesis.