StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
Authors: Yinghao Aaron Li
Cong Han
Vinay S Raghavan
Gavin Mischler
Nima Mesgarani
Venue: NeurIPS 2023 poster (2023)
Keywords: Speech Processing
Text-to-Speech
Diffusion Model
Large Language Model
Self-Supervised Speech Model
WavLM

Abstract: In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github.io/.

StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
Yinghao Aaron Li
Cong Han
Vinay S. Raghavan
Gavin Mischler
Nima Mesgarani
Columbia University
Abstract
In this paper, we present StyleTTSS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SL. to achieve human-level TTS synthesis. StyleTTSS 2 differs from its predecessor by modeling styles latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such
WavLM, as discriminators with our novel differentiable duration modeling for end- to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the
LJSpeech dataset and matches it on the multispeaker VCTK dataset judged by native English speakers. Moreover, when trained on the LibriTTSS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at htttt::///// http:::://sylett2..g1i2... io/
Introduction
Text-to-speech (TTS) synthesis has seen significant advancements in recent years, with numerous applications such as virtual assistants, audiobooks, and voice-over narration benefiting from increas-
ingly natural and expressive synthetic speech IT 22. Some previous works have made significant However, the quest for robust and accessible human-level TTS synthesis remains an ongoing challenge because there still room for improvement
progress towards human-level performance 3
in terms of diverse and expressive speech [5] 6) robustness for out-of nut-of-distributio (OOD) texts (
and the requirements of massive datasets for high-performing zero-shot TTS systems g
In this paper, we introduce Style'TTS 2, an innovative TTS model that builds upon the style-based generative model StyleTTS (6] present the next step towards human-level TTS systems. We model speech styles as latent random variable and sample them with a probabilistic diffusion model, allowing the model to efficiently synthesize highly realistic speech without the need for reference audio. Since it only needs to sample style vector instead of the entire speech as a latent variable, StyleTTS is faster than other diffusion TTS models while still benefiting from the diverse speech synthesis enabled by diffusion models. One of the key contributions of StyleTTS 2 is the use of large pre-trained speech language models (SLMs) like Wav2Vee 2.0 O. HuBERT 100 and WavLM I1 as discriminators, in conjunction with novel differentiable duration modeling approach. This end-to-end (E2E) training setup leverages SLM representations to enhance the naturalness of synthesized speech, transferring knowledge from large SLMs for speech generation tasks.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Our evaluations suggest that speech generated by StyleTTS 2 surpasses human recordings as judged
by native English speakers on the benchmark LJSpeech [22 dataset with statistically significant comparative mean opinion scores (CMOS) of +0.28 (p < 0.05). Additionally, StyleTTS 2 advances the state-of-the-art by achieving a CMOS of +1.07 (p < 0.01) compared to NaturalSpeech [5] Furthermore, it attains human-level performance on the multispeaker VCTK dataset [13] in terms of naturalness (CMOS= = -0.02, p > 0.05) and similarity (CMOS = +0.30, p < 0.1) to the reference speaker. When trained on a large number of speakers like the LibriTTS dataset [1], StyleTTS 2 demonstrates potential for speaker adaptation. It surpasses previous publicly available models in this task and outperforms Vall-E [8] in naturalness. Moreover, it achieves slightly worse similarity to the target speaker with only a 3-second reference speech, despite using around 250 times less data compared to Vall-E, making it a data-efficient alternative for large pre- pre-training in the zero-shot speaker adaptation task. As the first model to achieve human-level performance on publicly available single and multispeaker datasets, Style'TTS sets new benchmark for TTS synthesis, highlighting the potential of style diffusion and adversarial training with SLMs for human-level TTS synthesis.
2 Related Work
Diffusion Models for Speech Synthesis. Diffusion models have gained traction in speech synthesis due to their potential for diverse speech sampling and fine ane-grained speech control [13). They have
been applied to mel-based text-to-speech [16] 17
18
19. 20), mel-to-waveform vocoder 121 [22
23 24 22 26), and end-to-end speech generation 27 28 29 However, their efficiency is limited
compared to non-iterative methods, like GAN-based models 31), due to the need to iteratively
sample -pe-stoctsogs,mss waveforms, or other latent representations proportional to the target speech duration [15]. Furthermore, recent works suggest that tfte-of-theear GAN-based models still perform better than diffusion models in speech synthesis (26 33). To address these limitations, we introduce style diffusion, where a fixed-length style vector is sampled by diffusion model conditioned on the input text. This approach significantly improves model speed and enables end- to-end training. Notably, StyleTTS 2 synthesizes speech using GAN-based models, with only the style vector dictating the diversity of speech sampled. This unique combination allows StyleTTS 2 to achieve high-quality synthesis with fast inference speed while maintaining the benefits of diverse speech generation, further advancing the capabilities of diffusion models in speech synthesis.
Text- Texeech with Large Speech Language Models. Recent advancements have proven the effectiveness of large-scale self-supervised speech language models (SLMs) in enhancing text-to- speech (TTS) quality [34, 335 36 (37 and speaker adaptation 8 38 29 39]. These works typically convert text input into either continuous or quantized representations derived from pre- pre-trined SLMs for speech reconstruction. However, SLM features are not directly optimized for speech synthesis, contrast, our model benefits from the knowledge of large SLMs via adversarial training using SLM features without latent space mapping, thus directly learning a latent space optimized for speech synthesis like other end-to-end (E2E) TTS models. This innovative approach signifies a new direction in TTS with SLMs.
while tuning SLMs as a neural codee 34 35| 29 involves two-stage training.
Human-Level Text-to-Speech Several recent works have advanced towards human-level TTS 3 using techniques like BERT pre-training [4] 40 7 and E2E training 32 5 with differentiable
duration modeling [1]42 42). VITS (3] demonstrates MOS comparable to human recordings on the LJSpeech and VCTK datasets, while PnG-BERT 4 obtains human-level results on a proprictary dataset. NaturalSpeech (5), in particular, achieves both MOS and CMOS on L.Speech statistically indistinguishable from human recordings. However, we find that there is still room for improvement in speech quality beyond these state-of-the-arr models, as we attain higher performance and set a new standard for human-level TTS synthesis. Furthermore, recent work shows the necessity for disclosing the details of evaluation procedures for TTS research [43). Our evaluation procedures are detailed in Appendix H. which can be used for reproducible future research toward human-level TTS.
3 Methods
3.1 StyleTTS Overview
StyleTTS 6 a non-autoregresssiee TTS framework using a style encoder to derive a style vector from reference audio, enabling natural and expressive speech generation. The style vector is incorpo-
2

---

rated into the decoder and duration and prosody predictors using adaptive instance normalization
(AdaIN) (44. allowing the model to generate speech with varying duration, prosody, and emotions.
The model comprises eight modules, organized into three categories: (1) a speech generation system (acoustic modules) with a text encoder, style encoder, and speech decoder; (2) TTS prediction system with duration and prosody predictors; and (3) a utility system for training, including a discriminator, text aligner, and pitch extractor. undergoes two-stage training process: the first stage trains the acoustic modules for mel-spectrogram reconstruction, and the second trains TTS prediction modules using the fixed acoustic modules trained in the first stage.
In the first stage, the text encoder T encodes input phonemes t into phoneme representations Mext T()), while the text aligner A extracts speech-phonems alignment daga A(x t) from input speech x and phonemes t to produce aligned phoneme representations haggn = heet: Galgn via dot product. Concurrently, the style encoder F obtains the style vector F(x), and the pitch extractor F extracts the pitch curve P = F(x)) along with its energy da = ||a|||- Lastly, the speech decoder G reconstructs e G (helgn= 8, Pæ: ? n=),, which is trained match input æ using L1 reconstruction loss Cmel and adversarial objectives Ladv, Lim with discriminator D. Transferable monotonic aligner (TMA) objectives are also applied learn optimal alignments (see Appendix G for details).
In the second stage, all components except the discriminator D are fixed, with only the duration and prosody predictors being trained. The duration predictor S predicts the phoneme duration with dpeed = S(haa. whereas the prosody predictor P predicts pitch and energy as Pa. Ωg = P(hees s). The predicted duration is trained match the ground truth duration dg derived from the summed monotonic version of the alignment aaggn along the time axis with an In loss Laur- The predicted pitch Pa and energy an are trained to match the ground truth pitch Pa and energy ne derived from pitch extractor F' with L1 loss Lyo and L. During inference, dpred is used to upsample Mex through apred. the predicted alignment, obtained by repeating the value for dhred times fi-1, where E; is the end position of the 7th phoneme t; calculated by summing drreal [k] for k E (1, (1......). .)., and dprea(t] [i] are the predicted duration of 4,. The mel-spectrogram is synthesized by preed = Gheestt apred, E(ū), pa. âg) with z an arbitrary reference audio that influences the style of Epred. which is then converted into a waveform using a pre-trained vocoder.
Despite its sfate-of-the-art performance in synthesizing diverse and controllable speech, StyleTTS has several drawbacks, such as a two-stage training process with an additional vocoding stage that degrades sample quality, limited expressiveness due to deterministic generation, and reliance reference speech hindering real-time applications.
3.2 StyleTTS 2
StyleTTS 2 improves upon the StyleTTS framework, resulting in more expressive text-to- text-to-speech (TTS) synthesis model with human-level quality and improved duitof-distribution performance. We introduce an end-to-end (E2E) training process that jointly optimizes all components, along with direct waveform synthesis and adversarial training with large speech language models MSL enabled by our innovative differentiable duration modeling. The speech style is modeled as latent variable sampled through diffusion models, allowing diverse speech generation without reference audio. We outline these important changes in the following sections with an overview in Figure |
3.2.1 End-to-End Training
E2E training optimizes all TTS system components for inference without relying on any fixed components like pre-trained vocoders that convert mel-spectrograms into waveforms 3 32 To achieve this, we modify the decoder G to directly generate the waveform from the style vector, aligned phoneme representations, and pitch and energy curves. We remove the last projection layer for spessppctsogssmss of the decoder and append a waveform decoder after it. We propose two types of decoders: HifiGAN-bare and iSTFTNet-based. The first is based on Hifi-GAN (30), which directly generates the waveform. In contrast, the iSTFTNet-basee and phase, which are converted into waveforms using inverse short-time Fourier transform for faster training and inference. We employ the snake activation function [46), proven effective for waveform generation in 31). An AdaIN module [44 is added after each activation function to model the style dependence the speech, similar to the original StyleTTS decoder. We replace the mel-diseriminator in [6] with the multi-period discriminator (MPD) [30 and multi-resolution discriminator (MRD)
decoder (45] produces magnitude
3

---

Mal--paatrogram
Phonemes
Prosodio Style Enooder
Acoustic Style Enooder
Pish Extractor
Text Aligner
Acoustic Text Enooderr
Prosodic Text Enooder
Duraion Predictor
Prosody Predictor
Recoretructed Waselomm
Style Ditlusion Denotser
Decoder
(a) Acoustic modules pre-training and joint training. To accelerate training. the pre-training first optimize modules inside the blue box; the joint training then follows optimize all components except the pitch extractor, which is used to provide the ground truth label for pitch curves. The duration predictor trained with only Laur-
Phonomes
Acouatic
Prosedo Text
Preeedy
Encoder
TTS pww.ppteeinnnnneo mod.uns
Duration Predictor
Style Sampler
Usilly medills for hainiing
Upeamole
Sper
Accustio mod pe-iiniinng
pre-troiinng
WavLM
Autar
Dacriminntiv Head
Ponsnareeecoo Fake??
Synthsireed
Waveforms
(b) SLM adversarial training and inference. WavLM is pre-trained and not tuned. Unlike (a), the duration predictor is trained E2E with all components using Liim (eq. (5 via differentiable upsampling. This process is separate from (a) during training the input texts the gradients are accumulated both processes each batch update parameters.
different, but
Training and inference scheme of StyleTTS for the single-speaker case. For the multi- speaker case, the acoustic and prosodic style encoders (denoted as E) first take reference audio oe of the target speaker and produce a reference style vector E(wret). The style diffusion model then takes e as a reference sample Sp and sa that correspond to the speaker Fxet-
Figure 1:
47 along with the LSGAN loss functions 48 for decoder training, and incorporate the truncated pointwise relativistic loss function 49) enhance sound quality (see Appendix [F and G for details).
We found that well-trained acoustic modules, especially the style encoder, can accelerate the training process for TTS prediction modules. Therefore, before jointly optimizing all components, we first pre-train the acoustic modules along with the pitch extractor and text aligner via Cmel Cuiv. Cin and TMA objectives for N epochs where N depends on the size of the training set, in the same way as the first training stage of [6]. However, we note that this pre- pre-training is not an absolute necessity: despite being slower, starting joint training directly from scratch also leads model convergence,
After acoustic module pre-training, we jointly optimize Lmet. Ladv, Lim, Laur, Lm and L.. where Lmel modified to match the mel-spectrograms of waveforms reconstructed from predicted pitch p. and energy a (Fig Th). During joint training, stability issues emerge from diverging gradients, as the style encoder must encode both acoustic and prosodic information. To address this inconsistency, we introduce a prosodic style encoder E, alongside the original acoustic style encoder E.., previously denoted as E' in section 3.1 Instead of using 8, (x), predictors S and P take sp E,( as the input style vector. The style diffusion model generates the augmented style vector This modification effectively improves sample quality (see section (.3)). To further decouple the acoustic modules and predictors, we replace the phoneme representations heet from T. now referred to as acoustic text encoder, with hhen from another text encoder based on BERT transformers, denoted as prosodic text encoder. Specifically, we employ a phoneme-level BERT 7 pre-trained on extensive corpora of Wikipedia articles as the prosodic text encoder. This approach has been shown to enhance the naturalness of STyleTTS in the second stage 77, similar proposed usage here.
(sp. sal.

---

With differentiable upsampling and fast style diffusion, we can generate speech samples during training a fully differentiable manner, just as during inference. These samples are used to optimize fist (eq. 5) during joint training to update the parameters of all components for inference (Fig T)).
3.2.2 Style Diffusion
In StyleTTS 2, we model the speech æ as conditional distribution p(ar) = f p(artt, s)p(s(t) ds through a latent variable that follows the distribution p(s(t). We call this variable the generalized speech style, representing any characteristic in speech beyond phonetic content including but not limited to prosody, lexical stress, formant transitions, and speaking rate [6]. We sample by EDM 50 that follows the combined probability flow [51] and time-varying Langevin dynamics [52
8 s= f -T)
/ (28(
(T)) V. (s]p
(1)
where (7) is the noise level schedule and d(r) is its time derivative, (()) is the stochasticity term, Wr is the backward Wiener process for 7 E [T, and V. log (s|t) is the score function at time т. We follow the EDM [50] formulation with the denoiser K(s;t, or) preconditioned as:
(s) = data
data or
s Ino )
(2)
where follows log-normal distribution In (V mean: Padd) with Pimaan 1.2, Pad 1.2, o² v0² + o dana is the scaling term, dataa 0.2 is the empirical estimate of the standard deviation of style vectors, and V is a 3-layer transformer [53] conditioned on t and α which is trained with:
Cdmm (.,.ta..-N(...) ((O) [[<(E()) oo;t, of E(
where E(x) := EE (x) (x)), and A(o) := (0-1(o ddaa))² 2 is the weighting factor. Under this framework, equation 16 becomes ODE where the score function depends on instead of T:
ds
o)
do o V. logpo
s(o(T)) (T)))
Unlike (50] that uses 2nd-order Heun, we solve eq. with the ancestral DPM-2 solver [54] for fast and diverse sampling as we demand speed more than accuracy. On the other hand, we use the same scheduler as in [50] with mii 0.0001, max = 3 and م = 9. This combination allows us to sample style vector for high-qualin speech synthesis with only three steps, equivalent to running a 9 9-layer transformer model, minimally impacting the inference speed (see Appendix B for more discussions).
V conditions on t through Aben concatenated with the noisy input E(x) + af, and a is conditioned model p(s]t, e) by K (s; t, c, o)) with an additional speaker embedding c = E(T,e) where aeet is the reference audio of the target speaker. The speaker embedding e is injected into by adaptive layer normalization (6).
via sinusoidal positional embeddings (53). In the multispeaker setting,
3.2.3 SLM Discriminators
Speech language models (SLMs) encode valuable information ranging from acoustic to semantic aspects [55], and SLM representations shown mimic human perception for evaluating synthe- sized speech quality 45). We uniquely transfer knowledge from SLM encoders to generative tasks via adversarial training by employing 12-layer WayLM [II] pre-trained on 94k hours of data ass the discriminator. As the number of parameters of WavLM is greater than StyleTTS 2, to avoid dis- criminator overpowering, we fix the pre-trained WavLM model W and append a convolutional neural network (CNN) C' as the discriminative head. We denote the SLM discriminator DELM = C o W. The input audios are downsampled to 16 kHz before being fed into DSLM to match that of WavLM. C' pools features fSLM = W from all layers with a linear map from 13 x 768 to 256 channels. We train the generator components (T,, B, G. S, P, V. denoted as G) and D SLLM optimize:
Luhm min max (En llg DSLM (=)] + [log (1 DSLM (G()))), G DELM
(5)
Available http://///angggggoooooooooo//oo/ http:///wwaaaaaaocooo//////////a
5

---

where G(t is the generated speech with text t. and a is the human recording. [56] shows that:
PwoT(#)
D'LLs (x) (w) + (x))
(6)
where D'LM (a:) is the optimal discriminator, T and 9 represent true and generated data distributions, while Py and Pg are their respective densities. The optimal G" is achieved if W Wo mean- ing that when converged, G" matches the generated and true distributions in the SLM representation space, effectively mimicking human perception achieve human-like speech synthesis.
In equation 5 the generator loss is independent of ground truth æ and relies only on input text t. This enables training out-of-distributin (OOD) texts, which we show in section 5.3 can improve the performance on OOD texts. In practice, to prevent DSLM from over-fitting on the content of the speech, we sample in-distribution and OOD texts with equal probability during training.
3.2.4 Differentiable Duration Modeling
The duration predictor produces phoneme durations dpeed. but the upsampling method described in section to obtain apred is not differentiable, blocking gradient flow for E2E training. NaturalSpecch [5] employs an attention-basee upsampler 42] for human-lewe TTS. However, we find this approach unstable during adversarial training because train our model using differentiable upsampling with only the adversarial objective described in eq. 5 and without extra loss terms due to the length mismatch caused by deviations of daeed from dg. Although this mismatch
be mitigated with soft dynamic time warping as used [42]] we find this approach both computationally expensive and unstable with mel-reconstruucion and adversarial objectives. To achieve human-level performance with adversarial training, a non-parametric upsampling method is preferred for stable training.
Gaussian upsampling 41 is non non-parametric and converts the predicted duration dpred into apred (n, i] using a Gaussian kernel No. (n; o)) centered at := l; [ with the hyperparameter a:;
exp (- (n (/a=c)²² c:)2
(n; exp
(-222 20²
P dprcal [k),
where & the end position and o is the center position of the 7th phoneme t,, However, Gaussian upsampling has limitations due to its fixed width of Gaussian kernels determined by α. This constraint prevents from accurately modeling alignments with varying lengths depending on deread Non- attentive Tacotron [57 extends this by making a, trainable, but the trained parameters introduce instability for E2E training with adversarial loss, similar to issues of autention-based upsamplers.
We propose a new non- parametric differentiable upsampler without additional training while taking into account the varying length of the alignment. For each phoneme t., we model the alignment as random variable a, E N. indicating the index of the speech frame the phoneme t; is aligned with. We define the duration of the 7t phoneme as another random variable d, E {I,..., L}, where L 50 is the maximum phoneme duration hyperparameter, equivalent to 1.25 seconds in our setting. We
observe that a, Σ dy. but each dę is dependent on each other, making the sum difficult to model.
k=1
Instead, we approximate a; = d; + 11-1. The approximated probability mass function (PMF) of a, is
[n] = fdstte,, [n] fa. [n] fe.., [n]
fa. [k]] -8,,8,,,, k).
where 62,_, is the PMF of é;-1. a constant defined in eq. 8 This delta function is not differentiable,
so we replace it with Nr.,, defined in eq. with a = 1.5 (see Appendix C.2 for more discussions).
To model fa, we modify the duration predictor to output q[k: i], the probability of the 7th phoneme hav-
ing a duration of at least k for k E [I,...,L} optimized to be when dg 2 k with the cross-entropy
loss (see Appendix G Under this new scheme, we can approximate dprea
Σ glk, i]. trained to
match de: with Laur as in section 3.1 The vector o(:, can be viewed as the unnormalized version of fa,, though it is trained to be uniforlly distributed across the interval [1, d.). Since the number of speech frames M is usually larger than the number of input phonemes N, this uniform distribution

---

aligns single phonemes to multiple speech frames as desired. Finally, we normalize the differentiable approximation fa, [n] across the phoneme axis, as in [11, to obtain Opred using the softmax function:
(f),
M
apred [re, x]
(10)
N
fa, [m] = Σ q(n, i] NN k; o), k=
(11)
Σ
where M : [ey] is the predicted total duration of the speech and {L...., M) An illustration of our proposed differentiable duration modeling is given in Figure 4 (Appendix C
{L,
4 Experiments
4.1 Model Training
We performed experiments on three datasets: LJSpeech, VCTK, and LibriTTS. Our single-speaker model was trained on the LISpeech dataset, consisting of 13,100 short audio clips totaling roughly 24 hours. This dataset was divided into training 12,500 samples), validation (100 samples), and testing (500 samples) sets, with the same split as 3 5 6). The multispeaker model was trained on VCTK, comprising nearly 44,000 short clips from 109 native speakers with various accents. The data split was the same as BI. with 43,470 samples for training, 100 for validation, and 500 for testing. Lastly, we trained our model the combined LibriTTS train-clean-460 subset [14] for zero-shot adaptation. This dataset contains about 245 hours of audio from 1,151 speakers. Utterances longer than 30 seconds shorter than one second were excluded. We distributed this dataset into training (98%), validation (1% 1)), and testing (1%) sets, in line with [6). The test-clean subset was used for zero-shot adaptation evaluation with 3-second reference clips. All datasets were resampled to 24 kHz to match LibriTTS, and the texts were converted into phonemes using phonemizer (58)
We used texts in the training split of LibriTTS as the out-of-distribution (OOD) texts for SLM adversarial training. We used iSTFTNet decoder for LJSpeech due to its speed and sufficient performance on this dataset, while the HifiGAN decoder was used for the VCTK and LibriTTS models. Acoustic modules were pre-trained for 100, 50, and 30 epochs on the LJSpeech, VCTK, and LibriTTS datasets, and joint training followed for 60, 40, and 25 epochs, respectively. We employed the Adam W optimizer
(59] with 3g = 0, B2 = 0.99, weight decay X = 10 learning rate y = 10-4 and batch size of 16 samples for both pre-training and joint training. The loss weights were adopted from 6 to balance all loss terms (see Appendix GG for details). Waveforms were randomly segmented with max length of 3 seconds. For SLM adversarial training. both the ground truth and generated samples were ensured to be 3 to 6 seconds in duration, the same as in fine-tuning of WavLM models for various downstream tasks (IT]. Style diffusion steps were randomly sampled from training for speed and set to 5 during inference for quality. The training was conducted on four NVIDIA A40 GPUs.
during
4.2 Evaluations
We employed two metrics in our experiments: Mean Opinion Score of Naturalness (MOS-N) for human-likeness and Mean Opinion Score of Similarity (MOS-S) S) for similarity to the reference for multi-speaker models. These evaluations were conducted by native English speakers from the U.S. on Amazon Mechanical Turk. All evaluators reported normal hearing and provided informed consent as monitored by the local institutional review board and in accordance with the ethical standards of the Declaration of Helsinki [ In each test, 80 random text samples from the test set were selected and converted into speech using our model and the baseline models, along with ground truth for comparison. Because 77) reported that many TTS models perform poorly for OOD texts, LJSpeech experiments also included 40 utterances from Librivox spoken by the narrator of LJSpeech but from audiobooks not in the original dataset as the ground truth for OOD texts. To compare the difference between in-distribution and OOD performance, we asked the same raters to evaluate samples on both in-distribution and OOD texts.
Our baseline models consisted of the three and JETS 332 for L.Sppecc; and VITS, YourTTS 60), and StyleTTS for LibriTTS. Each synthesized
public models: VITS 3 StyleTTS (
We obtained approval for our protocol (number IRB-AAAR8657 from the Institutional Review Board.
7

---

Table I: Comparative mean opinion scores of naturalness and similarity for StyleTTS 2 with p-values from Wilcoxon test relative to other models. Positive scores indicate StyleTTS 2 is better.
Model
Dataset
CMOS-N (p-value)
CMOS-S (p-value)
Ground Truth
LJSpeech LJSpeech
+0.28 (p 0.021)
NaturalSpeech
+1.07 (p 10 10-6)
Ground Truth
VCTK
-0.02 (p = 0.628)
+0.30 (p - 0.081)
VITS
VCTK
+-0.45 (p 0.009)
+0.44 (p 0.032)
Vall-E
LibriSpeech (zero-shot)
+0.67 (p < 10 10-3)
-0.47 (p < 2
speech set was rated by 5 to 10 evaluators on a 1-5 scale, with increments of 0.3. We randomized the model order and kept their labels hidden, similar to the MUSHRA approach [6] 62) We also conducted Comparative MOS (CMOS) tests to determine statistical significance, as raters can ignore subtle differences in MOS experiments whether the second was better or worse than the first on a -6 to 6 scale with increments of 1. We compared our model the ground truth and NaturalSpeech [5) for LJSpeech, and the ground truth and VITS for VCTK. For the zero-shot experiment, we compared our LibriTTS model to Vall-E g
3 7 5] Raters were asked listen two samples and rate
All baseline models, except for the publicly available VITS model LibriTTS from ESPNet Toolkit [63]. were official checkpoints released by the authors, including vocoders used in StyleTTS. As NaturalSpeech and Vall-E are not publicly available, we obtained samples from the authors and the official Vall-F demo page, respectively. For faimess, we resampled all audio to 22.5 kHz for IJSpeech and VCTK, and 16 kHz for LibriTTS, match the baseline models. We conducted ablation studies using CMOS-N on IJSpeech on OOD texts from LibriTTS test-clean subset for more pronounced results as in 77. For more details of our evaluation procedures, please see Appendix E
5 Results
5.1
Model Performance
The results outlined in Table I establish that StyleTTS 2 outperforms NaturalSpeech by achieving a CMOS of +1.07 (p < 0.01), setting a new standard for this dataset. Interestingly, StyleTTS 2 was favored over the ground truth with CMOS of +0.28 (p- < 0.05). This preference may stem from dataset artifacts such as fragmented audiobook passages in the LJSpeech dataset that disrupt narrative continuity, thus rendering the ground truth narration seemingly unnatural. This hypothesis corroborated by the performance of StyleTTS 2 on the VCTK dataset, which lacks narrative context, where it performs comparably to the ground truth (CMOS = -0.02, > 0.05). Samples from our model were more similar to the reference audio speaker than the human recording, suggesting our model's iss effective use of the reference audio for style diffusion. Moreover, StyleTTS scored higher than the previous state-of-ther of-the--rtttttt-o art model VITS on VCTK, as evidenced by CMOS-N and CMOS-S.
Consistent with the CMOS results, our model achieved MOS of 3.83, surpassing all previous models on LJSpeech (Table 2). In addition, all models, except ours, exhibited some degrees of quality degradation for -f--triiitiiitio (OOD) texts. This corroborates the gap reported in 77. with our results providing additional ground truth references. On the other hand, our model did not show any degradation and sigmificantly outperformed other models in MOS for OOD texts (Table demonstrating its strong generalization ability and robustness towards OOD texts.
In zero-shot tests, StyleTTS surpasses Vall-E in naturalness with a CMOS of +0.67 (p & 0.01), although falls slightly short in similarity (Table Im- portantly, StyleTTS 2 achieved these re- sults with only 245 hours of training data, compared to Vall-E's 60k hours, a 250x difference. This makes TTY 2 data-efficient alternative to large pre- training methods like Vall-E. The MOS results Table 3 support these findings,
Table 2: Comparison of MOS with 95% confidence in- tervals (CI) on LJSpeech. MOSpp represents MOS-N for in-distribution texts, while MOSooD is that for OOD texts.
Model
MOSp (CI)
MOSooD (CI)
Ground Truth
3.81 (+: 0.09)
3.70 (+: 0.11)
StyleTTS 2 JETS VITS
3.83 (+: 0.08)
3.87 (+t 0.08)
3.57 (#: 0.09)
3.21 (+ 0.
3.34 (+: 0.10)
3.21 (1 0.11)
StyleTTS + HifiGAN
3.35 (#: 0.10)
3.32 (+ 0.12)
as our model exceeds all baseline models in both MOS-N and MOS-S. However, the difference in
8

---

Sputerr Sombard
IRODY
SNd Nagesaa
Inferene autio outlo
(a) LJSpeech model.
(b) Unseen speakers on LibriTTS, (c) Zoomed-in unseen speaker.
Figure 2: t-SNE visualization of style vectors sampled via style diffusion from texts in five emotions, showing that emotions are properly separated for seen and unseen speakers. (a) Clusters of emotion from styles sampled by the LJSpeech model.
(b) Distinct clusters of styles sampled from 5 unseen speakers by the LibriTTS model. (c) Loose clusters of emotions from Speaker in (b).
MOS-S between StyleTTS and StyleTTS 2 was not statistically significant, hinting at future directions for improvement in speaker similarity.
5.2 Style Diffusion
Figure shows t-SNE visual- izations of style vectors created using style diffusion pro- cess. Due to the lack of emotion- labeled audiobook text datasets, we used GPT-4 to generate 500 utterances across five emotions for this task [64]. Figure 2h, style vectors from the LJSpeech model illustrate distinct emo-
Table 3: Comparison of MOS with 95% confidence intervals (CI)
on rest-clean subset of LibriTTS for zero-shot speaker adaptation.
Model
MOS-N (CI)
MOS-S (CI)
Ground Truth
4.60 (+: 0.09)
4.35 (+: 0.10)
StyleTTS 2
4.15 (# 0.11)
4.03 0.11)
YourTTS
2.35 (# 0.07)
2.42 0.09)
VITS
3.69 (#: 0.12)
3.54 (1 0.13)
StyleTTS + HiFi-GAN
3.91 (# 0.11)
4.01 0.10)
tional styles in response input sentiment, demonstrating the model's capability to synthesize expressive speech in varied emotions without explicit emotion labels during training. This process was repeated with the LibriTTS model on five unseen speakers, each from a 3-second reference audio. As depicted in Figure 2p. distinct clusters form for each speaker, showcasing wide stylistic diversity derived from single reference audio. Figure 2 provides more nuanced view the first speaker, revealing visible emotion- baseed clusters despite some overlaps, indicating that we can manipulate the emotional tone of an unseen speaker regardless of the tones in the reference audio. These overlaps, however, can partly explain why the LibriTTS model does not perform as well as the LJSpeech model, as it is harder to disentangle texts from speakers in the zero-shot setting (see Appendix A.2 for more results). Table 4 displays synthesized speech diversity against several baseline models with the coefficient of variation of duration (CVaur) and pitch curve (CVm)) from the same input text (3]. Our model yields the highest variation, indicating superior potential for generating diverse speech. Despite being diffusion-basse our model is faster than VITS, FastDiff (28). and ProDiff [18), two of the fastest cestuusion-bbsee TTS models, with 5 iterations of diffusion and iSTFTNet decoder.
5.3 Ablation Study
Table 4: Speech diversity metrics and real-time factor (RTF). ProDiff and FastDiff were tested with steps of diffusion.
Table 5: CMOS-N relative to the StyleTTS 2 baseline on OOD texts the ablation study.
Model
CMOS-N
Model
CVam f
CV 1
RTF 1
w/o style diffusion
00.66
StyleTSS 2
0.0321
0.6962
0.0185
w/o differentiable upsampler
-0.21
VITS
0.0214
0.5976
0.0599
w/o SLM adversarial training
-0.32
FastDiff ProDiff
0.0295
0.6490
0.0769
w/o prosodic style encoder
-0.35
2e-16
0.5898
0. 0.1454
w/o OOD texts
-0.15

---

Table details the ablation study, underlying the importance of our proposed components. When style vectors from style diffusion are substituted with randomly encoded ones as in (6. the CMOS is -0.46, highlighting the contribution of text-dependent style diffusion to achieving human-level TTS. Training without our differentiable upsampler and without the SLM discriminator results in a CMOS of -0221 and -0 0..32, validating their key roles in natural speech synthesis. Removing the prosodic style encoder also yields a -0.35 CMOS. Last, excluding OOD texts from adversarial training leads to CMOS of -0.15, proving its efficacy for improving OOD speech synthesis. Table 6 in Appendix A.3 shows similar outcomes with objective evaluations, further affirming the effectiveness of various components proposed in this work. Figure 7 in Appendix D details layer-wise analysis of input weights of the SLM discriminator, providing a different view of the efficacy of SLM discriminators.
6 Conclusions and Limitations
In this study, we present StyleTTS 2, a novel text-to-speech (TTS) model with human-level perfor- mance via style diffusion and speech language model discriminators. In particular, it exceeds the ground truth on L.Speecc and performs on par with on the VCTK dataset. StyleTTSS 2 also shows potential for zero-shot speaker adaption, with remarkable performance even on limited training data compared to large-scale models like Vall-E. With our innovative style diffusion method, StyleTTS generates expressive and diverse speech of superior quality while ensuring fast inference time. While StyleTTS 2 excels several areas, our results indicate room for improvement in handling large-scale datasets such as LibriTTS, which contain thousands of or more speakers, acoustic environments, accents, and other various aspects of speaking styles. The speaker similarity in the aforementioned zero-shot adaptation speaker task could also benefit from further improvements.
However, zero-shot speaker adaptation has the potential for misuse and deception by mimicking the voices of individuals as potential source of misinformation or disinformation. This could lead to harmful, deceptive interactions such as theft, fraud, harassment, or impersonations of public figures that may influence political processes trust in institutions. In order to manage the potential for harm, we will require users of our model to adhere to a code of conduct that will be clearly displayed as conditions for using the publicly available code and models. In particular, we will require users to inform those listening to samples synthesized by StyleTTS 2 that they are listening to synthesized speech to obtain informed consent regarding the use of samples synthesized by StyleTTS 2 in experiments. Users will also be required to use reference speakers who have given consent to have their voice adapted, either directly by license. Finally, we will make the source code publicly available for further research in speaker fraud and impersonation detection.
In addition, while human evaluators have favored StyleTTS 2 over ground truth with statistical significance on the LJSpeech dataset, this preference may be context-dependent. Original audio segments from larger contexts like audiobooks could inherently differ in naturalness when isolated, potentially skewing the evaluations in favor of synthesized speech. Additionally, the inherent variability in human speech, which is context-independent might lead to lower ratings when compared to the more uniform output from StyleTTS 2. Future research should aim to improve evaluation methods to address these limitations and develop more natural and human-like like speech synthesis models with longer context dependencies.
7 Acknowledgments
We thank Menoua Keshishian, Vishal Choudhari, and Xilin Jiang for helpful discussions and feedback on the paper. We also thank Grace Wilsey, Elden Griggs, Jacob Edwards, Rebecca Saez, Moises Rivera, Rawan Zayter, and D.M. for assessing the quality of synthesized samples and providing feedback on the quality of models during the development stage of this work. This work was funded by the national institute of health (NIHNIDCD) and a grant from Marie-Josee and Henry R. Kravis.
References
[1] Yishuang Ning, Sheng He, Zhiyong Wu, Chunxiao Xing, and Liang-Jie Zhang. A review of deep learning based speech synthesis. Applied Sciences, 409), 2019.
10

---

[2] Xu Tan, Tao Qin, Frank Soong, and Tie Yan Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2106. 155666,,, 2021.
[3] Jaehyeon Kim, Jungil Kong, and Juhee Son, Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference Machine Learning, pages 5530-5540. PMLR, 2021.
[4] Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu. PnG BERT: Augmented BERT on phonemes and graphemes for neural TTS. arXiv preprint arXiv:2103./5060 2021.
[5] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang. Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, Frank K. Soong, Tao Qin, Sheng Zhao, and Tie-Yan Liu. NaturalSpeech: End-to-End Text Speech Synthesis with Human-Level Quality. ArXiv, abs/2205.04421, 2022.
[6] Yinghao Aaron Li, Cong Han, and Nima Mesgarani. StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesis. arXiv preprint arXiv:2205.15455 2022.
[7] Yinghao Aaron Li, Cong Han, Xilin Jiang, and Nima Mesgarani. Phoneme-Leve Bert for Enhanced Prosody of Text-To-Specee with Grapheme Predictions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.
[8] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers. arXiv preprint arXiv:2301 10001.02111 111, 2023.
[9] Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for sulf-supe learning of speech representations, Advances in neural information processing systems, 33: 12244-114666 2020.
[10] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. HuBERT: Supervisd Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460 2021.
[11] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. WavLM: Large-Scale Sepf-Supervieed Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing, 1500:1155-111 2022.
[12] Keith Ito and Linda Johnson. 2017.
The lj speech dataset.
https://keithito..c com/
LJ
[13] Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al. CSTR VCTK Corpus: English Multi-Speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92). 2019.
[14] Heiga Zen, Viet Dang. Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. Proc. Interspeech 2019, pages 1526-15220 1530, 2019.
[15] Chenshuang Zhang, Chaoning Zhang, Sheng Zheng, Mengehun Zhang, Maryam Qamar, Sung- Ho Bae, and In So Kweon. A Survey on Audio Diffiusion Models: Text To Speech Synthesis and Enhancement in Generative AI. arXiv preprini arXiv:2303.1 13336, 2, 2023.
[16] Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-TTS: A Denoising Diffusion Model for Text-to-Speech. In Interspeech, 2021.
[17] Vadim Popov, Ivan Vovk. Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- TTS: Diffusion Probabilistic Model for -epeech. In International Conference on Machine Learning, pages 8599-8608. PMLR, 2021.
[18] Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. ProDiff: Progressive Fast Diffusion Model for High-Quality Text-to-Speech. In Proceedings of the 30th ACM International Conference Multimedia, pages 2595-2605, 2022.
11

---

[19] Songxiang Liu. Dan Su, and Dong Yu. DifiGAN-TTS: High-Fidelity and Efficient Text-to- Speech with Denoising Diffusion GANs. arXiv preprint arXiv:2201.11972. 2022.
[20] Donghhao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng. Yuexian Zou, and Dong Yu. Diffsound: Discrete Diffusion Model for Text-to-Sound Generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[21] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating Gradients for Waveform Generation. In International Conference on Learning Representations.
[22] Zhifeng Kong. Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiveWave: Versatile Diffusion Model for Audio Synthesis. ArXiv, abs/2009.09761, 2020.
[23] Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng. Tao Qin, Wei Chen, Sungroh Yoon, and Tie- Liu. PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependee Adaptive Prior. In International Conference on Learning Representations.
[24] Zehua Chen, Xu Tan, Ke Wang. Shifeng Pan, Danilo Mandic, Lei He, and Sheng Zhao. Infergrad: Improving diffusion models for vocoder by considering inference in training. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8432-8436. IEEE, 2022.
[25] Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping, In Interspeech, 2022.
[26] Yuma Koizumi, Kohei Yatabe, Heiga Zen, and Michiel Bacchiani. WaveFit: An Iterative and Non-autoregressivve Neural Vocoder based on Fixed-Poin Iteration. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 884-891. IEEE, 2023.
[27] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, Najim Dehak, and William Chan. Wavegrad Iterative refinement for text-to-speech synthesis. In Proceedings of Interspeech, pages 3765-3769, 2021.
[28] Rongjie Huang, Max W. Lam, J. Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. FastDiff: A Fast Conditional Diffusion Model for High-Qualin Speech Synthesis. In International Joint Conference on Artificial Intelligence, 2022.
[29] Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers. arXiv preprint arXivv2304.091 16,0009116611000000000 2023.
[30] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. Advances in Neural Information Processing Systems, 13:1170221177 17033, 2020.
[31] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. BigVGAN: A Universal Neural Vocoder with Large-Scale Training, arXiv preprint arXiv:2206.0446 04658, 2022.
[32] Dan Lim, Sunghee Jung, and Eesung Kim. JETS: Jointly training FastSpeech2 and HiFi-GAN for end to end text to speech. arXiv preprint arXiv:2203. 2022.
[33] Matthew Baas and Herman Kamper. GAN You Hear Me? Reclaiming Unconditional Speech Synthesis from Diffusion Models. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 906-911. 111.1M IEEE, 2023.
[34] Chenpeng Du, Yiwei Guo, Xie Chen, and Kai Yu. VQTTS: High-Fidelity Texe-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature. arXiv preprint arXiv:2204.00766, 2022.
[35] Hubert Siuzdak, Piotr Dura, Pol van Rijn, and Nori Jacoby. WavThruVec: Latent speech repre- sentation as intermediate features for neural speech synthesis. arXiv preprint arXiv:2203. 16930, 2022.
12

---

[36] Sang-Hoon Lee, Seung- Bin Kim, Ji-Hyun Lee. Eunwoo Song, Min-Jae Hwang, and Seong- Whan Lee. HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis. Advances in Neural Information Processing Systems, 35: 16224-6666, 2022.
[37] Jinlong Xue, Yayue Deng, Fengping Wang, Ya Li, Yingming Gao, Jianhua Tao, Jianqing Sun, and Jiaen Liang. M2-CTTS: End-to-End Multi-scale Multi-modal Conversational Text-to- Speech Synthesis. arXiv preprint arXiv:2305.0222 02269, 2023.
[38] Ruiqing Xue, Yanqing Liu, Lei He, Xu Tan, Linquan Liu, Edward Lin, and Sheng Zhao. FoundationTTS: Text-to-Speed for ASR Custmization with Generative Language Model. arXiv preprint arXiv:2303.02 02939, 2023.
[39] Kenichi Fujita, Takanori Ashihara, Hiroki Kanagawa, Takafumi Moriya, and Yusuke Ijima. Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model. arXiv preprint arXiv:2304.11976 2023.
[40] Guangyan Zhang, Kaitao Song. Xu Tan, Daxin Tan, Yuzi Yan, Yanqing Liu, G. Wang, Wei Zhou, Tao Qin, Tan Lee, and Sheng Zhao. Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech. In Interspeech, 2022.
[41] Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End- to- tndEnd Adversarial Speeeh. arXiv preprint vrXiv:2006.03575 2020.
[42] Isaac Elias, Heiga 7en, Jonathan Shen, Yu Zhang, Jia Ye, R. J. Skerry-Ryan and Yonghui Wu. Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling. In Interspeech, 2021.
[43] Cheng-Hat Han Chiang, Wei-Ping Huang, and Hung-yi Lee. Why we should report the details in subjective evaluation of tts more rigorously. arXiv preprint arXiv:2306.0200 2023.
[44] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[45] Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki. iSTFTNet: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time Fourier transform. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6207-6211 1. IEEE, 2022.
[46] Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how fix it. Advances in Neural Information Processing Systems, 33:1583-1594, 2020.
[47] Won Jang, Daniel Chung Yong Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. UnivNet: A Neural Vocoder with Resoitieeoo Spectrogram Discriminators for High-Fidelity Waveform Generation. In Interspeech, 2021.
[48] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In Proceedings of the IEEE international conference computer vision, pages 2794-2802, 2017.
[49] Yanli Li and Congyi Wang. Improve gan- based neural vocoder using truncated pointwise relativistic least square gan. In Proceedings of the 4th International Conference on Advanced Information Science and System, pages 1-7, 2022.
[50] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00366 2022.
[51] Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective on diffusion- based generative models and score matching. Advances in Neural Information Processing Systems, 34:22863-2222 22876, 2021.
[52] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
13

---

[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[54] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, DPM-Solver: Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. arXiv preprint arXivv:206.00092
2022.
[55] Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. Layer-wise analysis of self-supervised speech representation model. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 914-921. IEEE, 2021.
[56] Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger. Projected GANs Converge Faster. Advances in Neural Information Processing Systems, 34:17480-17483 2021.
[57] Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu. Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling, arXiv preprint arXiv:2010.04300 2020.
[58] Mathieu Bernard and Hadrien Titeux. Phonemizer: Text to Phones Transcription for Multiple Languages in Python. Journal of Open Source Software, 6(68):3958, 2021.
[59] Ilya Loshchilov and Frank Hutter. Fixing Weight Decay Regularization in Adam, 2018.
[60] Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir Ponti. YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone. In International Conference on Machine Learning, pages 2709-2720.
PMLR, 2022.
[61] Yinghao Aaron Li, Ali Zare, and Nima Mesgarani. StarGANv2-VC: A Diverse, Unsu- pervised, Non-parallel Framework for Natural-Sounding Voice Conversion. arXiv preprint SIIV:2200..0394 2021.
[62] Yinghao Aaron Li, Cong Han, and Nima Mesgarani. Styletts-vc: One-shot voice conversion by knowledge transfer from style-based tts models. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 920-927. IEEE, 2023.
[63] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Ren- duchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit. In Proceedings of Interspeech, pages 2207-2211, 2018
[64] Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshen He, Zhengliang Liu, et al. Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models. arXiv preprint 27X1022304.01852, 2023.
14

---

Appendix A Additional Evaluation Results
A.1 Feedback Analysis from Survey Participants
On the last page of our survey, we encouraged the raters to provide their feedback on the study. While the majority opted not to comment, those who did provided valuable insights into our survey design and results interpretation. The main point of feedback revolved around the difficulty in discerning differences between different samples, specifically for the LJSpeech CMOS experiments. Below a few representative comments from our CMOS experiment participants:
Sounds great. At most points the differences were negligible.
really liked the person
was reading
clips.
She sounds professional and was very good at changing her tone of voice. Sometimes it was hard to evaluate the clip because you had to listen carefully when they sounded similar.
I didn't have any problems. enjoyed participating Thank you!
For most of the rounds, it was difficult to notice a difference between the voices.
Interested to see the results differences were sometimes
noticeable, often subtle
This feedback strengthens our conclusion that StyleTTS 2 has reached numan--lke text-to text-to-speech synthesis quality. Several participants offered specific observations regarding the differences:
Some sounded little unnatural because they sounded like actor auditioning for a role little too dramatic.
So, a couple of comments. I feel like the expressiveness of the speaker made think it was more natural, but hyper expressiveness would be unnatural in certain contexts (and did feel odd with some of them but it felt more real). Additionally, the different voices, in being expressive, sometimes tended to turn into different emotions and contextual these emotions sometimes felt unnatural but felt real at the same time.
Fun study though. wish you all well!
some sound relatively peexpectedd/mmatureee but most often it was just minor difference
These comments suggest that the differences between StyleTTS and the original audio mainly lie in subtle nuances like intonation and word emphasis.
Our results indicate that StyleTTS 2 scored statistically higher than the original audio in both MOS and CMOS, likely due to the absence of contextual continuity in the common evaluation procedure used in TTS studies. This disruption of narrative continuity in the LJSpeech dataset, which consists of isolated audiobook clips, might have led to the perceived unnaturalness of the ground truth compared to samples generated by STyleTTS 2. Future research should aim incorporate context-aware long-form generation into human-like text-to-speech synthesis to improve evaluation faimess and relevance.
A.2 Speech Expressiveness
Our model's ability to synthesize diverse and expressive speech is demonstrated by synthesizing speech from 500 text samples across five emotions generated using GPT-4. This reinforces the results visualized in Figure 2 which maps style vectors from texts associated with different emotions using t-SNE. We use the mean value of the FO and energy curves to evaluate emotive speech.
As depicted Figure 3 (a), speech synthesized by our model exhibits visible FO and energy charac- teristics for each emotion, notably for anger and surprise texts, which deviate from the ground truth average. This substantiates our model's capacity to generate emotive speech from texts associated with specific emotions. Conversely, VITS shows certain degrees of insensitivity to emotional variance,
15

---

Grund
Surarised
Neutral
386
260
220
6.64
0.55
0.35
6,7
8.06
FO Meen
Energy Keen
StyleTTS2
Ground Iuth
Argry
Happy
Sad
Surp food
Nic (tra
1.00
140
265
240
363
280
6/4
0.24 Energy Mean
(b) VITS
FD
Enargy Mean
(c) JETS
Figure 3: Histograms and kernel density estimation of the mean FO and energy values of speech, synthesized with texts in five different emotions. The blue color ("Ground Truth") denotes the distributions of the ground truth samples in the test set. StyleTTSS 2 shows distinct distributions for different emotions and produces samples that cover the entire range of the ground truth distributions.
as displayed in Figure 3 (b). JETS, shown in Figure 3 (c), fails to span the full distribution of FO and energy, producing samples clustered around the mode. In contrast, our model extends across both FO and energy distributions, approximating the range of the ground truth.
Since both StyleTTS and VIT'S are probabilistic while JETS is deterministic, this shows that probabilistic models generate our model demonstrates slightly better mode coverage compared to VITS at the right tail of the energy mean distribution, likely because we use diffusion-based models opposed the variational autoencoder used in VITS.
expressive speech compared to deterministic ones. Moreover,
For audio samples of the synthesized speech and their corresponding emotional text, please refer to our demonstration page at/tttss//styllttts22 github. 10/ #emo
A.3 Ablation Study
The ablation study results, presented in Table 6 illustrate the importance of each component in our proposed model using several measures: mel-cepstral distortion (MCD), MCD weighted by Speech Length (SL) that evaluates both the length and the quality of alignment between two speeches (MCD-SL), root mean square error of log FO pitch (Fo RMSE), mean absolute deviation of phoneme duration (DUR MAD), and word error rate (WER), following (36). The MCD and Fa RMSE were computed between the samples generated by each model and the ground truth aligned with dynamic time warping. The MCD-SL computed with a pre- pre-trained ASR model from ESPNet (63]
computed using a publicly available library The WER was
Most notably, using randomly encoded style vectors instead of those sampled through style diffusion considerably affects all metrics, including the subjective CMOS score, thereby underscoring its
hattge///gtthhbbbccmm///epnppppp
'Available wubbppes//eenddd.oo/rrroo//////
16

---

Table 6: Comparison of models in ablation study with mel-cepstral distortion (MCD), MCD weighted by Speech Length (MCD-SL), root mean square error of log FO pitch (Fo RMSE), mean absolute deviation of phoneme duration (DUR MAD), and word error rate (WER) on LJSpeech. The CMOS results are copied directly from T as a reference.
Model
MCD
MCD-SL
Fo RMSE
DUR MAD
WER
CMOS
Proposed model
4.93
5.34
0.651
0.521
6.50%
0
w/o style diffusion
8.30
9.33
0.899
0,634
8.77%
-0.6
w/o SLM adversarial training
4.95
5.40
0.692
0.513
6.52%
-0.32
w/o prosodic style encoder
5.04
5.42
0.663
0,543
6.92%
-0.35
w/o differentiable upsampler
4.94
5.34
0.880
0.525
6.54%
-0.21
w/o OOD texts
4.93
5.45
0.690
0.516
6.58%
-0.15
significance. Since [6] establishes that the style vectors crucially influence all aspects of the speech, such as pauses, emotions, speaking rates, and sound quality, style diffusion the most important factor in producing speech close to the ground truth and natural human speech.
Excluding adversarial training with large speech language models (SLMs) results in slight decline in MCD-SL and Fo RMSE but does not impact WER. Interestingly, this version produces the lowest duration error, implying that SLM discriminators may cause minor underfitting on in-distribution texts, which can also be observed in models trained without OOD texts during SLM adversarial training. Nonetheless, subjective evaluations reveal significant difference for out-of-distribution (OOD) texts for models trained without SLM discriminators (see Appendix D Dor more discussions).
Training without the differentiable upsampler increases the Fo RMSE, but leaves the MCD, MCD-SL, DUR MAD, and WER unaffected. The removal of the prosodic style encoder affects all metric scores, thus underscoring its effectiveness in our model. Lastly, training without OOD texts only affects the Fo RMSE in our objective evaluation.
Further details, discussions, and audio samples regarding these model variations can be found in Section of our demo page: https://stylettta2 github. io/#ab
Appendix B
Style Diffusion
B.1
EDM Formulation
EDM 50 formulates the sampling procedure of diffusion-basee generative model as an ordinary differential equation (ODE) (Eq. in (50)):
da = - log (())) dt,
(12)
where the noise level schedule and its derivative are represented by at) and (t), respectively. The score function p(x) is V. log p(æ; (t)), ), and p(æ) the targeted data distribution (please refer to Appendix B in (50] for detailed derivation). The equation can be rewritten in terms of do as:
dx = -V æ log () do.
(13)
The denoising score matching objective is defined as follows (Eq. 108 in [50]:
Eyr- (.--pam.._A((...1))).....
(oo [I (:o) v12
(14)
Here, denotes the training sample from data distribution Paaa: E is the noise, and D is the denoiser, defined as (Eq. in (50):
(15) In the above equation, Fg represents the trainable neural network, Casp(a) modulates the skip connection, Cn (o) and (a) regulate input and output magnitudes respectively, and Cnoise (ar) maps
D(x;)) Gaip (a (a) (oo
(o)æ;; Croise (n)ise(a)).
17

---

noise level a into conditioning input for Fa. With this formulation, the score function is given by (Eq. in [50):
V log pog; a (1)) (D( a) a:) /0².
(16)
In Table [50], the authors define the scaling factors as follows:
Cskip (0) = ddual /o² dduaa (O (aaua//") /0")²
(17)
Cour (a) = 0 Odata/
dma + + = Odata/ 'o",
(18)
Gn(o) 1/v data + 1/0",
(19)
Cnoise (a)
In(o),
(20)
4
where o² := Vo + o², shorthand notation for simplicity.
Inserting equations 17 18 19
and 20 into equation 13 yields:
D σ) Cakip( j + Cou (a)
Cnoise (o))
data 0
data
1
(21)
x
Ino
o 4
Combining equation 13 and 16 we obtain
dæ
do = o Va logp(x;;(tt)
(D(rr; m)) m)//" /02
(22)
x D(#;0)
Equations 2 and can be recovered by replacing æ with 8, p(x) with p(s(t), F& with and with K in equations 21 and 22 respectively.
In EDM, the time step (au, 01.... :2.....,O---.. N for the total step N defined as (Eq. in (50):
i m max
oce
m
(23)
where max Op. min ON-1, and p is the factor that shortens the step lengths near at the expense of longer steps near max- Optimal performance is observed in [50] when Faa >> d
and p E [5, 10). Through empirical tests, we set amin 0.0001, max 3 and 9 in our work,
allowing fast sampling with small step sizes while producing high-quality speech samples.
B.2 Effects of Diffusion Steps
Table 7: Comparision of mel-cepstral distortion (MCD), MCD weighted by Speech Length (MCD-SL), root mean square eиror of log FO pitch (F, RMSE), word error rate (WER), real-time time factor (RTF), coefficient of variation of duration (CVa..), and coefficient of variation of pitch (CV between different diffusion steps.
Step
MCD |
MCD-SL J
Fo RMSE
WER 1
RTF
CVa 1
CV
4
4.90
5.34
0.650
6.72%
0.0179
0.0207
0.5473
8
4.93
5.33
0.674
6.53%
0.0202
0.0466
0.7073
16
4.92
5.34
0.665
6.44%
0.0252
0.0505
0.7244
32
4.92
5.32
0.663
6.56%
0.0355
0.0463
0.7345
64
4.91
5,34
0.654
6.67%
0.0557
0.0447
0.7245
128
4.92
5.33
0.656
6.73%
0.0963
0.0447
0.7256
The impact of diffusion steps on sample quality, speed, and diversity was studied using multiple indicators. We assessed the sample quality using mel-cepstral distortion (MCD), MCD weighted by
18

---

Speech Length (MCD-SL). root mean square error of log FO pitch (Fo RMSE), and word error rate (WER). We also examined the influence of diffusion steps on computational speed using the real-time factor (RTF), which was computed on single Nvidia RTX 2080 Ti GPU. Given the application of the ancestral solver, which introduces noise at each integration step (52), it was postulated that a higher number of steps would lead to more diverse samples. This hypothesis was tested using the coefficient of variation on the FO curve (CV (o) and duration (CV)
The model was run with diffusion steps ranging from 4 to 128, as results with only two steps were disregarded due to the characteristics of the noise scheduler in EDM. Here, do max and NN-1 = min. and therefore, with merely two diffusion steps. the attainment of samples of acceptable quality is not feasible.
The results presented in T negligible disparities in sample quality across varying diffusion steps, with satisfactory quality samples being producible with as few three steps in our experiments. Consequently, we decided to randomly sample styles with diffusion steps ranging from 3 to during training in order to save time and GPU RAM. Furthermore, diversity relative to the number of diffusion steps, reaching a plateau around 16 steps. Thereafter, the increase is marginal, with a slight decrease in diversity noticed when the steps are large. potentially attributable to the refining of the noise schedule with higher step counts. This could cause the ancestral solver converge to a fixed set of solutions despite the noise introduced at each step.
observed incremental rise in speech
Notably, optimal results in terms of sample quality and diversity, paired with acceptable computational speed, were achieved with around 16 diffusion steps. Even though the RTF experienced 30% increase compared to 4 steps, it still outperformed VITS by twice the speed, making it a suitable choice for real-time applications.
B.3 Consistent Long-Form Generation
Algorithm 1 Long-form generation with style diffusion
procedure LONGFORM(t, ()) T + StLLI((t) for irii (0, (O,...,N. N 1} do
D is the input paragraph, [0, is the weighting factor b. Split t into sentences (To. Ti...., "1....,Ty_.. -1
8cum + STYLEDIIFUSION(T)
Sample a style vector with the current text T; Check if the previous style vector is defined b Convex combination interpolation
if Sprev then
Scur + a&Suu + (1 - a.) Sprev
end if
K; f SYNTHESIZE T,, Scurr)
> Synthesize with the interpolated ccur
8 Sprev f- 8Eur
b Set 8pre for the next iteration
end for return CONCAT((aa, "I. 2N-1)) end procedure
Return concatenated speech from all sentences
Our findings indicate that style diffusion creates significant variation in samples, characteristic that poses challenges for long-form synthesis. In this scenario, a long paragraph is usually divided into smaller sentences for generation, sentence by sentence, in the same way as real-time applications. Using an independent style for each sentence may generate speech that appears inconsistent due to differences in speaking styles. Conversely, maintaining the same style from the first sentence throughout the entire paragraph results in monotonic, unnatural, and robotic-sounding speech.
We empirically observe that the latent space underlying the style vectors generally forms convex space. Consequently, a convex combination of two style vectors yields another style vector, with the speaking style somewhere between the original two. This allows to condition the style of the current sentence on the previous sentence through a simple convex combination. The pseudocode of this algorithm, which uses interpolated style vectors, is provided in Algorithm
An example of a long paragraph generated using this method with a = 0.7 is available on our demo page: httts::/sstyltttee gi github.... folub.io/Htt /t #long
19

---

0.995 4 0.00
246 & 2477
140 4 0.33
Doslinn
Speech
Speach
Speach
Dunsioa prodictor outjut
(b) Gaussian Fiber contered
Unnormalized predicted alignment
(6) Normaizzd predicted alignment
Figure 4: Illustration of our proposed differentiable duration upsampler. (a) Probability output from the duration predictor for input tokens with 5. (b) Gaussian filter N centered f--1. (c) Unnormalized predicted alignment fa, [r] from the convolution operation between (a) and (b). (d)
Normalized predicted alignment apred over the phoneme axis.
B.4
Style Transfer
Our model, contrary to traditional end-to-end (E2E) TTS systems that directly map texts to speech- related representations, decouples speech content and style via a style vector. This setup enables the transfer of style sourced from text with specific emotions or stylistic nuances to any given input text. The procedure involves generating style vector from a text reflecting desired aspects of the target style (e.g.. emotions, speaking rates, recording environments, etc.) and synthesizing speech from any input text using this style vector. This process also extends to unseen speakers, as demonstrated on our demo page at hhtppss //stylyttstts. github. io/ #libri] It is important to note that the relationship between speech style and input text learned in a self-supervised manner without manual labeling, analogous to the CLIP embedding used in Stable Diffusion.
Appendix C Differentiable Duration Modeling
C.1 DSP-Based Formulation
formulated the differentiable duration model using probability theory. This section provides an alternative formulation based on principles of digital signal processing (DSP). The upsampler formulated using the following DSP properties:
In section 3.2.4
<m|] [m] xrnn R)),
(24)
5c(x) = lim Nc(x;
(25)
o 2
Here, 6x [n] is the Kronecker delta function, 8.(z)) is the Dirac delta function, and No; is the Gaussian kernel centered at c, as defined in eq. 7 These properties state that a delta function, &g- through convolution operation, shifts the value of function normalized Gaussian kernel converges to the delta function as a + O. Therefore, by properly selecting σ, we can approximate the shifting property of the delta function with NN(x;; a) and adjust the duration predictor output gl:, il to the starting position & of the jH phoneme. Figure illustrrtess our proposed differentiable duration modeling. and real example of the duration predictor output, the predicted alignment using the non-differentiable upsampling method described section 3.1 and the alignment using our proposed method is given in Figure
position by k and a properly
Toan maki
20
Theen
20
32
10
20
60
80
50
10
(a) Predictor output.
(b) Non-differentiahle upsampling.
(c) Proposed upsampling with a 1.5.
Figure 5: An example of duration predictor output and the predicted alignment with and without differentiable duration upsampler. (a) displays the log probability from the duration predictor for improved visualization. Although (b) and (c) differs, the duration predictor is trained end-to-end with the SLM discriminator, making the difference perceptually indistinguishable in synthesized speech.
20

---

C.2 Effects of a on Training Stability and Sample Quality
The gradient propagation from the SLM discriminator to the duration predictor crucial for our model's performance. The norm of this gradient, computed using the chain rule, is the product of the norms of individual functions' gradients. The magnitude of the partial derivatives No σ) is:
aNc(2: Oz
ox; = dc
+ (x - -ooooo o²
(e((;;o ---d o²
C 02'
(26)
where C' a constant dependent on L. The gradient norm is thus of order 0(1/o*), where depends on the input dimension. As O, or) approaches (z(),, but the gradient norm [VV.Ne(rr;; a) goes to infinity. Moreover, smaller a values may not yield better approximations due to numerical instability, particularly when dealing with small values such as the predictor output g [0, 1]. which is multiplied by No(2;; σ)) in eq. [1. exacerbating numerical instability.
We investigate the off by examining the impact of a training stability and sample quality. Training stability is represented by the gradient norm from the SLM discriminator to the duration predictor, and sample quality is measured by the mel cepstral distortion (MCD) between speech synthesized using the predicted alignment generated through the non-differentiable upsampling method (section 3.T) and our proposed differentiable upsampling method. We use the maximum gradient over an epoch to represent training stability, as large gradient batches often cause gradient explosion and unstable training. We compute these statistics for values from 0.01 10.0 log scale, with results shown in Figure
10
1200
1000
6-
600-
400
200
10a2 10n4
10
10*
10
(a) Sample quality (MCD)
(b) Training stability (max gradient norm)
Figure 6: Effects of a MCD and max gradient norm. Our choice of 1.5 is marked with a star symbol. (a) MCD between samples synthesized with differentiable and non-differentiable upsampling over different a. (b) The maximum norm of gradients from the SLM discriminator the duration predictor over an epoch of training with different a.
Our chosen Ø = 1.5 minimizes both MCD and gradient norm across a wide range of a values, making it optimal choice for upsampling. This value also aligns with the typical phoneme duration of 2 to 3 speech frames, as Gaussian kernels with o = 1.5 span approximately speech frames.
Appendix D SLM Discriminators
D.1 Layer-Wise Analysis
Following the approach of the WavLM study (L]. we performed a layer-wise feature importance analysis for the speech language model (SLM) discriminator. The weights of the input lincar projection layer into the convolutional discriminative head were normalized using the average norm values of WavLM features for samples in the test set across LJSpeech, VCTK, and LibriTTS datasets. Figure 7 depicts the normalized weight magnitudes across layers.
In the LJSpeech and LibriTTS models, the initial (1 and 2) and middle layers (6 and 7) showed the highest importance, while the final layers (10 and 11) were of diminishing importance. The least influential was the final layer, consistent with previous findings that ascribe the primary role of this layer to pretext tasks over downstream tasks (55 IO I1. Notably, layers 0, 1, and 2, which encode acoustic information like energy, pitch, and signal-to-noise ratio (SNR), emerged the most
21

---

Magnitude
3
Layer
(a) LJSpeech
Magnitude
2
10
Layer
(b) VCTK
Magnitude
3
2
10
12
Layer
(c) LibriTTS
Figure 7: Layer-wise input weight magnitude the SLM discriminators across different datasets. The layer importance shows divergent pattern for the VCTK model relative to the IJSpeech and LibriTTS models, showcasing the impact of contextual absence on the SLM discriminators.
crucial. Layers 5, 6, and 7, encoding semantic aspects like word identity and meaning [55 held secondary importance. This indicates that the SLM discriminator learns fuse acoustic and semantic information to derive paralinguistic attributes, such prosody, pauses, intonations, and emotions, which are critical distinguishing real and synthesized speech. The outcome reaffirms the SI.M discriminator's capacity to enhance the emotional expressiveness and prosody of synthesized speech.
In contrast, the SLM discriminator for the VCTK dataset demonstrated distinct layer preference, likely due to its limited contextual information. Unlike the storytelling and audiobook narrations in the LJSpeech and LibriTTS datasets, the VCTK dataset only involves reading standard paragraph devoid of specific context or emotions. This context shortage could explain the marginal performance increase of our model over VITS the VCTK dataset compared to LJSpeech and LibriTTS datasets (see Tables 3) as the advantages of style diffusion and SLM adversarial training in our model less pronounced in datasets with restricted expressiveness and emotional variety.
D.2 Training Stability
Figure 6(b) shows that the maximum norm gradient from the SLM discriminator to the duration predictor can reach up to 200 even with 1.5, potentially destabilizing training, given that the gradient norms for other modules are generally less than 10. This instability is particularly concerning when the gradient back-propagated the prosodic text encoder, which employs BERT model that is notably sensitive to gradient explosion. To mitigate this issue, we adopt strategy for scaling gradient from the SLM discriminator to the duration predictor, a common practice prevent gradient explosion in GAN training (31].
22

---

We found that the gradient norm of the entire predictor typically falls between 10 and 20, As such, we implement a scaling factor 0.2 the gradient norm when it surpasses 20. Additionally, we also account for potential instabilities arising from the sigmoid function used by the duration predictor for probability output. We mitigate these by scaling the gradient to the last projection layer and the LSTTM layer within the duration predictor by factor of 0.01. These scaling values have consistently shown stability across different datasets and can be modified to accommodate -ataset-specific attributes. All the models presented in this paper were trained with these scaling factors.
Appendix E Subjective Evaluation Procedures
In this section, we propose a detailed and standardized evaluation procedure and elucidate our adherence these guidelines in the current work.
E.1 Proposed Framework for Subjective Evaluation
In light of the findings in 4 along with guidelines from 5 we recommend the following procedures for assessing human-like text-to-speech models:
1. Raters should be native speakers of the language used in the model's training corpus, with provisions in place to exclude falsely claimed native speakers.
2. Attention checks should be incorporated into the evaluation to ensure thoughtful, careful completion, and raters who do not pass the attention checks should be excluded from the outcome analysis.
3. The objectives of the evaluation should be clearly stated. For instance, when assessing naturalness, its definition should be explicit within the evaluation framework.
4. Ideally, all models used in the experiments should be publicly available, and official imple- mentations are preferred over unofficial implementations. If official implementations publicly available, the testing samples can be requested from the authors of the papers, or from implementations that have been employed in other papers.
5. For model comparisons, the MUSHRA based approach, where paired samples from all models are presented, should be employed, in contrast to the traditional MOS evaluation scheme that presents model samples in isolation.
6. For nuanced model comparison and determining statistical significance, CMOS should be used in place of the MOS and MUSHRA evaluations. Each sample should be rated by at least 20 raters with more than 50 samples to claim statistical significance compared to ground truth to assess human-level quality (5).
E.2 Evaluation Details
The first criterion was ensured by selecting native English-speaking participants residing in the United States. Upon launching our survey on MTurr, we activated the following filters:
HIT Approval Rate (%) for all Requesters' HITS: greater than 95.
Location: is UNITED STATES (US).
Number of HITs Approved: greater than 50.
For each batch, completed responses were collected exclusively from self-identified native speakers in the United States, verified by residential IP addresses (i.e., not proxy or VPN) with an online service 5
To fulfill the second criterion, attention checks were implemented differently for MOS and CMOS evaluations. In the MOS assessment, we utilized the average score given by participant to ground truth audios, unbeknownst to the participants, to ascertain their attentiveness. We excluded ratings from those whose average score for the ground truth did not rank in the top three among all five models. In the CMOS evaluation, we assessed the consistency of the rater's scoring; if the score's
Avaiable httpa//iitttt////ooooooooooo/oop www.ooooooooooooooooooooooooooee
23

---

sign (indicating whether A was better or worse than B) differed for over half the sample set (in our case, 10 samples), the rater was disqualified. Six raters were eliminated through this process all of our experiments.
The third criterion was ensured by providing explicit definitions of "naturalness" and "similarity" within our surveys, exemplified by the following lines (see our survey links 100 for more detail);
Naturalness:
Some of them may synthesized while others may spoken by an American audiobook narrator.
Rate how natural each audio clip sounds scale of 1 to 5 with 1 indicating completely unnatural speech (bad) and 5 completely natural speech (excellent).
Here, naturalness includes whether you feel the speech is spoken by a native American English speaker from a human source.
Similarity:
Rate whether the two audio clips could have been produced by the same speaker or not on a scale of to 5 with indicating completely different speakers and indicating exactly the same speaker.
Some samples may sound somewhat degraded/distortedd; for this question, please try to listen beyond the distortion of the speech and concentrate on identifying the voice (including the person's accent and speaking habits, if possible).
In accordance with the fourth criterion, official model checkpoints were utilized in both MOS and CMOS evaluations. Specifically, for the MOS evaluation of LJSpeech, checkpoints for VITS? JE and StyleTTS with HifiGAN vocode were sourced directly from the authors. Likewise, for the MOS evaluation on LibriTTS, official checkpoints for StyleTTS with the HifiGAN vocoder and YourTTS (utilizing the Exp 4 checkpoint were used. Since there is official checkpoint of VITS for LibriTTS, we used the VITS model from ESPNet [13] on LibriTTS, which was previously employed as a baseline model in 6 For the CMOS experiment on VCTK with VITS, we used official checkpoints and implementation. For the CMOS evaluation with NaturalSpeech, we obtained 20 demo samples directly from the author. Regarding Vall-E's CMOS experiment, we collected 30 samples from LibriTTS, along with their corresponding 3-second prompts from the official demo page 14
Fulfilling the fifth criterion, we employed a MUSHRA-basee approach in our MOS evaluations. Lastly, we assured that 20 raters evaluated each sample in our CMOS experiments, after excluding ineligible raters. For NaturalSpeech and Vall-E, where additional samples were not available, we compensated by doubling the number of raters.
Appendix F Detailed Model Architectures
This section offers comprehensive outline of the enhanced StyleTTS 2 architecture. The model integrates eight original StyleTTT modules with the addition of an extra style diffusion denoiser, prosodic style encoder, and prosodic text encoder.
'LISpeech MOS: https://suveey. www.eee.....ccccc.cccmm.mmmm.. htt.aacceeeee..ce/e/3//6/6////// http:////w////////aoo.aa.a...aaa
ID-DDD-
7 'LISpeech CMOS: https::/sstreeee alemerr com
a http://co/////////////////0 http://11o//oSssggttyyetttttee 0430-
LibriTTS MOS:
chemer com/ ///////333333883/333 1tt.//880000000-11r111-111 050---
"Available

---

The model keeps the same architecture for the acoustic text encoder, text aligner, and pitch extractor as in the original StyleTTS [6]. The architecture of both the acoustic style encoder and prosodic style encoder follows that of the style encoder from StyleTTS [6]. The prosodic text encoder is a pre-trained PI-BERTI 15 model
[7] Additionally, we adopt the same discriminators, MPD and MRD, as specified in [3I] The decoder is combination of the original decoder from StyleTTTS and
either iSTFTNet [45] or HifiGAN [28], with the AdaIN [44] module appended after each activation function. The duration and prosody predictors follow the same architecture as in [6], albeit with alterations to the output of the duration predictor changing from 1 x to N for probability output modeling. Thus, this section focuses primarily providing comprehensive outlines for the style diffusion denoiser and the discriminative head of the SLM discriminators. Readers seeking detailed specifications for other modules can refer to the aforementioned references.
The Denoiser (Table is designed to process an input style vector 8, the noise level a, the phoneme embeddings from the prosodic text encoder hbee and, when applicable, speaker embedding from the acoustic and prosodic encoders in the multi-speaker scenario. Here, e also considered part of the input and is introduced to the denoiser through adaptive layer normalization (AdaLN):
AdaLN( s) (s) x p((x)
+ (,(ss)
(27)
O(x)
where r is the feature maps from the output of the previous layer, s is the style vector, pl-) and (+-) denotes the layer mean and standard deviation, and LT and Ly are learned linear projections for
computing the adaptive gain and bias using the style vector
Table Denoiser architecture. represents the input phoneme length of the mel- mel-sppctrogram, is the noise level, Neea is the phoneme embeddings, and is the speaker embedding. The size of s and is 256 1, that of hben 768 N, and x 1. Group(32) is the group normalization with group size of 32.
Submodule
Input
Layer
Norm
Output Shape
Sinusoidal Emhedding
256 x1
Embedding
Addition
256 x1
Repeat for N times
256 xN
Linear 256 1024
1024 x
Output Embedding
1024 xNN
Repeat for N Times
256 X
Input
heeert
Concat
Group(32)
1024x
Conv x024 1024
1024x N
k
Addition
1024 X
Transformer Block (x3)
8-Head Self-Atention (64 head features)
AdaLN c)
x
Linear 2048
2048 N
GELU
2048
Linear 2048 1024
N024x
Adaptive Average Pool
1024 X1
Output
Linear 1024x256
256x1
The Discriminative Head (Table 9 is comprised of a 3-layer convolutional neural network, conclud- ing with linear projection. Table 9: Discriminative head architecture. T represents the number of frames (length) of the output feature halm from WavLM.
Layer
Norm
Output Shape 13x 768 x
Input ha
Reshape
9984x7
Linear 9984x255
2
Conv 256x
256xT
Leaky ReLU (0.2)
256xT
Conv 256 x
512xT
Leaky ReLU (0.2)
512xT
Conv 512x512
512xT
Leaky ReLUU (0.2)
512xT
Conv 512x1
1xT
Available
25

---

Appendix G Detailed Training Objectives
In this section, provide detailed training objectives for both acoustic modules pre-training and joint training. We first pre-train the acoustic modules for accelerated training, and we then proceed with joint training with the pitch extractor fixed.
G.1 Acoustic module pre-training
Mel-spectrogram reconstruction. The decoder is trained on waveform y, its corresponding mel- spectrogram and the text t, using L, reconstruction loss as
Liell E [||= M (G (hext Oalgn: 8a Pw: Du })]]
(28)
Here, hes = T(t) is the encoded phoneme representation, and the attention alignment denoted by aalgn = A(x, t). The acoustic style vector of æ is represented by Sa = Ea (x). Pa symbolizes the pitch FO and fe indicates energy of æ, and (-) represents mel-spectrogram transformation. Follow- ing [6]. half of the time, raw attention output from A is used as alignment, allowing backpropagation through the text aligner. For another 50% of the time, a monotonic version of Oalgn utilized via
dynamic programming algorithms (see Appendix A in (6)
TMA objectives. We follow [6] and use the original requence-to-sequeen ASR loss function L2s to tune the pre-trained text aligner, preserving the attention alignment during end-to-end training:
L.ss Ex,t Σ CE(t,,
where is the number of phonemes t, is the i-th phoneme token of f, is the i-th predicted phoneme token, and CE(-) denotes the cross-entropy loss function.
Additionally, we apply the monotonic loss Lmono ensure that soft attention approximates its non-differentiable monotonic version:
Cuono Ex,t [||Calgn - Charall1]
(30)
where ahard is the monotonic version of Oalgn obtained through dynamic programming algorithms (see Appendix A in (6] for more details).
Adversarial objectives. Two adversarial loss functions, originally used in HifiGAN 30 are employed enhance the sound quality of the reconstructed waveforms: the LSGAN loss function Ladv for adversarial training and the feature-matching loss Lim-
Ladv (G; Et.
[D ((G (hrext aalgn San Pres nz)))
1)²]
(31)
Cadv (D; Ex (D((G) (Frex Galgn Sa. Per nzx 12)) Fuy (D(y) -1)²]
(32)
A
Lim Ey
('(y)) D' (G (hrest Oalgn Sa, Paz Te )))
(33)
N
where D represents both MPD and MRD, A is the total number of layers in D. and D denotes the output feature map of (-th layer with N₁ number of features.
In addition, we included the truncated pointwise relativistic loss function 49 to further improve the
sound quality:
G. D) Egg Eg pmm ReLU (T- (D-(D()) D - -mo)))] (34)
26

---

(D; B D(W)s( (r ReLU (r (T- (D - D (i) mpd)
(35)
where ý G (hrext Galgn: 8 Py: my) is the generated sample, ReLU(.) is the rectified linear unit function, is the truncation factor that is set to be 0.04 per [49 MGD and MDG are margin
parameters defined by the median of the score difference:
MGD Myy. (D(ÿ) D
(36)
mpg Myy (D()) D())):
(37)
where M []] is the median operation.
Acoustic module pre-training full objectives. Our full objective functions in the acoustic modules pre-training can be summarized as follows with hyperparameters Ags and Amono
min Liel + Ca2s + Cmonnomonn
min (Dv
G,A,E,,FFT
(38)
(39)
+ Lady () Liel D) + Lim
+ LD)(G)
Following (6), set Asz= 0.2 and Amono 5.
G.2 Joint training
Duration prediction. We use the following cross-entropy loss to train the duration predictor:
Lce = Edd ΣΣ CE (g R)))
(40)
where I (-) is the indicator function, q SShben sp) is the output of the predictor under the prosodic style vector 8p (x) and prosodic text embeddings heen = Bt),, is the number of phonemes in t, and dg is the ground truth duration obtained by summing aalgn along the mel frame axis.
Additionally, we employ the L-1 loss make sure approximated duration also optimized:
Lau Edg [l]dg dprodll1..
(41)
1. Σ glk, i] is the approximated predicted duration of the 7th phoneme.
where dpreat [i]
Prosody prediction. We use fo and Ln. which are FO and energy reconstruction loss, respectively:
C = Exe [llee p=1l1l
(42)
C, = || - fz ll where f., = P (hben, 8p) are the predicted pitch and energy of æ.
(43)
Mel-spectrogram reconstruction. During joint training, modify Liee follows:
Lmel Ez,t ||| M (G (hest Oalgn: 8n, Pa. Ω. [[1]
(44)
Now we use the predicted pitch Pex and energy tu reconstruct the speech. We also make similar changes to Ladv, Lim, and Lid during joint training.
SLM adversarial objective. The SLM discriminator can easily overpower the generator and disable the gradient flow to the generator. To facilitate learning when the discriminator overpowers the generator, we the LSGAN loss 48 instead of the min-max loss in eq.
Lylm (G; D) E2
(DELM ((G (hrea apred 8 Pa: n2))) 1)²]
(45)
27

---

Sulm (D; = Bt, (DSLM ((G (Fejext apredt Sa, Pa, Ra )1))
+
(46)
By (Ds -
where & is the acoustic style sampled from style diffusion, agred is the predicted alignment obtained through the differentiable duration upsampler from the predicted duration dprdd and and na are the predicted pitch and energy with the sampled prosodic style Sp-
Joint training full objectives. Our full objective functions in joint training can be summarized as follows with hyperparameters Adur, Aces A5o. An. 7ess, and Amono:
min F.N.B.B......VV....
Lmet XceLee + Aaur Cuur Ao + AnCn
+ Cazs + Amoon Lads (G; D)
(47)
+ LG; D) + Lim + falm (G; D) + feedm:
where Ledm is defined in eq. 3
The discriminator loss is given by:
min Ladv (D; + LD,(D; + Lstm (D;G)
(48)
Following 6 set As=- 0.2, Amono 5, 1, 0.1, An and Aoe 1.
28


Reviews:
[{"review_id": "SjjTAWozcL", "reviewer": "Reviewer_JmdN", "summary": "The paper introduces StyleTTS 2, a text-to-speech (TTS) model that leverages latent diffusion and adversarial training with SLMs. The most noteworthy thing is that the paper employs large pre-trained speech language models (SLMs), such as WavLM, as discriminators in the adversarial training process. This novel approach, combined with differentiable duration modeling for end-to-end training, improves the naturalness of the synthesized speech.", "questions": "None", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": " 1. It utilizes adversarial training with large speech language models (SLMs) in conjunction with differentiable duration modeling for end-to-end training. This approach improves the naturalness of synthesized speech. \n\n2. Text encoder in StyleTTS 2 is separated into an Acoustic Text Encoder and a Prosodic Text Encoder. The Prosodic Text Encoder is then used as a conditional input for the latent diffusion process. This design allows the Diffusion model to model the diverse prosody of different styles, and it can also be seen through the ablation study that latent diffusion is essential in this paper.\n\n", "weaknesses": "1. The article fails to provide a clear explanation of how the prosodic style encoder ensures the extraction of style from the Mel spectrogram rather than other elements. The specific mechanism behind this process is not adequately detailed in the paper. While the article briefly mentions certain style diffusion methods, they do not sufficiently explain how the prosodic style is accurately derived from the Mel spectrogram. Further exploration is required to devise more effective encoder architectures and training mechanisms to ensure the precise extraction of prosodic style information.\n\n2. The writing style of the article lacks coherence, making it challenging to grasp the main ideas and problem-solving strategies. The organization and presentation of the paper could have been improved.\n\n-----------\nThe author's response addressed some of my issues, so I raised my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your valuable time and effort in reviewing our paper. Your insights have helped us clarify certain points about our work. Here are our point-to-point responses. \n\n- **Concern about Zero-Shot TTS and Prosodic Styles**\n\nWe acknowledge that this aspect may not have been clearly articulated, and we apologize for not clarifying that the inference pipeline in Figure 1(b) is for the single-speaker model only. For multispeaker models, including zero-shot speaker adaptation, the style diffusion model does take both acoustic and prosodic styles as input, as described in the last paragraph of Section 3.2.2. We will make it clear that Figure 1(b) represents the single-speaker case only and we will add an additional description for the multispeaker case under the figure in the camera-ready version. Moreover, we would like to emphasize that our primary contribution lies in presenting human-level TTS systems for seen speakers with style diffusion and SLM discriminators. Beyond this, we present the zero-shot feature as an additional capability, and we leave further improvements in zero-shot speaker adaptation for future works, as discussed in Section 6. \n\n- **Concern about Prosodic Style Encoder and Decoupling from Speaker Timbre**\n\nWe regret that we have not made the function of our prosodic style encoder more clear. Its role is not to decouple from the speaker timbre but to capture additional stylistic information that the acoustic style encoder cannot. This relieves the \"burden\" of the acoustic style encoder so the information it extracts does not have to be used to both reconstruct the speech and predict the prosody (duration and F0) faithfully. As you have correctly pointed out,  the former needs to encode more of the timbre information, while the latter needs to extract more of the prosodic aspects. Hence, the prosodic style encoder is designed to complement, not decouple, the speaker's timbre by capturing additional stylistic information. We agree that this point can be made more clear and we will make the motivation of adding the prosodic style encoder more straightforward in the revision. \n\nIn addition, since the F0 is predicted based on the style extracted from the reference audio, our model is able to produce speech with pitch ranges similar to the reference speech. This was already shown in the previous work StyleTTS, and we have presented several samples in Section 4 on our demo page to demonstrate this effect. We acknowledge that this could have been elaborated more clearly, and we will do so in the revision by emphasizing that our model maintains the capability of the original StyleTTS that can match the styles in the synthesized speech with those extracted from the reference audio such as the pitch range. \n\n- **Concern about Clarity and Understanding of Prosodic Style Encoder**\n\nWe would like to clarify that the prosodic style encoders are trained specifically to reconstruct duration, F0, and energy, which we refer to as \"prosody\". It is worith noting that the prosodic style encoder is not the main focus of our paper. It is only to address the problem we found in the previous work (StyleTTS) that using a single style encoder to reconstruct both the speech and the prosodic information (duration, F0, and energy) is not sufficient. Hence, we simply duplicated the original style encoder to encode the acoustic and prosodic information separately. We appreciate the reviewer’s suggestion for further exploration of encoder architectures and training mechanisms, which we plan to do in future work. We will discuss this in Section 6 as a possible direction for future research in the revised version. \n\n- **Concern about the Readability of the Paper**\n\nWe appreciate your feedback regarding the readability of our paper. We have made significant efforts to improve the readability and accessibility of the writing. Based on your feedback and that of the other reviewers, we have improved the clarity of several sections. We believe the overall coherence is greatly improved. We welcome further comments or suggestions on how we can improve the presentation of our work.\n"}, {"review_id": "BCe944Y5kD", "reviewer": "Reviewer_5CmP", "summary": "The authors of the paper have proposed a high-quality TTS model StyleTTS 2. Unlike previous StyleTTS work, StyleTTS 2 has a DDPM-based style generator, speech language model discriminators, and a differentiable aligner. All of these design decisions help improve\nthe overall performance of the model. As the authors report, StyleTTS 2 sets new standards for TTS synthesis.", "questions": "Do MPD and MRD still exist in the jointly training stage?", "limitations": "No", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I appreciate the hard work of the authors on this paper. It was a well-written and enjoyable read.\nIt is a great challenge to further develop TTS models with high quality, as the current SOTA models are very close to human speeches.\nStyleTTS 2 optimizes all the components in the joint training stage.\nEvaluation is thorough enough to support the arguments with in-depth ablation studies.\nI have carefully checked the audio samples, they are promising.", "weaknesses": "The training procedures are complex. It requires several training stages, such as pre-training the acoustic modules, retraining the speech language models, and joint training.\nIt is unclear whether the speech language models (SLMs) are trained on audio clips (rather than the whole-length audio). If so, how do you learn semantic information from short audio clips?\nIt is also unclear whether the MPD and MRD still exist in the jointly training stage. I found MPD and MRD missing in Figure 1 (b).\nThe overall novelty is limited, as many previous works integrate the diffusion modules and speech language model modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in reviewing our paper and your constructive questions. We appreciate your positive feedback on our work and your insightful comments. We would like to address your concerns and clarify certain aspects of our paper.\n\n- **Concern about Complicated Training Procedure**\n\nWe acknowledge that the training procedure consists of more than one step. We hope that we can alleviate this issue by making the training code publicly available, ensuring that users can train our model easily. In addition, we would like to clarify that the SLMs are not re-trained. Instead, we appended a discriminator head on top of it, as mentioned in Section 3.2.3. We will ensure to highlight this point more prominently in the revised manuscript.\n\n- **Concern about Training SLM Discriminators on Audio Clips**\n\nWe followed the methodology of the original WavLM paper for downstream tasks. For example, the original WavLM paper used the SLM representations on 3-second clips and fine-tuned for extra 2 epochs with 6-second clips for the speaker verification task. They also used a batch size of 200 seconds of audio for base model finetuning and 80 seconds of audio for large model finetuning for the speech recognition task. In our training process, we randomly cropped the audio into 3-6 seconds (thus 96 to 192 seconds per batch) to mimic how the WavLM model was used for downstream tasks, as we have described in Section 4. \n\nHowever, we acknowledge that having longer clips can help with the performance, but it is also more costly in terms of GPU RAM usage. We have found that 3-6 seconds of clips are sufficiently long to produce naturalistic speech. Since the semantic information the discriminator captures is primarily for understanding the paralinguistic information as described in Appendix D.1, we stick to this setting in our implementation. We will make this point more apparent in the revised paper to avoid confusion.\n\n- **Concern about the Absence of MPD and MRD in Figure 1(b)**\n\nFigure 1(b) is an illustration of adversarial training with SLM only. MPD and MRD are not present for adversarial training with SLM but are present during the joint training process, as shown in Figure 1(a). The training consists of both joint training and adversarial training with SLM, where the gradients from both processes are combined to update the parameters of trainable modules in each batch. \n\nWe agree that the absence of these components in Figure 1(b) could be confusing. In the revised manuscript, we will amend the caption of Figure 1 to more clearly show that the actual training procedure is a combination of the processes illustrated in Figures 1(a) and 1(b). \n\n- **Concern about Novelty**\n\nWe appreciate your feedback, and we apologize for not presenting the novelties of our work as clearly as possible. The main novelties that we want to present are the following: Unlike previous works that sample waveforms, melspectrograms, or hidden representations proportional to the duration of the speech using diffusion models, we are the first to diffuse a fixed-length vector for speech synthesis, which greatly improves the model efficiency while maintaining the benefits of diverse samples provided by diffusion models. Moreover, unlike previous works, we employ SLM as a discriminator, rather than a source of hidden representations from which speech is decoded or reconstructed. While diffusion models and SLMs have indeed been proposed in previous works for speech synthesis, we use them in new ways that greatly improve the performance of the model. We will emphasize these points more in Section 2 of the final version of the manuscript.\n"}, {"review_id": "4Z4FmIoD2t", "reviewer": "Reviewer_ogQb", "summary": "The paper presents Style TTS 2, a new TTS model with three main innovations 1) diffusion of a latent variable to capture style (everything in the speech signal that’s not enumerated by a phone sequence t and speaker variable c), 2) the use of a pretrained speech language model (SLM) as a discriminator and 3) a novel formulation of a differentiable alignment function.  The authors also enable a new joint training curriculum.   These three innovations result in quality that surpasses existing available TTS approaches including VITS, VALL-E, Natural Speech and YouTTS.\n", "questions": "What does it mean for synthetic samples to “surpass” human recordings? Not in the context of how this happens as a score, but what does it say about the evaluation itself if it’s possible for an artificial sample to be more “natural” than a human derived sample which is “natural” with metaphysical certitude.  I appreciate the argument about unnatural segmentation of the audio in LJSpeech, but if this is the source of the “unnaturalness” in the test data, it’s maybe not worth making too much of a claim about its value as a quantitative evaluation.\n\nOn line 151, the augmented style vector is defined as s = [s_p,s_d], but s_d is not previously defined.  Should this be s_a?\n\nSection 3.2.3 - the generator loss being independent of ground truth is true of any GAN training, no? Is there something specific to the use of an SLM discriminator that enables this better than prior work training TTS with GAN objectives?\n\nSection 5.1 “setting a new standard for this dataset”.  What is the referent of “this dataset”? Is it the evaluation set of 80 text samples from LIbriTTS and 40 Librivox utterances spoken by the speaker of LJSpeech described in Section 4?\n", "limitations": "Very briefly in two sentences at the end of the conclusion. (My rating below is contingent on the outcome of an ethics review to assess if the limitations and discussion of ethics concerns are sufficient to comply with https://neurips.cc/public/EthicsGuidelines)", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper is well written, and, for the most part, easy to follow.   Each of the novel components are clearly described, well motivated and contextualized with other comparable approaches. Also, the quality of the samples is quite good.", "weaknesses": "Given that zero-shot speaker adaptation is extremely ripe for misuse as Societal Impact and Potential Harmful Consequences\nper https://neurips.cc/public/EthicsGuidelines, I would think the risk and mitigation section of this presentation should be more than 2 sentences.\n\nSection 3.2.1 using less resources is highlighted as a contribution of Style TTS 2 – does this include the “phoneme-level BERT pretrained on extensive corpora of Wikipedia articles”?  Similar question concerning Section 3.2.3 and the use of WavLM pre-trained on 94k hours of data?  It seems like this assessment is limited to the resources required for the model itself, but does not include components like the discriminator and text embedding.\n\nSome areas to tighten the presentation and contribution in the upfront material of the paper.\n\nIntroduction: the first paragraph establishes the areas of improvement for TTS as diverse, expressive speech, robustness to OOD text, and requirements of massive datasets for performing zero-shot TTS.   In the last paragraph of the introduction the contributions are oriented about quality, and novel use of SLMs. However, these are inconsistent with the areas for improvement enumerated by the authors.  It might be clearer to be specific about which open questions for in the first paragraph are going to be addressed in some form by the proposed technique.\n\nThe limitations of StyleTTS are again not the same as those established in the introduction. Namely they are 1) a two-stage training process, 2) limited expressiveness and 3) reliance on reference speech hindering realtime application.  StyleTTS 2 addresses these three limitations, unifying the training process, and reducing reliance of reference speech while improving OOD performance and expressiveness.  While this is compact and consistent as an improvement ot StyleTTS, it does not address the broad limitations of TTS established by the authors.\n\nSection 3.2 and 3.2.1 one of the advantages of Style TTS 2 is described as an E2E training process that jointly optimizes all components.  This is presented in contrast to STyle TTS which required a two-stage training process (Section 3.1).  However, later in Section 3.2.1 (line 142 and Figure 1a) Style TTS2 uses a two-stage training pre-training and joint training process.  If single stage training, or E2E training is meant to be an advantage and contribution of Style TTS 2, the distinction between its two-stage training and the two-stage training used by the original Style TTS needs to be more clearly defined.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful comments on our work. We appreciate your observations and feedback that have helped us identify areas for clarification and improvement, and here are our responses.\n\n- **Ethical Concerns on Misuse** \n\nWe are grateful for your attention to the potential ethical problems in our work. We acknowledge that having only two sentences is insufficient for addressing ethical issues of potential misuse. We will expand the discussion in the conclusion section by adding the following paragraph:\n\n> We acknowledge that zero-shot speaker adaptation has the potential for misuse and deception by mimicking the voices of individuals as a potential source of misinformation or disinformation. This could lead to harmful, deceptive interactions such as theft, fraud, harassment, or impersonations of public figures that may influence political processes or trust in institutions. In order to manage the potential for harm, we will require users of our model to adhere to a code of conduct that will be clearly displayed as conditions for using the publicly available code and models. In particular, we will require users to inform those listening to samples synthesized by StyleTTS 2 that they are listening to synthesized speech or to obtain informed consent regarding the use of samples synthesized by StyleTTS 2 in experiments. Users will also be required to use reference speakers who have given consent to have their voice adapted, either directly or by license. Finally, we will make the source code publicly available for further research in speaker fraud and impersonation detection.\n\n- **Resources for Zero-Shot Speaker Adaptation Task** \n\nWe acknowledge your concern regarding the resources utilized in our zero-shot speaker adaptation task. Our intention was to emphasize the reduction in the need for speech data in TTS model training rather than the entire training pipeline. Moreover, since pre-trained models like PL-BERT and WavLM are already publicly available, we do not consider the training data used for these models a part of required training resources. Compared to models like Vall-E that trains on 60k hours of data, our approach can be seen as a resource-efficient method, as we are using 250 times less speech data than Vall-E while achieving similar or better performance. We will revise our paper to make this point more straightforward.\n\n- **Addressing Limitations in Previous Works** \n\nThank you for pointing out this inconsistency in presenting the areas for improvement and our model’s contributions in the introduction. Our model indeed addresses the limitations mentioned in the introduction, but it would be easier to follow if we were more consistent in our wording so that it was clear that our model contributed solutions to these limitations. Using style diffusion, our model can synthesize diverse samples, outperforming models like VITS, as shown in Section 5.2 and Appendix A.2.1. The adversarial training with SLM also improves the performance on out-of-distribution (OOD) texts. Furthermore, our model performs comparably to Vall-E, despite using significantly less speech data, making our model a resource-efficient alternative to models requiring a large amount of speech data like Vall-E. We appreciate your suggestion and we will highlight these contributions more in the conclusion section for consistency.\n\n- **Meaning of Outperforming Human Recordings** \n\nThank you for bringing up the meaning of outperforming human data in the evaluation of naturalness. There are important considerations to make about the way in which speech segments were evaluated. First, the evaluated segments were originally spoken in a larger context. It is possible that a certain speech segment spoken in isolation sounds less natural than the same speech spoken in context. \nHowever, StyleTTS 2 synthesizes segments with a more limited context which may sound more natural in isolation than in a larger context. This would lead to higher ratings for StyleTTS 2 than for real speech, as the lack of larger context in evaluation would favor synthesized speech. Second, human speech naturally contains variability that is neither text nor context-dependent. This variability can lead to short segments of real speech sounding less natural than expected on average. On the other hand, synthesized speech lacks this variability, producing “average” speech given the current text and context, which sounds more natural on average. We will include these discussions in the revised paper to guide future research in this area. \n\n- **Clarification on Two-Stage Training Process** \n\nWe understand the confusion around our two-stage training process. Unlike the original StyleTTS, which fixed the acoustic modules during the second stage, StyleTTS 2 goes through joint training of both acoustic and predictor modules after pre-training. The initial pre-training is primarily aimed at accelerating the training process, not a strict necessity. We found that having a well-trained acoustic module first, particularly the acoustic style encoder as the starting point for the parameters of the prosodic encoder, promotes faster convergence of the prosody predictor. We will clarify this in the camera-ready version.\n\n- **Typos and Other Clarifications**\n\nWe appreciate your attention to detail, and we will correct the notation mistake on line 151. \n\nWe agree that the independence of the generator from the ground truth is common in GAN training. However, in our work, this independence allows us to train on OOD texts without specific speech ground truth, improving performance on OOD texts. We will emphasize more on this point in the revision.\n\nWe also apologize for the ambiguity in Section 5.1. By \"this dataset\", we were referring to the in-distribution samples from the LJSpeech dataset, as our model has outperformed the previous state-of-the-art model NaturalSpeech on this dataset. We will make this more specific in the revised paper.\n"}, {"review_id": "iLUhvyerOW", "reviewer": "Reviewer_4B59", "summary": "In this paper, the paper proposes StyleTTS 2 for human-level text-to-speech, which is an end-to-end system with two-stage training (pretraining some modules and finetuning the whole network). Two main modules proposed in paper is style diffusion module to model the prosodic style information and WavLM adversarial module to improve the quality of synthesized waveforms. Moreover, the paper proposes an interesting module on duration prediction, which is differentiable and stable.", "questions": "1. For the multi-speaker setting or the zero-shot setting, how is the inference details. I think it will be different with Figure 1b since reference audio is used in inference?\n\n2. Will the code be open-sourced to help the future research in TTS?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper proposes several interesting modules, including new differentiable duration modeling method, new adversarial module with WavLM and style diffusion to model the prosodic style information, which can benefit the future research in TTS.\n\n2. The experiments verify the effectiveness and efficiency of proposed method by comparing MOS and RTF with different baseline methods.\n\n3. The ablation study shows the effectiveness of proposed methods.", "weaknesses": "I do not see obvious weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive assessment of our work and constructive feedback on our paper. Here are the responses to your questions.\n\n1. Figure 1(b) represents the single speaker case only. In the multispeaker scenario, our model first encodes the speaker embedding (anchor style) and then samples a style using this anchor. This process is not well-articulated in the current version of the manuscript. We will revise the caption of Figure 1(b) to illustrate this process more clearly in the camera-ready version. \n\n2. We will make the code publicly available to further the research in this field. We will include a link to the code in the camera-ready version of the paper.\n"}, {"review_id": "1RL2zU3yZb", "reviewer": "Reviewer_mqLA", "summary": "This work presents a GAN-based TTS model, as an improvement on its predecessor StyleTTS. Notably, it includes a diffusion-based style encoder, which is one major difference compared to its predecessor. In addition, the proposed model directly produce waveform, in contrast to spectrogram in its predecessor.\n\n", "questions": "1) \"[StyleTTS 2] As the first model to achieve human-level performance on both single and multispeaker datasets\" (line 47) -- Aren't PnG BERT and VITS achieved so as well, as acknowledged in Sec 2? If there is a difference on the scope of the claim, it's important to make it clear here. Also note that PnG BERT reported both MOS and CMOS (line 77).\n\n2) Sec 3.2.1 claims \"E2E training optimizes all TTS system components for inference without relying on any fixed components\", but still \"before jointly optimizing all components, we first pre-train the acoustic modules along with the pitch extractor and text aligner\" -- is this still end-to-end training?\n\n3) Sec 5.2: It's not clear how the evaluation is done. Are the styles analyzed the prosodic style, the acoustic style, or both? Are the emotion labels based on text, or audio (which can diverge)? what's the emotion of the reference audio? what about other factors as \"style\", beside emotion? Figure 2(c) -- \"loose clusters\" -- these are highly overlap to be considered as (even loose) \"clusters\"\n\n4) Table 4: \"real-time factor (RTF) in second\" -- RTF is agnostic to the time unit\n \n5) \"SLM adversarial training\" on \"OOD texts\" -- how does this work together with the regular training? There would be two types of training data, one is text-speech pairs, one is text-only -- how are they coordinated in an end-to-end manner? Can the model benefit from using model text-only data?\n\n6) Evaluation on OOD text -- given the proposed model is trained on the OOD text (with SLM adversarial training), is it still a fair comparison to baseline models? importantly, are the eval text seen during SLM adversarial training?\n \n7) Figure 1(b): where does $s_a$ come from during inference?", "limitations": "Any plan on releasing the source code and the trained checkpoints, especially given the claim on \"StyleTTS 2 sets a new benchmark for TTS synthesis\"?", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The proposed method is sound;\n2) The audio samples sound great with clear improvements over multiple baselines\n3) Abundant evaluations are conducted with persuasive results, including ablation studies", "weaknesses": "1) The proposed model is quite complicated\n2) A few technical questions, see below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive remarks and appreciation of our work, and we appreciate your helpful feedback for clarification. Here are our responses to your insightful questions and concerns.\n\n**1. Claim of Human-Level Performance on Single and Multispeaker Datasets:**\n\nYou are correct in pointing out that the achievements of PnG BERT and VITS are also human-level. However, VITS shows differences from ground truth in CMOS in the Appendix, indicating possible room for improvement. In addition, PnG BERT conducted the evaluation on a proprietary dataset rather than an open-access dataset. We appreciate your attention to detail, and we intend to revise the claim to specify that StyleTTS 2 is the first human-level model on publicly available single and multispeaker benchmark datasets. This revision will help in differentiating our contribution from previous works. \n\n**2. End-to-End Training:**\n\nThe pre-training is only to speed up the training procedure because we found that having a well-trained acoustic model, particularly the style encoder in the first place can help the prosodic predictor converge faster. However, this is not absolutely necessary, and the model still converges when starting directly from the joint training without pre-training. Moreover, even with pre-training, all components are eventually trained together during joint training, so it is still considered end-to-end. We will make this point more clear in the revised version, by stating that this pre-training is not necessary for convergence.\n\n**3. Evaluation and Style Analysis:**\n\nThe styles analyzed are concatenated acoustic and prosodic styles, and the emotion labels are based on texts generated by GPT-4. There is no reference for the single-speaker case shown in Figure 2(a), and the reference for the multispeaker case is neutral emotion (randomly picked from the testing set). This analysis aims to show that our model can sample different styles based on the emotions of the texts. \n\nThere are definitely other aspects of styles that cannot be reflected in the text, though they can be reflected in the reference audio. Previous work (StyleTTS) has already shown that the framework can reproduce many aspects from the reference audio, so this work focuses more on the newly invented style diffusion. We have also provided samples on our demo page to show that our model can reflect several aspects of styles in the reference audio, including the acoustic environment and the speaker’s emotions. \n\n**4. Real-Time Factor (RTF):**\n\nWe acknowledge the oversight in describing RTF, and we will correct this mistake in the camera-ready version.\n\n**5. SLM Adversarial Training on OOD Texts:**\n\nThe training procedure in Figures 1(a) and 1(b) is indeed separate, as you have correctly noticed that their inputs are different, but the gradients are combined for each batch to update the parameters in the end. We will revise the caption under the figure to clarify this process and emphasize that adversarial training in Figure 1(b) is not together with joint training in Figure 1(a) because they have different inputs.\n\n**6. Evaluation on OOD Text:**\n\nThe evaluation data was not seen during SLM adversarial training. In particular, the adversarial training uses texts from *train-clean-460* subset in the LibriTTS dataset, while in our evaluation we use LJSpeech, VCTK, and the *test-clean* subset of LibriTTS, so none of them are in the OOD texts used for training. We will make sure this is further emphasized in Section 4 Model Training.\n\n**7. Origin of $s_a$ in Figure 1b:**\n\nWe apologize for the confusion in Figure 1b. The style diffusion model samples both $s_a$ and $s_p$, not just $s_p$. We will revise the figure and make the border around $s_a$ and $s_p$ darker in color to avoid any confusion. \n\n***\nLastly, we acknowledge that our method contains multiple components, but we hope that our paper has described each component sufficiently clearly. In addition, we do plan to make the source code publicly available to make sure all components in our method are completely transparent. We will add a link to the source code and checkpoints in the camera-ready version. \n"}]

Author rebuttal:
We thank all the reviewers for their time and effort in reviewing and improving our work. Here are some common concerns we would like to address: 

***

**Source Code and Checkpoints**

We will make the trained checkpoints and source code for training and inference publicly available. We will add a link to the GitHub repository in the camera-ready version that contains the source code and checkpoints. To prevent potential misuse for deception, we will release the model under a code of conduct that the listeners of the samples should be informed that the speech they are listening to is synthesized by StyleTTS 2, and the reference speakers should have given consent either directly or by license to have their voice adapted by StyleTTS 2. 

***

**Figure 1 Clarification**

We have noticed several confusions caused by unclarity in our presentation in Figure 1, and we will make the following changes:

- We will clarify that Figure 1 illustrates two separate training processes in Figure 1(a) and 1(b), with different inputs but combined gradients for each batch. 

- We will make it clear that Figure 1(b) only describes the inference pipeline for single-speaker models, and we will add additional descriptions for multispeaker cases involving reference speaker embeddings. 

- We will clarify that the joint training process described in Figure 1(a) is end-to-end with all components being trained together. We will emphasize that the pre-training described in Figure 1(a) is for faster convergence only, not an absolute necessity. 

***

**Ethics Review Rebuttal**

We appreciate the ethics reviewer's careful evaluation of any ethical issues in our work. Here is our rebuttal to the ethics review and our plan to address these concerns.

**1. IRB Details**:

We thank the ethics reviewer for noting that we omitted details about the IRB approval of the protocol used to obtain speech evaluations from human participants. We will make revisions to include further details about the human study protocol.

Previous:

> “These evaluations were conducted by native English speakers from the U.S. on Amazon Mechanical Turk.”

Updated: 

>“These evaluations were conducted by native English speakers from the U.S. on Amazon Mechanical Turk. All evaluators reported normal hearing and provided informed consent as monitored by the local institutional review board and in accordance with the ethical standards of the Declaration of Helsinki.”

We will further disclose the IRB protocol number in the camera-ready version as proof. 

**2. Potential for Deception and Ways for Mitigation:**

We appreciate the importance of the potential for deception this model creates. We will address this issue more wholistically by noting further impacts of possible deception and our specific plans for mitigating possible negative consequences of the release of this model. Specifically, we will expand our discussion in Section 6 with the following paragraph:

>We acknowledge that zero-shot speaker adaptation has the potential for misuse and deception by mimicking the voices of individuals as a potential source of misinformation or disinformation. This could lead to harmful, deceptive interactions such as theft, fraud, harassment, or impersonations of public figures that may influence political processes or trust in institutions. In order to manage the potential for harm, we will require users of our model to adhere to a code of conduct that will be clearly displayed as conditions for using the publicly available code and models. In particular, we will require users to inform those listening to samples synthesized by StyleTTS 2 that they are listening to synthesized speech or to obtain informed consent regarding the use of samples synthesized by StyleTTS 2 in experiments. Users will also be required to use reference speakers who have given consent to have their voice adapted, either directly or by license. Finally, we will make the source code publicly available for further research in speaker fraud and impersonation detection.

**3. Meaning and Implications of Outperforming Human Data**

We thank the ethics reviewer for bringing up the meaning of outperforming human data on an evaluation of naturalness. There are important considerations to make about the way in which speech segments were evaluated. First, the evaluated segments were originally spoken in a larger context. It is possible that a certain speech segment spoken in isolation sounds less natural than the same speech spoken in context. However, StyleTTS 2 synthesizes segments with a more limited context which may sound more natural in isolation than in a larger context. This would lead to higher ratings for StyleTTS 2 than for real speech, as the lack of larger context in evaluation would favor synthesized speech. Second, human speech naturally contains variability that is neither text- nor context-dependent. This variability can lead to short segments of real speech sounding less natural than expected on average. On the other hand, synthesized speech lacks this variability, instead producing “average” speech given the current text and context, which sounds more natural on average.

Given that we can synthesize speech barely distinguishable from real speech, it raises the question of how real speech could be reasonably detected. Since we believe StyleTTS 2 outperforms human data mainly due to context and variability, we believe these characteristics could also be used for detecting synthesized speech. In particular, shorter segments may not be detected accurately, as they lack context and only represent a small amount of the possible variation in the speaker’s voice. On the other hand, longer segments may be detected more accurately, as real speech will be more influenced by long-term context and contain more “human” sources of variability, such as pauses and breaths. We will include these discussions in the revised paper to guide future research in this area. 

Meta-review: This paper introduces a high-quality TTS model StyleTTS 2, which has a DDPM-based style generator, speech language model discriminators, and a differentiable aligner. Reviewers are all positive about this paper. Some reviewers also have concerns about the complex training procedure of the model. Suggest the author validate all the components to verify whether they are unnecessarily complicated. Overall, it is a good paper, and I recommend to accept.
Decision: Accept (poster)