SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(c) LibriTTS
Figure 7: Layer-wise input weight magnitude to the SLM discriminators across different datasets. The layer importance shows a divergent pattern for the VCTK model relative to the LJSpeech and LibriTTS models, showcasing the impact of contextual absence on the SLM discriminators.
crucial. Layers 5, 6, and 7, encoding semantic aspects like word identity and meaning [55], held secondary importance. This indicates that the SLM discriminator learns to fuse acoustic and semantic information to derive paralinguistic attributes, such as prosody, pauses, intonations, and emotions, which are critical in distinguishing real and synthesized speech. The outcome reaffirms the SLM discriminator's capacity to enhance the emotional expressiveness and prosody of synthesized speech.
In contrast, the SLM discriminator for the VCTK dataset demonstrated no distinct layer preference, likely due to its limited contextual information. Unlike the storytelling and audiobook narrations in the LJSpeech and LibriTTS datasets, the VCTK dataset only involves reading a standard paragraph devoid of specific context or emotions. This context shortage could explain the marginal performance increase of our model over VITS on the VCTK dataset compared to LJSpeech and LibriTTS datasets (see Tables 1, 2, 3), as the advantages of style diffusion and SLM adversarial training in our model are less pronounced in datasets with restricted expressiveness and emotional variety.
D.2 Training Stability
Figure 6(b) shows that the maximum norm gradient from the SLM discriminator to the duration predictor can reach up to 200 even with  $\sigma = 1.5$ , potentially destabilizing training, given that the gradient norms for other modules are generally less than 10. This instability is particularly concerning when the gradient is back-propagated to the prosodic text encoder, which employs a BERT model that is notably sensitive to gradient explosion. To mitigate this issue, we adopt a strategy for scaling the gradient from the SLM discriminator to the duration predictor, a common practice to prevent gradient explosion in GAN training [31].
We found that the gradient norm of the entire predictor typically falls between 10 and 20. As such, we implement a scaling factor of 0.2 to the gradient norm when it surpasses 20. Additionally, we also account for potential instabilities arising from the sigmoid function used by the duration predictor for probability output. We mitigate these by scaling the gradient to the last projection layer and the LSTM layer within the duration predictor by a factor of 0.01. These scaling values have consistently shown stability across different datasets and can be modified to accommodate dataset-specific attributes. All the models presented in this paper were trained with these scaling factors.
Appendix E Subjective Evaluation Procedures
In this section, we propose a detailed and standardized evaluation procedure and elucidate our adherence to these guidelines in the current work.
E.1 Proposed Framework for Subjective Evaluation
In light of the findings in [43] along with guidelines from [5], we recommend the following procedures for assessing human-like text-to-speech models:
1. Raters should be native speakers of the language used in the model's training corpus, with provisions in place to exclude falsely claimed native speakers.
2. Attention checks should be incorporated into the evaluation to ensure thoughtful, careful completion, and raters who do not pass the attention checks should be excluded from the outcome analysis.
3. The objectives of the evaluation should be clearly stated. For instance, when assessing naturalness, its definition should be explicit within the evaluation framework.
4. Ideally, all models used in the experiments should be publicly available, and official implementations are preferred over unofficial implementations. If no official implementations are publicly available, the testing samples can be requested from the authors of the papers, or from implementations that have been employed in other papers.
5. For model comparisons, the MUSHRA-based approach, where paired samples from all models are presented, should be employed, in contrast to the traditional MOS evaluation scheme that presents model samples in isolation.
6. For nuanced model comparison and determining statistical significance, CMOS should be used in place of the MOS and MUSHRA evaluations. Each sample should be rated by at least 20 raters with more than 50 samples to claim statistical significance compared to ground truth to assess human-level quality [5].
E.2 Evaluation Details
The first criterion was ensured by selecting native English-speaking participants residing in the United States. Upon launching our survey on MTurk, we activated the following filters:
- HIT Approval Rate (\%) for all Requesters' HITS: greater than 95.
- Location: is UNITED STATES (US).
- Number of HITs Approved: greater than 50.
For each batch, completed responses were collected exclusively from self-identified native speakers in the United States, verified by residential IP addresses (i.e., not proxy or VPN) with an online service  $^{5}$ .
To fulfill the second criterion, attention checks were implemented differently for MOS and CMOS evaluations. In the MOS assessment, we utilized the average score given by a participant to ground truth audios, unbeknownst to the participants, to ascertain their attentiveness. We excluded ratings from those whose average score for the ground truth did not rank in the top three among all five models. In the CMOS evaluation, we assessed the consistency of the rater's scoring; if the score's
Available at https://www.ipqualityscore.com/free-ip-lookup-proxy-vpn-test
sign (indicating whether A was better or worse than B) differed for over half the sample set (in our case, 10 samples), the rater was disqualified. Six raters were eliminated through this process in all of our experiments.
The third criterion was ensured by providing explicit definitions of "naturalness" and "similarity" within our surveys, exemplified by the following lines (see our survey links  $^{678}$  for more detail):
- Naturalness:
Some of them may be synthesized while others may be spoken by an American audiobook narrator.
Rate how natural each audio clip sounds on a scale of 1 to 5 with 1 indicating completely unnatural speech (bad) and 5 completely natural speech (excellent).
Here, naturalness includes whether you feel the speech is spoken by a native American English speaker from a human source.
- Similarity:
Rate whether the two audio clips could have been produced by the same speaker or not on a scale of 1 to 5 with 1 indicating completely different speakers and 5 indicating exactly the same speaker.
Some samples may sound somewhat degraded/distorted; for this question, please try to listen beyond the distortion of the speech and concentrate on identifying the voice (including the person's accent and speaking habits, if possible).
In accordance with the fourth criterion, official model checkpoints were utilized in both MOS and CMOS evaluations. Specifically, for the MOS evaluation of LJSpeech, checkpoints for VITS<sup>9</sup>, JETS<sup>10</sup>, and StyleTTS with HifiGAN vocoder<sup>11</sup> were sourced directly from the authors. Likewise, for the MOS evaluation on LibriTTS, official checkpoints for StyleTTS with the HifiGAN vocoder and YourTTS (utilizing the Exp 4 checkpoint)<sup>12</sup> were used. Since there is no official checkpoint of VITS for LibriTTS, we used the VITS model from ESPNet<sup>13</sup> on LibriTTS, which was previously employed as a baseline model in [6]. For the CMOS experiment on VCTK with VITS, we used official checkpoints and implementation. For the CMOS evaluation with NaturalSpeech, we obtained 20 demo samples directly from the author. Regarding Vall-E's CMOS experiment, we collected 30 samples from LibriTTS, along with their corresponding 3-second prompts from the official demo page<sup>14</sup>.
Fulfilling the fifth criterion, we employed a MUSHRA-based approach in our MOS evaluations. Lastly, we assured that 20 raters evaluated each sample in our CMOS experiments, after excluding ineligible raters. For NaturalSpeech and Vall-E, where additional samples were not available, we compensated by doubling the number of raters.
Appendix F Detailed Model Architectures
This section offers a comprehensive outline of the enhanced StyleTTS 2 architecture. The model integrates eight original StyleTTS modules with the addition of an extra style diffusion denoiser, prosodic style encoder, and prosodic text encoder.
$^{6}$ LJSpeech MOS: https://survey.chemist.com/s3/7376864/MOS-MIX-ID-00D-B1
LJSpeech CMOS: https://survey.alchemer.com/s3/7332116/CMOS-gt-stylets2-0430-b4
LibriTTS MOS: https://survey.alchemer.com/s3/7387883/MOS-LibriTTS-0508-b2
<sup>9</sup>https://github.com/jaywalnut310/vits
10 https://huggingface.co/imdanboy/jets
11https://github.com/yl4579/StyleTTS
<sup>12</sup>https://github.com/Edresson/YourTTS
<sup>13</sup>Available at https://huggingface.co/espnet/kan-bayashi_libritts_xvector_vits
<sup>14</sup>https://www.microsoft.com/en-us/research/project/vall-e-x/vall-e/
The model keeps the same architecture for the acoustic text encoder, text aligner, and pitch extractor as in the original StyleTTS [6]. The architecture of both the acoustic style encoder and prosodic style encoder follows that of the style encoder from StyleTTS [6]. The prosodic text encoder is a pre-trained PL-BERT $^{15}$  model [7]. Additionally, we adopt the same discriminators, MPD and MRD, as specified in [31]. The decoder is a combination of the original decoder from StyleTTS and either iSTFTNet [45] or HifiGAN [28], with the AdaIN [44] module appended after each activation function. The duration and prosody predictors follow the same architecture as in [6], albeit with alterations to the output of the duration predictor changing from  $1 \times N$  to  $L \times N$  for probability output modeling. Thus, this section focuses primarily on providing comprehensive outlines for the style diffusion denoiser and the discriminative head of the SLM discriminators. Readers seeking detailed specifications for other modules can refer to the aforementioned references.
The Denoiser (Table 8) is designed to process an input style vector  $s$ , the noise level  $\sigma$ , the phoneme embeddings from the prosodic text encoder  $h_{\mathrm{bert}}$  and, when applicable, a speaker embedding  $c$  from the acoustic and prosodic encoders in the multi-speaker scenario. Here,  $c$  is also considered a part of the input and is introduced to the denoiser through adaptive layer normalization (AdaLN):
$$
\operatorname {A d a L N} (x, s) = L _ {\sigma} (s) \frac {x - \mu (x)}{\sigma (x)} + L _ {\mu} (s) \tag {27}
$$
where  $x$  is the feature maps from the output of the previous layer,  $s$  is the style vector,  $\mu(\cdot)$  and  $\sigma(\cdot)$  denotes the layer mean and standard deviation, and  $L_{\sigma}$  and  $L_{\mu}$  are learned linear projections for computing the adaptive gain and bias using the style vector  $s$ .
Table 8: Denoiser architecture.  $N$  represents the input phoneme length of the mel-spectrogram,  $\sigma$  is the noise level,  $h_{\mathrm{bert}}$  is the phoneme embeddings, and  $c$  is the speaker embedding. The size of  $s$  and  $c$  is  $256\times 1$ , that of  $h_{\mathrm{bert}}$  is  $768\times N$ , and  $\sigma$  is  $1\times 1$ . Group(32) is the group normalization with a group size of 32.
Submodule	Input	Layer	Norm	Output Shape
Embedding	σ	Sinusoidal Embedding	-	256 × 1
c	Addition	-	256 × 1
-	Repeat for N times	-	256 × N
-	Linear 256 × 1024	-	1024 × N
-	Output Embedding k	-	1024 × N
Input	s	Repeat for N Times	-	256 × N
hbert	Concat	Group(32)	1024 × N
-	Conv 1024 × 1024	-	1024 × N
Transformer Block (×3)	k	Addition	-	1024 × N
c	8-Head Self-Attention (64 head features)	AdaLN (·, c)	1024 × N
-	Linear 1024 × 2048	-	2048 × N
-	GELU	-	2048 × N
-	Linear 2048 × 1024	-	1024 × N
Output	-	Adaptive Average Pool	-	1024 × 1
-	Linear 1024 × 256	-	256 × 1
The Discriminative Head (Table 9) is comprised of a 3-layer convolutional neural network, concluding with a linear projection.
Table 9: Discriminative head architecture.  $T$  represents the number of frames (length) of the output feature  ${h}_{\text{slm }}$  from WavLM.
Layer	Norm	Output Shape
Input hslm	-	13×768×T
Reshape	-	9984×T
Linear 9984×256	-	256×T
Conv 256×256	-	256×T
Leaky ReLU (0.2)	-	256×T
Conv 256×512	-	512×T
Leaky ReLU (0.2)	-	512×T
Conv 512×512	-	512×T
Leaky ReLU (0.2)	-	512×T
Conv 512×1	-	1×T
15Available at https://github.com/yl4579/PL-BERT
Appendix G Detailed Training Objectives
In this section, we provide detailed training objectives for both acoustic modules pre-training and joint training. We first pre-train the acoustic modules for accelerated training, and we then proceed with joint training with the pitch extractor fixed.
G.1 Acoustic module pre-training
Mel-spectrogram reconstruction. The decoder is trained on waveform  $\pmb{y}$ , its corresponding mel-spectrogram  $\pmb{x}$ , and the text  $\pmb{t}$ , using  $L_{1}$  reconstruction loss as
$$
\mathcal {L} _ {\mathrm {m e l}} = \mathbb {E} _ {\boldsymbol {x}, t} \left[ \| \boldsymbol {x} - M \left(G \left(\boldsymbol {h} _ {\text {t e x t}} \cdot \boldsymbol {a} _ {\text {a l g n}}, \boldsymbol {s} _ {a}, p _ {\boldsymbol {x}}, n _ {\boldsymbol {x}}\right)\right) \| _ {1} \right]. \tag {28}
$$
Here,  $h_{\mathrm{text}} = T(t)$  is the encoded phoneme representation, and the attention alignment is denoted by  $\pmb{a}_{\mathrm{algn}} = A(\pmb{x}, t)$ . The acoustic style vector of  $\pmb{x}$  is represented by  $s_a = E_a(\pmb{x})$ ,  $p_x$  symbolizes the pitch F0 and  $n_x$  indicates energy of  $x$ , and  $M(\cdot)$  represents mel-spectrogram transformation. Following [6], half of the time, raw attention output from  $A$  is used as alignment, allowing backpropagation through the text aligner. For another 50% of the time, a monotonic version of  $\pmb{a}_{\mathrm{algn}}$  is utilized via dynamic programming algorithms (see Appendix A in [6]).
TMA objectives. We follow [6] and use the original sequence-to-sequence ASR loss function  $\mathcal{L}_{\mathrm{s2s}}$  to fine-tune the pre-trained text aligner, preserving the attention alignment during end-to-end training:
$$
\mathcal {L} _ {\mathrm {s} 2 \mathrm {s}} = \mathbb {E} _ {\boldsymbol {x}, t} \left[ \sum_ {i = 1} ^ {N} \mathbf {C E} \left(\boldsymbol {t} _ {i}, \hat {\boldsymbol {t}} _ {i}\right) \right], \tag {29}
$$
where  $N$  is the number of phonemes in  $\pmb{t}$ ,  $\pmb{t}_i$  is the  $i$ -th phoneme token of  $\pmb{t}$ ,  $\hat{\pmb{t}}_i$  is the  $i$ -th predicted phoneme token, and  $\mathbf{CE}(\cdot)$  denotes the cross-entropy loss function.
Additionally, we apply the monotonic loss  $\mathcal{L}_{\mathrm{mono}}$  to ensure that soft attention approximates its non-differentiable monotonic version:
$$
\mathcal {L} _ {\mathrm {m o n o}} = \mathbb {E} _ {\boldsymbol {x}, t} \left[ \| \boldsymbol {a} _ {\text {a l g n}} - \boldsymbol {a} _ {\text {h a r d}} \| _ {1} \right], \tag {30}
$$
where  $\pmb{a}_{\mathrm{hard}}$  is the monotonic version of  $\pmb{a}_{\mathrm{algn}}$  obtained through dynamic programming algorithms (see Appendix A in [6] for more details).
Adversarial objectives. Two adversarial loss functions, originally used in HifiGAN [30], are employed to enhance the sound quality of the reconstructed waveforms: the LSGAN loss function  $\mathcal{L}_{\mathrm{adv}}$  for adversarial training and the feature-matching loss  $\mathcal{L}_{\mathrm{fm}}$ .
$$
\mathcal {L} _ {\mathrm {a d v}} (G; D) = \mathbb {E} _ {\boldsymbol {t}, \boldsymbol {x}} \left[ \left(D \left(\left(G \left(\boldsymbol {h} _ {\text {t e x t}} \cdot \boldsymbol {a} _ {\text {a l g n}}, \boldsymbol {s} _ {a}, p _ {\boldsymbol {x}}, n _ {\boldsymbol {x}}\right)\right)\right) - 1\right) ^ {2} \right], \tag {31}
$$
$$
\mathcal {L} _ {\mathrm {a d v}} (D; G) = \mathbb {E} _ {\boldsymbol {t}, \boldsymbol {x}} \left[ \right.\left( \right.D \left(\left(G \left(\left(h _ {\text {t e x t}} \cdot \boldsymbol {a} _ {\text {a l g n}}, \boldsymbol {s} _ {a}, p _ {\boldsymbol {x}}, n _ {\boldsymbol {x}}\right)\right)\right)\right) ^ {2} \left. \right] + \tag {32}
$$
$$
\mathbb {E} _ {\boldsymbol {y}} \left[ (D (\boldsymbol {y}) - 1) ^ {2} \right],
$$
$$
\mathcal {L} _ {\mathrm {f m}} = \mathbb {E} _ {\boldsymbol {y}, \boldsymbol {t}, \boldsymbol {x}} \left[ \sum_ {l = 1} ^ {\Lambda} \frac {1}{N _ {l}} \left\| D ^ {l} (\boldsymbol {y}) - D ^ {l} \left(G \left(\boldsymbol {h} _ {\text {t e x t}} \cdot \boldsymbol {a} _ {\text {a l g n}}, \boldsymbol {s} _ {a}, p _ {\boldsymbol {x}}, n _ {\boldsymbol {x}}\right)\right) \right\| _ {1} \right], \tag {33}
$$
where  $D$  represents both MPD and MRD,  $\Lambda$  is the total number of layers in  $D$ , and  $D^l$  denotes the output feature map of  $l$ -th layer with  $N_{l}$  number of features.
In addition, we included the truncated pointwise relativistic loss function [49] to further improve the sound quality:
$$
\mathcal {L} _ {\mathrm {r e l}} (G; D) = \mathbb {E} _ {\{\hat {\boldsymbol {y}}, \boldsymbol {y} \mid D (\hat {\boldsymbol {y}}) \leq D (\boldsymbol {y}) + m _ {G D} \}} \left[ \tau - \operatorname {R e L U} \left(\tau - \left(D (\hat {\boldsymbol {y}}) - D (\boldsymbol {y}) - m _ {G D}\right) ^ {2}\right) \right], \tag {34}
$$
$$
\mathcal {L} _ {\mathrm {r e l}} (D; G) = \mathbb {E} _ {\{\hat {\boldsymbol {y}}, \boldsymbol {y} \mid D (\boldsymbol {y}) \leq D (\hat {\boldsymbol {y}}) + m _ {D G} \}} \left[ \tau - \operatorname {R e L U} \left(\tau - \left(D (\boldsymbol {y}) - D (\hat {\boldsymbol {y}}) - m _ {D G}\right) ^ {2}\right) \right], \tag {35}
$$
where  $\hat{\pmb{y}} = G(\pmb{h}_{\mathrm{text}}\cdot \pmb{a}_{\mathrm{algn}},\pmb{s}_a,\pmb{p_y},n_{\pmb{y}})$  is the generated sample,  $\mathrm{ReLU}(\cdot)$  is the rectified linear unit function,  $\tau$  is the truncation factor that is set to be 0.04 per [49],  $m_{GD}$  and  $m_{DG}$  are margin parameters defined by the median of the score difference:
$$
m _ {G D} = \mathbb {M} _ {\boldsymbol {y}, \hat {\boldsymbol {y}}} [ D (\hat {\boldsymbol {y}}) - D (\boldsymbol {y}) ], \tag {36}
$$
$$
m _ {D G} = \mathbb {M} _ {\boldsymbol {y}, \hat {\boldsymbol {y}}} [ D (\boldsymbol {y}) - D (\hat {\boldsymbol {y}}) ], \tag {37}
$$
where  $\mathbb{M}[\cdot ]$  is the median operation.
Acoustic module pre-training full objectives. Our full objective functions in the acoustic modules pre-training can be summarized as follows with hyperparameters  $\lambda_{\mathrm{s2s}}$  and  $\lambda_{\mathrm{mono}}$ :
$$
\begin{array}{l} \min  _ {G, A, E _ {a}, F, T} \mathcal {L} _ {\mathrm {m e l}} + \lambda_ {\mathrm {s} 2 \mathrm {s}} \mathcal {L} _ {\mathrm {s} 2 \mathrm {s}} + \lambda_ {\mathrm {m o n o}} \mathcal {L} _ {\mathrm {m o n o}} \\ + \mathcal {L} _ {\mathrm {a d v}} (G; D) + \mathcal {L} _ {\mathrm {r e l}} (G; D) + \mathcal {L} _ {\mathrm {f m}} \\ \end{array}
$$
$$
\begin{array}{l} \min  _ {D} \mathcal {L} _ {\mathrm {a d v}} (D; G) \tag {39} \\ + \mathcal {L} _ {\text {r e l}} (D; G) \\ \end{array}
$$
Following [6], we set  $\lambda_{\mathrm{s2s}} = 0.2$  and  $\lambda_{\mathrm{mono}} = 5$ .
G.2 Joint training
Duration prediction. We use the following cross-entropy loss to train the duration predictor:
$$
\mathcal {L} _ {\mathrm {c e}} = \mathbb {E} _ {\boldsymbol {d} _ {\mathrm {g t}}} \left[ \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {L} \mathbf {C E} (q [ k, i ], \mathbb {I} (\boldsymbol {d} _ {\mathrm {g t}} [ i ] \geq k)) \right], \tag {40}
$$
where  $\mathbb{I}(\cdot)$  is the indicator function,  $q = S(\pmb{h}_{\mathrm{bert}},\pmb{s}_p)$  is the output of the predictor under the prosodic style vector  $\pmb{s}_p = E_p(\pmb{x})$  and prosodic text embeddings  $\pmb{h}_{\mathrm{bert}} = B(\pmb{t})$ ,  $N$  is the number of phonemes in  $\pmb{t}$ , and  $\pmb{d}_{\mathrm{gt}}$  is the ground truth duration obtained by summing  $\pmb{a}_{\mathrm{algn}}$  along the mel frame axis.
Additionally, we employ the  $L-1$  loss to make sure the approximated duration is also optimized:
$$
\mathcal {L} _ {\mathrm {d u r}} = \mathbb {E} _ {\boldsymbol {d} _ {\mathrm {g t}}} \left[ \left\| \boldsymbol {d} _ {\mathrm {g t}} - \boldsymbol {d} _ {\mathrm {p r e d}} \right\| _ {1} \right], \tag {41}
$$
where  $\pmb{d}_{\mathrm{pred}}[i] = \sum_{k=1}^{L} q[k, i]$  is the approximated predicted duration of the  $i^{\mathrm{th}}$  phoneme.
Prosody prediction. We use  $\mathcal{L}_{f_0}$  and  $\mathcal{L}_n$ , which are F0 and energy reconstruction loss, respectively:
$$
\mathcal {L} _ {f 0} = \mathbb {E} _ {\boldsymbol {x}} \left[ \left\| p _ {\boldsymbol {x}} - \hat {p} _ {\boldsymbol {x}} \right\| _ {1} \right] \tag {42}
$$
$$
\mathcal {L} _ {n} = \mathbb {E} _ {\boldsymbol {x}} \left[ \left\| n _ {\boldsymbol {x}} - \hat {n} _ {\boldsymbol {x}} \right\| _ {1} \right] \tag {43}
$$
where  $\hat{p}_{\boldsymbol{x}}, \hat{n}_{\boldsymbol{x}} = P(h_{\mathrm{bert}}, s_p)$  are the predicted pitch and energy of  $\boldsymbol{x}$ .
Mel-spectrogram reconstruction. During joint training, we modify  $\mathcal{L}_{\mathrm{mel}}$  as follows:
$$
\mathcal {L} _ {\mathrm {m e l}} = \mathbb {E} _ {\boldsymbol {x}, t} \left[ \| \boldsymbol {x} - M \left(G \left(\boldsymbol {h} _ {\text {t e x t}} \cdot \boldsymbol {a} _ {\text {a l g n}}, \boldsymbol {s} _ {a}, \hat {p} _ {\boldsymbol {x}}, \hat {n} _ {\boldsymbol {x}}\right)\right) \| _ {1} \right]. \tag {44}
$$
Now we use the predicted pitch  $\hat{p}_{\boldsymbol{x}}$  and energy  $\hat{n}_{\boldsymbol{x}}$  to reconstruct the speech. We also make similar changes to  $\mathcal{L}_{\mathrm{adv}}$ ,  $\mathcal{L}_{\mathrm{fm}}$ , and  $\mathcal{L}_{\mathrm{rel}}$  during joint training.
SLM adversarial objective. The SLM discriminator can easily overpower the generator and disable the gradient flow to the generator. To facilitate learning when the discriminator overpowers the generator, we use the LSGAN loss [48] instead of the min-max loss in eq. 5:
$$
\mathcal {L} _ {\mathrm {s l m}} (G; D) = \mathbb {E} _ {\boldsymbol {t}, \boldsymbol {x}} \left[ \left(D _ {S L M} \left(\left(G \left(\boldsymbol {h} _ {\text {t e x t}} \cdot \boldsymbol {a} _ {\text {p r e d}}, \hat {\boldsymbol {s}} _ {a}, p _ {\hat {\boldsymbol {x}}}, n _ {\hat {\boldsymbol {x}}}\right)\right)\right) - 1\right) ^ {2} \right], \tag {45}
$$
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {s l m}} (D; G) = \mathbb {E} _ {\boldsymbol {t}, \boldsymbol {x}} \left[ \right.\left( \right.D _ {S L M} \left(\left(G \left(\boldsymbol {h} _ {\text {t e x t}} \cdot \boldsymbol {a} _ {\text {p r e d}}, \hat {\boldsymbol {s}} _ {a}, p _ {\hat {\boldsymbol {x}}}, n _ {\hat {\boldsymbol {x}}}\right)\right)\right) ^ {2} \left. \right] + \tag {46} \\ \mathbb {E} _ {\boldsymbol {y}} \left[ \left(D _ {S L M} (\boldsymbol {y}) - 1\right) ^ {2} \right], \\ \end{array}
$$
where  $\hat{s}_a$  is the acoustic style sampled from style diffusion,  $a_{\mathrm{pred}}$  is the predicted alignment obtained through the differentiable duration upsampler from the predicted duration  $d_{\mathrm{pred}}$ , and  $p_{\hat{x}}$  and  $n_{\hat{x}}$  are the predicted pitch and energy with the sampled prosodic style  $\hat{s}_p$ .
Joint training full objectives. Our full objective functions in joint training can be summarized as follows with hyperparameters  $\lambda_{\mathrm{dur}}$ ,  $\lambda_{\mathrm{ce}}$ ,  $\lambda_{f0}$ ,  $\lambda_{n}$ ,  $\lambda_{\mathrm{s2s}}$ , and  $\lambda_{\mathrm{mono}}$ :
$$
\begin{array}{l} \min  _ {G, A, E _ {a}, E _ {p}, T, B, V, S, P} \mathcal {L} _ {\text {m e l}} + \lambda_ {\text {c e}} \mathcal {L} _ {\text {c e}} + \lambda_ {\text {d u r}} \mathcal {L} _ {\text {d u r}} + \lambda_ {f 0} \mathcal {L} _ {f 0} + \lambda_ {n} \mathcal {L} _ {n} \\ + \lambda_ {\mathrm {s} 2 \mathrm {s}} \mathcal {L} _ {\mathrm {s} 2 \mathrm {s}} + \lambda_ {\mathrm {m o n o}} \mathcal {L} _ {\mathrm {m o n o}} + \mathcal {L} _ {\mathrm {a d v}} (G; D) \tag {47} \\ + \mathcal {L} _ {\mathrm {r e l}} (G; D) + \mathcal {L} _ {\mathrm {f m}} + \mathcal {L} _ {s l m} (G; D) + \mathcal {L} _ {\mathrm {e d m}}, \\ \end{array}
$$
where  $\mathcal{L}_{\mathrm{edm}}$  is defined in eq. 3.
The discriminator loss is given by: