DPP-TTS: Diversifying prosodic features of speech via determinantal point processes
Authors: Seongho Joo
Hyukhun Koh
Kyomin Jung
Venue: EMNLP 2023 Main (2023)
Keywords: Speech prosody; prosodic segmentation; text-to-speech
TL;DR: We propose DPP-TTS, a text-to-speech model that boosts the diveristy of speech prosody.

Abstract: With the rapid advancement in deep generative models, recent neural Text-To-Speech(TTS) models have succeeded in synthesizing human-like speech. There have been some efforts to generate speech with various prosody beyond monotonous prosody patterns.
However, previous works have several limitations. First, typical TTS models depend on the scaled sampling temperature for boosting the diversity of prosody. Speech samples generated at high sampling temperatures often lack perceptual prosodic diversity, which can adversely affect the naturalness of the speech. Second, the diversity among samples is neglected since the sampling procedure often focuses on a single speech sample rather than multiple ones. In this paper, we propose DPP-TTS: a text-to-speech model based on Determinantal Point Processes (DPPs) with a prosody diversifying module. Our TTS model is capable of generating speech samples that simultaneously consider perceptual diversity in each sample and among multiple samples. We demonstrate that DPP-TTS generates speech samples with more diversified prosody than baselines in the side-by-side comparison test considering the naturalness of speech at the same time.

DPP-TTS: Diversifying prosodic features of speech via determinantal point processes
Seongho Joo
Hyukhun Koh Seoul National University hyukhunkoh-ai@snuu ac kr
Kyomin Jung' Seoul National University
Seoul National University seonghojoo@snu.ac kr
kjung@snu.ac ac. kr
Abstract
Prosody Space
With the rapid advancement deep generative models, recent neural Tex--To-Speech (TTS) models have succeeded in synthesizing human- like speech. There have been some efforts to generate speech with various prosody beyond monotonous prosody patterns. However, previ- ous works have several limitations. First, typ- ical TTS models depend on the scaled sam- pling temperature for boosting the diversity of prosody. Speech samples generated high sampling temperatures often lack perceptual prosodic diversity, thereby hampering the nat- uralness of the speech. Second, the diversity among samples is neglected since the sampling procedure often focuses on single speech sam- ple rather than multiple ones. In this paper, we propose DPP-TTS: a text-to-specch model based on Determinantal Point Processes (DPPs) with a new objective function and prosody di- versifying module. Our TTS model is capable of generating speech samples that simultane- ously consider perceptual diversity in each sam- ple and among multiple samples. We demon- strate that DPP-TTS generates speech samples with more diversified prosody than baselines in the side-br by-de comparison test considering the naturalness of speech the same time.
A..
POM
Prosedy predictor
WW in fow wests, the maniim crobodrnclly Inyphca made Piiner,
Input
Sapping
Figure 1: Overview of our DPP-TTS model. Along with the input text segmentation, our Prosody Diversifying Module(PDM) generates more diverse and smooth prosody pattern compared to the baseline.
rhythm beyond the fidelity of speech. To reflect such acoustic features on generated speech, there have been many attempts to synthesize speech with rich and diverse prosodic patterns. One of the widely used approaches for prosody modeling is to exploit generative models like VAEs and flow mod- els (Hsu et al., 2019; Lee al., 2021; Ren et al., 2021b; Valle et al., 2021; Vallés-Pérez, al., 2021). These generative TTS models control the extent of variation speech by sampling prior distribu- tion with adequate temperatures. In other works, auxiliary features such as syntax information and text semantics from BERT embeddings are used to enhance the prosody of speech (Ye et al., 2022; Xu et al., 2020).
1 Introduction
In the past few years, Text-To-Speech (TTS) mod- els have made a lot of progress in synthesizing human-like speech (Li et al., 2019; Ping et al., 2018; Ren et al.. 2019; Shen al., 2018; Kim al., 2021). Furthermore, the latest studies, sev- eral TTS models made high-quality speech samples even in the end-to-end setting without a two-stage synthesis process (Donahue et al., 2021; Kim et al., 2021). Based on these technical developments, TTS models are now able to generate high-fidelity speech.
However, previous approaches are subject to sev- eral limitations. First, sampling with high tempera- tures for generating diverse prosody patterns often severely degrades the naturalness of speech. In the experiment section, we show that previous TTS models often fail to generate diverse and smooth speech samples to the listeners in diverse sampling temperature settings. Second, previous works for prosody modeling treat each sample independently. thereby not guarantecing the diversity among sepa- rately generated samples.
Meanwhile, human speech contains diverse prosody patterns regarding intonation, stress, and
Corresponding author

---

In this paper, for generating diverse speech sam- ples, we resolve the Iwo aforementioned limita- tions by adopting Determinantal point processes (DPPs), which have typically shown great results in modeling diversity among multiple samples in various machine learning tasks such as text sum- marization (Cho et al., 2019) and recommendation systems (Gartrell et al., 2021). We devise novel TTS model equipped with the new objective func- tion and prosody diversifying module based on DPPs. Through our adaptive MTC objective func- tion and DPP kernel, we can effectively generate speech samples with diverse and natural prosody. However, with the standard DPP, it is challeng- ing to model fine-grained level of prosody for non-monotonous speech. For more sophisticated prosody modeling, also propose conditional DPPs which utilize the conditional information for the sampling. Specifically, we first segment the whole input text into more fine-grained text seg- ments, and then sample prosodic features for the targeted segment, reflecting its neighbor segments to generate more expressive samples.
from a single utterance by adopting condi- tional DPPs into our TTS model with the novel objective function.
We evaluate and demonstrate that our model outperforms the baselines in terms of prosody diversity while showing more stable natural- ness.
Background
2.1 Determinantal point processes
Given ground set y which consists of prosodic fea- tures in our case, a determinantal point processes (DPP) defines probability distribution for all sub- sets of ) via positive semi-definite matrix L as followings:
det( Lv) PL(Y) det( I)
(1)
where det(-) is the determinant of a matrix, Ly de- notes the submatrix of L whose entries are indexed by the subset Y and det(L I) in the denomi- nator acts as normalization constant. To model diversity between items, the DPP kernel is usu- ally constructed as symmetric similarity matrix S, where Sig represents the similarity between two items x; and Øz- Kulesza and Taskar (2010) pro- pose decomposing the kernel L as a Gram matrix incorporating quality vector to weigh each item according to its quality by defining the kernel ma- trix Lijj q: Si Qj where qi denotes the quality of the item. The quality can be chosen as the likelihood the prosodic feature in our context. Given two prosodic feature Yi = {i.j), the proba- bility ())(i,j)) to sample two prosodic features is proportional to det(Ly;) = of of (1 - ST3) Therefore, two prosodic features are unlikely sam- pled together if they are highly similar to each other. In contrast, they are more likely sampled together if they have high quality values.
In the process of adopting conditional DPPs into our TTS models, two technical difficulties must be handled. First, the range of targets for the sam- pling of conditional DPP is ambiguous. Second, prosodic features usually vary in length. We re- solve the first issue by extracting key segments from the input text using Prosodic Boundary De- tector (PBD). Next, we resolve the second issue by adopting the similarity metric, soft dynamic time warping discrepancy (Soft-DTW) (Cuturi and Blondel, 2017), among the segments of variable lengths.
Experimental results demonstrate that our train- ing methodology becomes stable with the new adaptive MIC objective function, overcoming the instability of sampling-based learning. In addition, our DPP-TTS is highly effective at generating more expressive speech and guaranteeing the quality of speech. We find that our model consistently gains higher diversity score across all temperature set- tings than baselines.
2.2 Conditional determinantal point processes
In summary, our contributions of the paper are as follows:
If DPPs are used for diversifying prosodic features corresponding to the target utterance, it would re- sult in diversity among generated samples. How- ever, there still can be monotonous patterns in each generated utterance. To resolve this issue, it is nec- essary to model the prosodic features of the target
We show that the fine-grained level of the prosody modeling method based PBD con- tributes more diverse prosody in each gen- erated speech sample.
accounting into the neighboring segments of the target. In previous studies, DPPs are extended to
We generate more diverse prosody patterns

---

Prior Distribution
Sampling
PDM
DPP Kerel
DTW Discrepancy
Density
Hidden Sequence
Encoder
Density Estimation
Prosody predictor
Attention
Text
Boundary Position
Candidates
Boundary Detector
Ground Set
DPP Kernel
Construction
Build
Segmentation
Candidates
Loss
Build
Figure 2: Diagrams describing the training procedure PDM.
conditional DPPs, for subset B C Y not inter- secting with A we have
3 DPP-TTS
Our model DPP-TTS is composed of base TTS based on FastSpeech2 (Ren al., 2019), Prosody (2) Diversifying module (PDM), and Prosody bound- ary Detector (PBD). Once the base TTS is trained, (3) PDM is inserted in front of the prosody predic- tor and trained with the method which will be de- scribed in detail in Section 3.2. We describe the main modules of DPP-TTS in the following subsec- tion 3.1. For the detailed architecture of DPP-TTS. refer to Appendix A.
= TY==AUBIACY P(Y= = AUB)
P(ACY)
det (AU(L)) det(L Ix)
where I is the matrix with ones in the diagonal entries indexed by elements A and zeros else- where. In conditional DPPs, items in the ground set y can be sampled according to kernel considering given contexts A. In this work, prosodic features corresponding neighbor segments of the target are used conditions for conditional DPPs.
3.1 Main modules of DPP-TTS
Prosody predictor Our prosody predictor esti- mates the distribution of the duration and pitch in prosody. For more human-like rhythm and pitch in prosody, the stochastic prosody predictor is built upon normalizing flows. Specifically, the stochas- tic duration predictor estimates the distribution of phoneme duration and the stochastic pitch predictor estimates the distribution of phoneme-level pitch from the hidden sequence. At the training stage, the prosody predictor learns the mapping from the distribution of prosodic features to normal distri- bution. At inference, it predicts the phoneme-level duration pitch by reversing the learned flows. In addition, it also serves the density estimator prosodic features during the training of PDM which will be described in detail the next subsec- tion 3.2. The prosody predictor is trained to max-
2.3 Prosody phrasing
As humans usually speak with some pauses convey a message and the speaker's intention, the utterance is required
divided into more fine-grained segments, referred as prosodic units. Ye et al. (2023) show that the prosody pattern re- flecting relevant text context contributes to the enhanced TTS. Suni et al. (2020); Nguyen et al. (2020) suggest that prosodic boundary plays an im- portant role in the naturalness and intelligibility of speech. To incorporate an inherent prosodic struc- ture of input utterance within its context, we build Prosodic Boundary Detector (PBD) trained on a large corpus along with prominence labels.

---

imize variational lower bound of the likelihood of the phoneme duration or pitch. More details regarding the prosody predictor are in Appendix A.
at the prominent word and ends right before another prominent word. In addition, adjacent left and right contexts with the same number of words for the target are chosen.
PDM Since the prosodic predictor is only trained with the lower bound of likelihood objective, it is not enough to generate diverse and non- monotonous prosody patterns. For more expressive speech modeling, PDM is added in front of the prosody predictor as shown in Figure 2. Its role to map samples from a standard normal distribution to another distribution for diverse prosodic features of speech. This module is trained with an objective based conditional DPPs which is described in Section 2.2. At inference of DPP-TTS, multiple prosodic candidates are generated by PDM. Subse- quently, the prosodic feature of speech is selected via MAP inference when a single sample is gen- crated, otherwise, multiple prosodic features are generated via the DPP sampling.
Algorithm 1 Training of PDM
Require: TextEncoder f(-), PDM parameterized
8, prosody predictor g(-), number of candi- dates nee noise scale e
1: while not converged do
2: Mrez +- f(lexl)
3: Split hezt into [harggtt, hcontezt and store indices (de. te) for the target and context
Sample latent code Zeontext IR with
noise scale
Get prosodic features of contexts:
deontezt < [g(heeats #contezt ))kc 6: Sample latent codes Starget with noise scale
7: Get latent codes after PDM: Starget PDM(Ztarget) Rexx Get ne prosodic features of targets: (dhargets tar [Q Starget)
Prosodic Boundary Detector We utilize the prosodic boundary based on the prominence of words, by reflecting the assumption that people un- consciously pronounce the sentence focusing on what intentions they want to convey. To decide the boundary, we create Prosodic Boundary Detector (PBD) whose backbone is pretrained Sentence- Transformer. A Prominence dataset (Talman et al., 2019), consisting of Librispeech scripts and promi- nence classes, is used for training input of PBD text sequence, and PBD predicts each word's prominence level. Based on the promi- nence level, the prosodic units are extracted from the input utterance.
Concatenate contexts and targets:
[dcomtents dorg
10: Get quality of targets along with contexts:
Farget Likel hooohoddeonttetee dearget))
II:
Build the kernel of conditional DPPs:
L < Build kernel Liar geto [dCon, Burreet)
PBD. The
12: Calculate the loss function:
Ldiversity f- -r([ [(L + +1x) (x)=
13:
Update 0 with the gradient V Ldiversity
14:
end while
3.2 Training process of PDM
3.2.2 Generation of prosodic feature candidates
In this section, we explain the methodology de- scribed in Algorithm 1 for training the prosody diversifying module (PDM). The training process mainly consists of three steps: segmentation of an input text, generation of prosodic feature candi- dates, and building DPP kernel. We also explain the conditional maximum induced cardinality (MIC) objective for training PDM. The overall training procedure is depicted in Figure 2.
In this stage, multiple prosodic candidates are gen- erated for DPP sampling as shown in Figure 2. First, a hidden sequence is generated from an input text through the text encoder. Second, the pre- trained prosody predictor generates ni prosodic features conditioned on the hidden sequence by uti- lizing samples from the normal distribution. Mean- while, other new samples from a normal distribu- tion are fed into PDM, and then the prosodic predic- tor conditioned the hidden sequence generates new prosodic features of the target from the output features of PDM. Finally, the latter-generated tar- get prosodic features substitute former semer--enerated target features, and then n. prosodic candidates are generated with the target and context entangled.
3.2.1 Segmentation of the input text In this stage. targets in input text for diversification of prosody are chosen. Given a sentence, PBD predicts the positions of prominent words. After that, those positions are used to divide the context and target sequences. Specifically, each target starts

---

3.2.3 Construction of DPP kernel
where is a quality weight and the threshold value was set as the average density of the train- ing dataset in the experiment. We need to mea- sure the diversity with respect to kernel L to train the PDM. One straightforward choice the max- imum likelihood (MLE) objective, log (Y = log (L log det (L + I). However, there are some cases where almost identical prosodic fea- tures are predicted. We recognize that such cases cause the objective value to become near zero, only to make the training process unstable. Instead, max- imum induced cardinality (MIC) (Gillenwater al., 2018) objective which is defined as Sy..Pr(( can be an alternative. It does not suffer from training instability. In this work, context segments dz, dR are used as the condition in the MIC objective of conditional DPPs The objective function with re- spect to the candidate set [dz, dR, d1, d2, ...,. du] and its derivative are as follows:
Generated candidates are split into left and right context d7, du and n targets d1, d2....dn. then the candidate set for DPP is constructed as shown in the Figure 2. Next, the kernel of conditional DPP is built by incorporating both the diversity and qual- ity(likeihhood) of each candidate feature. Here, quality features guarantee the smooth transition of prosody. The kernel of conditional DPPs de- fined as = diag(4) S diag(g), where S is the similarity matrix and is the quality vector.
3.3 Objective function
Similarity metric In the process of construct- ing the DPP kernel, we need define the similar- ity metric between features. However, target se- quences and context sequences often vary in length. As the Euclidean distance is not applicable to cal- culate the similarity between two sequences, we utilize Soft DTW:
Proposition (MIC objective of CDPPs) With respect to the candidate set [d,,, due d1, d2:..., d.., the MIC objective of conditional DPPs and its derivative are as follows:
S. cxpp(dttw/<( dz))
(4)
where dtw, denotes soft-DTWW discrepancy with a metric D and smoothing parameter 7. When the metric D is chosen as the In distance D ( y)
LMIC tr([I (((((0) + (1x), (6)
aLMIC = ((L 1 IA + AA) 1 7L 00
00
Σ, [ yil or half gaussian D(x, y) = | Elا + log(2 exp(-Ilæ y)1)),, the similarity matrix becomes positive semi-definite (Blondel et al., 2021; Cuturi et al., 2007). In this work, T distance is used as the metric of Soft-DTW so that S to be positive semi-definite.
(7)
where A denotes the set of contexts (di, dR). A
denotes the complement of the set A and 0 is the parameter of PDM.
Remark 1 The MLE objective becomes unstable since the determinant volume is close to zero if two similar items are included. In contrast, the MIC objective guarantees stability as the gradient of our objective guarantees the full-rank structure.
Quality metric To reflect the naturalness of prosodic features, quality scores are calculated based on the estimated density of predicted features. Given the features æ, posterior 2 ax; Φ) and joint likelihood p(z, z;; f) where o, 9 are parameters of the prosody predictor based on the variational method, the density values of predicted features are calculated with importance sampling using the prosody predictor: p(æ; 0, () Ri E p(x....)) In (==)) the experiment, we empirically find that it is more
Detailed proof is presented in Appendix B. At inference, for predicting single prosodic fea- ture, MAP inference is performed across sets with just single item as follows: x² arg max log det (Lzz)u)) Otherwise, the k- DPP sampling method used for sampling multi- ple prosodic features when multiple speech sam- ples are generated. The detailed procedure of train- ing PDM and the inference of DPP-TTS are in Appendix C.
helpful not to give a penalty to the quality score the likelihood is greater than the specific threshold, With log- likelhood (xx) = log p(x), the quality score of single sample is defined
4 Experiment setup
if (x:) >=
4.1 Dataset and preprocessing
w
((x)=
w (xx-exxppxx=)))-))) 1)) otherwise
We conduct experiments on the LJSpeech dataset which consists of audio clips with approximately
(5)

---

24 hours lengths. We split audio samples into 12500/100/500 samples for the training, valida- tion, and test set. Audio samples with 22kHz sampling rate are transformed into 80 bands mel- spectrograms through the Short-time Fourier trans- form (STFT) with 1024 window size and 256 hop length. International Phonetic Alphabet (IPA) se- quences are used as input for phoneme encoder. Text sequences
the mel-sppetroggamss In addition, we also com- pare our model with bascline DPP-TTS w/o PDM which does not utilize PDM for prosody modeling. Audio samples used for the evaluation are in the supplementary material and the Demo page?.
4.4 Evaluation Method
Side-by-Side evaluation To evaluate the percep- tual diversity of our model, we conduct a side-by- side evaluation of the prosody of speech samples. Via Amazon Mechanical Turk (AMT), we assign ten testers living in the United States to pair of audio samples (i.e., DPP-TTS and a baseline), and ask them listen to audio samples and choose among three options: A: sample A has more varied prosody than sample B, Same: sample and B are equally varied in prosody, B: the opposite of first option. For DPP-TTS-d, testers are asked to focus on the rhythmic variation of the speech sample other than different aspects of prosody. Likewise for DPP-TTS-P -p, the testers are asked to focus the pitch variation of speech samples. Importantly, the testers are asked to ignore about pace, and vol- ume of speech samples to eliminate possible bias on speech samples as possible.
converted IPA phoneme se- quences using Phonemizer' software. Follow- ing (Kim et al., 2020), the converted sequences are interspersed with blank tokens, which repre- sent the transition from one phoneme to another phoneme.
4.2 Implementation Details
Both the base TTS and PDM are trained using the AdamW optimizer (Loshchilov and Hutter, 2019) with 31 = -08,, A2 = 0.99 and À = 0.01. The ini- tial learning rate is 2 x 10-4 for the basic TTS and 1 X 10-5 for the PDM with exponential learning decay. Base TTS is trained with 128 batch size for 270k steps on 4 NVIDIA RTX A5000 GPUs. PDM is trained with 8 batch size for 2k steps on single GPU. The quality weight of the model is set as w = 10. We prepare two DPP-TTS versions for the ovaluation: (1): DPP-TTS-d: a model that has the duration diversifying module and DPP-TTS-p: a model that has the pitch diversifying module.
MOS In addition, we also conduct the Mean- Opinion-Score (MOS) to evaluate the naturalness of prosody for generated samples. Ten testers are assigned to each audio sample. Given reference speech samples to each score, testers are asked to give score between to 5 on a 9-scale based on the sample's 's naturalness, In addition, they are asked to focus on the prosody aspect of audio samples.
4.3 Baselines
We compare our model with the following state- of-the-art models: 1) VITS² (Kim et al., 2021), an end-to-end TTS model based on conditional VAE and normalizing flows. It mainly has two pa- rameters for the sampling: the standard deviation of input noise to duration predictor and a scale factor F the standard deviation of prior distri- bution which controls other variations of speech (e.g.. pitch and energy); 2) Flowtron" (Valle al., 2021), an autoregressive flow-based TTS model; 3) DiffSpeech+ (Liu et al., 2021), a diffusion- based probabilistic text-to-speech model; 4) Syn- taSpeech² (Ye et al., 2022), TTS model using a syntactic graph of input sentence for prosody modeling. HiFi-GAN (Kong et al., 2020) is used the vocoder for synthesizing waveforms from
Quantitative evaluation In addition to human evaluation, We conduct quantitative evaluations for our DPP-TTS. DPP-TTS w/o PDM, and VITS with high temperature 7, We have used the following metrics for evaluating our model and the bascline. Tpi phoneme-level standard deviation of dura- tion or pitch in speech. This metric reflects the prosodic diversity inside each speech sample.
Determinant: determinant of the similarity matrix is used evaluate the diversity among prosodic features of 10 generated samples. Cosine similarity is used as a metric between two features. Since dissimilar items increase the volume of the matrix, higher determinant values indicate that gen- erated samples are more diverse.
htttps://githubbcoo httpa//b..ommoottppo///o/oooaama
http://wwww....oooooooattttettte Phttps: http://ggghooocmooeeerrr/srttsss
'It tas: //ppp-ddpppptpp tts. gi thub. io/
7 Same counterparts as the side-hy-side evaluation are used.

---

Table Side-by-side comparison on the LJSpeech dataset along with MOS results. Model A/B denotes the models compared for the side-by-side evaluation. 'A' denotes the percentage of testers who vote for the sample of Model A is more varied than B, and 'B' denotes the vice-versa. 'Same' denotes the percentage of testers who vote for two samples that are equally varied prosody.
Model A/B
A
Same
B
Model MOS
Model MOS
DPP-TTS-d/Flowtron = 0.6)
52.0%
16.5%
31.5%
3.97 + 0.08
3.85 = 0.06
DPP-TTSS-P/Flowtronn( = 1.0)
49.3%
22.6%
28%
4.02 + 0.08
3.66 + 0.09
DPP-TTS-P/VITS(T 0.667, 0.8)
57.25%
6.75%
36%
4.02 + 0.08
4.38 + 0.07
DPP-TTS-UVIISS( 0.667, 1.2)
44%
34.6%
21.3%
3.97 = 0.08
3.95 = 0.08
DPP-TTS-P/VITS(- = 1.2, a 0.8)
34.6%
42.3%
23%
4.02 + 0.08
3.98 + 0.08
DPP-TTS-d/DiffSpeechh DPP-TTS-p/DifSSeecch
55.5%
12.7%
31.8
3.97 + 0.08
3.92 0.06
66.4%
5.4 %
28.2 %
4.02 + 0.08
3.92 = 0.06
DPP-TTS--dSSnuaSpeech
55.5%
9.1%
35.4 %
3.97 + 0.08
4.04 + 0.09
DPP-TTS-p/SymiaSpeech
63.6%
6.4 %
30.0 %
4.02 1: 0.08
4.04 1: 0.09
DPP-TTS-p/DPP-TTS w/o PDM
55.5 %
17.5 %
27 %
4.02 1 0.08
4.09 + 0.09
Inference time: the inference time for synthesiz- ing waveform is calculated the TTS model. The inference speed evaluated on Intel(R) Core(TM) 17-7800X CPU and a single NVIDIA RTX 3080 GPU. Computation time is averaged over 100 for- ward passes.
being more prominent in longer texts.
Table Side-by-Sinn comparison on the paragraph.
Model A/B
DPP-TTS- = 0.687, = 0.5)
25.25% 48.4%%
61.0%
13.75%
DPP-TTS--VVITS:: 0.667.
25.27%
26.27% 36.34%
DPR-TTS-PVITSS- - -12, - 0.8)
40.91%
22.73%
5 Results
5.2 Quantitative evaluation
5.1
Perceptual Diversity Results
We first report the side-by-side evaluation results between our model and baselines along with the MOS test. Results observe three findings from the evaluation: 1) Scaling temperature of Flowtron and VITS model does not contribute much to the perceptual diver- sity of rhythm and pitch while the MOS degrades by& large margin; 2) Our DPP-TTS model out- performs the DPP-TTS w/o PDM in the side-by- side evaluation which proves the usefulness PDM with MIC training objective; 3) Our model DPP- TTS outperforms the four baselines in the side- by-side evaluation for rhythm and pitch compari- son. Since LJSpeech dataset consists of short text transcripts, we conduct the side-by-side evaluation for paragraph samples to evaluate prosody mod- eling in longer input texts. As the evaluation of the L.ISpeech dataset, listeners are asked to choose among the three options. In the side-by-side eval- uation of prosody in a paragraph, both the DPP- TTS-d/p outperform the baseline as Table shows. More testers voted for our model in the evaluation for paragraph than the LJSpeech dataset. We spec- ulate that this is due to diverse prosody patterns
Table 3:
Quantitative evaluation results. Numbers in bold denote the better result.
shown in Table 1. We
Model VITS
Op
Duration Determinant
0.039
1.86 x 10-5
DPP-TTS-d
0.041
2.98 x 10
DPP-TTS w/o PDM
0.033
2.02 x 10
Model
dp
Pitch Determinunt
Inference time(s)
Flowtroe
0.0121
10-14
5.3 10-1
VITS
0,0132
4.12 10-15
1.3 10
DiftSpazeh
0.0133
8.51
10-14
4.1 10
SyntSSpesch
0.0145
2.23 x 10-14
3.9 10-2
DPP-TTS-p
0.0145
1.88 10
4.4 10
wio PDM
0.011
3.13 10-14
2.3 10
Table 3 shows the result of op. determinant, and inference time In both duration and pitch op- DPP-TTS outperforms baselines by having higher standard deviation in the phoneme-level features. demonstrates that DPP-TTS generates a speech with more dynamic pitch and rhythm than the base- line. The determinants of duration and pitch sets of
"If the model does not have a separate duration predic- tor like VITS, it is challenging to evaluate the duration of phonemes, therefore only the pitch evaluation is included for other baselines.

---

Table 4: Side-by-side comparison between the model with PBD and the fixed-length baseline without PBD.
Model A/B
A
Same
B
Model A MOS
Model B MOS
DPP-TTS--dFiied--ee
58.2%
24.5%
17.3%
4.02 + 0.08
3.78 0.06
DPP-TTS-p/Fixedd-en
61.8%
23.4%
14.8%
3.98 + 0.08
3.78 + 0.06
model starts to generate more dynamic and diverse prosody patterns. For examples of the pitch contour from generated samples, refer to Appendix G.
0.00
Case study To analyze why and how the natu- ralness drops after applying PDM, we investigate some samples that get low MOS by testers. The degradation cases mainly fall into cases. First, in some utterances, the prosodic transition between adjacent words arises too rapidly. A quality met- ric that imposes a penalty for the rapid transition may address this problem. Second, speech real- ization at a particular level of pitch is somewhat awkward. This problem is likely to appear since the TTS model has not seen enough prosody pat- terns during the training stage. We believe that speech augmentation related to pitch or duration will address this problem.
-0.50
-0.75
-1.25
-1.50
-1.75
Figure 3: Loss trajectory for the MIC and MLE objec- tive. While the training dynamic of MLE objective is unstable due to the structure, the MIC objective shows faster convergence.
DPP-TTS also outperform the baseline. It shows that DPP-TTS generates more samples with diverse prosody than baselines. Finally, DPP-TTS results 0.044 seconds of inference speed. Although its inference speed is slower than the baseline, our model is applicable in practice since the inference speed of our model is 22.7x faster than real-time.
Effectiveness of PBD to verify the effectiveness of PBD over the fixed- length baseline. While PBD dynamically adjusts the length of the target considering the prosodic boundary, the fixed-length baseline just extracts the target of fixed length. For example, with length = 3, the first three words are selected as the text, the next three words selected as the target, and the next three words are selected as the context, and so on. We can see from Table 4, both the per- ceptual diversity and MOS the baseline decrease compared to the model using PBD. indicates the advantage of PBD which dynamically adjusts the target length considering the prosodic boundary.
We conduct an experiment
5.3
Model analysis
MIC vs MLE objective for training PDM We conduct ablation studies to verify the effectiveness MIC training objective for conditional DPPs. Fig 3 shows the training loss trajectory. The norm of the gradient for MLE objective blows up when prosodic features in the candidate set are nearly identical leading to unstable training. In contrast, since our MIC objective guarantees the full rank structure of the gradient matrix, the training does not suffer from instability, leading to faster conver- gence.
Related Works
There have been many efforts TTS research to enhance the expressivity of generated speech. Learning latent prosody embedding at the sen- tence level is proposed to generate more expres- sive speech (Skerry- Ryan et al., 2018; Wang et al., 2018), Wan al. (2019) presents hierarchical con- ditional VAE model to generate speech with more expressive prosody and Sun al. (2020) propose hierarchical VAE-based model for fine- ane-grained prosody modeling. In addition, a speech syn-
Adjusting the extent of variation To study the impact of quality weight in quality metric (2x) for DPP sampling, we generate multiple speech samples with different values of quality weight, For a large magnitude of quality weight, our model likely generates smoother prosody patterns. As the magnitude of quality weight w decreases, the

---

thesis model incorporating linguistic information BERT (Devlin et al., 2019) is proposed get en- riched text representation by (Kenter et al., 2020), However, the controllability of speech attributes like pitch and rhythm not fully resolved and the expressivity or diversity of generated speech is still far off the human.
quality metric for building DPP kernel, we have used the density value of each sample. We use an importance sample scheme for the value estimation, however, it needs multiple samples for exact evalu- ation. We believe more efficient sampling scheme contributes to generating more exact evaluation of the naturalness of prosodic features.
Meanwhile, since prosody is directly related to duration, pitch, and energy, text-lo-speech mod- els that explicitly control these features are pro- posed (Lańcucki, 2021; Ren et al., 2021a). A flow- based stochastic duration predictor is also proposed to generate speech with more diverse rhythms and it shows superior performance compared to a de- terministic duration predictor (Kim et al., 2021). In this work, incorporate stochastic duration, and pitch predictor upon Fastspeech2 (Ren al., 2021a) model more expressive prosody.
A Acknowledgements
We thank anonymous reviewers for their construc- tive and insightful comments. K. Jung is with ASRI, Seoul National University, Korea. This work was supported by Samsung Electronics. This work was partly supported by Institute of Informa- tion communications Technology Planning Eval- uation (IITP) grant funded by the Korea govern- men((MSIT) NN0.2021-0-01344 Artificial Intelli- gence Graduate School Program (Seoul National University)J.
7 Conclusion
We propose Prosody Diversifying Module (PDM) that explicitly diversifies prosodic features like duration or pitch to avoid a monotonous speak- ing style, using conditional MIC objective. Pre- vious research has indicated that well-positioned prosodic boundaries are helpful for understanding the meaning of speech (Sanderman and Collier, 1997). There are some possible avenues improve our model for more realistic prosody. A more so- phisticated segmentation rule that reflects human prosodic boundaries can improve the naturalness of our model. Incorporating linguistic representation well-aligned with human intonation contributes to accurate prosodic boundaries. Such methods will boost the performance of our model.
References
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. ArXiv, abs//667.066/ .6650.
Mathieu Blondel, A. Mensch, and Jean-Philippe Vert. 2021. Differentiable divergences between time series. In ATSTATS.
Jianfei Chen, Cheng Lu, Biqi Chenli. J. Zhu, and Tian Tian. 2020. Vflow: More expressive generative flows with variational data augmentation. ArXiv, abs/2002.09741.
Sangwoo Cho, Chen Li. Dong Yu, H. Foroosh, and Fei Liu. 2019. Multi-document summarization with determinantal point processes and contextualized rep- resentations. ArXiv, abs/1910.11411.
Marco Cuturi and Mathieu Blondel. 2017. Soft-dtw: a differentiable loss function for time-series. In ICML.
Limitations
Since methodology depends the segmenta- tion of input utterances, a sophisticated boundary for input utterances necessary. We believe that more advanced research for prosody boundary de- tectors will contribute to smoother and more expres- sive prosody for TTS. The expression of speech at certain pitch level can be somewhat unnatu- ral. This issue probably arises because the TTS model hasn't 't been sufficiently exposed to prosody patterns during its training phase. We believe that improvements related to pitch or duration in speech augmentation could solve this issue. We plan to resolve this problem by using an advanced augmen- tation method and large pretrained models. For the
Marco Cuturi, Jean- Philippe Vert, Øystein Birkenes. and T. Matsui. 2007. kernel for time series based on global alignments. 2007 IEEE International Con- ference Acoustics, Speech and Signal Processing ICASSP '07, 2:11-413-111
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. ArXiv, abs//880.004805
Jeff Donahue, Dieleman, Mikolaj Binkowski, Erich Ellsn, and K. Simonyan. 2021. End-to-end adversar- ial text-to-speech. ArXiv, abs/2006.03575.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. 2019. Neural spline flows. NeurIPS.

---

Mike Gartrell, Insu Han, Elvis Dohmatob, Jennifer Gillenwater, and Victor-Emmant Emmanuel Brunel. 2021. Scalable learning and map inference for non- symmetric determinantal point processes. ArXiv, abs/2006.09862..
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Confer- ence Learning Representations.
Diganta Misra. 2019.
Mish: A self regularized non-monotonic neural activation function. ArXiv, abs/1908.08681.
Jennifer Gillenwater, A. Kulesza, Sergei Vassilvitskii, and Zelda E. Mariet. 2018. Maximizing induced cardinality under a determinantal point process. In NeurlPS.
Thi-Thu- Thi-Thu-Trang Nguyen, Nguyen Hoang Ky, Albert Ril- liard, and Christophe Alessandro. 2020. Prosodic boundary prediction model for vietnamese text-to- speech. In Interspeech.
Jonathan Ho. Xi Chen, A. Srinivas, Yan Duan, and P. Abbeel. 2019. Flow++: Improving flow-based generative models with variational dequantization and architecture design. ArXiv, abs/1902.00075..
Wei Ping, Kainan Peng, Andrew Gibiansky. Ser- can Ô. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. 2018. Deep voice 3: Scal- ing texe-to-speech with convolutional sequence learn- ing. arXiv: Sound.
Wei-Ning Hsu, Y. Zhang, Ron J. Weiss, H. Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, 7. Chen, Jonathan Shen, P. Nguyen, and Ruoming Pang. 2019. Hierarchical generative modeling for controllable speech synthesis. ArXiv, abs/1810.07211.
Yi Ren, Chenxu Hu, Xu Tam, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2021a. Fastspeech 2: Fast and high-quality end end-to-end text to speech. ArXiv, abs/2006.04558.
Tom Kenter, Manish Sharma, and Robert A. Clark. 2020. Improving the prosody of ran-based en- glish text-to-speech synthesis by incorporating bert model. In INTERSPÉECH.
Yi Ren, Jinglin Liu, and Zhou Zhao. 2021b. Por- taspeech: Portable and high-quality generative text- to-speech. In Advances in Neural Information Pro- cessing Systems.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sun- groh Yoon. 2020. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. ArXiv, abs/2005. 1122.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie Yan Liu. 2019. Fastspeech: Fast, robust and controllable text to speech. In NeurlPS.
Jachyeon Kim, Jungil Kong, and Juhee Son. 2021. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-specch. ArXiv, abs/2106.06103.
Angelien A. Sanderman and René Collier. 1997.
Prosodic phrasing and comprehension. Language and Speech, 40:391 -409.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In NAACL.
Jungil Kong, Jachyeon Kim, and Jackyoung Bac. 2020. Hifi-gan: Generative adversarial networks for ef- ficient and high fidelity speech synthesis. In Ad- vances in Neural Information Processing Systems, volume 33, pages 17022-17033. Curran Associates, Inc.
Jonathan Shen, Ruoming Pang. Ron J. Weiss, M. Schuster, Navdeep Jaitly, Zongheng Yang. Z. Chen, Yu Zhang. Yuxuan Wang, R. Skerry-Ryan, R. Saurous, Yannis Agiomyrgiamakiis and Yonghui Wu. 2018. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783.
A. Kulesza and B. Taskar. 2012. Determinantal point processes for machine learning. Found. Trends Mach. Learn., 5:123-286.
Alex Kulesza and Ben Taskar. 2010. Structured deter- minantal point processes. In NIPS.
R. Skerry-Ryan, Eric Battenberg, Y. Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J. Weiss, R. Clark, and R. Saurous. 2018. Towards end-to- end prosody transfer for expressive speech synthesis with tacotron. ArXiv, abs/ 100/180339947..
Adrian Lańcuuci.. 2021. Fastpitch: Parallel text-to- speech with pitch prediction. In ICASSP.
Yoonhyung Lee. Joongbo Shin, and Kyomin Jung. 2021. Bidirectional variational inference for non- autoregressive text-to-speech. In ICLR.
G. Sun, Y. Zhang, Ron J. Weiss, Yuanbin Cao, H. Zen, and Yonghui Wu. 2020. hallychierar fine- grained prosody modeling for interpretable speech synthesis. ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 6264-6268.
N. Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. 2019. Neural speech synthesis with transformer network. In AAAI.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. 2021. Diffssiger: Singing voice syn- thesis via shallow diffusion mechanism. AAAI Conference on Artificial Intelligence.
Antti Suni, Sofoklis Kakouros, Martti Vainio, and Juraj Šimko. 2020. Prosodic prominence and boundaries in sequence-to-sequencc speech synthesis. ArXiv. abs/2006. 15967.

---

Aarne Talman, Antti Suni, Hande Celikkanat, Sofoklis Kakouros, Jörg Tiedemann, and Martti Vainio. 2019. Predicting prosodic prominence from text with pre- trained contextualized word representations. ceedings of the 22nd Nordic Conference on Compu- lational Linguistics, pages 181-290, Turku, Finland. Linköping University Electronic Press.
Pro-
Rafael Valle, Kevin Shih, R. Prenger. and Bryan Catanzaro. 2021. Flowtron: an autonegressive flow- based generative network for text-to-speech synthesis. ArXiv, abs/2005.0995
Iván Vállés-Pérez, Julian Roth, Grzegorz Beringer, R. Barra-Chicote, and J. Droppo. 2021. Im- proving multi-speaker its prosody variance with residual encoder and normalizing flows. ArXiv, abs/2106.05762.
Vincent Wan, Chun an Chan, Tom Kenter, Jakub Vít, and Robert A. J. Clark. 2019. Chive: Varying prosody in speech synthesis with linguistically driven dynamic hierarchical conditional variational network. ArXiv, abs/1905.07195.
Yuxuan Wang, Daisy Stanton, Yu Zhang, R. Skerry- Ryan, Eric Battenberg, Joel Shor, Xiao, Fei Ren, Ye Jia, and R. Saurous. 2018. Style tokens: Un- supervised style modeling, control and transfer in end-to-end speech synthesis. ICML.
Yunyang Xiong, Zhampeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn M. Fung, Yin Li, and Vikas Singh. 2021. Nyströmformer: A nyström-based algo- rithm for approximating self-attention. AAAI.
Guanghui Xu, Wei Song, Zhengehen Zhang, Chao Zhang. Xiaodong He, and Bowen Zhou. 2020. Im- proving prosody modelling with cross-utterance bert embeddings for end-to-end speech synthesis. ICASSP 2021 2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 6079-6083.
Zhenhui Ye, Rongjic Huang, Yi Ren, Ziyue Jiang, Jinglin Liu, Jinzheng He, Xiang Yin, and Zhou Zhao. 2023. Clapspeech: Learning prosody from text con- text with contrastive language-aute pre- pre-training.
Zhenhui Ye, Zhou Zhao, Yi Ren, and Fei Wu. 2022. Syntaspeech: Syntax-aware generative adversarial text-to-speech. ArXiv, abs/2204.11792

---

A Details of DPP-TTS
predictor estimates the distribution of phoneme du- ration and the stochastic pitch predictor estimates the distribution of phoneme-level pitch from the hidden sequence. At the training stage, the prosody predictor learns the mapping from the distribution of prosodic features to normal distribution. At in- ference. it predicts the phoneme-level duration or pitch by reversing the learned flows. In addition, it also serves as the density estimator for prosodic features during the training of PDM which will be described in detail in Section 4. The prosody pre- dictor is trained to maximize a variational lower bound of the likelihood of the phoneme duration or pitch. More details regarding the prosody predictor are in Appendix A.
Our model DPP-TTS is composed of a Seq2Seq module for generating the mel-spectrogram, prosody predictor for predicting duration and pitch sequences, and prosody diversifying module (PDM). At the first stage, the base TTS model which consists of the Seq2Seq module and the prosody predictor is trained as shown in Figure 4a. Once the base TTS is trained, PDM is inserted in front of the prosody predictor and trained with the method which will be described in detail in Sec- tion 3.2. We describe the main modules of DPP- TTS and their roles in the following subsection.
A.1 Main modules of DPP-TTS
Seq2Seq module The role of the Seq2seq mod- ule is generating mel-spectrograms from phoneme sequences. The module is adapted from Fast- Speech2 (Ren et al., 2021a) with some modifi- cations. The model consists of four main parts: a phoneme prenet, phoneme encoder, vari- ance adaptor, and a mel-spectroogram decoder. In the phoneme encoder, phoneme sequences are processed through a stack of feed-forward trans- former blocks with relative positional representa- tion (Shaw et al., 2018). In variance adaptor at train- ing, pitch embeddings and energy embeddings 9 are added to encoded hidden representations and then hidden representations are expanded according to ground-truil duration labels 10, At inference, these prosodic features are provided from predictions of the prosody predictor. Finally, expanded repre- sentations are processed through stack of feed- forward transformer blocks and mel-spectrograms with 80 channels are generated after the linear pro- jection. The Seq2Seq module is trained to mini- mize L1 distance between the predicted and target mel-spectrogram.
PDM Although the stochastic duration and pitch predictor are trained to generate speech with diverse rhythm and pitch, the prosody predictor may favors major modes and it can lead to the monotonous prosodic pattern in the speech. For more expressive speech modeling, PDM is added in front of the prosody predictor as shown Fig- ure 4b. Its role is to map latent codes from a stan- dard normal distribution to another distribution for diverse prosodic features of speech. This module is trained with an objective based on conditional DPPs which is described in Section 2.2. At infer- ence of DPP-TTS, multiple prosodic candidates are generated by PDM. Subsequently, the prosodic feature of speech is selected via MAP inference, or multiple prosodic features are sampled when multiple speech samples are generated.
Prosodic Boundary Detector There are two ways of diversifying prosodic features: autoregres- sive and non-autoregressive methods. Since autore- gressive methods require much time to generate the output sequence, we employ the non-autoregressivve method in our model. For the non-autoregressive approach, some prosodic features of speech need be set when trying to diversify overall prosodic fea- tures. In this perspective, a sentence needs to be di- vided into a context sequence and target sequence, utilizing the prosodic boundary based on the promi- nence of words by reflecting the assumption that people unconsciously pronounce the sentence fo- cusing on what intentions they want convey. To decide the boundary, we make Prosodic Boundary Detector (PBD) whose backbone is pretrained Sentence-Transforee Transformer. A Prominence dataset (Tal- man et al., 2019) consisting of Librispeech scripts
Prosody predictor In FastSpeech2, the variance adaptor consists of deterministic predictors for pre- dicting prosodic features. However, a determin- istic prosodic predictor is not expressive enough to learn the speaking style of a person. For di- verse rhythm and pitch, a stochastic duration pre- dictor and pitch predictor are built upon normal- izing flows. Specifically, the stochastic duration
"For the brevity, deterministic energy predictor omit- ted in the figure. 10 "Ground-truth labels are obtained via monotonic alignment search (Kim et al., 2020) between the phonemes and me]- spectrogram.

---

Mel-spectrogram Decoder
MAP Inference
Prosadic
Expand
Positional Encoding
S
Duration predictor
DPP Kernel bulld
DDuraiin
Variance Adaptor
Energy Pitch
Normal Distribution
Duration Pitch predictor
Hidden Sequence
Pitch predictor
Stop Gradient
Phonome Encoder
PDM
Distribution
Phoneme Prenet
Phoneme Embeddings
(b) Variance adaptor at inference of DPP-TTS
Base soeechh model training
Figure 4: Diagrams describing (a): training of base text-to-speech model and (b) variance adaptor at the inference DPP-TTS after PDM added.
and prominence classes is used for training PBD. The input of PBD is a text sequence, and PBD predicts each word's prominence level.
Like the duration predictor, the ELBO for pitch predictor can be calculated as follows:
log Po hrezt) 2
poolllp. [log Po (p, v/ o(feeczt) go (vlp, hrezt)
B Details of prosody predictor and PDM
B
where p denotes the pitch sequences. Both the duration and pitch predictor are trained on negative lower bound of likelihood.
B.1 Prosody predictor
Training It is hard to the maximum likeli- hood objective directly to train duration predictor because the duration of each phoneme is I) a dis- crete integer and a scalar, which hinders expressive transformation because invertibility should remain in normalizing flow. To train duration predictor, duration values are extended to continuous val- using variational dequantization (Ho et al., 2019) and are augmented with extra dimensions using variational data augmentation (Chen al., 2020). Specifically, duration sequence d becomes continuous as u where u's value is restricted to [0, 1) and augmented as [d u, v] with a ex- tra random variable v. Two random variables and are sampled through approximate posterior go (u vld, hezz)). The ELBO can be calculated as follows:
The pitch predictor and duration predictor share identical architecture except for the additional random variable u. We will introduce the architecture the duration predictor whose diagram is shown in Figure 5a. Duration predictor consists of condition encoder for hidden sequence, posterior encoder for the duration sequence, and the main flow blocks 9(.). Specifically, posterior encoder maps latent codes from normal distribu- tion to random variable [u, 2) and flow g(-) maps [d - u, u] to normal distribution. Figure 5b shows the coupling of normalizing flows. First, input x is split into [40, æ1] and then Tg is processed by a Ixl convolution block. The output of the convo- lution block is processed by dilated convolution and Nystromer block and their outputs are con- catenated. The concatenated output processed by LayerNorm. Mish activation, and Convolution block. Finally, spline flows (Durkan et al., 2019) are parameterized by the output and then the output of spline flows and zo are concatenated.
Architecture
log py(d) (teas))))
po(d u, (//eett)) ao (u, v/dd Mecer)
E Egg(u (u.lld (e Dlog

---

Split
X;
Conv 1x1
Latent
Dilated
Nyströner
Conv
Missh
Concat
Conv 1x1
LayerNorm
Posterior
encodar
Mish
LayerWora
LayerNorn
Concet
Dilated
Spline flows
Dilated ConvBlock
Condition Encoder
#0
Spline flww
parameters
Conv 1x1
Concat
Hidden sequence
(001))
Hidden sequeecce
Nolse
(a) Duration predictor
(b) Coupling block
(c) PDM block
Figure 5: Diagrams describing (a): the duration predictor, (b): Coupling block in normalizng flows and (c): Prosody diversifying module.
B.2 PDM
2019) is used for the duration predictor since the phoneme duration is discrete integer. In addi- tion, variational augmentation (Chen et al., 2020) is used expand channel dimensions for expres- sive flows in both the duration and pitch predictor, The stochastic and pitch predictor are trained by maximizing their variational lower bounds (ELBO). Details of the duration and pitch predictor are de- scribed in Appendix B.
The architecture of PDM is shown in Figure Sc. First, the hidden sequence is processed by con- volution block followed by the dilated convolu- tion block conditioned on hidden sequences, Lay- ernorm, and 1x1 convolution block. After that, noise from normal distribution processed by Ixl convolution followed by dilated convolution block, LayerNorm, and Mish activation.
As the prosody pattern of human speech is cor- related with the local context and global context of the text, design the architecture of the model to encode both the local and global features. Specifi- cally, dilated convolution blocks are stacked en- code local features and a transformer block used to encode global features in the coupling layers of normalizing flows. However, using the vanilla transformer to encode the global features in the cou- pling layer requires too large computational com- plexity. Therefore, Nyströmformer (Xiong al., 2021) which Nyström- Based algorithm for ap- proximating self- self-attention is used to encode global context for more efficient memory usage in the coupling layer of normalizing flow. Encoded local features and global features are concatenated and processed through LayerNorm (Ba et al., 2016), Mish activation (Misra, 2019) and 1xl convolu- tion module.
C Proof of Proposition 1
Objective From equation [45] in (Kulesza and Taskar, 2012), the marginal kernel of conditional DPPs given the appearance of set A has a following form:
KA-I- [L (I++A)~ "A
(8)
In addition, from equation [34] in Kulesza and Taskar (2012) the expected cardinality of Y given the marginal kernel K is:
An
E E|
Σ
ti(K)
(9)
From equation 8, 9, the expected cardinality of conditional DPP given the appearance of set is:
[|| (r(KA)) tr [L + 1a) (10)
Following the duration predictor model Kim et al. (2021), variational dequantization (Ho et al.,
Derivative For the proof, we will start with the following lemma:

---

Lemma Given a matrix E and non-singular ma- trix A. following equation holds:
D Inference of DPP-TTS
Algorithm 2 Inference of DPP-TTS
① A (A
(11)
Require: TextEncoder f()), Decoder h(-), PDM,
a prosody predictor g(-), noise scale
Proof. First, consider TA, trace and derivative operator are interchangeable,
tattttEA""""" A E). Since
1: heer < f(tert)
2: Split htezt into [htarget: hcontezt)
3: Sample latent code Zeontent E IRrn'l with
0 tr(FTA A FE) tr( 0Ai AAj
① (ETA-!A))
noise scale
Get prosodic features of contexts: domierst f
0A A 1 E)
-rr(EE" A
9 (Noontett #conteat))
aAi
Get quality of contexts:
(12)
Acontert = Density ty estimation (dconteet)))
By setting 04 = E where E denotes the ma-
6: Sample latent codes Starges with noise scale
trix whose (i.j)
and C = component is and O elsewhere
7: Get latent codes after PDM: Starget t
PDM(Zurgee) E R
8: Get ne prosodic features of targets: (dharget: Kaargety-.. -..dae ...gaareet)
1 EcH C77
Get quality of targets:
E A
18 Kika
E)
Storget Density ty innimmttion(darrge
1a2
10: Concatenate contexts and targets:
{(E" ( 1B)
Acontent: Qtarget], |dcontezts diarget)
11: Build the kernel of conditional DPPs:
L + Build kernel 9r), [de: del)
E(A-TE); E) (E* A
12: Perform the MAP inference:
d' +- arg maxx logdet( (addenntea)
(A-TEETAT 1-T);
13: Synthesize wavs with prosodic features: y
= -- EET A 11)
h(hteats d²)
(13)
aA., tr(EE =
Now, with respect to set A whose cardinality is p and matrix L E IR xp
0
tr I
[[L (A)
00
0
tr(((L [x) (Aa)
(14)
00
tr(E")
(L + In)-EE) E).
00
0
where E denotes
E R(llg)
Then by
I
Lemma 1.
ET (L 1A)
30
= =((L + -1)--EE
LEET LL + 11) -TT (15)
=(( I1) 'I (0)
The proof Proposition is now finished.

---

Table 5: Side-by-side comparison between models with different quality weight values. DPP-TTS with quality weight 2.0 is used the Model A.
Model A/B
Same
B
Model A MOS
Model B MOS
DPP-TTS/DDPP-TTS---L..
24.3%
17.7%
57.8%
4.02 + 0.08
3.95 1 0.06
DPP-TTIS/DPP-TTS--W-..
44.5%
24.2%
31.8%
4.02 + 0.08
4.07 + 0.06
1001
57.6%
19.5%
22.9%
4.02 0.08
4.11 + 0.06
(a) Pitch plot with 10
(b) Pitch plot with W=
(c) Pitch plot with w=2
(d) Pitch plot with w=l
Figure 6: Pitch and log duration plots with different values quality weight. Green plots indicate prosody prediction before MAP inference DPP and red plots indicate the prosody prediction after MAP inference of DPP.
E Sample paragraph for the side-by-side comparison test
Known individually and collectively
ShaiHulud, the sandworms are these supermassive beings that plow through the deserts of Arrakis, consuming everything that dares venture unprepared into their territory. The worms are what make harvesting spice so difficult because they tend to eat whatever tools off-worlders use to mine it. They are also sacred to the Fremen, who seem to know ways to navigate around them, and, somehow, they're re linked the creation of spice. Think of them as big honking metaphors for the sublime powers of nature that loom beyond human understanding, like desert full of Moby Dicks
F Adjusting the extent of variation
Table 5 demonstrates that adjusting quality weight can trade-off between diversity and naturalness without having a severe impact on naturalness. The pitch and log duration plots are shown in Figure 6.


Reviews:
[{"review_id": "kcF9hTzLFG", "reviewer": "Reviewer_gArD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive and detailed comments. Here are responses regarding the concerns. \n\n**Reply reasons to reject**\n>No ablation experiments focusing on the PBD component were conducted\n\nWe conduct an experiment to demonstrate **the effectiveness of PBD over the fixed-length baseline**. While PBD *dynamically* adjust the length of the target considering prosodic boundary, the fixed-length baseline just extracts the target of fixed length.\n   \n|      Model A/B      |   A  | Same |   B  | Model A MOS | Model B MOS |\n|:-------------------:|:----:|:----:|:----:|:-----------:|:-----------:|\n|  DPP-TTS-d/baseline | 58.2 | 24.5 | 17.3 |  4.02 $\\pm$ 0.08  |  3.78 $\\pm$ 0.06  |\n| DPP_TTS-p/baseline  | 61.8 | 23.4 | 14.8 |  3.98 $\\pm$ 0.08  |  3.78 $\\pm$ 0.06  |\n|                     |      |      |      |             |             |\n\n(baseline= DPP-TTS w/o PBD)\\\nWe can see from the table, both the perceptual diversity and MOS of the baseline decrease compared to the model using PBD. It indicates the advantage of PBD which dynamically adjusts the target length considering the prosodic boundary. \n\n>For unclear reasons, the ancillary materials only provide two types of samples from the baseline models, and not all of them are included.\n\nSome samples from Diffspeech and Syntaspeech are provided in the [Demo Webpage](https://dpp-tts.github.io/). We will update the supplementary materials so that all samples subject to the evaluation are included. \n\n>I am curious about the advantages of the author's complex structural design compared to such simple yet effective approaches in real-world scenarios.\n\nAlthough the goal of the work in [1] and our work are both prosody modeling, there exists a clear distinction between the goals.  The goal of the work in [1] is to generate samples similar to the ground truth samples as possible. In contrast, our research aims to explore the perceptual diversity of audio samples beyond mere imitation of the ground truth, once a baseline level of naturalness has been achieved.Our research aims to explicitly diversify prosody based on the context, following the generation of prosodic features by the baseline prosody predictor. This suggests that PBD module can be seamlessly integrated as a plug-and-play component. Besides, as the number of parameters of PDM is only 0.3M and the inference time is still reasonable due to the non-autoregressive prosody modeling, the module can be easily integrated into the other TTS model.\n\n>In section 4, it is apparent that the MOS value of DPP-TTS is relatively low compared to the Syntaspeech.\n\nWe once again emphasize that our main goal is to achieve **better perceptual diversity**,  maintaining the naturalness of audio samples. We consider that the audio samples have achieved perceptual diversity if they show good performance in the side-by-side evaluation and the MOS does not degrade too much. When audio samples show good performance in the side-by-side evaluation and the MOS does not decrease too much, we consider that the audio samples have achieved perceptual diversity. The MOS value of Table 1 is served as supplementary metric since big MOS differences between samples may bias listeners to one sample without considering the diversity. Actually, if we increase the quality weight, it is possible to achieve a better MOS sample over the Syntasepech: \n| Model A/B             | Model A MOS | Model B MOS |   |   |\n|-----------------------|-------------|-------------|---|---|\n| Dpp-TTS-d/Syntapeech  | 4.07 $\\pm$ 0.07   | 4.04 $\\pm$ 0.09   |   |   |\n| Dpp-TTS-p/Syntaspeech | 4.09 $\\pm$ 0.07   | 4.04 $\\pm$ 0.09   |   |   |\n|                       |             |             |   |   |\n\nIf we are merely aiming for naturalness, we can achieve better MOS at the cost of losing some diversity. \n\n>The author should consider training and testing on larger and more diverse datasets. \n\nWe are conducting an additional experiment on the VCTK benchmark, and we will merge the results in the future manuscript and the [Demo webpage](https://dpp-tts.github.io). However, note that we also presented paragraph evaluation outside of the LJSpeech to demonstrate the model's generalization ability. Performing well on longer textual data sets,  which is different from LJSpeech, suggests that our model is not limited to a specific datasets.\n\n>Additionally, relying solely on MOS-P and MOS-D as indicators of speech quality may not be comprehensive, and factors such as speech naturalness should be taken into account. \n\nFor the evaluation of speech naturalness, the MOS-P metric has been used in many literatures [2,3,4]. Since the main focus of our research is the prosody aspect of speech, we stick to the prosody aspect of MOS rather than the original MOS. \n\n> Furthermore, more experiments should be conducted to compare DPP-TTS and DPP-TTS w/o PDM\n\nWe present an experiment result regarding the **quality weight $w$** of PDM. \n\n| Model A/B              | A    | Same | B    | Model A MOS | Model B MOS |\n|------------------------|------|------|------|-------------|-------------|\n| DPP-TTS/DPP-TTS-w=1.0  | 24.5 | 17.7 | 57.8 | 4.02 $\\pm$ 0.08   | 3.95 $\\pm$ 0.09   |\n| DPP-TTS/DPP-TTS-w=5.0  | 44.5 | 24.2 | 31.3 | 4.02 $\\pm$ 0.08   | 4.07 $\\pm$ 0.06   |\n| DPP-TTS/DPP-TTS-w=10.0 | 57.6 | 19.5 | 22.9 | 4.02 $\\pm$ 0.08   | 4.11 $\\pm$ 0.06   |\n\nDPP-TTS with quality weight $w=2.0$ is used for Model A. As the table demonstrates, we can trade off between diversity and naturalness without having a severe impact on naturalness.  These experimental observations sufficiently demonstrate the effectiveness of our methodology.\n\n[1] CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training. 2023 ACL (https://arxiv.org/abs/2305.10763) \\\n[2] PortaSpeech: Portable and High-Quality Generative Text-to-Speech. 2021 Neurips (https://arxiv.org/abs/2109.15166)  \\\n[3] SyntaSpeech:  Syntax-Aware Generative Adversarial Text-to-Speech. 2022 IJCAI (https://arxiv.org/abs/2204.11792) \\\n[4] Prosody-TTS: Improving Prosody with Masked Autoencoder and\nConditional Diffusion Model For Expressive Text-to-Speech. 2023 ACL (https://aclanthology.org/2023.findings-acl.508.pdf)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. To the best of my knowledge, the proposed method incorporating the PDM and DPP kernel represents a novel architecture compared to existing methods for the task of speech synthesis, despite the absence of a notable improvement.\n2. The author addresses a pivotal issue in prosodic modeling within the domain of speech synthesis, and provides a comprehensive discussion on the challenges associated with prosodic modeling. For instance, the range of targets for conditional DPP sampling lacks clarity, and prosodic features typically exhibit variations in length.", "reasons_to_reject": "The author's performance in the experiment is somewhat unsatisfactory.\n1. Ablation studies are missing. The author highlights the fine-grained level of the prosody modeling method based on PBD and its contribution to diverse prosody in each generated speech sample. However, no ablation experiments focusing on the PBD component were conducted.\n2. For unclear reasons, the ancillary materials only provide two types of samples from the baseline models, and not all of them are included.\n3. In my perspective, the PDM appears to be a plug-and-play component in the TTS system. Similar recent work, such as CLAPSpeech [1], has explored prosody modeling through contrastive learning of multiple sampling. I am curious about the advantages of the author's complex structural design compared to such simple yet effective approaches in real-world scenarios.\n\n[1] CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training. 2023 ACL (https://arxiv.org/abs/2305.10763)\n\n4. In Section 4, based on the experimental results presented in Table 1 for DPP-TTS-d/SyntaSpeech and DPP-TTS-p/SyntaSpeech, it is apparent that the MOS value of DPP-TTS is relatively low. However, the author has not provided an explanation for this observation. I am also curious about the author's interpretation of the MOS value of 4.09 ± 0.09 for DPP-TTS w/o PDM.\n5. The level of persuasiveness in the conducted experiment is insufficient. The author should consider training and testing on larger and more diverse datasets. Additionally, relying solely on MOS-P and MOS-D as indicators of speech quality may not be comprehensive, and factors such as speech naturalness (despite the author's analysis of some unfavorable cases) should be taken into account. Furthermore, more experiments should be conducted to compare DPP-TTS and DPP-TTS w/o PDM, which would provide evidence of the wide range of functionalities offered by the PDM model.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes the Prosody Diversifying Module(PDM), which aims to generate a more diverse and smooth prosody pattern in text-to-speech system.\n\nThis paper analyzes the limitations of some of the current work on prosodic modeling in the field of speech synthesis, points out that sampling with high temperatures may degrades the naturalness of speech and these work treat each sample independently. This paper attempts to solve these problem by proposing a prosody diversifying module (PDM) which is based on conditional determinantal point processes (DPP). This module is utilized in tandem with a Fastspeech2-based TTS model.  In addition, the authors extracted key segments by using Prosodic Boundary Detector (PBD) to get a more accurate target. The paper mainly shows Mean Opinion Scores (MOS) comparing the proposed approach against four strong models for rhythmic and pitch.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aCsPnyYHfC", "reviewer": "Reviewer_QLVm", "summary": "", "questions": "Quection A. In the subjective comparison of single sentences and paragraphs, how many audio samples were provided to the testers to score?\n\nQuection B. I am confused with the function of Section 6 Related Work, also, since both the introduction and related work part mentioned works that using VAEs to improve prosody, probably apart from flow and diffusion based baselines, another baseline based on VAE may be useful to be included.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's thorough and constructive comments. Here are the responses.\n\n**Reply to reasons to Reject**\n>The writing may need advision. For example, section 6 (Related Work) is highly similar to the Introduction's second paragraph.\n\nThank you for the feedback. We'll make sure to update the content so it doesn't overlap.\n\n>There is room for improvement in the experimental details.\n\nHere is an integrated quantitative evaluation including the baselines. \n\n| Model        | $\\sigma_p$ | Pitch Determinant | Inference time  |\n|-------------|---------|---------------|-----------------|\n| Flowtron    | 0.0121  | $6.76 \\cdot 10^{-14}$   | $5.3 \\cdot 10^{-1}$       |\n| DiffSpeech  | 0.0116  | $8.51 \\cdot 10^{-14}$   | $4.1 \\cdot 10^{-2}$       |\n| SyntaSpeech | 0.0133  | $2.23 \\cdot 10^{-14}$   | $3.9 \\cdot 10^{-2}$       |\n| DPP-TTS-p   | 0.0145  | $1.88 \\cdot 10^{-13}$   | $4.4 \\cdot 10^{-2}$       |\n\nIf the model does not have a separate duration predictor like VITS, it is challenging to evaluate the duration of phonemes, therefore only the pitch evaluation is included. \n\nOne of the main reasons we chose the LJSpeech is to demonstrate our model's ability to generate diverse prosody patterns beyond the ground truth of non-narrative dataset. Even on a dataset which lacks diverse prosody pattern, if we can generate diverse prosodic features that do not exist in the dataset, it would serve as an effective demonstration of our model's capabilities.\nIn addition, we also observe good performance in the paragraph evaluation which is not included in the LJSpeech. For the case of VCTK benchmark, and the audio samples will be included in the [Demo Webpage](https://dpp-tts.github.io/).\n\n**Reply to Questions**\n> In the subjective comparison of single sentences and paragraphs, how many audio samples were provided to the testers to score?\n\nIn the case of the LJSpech dataset, 75 audio samples that are randomly sampled from the held-out dataset are evaluated and 10 testers are assigned to each sample. For the comparison of paragraphs,  5 long paragraphs which are not included in the LJSpeech are used for generating audio samples.  Likewise the LJSpeech dataset, 10 testers are assigned to each audio sample. \n\n>Since both the introduction and related work part mentioned works that using VAEs to improve prosody, probably apart from flow and diffusion based baselines, another baseline based on VAE may be useful to be included.\n\nThank you for the suggestion. We experiment **VITS** because it is a sophisticated TTS model based on VAEs, consisting of a variational autoencoder backbone along with normalizing flow to enhance the prior.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors attempt to adopt DDP to improve prosody diversity in TTS, which is not only applicable to single sample diversity but also to the diversity among samples. This is a novel approach.\n\nThrough subjective experiments, the authors demonstrate that previous TTS models that increase temperature do not result in perceptual prosodic diversity but rather reduce naturalness.\n\nThe model is well-designed, including predictors of duration and pitch, PDM module, prosodic boundary detection, and an adaptive MIC loss function. A series of ablation experiments validate the effectiveness of each module and the MIC objective function.\n\nExperiments were implemented on various baselines, including Flowtron, VITS, DiffSpeech, which use Flow and Diffusion to enhance prosodic diversity, and Synta Speech. A series of ablation experiments have verified the effectiveness of the proposed modules and the MIC objective function.", "reasons_to_reject": "The writing may need advision. For example, section 6 (Related Work) is highly similar to the Introduction's second paragraph.\n\nThere is room for improvement in the experimental details. For example,  additional quantitative experiments compared to baselines (Flowtron, DiffSpeech, or SyntaSpeech) could be considered. As mentioned in line 525, LJSpeech, as a single-speaker non-narrative dataset, may lack sufficient prosody patterns. Alternatively, the VCTK multi-accent dataset or the LibriTTS multi-person novel dataset may be considered.\n", "questions_to_authors": "Quection A. In the subjective comparison of single sentences and paragraphs, how many audio samples were provided to the testers to score?\n\nQuection B. I am confused with the function of Section 6 Related Work, also, since both the introduction and related work part mentioned works that using VAEs to improve prosody, probably apart from flow and diffusion based baselines, another baseline based on VAE may be useful to be included.", "paper_topic_and_main_contributions": "The authors adopted DDP to improve prosody diversity in TTS, trying to improve perceptual prosodic diversity while maintaining naturalness and focusing on both the diversity of a single sample and among samples. Also, modified prosody predictor, prosodic boundary detector, and MIC objective function are proposed to ensure the effectiveness.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "When the MIC objective function first appeared in the full text of Line 78, it may be better to be fully named."}, "extra_text": {}}, {"review_id": "EAjTPqWHGN", "reviewer": "Reviewer_a3rZ", "summary": "", "questions": "1. How are the predicted prosody features being added to the main TTS system? It says in the appendix but I could not find it.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments for the paper. \n\n> The paper seems not complete: I can not find any appendices. How are the predicted prosody features being added to the main TTS system? \n\nThe appendix is included in the Supplemenrary Materials. The usage of prosodic features is similar to FastSpeech2 [1]. Predicted duration, pitch an energy features are added to the text encoder output in the hidden dimensions.\n\n>It is usually needed to have a demonstration webpage as TTS work needs subjective listening to demonstrate its superiority.\n\n[The demo webpage](https://dpp-tts.github.io) is released. Audio samples from LJSpeech and the paragraph are provided. \n\n[1] FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (https://arxiv.org/abs/2006.04558)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The conditional DPP used for diverse prosody modelling is novel and\n2. The derivations and explanations are plausible. \n3. Experiments are adequate", "reasons_to_reject": "1. The paper seems not complete: I can not find any appendices.\n2. It is usually needed to have a demonstration webpage as TTS work needs subjective listening to demonstrate its superiority.", "questions_to_authors": "1. How are the predicted prosody features being added to the main TTS system? It says in the appendix but I could not find it.", "paper_topic_and_main_contributions": "This paper proposes to use the Determinantal Point Processes (DPP) for diverse prosody modelling in text-to-speech synthesis. It proposes a prosody diversifying module that is trained based on conditional DPP.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper identifies the limitation of prior work on generating diversity prosody (pitch and duration) patterns, and presents a plug-and-play prosody diversifying module (PDM) to effectively tackle the issue. All reviewers agree that the authors address an important issue in speech synthesis, present a novel DPP-based solution that is well-designed and thoroughly evaluated when combined with various TTS backbones. 

The authors have presented many additional studies and samples during the rebuttal period, which sufficiently addressed reviewers concern on ablation studies. These results would be valuable to the readers and I suggest authors to incorporate them into the paper.
Decision: Accept-Main