DPP-TTS: Diversifying prosodic features of speech via determinantal point processes
Authors: Seongho Joo
Kyomin Jung
Venue: ICLR 2022 Submitted (2022)
Keywords: Text to speech synthesis
determinantal point processes
prosody modeling

Abstract: With the rapid advancement in deep generative models, recent neural text-to-speech models have succeeded in synthesizing human-like speech, even in an end-to-end manner. However, many synthesized samples often have a monotonous speaking style or simply follow the speaking style of their ground-truth samples. Although there have been many proposed methods to increase the diversity of prosody in speech, increasing prosody variance in speech often hurts the naturalness of speech. Determinantal point processes (DPPs) have shown remarkable results for modeling diversity in a wide range of machine learning tasks. However, their application in speech synthesis has not been explored. To enhance the expressiveness of speech, we propose DPP-TTS: a text-to-speech model based on a determinantal point process. The extent of prosody diversity can be easily controlled by adjusting parameters in our model. We demonstrate that DPP-TTS generates more expressive samples than baselines in the side-by-side comparison test while not harming the naturalness of the speech. 

Under review as a conference paper at ICLR 2022
DPP-TTS: DIVERSIFYING PROSODIC FEATURES OF SPEECH VIA DETERMINANTAL POINT PROCESSES
Anonymous authors Paper under double-blind review
ABSTRACT
With the rapid advancement in deep generative models, recent neural text-to- speech models have succeeded in synthesizing human-like speech, even in an end- to-end manner. However, many synthesized samples often have a monotonous speaking style or simply follow the speaking style of their ground-truth samples. Although there have been many proposed methods increase the diversity of prosody in speech, increasing prosody variance in speech often hurts the natu- ralness of speech. Recently, Determinantal point processes (DPPs) have shown remarkable results for modeling diversity in wide range of machine learning tasks. However, their application in speech synthesis has not been explored. To enhance the expressiveness of speech, propose DPP-TTS: text-to-speech model based on a determinantal point process to diversify prosody in speech. The extent of prosody diversity can be easily controlled by adjusting parameters in model. We demonstrate that DPP-TTS generates more expressive samples than baselines in the side-by-side comparison test while not harming the naturalness of the speech.
INTRODUCTION
In the past few years, text-to-speech models have made a lot of progress in synthesizing human- like speech (Shen et al. 2018: Ping et al., 2018 Li et al. 2019 Ren et al. (019). Furthermore, in the latest studies, several text-to-speech models made high-quality speech samples even in the end-to- end setting without two-stage synthesis process (Donahue et al. 2021 Kim al. (021). Based on these technical developments, text-to-speech models are now able to generate high-fidelity speech.
However, human speech contains many prosodic features like intonation, stress, and rhythm beyond textual features. Therefore, it is crucial to generate speech samples with rich prosody. There have been many attempts synthesize speech with rich prosodic features. To incorporate rich prosodic features into speech, reference acoustic samples like mel-spectrograms are processed through refer- ence encoder (Skerry- Ryan al. 2018 WWag al.] 2018) text-to- sppeehh-peeech models conditioned on prosodic features like duration, pitch and energy then these features are predicted or manually controlled inference (Ren et al., 2021 Lańcucki (021). However, these methods have drawbacks that the bottleneck dimension should be carefully tuned for desirable results, or the prosody predictor just learns averaged prosodic features of training sets.
Meanwhile, generative models like VAEs and flow models (Hsu et al. 2019 Sun et al. 2020 Vallés-
Pérez al. 202T: Lee al. 2021 Valle al. (02T) have been recently used for speech synthesis and their latent spaces are manipulated for generating more expressive speech. Controlling the amount of variation in the speech achieved by generating samples Gaussian prior with an ade- quate temperature, however, it has two major drawbacks. First, generating latent samples with high variance often make generated samples to be unstable in terms of the naturalness and intelligibility of speech. Second, sampling in the latent space might only cover major modes learned during the training stage and only monotonous prosodic patterns can occur in some segments speech.
Determinantal point processes (DPPs) have shown great results for modeling diversity in various machine learning tasks and their applications. Its uses include text summarization (Cho et al. recommendation systems (Gartrell et al. (22I). multi-modal output generation (Elfeki al. diverse trajectory forecasting (Yuan Kitani 2020), and machine translation (Meister et al. Determinantal point process (DPPs) offers efficient subset selection method by considering
2019
2019
2021

---

Under review as a conference paper at ICLR 2022
both the quality and diversity of items within set. Specifically, items within a set sampled according to DPPs kernel which reflects the quality and diversity of items in the ground set. In some cases, we may need to incorporate the conditional information into the sampling process. To this end, conditional DPP (Kulesza & Taskar 2012) has been widely used for sampling subset by setting the kernel conditioned on the specific information.
To use conditional DPP for sampling prosodic features of speech, two main issues should be con- sidered. First, prosodic features are usually varying in length. Second, it is ambiguous what the ground set is in this problem. In this work, we resolve these issues by using a new similarity metric between prosodic features and adding prosodic diversifying module (PDM) into the framework. Specifically, PDM generates candidates of prosodic features of speech by mapping latent codes from normal distribution to a new latent space and these candidates are used as the ground set in the sampling process. In addition, the similarity between two prosodic features is evaluated using soft dynamic time warping discrepancy (Soft-DTW) (Cuturi & Blondel 2017] which enables to evaluate the similarity between features in different lengths.
The kernel matrix of conditional DPP learned during the training by parameters of PDM getting updated. Specifically, parameters of the prosody diversifying module (PDM) are updated in the training stage with conditional maximum induced cardinality (MIC) objective which is adapted from MIC objective introduced in Gillenwater et al. (018). We formally introduce the conditional MIC objective and its derivative for clarity in Section
To implement the aforementioned DPP, stochastic duration predictor and pitch predictor intro- duced in this work. Both the duration and pitch predictor are built upon normalizing flows trained with the maximum likelihood (MLE) objective. At inference, the prosody predictor maps latent codes from a latent space to feature spaces with the inverse flow.
In experiments, we compare DPP-TTS with the state-of-the-art models including VITS (Kim al., 2021] and Flowtron (Valle et al. 2021) in terms of prosody diversity and speech quality. The results demonstrate that our model generates speech with richer prosody than baselines while maintaining speech naturalness. We also demonstrate that our DPP-TTS can be used in real-time applications by evaluating the inference speed of our model.
In summary, our contributions follow:
We propose a novel method for diversifying prosodic features of speech based on condi- tional DPPs by considering prosodic features of context words conditions.
To learn the kernel matrix of conditional DPPs, we propose train prosody diversifying module (PDM) with the conditional maximum induced cardinality (MIC) objective.
Experiments on the side-by-side comparison and the MOS test verify that our model out- performs the baseline models in terms of prosodic diversity while maintaining the naturalness of speech.
2 BACKGROUND
2.1 DETERMINANTAL POINT PROCESSES
DPPs encourage diversity within a set by discouraging sampling similar items within the ground set. Formally, point process P is called determinantal point process when Y is random subset drawn according to P. we have, for every A C y,
C(ACY) x (,),
(1)
where K is positive definite matrix whose eigenvalues are all between 0 and I and KA is a positive definite matrix indexed by elements in A. The marginal probability of including two elements e, and éj is Kūj - K3 = p(e: E Y)p(e; E Y) K2-- Therefore, the value of K? models the extent of negative correlation between item and j. More frequently, DPPs are defined by L-ensemble through real and symmetric matrix L instead of the marginal kernel K:
det Ly)
PL = Y) det( L + I))
(2)
2

---

Under review as a conference paper at ICLR 2022
where det(L I) in the denominator acts as normalization constant. The marginal kernel K and kernel has the following relation:
K = (
(L (L+
To model diversity between items, the DPP kernel L is usually constructed symmetric similarity matrix, where Sij represents the similarity between two items 2; and zj. In Kulesza & Taskar (2010), authors proposed decomposing the kernel L as a Gram matrix incorporating quality vector weigh each item according to its quality:
P(J CY) (J) II (ee) Diag(g) S Diag(g),
6,EJ
where d, E D N and ||o/|12 = 1. this manner, the similarity matrix is guaranteed to be real positive semidefinite matrix.
2.2 CONDITIONAL DETERMINANTAL POINT PROCESSES
If DPPs are used for diversifying prosodic features by setting targets of DPPs as prosodic features of sentences, it would result in diversity among generated samples. However, there still can be monotonous patterns in each generated speech. To resolve this issue, it is required to diversify prosodic features accounting into their neighbor prosodic features. Therefore, we need conditional DPPs to take into account neighboring prosodic features (contexts).
By setting conditions of point process P, DPPs can be extended to conditional DPPs, For subset B F Y not intersecting with A we have
P(Y AUBIACY) P(Y P(Y= AUB) det(L
(5)
P(ACY) det(L + IA)
where A is the matrix with ones in the diagonal entries indexed by elements of y A and zeros elsewhere. In Borodin & Rains (2004), authors showed that this conditional distribution is again DPP, with a kernel given by
L LA= [A[(((++ (1+++))~~))--
(6)
In conditional DPPs, items in the ground 3 sampled according kernel given contexts. In this work, contexts of target words are used as conditions for conditional DPPs.
2.3 SOFT DYNAMIC TIME WARPING
To build the kernel of conditional DPPs, we need a measure of similarity between two temporal signals. Simple Euclidean distance is not applicable because two time signals often vary in their lengths. In this work, soft dynamic time warping (Soft DTW) discrepancy which allows measuring the similarity of shape between time series of different lengths is adapted to calculate the similarity between prosodic sequences. DTW computes the best possible alignment between two temporal signals. First, for the length of and signals, DTW computes by pairwise distance matrix between these points with a specific metric (e.g., Euclidean distance, L1 distance). After that, this matrix is used to solve the dynamic program using Bellman's recursion with quadratic ((m)) cost.
Unfortunately, vanilla DTW cost is not easy optimize because it only considers single alignment between two temporal signals. Cuturii & Blondel (2017) proposed differentiable Soft-DTW which considers all possible alignments between two temporal series. Soft-DTW is differentiable in all of its arguments with quadratic cost. Formally, Soft-DTW is defined as follows:
dtw, y) min" ((A,80 8(x, v)), An,m.
where An.m denotes all possible alignments between x and y and min" is defined as
min
if 7=0 otherwise
min" (.1,-... 0n) log Σn
small magnitude of y reflects the true discrepancy two temporal signals, however, optimization becomes unstable.
3

---

Under review as a conference paper at ICLR 2022
Mel- speestrccto Decoder
Inforence
Expand
Prosodic Fentures
Posttional
Duration predictor
Encoding
Duration
DPP Komel build
Distribution
Varlance Adaptor
Duration or Pitch predictor
Pitch predictor
Sequance
Stop Gradient
Phoneme Encoder
Normal Distribution
Phoneme Prenet
Phoneme Embeddings
(b) Variance adaptor at inference of DPP-TTS
(a) Base text-to- text-to-speech speech model training
Diagrams describing (a): training of base text-to-speech model and (b) variance adaptor at the inference DPP-TTS after PDM added.
Figure 1:
DPP-TTS
Our model DPP-TTS is composed of Seq2Seq module for generating the mel-spectrogram, a prosody predictor for predicting duration and pitch sequences, and prosody diversifying module (PDM). At the first stage, the base TTS model which consists of the Seq2Seq module and the prosody predictor is trained shown in Figure Ta Once the base TTS is trained, PDM is inserted in front of the prosody predictor and trained with the method which will be described detail in Section We describe the main modules of DPP-TTS and their roles in the following subsection.
3.1 MAIN MODULES OF DPP-TTS
Seq2Seq module The role of the Seq2seq module is generating mel-spectrograms from phoneme sequences. The module is adapted from FastSpeech2 (Ren et al., 2021) with some modifications. The model consists of four main parts: phoneme prenet, a phoneme encoder, a variance adap- tor, and mel-spectrogram decoder. In the phoneme encoder, phoneme sequences processed through a stack of feed-forward transformer blocks with a relative positional representation (Shaw) al. 2018). In variance adaptor at training, pitch embeddings and energy embeddings are added to encoded hidden representations and then hidden representations are expanded according to ground- truth duration labels |] At inference, these prosodic features are provided from predictions of the prosody predictor. Finally. expanded representations are processed through a stack of feed-forward transformer blocks and mel-spectrograms with 80 channels are generated after the linear projec- tion. The Seq2Seq module is trained to minimize T, distance between the predicted and target mel-spectrogram.
Prosody predictor In FastSpeech2, the variance adaptor consists of deterministie predictors for predicting prosodic features. However, deterministic prosodic predictor is not expressive enough learn the speaking style of a person. For diverse rhythm and pitch, stochastic duration predictor and pitch predictor are built upon normalizing flows. Specifically, the stochastic duration predictor estimates the distribution of phoneme duration and the stochastic pitch predictor estimates the dis- tribution of phoneme pitch from the hidden sequence. At the training stage, the prosody predictor learns the mapping from the distribution prosodic features to normal distribution. At inference, it predicts the phoneme-level duration pitch by reversing the learned flows. In addition, it also
"Ground-truth labels are obtained via monotonic alignment search Kim et al. 2020 between the phonemes and mel-spectrogram.
4

---

Under review as a conference paper at ICLR 2022
serves the density estimator for prosodic features during the training of PDM which will be de- scribed in detail in Section 4. The prosody predictor is trained to maximize a variational lower bound of the likelihood of the phoneme duration or pitch. More details regarding the prosody predictor in Appendix A
PDM Although the stochastic duration and pitch predictor are trained to generate a speech with diverse rhythm and pitch, the prosody predictor may favors major modes and it leads to the monotonous prosodic pattern in the speech. For more expressive speech modeling, PDM is added in front of the prosody predictor shown in Figure Tb Its role guide latent codes from a standard normal distribution to another distribution for diverse prosodic features of speech. This module is trained with an objective based on conditional DPPs which is described in Section 2.2 At inference of DPP-TTS, multiple prosodic candidates are generated by PDM and the prosodic feature of speech selected via MAP inference.
Normal Distribution
Target indices
indices
[12--19),(22221, 52-67 .-]]
[[5-11],44--3]__]
PDM
Irulan
Shall opening Nife, opening for an way for, to take
Target and Context Suquance
Duraffon or pilch predictor
ay wiffe
the
Extract
Predicted Sequences
Sequences target
an Atreides
the throne
Store Indices
Hidden Sequence
Target Hit
Context
10g
Sa Segmantation
PreNel Encoder
DPP Kernel build
Loss
Irulen shell be my wiffe, upening The way for an Atreides to take the throne.
Irulan shall be udfe, way for Abreides to take the throne.
Input text
Input
(a) Segmentation an input text
(b) Training procedure of PDM
Figure 2: Diagrams describing (a): segmentation of an input text for generating target and context indices and (b) training procedure of PDM.
4 DIVERSIFYING PROSODIC FEATURES OF SPEECH WITH PDM
In this section, we explain the methodology for training the prosody diversifying module (PDM). The training process mainly consists of three steps: segmentation of an input text, generation of prosodic feature candidates, building DPP kernel. We also explain the conditional maximum in- duced cardinality (MIC) objective for training PDM. The overall training procedure is depicted in Figure 2b
4.1 SEGMENTATION OF AN INPUT TEXT
In this stage, targets in input text for diversification of prosody are chosen. The scale of each target is important for the desired goal. Phoneme-level targets would result in too dynamic prosody of generated speech. In contrast, sentence-level targets may result in a monotonous pattern in each speech corresponding to each sentence. Therefore, we choose a few words as each target in the input text. However, naively choosing random words in the input text is not desirable because prosodic boundaries and syntax of the sentence are closely related (Cole et al. 2010). We choose the noun phrases the input text as each target using Spacy P library to take account into the syntactic structure of the input text. Neighboring words with the same number of words the target are selected as the left of right context. For example, as shown in Figure 2a if the target is an Atreides, then way for and to take selected as the left and right contexts respectively. Indices of targets and contexts are stored after the segmentation.
5

---

Under review as a conference paper at ICLR 2022
4.2 GENERATION OF PROSODIC FEATURE CANDIDATES
In this stage, multiple prosodic candidates are generated for the ground set of DPP as shown in Figure 2b First, an input text is fed into the pre-trained phoneme prenet and encoder and hid- den sequence is generated. Second, the pre-trained prosody predictor conditioned on the hidden sequence predicts the whole prosodic features of the imput text from the latent codes samples from a normal distribution. Third, new latent codes from a normal distribution are fed into PDM, then the prosody predictor conditioned on hidden sequence corresponds the target and neighboring context predicts new n. prosodic features. Finally, newly generated target prosodic sequences and previously predicted prosodic sequences are mixed, then n. prosodic candidates are generated with the target and context entangled.
4.3 CONSTRUCTION OF DPP KERNEL
Generated candidates are split into left and right context dr, dR and targets di, d2....nm then the ground for DPP is constructed merging the targets and contexts as shown in F The kernel of conditional DPP is built incorporating both diversity and quality of the ground set. mentioned in Section 2.1 the kernel of DPP can be de- composed as L, = Diag(q) (q) S Diag(q). (q), where denotes the quality vectors of predicted features and S denotes the similarity matrix.
6x6 DPP kernel
Density values
Soft Discrepancy
Duration or Pitch predictor
attention
Similarity metric Target sequences and context se- quences often vary in lengths, therefore the Huclidean dis- tance is not applicable to calculate the similarity between two sequences. The similarity metric is calculated as fol- lows based on Soft DTW:
Ground set
Ground set construction
(9) dtw denotes soft-DTW discrepancy with a metric D and smoothing parameter y. When the metric D is chosen as the 1, distance (xx, x y) = Σ; i yi or half gaus- sian D(x, y) = ||0 y 2 + log - exp(-Ilæ - - y)())), the similarity matrix is positive semi-definite (Cuturi
St exp(-k dtw y (d;, d,))
prosodic candidates
Figure 3: Construction of DPP kernel
with ne = 4
jtt al. 2007 Blondel et al., (021). In this work, L, dis-
tance is used as the metric of Soft-DTW that S to be positive semi definiefeete.
Quality metric Based on the estimated density of predicted features, quality scores are calculated. The density values of predicted features are calculated with importance sampling using the prosody
predictor: p(x; ()) ΣE= g(=, (2:d) In the experiment, using density value resulted in not enough diverse sample, therefore quality values are not penalized if the density value is greater than specific threshold. With log likelihood π (ιx) = log p(z), the quality score is defined as
N p(æ.z,;9)
if (()) >= k
(æz)
(10)
w exp(T(x) k) otherwise
The threshold value k set the average density of the dataset in the experiment. Finally, the kernel of conditional DPPs is defines as L diag(g): S diag(g).
4.4 OBJECTIVE FUNCTION
We need a measure of diversity with respect to kernel L to train the PDM. One straightforward choice is maximum likelihood (MLE) objective, log PY) log det(L) log det + I). However, there are some cases where almost identical prosodic features are predicted. This leads the objective value be close zero which makes the training process unstable. Instead, maximum induced cardinality (MIC) (Gillenwater et al., 2018) objective which defined as Ey can be an alternative. It does not sufter from traiing instability since identical items in the ground

---

Under review as a conference paper at ICLR 2022
set )' are allowed. In this work, conditional MIC objective is used as setting contexts di, dæ as the condition. The objective function with respect to the ground set [dz, dR, d1, d2, .... dyl and its derivative are as follows:
Proposition (MIC objective of conditional DPPs). With respect to the ground set [dz, dR. d1, d22 ... dy}, the MIC objective of conditional DPPs and its derivative are as follows:
aLMIC ((L+IA)
LMIC = =t((1 - [LB + (A)),
+ +A)
(11)
00
where denotes the set of contexts (dz, dn), A denotes the complement of the set A and is the parameter of PDM.
Detailed proof is presented Appendix B At inference, for predicting a single prosodic fea- ture, MAP inference is performed across sets with just single item as follows: x² arg = log det Lijju)). The detailed procedure of training PDM and the inference of DPP- TTS are in Appendix
5
EXPERIMENTS
In this section, we first explain the experimental setup and then human evaluation results and further evaluation results that are conducted to evaluate the proposed DPP-TTS. Audio samples used for the evaluation are in the supplementary material.
5.1 EXPERIMENTAL SETUP
Dataset and preprocessing We conducted experiments on the LJ Speech dataset which consists of audio clips with approximately 24 hours lengths. We split audio samples into 12500/100/500 samples for the training. validation. and test set. Audio samples with 22kHz sampling rate are transformed into 80 bands -peettrecss through the Short-time Fourier transform (STFT) with 1024 window size and 256 hop length. International Phonetic Alphabet (IPA) sequences are used input for phoneme encoder. Text sequences are converted to IPA phoneme sequences using P Phonemi ze zeifi software. Following Kim et al. (2020). the converted sequences are interspersed with blank tokens, which represent the transition from one phoneme another phoneme.
Training Setup Both the basic TTS and PDM are trained using the Adam optimizer (Loshchilov
& Hutter 2019) with B1 0.8, 82 0.99 and À 0.01. The initial learning rate is 2 x 10 for the basic TTS and x 10-5 for the PDM and the exponential learning decay used for both of them. Basic TTS is trained with 16 batch size for 200k steps 2 NVIDIA RTX 3080 GPUs. PDM is trained with batch size for 5k steps on same devices. both the duration and pitch diversification modeling, the quality weight of the model is set as 10.
Baseline We compare our model with two state-of-the-art models in terms of naturalness and prosody diversity. The first one is VITS (Kim et al.) (02T) which is an end-to-end TTS model based conditional VAE and normalizing flows. In all experiments, the temperature of the duration predictor in VITS was set 0.8. The second one is Flowtron (Valle et al., (02T) which is an autoregressive flow-based TTS model. In experiments. the temperature o of Flowtron was set to 0.6. For our DPP-TTS and Flowtron, HiFi-GAN (Kong et al. 2020) is used as the vocoder for synthesizing waveforms from the mel-spectrograms.
5.2 HUMAN EVALUATION RESULTS
First, we conduct a side-by-side evaluation of the prosody of a sentence. In this experiment, texts from test samples from LJSpeech are synthesized. Target phrases extracted from the sentence are processed by PDM and pitch diversity modeling is used in DPP-TTS. Via Amazon Mechanical Turk (AMT), we assign five testers living in the United States to a pair of audio samples (i.e., DPP-TTS and baseline), and ask them to listen audio samples and choose among three options: A: sample A has more varied prosody than sample B, Same: sample A and B are equally varied in prosody, B: the opposite of option (I).
Side by side evaluation

---

Under review as a conference paper at ICLR 2022
In addition, we also conduct side-by-sid evaluation of prosody in a paragraph that is not included in the LJSpeech dataset since a monotonous pattern is prominent in long texts rather than a single model that has only duration diversifying module and DPP-TTS2: a model that has both duration and pitch diversifying modules. In this experiment, only VITS is compared as the baseline. Likewise the evaluation of sentence, listeners are asked to choose among the three options.
sentence. We prepare two DPP-TTS versions: (I): DPP-TTSI:
Results are shown in Table 0 In the side- by-side evaluation of prosody in à sentence, DPP-TTS outperforms VITS and Flowtron. In the side-by-side evaluation of prosody in a paragraph, both the DPP-TTS1 and DPP- TTS2 outperform the baseline.
Table I: Side-by-side comparison on the LJSpeech dataset.
Model A/B
A
Same
B
DPP-TTS/Flowtron
55.3%
12.7%
32.0% 28.0%
DPP-
40.0%
32.0%
Naturalness test this experiment, we measure the Mcan-Opinion-Seore (MOS) for audio samples evaluated in the side-by- side evaluation. Likewise in the previous test, five testers are assigned to each audio sample. Testers are asked to give score be- tween 1 to on a 9-scale based on the sam- ple's naturalness. We used the same qual- ity weight value w = 10 for DPP-TTS as in the side-by-side comparison test. MOS- S denotes the quality of audio from the L.Speech dataset and MOS-P denotes the quality of audio from the paragraph. The MOS result is shown in Table 3 In MOS- S, our model gets lower MOS than VITS, it yet still maintain a MOS of 3.92 and outperforms Flowtron. In MOS-P, our model outperforms
Table 2: Side-by-Side comparison on the paragraph.
Model A/B
A
Same
B
DPP-TTST/VITS
56.25%
6.25%
37.5% 37.5%
DPP-TTS2/VITS 50.0%
12.5%
Table 3: Mos evaluation results with 95% confidence intervals.
Model
MOS-S
MOS-P 3.06 + 0.52
VITS
4.08 0.53
Flowtron
3.81 + 0.72
DPP-TTS
3.92 + 0.58
3.13 0.54
Ground-truth
4.46 + 0.51
VITS. A significant reduction of MOS in MOS-P compared to MOS-S indicates that challenging to maintain the naturalness in long texts.
5.3 ADDITIONAL EVALUATION
In this section, we conduct additional evaluations for our DPP-TTS and VITS We use the following metries for the evaluation (1) a frame: frame-level standard deviation of the pitch in speech. All pitch values are normalized since speech samples have different absolute pitch values. Standard deviations are averaged over 50 generated samples. DIO algorithm (Morise et al., 2009) is used for pitch extraction. (2): pponneme phoneme-level standard deviation of duration pitch in speech. Likewise, the standard deviation values are averaged over 50 generated samples. (3): Determinant: determinant of the matrix with respect to feature set J generated by multiple samples is used to evaluate the diversity of phoneme-level prosodic features of multiple generated samples. Cosine metric between two sequences. Each set is constructed from 10 generated samples. (4): Inference time: the inference time for synthesizing one-second length waveform is calculated in the text-to-speech model. The inference speed is evaluated on Intel(R) Core(TM) i7- 7800X CPU and a single NVIDIA RTX 3080 GPU. Computation time is averaged over 100 forward
similarity is used
passes.
Model
Piich frome
Pitch Pphoneme
Pitch Determimunt
Duration phooneme
Duration Determinant Inference Timels)
DFP-TTS
0.012
0.014
2.92 x 10
1.81
7.21 10
1.1 x 10
VITS
0.010
0.011
5.38 x 10
0.22
6.68 10
1.3 x 10-2
Table 4: Additional evaluation results. Numbers in bold denote the better result.
Table 4 shows the result of frame: Ohoneme. determinant and inference time. At both frame and phoneme levels, the standard deviation of the pitch in DPP-TTS results in higher values than the baseline. In addition, the standard deviation of duration at phoneme-level in DPP-TTS also results in a higher value than the baseline. It demonstrates that DPP-TTS generates speech with more
Hyperparamettr settings remain the Section 5.2
8

---

Under review as a conference paper at ICLR 2022
dynamic pitch and rhythm than the baseline. The determinants of duration and pitch sets of DPP- TTS outperform the bascline. shows that DPP-TTS generates more diverse samples in terms of prosody than bascline. Finally, DPP-TTS results in 0.044 seconds of inference speed. Although its inference speed is slower than the baseline, our model is applicable in practice since the inference speed of our model is 22.7x faster than real-time.
5.4 MODEL ANALYSIS
Ablation study We conduct ablation studies to verify the effectiveness of PDM, the results are shown in T5 In the side-by-side comparison, 34% more listeners choose the option that DPP- TTS has more varied prosody compared to DPP-TTS without PDM. It demonstrates that PDM contributes to prosody diversification significantly. DPP-TTS also achieves a higher phoneme-level standard deviation of pitch and duration than the baseline. Although there is quality degradation (-0.26 MOS) compared to the baseline, DPP-TTS still achieve a MOS of 3.92.
Side-by-Side Comparison
Model
MOS
Pitch phoneme
Duration phoneme
DPP-ITS
58.5%
3.9210.58
0.014
L881
17.0%
DPP-TTS PDM
4.18 0.49
0.01
1.76
24.5%
Table 5: Comparison in the ablation studies.
Adjusting the extent of variation To study the impact of quality weight w. we plot (normalized) pitch and log duration values phoneme-level with different values of quality weight. As expected, for a large magnitude of quality weight, the model learns diversifying prosodic features with small deviations from contexts. In contrast, for small magnitude of quality weight, the model diversifies prosodic features with large deviations from contexts. We also found that a small magnitude of quality weight leads to faster convergence of the model but it easily hurts the naturalness of generated samples. Figures are shows in Appendix
Effeet of number of candidates on diver- sity In this experiment, we study the ef- the
Number of candidates
Pitch determinant
feet of the number of candidates prosody diversity in generated samples. For different number of candidates n, 4, 8, 12, DPP-TTS is trained up to 4k steps and all other hyperparameters remain un- changed. After training, we calculate pitch determinants over generated samples for each setup. The result is reported in Table 6 We ob-
4
1.69 x 10-15
4.56 x 10-15 15
ne =
7.8 X 10-15
nee
Table 6: Pitch determinant values with different num- ber of candidates
serve that using a large number of candidates results in more diverse samples although only single prosody item is selected at MAP inference. We also find that using a large number of candidates has regularization effect for training. Using small number of candidates generates unstable speech samples terms of naturalness, in contrast, using a large number of candidates results in more stable speech samples.
6 CONCLUSION
In this paper, we have proposed a novel prosody diversifying method based on conditional determi- nantal point processes (DPPs). To learn the kernel matrix of conditional DPPs, conditional deter- minantal point process (DPPs) is parameterized additional prosody diversifying module (PDM). To build the kernel, Soft dynamic time warping is adopted to measure the extent of similarity between two prosodic features, and the kernel of conditional DPPs is learned with conditional maximum induced cardinality (MIC) objective. In the experiment, we demonstrated that our new prosody diversifying method generates more dynamic samples than the baseline while maintaining the natu- ralness. In addition, we demonstrate that our model can be used in real-time applications by showing the inference speed of our model.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. abs/ abx,v,,bb/160000445505 2016.
Mathieu Blondel, A. Mensch, and Jcan-Philippe Vert. Differentiable divergences between time series. In AISTATS, 2021.
Alexei Borodin and Fric M. Rains. Eynard-mehta theorem, schur process, and their pfaffian analogs. Journal of Statistical Physics, 121:291-317, 2004.
Jianfei Chen, Cheng Lu, Biqi Chenli, J. Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. ArXiv, abs/2002.09741, 2020.
Sangwoo Cho, Chen Li, Dong Yu, H. Foroosh, and Fei Liu. Multi-document summarization with determinantal point processes and contextualized representations. ArXiv, abs/1910.1 1411,, 2019.
Jennifer S. Cole, Yoonsook Mo, and Soondo Back. The role syntactic structure in guiding prosody perception with ordinary listeners and everyday speech. Language and Cognitive Processes, 25: 1141 1177, 2010.
Marco Cuturi and Mathieu Blondel. Soft-dtw: a differentiable loss function for time-series. In ICMI, 2017.
Marco Cuturi, Jean-Philippe Vert, Øystein Birkenes, and T. Matsui. A kernel for time series based on global alignments. 2007 IEEE International Conference on Acousties, Speech and Signal Processing ICASSP '07, 2:11-413-11-416, 2007.
Jeff Donahue, S. Dieleman, Mikolaj Binkowski, Brich Elsen, and K. Simonyan. End-to-end adver- sarial text-to-speech. ArXiv, abs/2006.03355, 2021.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In NewrlPS, 2019.
M. Elfeki, C. Couprie, M. Rivière, and Mohamed Elhoseiny, Gdpp: Learning diverse generations using determinantal point process. In ICML, 2019.
Mike Gartrell, Insu Han, Elvis Dohmatob, Jennifer Gillenwater, and Victor-Emmanuel Brunel. Scalable learning and map inference for nonsymmetric determinantal point processes. ArXiv, abs/2006.09862. 2021.
Jennifer Gillenwater, Kulesza, Sergei Vassilvitskii, and Zelda Mariet. Maximizing induced cardinality under a determinantal point process. In NeurlPS, 2018.
Jonathan Ho, Xi Chen, A. Srinivas, Yan Duan, and P. Abbeel. Flow++: erative models with variational dequantization and architecture design. ArXiv, Abs//190..00255, 2019.
Improving flow-based gen-
Wei-Ning Hsu, Y. Zhang, Ron J. Weiss, H. Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Z. Chen, Jonathan Shen, P. Nguyen, and Ruoming Pang. Hierarchical generative modeling for controllable speech synthesis. ArXiv, abs/18 abs//810.007.1, 2019.
Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: generative flow for text-to-speech via monotonic alignment search. ArXiv, abs/2005.1 11 2020.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. ArXiv, abs/2106.06103. 2021.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 77022-17033. Curran Associates, Inc., 2020. URL ht
10

---

Under review as a conference paper at ICLR 2022
A. Kulesza and B. Taskar. Determinantal point processes for machine learning, Found. Trends Mach. Learn., 5:123-286, 2012.
Alex Kulesza and Ben Taskar. Structured determinantal point processes. In NIPS, 2010.
Adrian Lañcucki. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP, 2021.
Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. Bidirectional variational inference for non- autoregressive text-to-speech. In ICLR, 2021.
N. Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with trans- former network. In AAAI, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer-
ence on Learning Representations, 2019. URL htt.::ttpp///tc /ettpps:/oppnreviii ew net. www.ne/fforrm! fwr.mrrrummmmmmm
3kk6RicqYY
Clara Meister, Martina Forster, and Ryan Cotterell. Determinantal beam search. In ACL/IUNLLP 2021.
Diganta Mísra. Mish: A self regularized non-monotonic neural activation function. 0ess/190.00681// 2019.
ArXiv,
Masanori Morise, Hideki Kawahara, and Haruhiro Katayose. Fast and reliable estimation method based the period extraction of vocal fold vibration of singing voice and speech. 2009.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan Ö. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: Scaling text-to-speech with convolutional sequence learning. arXiv: Sound, 2018.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In NeurIPS, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text speech. ArXiv, abs/22006.04558, 2021.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In NAACL, 2018
Jonathan Shen, Ruoming Pang, Ron J. Weiss, M. Schuster, Navdeep Jaitly, Zongheng Yang, Z. Chen, Yu Zhang, Yuxuan Wang, R. Skerry-Ryan R. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783, 2018.
R. Skerry-Ryan, Eric Battenberg, Y. Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J. Weiss, R. Clark, and R. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. ArXiv, abs/1803.09047, 2018.
G. Sun, Y. Zhang, Ron Weiss. Yuanbin Cao, H. Zen, and Yonghui Wu. huraa-hherarl fine- grained prosody modeling for interpretable speech synthesis. ICASSP 2020 2020 IEEE Interna- lional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6264-6268, 2020.
Rafael Valle, Kevin J. Shih, R. Prenger, and Bryan Catanzaro. Flowtron: an autoregressive flow- based generative network for text-to-speech synthesis. ArXiv, abs/2005.05951 2021.
Iván Vallés-Pérez, Julian Roth, Grzegorz Beringer, R. Barra-Chicote, and J. Droppo. Improv- ing multi-speaker uts prosody variance with residual encoder and normalizing flows. ArXiv, abs/2106.05762, 2021.
Yuxuan Wang, Daisy Stanton, Yu Zhang, R. Skemy-Ryan, Eric Battenberg, Joel Shor, Y. Xiao, Fei Ren, Ye Jia, and R. Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In ICMI, 2018.

---

Under review as a conference paper at ICLR 2022
Yunyang Xiong, Zhanpeng Zeng, Vikas Singh. Nyströmformer: A nystróm-based algorithm for approximating self-attention. In AAAI, 2021.
Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn M. Fung, Yin Li, and
Ye Yuan and Kris Kitani. Diverse trajectory forecasting with determinantal point processes. ArXiv, abs/1907.04967 2020.
12

---

Under review as a conference paper at ICLR 2022
A DETAILS OF PROSODY PREDICTOR AND PDM
Split
Conv 1x1
2
Labent
Dilated Comv
Nyströmer
Mish
Concat
Conv 1x1
LayerWors
Conv
Mish
LayerNors
LayerNorm
encoder
Cansat
Conv 1x1
Dilated Conv
+
Spline flows
Dilated ConvBlock
Flow gO
Spline flow paramaters
Condttion
Encoder
Conv 1x1
Conv 1x1
Concal
Hidden inqunce
M20LE
Hidden
Neise
(a) Duration predictor
(b) Coupling block
(c) PDM block
Figure 4: Diagrams describing (a): the duration predictor, (b): Coupling block in normalizng flows
and (c): Prosody diversifying module
A.I PROSODY PREDICTOR
Training is hard to use the maximum likelihood objective directly to train duration predictor because the duration of each phoneme is 1) discrete integer and scalar, which hinders expressive transformation because invertibility should remain in normalizing flow. To train duration predictor, duration values extended to continuous values using variational dequantization (Ho al. 2019) and are augmented with extra dimensions using variational data augmentation (Chen et al. 2020
Specifically, duration sequence d becomes continuous d u where u's value restricted [0
and augmented as [d - v] with extra random variable v. Two random variables u and are sampled through approximate posterior qo(u, v/d, httxt). The ELBO can be calculated as follows:
Pe dd u. 2 v(hhezt)
log pe (d htezt) Eog((,,odd,hrrz))) [log
(12)
(u, v vlld, hreat)
Like the duration predictor, the ELBO for pitch predictor can be calculated as follows:
po (p, (iteet)
log pe (p(pertezt) 2 (rp. [log
(13)
2 (vlp, breat)
where p denotes the pitch sequences. Both the duration and pitch predictor are trained on negative lower bound of likelihood.
Architecture The pitch predictor and duration predictor share identical architecture except for the additional random variable u. We will introduce the architecture of the duration predictor whose diagram is shown in Figure 4a Duration predictor consists of condition encoder for hidden se- quence, posterior encoder for the duration sequence, and the main flow blocks g(-). Specifically, posterior encoder maps latent codes from normal distribution to random variable [u, u] and flow g(-) maps [d - u, v) to normal distribution. Figure 4b shows the coupling normalizing flows. First, input is split into ( 21] and then .n is processed by 1x1 convolution block. The output of the convolution block is processed by dilated convolution and Nystromer block and their outputs are concatenated. The concatenated output is processed by LayerNorm, Mish activation. and Convolu- tion block. Finally, spline flows (Durkan et al. 2019) are parameterized by the output and then the output of spline flows y and 2o concatenated.
A.2 PDM
The architecture PDM is shown Figure First, the hidden sequence is processed by convolu- tion block followed by the dilated convolution block conditioned hidden sequences, Layernorm,
13

---

Under review as a conference paper at ICLR 2022
and Ixl convolution block. After that, noise from normal distribution is processed by 1x1 convo- lution followed by dilated convolution block, LayerNorm, and Mish activation.
As the prosody pattern of human speech is correlated with the local context and global context of the text, we design the architecture of the model to encode both the local and global features. Specifi- cally, dilated convolution blocks stacked to encode local features and transformer block is used to encode global features the coupling layers of normalizing flows. However, using the vanilla transformer to encode the global features in the coupling layer requires too large computational complexity. Therefore, Nyströmformer (Xiong et al.) 2021) which is a Nyström-Bas -estrrd-- algorithm for approximating self-attention is used to encode global context for more efficient memory usage in the coupling layer of normalizing flow. Encoded local features and global features are concate- nated and processed through LayerNorm (Ba al. 2016), Mish activation (Misra 2019) and a convolution module.
Following the duration predictor model in Kim al. (2021). variational dequantization (Ho al, 2019) is used for the duration predictor since the phoneme duration is discrete integer. In addition, variational augmentation (Chen et al. 2020) used to expand channel dimensions for expressive flows in both the duration and pitch predictor. The stochastic and pitch predictor are trained by maximizing their variational lower bounds (ELBO). Details of the duration and pitch predictor are described in Appendix A
B PROOF Proposition
Objective From equation [45] in (Kulesza & Taskar 2012). the marginal kernel of conditional DPPs given the appearance set has a following form:
KA= I [LL 11
(14)
In addition, from equation [34] in Kulesza & Taskar 2012) the expected cardinality of Y given the marginal kernel K is:
Σ
tr(K)
(15)
An +1
From equation 14 15 expected cardinality of conditional DPPs given the appearance of set A is:
Y = trttr(K tr([I [L 1x) 1A)
(16)
Derivative
For the proof, we will start with a following lemma:
Lemma 1, Given a matrix E and non-singular matrix A, following equation holds:
0
tr A A--EE---EE - A-
(17)
aA
Proof. First, consider
0 TT A EE). Since trace and derivative operator are interchangeable,
tr(E E A
E) = tr( DAi
(ETA (TA-EE)
aAg
(18)
0A F))
= -ur(F= -1
AAi
14

---

Under review as a conference paper at ICLR 2022
By setting 0, = E where E denotes the matrix whose (i.j) component is 1 and 0 elsewhere 0.
and C = -EE A' A E,
Ios E)
8 ΣEΣ Σ EETAT )in Ek (A
R1 kg
2(E) ( A 1s(AA EE)s
(19)
E(A F.) iz (F.
)-
- EE" 1ー T
BAg tr = (A EET A 1r--
Now, with respect to set a whose cardinality is p and matrix ((P+s)((+t))
tr(1 11---LL- + 11)
00
0
tr([(L+ Ix)=
(20)
00
tr(EE (L EE))
00
where denotes
xx+ Then by Lemma
Lo]
tr (ET
[L
B) = ((L - 1A) EET (L -1)-
00
(21)
The proof of Proposition
1 is now finished.
15

---

Under review as a conference paper at ICLR 2022
C ALGORITHMS FOR PDM TRAINING AND INFERENCE OF DPP-TTS
C.I PDM TRAINING
Algorithm Training of PDM
Require: TextEncoder f()), PDM parameterized 0, prosody predictor (trained flow) g(-), number of candidates Re, noise scale
while not converged do
2: frear + f(tezt)
3: Split hteet into [Mtrggtt hcontert]
4: Sample latent code Zcontent Rº with noise scale e
5: Get prosodic features of contexts; deontent f- 9 (heontent: Zcontest)
6: Get quality of contexts: gcontest Density estimatioon (ddennttent
7:
Sample latent codes Starget with noise scale
8: Get latent codes after PDM: Starget +- PPDM(Ztarget) R
9:
Get ne prosodic features of targets: (dharget: -argets-..daan
10:
Get quality of targets: Qtarget = +Density est mat on on (dhagget)
Concatenate contexts and targets: [goonteat, Ntarget], [dcontext; dearget)
12:
Build the kernel of conditional DPPs:
f Build kernel( Acontert: Qtarget], [dContert: dearget))
13:
Calculate the loss function: Ldiversity Lr([ [[L 1xx
14:
Update / with the gradient Iddieersity
end while
C.2 INFERENCE OF DPP-TTS Algorithm 2 Inference of DPP-TTS
Require: TextEncoder f(-), Decoder h(-), PDM, a prosody predictor g(-), noise scale
1: Mext + f(text)
2: Split htext into [htarget, hcontext]
3: Sample latent code Zeontent RrmT with noise scale
4: Get prosodic features of contexts: deontent + "(hcontexts Zcontent)
5: Get quality contexts: Gcontezt Density estimati on (cleontett)
6: Sample latent codes Starget with noise scale 6.
7: Get latent codes after PDM: Starget t- PDMM(targget) E R
8: Get The prosodic features of targets: (diorgets
9: Get quality of targets: Itarget = -Densstty est irtimmatio matiio(drargee)) Concatenate contexts and targets: [goontezt; Starget], (dconteat; dearget]
1I: Build the kernel of conditional DPPs:
L +
Build kernel (ermel((conneett Qtarget], [dContext- dearget))
12: Perform the MAP inference d' < arg max, logdet( LnJdeontezs)
Synthesize wavs with prosodic features: y f- licheert d²)
16

---

Under review as a conference paper at ICLR 2022
D SAMPLE PARAGRAPH FOR THE SIDE-BY-SIDE COMPARISON TEST
Known individually and collectively as Shai-Hulud, the sandworms are these supermassive beings that plow through the deserts of Arrakis, consuming everything that dares venture unprepared into their terri- tory. The worms are what make harvesting spice so difficult because they tend to eat whatever tools off-woriders use to mine it. They are also sacred them, and, somehow, they're re linked to the creation of spice. Think of them as big honking metaphors for the sublime powers nature that 1oom beyond human understanding, like desert full of Moby Dicks
the Fremen, who seem to know ways to navigate around
E ADJUSTING THE EXTENT OF VARIATION
(a) Pitch plot with 10
(h) Pitch plot with w=
(c) Log duration plot with w=10
(d) Log duration plot with m==2
Figure 5: Pitch and log duration plots with different values of quality weight. Green plots indicate the prosody prediction before MAP inference of DPP and red plots indicate the prosody prediction after MAP inference of DPP.
17


Reviews:
[{"review_id": "dcAugH9xBFT", "reviewer": "Reviewer_djSa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper targets a very relevant problem in speech synthesis. The proposed method involving the PDM and DPP kernel is a novel application of existing methods to the task of speech synthesis. The selected baselines are strong baselines both of which claim to have state-of-the-art performance in speech naturalness while also being able to have diversity in prosody of the synthesized speech. The audio examples demonstrate that the model is able to synthesize diverse speech even though LJSpeech is notoriously neutral in terms of prosody.\n\nWeaknesses:\n\n- The paper seems to oversell the method claiming that the speech naturalness does not suffer (in the abstract and conclusion). The MOS scores and the speech samples do not back up those claims. \n- Section 4 is a little hard to understand for someone unfamiliar with DPP literature. The figures really come in handy to understand what is going on in the model. There also seem to be some inconsistency in the text and the figure. Specifically, in \"Overview of the PDM process\", it is mentioned that PDM takes context and target hidden sequences, however, figure 2 shows that PDM only takes in the target hidden sequence, which makes sense to me. Similarly it is mentioned that the trained prosody predictor takes in the context hidden sequences, but figure 1b shows that the prosody predictor takes in context and target hidden sequences. \n- The authors need to better motivate the reason for choosing only noun phrases as the target for prosody diversification.\n- Evaluation section has some issues as well:\n  - In the side-by-side evaluation the authors find that listeners rate VITS higher than DPP-TTS2 model in terms of prosody variation. Note that this model should actually be more diverse than DPP-TTS1 since we add an additional degree of freedom: pitch. The authors attribute this poor performance to the drop in naturalness, but this is not a valid explanation, since the listers are instructed to rate the prosody diversity and not naturalness. \n  - The determinant-based metric is not clear to me. Based on my understanding, first, a square cosine similarity matrix is computed between feature sequences from N generated samples for the same utterance. Then the determinant is computed. Please correct me if I'm wrong. Also, if we look at table 2: there is some inconsistency regarding the determinant metrics. For pitch determinant, the larger value is bold, while for duration determinant, the smaller value is bold. One of these is probably incorrect, unless I have completely misunderstood this metric. Based on my understanding, the larger value should indicate better diversity. Another question though is that these determinant values are very tiny. Are the differences significant, and noticeable? The samples should ideally have also contained a set of samples generated for the same speech which better demonstrates the ability of the model to generate diverse prosody.\n\nTypos, etc:\n- Accents in references are not properly rendered. Please check your bib file. E.g. Valles-Peres et al., 2021\n- P5: The stochastic and pitch -> The stochastic duration and pitch\n- P6: importance sampling equation is missing a closing brace.\n", "summary_of_the_paper": "The paper addresses the challenge of synthesizing diverse prosody in text-to-speech systems. Most recent works have now successfully modeled human speech but the delivery usually ends up being monotonous or the average of the training set, and that is the problem that this paper attempts to solve.\n\nThe authors propose a prosody diversifying module (PDM) which is based on conditional determinantal point processes (DPP). This module is utilized in tandem with a Fastspeech 2-based TTS model. They modified the base TTS model to incorporate a stochastic duration and pitch predictor. Once this TTS model is trained, the PDM model is trained separately using the prosody feature predictors and text encoder. The authors focus only on noun phrases for where the PDM operates on to diversify prosody. Additionally, the authors use soft-DTW cost to measure the distance between predicted prosody features for context and focused-on phrases which is vital to the construction of the DPP kernel.\n\nThe authors compare their approach to other methods of TTS which claim to have prosodic diversity, i.e. VITS and Flowtron. They compare the diversity as well as the naturalness of the samples. The DPP model generally outperforms the baselines in terms of diversity but suffers from loss of naturalness.", "summary_of_the_review": "The work is well-motivated and novel. The method also achieves diverse prosody which is clear in the samples provided and the MOS scores. However, some of the claims are a little exaggerated. The naturalness does indeed suffer quite a bit. Additionally, there are some lingering confusions that I raise in the review. Thus I am giving a score of 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rj1y2qmvSgZ", "reviewer": "Reviewer_kefC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an important and practical problem of generating expressive speech. Using DPPs to improve the diversity of predicted prosody is interesting; however, the subjective measures don't demonstrate the effectiveness of the proposed approach. In Table 1, the proposed method comes last in naturalness against the other two baseline systems. Although figure 4 shows that the proposed system can generate more diverse prosody, it might not be naturally diverse. Using both duration and pitch predictors shows inferior results on the right side of figure 4-b. More work is needed to figure out a good recipe for applying DPP to the problem at hand and ensure all components contribute constructively to the overall goal of generating natural and diverse speech.\n\nAnother critical area for improvement is writing. The TTS problem and its components are disconnected in presentation from the technical DPP components. Presenting the baseline TTS system and its prosodic components before DPP allows a more integrated presentation of the DPP section motivated from the problem of interest rather than done abstractly. Figure 3 should be brought earlier to clarify what you mean by context and how you are using the Spacy library for noun phrase segmentation. The authors need to unify the symbols used in the DPP sections and TTS sections to enable the reader to understand the proposed approach. The figure font sizes need to get larger, and the figures themselves need to be redone to be more printing friendly.", "summary_of_the_paper": "This paper addresses the problem of generating expressive speech using Determinantal Point Processes (DPP). Compared to a baseline Fastspeech2 baseline, the authors extend the duration and pitch predictors with a DPP model to favor more diverse candidates. Given that such candidates are variable-length sequences, comparing them is conducted through the differentiable soft-DTW algorithm. The paper shows Mean Opinion Scores (MOS) comparing the proposed approach against the ground-truth audio and two strong models for naturalness. Subjective studies were also conducted to measure the system's ability to generate more speech variability than baselines. \n", "summary_of_the_review": "This paper presents an exciting approach to improving the prosodic diversity of generated speech; however, more work is needed before bringing this research to the publication stage. The paper needs a complete rewrite to integrate the technical approach and the problem of interest and use unified symbols across different sections.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PPyp8hGvaq1", "reviewer": "Reviewer_Dfaw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The idea of formulating prosody generation as a determinantal point process (DPP) is novel to the best of my knowledge. DPP indeed has nice properties that explicitly encourage diversity when generating a subset of samples.\n2. The design of the similarity metric (soft-DTW) is reasonable and addresses the issue of comparing sequences with varying lengths\n\nWeaknesses\n1. Writing requires substantial improvement. Many symbols are undefined when they are first mentioned, notations are messy, and there are a few potential errors in the equations, making it even harder to comprehend. Descriptions of the proposed methods are unclear in several places, and details of how prosody selection fits into the conditional DDP formulation is missing. Please see the detailed comments and questions below for writing issues.\n2. DPP is useful for selecting a high quality and diverse subset. Hence, I was expecting the paper to show how this approach can outperform VAE of flow-based methods that draw i.i.d. samples from the latent space both in quality and in diversity when **generating multiple samples** conditioned on the same text. Unfortunately, this capability was not demonstrated empirically: as Section 4.3 shows, only one prosody feature is selected during inference.\n3. There is a more serious concern about how conditional DDP is deployed for subset selection. Based on the description of Section 4.3, the goal of applying DDP is to select a subset that includes d_L, d_R and one candidate from d_i where i \\in [1, N], such that the subset is high quality and diverse. In other words, given d_L and d_R, the authors want to find a d_i that is of high quality and different from the context, which does not make much sense from the speech synthesis point of view. The context should be coherent with the target, not different. A more reasonable way to apply DDP would be selecting K samples from a set of N candidates which are the prosodic features for the same piece of text. The original design makes s-DTW an unreasonable choice, because DTW aims to align temporal signals corresponding to the same content but may be non-uniformly distorted temporally. \n4. The authors argue that previous work such as GST-Tacotron and VAE-based methods require careful parameter tuning (bottleneck or variance). However, the proposed DDP-TTS also requires tuning the weight of the quality term $w$, which does not seem to solve the issues mentioned in the previous work.\n5. The authors state “The results demonstrate that our model generates speech with richer prosody than baselines while maintaining speech naturalness...” in the introduction. This is clearly an overstatement in terms of speech naturalness - the proposed model is 0.84 behind VITS on MOS score, which apparently are not of the same level of naturalness.\n6. Ablation studies are missing. The authors should have compared with a baseline of their system that does not use the DDP module. More specifically, the authors could sample from the latent distribution (normal) or take the mode of the prior to generate prosodic features. \n\nQuestions/Comments\n1. The concept of “ground set” is never explicitly defined. The author could have mentioned that as a set of candidate prosodies.\n2. For every target sequence, does the prosody predictor encode the left and right context separately?\n3. Why conditional DPP is required is not clear when reading Section 2.2. The authors should draw more connections between (conditional) prosody selection to (conditional) DDP when writing Section 2, instead of just copying most of the text from Kulesza & Taskar (2010).\n4. Which part of the “DPP kernel built” is learned? The similarity matrix is defined by soft-DTW between prosody feature sequences, and the quality measure is defined by a density estimator (which I assume is the flow module but again was never stated formally).\n5. In Sec 4.3, the authors put $logP_L(Y) = det(L) / det(L+I)$, which is inconsistent with Eq.2: it is probability instead of log probability, and the numerator should be $L_Y$ instead of $L$\n\n\n** Post Rebuttal **\nI thank the authors for providing detailed responses to my concerns. After reading the rebuttal and the updated manuscripts carefully, the concerns about clarity and lack of ablation studies have been addressed. However, I still do not agree with the authors' view that neighboring words prosody should be diversified and S-DTW is a suitable similarity metric for the purpose. For utterances with more lively speaking styles, there is a bigger variance of speaking rate and pitch within an utterance, but such variation is typically smooth temporally, instead of bouncing up and down every word, which would resulted in unnatural prosody, which might have been hinted by the worse MOS score in Table 5 when including PDM. In addition, using DTW to align two different spans within an utterance also makes little sense - it should be used to align sequences with the same content but having non-uniform rate distortion. Given these considerations, I would increase the rating from 3 to 5, but I am still in favor of rejection.\n", "summary_of_the_paper": "The goal of this paper is to build a text-to-speech model that can generate diverse prosodic features (in particular pitch and duration patterns) without sacrificing the audio quality. The authors propose to cast the prosody selection problem as a determinantal point process, which explicitly takes quality and diversity between elements into account when sampling a subset. To formulate the problem as such, the authors present a prosody diversifying module to generate candidate prosodies for selection, and use soft-DTW as the similarity metric.", "summary_of_the_review": "The current writeup lacks clarity, ablation studies, and proper evaluations. The way of how conditional DDP is applied also seems problematic.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes the use of the determinantal point process to introduce the diversity in the prosodic features, including intonation, stress, and rhythm, in text to speech synthesis.  The proposed approach is certainly new, but the experimental support is of critical importance for this work.  One of the major points of discussion was the reliability of the experimental results.  In the original submission, the mean opinion score (MOS) of the proposed approach was inferior to the baseline.  The authors updated the experiments, which significantly (more than the confidence interval) lowers the MOS of a baseline.  This however makes the experimental results questionable.
Decision: Reject