DPP-TTS: Diversifying prosodic features of speech via determinantal point processes

1 INTRODUCTION . In the past few years , text-to-speech models have made a lot of progress in synthesizing human-like speech ( Shen et al. , 2018 ; Ping et al. , 2018 ; Li et al. , 2019 ; Ren et al. , 2019 ) . Furthermore , in the latest studies , several text-to-speech models made high-quality speech samples even in the end-toend setting without a two-stage synthesis process ( Donahue et al. , 2021 ; Kim et al. , 2021 ) . Based on these technical developments , text-to-speech models are now able to generate high-fidelity speech . However , human speech contains many prosodic features like intonation , stress , and rhythm beyond textual features . Therefore , it is crucial to generate speech samples with rich prosody . There have been many attempts to synthesize speech with rich prosodic features . To incorporate rich prosodic features into speech , reference acoustic samples like mel-spectrograms are processed through reference encoder ( Skerry-Ryan et al. , 2018 ; Wang et al. , 2018 ) or text-to-speech models are conditioned on prosodic features like duration , pitch and energy then these features are predicted or manually controlled at inference ( Ren et al. , 2021 ; Lańcucki , 2021 ) . However , these methods have drawbacks that the bottleneck dimension should be carefully tuned for desirable results , or the prosody predictor just learns averaged prosodic features of training sets . Meanwhile , generative models like VAEs and flow models ( Hsu et al. , 2019 ; Sun et al. , 2020 ; VallésPérez et al. , 2021 ; Lee et al. , 2021 ; Valle et al. , 2021 ) have been recently used for speech synthesis and their latent spaces are manipulated for generating more expressive speech . Controlling the amount of variation in the speech is achieved by generating samples in Gaussian prior with an adequate temperature , however , it has two major drawbacks . First , generating latent samples with high variance often make generated samples to be unstable in terms of the naturalness and intelligibility of speech . Second , sampling in the latent space might only cover major modes learned during the training stage and only monotonous prosodic patterns can occur in some segments of speech . Determinantal point processes ( DPPs ) have shown great results for modeling diversity in various machine learning tasks and their applications . Its uses include text summarization ( Cho et al. , 2019 ) , recommendation systems ( Gartrell et al. , 2021 ) , multi-modal output generation ( Elfeki et al. , 2019 ) , diverse trajectory forecasting ( Yuan & Kitani , 2020 ) , and machine translation ( Meister et al. , 2021 ) . Determinantal point process ( DPPs ) offers us an efficient subset selection method by considering both the quality and diversity of items within a set . Specifically , items within a set are sampled according to DPPs kernel which reflects the quality and diversity of items in the ground set . In some cases , we may need to incorporate the conditional information into the sampling process . To this end , conditional DPP ( Kulesza & Taskar , 2012 ) has been widely used for sampling subset by setting the kernel conditioned on the specific information . To use conditional DPP for sampling prosodic features of speech , two main issues should be considered . First , prosodic features are usually varying in length . Second , it is ambiguous what the ground set is in this problem . In this work , we resolve these issues by using a new similarity metric between prosodic features and adding a prosodic diversifying module ( PDM ) into the framework . Specifically , PDM generates candidates of prosodic features of speech by mapping latent codes from normal distribution to a new latent space and these candidates are used as the ground set in the sampling process . In addition , the similarity between two prosodic features is evaluated using soft dynamic time warping discrepancy ( Soft-DTW ) ( Cuturi & Blondel , 2017 ) which enables to evaluate the similarity between features in different lengths . The kernel matrix of conditional DPP is learned during the training by parameters of PDM getting updated . Specifically , parameters of the prosody diversifying module ( PDM ) are updated in the training stage with conditional maximum induced cardinality ( MIC ) objective which is adapted from MIC objective introduced in Gillenwater et al . ( 2018 ) . We formally introduce the conditional MIC objective and its derivative for clarity in Section 4 . To implement the aforementioned DPP , a stochastic duration predictor and pitch predictor are introduced in this work . Both the duration and pitch predictor are built upon normalizing flows trained with the maximum likelihood ( MLE ) objective . At inference , the prosody predictor maps latent codes from a latent space to feature spaces with the inverse flow . In experiments , we compare DPP-TTS with the state-of-the-art models including VITS ( Kim et al. , 2021 ) and Flowtron ( Valle et al. , 2021 ) in terms of prosody diversity and speech quality . The results demonstrate that our model generates speech with richer prosody than baselines while maintaining speech naturalness . We also demonstrate that our DPP-TTS can be used in real-time applications by evaluating the inference speed of our model . In summary , our contributions follow : • We propose a novel method for diversifying prosodic features of speech based on conditional DPPs by considering prosodic features of context words as conditions . • To learn the kernel matrix of conditional DPPs , we propose to train prosody diversifying module ( PDM ) with the conditional maximum induced cardinality ( MIC ) objective . • Experiments on the side-by-side comparison and the MOS test verify that our model outperforms the two baseline models in terms of prosodic diversity while maintaining the naturalness of speech . 2 BACKGROUND . 2.1 DETERMINANTAL POINT PROCESSES . DPPs encourage diversity within a set by discouraging sampling similar items within the ground set . Formally , point process P is called a determinantal point process when Y is a random subset drawn according to P , we have , for every A ⊆ Y , P ( A ⊆ Y ) ∝ det ( KA ) , ( 1 ) whereK is a positive definite matrix whose eigenvalues are all between 0 and 1 andKA is a positive a definite matrix indexed by elements in A . The marginal probability of including two elements ei and ej is KiiKjj − K2ij = p ( ei ∈ Y ) p ( ej ∈ Y ) − K2ij . Therefore , the value of K2ij models the extent of negative correlation between item i and j . More frequently , DPPs are defined by L-ensemble through real and symmetric matrix L instead of the marginal kernelK : PL ( Y = Y ) = det ( LY ) det ( L+ I ) , ( 2 ) where det ( L+ I ) in the denominator acts as a normalization constant . The marginal kernel K and kernel L has the following relation : K = L ( L+ I ) −1 = I − ( L+ I ) −1 ( 3 ) To model diversity between items , the DPP kernelL is usually constructed as a symmetric similarity matrix , where Sij represents the similarity between two items xi and xj . In Kulesza & Taskar ( 2010 ) , authors proposed decomposing the kernel L as a a Gram matrix incorporating a quality vector to weigh each item according to its quality : P ( J ⊆ Y ) ∝ det ( φ ( J ) Tφ ( J ) ) ∏ ei∈J q2 ( ei ) = Diag ( q ) · S · Diag ( q ) , ( 4 ) where φi ∈ RD ; D ≤ N and ||φi||2 = 1 . In this manner , the similarity matrix S is guaranteed to be real positive semidefinite matrix . 2.2 CONDITIONAL DETERMINANTAL POINT PROCESSES . If DPPs are used for diversifying prosodic features by setting targets of DPPs as prosodic features of sentences , it would result in diversity among generated samples . However , there still can be monotonous patterns in each generated speech . To resolve this issue , it is required to diversify prosodic features accounting into their neighbor prosodic features . Therefore , we need conditional DPPs to take into account neighboring prosodic features ( contexts ) . By setting conditions of point process P , DPPs can be extended to conditional DPPs , For a subset B ⊆ Y not intersecting with A we have P ( Y = A ∪B|A ⊆ Y ) = P ( Y = A ∪B ) P ( A ⊆ Y ) = det ( LA∪B ) det ( L+ IĀ ) , ( 5 ) where IĀ is the matrix with ones in the diagonal entries indexed by elements of Y − A and zeros elsewhere . In Borodin & Rains ( 2004 ) , authors showed that this conditional distribution is again a DPP , with a kernel given by LA = ( [ ( L+ IĀ ) −1 ] Ā ) −1 − I . ( 6 ) In conditional DPPs , items in the ground set Y are sampled according to kernel given contexts . In this work , contexts of target words are used as conditions for conditional DPPs . 2.3 SOFT DYNAMIC TIME WARPING . To build the kernel of conditional DPPs , we need a measure of similarity between two temporal signals . Simple Euclidean distance is not applicable because two time signals often vary in their lengths . In this work , soft dynamic time warping ( Soft DTW ) discrepancy which allows measuring the similarity of shape between time series of different lengths is adapted to calculate the similarity between prosodic sequences . DTW computes the best possible alignment between two temporal signals . First , for the length of n and m signals , DTW computes n by m pairwise distance matrix between these points with a specific metric ( e.g. , Euclidean distance , L1 distance ) . After that , this matrix is used to solve the dynamic program using Bellman ’ s recursion with a quadratic O ( mn ) cost . Unfortunately , vanilla DTW cost is not easy to optimize because it only considers a single alignment between two temporal signals . Cuturi & Blondel ( 2017 ) proposed differentiable Soft-DTW which considers all possible alignments between two temporal series . Soft-DTW is differentiable in all of its arguments with quadratic cost . Formally , Soft-DTW is defined as follows : dtwγ ( x , y ) : = minγ { 〈A , δ ( x , y ) 〉 , A ∈ An , m } , ( 7 ) where An , m denotes all possible alignments between x and y and minγ is defined as minγ { a1 , ... , an } : = { mini≤nai , if γ = 0 −γ log ∑n i=1 e −ai γ , otherwise ( 8 ) A small magnitude of γ reflects the true discrepancy of two temporal signals , however , optimization becomes unstable . 3 DPP-TTS . Our model DPP-TTS is composed of a Seq2Seq module for generating the mel-spectrogram , a prosody predictor for predicting duration and pitch sequences , and a prosody diversifying module ( PDM ) . At the first stage , the base TTS model which consists of the Seq2Seq module and the prosody predictor is trained as shown in Figure 1a . Once the base TTS is trained , PDM is inserted in front of the prosody predictor and trained with the method which will be described in detail in Section 4 . We describe the main modules of DPP-TTS and their roles in the following subsection .

Summary: The paper addresses the challenge of synthesizing diverse prosody in text-to-speech systems. Most recent works have now successfully modeled human speech but the delivery usually ends up being monotonous or the average of the training set, and that is the problem that this paper attempts to solve.  The authors propose a prosody diversifying module (PDM) which is based on conditional determinantal point processes (DPP). This module is utilized in tandem with a Fastspeech 2-based TTS model. They modified the base TTS model to incorporate a stochastic duration and pitch predictor. Once this TTS model is trained, the PDM model is trained separately using the prosody feature predictors and text encoder. The authors focus only on noun phrases for where the PDM operates on to diversify prosody. Additionally, the authors use soft-DTW cost to measure the distance between predicted prosody features for context and focused-on phrases which is vital to the construction of the DPP kernel.  The authors compare their approach to other methods of TTS which claim to have prosodic diversity, i.e. VITS and Flowtron. They compare the diversity as well as the naturalness of the samples. The DPP model generally outperforms the baselines in terms of diversity but suffers from loss of naturalness.