DPP-TTS: Diversifying prosodic features of speech via determinantal point processes

1 INTRODUCTION . In the past few years , text-to-speech models have made a lot of progress in synthesizing human-like speech ( Shen et al. , 2018 ; Ping et al. , 2018 ; Li et al. , 2019 ; Ren et al. , 2019 ) . Furthermore , in the latest studies , several text-to-speech models made high-quality speech samples even in the end-toend setting without a two-stage synthesis process ( Donahue et al. , 2021 ; Kim et al. , 2021 ) . Based on these technical developments , text-to-speech models are now able to generate high-fidelity speech . However , human speech contains many prosodic features like intonation , stress , and rhythm beyond textual features . Therefore , it is crucial to generate speech samples with rich prosody . There have been many attempts to synthesize speech with rich prosodic features . To incorporate rich prosodic features into speech , reference acoustic samples like mel-spectrograms are processed through reference encoder ( Skerry-Ryan et al. , 2018 ; Wang et al. , 2018 ) or text-to-speech models are conditioned on prosodic features like duration , pitch and energy then these features are predicted or manually controlled at inference ( Ren et al. , 2021 ; Lańcucki , 2021 ) . However , these methods have drawbacks that the bottleneck dimension should be carefully tuned for desirable results , or the prosody predictor just learns averaged prosodic features of training sets . Meanwhile , generative models like VAEs and flow models ( Hsu et al. , 2019 ; Sun et al. , 2020 ; VallésPérez et al. , 2021 ; Lee et al. , 2021 ; Valle et al. , 2021 ) have been recently used for speech synthesis and their latent spaces are manipulated for generating more expressive speech . Controlling the amount of variation in the speech is achieved by generating samples in Gaussian prior with an adequate temperature , however , it has two major drawbacks . First , generating latent samples with high variance often make generated samples to be unstable in terms of the naturalness and intelligibility of speech . Second , sampling in the latent space might only cover major modes learned during the training stage and only monotonous prosodic patterns can occur in some segments of speech . Determinantal point processes ( DPPs ) have shown great results for modeling diversity in various machine learning tasks and their applications . Its uses include text summarization ( Cho et al. , 2019 ) , recommendation systems ( Gartrell et al. , 2021 ) , multi-modal output generation ( Elfeki et al. , 2019 ) , diverse trajectory forecasting ( Yuan & Kitani , 2020 ) , and machine translation ( Meister et al. , 2021 ) . Determinantal point process ( DPPs ) offers us an efficient subset selection method by considering both the quality and diversity of items within a set . Specifically , items within a set are sampled according to DPPs kernel which reflects the quality and diversity of items in the ground set . In some cases , we may need to incorporate the conditional information into the sampling process . To this end , conditional DPP ( Kulesza & Taskar , 2012 ) has been widely used for sampling subset by setting the kernel conditioned on the specific information . To use conditional DPP for sampling prosodic features of speech , two main issues should be considered . First , prosodic features are usually varying in length . Second , it is ambiguous what the ground set is in this problem . In this work , we resolve these issues by using a new similarity metric between prosodic features and adding a prosodic diversifying module ( PDM ) into the framework . Specifically , PDM generates candidates of prosodic features of speech by mapping latent codes from normal distribution to a new latent space and these candidates are used as the ground set in the sampling process . In addition , the similarity between two prosodic features is evaluated using soft dynamic time warping discrepancy ( Soft-DTW ) ( Cuturi & Blondel , 2017 ) which enables to evaluate the similarity between features in different lengths . The kernel matrix of conditional DPP is learned during the training by parameters of PDM getting updated . Specifically , parameters of the prosody diversifying module ( PDM ) are updated in the training stage with conditional maximum induced cardinality ( MIC ) objective which is adapted from MIC objective introduced in Gillenwater et al . ( 2018 ) . We formally introduce the conditional MIC objective and its derivative for clarity in Section 4 . To implement the aforementioned DPP , a stochastic duration predictor and pitch predictor are introduced in this work . Both the duration and pitch predictor are built upon normalizing flows trained with the maximum likelihood ( MLE ) objective . At inference , the prosody predictor maps latent codes from a latent space to feature spaces with the inverse flow . In experiments , we compare DPP-TTS with the state-of-the-art models including VITS ( Kim et al. , 2021 ) and Flowtron ( Valle et al. , 2021 ) in terms of prosody diversity and speech quality . The results demonstrate that our model generates speech with richer prosody than baselines while maintaining speech naturalness . We also demonstrate that our DPP-TTS can be used in real-time applications by evaluating the inference speed of our model . In summary , our contributions follow : • We propose a novel method for diversifying prosodic features of speech based on conditional DPPs by considering prosodic features of context words as conditions . • To learn the kernel matrix of conditional DPPs , we propose to train prosody diversifying module ( PDM ) with the conditional maximum induced cardinality ( MIC ) objective . • Experiments on the side-by-side comparison and the MOS test verify that our model outperforms the two baseline models in terms of prosodic diversity while maintaining the naturalness of speech . 2 BACKGROUND . 2.1 DETERMINANTAL POINT PROCESSES . DPPs encourage diversity within a set by discouraging sampling similar items within the ground set . Formally , point process P is called a determinantal point process when Y is a random subset drawn according to P , we have , for every A ⊆ Y , P ( A ⊆ Y ) ∝ det ( KA ) , ( 1 ) whereK is a positive definite matrix whose eigenvalues are all between 0 and 1 andKA is a positive a definite matrix indexed by elements in A . The marginal probability of including two elements ei and ej is KiiKjj − K2ij = p ( ei ∈ Y ) p ( ej ∈ Y ) − K2ij . Therefore , the value of K2ij models the extent of negative correlation between item i and j . More frequently , DPPs are defined by L-ensemble through real and symmetric matrix L instead of the marginal kernelK : PL ( Y = Y ) = det ( LY ) det ( L+ I ) , ( 2 ) where det ( L+ I ) in the denominator acts as a normalization constant . The marginal kernel K and kernel L has the following relation : K = L ( L+ I ) −1 = I − ( L+ I ) −1 ( 3 ) To model diversity between items , the DPP kernelL is usually constructed as a symmetric similarity matrix , where Sij represents the similarity between two items xi and xj . In Kulesza & Taskar ( 2010 ) , authors proposed decomposing the kernel L as a a Gram matrix incorporating a quality vector to weigh each item according to its quality : P ( J ⊆ Y ) ∝ det ( φ ( J ) Tφ ( J ) ) ∏ ei∈J q2 ( ei ) = Diag ( q ) · S · Diag ( q ) , ( 4 ) where φi ∈ RD ; D ≤ N and ||φi||2 = 1 . In this manner , the similarity matrix S is guaranteed to be real positive semidefinite matrix . 2.2 CONDITIONAL DETERMINANTAL POINT PROCESSES . If DPPs are used for diversifying prosodic features by setting targets of DPPs as prosodic features of sentences , it would result in diversity among generated samples . However , there still can be monotonous patterns in each generated speech . To resolve this issue , it is required to diversify prosodic features accounting into their neighbor prosodic features . Therefore , we need conditional DPPs to take into account neighboring prosodic features ( contexts ) . By setting conditions of point process P , DPPs can be extended to conditional DPPs , For a subset B ⊆ Y not intersecting with A we have P ( Y = A ∪B|A ⊆ Y ) = P ( Y = A ∪B ) P ( A ⊆ Y ) = det ( LA∪B ) det ( L+ IĀ ) , ( 5 ) where IĀ is the matrix with ones in the diagonal entries indexed by elements of Y − A and zeros elsewhere . In Borodin & Rains ( 2004 ) , authors showed that this conditional distribution is again a DPP , with a kernel given by LA = ( [ ( L+ IĀ ) −1 ] Ā ) −1 − I . ( 6 ) In conditional DPPs , items in the ground set Y are sampled according to kernel given contexts . In this work , contexts of target words are used as conditions for conditional DPPs . 2.3 SOFT DYNAMIC TIME WARPING . To build the kernel of conditional DPPs , we need a measure of similarity between two temporal signals . Simple Euclidean distance is not applicable because two time signals often vary in their lengths . In this work , soft dynamic time warping ( Soft DTW ) discrepancy which allows measuring the similarity of shape between time series of different lengths is adapted to calculate the similarity between prosodic sequences . DTW computes the best possible alignment between two temporal signals . First , for the length of n and m signals , DTW computes n by m pairwise distance matrix between these points with a specific metric ( e.g. , Euclidean distance , L1 distance ) . After that , this matrix is used to solve the dynamic program using Bellman ’ s recursion with a quadratic O ( mn ) cost . Unfortunately , vanilla DTW cost is not easy to optimize because it only considers a single alignment between two temporal signals . Cuturi & Blondel ( 2017 ) proposed differentiable Soft-DTW which considers all possible alignments between two temporal series . Soft-DTW is differentiable in all of its arguments with quadratic cost . Formally , Soft-DTW is defined as follows : dtwγ ( x , y ) : = minγ { 〈A , δ ( x , y ) 〉 , A ∈ An , m } , ( 7 ) where An , m denotes all possible alignments between x and y and minγ is defined as minγ { a1 , ... , an } : = { mini≤nai , if γ = 0 −γ log ∑n i=1 e −ai γ , otherwise ( 8 ) A small magnitude of γ reflects the true discrepancy of two temporal signals , however , optimization becomes unstable . 3 DPP-TTS . Our model DPP-TTS is composed of a Seq2Seq module for generating the mel-spectrogram , a prosody predictor for predicting duration and pitch sequences , and a prosody diversifying module ( PDM ) . At the first stage , the base TTS model which consists of the Seq2Seq module and the prosody predictor is trained as shown in Figure 1a . Once the base TTS is trained , PDM is inserted in front of the prosody predictor and trained with the method which will be described in detail in Section 4 . We describe the main modules of DPP-TTS and their roles in the following subsection .

Summary: This paper addresses the problem of generating expressive speech using Determinantal Point Processes (DPP). Compared to a baseline Fastspeech2 baseline, the authors extend the duration and pitch predictors with a DPP model to favor more diverse candidates. Given that such candidates are variable-length sequences, comparing them is conducted through the differentiable soft-DTW algorithm. The paper shows Mean Opinion Scores (MOS) comparing the proposed approach against the ground-truth audio and two strong models for naturalness. Subjective studies were also conducted to measure the system's ability to generate more speech variability than baselines.