Adaptive Speech Duration Modification using a Deep-Generative Framework

1 INTRODUCTION . Human speech is a rich and varied mode of communication that encompasses both language/semantic information and the mood/intent of the speaker . The latter is primarily conveyed by prosodic features , such as pitch , energy , speaking rate , and voice quality . There are many applications where understanding and manipulating these prosodic features is required . Consider voice conversion systems as an example . Pitch and energy modifications are used to passively inject emotional cues into the neutral speech or to change the overall speaking style ( A. Russell et al. , 2003 ; Schacter et al. , 2011 ; Shankar et al. , 2019a ; b ; Valle et al. , 2019 ) . Prosodic features are also used to evaluate the quality/engagement in human machine dialog systems ( Swerts & Krahmer , 2000 ) , and they play a significant role in speaker identification and recognition systems ( Park et al. , 2016 ) . While there are many approaches for automated pitch and energy modification ( Toda et al. , 2007 ; Aihara et al. , 2012 ; Kaneko & Kameoka , 2017 ; Shankar et al. , 2020b ; a ) , comparatively little progress has been made in changing the duration/speaking rate of an utterance . In fact , the speaking rate plays a crucial role in conveying emotions ( Schmidt et al. , 2016 ) and in diagnosing human speech pathologies ( Bayerl et al. , 2020 ) . The speaking rate is difficult to manipulate because , unlike pitch or energy , there is no explicit coding for either the signal duration or speaking rhythm . Rather , these features are implicitly defined by the cardinality of the set of frames over a particular interval of interest . This cardinality is a global parameter that masks subtle variations in the speaker rate over an utterance . As a result , duration modification algorithms are not adaptive . Instead , they either require considerable user supervision or they are geared towards aligning to known speech signals . Perhaps the earliest duration modification method is the time-domain pitch synchronous overlap and add ( TD-PSOLA ) algorithm ( Charpentier & Stella , 1986 ) . TD-PSOLA modifies the pitch and duration of a speech signal by replicating and interpolating between individual frames centered at the peaks of auto-correlation signal . However , the user must manually specify both the portion of speech to modify and the exact manner in which it should be altered . Hence , the method is neither automated nor adaptive . An alternative approach is dynamic time warping ( DTW ) , which finds the optimal time alignment between two parallel speech utterances ( dtw , 2008 ) . DTW constructs a pairwise similarity matrix between all frames of the two utterances and estimates a warping path between the starting ( 0 , 0 ) and ending ( Ts , Tt ) points of the utterances based on a Viterbi-like de- coding of the similarity matrix . While simple , DTW requires both the source and target utterances to be known a priori . Hence , it can not be used for on-the-fly modification of new signals . Finally , recent advancements in deep learning have led to a new generation of neural vocoders that disentangle the semantic content from the speaking style ( Oord et al. , 2016 ; Shen et al. , 2017 ; Wang et al. , 2017 ) . These vocoders can alter the speaking rate via the learned style embeddings . While these models represent seminal contributions to speech synthesis , the latent representations are learned in an unsupervised manner , which makes it difficult to control the output speaking style in predictable manner . Another drawback of these methods is the large amount of data and computational resources required to train the models and generate new speech ( Yasuda et al. , 2020 ) . In this paper , we introduce the first fully-automated adaptive speech duration modification scheme . Our approach combines the representation capabilities of deep neural networks with the structured simplicity of dynamic decoding . Namely , we model the alignment between a source and target utterance via a latent attention map ; these maps are used as replacement of similarity matrix for backtracking . We train a masked convolutional encoder-decoder network to estimate these attention maps using a stochastic mean absolute error ( MAE ) formulation . Unlike the conventional DTW ( dtw , 2008 ) algorithm , once trained our framework operates entirely on the source utterance without needing to reference the target . We demonstrate our framework on a voice conversion task using the CMU-Arctic dataset ( Kominek & W Black , 2004 ) and on three multi-speaker emotion conversion tasks using the VESUS dataset ( Sager et al. , 2019 ) . Our experiments confirm that the proposed model can perform open-loop duration modification and produces high-quality speech . 2 METHOD . Fig . 1 illustrates our underlying generative process . Given an utterance X , we first estimate the length T of the ( unknown ) target utterance Y and subsequently use it to estimate a mask M for the attention map . The mask restricts the domain of the attention vectors At at each frame t during the inference stage to mitigate distortion of the output speech . We use paired data ( Xtr , Ytr ) to train a convolutional encoder-decoder network to generate the attention vectors . During testing , we first generate the attention map from the input X and use it to produce the target speech Y . 2.1 LOSS FUNCTION . Formally , let X ∈ RD×Ts denote the input speech . In this work , X corresponds to the Mel filterbank energies extracted from short-time moving window analysis , where D is the number of filterbanks , and Ts is the number of temporal frames in the utterance . Similarly , we denote the target speech as Y ∈ RD×T . Notice that the target utterance length T is usually different from Ts . Our generative process for a single frame of the target speech is represented as follows : T ∼ Laplace ( T 0 , bT ) and Yt ∼ Laplace ( Y 0t , by ) , ( 1 ) where T is the estimated length of the target utterance , and Yt is the target Mel filter-bank energy features at time t. The parameters { T 0 , bT , Y 0t , by } of the distributions are unknown and implicitly estimated via a deep neural network . The neural network is parameterized by γ and θ ( Fig . 1 ) . By treating the unknown parameters as functions of the input X , we obtain the following estimating equations for the target sequence length and frame-wise Mel filter-bank energies : T̂ = fγ ( X ) and Ŷt = X ·At + fθ ( X , Ŷ0 : t−1 ) . ( 2 ) The functions fγ ( · ) and fθ ( · , · ) correspond to the length prediction and energy estimation component of the same deep neural network . The variableAt ∈ RTs is an attention vector that combines framewise features of the source utterance X to generate the target frame Ŷt . Our model differs from standard sequence-to-sequence model by treating neural net predictions as residuals added to input sequence itself . Notice that the residuals depend on input and the history of predictions Ŷ0 : t−1 at previous time steps . This autoregressive property allows the neural network to learn segmental and supra-segmental variations that can potentially distinguish between the speakers or emotions . During training , we use paired data ( X , Y ) and maximize the likelihood of the target speech signal with respect to the neural network weights { θ , γ } . This likelihood can be written as : P ( Ŷ , T̂ |X ) = P ( T̂ |X ) T̂∏ t=1 P ( Ŷt|X , T̂ , Ŷ0 : t−1 ) , ( 3 ) where , the second term in Eq . ( 3 ) can be obtained via marginalization over At as follows : P ( Ŷt|X , T̂ , Ŷ0 : t−1 ) = ∑ At P ( Ŷt , At|X , T̂ , Ŷ0 : t−1 , M ) = ∑ At P ( Ŷt|X , T̂ , At , Ŷ0 : t−1 ) P ( At|X , Ŷ0 : t−1 , M ) ( 4 ) The variable M here denotes the attention mask . We introduce M for convenience , as it is a deterministic function of the source length Ts and the estimated target length T̂ . We encode the attention At as a one-hot vector across the Ts frames of the source speech . Thus , it follows a multinomial distribution . For simplicity , we model At as conditionally independent of the target length T̂ given the maskM and the inputX ( see Fig . 1 ) . Taking log ( · ) of Eq . ( 3 ) and combining with Eq . ( 4 ) yields : L ( θ , γ ) = − log ( ∑ At P ( Ŷt , At|X , T̂ , Ŷ0 : t−1 , M ) ) − log ( P ( T̂ |X ) ) = − log ( ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) qθ ( At|X , Ŷ0 : t−1 , M ) P ( Ŷt , At|X , T̂ , Ŷ0 : t−1 , M ) ) − log ( P ( T̂ |X ) ) ≤ − ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) − log ( P ( T̂ |X ) ) +KL ( qθ ( At ) ||P ( At ) ) = − ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) − log ( P ( T̂ |X ) −H ( qθ ) + const . ≤ − ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) − log ( P ( T̂ |X ) + const . ( 5 ) The distribution qθ ( · ) above is an approximating distribution for the attention vectors implemented by a convolutional network . The first inequality uses the convexity of the − log function , and the second inequality comes from the fact that entropy H ( qθ ) ≥ 0 . Notice that we have implicitly assumed P ( At|X , Ŷ0 : t−1 , M ) has a uniform distribution over the masked region . This is a reasonable assumption given that the masking process reduces the attention domain to a small region ( see Section 2.3 ) . However , qθ is not penalized for deviating from this uniform distribution during training . This flexibility allows the network to learn realistic attention vectors during autoregressive decoding . Eq . ( 5 ) can be easily translated into a neural network loss function which we minimize for { θ , γ } : L ( θ , γ ) = λ1 × EAt∼qθ [ log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) ] + λ2 × log ( P ( T̂ |X ) ) = λ1 × EAt [ ‖Ŷt − Y 0t ‖1 ] + λ2 × ‖T̂ − T 0‖1 , ( 6 ) where λ1 and λ2 are the model hyperparameters that adjusts the trade-off between the two objectives and implicitly contain the variances of the Laplace distributions in Eq . ( 1 ) . Notice that the loss in Eq . ( 6 ) computes an expectation over the attention maps . We use the Monte-Carlo estimate by sampling from the attention map at each time-step . The training procedure is therefore stochastic in nature due to this random sampling . We mix this stochastic version with the maximum aposteriori estimate ( MAP ) of the attention vector with a probability of 0.2 during the start of training procedure . Algorithm 1 : Strategy for model training 1 function trainModelParameters ( X , Y ) ; Input : filterbank energies ( X ∈ RD×Ts , Y ∈ RD×Tt ) Output : model parameters ( θ , γ ) 2 if epoch < MaxEpochs then 3 Set t = 0 , predict target length T̂ = fγ ( X ) and create the mask M ∈ RTs×Tt ; 4 Estimate A ∈ RTs×Tt using masked convolution and sample u ∼ U ( 0 , 1 ) ; 5 if u < 0.2 then 6 Sample a ∈ RTs×Tt as 1-hot vectors from A ; 7 Reconstruct using Ŷt = X · a+ fθ ( X , Y0 : t−1 ) ; 8 else 9 Reconstruct using Ŷt = X ·A+ fθ ( X , Y0 : t−1 ) ; 10 end 11 Compute prediction errors and update parameters θ , γ ; 12 epoch← epoch + 1 ; 13 end 14 return θ and γ ;

Summary: This paper proposes a model for adaptive duration modification of an input signal. The model is a graphical model with neural components. By making some assumptions, authors derived an upper bound for the likelihood (conditional probability of output sequence and estimated length given input). All the model parameters are updated, by minimizing this upper bound.