Adaptive Speech Duration Modification using a Deep-Generative Framework

1 INTRODUCTION . Human speech is a rich and varied mode of communication that encompasses both language/semantic information and the mood/intent of the speaker . The latter is primarily conveyed by prosodic features , such as pitch , energy , speaking rate , and voice quality . There are many applications where understanding and manipulating these prosodic features is required . Consider voice conversion systems as an example . Pitch and energy modifications are used to passively inject emotional cues into the neutral speech or to change the overall speaking style ( A. Russell et al. , 2003 ; Schacter et al. , 2011 ; Shankar et al. , 2019a ; b ; Valle et al. , 2019 ) . Prosodic features are also used to evaluate the quality/engagement in human machine dialog systems ( Swerts & Krahmer , 2000 ) , and they play a significant role in speaker identification and recognition systems ( Park et al. , 2016 ) . While there are many approaches for automated pitch and energy modification ( Toda et al. , 2007 ; Aihara et al. , 2012 ; Kaneko & Kameoka , 2017 ; Shankar et al. , 2020b ; a ) , comparatively little progress has been made in changing the duration/speaking rate of an utterance . In fact , the speaking rate plays a crucial role in conveying emotions ( Schmidt et al. , 2016 ) and in diagnosing human speech pathologies ( Bayerl et al. , 2020 ) . The speaking rate is difficult to manipulate because , unlike pitch or energy , there is no explicit coding for either the signal duration or speaking rhythm . Rather , these features are implicitly defined by the cardinality of the set of frames over a particular interval of interest . This cardinality is a global parameter that masks subtle variations in the speaker rate over an utterance . As a result , duration modification algorithms are not adaptive . Instead , they either require considerable user supervision or they are geared towards aligning to known speech signals . Perhaps the earliest duration modification method is the time-domain pitch synchronous overlap and add ( TD-PSOLA ) algorithm ( Charpentier & Stella , 1986 ) . TD-PSOLA modifies the pitch and duration of a speech signal by replicating and interpolating between individual frames centered at the peaks of auto-correlation signal . However , the user must manually specify both the portion of speech to modify and the exact manner in which it should be altered . Hence , the method is neither automated nor adaptive . An alternative approach is dynamic time warping ( DTW ) , which finds the optimal time alignment between two parallel speech utterances ( dtw , 2008 ) . DTW constructs a pairwise similarity matrix between all frames of the two utterances and estimates a warping path between the starting ( 0 , 0 ) and ending ( Ts , Tt ) points of the utterances based on a Viterbi-like de- coding of the similarity matrix . While simple , DTW requires both the source and target utterances to be known a priori . Hence , it can not be used for on-the-fly modification of new signals . Finally , recent advancements in deep learning have led to a new generation of neural vocoders that disentangle the semantic content from the speaking style ( Oord et al. , 2016 ; Shen et al. , 2017 ; Wang et al. , 2017 ) . These vocoders can alter the speaking rate via the learned style embeddings . While these models represent seminal contributions to speech synthesis , the latent representations are learned in an unsupervised manner , which makes it difficult to control the output speaking style in predictable manner . Another drawback of these methods is the large amount of data and computational resources required to train the models and generate new speech ( Yasuda et al. , 2020 ) . In this paper , we introduce the first fully-automated adaptive speech duration modification scheme . Our approach combines the representation capabilities of deep neural networks with the structured simplicity of dynamic decoding . Namely , we model the alignment between a source and target utterance via a latent attention map ; these maps are used as replacement of similarity matrix for backtracking . We train a masked convolutional encoder-decoder network to estimate these attention maps using a stochastic mean absolute error ( MAE ) formulation . Unlike the conventional DTW ( dtw , 2008 ) algorithm , once trained our framework operates entirely on the source utterance without needing to reference the target . We demonstrate our framework on a voice conversion task using the CMU-Arctic dataset ( Kominek & W Black , 2004 ) and on three multi-speaker emotion conversion tasks using the VESUS dataset ( Sager et al. , 2019 ) . Our experiments confirm that the proposed model can perform open-loop duration modification and produces high-quality speech . 2 METHOD . Fig . 1 illustrates our underlying generative process . Given an utterance X , we first estimate the length T of the ( unknown ) target utterance Y and subsequently use it to estimate a mask M for the attention map . The mask restricts the domain of the attention vectors At at each frame t during the inference stage to mitigate distortion of the output speech . We use paired data ( Xtr , Ytr ) to train a convolutional encoder-decoder network to generate the attention vectors . During testing , we first generate the attention map from the input X and use it to produce the target speech Y . 2.1 LOSS FUNCTION . Formally , let X ∈ RD×Ts denote the input speech . In this work , X corresponds to the Mel filterbank energies extracted from short-time moving window analysis , where D is the number of filterbanks , and Ts is the number of temporal frames in the utterance . Similarly , we denote the target speech as Y ∈ RD×T . Notice that the target utterance length T is usually different from Ts . Our generative process for a single frame of the target speech is represented as follows : T ∼ Laplace ( T 0 , bT ) and Yt ∼ Laplace ( Y 0t , by ) , ( 1 ) where T is the estimated length of the target utterance , and Yt is the target Mel filter-bank energy features at time t. The parameters { T 0 , bT , Y 0t , by } of the distributions are unknown and implicitly estimated via a deep neural network . The neural network is parameterized by γ and θ ( Fig . 1 ) . By treating the unknown parameters as functions of the input X , we obtain the following estimating equations for the target sequence length and frame-wise Mel filter-bank energies : T̂ = fγ ( X ) and Ŷt = X ·At + fθ ( X , Ŷ0 : t−1 ) . ( 2 ) The functions fγ ( · ) and fθ ( · , · ) correspond to the length prediction and energy estimation component of the same deep neural network . The variableAt ∈ RTs is an attention vector that combines framewise features of the source utterance X to generate the target frame Ŷt . Our model differs from standard sequence-to-sequence model by treating neural net predictions as residuals added to input sequence itself . Notice that the residuals depend on input and the history of predictions Ŷ0 : t−1 at previous time steps . This autoregressive property allows the neural network to learn segmental and supra-segmental variations that can potentially distinguish between the speakers or emotions . During training , we use paired data ( X , Y ) and maximize the likelihood of the target speech signal with respect to the neural network weights { θ , γ } . This likelihood can be written as : P ( Ŷ , T̂ |X ) = P ( T̂ |X ) T̂∏ t=1 P ( Ŷt|X , T̂ , Ŷ0 : t−1 ) , ( 3 ) where , the second term in Eq . ( 3 ) can be obtained via marginalization over At as follows : P ( Ŷt|X , T̂ , Ŷ0 : t−1 ) = ∑ At P ( Ŷt , At|X , T̂ , Ŷ0 : t−1 , M ) = ∑ At P ( Ŷt|X , T̂ , At , Ŷ0 : t−1 ) P ( At|X , Ŷ0 : t−1 , M ) ( 4 ) The variable M here denotes the attention mask . We introduce M for convenience , as it is a deterministic function of the source length Ts and the estimated target length T̂ . We encode the attention At as a one-hot vector across the Ts frames of the source speech . Thus , it follows a multinomial distribution . For simplicity , we model At as conditionally independent of the target length T̂ given the maskM and the inputX ( see Fig . 1 ) . Taking log ( · ) of Eq . ( 3 ) and combining with Eq . ( 4 ) yields : L ( θ , γ ) = − log ( ∑ At P ( Ŷt , At|X , T̂ , Ŷ0 : t−1 , M ) ) − log ( P ( T̂ |X ) ) = − log ( ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) qθ ( At|X , Ŷ0 : t−1 , M ) P ( Ŷt , At|X , T̂ , Ŷ0 : t−1 , M ) ) − log ( P ( T̂ |X ) ) ≤ − ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) − log ( P ( T̂ |X ) ) +KL ( qθ ( At ) ||P ( At ) ) = − ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) − log ( P ( T̂ |X ) −H ( qθ ) + const . ≤ − ∑ At qθ ( At|X , Ŷ0 : t−1 , M ) log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) − log ( P ( T̂ |X ) + const . ( 5 ) The distribution qθ ( · ) above is an approximating distribution for the attention vectors implemented by a convolutional network . The first inequality uses the convexity of the − log function , and the second inequality comes from the fact that entropy H ( qθ ) ≥ 0 . Notice that we have implicitly assumed P ( At|X , Ŷ0 : t−1 , M ) has a uniform distribution over the masked region . This is a reasonable assumption given that the masking process reduces the attention domain to a small region ( see Section 2.3 ) . However , qθ is not penalized for deviating from this uniform distribution during training . This flexibility allows the network to learn realistic attention vectors during autoregressive decoding . Eq . ( 5 ) can be easily translated into a neural network loss function which we minimize for { θ , γ } : L ( θ , γ ) = λ1 × EAt∼qθ [ log ( P ( Ŷt|X , At , Ŷ0 : t−1 ) ) ] + λ2 × log ( P ( T̂ |X ) ) = λ1 × EAt [ ‖Ŷt − Y 0t ‖1 ] + λ2 × ‖T̂ − T 0‖1 , ( 6 ) where λ1 and λ2 are the model hyperparameters that adjusts the trade-off between the two objectives and implicitly contain the variances of the Laplace distributions in Eq . ( 1 ) . Notice that the loss in Eq . ( 6 ) computes an expectation over the attention maps . We use the Monte-Carlo estimate by sampling from the attention map at each time-step . The training procedure is therefore stochastic in nature due to this random sampling . We mix this stochastic version with the maximum aposteriori estimate ( MAP ) of the attention vector with a probability of 0.2 during the start of training procedure . Algorithm 1 : Strategy for model training 1 function trainModelParameters ( X , Y ) ; Input : filterbank energies ( X ∈ RD×Ts , Y ∈ RD×Tt ) Output : model parameters ( θ , γ ) 2 if epoch < MaxEpochs then 3 Set t = 0 , predict target length T̂ = fγ ( X ) and create the mask M ∈ RTs×Tt ; 4 Estimate A ∈ RTs×Tt using masked convolution and sample u ∼ U ( 0 , 1 ) ; 5 if u < 0.2 then 6 Sample a ∈ RTs×Tt as 1-hot vectors from A ; 7 Reconstruct using Ŷt = X · a+ fθ ( X , Y0 : t−1 ) ; 8 else 9 Reconstruct using Ŷt = X ·A+ fθ ( X , Y0 : t−1 ) ; 10 end 11 Compute prediction errors and update parameters θ , γ ; 12 epoch← epoch + 1 ; 13 end 14 return θ and γ ;

Summary: This paper proposes a generative sequence-to-sequence encoder-decoder architecture with attention for modifying the length of an input sequence. The authors derive the training loss for learning the network parameters from a principled Bayesian formulation and a variational inference bound, and use this to train the model from paired {input, output} sequences. During inference, a task-specific network is capable to estimating a target length, and modify the signal accordingly. It would seem that the architecture is capable of handling other transformations, and in fact the authors apply the model to speaker- and emotion-morphing tasks, but the bulk of the evaluation is on how accurate the  target length is estimated, and (briefly) the perceptual quality of the resulting samples.