Semi-Supervised Generative Modeling for Controllable Speech Synthesis

1 INTRODUCTION . The ability to reliably control high level attributes of speech , such as emotional expression ( affect ) or speaking rate , is often desirable in speech synthesis applications . Achieving this control however is made difficult by the necessity of acquiring a large quantity of high quality labels . In this paper we show that semi-supervised latent variable models can take us a significant step closer towards solving this problem . Combining state-of-the-art neural text-to-speech ( TTS ) systems with probabilistic latent variable models provides a natural framework for discovering aspects of speech that are rarely labelled or even difficult to describe . Both inferring the latent prosody and generating samples with sufficient variety requires reasoning about uncertainty and is thus a natural fit for deep generative models . There has been recent progress in applying stochastic gradient variational Bayes ( SGVB ) ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) to training probabilistic neural TTS models . Battenberg et al . ( 2019 ) and Hsu et al . ( 2018 ) have shown that it is possible to use latent variable models to discover features such as speaking style , speaking rate , arousal , gender and even the quality of the recording environment . However , these models are formally non-identifiable ( Hyvärinen & Pajunen , 1999 ; Locatello et al. , 2019 ) and this implies that repeated training runs will not reliably discover the same latent attributes . Even if they did , a lengthy human post-processing stage is necessary to identify what the model has learned on any given training run . In order to be of practical use for control , it is not enough for the models to discover latent attributes , they need to do so reliably and in a way that is robust to random initialization and to changes in the model . We demonstrate that the addition of even modest amounts of supervision can be sufficient to achieve this reliability . By augmenting state-of-the art neural TTS with semi-supervised deep generative models within the VAE framework ( Kingma et al. , 2014 ; Narayanaswamy et al. , 2017 ) , we show that it is possible to not only discover latent attributes of speech but to do so in a reliable and controllable manner . In particular we are able to achieve reliable control over affect , speaking rate and F0 variation ( F0 is the ∗Work performed while interning at Google Research . 1 https : //google.github.io/tacotron/publications/semisupervised_generative_modeling_for_ controllable_speech_synthesis/ fundamental frequency of oscillation of the vocal folds ) . Further , we provide demonstrations that it is possible to transfer controllability to speakers for whom we have no labels . Our core contributions are : • To combine semi-supervised latent variable models with Neural TTS systems , producing a system that can reliably discover attributes of speech we wish to control . • To demonstrate that as little as 30 minutes supervision can be sufficient to improve prosody and allow control over speaking rate , fundamental frequency ( F0 ) variation and affect , a problem of interest to the speech community for well over two decades ( Schröder , 2001 ) . • To imbue TTS models with control over affect , F0 and speaking rate whilst still maintaining prosodic variation when sampling . 2 GENERATIVE MODEL . Our generative model , shown in figures 1 and 2a , consists of an autoregressive distribution over a sequence of acoustic features , x1 ... t , that are generated conditioned on a sequence of text , y1 ... k , and on two latent variables , zu and zs . The latent variables can be discrete or continuous . zs represents the variations in prosody that we seek to control and is semi-supervised . zu is fully unobserved and represents latent variations in prosody ( intonation , rhythm , stress ) that we wish to model but not explicitly control . Once trained , our model can be used to synthesize acoustic features from text . Similar to Tacotron 2 ( Shen et al. , 2018 ) , we then generate waveforms by training a second network such as WaveNet ( van den Oord et al. , 2016 ) or WaveRNN ( Kalchbrenner et al. , 2018 ) to act as a vocoder . In our case we use WaveRNN . We parameterize our likelihood p ( x1 ... t|y1 ... k , zu , zs , θ ) by a sequence-to-sequence neural network with attention ( Shen et al. , 2018 ; Graves , 2013 ; Bahdanau et al. , 2014 ) that is shown schematically in figure 1 . Details largely follow Tacotron ( Wang et al. , 2017 ) and are given in appendix A . At each time step we model a mel-spectrogram frame with a fixed variance isotropic Laplace distribution whose mean is output by the neural network . We condition each of the latent variables by concatenating the vectors zu and zs to the representation of the text-encoder , before the application of the attention mechanism . In the case of continuous z we use a standard normal prior and in the case of discrete z we use a uniform categorical prior with one-hot encoding . 2.1 SEMI-SUPERVISED TRAINING . Following Kingma et al . ( 2014 ) ; Narayanaswamy et al . ( 2017 ) , we train our model via stochastic gradient variational Bayes ( SGVB ) . That is we approximately maximize the log-likelihood of our training data by maximizing a variational lower bound using stochastic gradient ascent . Since we are training with semi-supervision we in fact need two lower bounds : one for the data points for which zs is observed ; one for the case where zs is unobserved . In our models the fully latent variable zu is always continuous but the semi-supervised latent zs can be continuous or discrete . The conditional independence structure of our variational distributions is shown in figures 2b and 2c . On supervised data , the per-datapoint bound is : log p ( x , zs|y , θ ) = log ∫ p ( x , zu , zs|y , θ ) dzu ≥ Eq ( zu|x , y , zs , φ ) [ log ( p ( x|y , zu , zs , θ ) p ( zu ) p ( zs ) q ( zu|x , y , zs , φ ) ) ] = Eq ( zu|x , y , zs , φ ) [ log p ( x|y , zu , zs , θ ) ] + log p ( zs ) −DKL ( q ( zu|x , y , zs , φ ) ‖p ( zu ) ) = Ls ( θ , φ ; x , y , zs ) Where q ( zu|x , y , zs , φ ) is a parametric variational distribution introduced to approximately marginalize zu . θ are the parameters of the generative model and φ are the parameters of the variational distributions . The intractable integrals are approximated with reparameterized samples . For the cases where zs is unobserved and discrete , the bound is : log p ( x|y , θ ) = log ∫ ∑ zs p ( x , zu , zs|y , θ ) dzu ( 1 ) ≥ ∑ zs q ( zs|x , y , φ ) Ls ( θ , φ ; x , y , zs ) +H ( q ( zs|x , y , φ ) ) ( 2 ) = Lu ( θ , φ ; x , y ) ( 3 ) and when zs is continuous we replace the sum above with an integral and again approximate with reparameterized samples . The variational distributions are parameterized by a neural network that takes as input the text , spectrograms and other conditioning variables and outputs the parameters of the distribution . The exact structure of this network is given in appendix A . We have implicitly assumed that q ( zu , zs|x , y , φ ) may be factorized as q ( zu , zs|x , y , φ ) = q ( zu|x , y , zs , φ ) q ( zs|x , y , φ ) with shared parameters between these two distributions ( see appendix A ) . Optimizing the variational objective with respect to the parameters φ encourages the variational distributions to match the posterior of the generative model p ( zu , zs|x , y , θ ) . Unlike previous work ( Hsu et al. , 2018 ) , we do not assume that the posterior on the latents is independent of the text , as this dependence likely exists in the model due to explaining away . That is to say that although the text and the latents are independent in our prior , observing the spectrogram correlates them in the posterior because they both explain variation in the spectrogram . This has been shown to be significant by Battenberg et al . ( 2019 ) . If we define : q̃ ( zs|x , y ) = { q ( zs|x , y , φ ) if unsupervised γδ ( zs − zsobserved ) if supervised ( 4 ) then we can write the overall objective over both the supervised and unsupervised points succinctly as2 : L ( θ , φ ) = Ex , y [ ∑ zs q̃ ( zs|x , y , φ ) Ls ( θ , φ ; x , y , zs ) +H ( q̃ ( zs|x , y , φ ) ) ] ( 5 ) where summation would again be replaced by integration for continuous zs and γ ( shown in equation 4 ) is a weighting factor that pre-multiplies the loss for any supervised point . This weighting was also used in previous work such as Narayanaswamy et al . ( 2017 ) , who showed it to be beneficial at very low levels of supervision . Writing the objective in this form allows an intuitive interpretation for the semi-supervised training procedure . When supervision is provided , our objective function is evaluated at the observed value of zs . When supervision is not provided , we evaluate the objective function for every possible value of zs and take a ( potentially infinite for continuous zs ) weighted average . The weighting in the average is given by q ( zs|x , y , φ ) , which is simultaneously trained to approximate the posterior p ( zs|x , y , θ ) . In other words , on unsupervised utterances , we evaluate our objective for each possible value of the latent attribute and weight by the ( approximate ) posterior probability that this value of the latent was responsible for generating the utterance . As q ( zs|x , y , φ ) is trained to approximate p ( zs|x , y , θ ) we can expect it to become a reasonable classifier/regressor for the semi-supervised latent attribute as the model improves . For example when zs represents an affect label , p ( zs|x , y , θ ) is the posterior probability , of the model , over affect given text and speech . By taking the most likely posterior class , this distribution can be used as an affect classifier . However , this variational distribution is only trained on unsupervised training points and so does not benefit directly from the supervised data . To overcome this problem we follow Kingma et al . ( 2014 ) and add a classification loss to our objective . The overall objective becomes : Ltotal ( θ , φ ) = L ( θ , φ ) + αEx , y , zs [ log q ( zs|x , y , φ ) ] ( 6 ) Where α is a hyperparameter which adjusts the contribution of this term . 2We define the differential entropy of the delta function to be 0

Summary: This paper proposes to use semi-supervised learning to enforce interpretability on latent variables corresponding to properties like affect and speaking rate for text-to-speech synthesis. During training, only a few training items are annotated for these types of properties; for items where these labels are not given, the variables are marginalised out. TTS experiments are performed and the approach is evaluated objectively by training classifiers on top of the synthesised speech and subjectively in terms of mean opinion score.