Hierarchical Generative Modeling for Controllable Speech Synthesis

1 INTRODUCTION . Recent development of neural sequence-to-sequence TTS models has shown promising results in generating high fidelity speech without the need of handcrafted linguistic features ( Sotelo et al. , 2017 ; Wang et al. , 2017 ; Arık et al. , 2017 ; Shen et al. , 2018 ) . These models rely heavily on a encoderdecoder neural network structure ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) that maps a text sequence to a sequence of speech frames . Extensions to these models have shown that attributes such as speaker identity can be controlled by conditioning the decoder on additional attribute labels ( Arik et al. , 2017 ; 2018 ; Jia et al. , 2018 ) . There are many speech attributes aside from speaker identity that are difficult to annotate , such as speaking style , prosody , recording channel , and noise levels . Skerry-Ryan et al . ( 2018 ) ; Wang et al . ( 2018 ) model such latent attributes through conditional auto-encoding , by extending the decoder inputs to include a vector inferred from the target speech which aims to capture the residual attributes that are not specified by other input streams , in addition to text and a speaker label . These models have shown convincing results in synthesizing speech that resembles the prosody or the noise conditions of the reference speech , which may not have the same text or speaker identity as the target speech . Nevertheless , the presence of multiple latent attributes is common in crowdsourced data such as ( Panayotov et al. , 2015 ) , in which prosody , speaker , and noise conditions all vary simultaneously . Using such data , simply copying the latent attributes from a reference is insufficient if one desires to synthesize speech that mimics the prosody of the reference , but is in the same noise condition as another . If the latent representation were disentangled , these generating factors could be controlled independently . Furthermore , it is can useful to construct a systematic method for synthesizing speech with random latent attributes , which would facilitate data augmentation ( Tjandra et al. , 2017 ; 2018 ; Hsu et al. , 2017b ; 2018 ; Hayashi et al. , 2018 ) by generating diverse examples . These properties were not explicitly addressed in the previous studies , which model variation of a single latent attribute . ∗Work performed while interning at Google Brain . Motivated by the applications of sampling , inferring , and independently controlling individual attributes , we build off of Skerry-Ryan et al . ( 2018 ) and extend Tacotron 2 ( Shen et al. , 2018 ) to model two separate latent spaces : one for labeled ( i.e . related to speaker identity ) and another for unlabeled attributes . Each latent variable is modeled in a variational autoencoding ( Kingma & Welling , 2014 ) framework using Gaussian mixture priors . The resulting latent spaces ( 1 ) learn disentangled attribute representations , where each dimension controls a different generating factor ; ( 2 ) discover a set of interpretable clusters , each of which corresponds to a representative mode in the training data ( e.g. , one cluster for clean speech and another for noisy speech ) ; and ( 3 ) provide a systematic sampling mechanism from the learned prior . The proposed model is extensively evaluated on four datasets with subjective and objective quantitative metrics , as well as comprehensive qualitative studies . Experiments confirm that the proposed model is capable of controlling speaker , noise , and style independently , even when variation of all attributes is present but unannotated in the train set . Our main contributions are as follows : • We propose a principled probabilistic hierarchical generative model , which improves ( 1 ) sampling stability and disentangled attribute control compared to e.g . the GST model of Wang et al . ( 2018 ) , and ( 2 ) interpretability and quality compared to e.g . Akuzawa et al . ( 2018 ) . • The model formulation explicitly factors the latent encoding by using two mixture distribu- tions to separately model supervised speaker attributes and latent attributes in a disentangled fashion . This makes it straightforward to condition the model output on speaker and latent encodings inferred from different reference utterances . • To the best of our knowledge , this work is the first to train a high-quality controllable text- to-speech system on real found data containing significant variation in recording condition , speaker identity , as well as prosody and style . Previous results on similar data focused on speaker modeling ( Ping et al. , 2018 ; Nachmani et al. , 2018 ; Arik et al. , 2018 ; Jia et al. , 2018 ) , and did not explicitly address modeling of prosody and background noise . Leveraging disentangled speaker and latent attribute encodings , the proposed model is capable of inferring the speaker attribute representation from a noisy utterance spoken by a previously unseen speaker , and using it to synthesize high-quality clean speech that approximates the voice of that speaker . 2 MODEL . Tacotron-like TTS systems take a text sequence Yt and an optional observed categorical label ( e.g . speaker identity ) yo as input , and use an autoregressive decoder to predict a sequence of acoustic features X frame by frame . Training such a system to minimize a mean squared error reconstruction loss can be regarded as fitting a probabilistic model p ( X | Yt , yo ) = ∏ n p ( xn | x1 , x2 , . . . , xn−1 , Yt , yo ) that maximizes the likelihood of generating the training data , where the conditional distribution of each frame xn is modeled as fixed-variance isotropic Gaussian whose mean is predicted by the decoder at step n. Such a model effectively integrates out other unlabeled latent attributes like prosody , and produces a conditional distribution with higher variance . As a result , the model would opaquely produce speech with unpredictable latent attributes . To enable control of those attributes , we adopt a graphical model with hierarchical latent variables , which captures such attributes . Below we explain how the formulation leads to interpretability and disentanglement , supports sampling , and propose efficient inference and training methods . 2.1 CONDITIONAL GENERATIVE MODEL WITH HIERARCHICAL LATENT VARIABLES . Two latent variables yl and zl are introduced in addition to the observed variables , X , Yt , and yo , as shown in the graphical model in the left of Figure 1. yl is a K-way categorical discrete variable , named latent attribute class , and zl is a D-dimensional continuous variable , named latent attribute representation . Throughout the paper , we use y∗ and z∗ to denote discrete and continuous variables , respectively . To generate speechX conditioned on the text Yt and observed attribute yo , yl is first sampled from its prior , p ( yl ) , then a latent attribute representation zl is sampled from the conditional distribution p ( zl | yl ) . Finally , a sequence of speech frames is drawn from p ( X | Yt , yo , zl ) , parameterized by the synthesizer neural network . The joint probability can be written as : p ( X , yl , zl | Yt , yo ) = p ( X | Yt , yo , zl ) p ( zl | yl ) p ( yl ) . ( 1 ) Specifically , it is assumed that p ( yl ) = K−1 to be a non-informative prior to encourage every component to be used , and p ( zl | yl ) = N ( µyl , diag ( σyl ) ) to be diagonal-covariance Gaussian with learnable means and variances . As a result , the marginal prior of zl becomes a GMM with diagonal covariances and equal mixture weights . We hope this GMM latent model can better capture the complexity of unseen attributes . Furthermore , in the presence of natural clusters of unseen attributes , the proposed model can achieve interpretability by learning to assign instances from different clusters to different mixture components . The covariance matrix of each mixture component is constrained to be diagonal to encourage each dimension to capture a statistically uncorrelated factor . 2.2 VARIATIONAL INFERENCE AND TRAINING . The conditional output distribution p ( X | Yt , yo , zl ) is parameterized with a neural network . Following the VAE framework of Kingma & Welling ( 2014 ) , a variational distribution q ( yl | X ) q ( zl | X ) is used to approximate the posterior p ( yl , zl | X , Yt , yo ) , which assumes that the posterior of unseen attributes is independent of the text and observed attributes . The approximated posterior for zl , q ( zl | X ) , is modeled as a Gaussian distribution with diagonal covariance matrix , whose mean and variance are parameterized by a neural network . For q ( yl | X ) , instead of introducing another neural network , we configure it to be an approximation of p ( yl | X ) that reuses q ( zl | X ) as follows : p ( yl|X ) = ∫ zl p ( yl | zl ) p ( zl|X ) dzl = Ep ( zl|X ) [ p ( yl | zl ) ] ≈ Eq ( zl|X ) [ p ( yl | zl ) ] : = q ( yl|X ) ( 2 ) which enjoys the closed-form solution of Gaussian mixture posteriors , p ( yl | zl ) . Similar to VAE , the model is trained by maximizing its evidence lower bound ( ELBO ) , as follows : L ( p , q ; X , Yt , yo ) = Eq ( zl|X ) [ log p ( X | Yt , yo , zl ) ] − Eq ( yl|X ) [ DKL ( q ( zl | X ) || p ( zl | yl ) ) ] −DKL ( q ( yl | X ) || p ( yl ) ) ( 3 ) where q ( zl | X ) is estimated via Monte Carlo sampling , and all components are differentiable thanks to reparameterization . Details can be found in Appendix A . 2.3 A CONTINUOUS ATTRIBUTE SPACE FOR CATEGORICAL OBSERVED LABELS . Categorical observed labels , such as speaker identity , can often be seen as a categorization from a continuous attribute space , which for example could model a speaker ’ s characteristic F0 range and vocal tract shape . Given an observed label , there may still be some variation of these attributes . We are interested in learning this continuous attribute space for modeling within-class variation and inferring a representation from an instance of an unseen class for one-shot learning . To achieve this , a continuous latent variable , zo , named the observed attribute representation , is introduced between the categorical observed label yo and speech X , as shown on the right of Figure 1 . Each observed class ( e.g . each speaker ) forms a mixture component in this continuous space , whose conditional distribution is a diagonal-covariance Gaussian p ( zo | yo ) = N ( µyo , diag ( σyo ) ) . With this formulation , speech from an observed class yo is now generated by conditioning on Yt , zl , and a sample zo drawn from p ( zo | yo ) . As before , a variational distribution q ( zo | X ) , parameterized by a neural network , is used to approximate the true posterior , where the ELBO becomes : Lo ( p , q ; X , Yt , yo ) = Eq ( zo|X ) q ( zl|X ) [ log p ( X | Yt , zo , zl ) ] −DKL ( q ( zo | X ) || p ( zo | yo ) ) − Eq ( yl|X ) [ DKL ( q ( zl | X ) || p ( zl | yl ) ) ] −DKL ( q ( yl | X ) || p ( yl ) ) . ( 4 ) To encourage zo to disentangle observed attributes from latent attributes , the variances of p ( zo | yo ) are initialized to be smaller than those of p ( zl | yl ) . The intuition is that this space should capture variation of attributes that are highly correlated with the observed labels , so the conditional distribution of all dimensions should have relatively small variance for each mixture component . Experimental results verify the effectiveness , and similar design is used in Hsu et al . ( 2017a ) . In the extreme case where the variance is fixed and approaches zero , this formulation converges to using an lookup table .

Summary: The authors describe the conditioned GAN model to generate speaker conditioned Mel spectra. They augment the z-space corresponding to the identification with latent variables that allow a richer set of produced audio. In a way this is like a partially conditioned model that has "extra" degrees of freedom. It looks that the "latent" variables are just concaneted to the "original" set of z-values (altough with particular conditions to maximize independence). The conditioning of the z-space has originality in it and may provide interesting to the audience. Ultimately one coud think about z-space direction being totally mapped to specific features of the produced signal.