Hierarchical Generative Modeling for Controllable Speech Synthesis

1 INTRODUCTION . Recent development of neural sequence-to-sequence TTS models has shown promising results in generating high fidelity speech without the need of handcrafted linguistic features ( Sotelo et al. , 2017 ; Wang et al. , 2017 ; Arık et al. , 2017 ; Shen et al. , 2018 ) . These models rely heavily on a encoderdecoder neural network structure ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) that maps a text sequence to a sequence of speech frames . Extensions to these models have shown that attributes such as speaker identity can be controlled by conditioning the decoder on additional attribute labels ( Arik et al. , 2017 ; 2018 ; Jia et al. , 2018 ) . There are many speech attributes aside from speaker identity that are difficult to annotate , such as speaking style , prosody , recording channel , and noise levels . Skerry-Ryan et al . ( 2018 ) ; Wang et al . ( 2018 ) model such latent attributes through conditional auto-encoding , by extending the decoder inputs to include a vector inferred from the target speech which aims to capture the residual attributes that are not specified by other input streams , in addition to text and a speaker label . These models have shown convincing results in synthesizing speech that resembles the prosody or the noise conditions of the reference speech , which may not have the same text or speaker identity as the target speech . Nevertheless , the presence of multiple latent attributes is common in crowdsourced data such as ( Panayotov et al. , 2015 ) , in which prosody , speaker , and noise conditions all vary simultaneously . Using such data , simply copying the latent attributes from a reference is insufficient if one desires to synthesize speech that mimics the prosody of the reference , but is in the same noise condition as another . If the latent representation were disentangled , these generating factors could be controlled independently . Furthermore , it is can useful to construct a systematic method for synthesizing speech with random latent attributes , which would facilitate data augmentation ( Tjandra et al. , 2017 ; 2018 ; Hsu et al. , 2017b ; 2018 ; Hayashi et al. , 2018 ) by generating diverse examples . These properties were not explicitly addressed in the previous studies , which model variation of a single latent attribute . ∗Work performed while interning at Google Brain . Motivated by the applications of sampling , inferring , and independently controlling individual attributes , we build off of Skerry-Ryan et al . ( 2018 ) and extend Tacotron 2 ( Shen et al. , 2018 ) to model two separate latent spaces : one for labeled ( i.e . related to speaker identity ) and another for unlabeled attributes . Each latent variable is modeled in a variational autoencoding ( Kingma & Welling , 2014 ) framework using Gaussian mixture priors . The resulting latent spaces ( 1 ) learn disentangled attribute representations , where each dimension controls a different generating factor ; ( 2 ) discover a set of interpretable clusters , each of which corresponds to a representative mode in the training data ( e.g. , one cluster for clean speech and another for noisy speech ) ; and ( 3 ) provide a systematic sampling mechanism from the learned prior . The proposed model is extensively evaluated on four datasets with subjective and objective quantitative metrics , as well as comprehensive qualitative studies . Experiments confirm that the proposed model is capable of controlling speaker , noise , and style independently , even when variation of all attributes is present but unannotated in the train set . Our main contributions are as follows : • We propose a principled probabilistic hierarchical generative model , which improves ( 1 ) sampling stability and disentangled attribute control compared to e.g . the GST model of Wang et al . ( 2018 ) , and ( 2 ) interpretability and quality compared to e.g . Akuzawa et al . ( 2018 ) . • The model formulation explicitly factors the latent encoding by using two mixture distribu- tions to separately model supervised speaker attributes and latent attributes in a disentangled fashion . This makes it straightforward to condition the model output on speaker and latent encodings inferred from different reference utterances . • To the best of our knowledge , this work is the first to train a high-quality controllable text- to-speech system on real found data containing significant variation in recording condition , speaker identity , as well as prosody and style . Previous results on similar data focused on speaker modeling ( Ping et al. , 2018 ; Nachmani et al. , 2018 ; Arik et al. , 2018 ; Jia et al. , 2018 ) , and did not explicitly address modeling of prosody and background noise . Leveraging disentangled speaker and latent attribute encodings , the proposed model is capable of inferring the speaker attribute representation from a noisy utterance spoken by a previously unseen speaker , and using it to synthesize high-quality clean speech that approximates the voice of that speaker . 2 MODEL . Tacotron-like TTS systems take a text sequence Yt and an optional observed categorical label ( e.g . speaker identity ) yo as input , and use an autoregressive decoder to predict a sequence of acoustic features X frame by frame . Training such a system to minimize a mean squared error reconstruction loss can be regarded as fitting a probabilistic model p ( X | Yt , yo ) = ∏ n p ( xn | x1 , x2 , . . . , xn−1 , Yt , yo ) that maximizes the likelihood of generating the training data , where the conditional distribution of each frame xn is modeled as fixed-variance isotropic Gaussian whose mean is predicted by the decoder at step n. Such a model effectively integrates out other unlabeled latent attributes like prosody , and produces a conditional distribution with higher variance . As a result , the model would opaquely produce speech with unpredictable latent attributes . To enable control of those attributes , we adopt a graphical model with hierarchical latent variables , which captures such attributes . Below we explain how the formulation leads to interpretability and disentanglement , supports sampling , and propose efficient inference and training methods . 2.1 CONDITIONAL GENERATIVE MODEL WITH HIERARCHICAL LATENT VARIABLES . Two latent variables yl and zl are introduced in addition to the observed variables , X , Yt , and yo , as shown in the graphical model in the left of Figure 1. yl is a K-way categorical discrete variable , named latent attribute class , and zl is a D-dimensional continuous variable , named latent attribute representation . Throughout the paper , we use y∗ and z∗ to denote discrete and continuous variables , respectively . To generate speechX conditioned on the text Yt and observed attribute yo , yl is first sampled from its prior , p ( yl ) , then a latent attribute representation zl is sampled from the conditional distribution p ( zl | yl ) . Finally , a sequence of speech frames is drawn from p ( X | Yt , yo , zl ) , parameterized by the synthesizer neural network . The joint probability can be written as : p ( X , yl , zl | Yt , yo ) = p ( X | Yt , yo , zl ) p ( zl | yl ) p ( yl ) . ( 1 ) Specifically , it is assumed that p ( yl ) = K−1 to be a non-informative prior to encourage every component to be used , and p ( zl | yl ) = N ( µyl , diag ( σyl ) ) to be diagonal-covariance Gaussian with learnable means and variances . As a result , the marginal prior of zl becomes a GMM with diagonal covariances and equal mixture weights . We hope this GMM latent model can better capture the complexity of unseen attributes . Furthermore , in the presence of natural clusters of unseen attributes , the proposed model can achieve interpretability by learning to assign instances from different clusters to different mixture components . The covariance matrix of each mixture component is constrained to be diagonal to encourage each dimension to capture a statistically uncorrelated factor . 2.2 VARIATIONAL INFERENCE AND TRAINING . The conditional output distribution p ( X | Yt , yo , zl ) is parameterized with a neural network . Following the VAE framework of Kingma & Welling ( 2014 ) , a variational distribution q ( yl | X ) q ( zl | X ) is used to approximate the posterior p ( yl , zl | X , Yt , yo ) , which assumes that the posterior of unseen attributes is independent of the text and observed attributes . The approximated posterior for zl , q ( zl | X ) , is modeled as a Gaussian distribution with diagonal covariance matrix , whose mean and variance are parameterized by a neural network . For q ( yl | X ) , instead of introducing another neural network , we configure it to be an approximation of p ( yl | X ) that reuses q ( zl | X ) as follows : p ( yl|X ) = ∫ zl p ( yl | zl ) p ( zl|X ) dzl = Ep ( zl|X ) [ p ( yl | zl ) ] ≈ Eq ( zl|X ) [ p ( yl | zl ) ] : = q ( yl|X ) ( 2 ) which enjoys the closed-form solution of Gaussian mixture posteriors , p ( yl | zl ) . Similar to VAE , the model is trained by maximizing its evidence lower bound ( ELBO ) , as follows : L ( p , q ; X , Yt , yo ) = Eq ( zl|X ) [ log p ( X | Yt , yo , zl ) ] − Eq ( yl|X ) [ DKL ( q ( zl | X ) || p ( zl | yl ) ) ] −DKL ( q ( yl | X ) || p ( yl ) ) ( 3 ) where q ( zl | X ) is estimated via Monte Carlo sampling , and all components are differentiable thanks to reparameterization . Details can be found in Appendix A . 2.3 A CONTINUOUS ATTRIBUTE SPACE FOR CATEGORICAL OBSERVED LABELS . Categorical observed labels , such as speaker identity , can often be seen as a categorization from a continuous attribute space , which for example could model a speaker ’ s characteristic F0 range and vocal tract shape . Given an observed label , there may still be some variation of these attributes . We are interested in learning this continuous attribute space for modeling within-class variation and inferring a representation from an instance of an unseen class for one-shot learning . To achieve this , a continuous latent variable , zo , named the observed attribute representation , is introduced between the categorical observed label yo and speech X , as shown on the right of Figure 1 . Each observed class ( e.g . each speaker ) forms a mixture component in this continuous space , whose conditional distribution is a diagonal-covariance Gaussian p ( zo | yo ) = N ( µyo , diag ( σyo ) ) . With this formulation , speech from an observed class yo is now generated by conditioning on Yt , zl , and a sample zo drawn from p ( zo | yo ) . As before , a variational distribution q ( zo | X ) , parameterized by a neural network , is used to approximate the true posterior , where the ELBO becomes : Lo ( p , q ; X , Yt , yo ) = Eq ( zo|X ) q ( zl|X ) [ log p ( X | Yt , zo , zl ) ] −DKL ( q ( zo | X ) || p ( zo | yo ) ) − Eq ( yl|X ) [ DKL ( q ( zl | X ) || p ( zl | yl ) ) ] −DKL ( q ( yl | X ) || p ( yl ) ) . ( 4 ) To encourage zo to disentangle observed attributes from latent attributes , the variances of p ( zo | yo ) are initialized to be smaller than those of p ( zl | yl ) . The intuition is that this space should capture variation of attributes that are highly correlated with the observed labels , so the conditional distribution of all dimensions should have relatively small variance for each mixture component . Experimental results verify the effectiveness , and similar design is used in Hsu et al . ( 2017a ) . In the extreme case where the variance is fixed and approaches zero , this formulation converges to using an lookup table .

Summary: Quality: This submission claims to present a model that can control non-annotated attributes such as speaking style, accent, background noise, etc. Though empirical evidence in the form of numerical measurements is presented for some controllable attributes more evidence other than individual samples and authors claims is needed. For example a reliable numerical evidence is needed on page 4 following "We also found...", page 5 following "We discovered....", page 5 following "It clearly presents...", page 5 following "Drawing samples..." evidence is given only for 1 dimension, page 6 following "Figure 7(b)...".