Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis

Recent work has explored sequence-to-sequence latent variable models for expressive speech synthesis ( supporting control and transfer of prosody and style ) , but has not presented a coherent framework for understanding the trade-offs between the competing methods . In this paper , we propose embedding capacity ( the amount of information the embedding contains about the data ) as a unified method of analyzing the behavior of latent variable models of speech , comparing existing heuristic ( non-variational ) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information . In our proposed model ( Capacitron ) , we show that by adding conditional dependencies to the variational posterior such that it matches the form of the true posterior , the same model can be used for high-precision prosody transfer , text-agnostic style transfer , and generation of natural-sounding prior samples . For multi-speaker models , Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior . Lastly , we introduce a method for decomposing embedding capacity hierarchically across two sets of latents , allowing a portion of the latent variability to be specified and the remaining variability sampled from a learned prior . Audio examples are available on the web1 . 1 INTRODUCTION . The synthesis of realistic human speech is a challenging problem that is important for natural humancomputer interaction . End-to-end neural network-based approaches have seen significant progress in recent years ( Wang et al. , 2017 ; Taigman et al. , 2018 ; Ping et al. , 2018 ; Sotelo et al. , 2017 ) , even matching human performance for short assistant-like utterances ( Shen et al. , 2018 ) . However , these neural models are sometimes viewed as less interpretable or controllable than more traditional models composed of multiple stages of processing that each operate on reified linguistic or phonetic representations . Text-to-speech ( TTS ) is an underdetermined problem , meaning the same text input has an infinite number of reasonable spoken realizations . In addition to speaker and channel characteristics , important sources of variability in TTS include intonation , stress , and rhythm ( collectively referred to as prosody ) . These attributes convey linguistic , semantic , and emotional meaning beyond what is present in the lexical representation ( i.e. , the text ) ( Wagner & Watson , 2010 ) . Recent end-to-end TTS research has aimed to model and/or directly control the remaining variability in the output . Skerry-Ryan et al . ( 2018 ) augment a Tacotron-like model ( Wang et al. , 2017 ) with a deterministic encoder that projects reference speech into a learned embedding space . The system can be used for prosody transfer between speakers ( “ say it like this ” ) , but does not work for transfer between unrelated sentences , and does not preserve the pitch range of the target speaker . Lee & Kim ( 2019 ) partially address the pitch range problem by centering the learned embeddings using speaker-wise means . 1https : //variational-embedding-capacity.github.io/demos/ Other work targets style transfer , a text-agnostic variation on prosody transfer . The Global Style Token ( GST ) system ( Wang et al. , 2018 ) uses a modified attention-based reference encoder to transfer global style properties to arbitrary text , and Ma et al . ( 2019 ) use an adversarial objective to disentangle style from text . Hsu et al . ( 2019 ) and Zhang et al . ( 2019 ) use a variational approach ( Kingma & Welling , 2014 ) to tackle the style task . Advantages of this approach include its ability to generate style samples via the accompanying prior and the potential for better disentangling between latent style factors ( Burgess et al. , 2018 ) . Additionally , Hsu et al . ( 2019 ) use a Gaussian mixture prior over the latents , which ( when interpreting the mixture component index as a high-level discrete latent ) allows a form of hierarchical control . This work extends the above approaches by providing the following contributions : 1 . We propose a unified approach for analyzing the characteristics of TTS latent variable models , independent of architecture , using the capacity of the learned embeddings ( i.e. , the representational mutual information between the embedding and the data ) . 2 . We target specific capacities for our proposed model using a Lagrange multiplier-based optimization scheme , and show that capacity is correlated with perceptual reference similarity . 3 . We show that modifying the variational posterior to match the form of the true posterior enables style and prosody transfer in the same model , helps preserve target speaker identity during inter-speaker transfer , and leads to natural-sounding prior samples even at high embedding capacities . 4 . We introduce a method for controlling what fraction of the variation represented in an embedding is specified , allowing the remaining variation to be sampled from the model . 2 MEASURING REFERENCE EMBEDDING CAPACITY . 2.1 LEARNING A REFERENCE EMBEDDING SPACE . Existing heuristic ( non-variational ) end-to-end approaches to prosody and style transfer ( Skerry-Ryan et al. , 2018 ; Wang et al. , 2018 ; Lee & Kim , 2019 ; Henter et al. , 2018 ) typically start with the teacherforced reconstruction loss , ( 1 ) , used to train Tacotron-like sequence-to-sequence models and simply augment the model with a deterministic reference encoder , ge ( x ) , as shown in eq . ( 2 ) . L ( x , yT , yS ) ≡ − log p ( x|yT , yS ) = ‖fθ ( yT , yS ) − x‖1 +K ( 1 ) L′ ( x , yT , yS ) ≡ − log p ( x|yT , yS , ge ( x ) ) = ‖fθ ( yT , yS , ge ( x ) ) − x‖1 +K ( 2 ) where x is an audio spectrogram , yT is the input text , yS is the target speaker ( if training a multispeaker model ) , fθ ( · ) is a deterministic function that maps the inputs to spectrogram predictions , and K is a normalization constant . Teacher-forcing implies that fθ ( · ) is dependent on x < t when predicting spectrogram frame xt . In practice , fθ ( · ) serves as the greedy deterministic output of the model , and transfer is accomplished by pairing the embedding computed by the reference encoder with different text or speakers during synthesis . In these heuristic models , the architecture chosen for the reference encoder determines the transfer characteristics of the model . This decision affects the information capacity of the embedding and allows the model to target a specific trade-off between transfer precision ( how closely the output resembles the reference ) and generality ( how well an embedding works when paired with arbitrary text ) . Higher capacity embeddings prioritize precision and are better suited for prosody transfer to similar text , while lower capacity embeddings prioritize generality and are better suited for text-agnostic style transfer . The variational extensions from Hsu et al . ( 2019 ) and Zhang et al . ( 2019 ) augment the reconstruction loss in eq . ( 2 ) with a KL divergence term . This encourages a stochastic reference encoder ( variational posterior ) , q ( z|x ) , to align well with a prior , p ( z ) ( eq . ( 3 ) ) . The overall loss is then equivalent to the negative evidence lower bound ( ELBO ) of the marginal likelihood of the data ( Kingma & Welling , 2014 ) . LELBO ( x , yT , yS ) ≡ Ez∼q ( z|x ) [ − log p ( x|z , yT , yS ) ] +DKL ( q ( z|x ) ‖p ( z ) ) ( 3 ) − log p ( x|yT , yS ) ≤ LELBO ( x , yT , yS ) ( 4 ) Controlling embedding capacity in variational models can be accomplished more directly by manipulating the KL term in ( 3 ) . Recent work has shown that the KL term provides an upper bound on the mutual information between the data , x , and the latent embedding , z ∼ q ( z|x ) ( Hoffman & Johnson , 2016 ; Makhzani et al. , 2015 ; Alemi et al. , 2018 ) . RAVG ≡ Ex∼pD ( x ) [ DKL ( q ( z|x ) ‖p ( z ) ) ] , R ≡ DKL ( q ( z|x ) ‖p ( z ) ) ( 5 ) Iq ( X ; Z ) ≡ Ex∼pD ( x ) [ DKL ( q ( z|x ) ‖q ( z ) ) ] , q ( z ) ≡ Ex∼pD ( x ) q ( z|x ) ( 6 ) RAVG = Iq ( X ; Z ) +DKL ( q ( z ) ‖p ( z ) ) ( 7 ) =⇒ Iq ( X ; Z ) ≤ RAVG ( 8 ) where pD ( x ) is the data distribution , R is the the KL term in ( 3 ) , RAVG is the KL term averaged over the data distribution , Iq ( X ; Z ) is the representational mutual information ( the capacity of z ) , and q ( z ) is the aggregated posterior . This brief derivation is expanded in Appendix C.1 . The bound in ( 8 ) follows from ( 7 ) and the non-negativity of the KL divergence , and ( 7 ) shows that the slack on the bound is DKL ( q ( z ) ‖p ( z ) ) , the aggregate KL . In addition to providing a tighter bound , having a low aggregate KL is desirable when sampling from the model via the prior , because then the samples of z that the decoder sees during training will be very similar to samples from the prior . Various approaches to controlling the KL term have been proposed , including varying a weight on the KL term , β ( Higgins et al. , 2017 ) , and penalizing its deviation from a target value ( Alemi et al. , 2018 ; Burgess et al. , 2018 ) . Because we would like to smoothly optimize for a specific bound on the embedding capacity , we adapt the Lagrange multiplier-based optimization approach of Rezende & Viola ( 2018 ) by applying it to the KL term rather than the reconstruction term . min θ max β≥0 { Ez∼qθ ( z|x ) [ − log pθ ( x|z , yT , yS ) ] + β ( DKL ( qθ ( z|x ) ‖p ( z ) ) − C ) } ( 9 ) where θ are the model parameters , β serves as an automatically-tuned weight on the KL term , C is the capacity limit , and updates to θ and β are interleaved during training . We constrain β to be non-negative by passing an unconstrained parameter through a softplus non-linearity , which makes the capacity constraint a limit rather than a target . This approach is less tedious than tuning β by hand and leads to more consistent behavior from run-to-run . It also allows more stable optimization than directly penalizing the ` 1 deviation from the target KL . 2.2 ESTIMATING EMBEDDING CAPACITY . Estimating heuristic embedding capacity Unfortunately , the heuristic methods do not come packaged with an easy way to estimate embedding capacity . We can estimate an effective capacity ordering , however , by measuring the test-time reconstruction loss when using the reference encoder from each method . In Figure 1 , we show how the reconstruction loss varies with embedding dimensionality for the tanh-based prosody transfer ( PT ) and softmax-based global style token ( GST ) bottlenecks ( Skerry-Ryan et al. , 2018 ; Wang et al. , 2018 ) and for variational models ( Var . ) with different capacity limits , C. We also compare to a baseline Tacotron model without a reference encoder . For this preliminary comparison , we use the expressive single-speaker dataset and training setup described in Section 4.2 . Looking at the heuristic methods in Figure 1 , we see that the GST bottleneck is much more restrictive than the PT bottleneck , which hurts transfer precision but allows sufficient embedding generality for text-agnostic style transfer . Bounding variational embedding capacity We saw in ( 8 ) that the KL term is an upper bound on embedding capacity , so we can directly target a specific capacity limit by constraining the KL term using the objective in eq . ( 9 ) . For the three values of C in Figure 1 , we can see that the reconstruction loss flattens out once the embedding reaches a certain dimensionality . This gives us a consistent way to control embedding capacity as it only requires using a reference encoder architecture with sufficient structural capacity ( at least C ) to achieve the desired representational capacity in the variational embedding . Because of this , we use 128-dimensional embeddings in all of our experiments , which should be sufficient for the range of capacities we target .

Summary: In this work authors present a regularized, variational autoencoder method for speech synthesis. To endow the latent space with more capacity, the authors employ a modified variational autoencoder objective, which uses a learnable Lagrange multiplier to impose a capacity limit on KL divergence between latent posterior and prior. The authors furthermore propose to decompose the latent embedding space into a two-level hierarchical representation to give generative process more control over style transfer and sample-to-sample variance. They extend earlier theoretical results providing upper bounds on the mutual information between data and its latent embedding to their hierarchical latent representation. In numerical experiments the authors evaluate their approach on a number of speech synthesis tasks involving same-text prosody transfer, inter-text style transfer, inter-speaker prosody transfer. They also analyze speech samples generated from latent samples drawn from the prior.