Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis

Recent work has explored sequence-to-sequence latent variable models for expressive speech synthesis ( supporting control and transfer of prosody and style ) , but has not presented a coherent framework for understanding the trade-offs between the competing methods . In this paper , we propose embedding capacity ( the amount of information the embedding contains about the data ) as a unified method of analyzing the behavior of latent variable models of speech , comparing existing heuristic ( non-variational ) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information . In our proposed model ( Capacitron ) , we show that by adding conditional dependencies to the variational posterior such that it matches the form of the true posterior , the same model can be used for high-precision prosody transfer , text-agnostic style transfer , and generation of natural-sounding prior samples . For multi-speaker models , Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior . Lastly , we introduce a method for decomposing embedding capacity hierarchically across two sets of latents , allowing a portion of the latent variability to be specified and the remaining variability sampled from a learned prior . Audio examples are available on the web1 . 1 INTRODUCTION . The synthesis of realistic human speech is a challenging problem that is important for natural humancomputer interaction . End-to-end neural network-based approaches have seen significant progress in recent years ( Wang et al. , 2017 ; Taigman et al. , 2018 ; Ping et al. , 2018 ; Sotelo et al. , 2017 ) , even matching human performance for short assistant-like utterances ( Shen et al. , 2018 ) . However , these neural models are sometimes viewed as less interpretable or controllable than more traditional models composed of multiple stages of processing that each operate on reified linguistic or phonetic representations . Text-to-speech ( TTS ) is an underdetermined problem , meaning the same text input has an infinite number of reasonable spoken realizations . In addition to speaker and channel characteristics , important sources of variability in TTS include intonation , stress , and rhythm ( collectively referred to as prosody ) . These attributes convey linguistic , semantic , and emotional meaning beyond what is present in the lexical representation ( i.e. , the text ) ( Wagner & Watson , 2010 ) . Recent end-to-end TTS research has aimed to model and/or directly control the remaining variability in the output . Skerry-Ryan et al . ( 2018 ) augment a Tacotron-like model ( Wang et al. , 2017 ) with a deterministic encoder that projects reference speech into a learned embedding space . The system can be used for prosody transfer between speakers ( “ say it like this ” ) , but does not work for transfer between unrelated sentences , and does not preserve the pitch range of the target speaker . Lee & Kim ( 2019 ) partially address the pitch range problem by centering the learned embeddings using speaker-wise means . 1https : //variational-embedding-capacity.github.io/demos/ Other work targets style transfer , a text-agnostic variation on prosody transfer . The Global Style Token ( GST ) system ( Wang et al. , 2018 ) uses a modified attention-based reference encoder to transfer global style properties to arbitrary text , and Ma et al . ( 2019 ) use an adversarial objective to disentangle style from text . Hsu et al . ( 2019 ) and Zhang et al . ( 2019 ) use a variational approach ( Kingma & Welling , 2014 ) to tackle the style task . Advantages of this approach include its ability to generate style samples via the accompanying prior and the potential for better disentangling between latent style factors ( Burgess et al. , 2018 ) . Additionally , Hsu et al . ( 2019 ) use a Gaussian mixture prior over the latents , which ( when interpreting the mixture component index as a high-level discrete latent ) allows a form of hierarchical control . This work extends the above approaches by providing the following contributions : 1 . We propose a unified approach for analyzing the characteristics of TTS latent variable models , independent of architecture , using the capacity of the learned embeddings ( i.e. , the representational mutual information between the embedding and the data ) . 2 . We target specific capacities for our proposed model using a Lagrange multiplier-based optimization scheme , and show that capacity is correlated with perceptual reference similarity . 3 . We show that modifying the variational posterior to match the form of the true posterior enables style and prosody transfer in the same model , helps preserve target speaker identity during inter-speaker transfer , and leads to natural-sounding prior samples even at high embedding capacities . 4 . We introduce a method for controlling what fraction of the variation represented in an embedding is specified , allowing the remaining variation to be sampled from the model . 2 MEASURING REFERENCE EMBEDDING CAPACITY . 2.1 LEARNING A REFERENCE EMBEDDING SPACE . Existing heuristic ( non-variational ) end-to-end approaches to prosody and style transfer ( Skerry-Ryan et al. , 2018 ; Wang et al. , 2018 ; Lee & Kim , 2019 ; Henter et al. , 2018 ) typically start with the teacherforced reconstruction loss , ( 1 ) , used to train Tacotron-like sequence-to-sequence models and simply augment the model with a deterministic reference encoder , ge ( x ) , as shown in eq . ( 2 ) . L ( x , yT , yS ) ≡ − log p ( x|yT , yS ) = ‖fθ ( yT , yS ) − x‖1 +K ( 1 ) L′ ( x , yT , yS ) ≡ − log p ( x|yT , yS , ge ( x ) ) = ‖fθ ( yT , yS , ge ( x ) ) − x‖1 +K ( 2 ) where x is an audio spectrogram , yT is the input text , yS is the target speaker ( if training a multispeaker model ) , fθ ( · ) is a deterministic function that maps the inputs to spectrogram predictions , and K is a normalization constant . Teacher-forcing implies that fθ ( · ) is dependent on x < t when predicting spectrogram frame xt . In practice , fθ ( · ) serves as the greedy deterministic output of the model , and transfer is accomplished by pairing the embedding computed by the reference encoder with different text or speakers during synthesis . In these heuristic models , the architecture chosen for the reference encoder determines the transfer characteristics of the model . This decision affects the information capacity of the embedding and allows the model to target a specific trade-off between transfer precision ( how closely the output resembles the reference ) and generality ( how well an embedding works when paired with arbitrary text ) . Higher capacity embeddings prioritize precision and are better suited for prosody transfer to similar text , while lower capacity embeddings prioritize generality and are better suited for text-agnostic style transfer . The variational extensions from Hsu et al . ( 2019 ) and Zhang et al . ( 2019 ) augment the reconstruction loss in eq . ( 2 ) with a KL divergence term . This encourages a stochastic reference encoder ( variational posterior ) , q ( z|x ) , to align well with a prior , p ( z ) ( eq . ( 3 ) ) . The overall loss is then equivalent to the negative evidence lower bound ( ELBO ) of the marginal likelihood of the data ( Kingma & Welling , 2014 ) . LELBO ( x , yT , yS ) ≡ Ez∼q ( z|x ) [ − log p ( x|z , yT , yS ) ] +DKL ( q ( z|x ) ‖p ( z ) ) ( 3 ) − log p ( x|yT , yS ) ≤ LELBO ( x , yT , yS ) ( 4 ) Controlling embedding capacity in variational models can be accomplished more directly by manipulating the KL term in ( 3 ) . Recent work has shown that the KL term provides an upper bound on the mutual information between the data , x , and the latent embedding , z ∼ q ( z|x ) ( Hoffman & Johnson , 2016 ; Makhzani et al. , 2015 ; Alemi et al. , 2018 ) . RAVG ≡ Ex∼pD ( x ) [ DKL ( q ( z|x ) ‖p ( z ) ) ] , R ≡ DKL ( q ( z|x ) ‖p ( z ) ) ( 5 ) Iq ( X ; Z ) ≡ Ex∼pD ( x ) [ DKL ( q ( z|x ) ‖q ( z ) ) ] , q ( z ) ≡ Ex∼pD ( x ) q ( z|x ) ( 6 ) RAVG = Iq ( X ; Z ) +DKL ( q ( z ) ‖p ( z ) ) ( 7 ) =⇒ Iq ( X ; Z ) ≤ RAVG ( 8 ) where pD ( x ) is the data distribution , R is the the KL term in ( 3 ) , RAVG is the KL term averaged over the data distribution , Iq ( X ; Z ) is the representational mutual information ( the capacity of z ) , and q ( z ) is the aggregated posterior . This brief derivation is expanded in Appendix C.1 . The bound in ( 8 ) follows from ( 7 ) and the non-negativity of the KL divergence , and ( 7 ) shows that the slack on the bound is DKL ( q ( z ) ‖p ( z ) ) , the aggregate KL . In addition to providing a tighter bound , having a low aggregate KL is desirable when sampling from the model via the prior , because then the samples of z that the decoder sees during training will be very similar to samples from the prior . Various approaches to controlling the KL term have been proposed , including varying a weight on the KL term , β ( Higgins et al. , 2017 ) , and penalizing its deviation from a target value ( Alemi et al. , 2018 ; Burgess et al. , 2018 ) . Because we would like to smoothly optimize for a specific bound on the embedding capacity , we adapt the Lagrange multiplier-based optimization approach of Rezende & Viola ( 2018 ) by applying it to the KL term rather than the reconstruction term . min θ max β≥0 { Ez∼qθ ( z|x ) [ − log pθ ( x|z , yT , yS ) ] + β ( DKL ( qθ ( z|x ) ‖p ( z ) ) − C ) } ( 9 ) where θ are the model parameters , β serves as an automatically-tuned weight on the KL term , C is the capacity limit , and updates to θ and β are interleaved during training . We constrain β to be non-negative by passing an unconstrained parameter through a softplus non-linearity , which makes the capacity constraint a limit rather than a target . This approach is less tedious than tuning β by hand and leads to more consistent behavior from run-to-run . It also allows more stable optimization than directly penalizing the ` 1 deviation from the target KL . 2.2 ESTIMATING EMBEDDING CAPACITY . Estimating heuristic embedding capacity Unfortunately , the heuristic methods do not come packaged with an easy way to estimate embedding capacity . We can estimate an effective capacity ordering , however , by measuring the test-time reconstruction loss when using the reference encoder from each method . In Figure 1 , we show how the reconstruction loss varies with embedding dimensionality for the tanh-based prosody transfer ( PT ) and softmax-based global style token ( GST ) bottlenecks ( Skerry-Ryan et al. , 2018 ; Wang et al. , 2018 ) and for variational models ( Var . ) with different capacity limits , C. We also compare to a baseline Tacotron model without a reference encoder . For this preliminary comparison , we use the expressive single-speaker dataset and training setup described in Section 4.2 . Looking at the heuristic methods in Figure 1 , we see that the GST bottleneck is much more restrictive than the PT bottleneck , which hurts transfer precision but allows sufficient embedding generality for text-agnostic style transfer . Bounding variational embedding capacity We saw in ( 8 ) that the KL term is an upper bound on embedding capacity , so we can directly target a specific capacity limit by constraining the KL term using the objective in eq . ( 9 ) . For the three values of C in Figure 1 , we can see that the reconstruction loss flattens out once the embedding reaches a certain dimensionality . This gives us a consistent way to control embedding capacity as it only requires using a reference encoder architecture with sufficient structural capacity ( at least C ) to achieve the desired representational capacity in the variational embedding . Because of this , we use 128-dimensional embeddings in all of our experiments , which should be sufficient for the range of capacities we target .

Summary: 1. Summary: This paper proposes Capacitron, a conditional variational latent variable model for TTS which allow for controllable latent variable capacity. They optimize the Lagrangian dual of the ELBO and restrict the capacity of the rate-term through a learnable, non-negative multiplier. They demonstrate the effectiveness of their approach on a range of TTS tasks such as same-text prosody transfer and inter-text style transfer, and provide extensive analyses on their latent variable capacity (in addition to comparisons to non-variational approaches based on Tacotron).