Title: EFFECTIVE USE OF VARIATIONAL EMBEDDING CAPACITY IN EXPRESSIVE END-TO-END SPEECH SYNTHESIS Anonymous authors Paper under double-blind review

Abstract: Recent work has explored sequence-to-sequence latent variable models for expressive speech synthesis (supporting control and transfer of prosody and style), but has not presented a coherent framework for understanding the trade-offs between the competing methods. In this paper, we propose embedding capacity (the amount of information the embedding contains about the data) as a unified method of analyzing the behavior of latent variable models of speech, comparing existing heuristic (non-variational) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information. In our proposed model (Capacitron), we show that by adding conditional dependencies to the variational posterior such that it matches the form of the true posterior, the same model can be used for high-precision prosody transfer, text-agnostic style transfer, and generation of natural-sounding prior samples. For multi-speaker models, Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior. Lastly, we introduce a method for decomposing embedding capacity hierarchically across two sets of latents, allowing a portion of the latent variability to be specified and the remaining variability sampled from a learned prior. Audio examples are available on the web 1 .

1. INTRODUCTION:

The synthesis of realistic human speech is a challenging problem that is important for natural humancomputer interaction. End-to-end neural network-based approaches have seen significant progress in recent years (Wang et al., 2017; Taigman et al., 2018; Ping et al., 2018; Sotelo et al., 2017) , even matching human performance for short assistant-like utterances (Shen et al., 2018) . However, these neural models are sometimes viewed as less interpretable or controllable than more traditional models composed of multiple stages of processing that each operate on reified linguistic or phonetic representations.

Text-to-speech (TTS) is an underdetermined problem, meaning the same text input has an infinite number of reasonable spoken realizations. In addition to speaker and channel characteristics, important sources of variability in TTS include intonation, stress, and rhythm (collectively referred to as prosody). These attributes convey linguistic, semantic, and emotional meaning beyond what is present in the lexical representation (i.e., the text) (Wagner & Watson, 2010) . Recent end-to-end TTS research has aimed to model and/or directly control the remaining variability in the output. Skerry-Ryan et al. (2018) augment a Tacotron-like model (Wang et al., 2017) with a deterministic encoder that projects reference speech into a learned embedding space. The system can be used for prosody transfer between speakers ("say it like this"), but does not work for transfer between unrelated sentences, and does not preserve the pitch range of the target speaker. Lee & Kim (2019) partially address the pitch range problem by centering the learned embeddings using speaker-wise means.

Other work targets style transfer, a text-agnostic variation on prosody transfer. The Global Style Token (GST) system (Wang et al., 2018) uses a modified attention-based reference encoder to transfer global style properties to arbitrary text, and Ma et al. (2019) use an adversarial objective to disentangle style from text. Hsu et al. (2019) and Zhang et al. (2019) use a variational approach (Kingma & Welling, 2014) to tackle the style task. Advantages of this approach include its ability to generate style samples via the accompanying prior and the potential for better disentangling between latent style factors (Burgess et al., 2018) . Additionally, Hsu et al. (2019) use a Gaussian mixture prior over the latents, which (when interpreting the mixture component index as a high-level discrete latent) allows a form of hierarchical control. This work extends the above approaches by providing the following contributions:

1. We propose a unified approach for analyzing the characteristics of TTS latent variable models, independent of architecture, using the capacity of the learned embeddings (i.e., the representational mutual information between the embedding and the data). 2. We target specific capacities for our proposed model using a Lagrange multiplier-based optimization scheme, and show that capacity is correlated with perceptual reference similarity. 3. We show that modifying the variational posterior to match the form of the true posterior enables style and prosody transfer in the same model, helps preserve target speaker identity during inter-speaker transfer, and leads to natural-sounding prior samples even at high embedding capacities. 4. We introduce a method for controlling what fraction of the variation represented in an embedding is specified, allowing the remaining variation to be sampled from the model.

2. MEASURING REFERENCE EMBEDDING CAPACITY:



2.1. LEARNING A REFERENCE EMBEDDING SPACE:

Existing heuristic (non-variational) end-to-end approaches to prosody and style transfer (Skerry-Ryan et al., 2018; Wang et al., 2018; Lee & Kim, 2019; Henter et al., 2018) typically start with the teacherforced reconstruction loss, (1), used to train Tacotron-like sequence-to-sequence models and simply augment the model with a deterministic reference encoder, g e (x), as shown in eq. (2).

L(x, y T , y S ) ≡log p(x|y T , y S ) = f θ (y T , y S )x 1 + K (1)

L (x, y T , y S ) ≡log p(x|y T , y S , g e (x)) = f θ (y T , y S , g e (x))x 1 + K (2) where x is an audio spectrogram, y T is the input text, y S is the target speaker (if training a multispeaker model), f θ (•) is a deterministic function that maps the inputs to spectrogram predictions, and K is a normalization constant. Teacher-forcing implies that f θ (•) is dependent on x <t when predicting spectrogram frame x t . In practice, f θ (•) serves as the greedy deterministic output of the model, and transfer is accomplished by pairing the embedding computed by the reference encoder with different text or speakers during synthesis.

In these heuristic models, the architecture chosen for the reference encoder determines the transfer characteristics of the model. This decision affects the information capacity of the embedding and allows the model to target a specific trade-off between transfer precision (how closely the output resembles the reference) and generality (how well an embedding works when paired with arbitrary text). Higher capacity embeddings prioritize precision and are better suited for prosody transfer to similar text, while lower capacity embeddings prioritize generality and are better suited for text-agnostic style transfer.

The variational extensions from Hsu et al. (2019) and Zhang et al. (2019) augment the reconstruction loss in eq. ( 2) with a KL divergence term. This encourages a stochastic reference encoder (variational posterior), q(z|x), to align well with a prior, p(z) (eq. ( 3)). The overall loss is then equivalent to the negative evidence lower bound (ELBO) of the marginal likelihood of the data (Kingma & Welling, 2014) .

L ELBO (x, y T , y S ) ≡ E z∼q(z|x) [-log p(x|z, y T , y S )] + D KL (q(z|x) p(z))

(3)

log p(x|y T , y S ) ≤ L ELBO (x, y T , y S ) (4)

Controlling embedding capacity in variational models can be accomplished more directly by manipulating the KL term in (3). Recent work has shown that the KL term provides an upper bound on the mutual information between the data, x, and the latent embedding, z ∼ q(z|x) (Hoffman & Johnson, 2016; Makhzani et al., 2015; Alemi et al., 2018) .

R AVG ≡ E x∼p D (x)

[D KL (q(z|x) p(z))], R ≡ D KL (q(z|x) p(z))

(5)

I q (X; Z) ≡ E x∼p D (x) [D KL (q(z|x) q(z))], q(z) ≡ E x∼p D (x) q(z|x) (6) R AVG = I q (X; Z) + D KL (q(z) p(z)) (7) =⇒ I q (X; Z) ≤ R AVG (8)

where p D (x) is the data distribution, R is the the KL term in (3), R AVG is the KL term averaged over the data distribution, I q (X; Z) is the representational mutual information (the capacity of z), and q(z) is the aggregated posterior. This brief derivation is expanded in Appendix C.1.

The bound in (8) follows from ( 7) and the non-negativity of the KL divergence, and ( 7) shows that the slack on the bound is D KL (q(z) p(z)), the aggregate KL. In addition to providing a tighter bound, having a low aggregate KL is desirable when sampling from the model via the prior, because then the samples of z that the decoder sees during training will be very similar to samples from the prior.

Various approaches to controlling the KL term have been proposed, including varying a weight on the KL term, β (Higgins et al., 2017) , and penalizing its deviation from a target value (Alemi et al., 2018; Burgess et al., 2018) . Because we would like to smoothly optimize for a specific bound on the embedding capacity, we adapt the Lagrange multiplier-based optimization approach of Rezende & Viola (2018) by applying it to the KL term rather than the reconstruction term.

EQUATION

where θ are the model parameters, β serves as an automatically-tuned weight on the KL term, C is the capacity limit, and updates to θ and β are interleaved during training. We constrain β to be non-negative by passing an unconstrained parameter through a softplus non-linearity, which makes the capacity constraint a limit rather than a target. This approach is less tedious than tuning β by hand and leads to more consistent behavior from run-to-run. It also allows more stable optimization than directly penalizing the 1 deviation from the target KL.

2.2. ESTIMATING EMBEDDING CAPACITY:

Estimating heuristic embedding capacity Unfortunately, the heuristic methods do not come packaged with an easy way to estimate embedding capacity. We can estimate an effective capacity ordering, however, by measuring the test-time reconstruction loss when using the reference encoder from each method. In Figure 1 , we show how the reconstruction loss varies with embedding dimensionality for the tanh-based prosody transfer (PT) and softmax-based global style token (GST) bottlenecks (Skerry-Ryan et al., 2018; Wang et al., 2018) and for variational models (Var.) with different capacity limits, C. We also compare to a baseline Tacotron model without a reference encoder. For this preliminary comparison, we use the expressive single-speaker dataset and training setup described in Section 4.2. Looking at the heuristic methods in Figure 1 , we see that the GST bottleneck is much more restrictive than the PT bottleneck, which hurts transfer precision but allows sufficient embedding generality for text-agnostic style transfer.

Bounding variational embedding capacity We saw in (8) that the KL term is an upper bound on embedding capacity, so we can directly target a specific capacity limit by constraining the KL term using the objective in eq. ( 9). For the three values of C in Figure 1 , we can see that the reconstruction loss flattens out once the embedding reaches a certain dimensionality. This gives us a consistent way to control embedding capacity as it only requires using a reference encoder architecture with sufficient structural capacity (at least C) to achieve the desired representational capacity in the variational embedding. Because of this, we use 128-dimensional embeddings in all of our experiments, which should be sufficient for the range of capacities we target. 

3. MAKING EFFECTIVE USE OF EMBEDDING CAPACITY:



3.1. MATCHING THE FORM OF THE TRUE POSTERIOR:

In previous work (Hsu et al., 2019; Zhang et al., 2019) , the variational posterior has the form q(z|x), which matches the form of the true posterior for a simple generative model p(x|z)p(z). However, for the conditional generative model used in TTS, p(x|z, y T , y S )p(z), it is missing conditional dependencies present in the true posterior, p(z|x, y T , y S ). Figure 2 shows this visually. In order to

y T < l a t e x i t s h a _ b a s e = " d a Y p T y C b A H O Q E z b t X m V H M = " > A A A D Z H i c j Z L P a s w F M f V e D r v a l Z G x S w M d g p e h v K y H w H p o a d O G x i H I s p y K S r K Q n t s x s c d d + l Z f + w f G X / x O S k L I d w R Y v r z P n P R p H i z I D v / x r O Q e P n q / s R u v H s + Y v N r Z d n J s o T S l T I w o Z L g A G n f a U p F h G n H V x q f X N t W C p P I V d K P B Y s o Q R D L Z E u Y w m z H X W X j M E d H c f t e v O d H o W z B O S S a o B M K x M Y P A V z A s s A Z G O C d M D N U Y X K F x R g o S C m m E x X t p O C W p t o E b a P F g Y U w u I m s K D J e m z q r m K j b I I N k f F k y q D K g k O S j H u Q e t W H e z H T l A D P b c B E M u r R y x x g T s F D S W I K g S W c R F O y y K s D o i S Y l q W y y y Y J M y E B I U A p A F s V p w z M r v Z O N X F / n r H T a w / / z E q v / C / / T c Z f N i I V U J f s H u Q R V X B Z q K l f U L p e t G c O F e Z / c a B / s o m t o f o D X q H A v Q e H a A u O k I R N A Y f U X f P f W L f D X Z e z d X W t M N l o q Z + c P X x q d w = = < / l a t e x i t > y S

< l a t e x i t s h a 1 _ b a s e 6 4 = " w Z q e 2 E I n g q 9 x M t P t d P C q r / e t E O s = " > A A A D Z H i c j Z L P a 9 s w F M f V e D 8 6 r 9 v a l Z 0 x < l a t e x i t s h a 1 _ b a s e 6 4 = " d 7

G x S w M d g p 2 e 1 h v K 2 y H w H p o a d O G x i H I s p y K S r K Q n t s 4 x s c d d 9 3 + l Z 7 2 f + w f 2 G X / x O S k L I 2 d w R 4 Y v r z P 5 1 n P R p H i z I D v / 1 x r O Q 8 e P n q 8 / s R 9 u v H s + Y v N r Z d n J s 0 0 o T 2 S 8 l T 3 I 2 w o Z 5 L 2 g A G n f a U p F h G n 5 9 H V x 4 q f X 1 N t W C p P I V d 0 K P B Y s o Q R D L Z 1 E u Z m t N n 2 O / 6 s v G Y I 7 k L 7 w 4 / b 4 9 9 f d m 6 P R l u t z 2 G c k k x Q C Y R j Y w a B r 2 B Y Y A 2 M c F q 6 Y W a o w u Q K j + n A R o k F N c N i t m v p v b W d 2 E t S b R 8 J 3 q x 7 f 6 L A w p h c R N Y U G C 5 N n V X N V W y Q Q b I / L J h U G V B J 5 g c l G f c g 9 a o P 9 2 K m K Q G e 2 4 C J Z n Z X j 1 x i j Q n Y 3 + O G k t 6 Q V A g s 4 y K c l k V Y H R

x F k B U c a B C d Y Y t H u Y y q V F q i o R o = " > A A A C 7 n i c b Z L P T 9 s w F M f d M F g J v 9 l x 0 h S t Q u J U J X C A G 0 j j w L E w C h V N F T m O A x a 2 Y 9 n O 1 j T y k R v X c Z t 2 5 U / g x J + B x N + w f 2 J u i 1 Q S 9 i R L X 3 0 / 7 + m 9 Z z s W l C j t + y 8 N Z + 7 D / M L H 5 q K 7 t L y y u r a + s X m u s l w i 3 E U Z z W Q v h g p T w n F X E 0 1 x T 0 g M W U z x R X z z b c w v f m C p S M b P d C H w g M E r T l K C o L b W a T i M 1 l t + 2 5 + E 9 1 4 E r 6 J 1 8 P R 4 8 v f 2 y 2 M n 2 m g 8 h 0 m G c o a 5 R h Q q 1 Q 9 8 o Q c l l J o g i o 0 b 5 g o L i G 7 g F e 5 b y S H D a l B O R j X e l n U S L 8 2 k P V x 7 E / d t R Q m Z U g W L b S a D + l r V 2 d j 8 H + v n O t 0 f l I S L X G O O p o 3 S n H o 6 8 8 Z 7 e w m R G G l a W A G R J H Z W D 1 1 D C Z G 2 t + O G H P 9 E G W O Q J 2 U 4 M m U 4 b h C n 5 c i Y K h v O 2 L D O C j 2 D h Y n O 6 l h V 8 P c a H o m 3 b S N R x 6 q C l X G r / H K G L + u D 9 W a s Z + q F I j G l i I 6 M a 3 9 C U H / 3 9 + J 8 p x 3 s t n d O g t b h P p h G E 3 w G X 8 E 2 C M A e O A T H o A O 6 A I E U 3 I F f 4 N 4 R z r 3 z 2 / k z T X U a r z W f Q C W c h 3 / 2 G f u L < / l a t e x i t > q(z|x)

x < l a t e x i t s h a 1 _ b a s e 6 4 = " d 7 

x F k B U c a B C d Y Y t H u Y y q V F q i o R o = " > A A A C 7 n i c b Z L P T 9 s w F M f d M F g J v 9 l x 0 h S t Q u J U J X C A G 0 j j w L E w C h V N F T m O A x a 2 Y 9 n O 1 j T y k R v X c Z t 2 5 U / g x J + B x N + w f 2 J u i 1 Q S 9 i R L X 3 0 / 7 + m 9 Z z s W l C j t + y 8 N Z + 7 D / M L H 5 q K 7 t L y y u r a + s X m u s l w i 3 E U Z z W Q v h g p T w n F X E 0 1 x T 0 g M W U z x R X z z b c w v f m C p S M b P d C H w g M E r T l K C o L b W a T i M 1 l t + 2 5 + E 9 1 4 E r 6 J 1 8 P R 4 8 v f 2 y 2 M n 2 m g 8 h 0 m G c o a 5 R h Q q 1 Q 9 8 o Q c l l J o g i o 0 b 5 g o L i G 7 g F e 5 b y S H D a l B O R j X e l n U S L 8 2 k P V x 7 E / d t R Q m Z U g W L b S a D + l r V 2 d j 8 H + v n O t 0 f l I S L X G O O p o 3 S n H o 6 8 8 Z 7 e w m R G G l a W A G R J H Z W D 1 1 D C Z G 2 t + O G H P 9 E G W O Q J 2 U 4 M m U 4 b h C n 5 c i Y K h v O 2 L D O C j 2 D h Y n O 6 l h V 8 P c a H o m 3 b S N R x 6 q C l X G r / H K G L + u D 9 W a s Z + q F I j G l i I 6 M a 3 9 C U H / 3 9 + J 8 p x 3 s t n d O g t b h P p h G E 3 w G X 8 E 2 C M A e O A T H o A O 6 A I E U 3 I F f 4 N 4 R z r 3 z 2 / k z T X U a r z W f Q C W c h 3 / 2 G f u L < / l

t J Y k G J 0 r 7 / 0 n C + T U x O f Z + e c X / M z s 0 v L C 7 9 P F V Z L h F u o 4 x m s h N D h S n h u K 2 J p r g j J I Y s p v g s v t o b 8 L M / W C q S 8 R N d C N x j 8 I K T l C C o r f U 7 7 E e L T b / l D 8 P 7 L I J 3 0 d x 5 e j x 6 v V 1 9 P I y W G s 9 h k q G c Y a 4 R h U p 1 A 1 / o X g m l J o h i 4 4 a 5 w g K i K 3 i B u 1 Z y y L D q l c N R j f f L O o m X Z t I e r r 2 h + 7 G i h E y p g s U 2 k 0 F 9 q e p s Y H 7 F u r l O t 3 s l 4 S L X m K N R o z S n n s 6 8 w d 5 e Q i R G m h Z W Q C S J n d V D l 1 B C p O 3 t u C H H 1 y h j D P K k D P u m D A c N 4 r T s G 1 N l N 2 N 2 U 2 e F H s P C R C d 1 r C r 4 u I b 7 4 m P b S N S x q m B l 3 C o / H + P z + m C d M e u Y e q F I T C m i f e P a n x D U 3 / 2 z O N 1 o B Z u t j a O g u b s N R j E N V s A a W A c B 2 A K 7 4 A A c g j Z A I A V 3 4 B 4 8 O M J 5 c P 4 6 / 0 a p T u O 9 Z h l U w v n / B v s r + 4 0 = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " d 7 x F k B U c a B C d Y Y t H u Y y q V F q i o R o = " > A A A C 7 n i c b Z L P T 9 s w F M f d M F g J v 9 l x 0 h S t Q u J U J X C A G 0 j j w L E w C h V N F T m O A x a 2 Y 9 n O 1 j T y k R v X c Z t 2 5 U / g x J + B x N + w f 2 J u i 1 Q S 9 i R L X 3 0 / 7 + m 9 Z z s W l C j t + y 8 N Z + 7 D / M L H 5 q K 7 t L y y u r a + s X m u s l w i 3 E U Z z W Q v h g p T w n F X E 0 1 x T 0 g M W U z x R X z z b c w v f m C p S M b P d C H w g M E r T l K C o L b W a T i M 1 l t + 2 5 + E 9 1 4 E r 6 J 1 8 P R 4 8 v f 2 y 2 M n 2 m g 8 h 0 m G c o a 5 R h Q q 1 Q 9 8 o Q c l l J o g i o 0 b 5 g o L i G 7 g F e 5 b y S H D a l B O R j X e l n U S L 8 2 k P V x 7 E / d t R Q m Z U g W L b S a D + l r V 2 d j 8 H + v n O t 0 f l I S L X G O O p o 3 S n H o 6 8 8 Z 7 e w m R G G l a W A G R J H Z W D 1 1 D C Z G 2 t + O G H P 9 E G W O Q J 2 U 4 M m U 4 b h C n 5 c i Y K h v O 2 L D O C j 2 D h Y n O 6 l h V 8 P c a H o m 3 b S N R x 6 q C l X G r / H K G L + u D 9 W a s Z + q F I j G l i I 6 M a 3 9 C U H / 3 9 + J 8 p x 3 s t n d O g t b h P p h G E 3 w G X 8 E 2 C M A e O A T H o A O 6 A I E U 3 I F f 4 N 4 R z r 3 z 2 / k z T X U a r z W f Q C W c h 3 / 2 G f u L < / l a t e x i t > y T < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 d a Y p T y C b A H O 6 Q 2 E z b 7 t X m V H 6 4 M = " > A A A D Z H i c j Z L P a 9 s w F M f V e D 8 6 r 9 v a l Z 0 G x S w M d g p 2 e 1 h v K 2 y H w H p o a d O G x i H I s p y K S r K Q n t s 4 x s c d d 9 3 + l Z 7 2 f + w f 2 G X / x O S k L I 2 d w R 4 Y v r z P 5 1 n P R p H i z I D v / 1 x r O Q 8 e P n q 8 / s R 9 u v H s + Y v N r Z d n J s 0 0 o T 2 S 8 l T 3 I 2 w o Z 5 L 2 g A G n f a U p F h G n 5 9 H V x 4 q f X 1 N t W C p P I V d 0 K P B Y s o Q R D L Z 1 E u Y w 2 m z 7 H X 9 W X j M E d 6 H 9 4 c f t 8 e 8 v O 7 d H o 6 3 W 5 z B O S S a o B M K x M Y P A V z A s s A Z G O C 3 d M D N U Y X K F x 3 R g o 8 S C m m E x 2 7 X 0 3 t p O 7 C W p t o 8 E b 9 a 9 P 1 F g Y U w u I m s K D J e m z q r m K j b I I N k f F k y q D K g k 8 4 O S j H u Q e t W H e z H T l A D P b c B E M 7 u r R y 6 x x g T s 7 3 F D S W 9 I K g S W c R F O y y K s D o i S Y l q W y 2 y y Y J M 6 y 2 E B 8 3 I U A p 2 A F s V p w z M r v Z O 6 N 1 X 3 F / n r H T a 8 w / / 0 z E q v 2 / C 6 / / T c Z f N i I V 7 U 3 9 J f s H 4 1 u Q R V X B Z q 9 K l 0 7 f U L 6 p e t G c 5 2 O 8 F e Z / c 4 a B / s o 3 m t o 9 f o D X q H A v Q e H a A u O k I 9 R N A Y f U X f 0 P f W L 2 f D 2 X Z e z d X W 2 t 3 M N l o q Z + c P X x 8 q d w = = < / l a t e x i t > y S

< l a t e x i t s h a 1 _ b a s e 6 4 = " w Z q e 2 E I n g q 9 x M t P t d P C q r / e t E O s match the form of the true posterior, we inject information about the text and the speaker into the network that predicts the parameters of the variational posterior. Speaker information is represented as learned speaker-wise embedding vectors, while the text information is summarized into a vector by passing the output of the Tacotron text encoder through a unidirectional RNN as done by Stanton et al. (2018) . Appendix A.1 gives additional details.

= " > A A A D Z H i c j Z L P a 9 s w F M f V e D 8 6 r 9 v a l Z 0 G x S w M d g p 2 e 1 h v K 2 y H w H p o a d O G x i H I s p y K S r K Q n t s 4 x s c d d 9 3 + l Z 7 2 f + w f 2 G X / x O S k L I 2 d w R 4 Y v r z P 5 1 n P R p H i z I D v / 1 x r O Q 8 e P n q 8 / s R 9 u v H s + Y v N r Z d n J s 0 0 o T 2 S 8 l T 3 I 2 w o Z 5 L 2 g A G n f a U p F h G n 5 9 H V x 4 q f X 1 N t W C p P I V d 0 K P B Y s o Q R D L Z 1 E u Z m t N n 2 O / 6 s v G Y I 7 k L 7 w 4 / b 4 9 9 f d m 6 P R l u t z 2 G c k k x Q C Y R j Y w a B r 2 B Y Y A 2 M c F q 6 Y W a o w u Q K j + n A R o k F N c N i t m v p v b W d 2 E t S b R 8 J 3 q x 7 f 6 L A w p h c R N Y U G C 5 N n V X N V W y Q Q b I / L J h U G V B J 5 g c l G f c g 9 a o P 9 2 K m K Q G e 2 4 C J Z n Z X j 1 x i j Q n Y 3 + O G k t 6 Q V A g s 4 y K c l k V Y H R A l x b Q s l 9 l k w S Z 1 l s M C 5 u U o B D o B L Y r T h m d W e i d 1 b 6 r u L / L X O 2 x 4 h / / p m Z V e t + F 1 / + m 5 y + b F Q r y o v 6 W / Y P 1 q c g m q u C z U 6 F P p 2 u s X 1 C 9 b M 5 z t d o K 9 z u 5 x 0 D 7 Y R / N a R 6 / R G / Q O B e g 9 O k B d d I R 6 i K A x + o q + o e + t X 8 6 G s + 2 8 m q u t t b u Z b b R U z s 4 f X C E q d g = = < / l a t e x i t > p(x|z, y T , y S )p(z) < l a t e x i t s h a 1 _ b a s e 6 4 = " X f g w J 7 / b T S Q K S r s u h n l r D x V 8 A V o = " > A A A D 3 X i c j Z L N b 9 M w G M a d B s Y I H + v g y C W i q r R K V Z V u S H C c g E M l d h h i 3 a o 1 V e Q 4 z m Y t d r z Y g X w s R 2 6 I K 3 8 b N / 4 U n D b Q J i k S r x T l y f v 8 H t t y X p c H R E j L + q l 1 9 H v 3 d x 7 s P j Q e P X 7 y d K + 7 / + x c h H G E 8 B S F Q R j N X C h w Q B i e S i I D P O M R h t Q N 8 I V 7 8 6 7 0 L z 7 j S J C Q n c m U 4 w W F V 4 z 4 B E G p W k 4 3 6 / M D O 7 m z U z m 0 U z F Q H x m / s 7 O V E g O j f 1 u + V Y c P b l d e M v w D G 1 U 2 G 2 7 G q 0 w N 3 M o 5 3 Z 4 1 s p Z l t s W 4 E j 1 Q 1 a m z 3 / l g e y G K K W Y S B V C I + d j i c p H D S B I U 4 M K w Y 4 E 5 R D f w C s + V Z J B i s c i X l 1 S Y f d X x T D + M 1 M O k u e x u J n J I h U i p q 0 g K 5 b V o e m V z m z e P p f 9 m k R P G Y 4 k Z W m 3 k x 4 E p Q 7 O 8 c d M j E U Y y S J W A K C L q r C a 6 h h F E U v 0 X w 2 b 4 C w o p h c z L 7 a z I 7 X I D 1 8 + z o q h 7 y d p L m l 4 q 1 2 Z a O L b E i Y x o f t b i x F b u U 5 P L + O Z B / n I n L e 7 k P z m x l Z u 0 u M k / O a N O X q 7 B y + Y q s 7 U 3 K 5 M 1 k 3 t F z p 3 3 h a H G b 9 w c t r Y 4 P x y N j 0 a H H 1 / 1 j t 9 W g 7 g L X o C X 4 A C M w W t w D C b g F E w B A r + 0 H W 1 P 6 + q O / l X / p n 9 f o R 2 t y j w H t d J / / A Z p 9 0 / X < / l a t e x i t > q(z|x, y T , y S ) < l a t e x i t s h a 1 _ b a s e 6 4 = " 8 T q M h x 4 f j M 8 o K x G 2 Y j 1 9 c A n X x C 8 = " > A A A D 7 3 i c j Z L d b t M w G I a d h p 8 R / j o 4 Q e I k r K q 0 S l W V D C Q 4 n I C D S u x g i H W r 1 l S R 4 7 h b t N h x b Q f y s 9 w B 5 5 w h T r k k b o M r w G k D b Z M i 7 Z O i v P n e 5 7 W t + P N Y G A h p W b + 0 l n 7 r 9 p 2 7 O / e M + w 8 e P n r c 3 n 1 y K q K Y I z x C U R j x s Q c F D g O K R z K Q I R 4 z j i H x Q n z m X b 0 r / b P P m I s g o i c y Z X h K 4 A U N Z g G C U r X c 9 t c u 2 3 e S a y e V f S c V P f W R s W s n W y r R M 7 r z 8 q 0 6 r D d f e k n / L 2 x U 2 a y / H q 8 y N w H r n N v u W A N r U W Z T 2 J X o g K q O 3 d 3 W B 8 e P U E w w l S i E Q k x s i 8 l p D r k M U I g L w 4 k F Z h B d w Q s 8 U Z J C g s U 0 X / y 2 w u y q j m / O I q 4 e K s 1 F d z 2 R Q y J E S j x F E i g v R d 0 r m 9 u 8 S S x n b 6 Z 5 Q F k s M U X L j W Z x a M r I L O / A 9 A O O k Q x T J S D i g T q r i S 4 h h 0 i q m z I c i r + g i B B I / d z J i t w p N / B m e V Y U m 1 6 y 8 p K 6 l 8 q V m R a u I 3 E i O c l P G p z Y y n 2 q c x l b P 8 g / 7 q j B H d 2 Q E 1 u 5 Y Y M b / p c z N s n z F X h e X 2 W 8 8 s Z l c s N k f p E z 9 3 1 h q P G z 6 8 P W F K c H A / v l 4 O D j q 8 7 h 2 2 o Q d 8 B z s A f 2 g Q 1 e g 0 M w B M d g B B D 4 r T 3 T X m h 7 + l z / p n / X f y z R l l Z l n o K N 0 n / + A U Z + V o s = < / l a t e x i t >

For this work, we use a simple diagonal Gaussian for the approximate posterior, q(z|x, y T , y S ) and a standard normal distribution for the prior, p(z). We use these distributions for simplicity and efficiency, but using more powerful distributions such as Gaussian mixtures or normalizing flows (Rezende & Mohamed, 2015) should decrease the aggregate KL, leading to better prior samples.

Because we are learning a conditional generative model, p(x|y T , y S ), we could have used a learned conditional prior, p(z|y T , y S ), in order to improve the quality of the output generated when sampling via the prior. However, in this work we focus on the transfer use case where we infer

z ref ∼ q(z|x ref , y ref T , y ref S

) from a reference utterance and use it to re-synthesize speech using different text or speaker inputs, x ∼ p(x|z ref , y T , y S ). Using a fixed prior allows z to share a high probability region across all text and speakers so that an embedding inferred from one utterance is likely to lead to non-degenerate output when being used with any other text or speaker.

3.2. DECOMPOSING EMBEDDING CAPACITY HIERARCHICALLY:

In inter-text style transfer uses cases, we infer z ref from a reference utterance and then use it to generate a new utterance with the same style but different text. One problem with this approach is that z ref completely specifies all variation that the latent embedding is capable of conveying to the decoder, p(x|z ref , y T , y S ). So, even though there are many possible realizations of an utterance with a given style, this approach can produce only one2 .

To address this issue, we decompose the latents, z, hierarchically (Sønderby et al., 2016) into highlevel latents, z H , and low-level latents, z L , as shown in Figure 3 . This differs from the hierarchical interpretation of the Gaussian mixture prior used by Hsu et al. (2019) in that here the high-level latents are continuous vectors rather than a single categorical variable. Factorizing continuous latents in this way allows us to specify how the joint capacity, I q (X; [Z H , Z L ]), is divided between z H and z L . This approach can also be extended to additional levels of latents, each containing a prescribed proportion of the overall joint capacity.

x < l a t e x i t s h a 1 _ b a s e 6 4 = " d 7

x F k B U c a B C d Y Y t H u Y y q V F q i o R o = " > A A A C 7 n i c b Z L P T 9 s w F M f d M F g J v 9 l x 0 h S t Q u J U J X C A G 0 j j w L E w C h V N F T m O A x a 2 Y 9 n O 1 j T y k R v X c Z t 2 5 U / g x J + B x N + w f 2 J u i 1 Q S 9 i R L X 3 0 / 7 + m 9 Z z s W l C j t + y 8 N Z + 7 D / M L H 5 q K 7 t L y y u r a + s X m u s l w i 3 E U Z z W Q v h g p T w n F X E 0 1 x T 0 g M W U z x R X z z b c w v f m C p S M b P d C H w g M E r T l K C o L b W a T i M 1 l t + 2 5 + E 9 1 4 E r 6 J 1 8 P R 4 8 v f 2 y 2 M n 2 m g 8 h 0 m G c o a 5 R h Q q 1 Q 9 8 o Q c l l J o g i o 0 b 5 g o L i G 7 g F e 5 b y S H D a l B O R j X e l n U S L 8 2 k P V x 7 E / d t R Q m Z U g W L b S a D + l r V 2 d j 8 H + v n O t 0 f l I S L X G O O p o 3 S n H o 6 8 8 Z 7 e w m R G G l a W A G R J H Z W D 1 1 D C Z G 2 t + O G H P 9 E G W O Q J 2 U 4 M m U 4 b h C n 5 c i Y K h v O 2 L D O C j 2 D h Y n O 6 l h V 8 P c a H o m 3 b S N R x 6 q C l X G r / H K G L + u D 9 W a s Z + q F I j G l i I 6 M a 3 9 C U H / 3 9 + J 8 p x 3 s t n d O g t b h P p h G E 3 w G X 8 E 2 C M A e O A T H o A O 6 A I E U 3 I F f 4 N 4 R z r 3 z 2 / k z T X U a r z W f Q C W c h 3 / 2 G f u L < / l a t e x i t > y T < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 d a Y p T y C b A H O 6 Q 2 E z b 7 t X m V H 6 4 M = " > A A A D Z H i c j Z L P a 9 s w F M f V e D 8 6 r 9 v a l Z 0 G x S w M d g p 2 e 1 h v K 2 y H w H p o a d O G x i H I s p y K S r K Q n t s 4 x s c d d 9 3 + l Z 7 2 f + w f 2 G X / x O S k L I 2 d w R 4 Y v r z P 5 1 n P R p H i z I D v / 1 x r O Q 8 e P n q 8 / s R 9 u v H s + Y v N r Z d n J s 0 0 o T 2 S 8 l T 3 I 2 w o Z 5 L 2 g A G n f a U p F h G n 5 9 H V x 4 q f X 1 N t W C p P I V d 0 K P B Y s o Q R D L Z 1 E u Y w 2 m z 7 H X 9 W X j M E d 6 H 9 4 c f t 8 e 8 v O 7 d H o 6 3 W 5 z B O S S a o B M K x M Y P A V z A s s A Z G O C 3 d M D N U Y X K F x 3 R g o 8 S C m m E x 2 7 X 0 3 t p O 7 C W p t o 8 E b 9 a 9 P 1 F g Y U w u I m s K D J e m z q r m K j b I I N k f F k y q D K g k 8 4 O S j H u Q e t W H e z H T l A D P b c B E M 7 u r R y 6 x x g T s 7 3 F D S W 9 I K g S W c R F O y y K s D o i S Y l q W y 2 y y Y J M 6 y 2 E B 8 3 I U A p 2 A F s V p w z M r v Z O 6 N 1 X 3 F / n r H T a 8 w / / 0 z E q v 2 / C 6 / / T c Z f N i I V 7 U 3 9 J f s H 4 1 u Q R V X B Z q 9 K l 0 7 f U L 6 p e t G c 5 2 O 8 F e Z / c 4 a B / s o 3 m t o 9 f o D X q H A v Q e H a A u O k I 9 R N A Y f U X f 0 P f W L 2 f D 2 X Z e z d X W 2 t 3 M N l o q Z + c P X x 8 q d w = = < / l a t e x i t > y S

< l a t e x i t s h a 1 _ b a s e 6 4 = " w Z q e 2 E I n g q 9 x M t P t d P C q r / e t E O s

= " > A A A D Z H i c j Z L P a 9 s w F M f V e D 8 6 r 9 v a l Z 0 G x S w M d g p 2 e 1 h v K 2 y H w H p o a d O G x i H I s p y K S r K Q n t s 4 x s c d d 9 3 + l Z 7 2 f + w f 2 G X / x O S k L I 2 d w R 4 Y v r z P 5 1 n P R p H i z I D v / 1 x r O Q 8 e P n q 8 / s R 9 u v H s + Y v N r Z d n J s 0 0 o T 2 S 8 l T 3 I 2 w o Z 5 L 2 g A G n f a U p F h G n 5 9 H V x 4 q f X 1 N t W C p P I V d 0 K P B Y s o Q R D L Z 1 E u Z m t N n 2 O / 6 s v G Y I 7 k L 7 w 4 / b 4 9 9 f d m 6 P R l u t z 2 G c k k x Q C Y R j Y w a B r 2 B Y Y A 2 M c F q 6 Y W a o w u Q K j + n A R o k F N c N i t m v p v b W d 2 E t S b R 8 J 3 q x 7 f 6 L A w p h c R N Y U G C 5 N n V X N V W y Q Q b I / L J h U G V B J 5 g c l G f c g 9 a o P 9 2 K m K Q G e 2 4 C J Z n Z X j 1 x i j Q n Y 3 + O G k t 6 Q V A g s 4 y K c l k V Y H R A l x b Q s l 9 l k w S Z 1 l s M C 5 u U o B D o B L Y r T h m d W e i d 1 b 6 r u L / L X O 2 x 4 h / / p m Z V e t + F 1 / + m 5 y + b F Q r y o v 6 W / Y P 1 q c g m q u C z U 6 F P p 2 u s X 1 C 9 b M 5 z t d o K 9 z u 5 x 0 D 7 Y R / N a R 6 / R G / Q O B e g 9 O k B d d I R 6 i K A x + o q + o e + t X 8 6 G s + 2 8 m q u t t b u Z b b R U z s 4 f X C E q d g = = < / l a t e x i t > x < l a t e x i t s h a 1 _ b a s e 6 4 = " d 7 x F k B U c a B C d Y Y t H u Y y q V F q i o R o = " > A A A C 7 n i c b Z L P T 9 s w F M f d M F g J v 9 l x 0 h S t Q u J U J X C A G 0 j j w L E w C h V N F T m O A x a 2 Y 9 n O 1 j T y k R v X c Z t 2 5 U / g x J + B x N + w f 2 J u i 1 Q S 9 i R L X 3 0 / 7 + m 9 Z z s W l C j t + y 8 N Z + 7 D / M L H 5 q K 7 t L y y u r a + s X m u s l w i 3 E U Z z W Q v h g p T w n F X E 0 1 x T 0 g M W U z x R X z z b c w v f m C p S M b P d C H w g M E r T l K C o L b W a T i M 1 l t + 2 5 + E 9 1 4 E r 6 J 1 8 P R 4 8 v f 2 y 2 M n 2 m g 8 h 0 m G c o a 5 R h Q q 1 Q 9 8 o Q c l l J o g i o 0 b 5 g o L i G 7 g F e 5 b y S H D a l B O R j X e l n U S L 8 2 k P V x 7 E / d t R Q m Z U g W L b S a D + l r V 2 d j 8 H + v n O t 0 f l I S L X G O O p o 3 S n H o 6 8 8 Z 7 e w m R G G l a W A G R J H Z W D 1 1 D C Z G 2 t + O G H P 9 E G W O Q J 2 U 4 M m U 4 b h C n 5 c i Y K h v O 2 L D O C j 2 D h Y n O 6 l h V 8 P c a H o m 3 b S N R x 6 q C l X G r / H K G L + u D 9 W a s Z + q F I j G l i I 6 M a 3 9 C U H / 3 9 + J 8 p x 3 s t n d O g t b h P p h G E 3 w G X 8 E 2 C M A e O A T H o A O 6 A I E U 3 I F f 4 N 4 R z r 3 z 2 / k z T X U a r z W f Q C W c h 3 / 2 G f u L < / l a t e x i t > y T < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 d a Y p T y C b A H O 6 Q 2 E z b 7 t X m V H 6 4 M = " > A A A D Z H i c j Z L P a 9 s w F M f V e D 8 6 r 9 v a l Z 0 G x S w M d g p 2 e 1 h v K 2 y H w H p o a d O G x i H I s p y K S r K Q n t s 4 x s c d d 9 3 + l Z 7 2 f + w f 2 G X / x O S k L I 2 d w R 4 Y v r z P 5 1 n P R p H i z I D v / 1 x r O Q 8 e P n q 8 / s R 9 u v H s + Y v N r Z d n J s 0 0 o T 2 S 8 l T 3 I 2 w o Z 5 L 2 g A G n f a U p F h G n 5 9 H V x 4 q f X 1 N t W C p P I V d 0 K P B Y s o Q R D L Z 1 E u Y w 2 m z 7 H X 9 W X j M E d 6 H 9 4 c f t 8 e 8 v O 7 d H o 6 3 W 5 z B O S S a o B M K x M Y P A V z A s s A Z G O C 3 d M D N U Y X K F x 3 R g o 8 S C m m E x 2 7 X 0 3 t p O 7 C W p t o 8 E b 9 a 9 P 1 F g Y U w u I m s K D J e m z q r m K j b I I N k f F k y q D K g k 8 4 O S j H u Q e t W H e z H T l A D P b c B E M 7 u r R y 6 x x g T s 7 3 F D S W 9 I K g S W c R F O y y K s D o i S Y l q W y 2 y y Y J M 6 y 2 E B 8 3 I U A p 2 A F s V p w z M r v Z O 6 N 1 X 3 F / n r H T a 8 w / / 0 z E q v 2 / C 6 / / T c Z f N i I V 7 U 3 9 J f s H 4 1 u Q R V X B Z q 9 K l 0 7 f U L 6 p e t G c 5 2 O 8 F e Z / c 4 a B / s o 3 m t o 9 f o D X q H A v Q e H a A u O k I 9 R N A Y f U X f 0 P f W L 2 f D 2 X Z e z d X W 2 t 3 M N l o q Z + c P X x 8 q d w = = < / l a t e x i t > y S

< l a t e x i t s h a 1 _ b a s e 6 4 = " w Z q e 2 E I n g q 9 x M t P t d P C q r / e t E O s As shown in eq. ( 8), the KL term, R AVG , is an upper bound on I q (X; Z). We can also derive similar bounds for I q (X; Z H ) and I q (X; Z L ). Derivations of these bounds are provided in Appendix C.2.

= " > A A A D Z H i c j Z L P a 9 s w F M f V e D 8 6 r 9 v a l Z 0 G x S w M d g p 2 e 1 h v K 2 y H w H p o a d O G x i H I s p y K S r K Q n t s 4 x s c d d 9 3 + l Z 7 2 f + w f 2 G X / x O S k L I 2 d w R 4 Y v r z P 5 1 n P R p H i z I D v / 1 x r O Q 8 e P n q 8 / s R 9 u v H s + Y v N r Z d n J s 0 0 o T 2 S 8 l T 3 I 2 w o Z 5 L 2 g A G n f a U p F h G n 5 9 H V x 4 q f X 1 N t W C p P I V d 0 K P B Y s o Q R D L Z 1 E u Z m t N n 2 O / 6 s v G Y I 7 k L 7 w 4 / b 4 9 9 f d m 6 P R l u t z 2 G c k k x Q C Y R j Y w a B r 2 B Y Y A 2 M c F q 6 Y W a o w u Q K j + n A R o k F N c N i t m v p v b W d 2 E t S b R 8 J 3 q x 7 f 6 L A w p h c R N Y U G C 5 N n V X N V W y Q Q b I / L J h U G V B J 5 g c l G f c g 9 a o P 9 2 K m K Q G e 2 4 C J Z n Z X j 1 x i j Q n Y 3 + O G k t 6 Q V A g s 4 y K c l k V Y H R A l x b Q s l 9 l k w S Z 1 l s M C 5 u U o B D o B L Y r T h m d W e i d 1 b 6 r u L / L X O 2 x 4 h / / p m Z V e t + F 1 / + m 5 y + b F Q r y o v 6 W / Y P 1 q c g m q u C z U 6 F P p 2 u s X 1 C 9 b M 5 z t d o K 9 z u 5 x 0 D 7 Y R / N a R 6 / R G / Q O B e g 9 O k B d d I R 6 i K A x + o q + o e + t X 8 6 G s + 2 8 m q u t t b u Z b b R U z s 4 f X C E q d g = = < / l a t e x i t > z H < l a t e x i t s h a 1 _ b a s e 6 4 = " s 2 c 3 R I C O n g 9 4 3 p G x T i p V e G A t / 0 Q = " > A A A D A X i c b Z L N a h s x E M f l T T 9 S 9 y t p j 4 G y 1 B R 6 M r v p o b k 1 0 B 5 y 6 C E h c W L i N Y t W q 3 V E J K 2 Q Z l v v i j 2 F H n N N n 6 G 3 k l O g 0 P f o G x T y E p H t g O N 1 B g R / 5 v c f z U h M o j g z E A T / W t 7 K g 4 e P H q 8 + a T 9 9 9 v z F y 7 X 1 V 4 c m L z S h P Z L z X P c T b C h n k v a A A a d 9 p S k W C a d H y e n n C T / 6 R r V h u T y A U t G h w C P J M k Y w u N R + V J l 4 r R N 0 g 2 n 4 y y K 8 F Z 1 P f 6 / 2 r n + 8 u d q N 1 1 v / o z Q n h a A S C M f G D M J A w d B i D Y x w W r e j w l C F y S k e 0 Y G T E g t q h n Y 6 a + 2 / c 5 n U z 3 L t j g R / m r 1 b Y b E w p h S J c w o M J 6 b J J s n 7 2 K C A b G t o m V Q F U E l m j b K C + 5 D 7 k 4 f 7 K d O U A C + d w E Q z N 6 t P T r D G B N z 3 t C N J v 5 N c C C x T G 1 W 1 j S Y N k s x W d b 3 I x n M 2 b r I S 5 r C s 4 4 M m N g t 4 v 4 E r d b d t H A E d g x b 2 a 7 N L Z e 7 1 7 T j f o v F 4 7 j t u X t K f s / 5 S o U p r q + I v d d v t R t j c h G V x u N k N P 3 Q 3 9 8 L O 9 h a a x S r a Q G / R e x S i j 2 g b 7 a B d 1 E M E j d A 5 u k A / v T P v l / f b u 5 x Z v d Z t z W u 0 E N 6 f G z 7 a B C Y = < / l a t e x i t > z H < l a t e x i t s h a 1 _ b a s e 6 4 = " s 2 c 3 R I C O n g 9 4 3 p G x T i p V e G A t / 0 Q = " > A A A D A X i c b Z L N a h s x E M f l T T 9 S 9 y t p j 4 G y 1 B R 6 M r v p o b k 1 0 B 5 y 6 C E h c W L i N Y t W q 3 V E J K 2 Q Z l v v i j 2 F H n N N n 6 G 3 k l O g 0 P f o G x T y E p H t g O N 1 B g R / 5 v c f z U h M o j g z E A T / W t 7 K g 4 e P H q 8 + a T 9 9 9 v z F y 7 X 1 V 4 c m L z S h P Z L z X P c T b C h n k v a A A a d 9 p S k W C a d H y e n n C T / 6 R r V h u T y A U t G h w C P J M k Y w u N R + V J l 4 r R N 0 g 2 n 4 y y K 8 F Z 1 P f 6 / 2 r n + 8 u d q N 1 1 v / o z Q n h a A S C M f G D M J A w d B i D Y x w W r e j w l C F y S k e 0 Y G T E g t q h n Y 6 a + 2 / c 5 n U z 3 L t j g R / m r 1 b Y b E w p h S J c w o M J 6 b J J s n 7 2 K C A b G t o m V Q F U E l m j b K C + 5 D 7 k 4 f 7 K d O U A C + d w E Q z N 6 t P T r D G B N z 3 t C N J v 5 N c C C x T G 1 W 1 j S Y N k s x W d b 3 I x n M 2 b r I S 5 r C s 4 4 M m N g t 4 v 4 E r d b d t H A E d g x b 2 a 7 N L Z e 7 1 7 T j f o v F 4 7 j t u X t K f s / 5 S o U p r q + I v d d v t R t j c h G V x u N k N P 3 Q 3 9 8 L O 9 h a a x S r a Q G / R e x S i j 2 g b 7 a B d 1 E M E j d A 5 u k A / v T P v l / f b u 5 x Z v d Z t z W u 0 E N 6 f G z 7 a B C Y = < / l a t e x i t > z L < l a t e x i t s h a 1 _ b a s e 6 4 = " o w / U o g L F W k Z Q X C p W A Q A E S x 0 q A 7 Q = " > A A A D A X i c b Z L N a h s x E M f l T T 9 S 9 y t p j 4 G y 1 B R 6 M r v p o b k 1 0 B 5 y 6 C E h c W L i N Y t W q 3 V E J K 2 Q Z l v v i j 2 F H n N N n 6 G 3 k l O g 0 P f o G x T y E p H t g O N 1 B g R / 5 v c f z U h M o j g z E A T / W t 7 K g 4 e P H q 8 + a T 9 9 9 v z F y 7 X 1 V 4 c m L z S h P Z L z X P c T b C h n k v a A A a d 9 p S k W C a d H y e n n C T / 6 R r V h u T y A U t G h w C P J M k Y w u N R + V K l 4 r R N 0 g 2 n 4 y y K 8 F Z 1 P f 6 / 2 r n + 8 u d q N 1 1 v / o z Q n h a A S C M f G D M J A w d B i D Y x w W r e j w l C F y S k e 0 Y G T E g t q h n Y 6 a + 2 / c 5 n U z 3 L t j g R / m r 1 b Y b E w p h S J c w o M J 6 b J J s n 7 2 K C A b G t o m V Q F U E l m j b K C + 5 D 7 k 4 f 7 K d O U A C + d w E Q z N 6 t P T r D G B N z 3 t C N J v 5 N c C C x T G 1 W 1 j S Y N k s x W d b 3 I x n M 2 b r I S 5 r C s 4 4 M m N g t 4 v 4 E r d b d t H A E d g x b 2 a 7 N L Z e 7 1 7 T j f o v F 4 7 j t u X t K f s / 5 S o U p r q + I v d d v t R t j c h G V x u N k N P 3 Q 3 9 8 L O 9 h a a x S r a Q G / R e x S i j 2 g b 7 a B d 1 E M E j d A 5 u k A / v T P v l / f b u 5 x Z v d Z t z W u 0 E N 6 f G z c J B C M = < / l a t e x i t > z L < l a t e x i t s h a 1 _ b a s e 6 4 = " o w / U o g L F W k Z Q X C p W A Q A E S x 0 q A 7 Q = " > A A A D A X i c b Z L N a h s x E M f l T T 9 S 9 y t p j 4 G y 1 B R 6 M r v p o b k 1 0 B 5 y 6 C E h c W L i N Y t W q 3 V E J K 2 Q Z l v v i j 2 F H n N N n 6 G 3 k l O g 0 P f o G x T y E p H t g O N 1 B g R / 5 v c f z U h M o j g z E A T / W t 7 K g 4 e P H q 8 + a T 9 9 9 v z F y 7 X 1 V 4 c m L z S h P Z L z X P c T b C h n k v a A A a d 9 p S k W C a d H y e n n C T / 6 R r V h u T y A U t G h w C P J M k Y w u N R + V K l 4 r R N 0 g 2 n 4 y y K 8 F Z 1 P f 6 / 2 r n + 8 u d q N 1 1 v / o z Q n h a A S C M f G D M J A w d B i D Y x w W r e j w l C F y S k e 0 Y G T E g t q h n Y 6 a + 2 / c 5 n U z 3 L t j g R / m r 1 b Y b E w p h S J c w o M J 6 b J J s n 7 2 K C A b G t o m V Q F U E l m j b K C + 5 D 7 k 4 f 7 K d O U A C + d w E Q z N 6 t P T r D G B N z 3 t C N J v 5 N c C C x T G 1 W 1 j S Y N k s x W d b 3 I x n M 2 b r I S 5 r C s 4 4 M m N g t 4 v 4 E r d b d t H A E d g x b 2 a 7 N L Z e 7 1 7 T j f o v F 4 7 j t u X t K f s / 5 S o U p r q + I v d d v t R t j c h G V x u N k N P 3 Q 3 9 8 L O 9 h a a x S r a Q G / R e x S i j 2 g b 7 a B d 1 E M E j d A 5 u k A / v T P v l / f b u 5 x Z v d Z t z W u 0 E N 6 f G z c J B C M = < / l a t e x i t > q(z H |z L )q(z L |x, y T , y S ) < l a t e x i t s h a 1 _ b a s e 6 4 = " S p p 0 + Q r 5 / E d e V U m c 6 4 3 Z 7 a 5 C p 2 Y = " > A A A D n X i c j Z J b b 9 M w G I a 9 h s M I p w 4 u u S C i m r R K U 9 U M J B B X E w e p E h U a Y l 2 r 1 V X k O M 5 m L X Y 8 2 4 E c l v / C r + E W b v k 3 O G 2 h a 1 I k P i n K m / d 9 P t u x P l 9 E V O l + / 9 d W y 7 p x 8 9 b t 7 T v 2 3 X v 3 H z x s 7 z w 6 U X E i M R n h O I r l x E e K R J S T k a Y 6 I h M h C W J + R M b + x d s q H 3 8 h U t G Y H + t M k B l D Z 5 y G F C N t L K / 9 e l f s w f Q K Z n o f Z q p r P n J x B f O F U l 3 7 s n o Z Q 3 Q v F 1 G 6 / 4 f 1 2 p 1 + r z 8 v p y n c p e i A Z R 1 5 O 6 0 P M I h x w g j X O E J K T d 2 + 0 L M C S U 1 x R E o b J o o I h C / Q G Z k a y R E j a l b M f 7 J 0 d o 0 T O G E s z c O 1 M 3 e v d x S I K Z U x 3 5 A M 6 X N V z y p z U z Z N d P h q V l A u E k 0 4 X m w U J p G j Y 6 e 6 M S e g k m A d Z U Y g L K k 5 q 4 P P k U R Y m 3 u 1 I S d f c c w Y 4 k E B 8 7 K A 1 Q Z + W O R l u Z 6 l q y y t Z 5 l e h V n p Q U 1 S L V l x 3 O D U R u 5 z n c v F 9 Y P 8 5 Y Y N b v i f n N r I D R r c 4 J + c v U 6 e r s D T + i q T V T a p O t d C E Z S F 8 N 6 V t h k / t z 5 s T X F y 0 H O f 9 w 4 + v e g c v l k O 4 j Z 4 A p 6 B P e C C l + A Q D M A R G A E M v o H v 4 A f 4 a T 2 1 3 l t D 6 + M C b W 0 t e x 6 D t b L G v w E E A j s o < / l a t e x i t > p(x|y T , y S )p(z L |z H )p(z H ) < l a t e x i t s h a 1 _ b a s e 6 4 = " c z r o M Q 8 z 1 B T R G m k w w Z 8 w p k 3 Q + / 4 = " > A A A D n X i c j Z J b b 9 M w G I a 9 h s M I p w 4 u u S C i m r R K U 9 U M J B B X E w e p E h U a Y l 2 r 1 V X k O M 5 m L X Y 8 2 4 E c l v / C r + E W b v k 3 O G 2 h a 1 I k P i n K m / d 9 P t u x P l 9 E V O l + / 9 d W y 7 p x 8 9 b t 7 T v 2 3 X v 3 H z x s 7 z w 6 U X E i M R n h O I r l x E e K R J S T k a Y 6 I h M h C W J + R M b + x d s q H 3 8 h U t G Y H + t M k B l D Z 5 y G F C N t L K / 9 W u z B 9 A p m e h 9 m q m s + c n E F 8 4 V S X X v 3 s n o b R 3 Q v F 1 m 6 / w f 2 2 p 1 + r z 8 v p y n c p e i A Z R 1 5 O 6 0 P M I h x w g j X O E J K T d 2 + 0 L M C S U 1 x R E o b J o o I h C / Q G Z k a y R E j a l b M f 7 J 0 d o 0 T O G E s z c O 1 M 3 e v d x S I K Z U x 3 5 A M 6 X N V z y p z U z Z N d P h q V l A u E k 0 4 X m w U J p G j Y 6 e 6 M S e g k m A d Z U Y g L K k 5 q 4 P P k U R Y m 3 u 1 I S d f c c w Y 4 k E B 8 7 K A 1 Q Z + W O R l u Z 6 l q y y t Z 5 l e h V n p Q U 1 S L V l x 3 O D U R u 5 z n c v F 9 Y P 8 5 Y Y N b v i f n N r I D R r c 4 J + c v U 6 e r s D T + i q T V T a p O t d C E Z S F 8 N 6 V t h k / t z 5 s T X F y 0 H O f 9 w 4 + v e g c v l k O 4 j Z 4 A p 6 B P e C C l + A Q D M A R G A E M v o H v 4 A f 4 a T 2 1 3 l t D 6 + M C b W 0 t e x 6 D t b L G v w E K W j s o < / l a t e x i t >

I q (X; Z L ) ≤ R AVG = E x∼p D (x) [D KL (q(z H |z L )q(z L |x) p(z L |z H )p(z H ))]

(10)

EQUATION

If we define R AVG L ≡ R AVG -R AVG H , we end up with the following capacity limits for the hierarchical latents:

EQUATION

)

The negative ELBO for this model can be written as:

EQUATION

)

where R H and R L are single data point estimates of R AVG H and R AVG L computed from x. In order to specify how the joint capacity is distributed between the latents, we extend (9) to have two Lagrange multipliers and capacity targets.

min θ max βH,βL≥0 E zL∼q θ (zL|x,yT,yS) [-log p θ (x|z L , y T , y S )] + β H (R H -C H ) + β L (R L -C L ) (14)

C H limits the information capacity of z H , and C L limits how much capacity z L has in excess of z H (i.e., the total capacity of z L is capped at C H + C L ). This allows us to infer

z ref H ∼ q(z H |z L )q(z L |x ref , y ref T , y ref S )

from a reference utterance and use it to sample multiple realizations, x ∼ p(x|z L , y T , y S )p(z L |z ref H ). Intuitively, the higher C H is, the more the output will resemble the reference, and the higher C L is, the more variation we would expect from sample to sample when fixing z ref H and sampling z L from p(z L |z ref H ).

4. EXPERIMENTS 4.1 MODEL ARCHITECTURE AND TRAINING:

Model architecture The baseline model we start with is a Tacotron-based system (Wang et al., 2017) that incorporates modifications from Skerry-Ryan et al. ( 2018), including phoneme inputs instead of characters, GMM attention (Graves, 2013) , and a WaveNet neural vocoder (van den Oord et al., 2016) to convert the output mel spectrograms into audio samples (Shen et al., 2018) . The decoder RNN uses a reduction factor of 2, meaning that it produces two spectrogram frames per timestep. We use the CBHG text encoder from Wang et al. (2018) and the GMMv2b attention mechanism from Battenberg et al. (2019) .

For the heuristic models compared in Section 2.2, we augment the baseline Tacotron with the reference encoders described by Skerry-Ryan et al. ( 2018) and Wang et al. (2018) . For the variational models that we compare in the following experiments, we start with the reference encoder from Skerry-Ryan et al. ( 2018) and replace the tanh bottleneck layer with an MLP that predicts the parameters of the variational posterior. When used, the additional conditional dependencies (text and speaker) are fed into the MLP as well.

Training To train the models, the primary optimizer is run synchronously across 10 GPU workers (2 of them backup workers) for 300,000 training steps with an effective batch size of 256. It uses the Adam algorithm (Kingma & Ba, 2015) with a learning rate that is annealed from 10 -3 to 5 × 10 -5 over 200,000 training steps. The optimizer for β is run asychronously on the 10 workers and uses SGD with momentum 0.9 and a fixed learning rate of 10 -5 . The updates for these two optimizers are interleaved, allowing β to converge to a steady state value that achieves the target value for the KL term, as demonstrated in Figure B .2 in the appendix. Additional architectural and training details are provided in Appendix A.

4.2. EXPERIMENT SETUP:

Datasets For single-speaker models, we use an expressive English language audiobook dataset consisting of 50,086 training utterances (36.5 hours) and 912 test utterances spoken by Catherine Byers, the speaker from the 2013 Blizzard Challenge. Multi-speaker models are trained using highquality English data from 58 voice assistant-like speakers, consisting of 419,966 training utterances (327 hours). We evaluate on a 9-speaker subset of the multi-speaker test data which contains 1808 utterances (comprising US, UK, Australian, and Indian speakers).

Tasks The tasks that we explore include same-text prosody transfer, inter-text style transfer, and inter-speaker prosody transfer. We also evaluate the quality of samples produced when sampling via the prior. For these tasks, we compare performance when using variational models with and without the additional conditional dependencies in the variational posterior at a number of different capacity limits. For models with hierarchical latents, we demonstrate the effect of varying C H and C L for same-text prosody transfer when inferring z H and sampling z L or when inferring z L directly.

Evaluation We use crowd-sourced native speakers to collect two types of subjective evaluations. First, mean opinion score (MOS) rates naturalness on a scale of 1-5, 5 being the best. Second, we use the AXY side-by-side comparison proposed by Skerry-Ryan et al. (2018) to measure subjective similarity to a reference utterance relative to the baseline model on a scale of [-3,3] . For example, a score of 3 would mean that, compared to the baseline model, the model being tested produces samples much more perceptually similar to the ground truth reference. We also use an objective similarity metric that uses dynamic time warping to find the minimum mel cepstral distortion (Kubichek, 1993) between two sequences (MCD-DTW). Lastly, for inter-speaker transfer, we follow Skerry-Ryan et al. ( 2018) and use a simple speaker classifier to measure how well speaker identity is preserved. Additional details on evaluation methodologies are provided in Appendix A.

4.3. RESULTS:

Single speaker For single-speaker models, we compare the performance on same and inter-text transfer and the quality of samples generated via the prior for models with and without text conditioning in the variational posterior (Var+Txt and Var, respectively) at different capacity limits, C. Similarity results for the transfer task are shown on the left side of Figure 4 and demonstrate increasing reference similarity as C is increased, with the exception of the model without text conditioning on the inter-text transfer task. Looking at the MOS naturalness results on the right side of Figure 4 , we see that both inter-text transfer and prior sampling take a serious hit as capacity is increased for the Var model, while the Var+Txt model is able to maintain respectable performance even at very high capacities on all tasks.

Listening to the audio examples, we can hear that the Var model produces degenerate output at high capacities when attempting to transfer the style from a short utterance to a long utterance. This indicates that the decoder probably hasn't seen similar embeddings paired with long utterances during training, which suggests that z is improperly correlated with text length. Similar behavior is also observed when generating prior samples using shorter or longer text. This means that an arbitrary z (sampled from the prior or inferred from a reference) is unlikely to pair well with text of an arbitrary length.

Multi-speaker For multi-speaker models, we compare inter-speaker same-text transfer performance and prior sample quality with and without speaker conditioning in the variational posterior (Var+Txt+Spk and Var+Txt, respectively) at a fixed capacity limit of 150 nats. In Table 1 , we see that both models are able to preserve characteristics of the reference utterance during transfer (AXY Ref. Similarity column), while the Var+Txt+Spk model has an edge in MOS for both inter-speaker transfer and prior samples (almost matching the MOS of the deterministic baseline model even at high embedding capacity).

Similar to the utterance length argument in the single speaker section above, it is likely that adding speaker dependencies to the posterior allows the model to use the entire latent space for each speaker (meaning z is not correlated with speaker identity), thereby forcing the decoder to learn to map all plausible points in the latent space to natural-sounding utterances that preserve the target speaker's pitch range. The speaker classifier results show that the Var+Txt+Spk model preserves target speaker identity about as well as the baseline model and ground truth data (~5% of the time the classifier chooses a speaker other than the target speaker), whereas for the Var+Txt model this happens about 22% of the time. Though 22% seems like a large speaker error rate, it is much lower than the 79% figure presented by Skerry-Ryan et al. (2018) for a heuristic prosody transfer model. This demonstrates that even with a weakly conditioned posterior, the capacity limiting properties of variational models lead to better transfer generality and robustness.

Hierarchical latents To evaluate hierarchical decomposition of capacity in a single speaker setting, we use the MCD-DTW distance to quantify reference similarity and same-reference inter-sample variability. As shown in Table B .1 in the appendix, MCD-DTW strongly (negatively) correlates with subjective similarity.

The left side of Figure 5 shows results for samples generated using high-level latents, z H , inferred from the reference. As C H is increased, we see a strong downward trend in the average distance to ) from the prior to use in the reconstructions. The right side of Figure 5 shows the same metrics but for samples generated using low-level latents, z ref L , inferred from the variational posterior. In this case, we see a slight downward trend in the reference distance as the total capacity limit, C, is increased (the trend is less dramatic because the capacity is already fairly high). We also see significantly lower inter-sample distance because the variation modeled by the latents is completely specified by z L . In this case, we sample multiple z ref L 's from q(z L |x ref , y ref T ) for the same x ref because using the same z L would lead to identical output from the deterministic decoder.

Using Capacitron with hierarchical latents increases the model's versatility for transfer tasks. By inferring just the high-level latents, z H , from a reference, we can sample multiple realizations of an utterance that are similar to the reference, with the level of similarity controlled by C H , and the amount of sample-to-sample variation controlled by C L . The same model can also be used for higher fidelity, lower variability transfer by inferring the low-level latents, z L , from a reference, with the level of similarity controlled by C = C H + C L . As mentioned before, this idea could also be extended to use additional levels of latents, thereby increasing transfer and sampling flexibility. To appreciate the results fully, it is strongly recommended to listen to the audio examples available on the web3 .

C H = 20 C L = 50 C = 70 C H = 20 C L = 100 C = 120 C H = 50 C L = 50 C = 100 C H = 50 C L = 100 C = 150 C H = 100 C L = 50 C = 150 C H = 100 C L = 100 C = 200

5. CONCLUSION:

We have proposed embedding capacity (i.e., representational mutual information) as a useful framework for comparing and configuring latent variable models of speech. Our proposed model, Capacitron, demonstrates that including text and speaker dependencies in the variational posterior allows a single model to be used successfully for a variety of transfer and sampling tasks. Motivated by the multi-faceted variability of natural human speech, we also showed that embedding capacity can be decomposed hierarchically in order to enable the model to control a trade-off between transfer fidelity and sample-to-sample variation.

There are many directions for future work, including adapting the fixed-length variational embeddings to be variable-length and synchronous with either the text or audio, using more powerful distributions like normalizing flows, and replacing the deterministic decoder with a proper likelihood distribution. For transfer and control uses cases, the ability to distribute certain speech characteristics across specific subsets of the hierarchical latents would allow more fine-grained control of different aspects of the output speech. And for purely generative, non-transfer use cases, using more powerful conditional priors could improve sample quality.

A EXPERIMENT DETAILS A.1 ARCHITECTURE DETAILS Baseline Tacotron The baseline Tacotron we start with (which serves as f θ (•) in eq. ( 1)) is similar to the original sequence-to-sequence model described by Wang et al. (2017) but uses some modifications introduced by Skerry- Ryan et al. (2018) . Input to the model consists of sequences of phonemes produced by a text normalization pipeline rather than character inputs. The CBHG text encoder from Wang et al. (2017) is used to convert the input phonemes into a sequence of text embeddings. Before being fed to the CBHG encoder, the phoneme inputs are converted to learned 256-dimensional embeddings and passed through a pre-net composed of two fully connected ReLU layers (with 256 and 128 units, respectively), with dropout of 0.5 applied to the output of each layer. For multi-speaker models, a learned embedding for the target speaker is broadcast-concatenated to the output of the text encoder.

The attention module uses a single LSTM layer with 256 units and zoneout of 0.1 followed by an MLP with 128 tanh hidden units to compute parameters for the monotonic 5-component GMM attention window. Instead of using the exponential function to compute the shift and scale parameters of the GMM components as in (Graves, 2013) , we use the softplus function, which we found leads to faster alignment and more stable optimization.

The autoregressive decoder module consists of 2 LSTM layers each with 256 units, zoneout of 0.1, and residual connections between the layers. The spectrogram output is produced using a linear layer on top of the 2 LSTM layers, and we use a reduction factor of 2, meaning we predict two spectrogram frames for each decoder step. The decoder is fed the last frame of its most recent prediction (or the previous ground truth frame during training) and the current context as computed by the attention module. Before being fed to the decoder, the previous prediction is passed through the same pre-net used before the text encoder above.

Mel spectrograms:

The mel spectrograms the model predicts are computed from 24kHz audio using a frame size of 50ms, a hop size of 12.5ms, an FFT size of 2048, and a Hann window. From the FFT energies, we compute 80 mel bins distributed between 80Hz and 12kHz.

Reference encoder:

The common reference encoder we use to compute reference embeddings starts with the mel spectrogram from the reference and passes it through a stack of 6 convolutional layers, each using ReLU non-linearities, 3x3 filters, 2x2 stride, and batch normalization. The 6 layers have 32, 32, 64, 64, 128, and 128 filters, respectively . The output of this convolution stack is fed into a unidirectional LSTM with 128 units, and the final output of the LSTM serves as the output of our basic reference encoder.

To replicate the prosody transfer model from Skerry-Ryan et al. (2018) , we pass the reference encoder output through an additional tanh bottleneck layer to compute the embedding. For the Style Tokens model in Wang et al. (2018) , we pass the output through the Style Tokens bottleneck described in the paper. For the approximate posterior in our variational models, we pass the output of the reference encoder (and potentially vectors describing the text and/or speaker) through an MLP with 128 tanh hidden units to produce the parameters of the diagonal Gaussian posterior which we sample from to produce a reference embedding. For all models with reference encoders, the resulting reference embedding is broadcast-concatenated to the output of the text encoder.

Conditional inputs When providing information about the text to the variational posterior, we pass the sequence of text embeddings produced by the text encoder to a unidirectional RNN with 128 units and use its final output as a fixed-length text summary that is passed to the posterior MLP. Speaker information is passed to the posterior MLP via a learned speaker embedding.

A.2 TRAINING DETAILS:

For the optimization problems shown in eqs. ( 9) and ( 14), we use two separate optimizers. The first minimizes the objective with respect to the model parameters using the SyncReplicasOptimizer 4 from Tensorflow with 10 workers (2 of them backup workers) and an effective batch size of 256. We also use gradient clipping with a threshold of 5. This optimizer uses the Adam algorithm (Kingma & Ba, 2015) with β 1 = 0.9, β 2 = 0.999, = 10 -8 , and a learning rate that is set to 10 -3 , 5 × 10 -4 , 3 × 10 -4 , 10 -4 , and 5 × 10 -5 at 50k, 100k, 150k, and 200k steps, respectively. Training is run for 300k steps total.

The optimizer that maximizes the objective with respect to the Lagrange multiplier is run asynchronously across the 10 workers (meaning each worker computes an independent update using its 32-example sub-batch) and uses SGD with a momentum of 0.9 and a learning rate of 10 -5 . The Lagrange multiplier is computed by passing an unconstrained parameter through the softplus function in order to enforce non-negativity. The initial value of the parameter is chosen such that the Lagrange multiplier equals 1 at the start of training. MCD-DTW We evaluate the models with hierarchical latents using the MCD-DTW distance to quantify reference similarity and the amount of inter-sample variation. To compute mel cepstral distortion (MCD) (Kubichek, 1993) , we use the same mel spectrogram parameters described in A.1 and take the DCT to compute the first 13 MFCCs (not including the 0th coefficient). The MCD between two frames is the Euclidean distance between their MFCC vectors. Then we use the dynamic time warping (DTW) algorithm (Müller, 2007) (with a warp penalty of 1.0) to find an alignment between two spectrograms that produces the minimum MCD cost (including the total warp penalty). We report the average per-frame MCD-DTW.

To evaluate reference similarity, we simply compute the MCD-DTW between the synthesized audio and the reference audio (a lower MCD-DTW indicates higher similarity). The strong (negative) correlation between MCD-DTW and subjective similarity is demonstrated in Table B .1. To quantify inter-sample variation, we compute 5 output samples using the same reference and compute the average MCD-DTW between the first sample and each subsequent sample. A human rater is presented with three stimuli: a reference speech sample (A), and two competing samples (X and Y) to evaluate. The rater is asked to rate whether the prosody of X or Y is closer to that of the reference on a 7-point scale. The scale ranges from "X is much closer" to "Both are about the same distance" to "Y is much closer", and can naturally be mapped on the integers from -3 to 3. Prior to collecting any ratings, we provide the raters with 4 examples of prosodic attributes to evaluate (intonation, stress, speaking rate, and pauses), and explicitly instruct the raters to ignore audio quality or pronunciation differences. For each triplet (A, X, Y) evaluated, we collect 1 rating, and no rater is used for more than 6 items in a single evaluation. To analyze the data from these subjective tests, we average the scores and compute 95% confidence intervals. A human rater is presented with a single speech sample and is asked to rate perceived naturalness on a scale of 1-5, where 1 is "Bad" and 5 is "Excellent". For each sample, we collect 1 rating, and no rater is used for more than 6 items in a single evaluation. To analyze the data from these subjective tests, we average the scores and compute 95% confidence intervals. Natural human speech is typically rated around 4.5. In these plots we can see how R AVG varies with embedding dimensionality for constant KL weight, β, while using the KL limit, C, from the optimization problem in eq. 9 achieves constant R AVG .

B ADDITIONAL RESULTS:



Rate-distortion plots In:

Table B .3: Transfer using hierarchical latents. "Ref." is the the average MCD-DTW distance from the reference, and "X-samp." is the average inter-sample MCD-DTW. These are the numbers used in the plots in Figure 5 . R ≡ q(z|x) log q(z|x) p(z) dz (KL term) (15) R AVG ≡ p D (x)q(z|x) log q(z|x) p(z) dxdz (Average KL term) (16) I q (X; Z) ≡ p D (x)q(z|x) log q(z|x) q(z) dxdz (Representational mutual information) ( 17) q(z) ≡ p D (x)q(z|x)dx (Aggregated posterior) (18) KL non-negativity: q(x) log q(x) p(x) dx ≥ 0 (19)

=⇒ q(x) log q(x) ≥ q(x) log p(x)dx (20)

Mutual information is upper bounded by the average KL (Alemi et al., 2018) :

I q (X; Z) ≡ p D (x)q(z|x) log q(z|x) q(z) dxdz

= p D (x)q(z|x) log q(z|x)dxdzp D (x)q(z|x) log q(z)dxdz

= p D (x)q(z|x) log q(z|x)dxdzq(z) log q(z)dz (23) ≤ p D (x)q(z|x) log q(z|x)dxdzq(z) log p(z)dz (24) = p D (x)q(z|x) log q(z|x)dxdzp D (x)q(z|x) log p(z)dxdz (25) = p D (x)q(z|x) log q(z|x) p(z) dxdz (26)

EQUATION

)

where the inequality in (24) follows from (20).

The difference between the average KL and the mutual information is the aggregate KL:

R AVG -I q (X; Z) = p D (x)q(z|x) log q(z) p(z) dxdz (29)

= q(z) log q(z) p(z) dz 

q(z|x) = q(z H , z L |x) = q(z L |x)q(z H |z L )

Under review as a conference paper at ICLR 2020 Again, using the non-negativity of the KL, we can bound I q (Z H ; Z L ): 

I q (Z H ; Z L ) = q(z H |z L )q(z L ) log q(z H |z L ) q(z

EQUATION

)

where ( 50) can be demonstrated by applying the data processing inequality to a reversed version of the Markov chain, X → Z L → Z H Define R L :

R L ≡ R -R H (52) = q(z L |x)q(z H |z L ) log q(z L |x) p(z L |z H ) dz H dz L (53)

Giving us the following bounds on I q (X; Z L ) and I q (X; Z H ):

=⇒ I q (X; Z H ) ≤ R AVG H , I q (X; Z L ) ≤ R AVG H + R AVG L (54)

:

https://variational-embedding-capacity.github.io/demos/

If the decoder were truly stochastic (not greedy), we could actually sample multiple realizations given the same z ref , but, at high embedding capacities the variations would likely be very similar perceptually.

https://variational-embedding-capacity.github.io/demos/

https://www.tensorflow.org/api_docs/python/tf/train/ SyncReplicasOptimizer

Figures and Tables:

Figure 1: Reconstruction loss vs. embedding dimensionality for a variety of heuristic and variational models. For the variational model (Var.), we vary the capacity limit, C. Notice how the reconstruction loss flattens out at lower values for higher values of C. The heuristic models are denoted by PT and GST for Prosody Transfer and Global Style Tokens. Figure B.1 in the appendix shows how the KL term changes when varying C as well as the KL weight, β.

A l x b Q s l 9 l k w S Z 1 l s M C 5 u U o B D o B L Y r T h m d W e i d 1 b 6 r u L / L X O 2 x 4 h / / p m Z V e t + F 1 / + m 5 y + bF Q r y o v 6 W / Y P 1 q c g m q u C z U 6 F P p 2 u s X 1 C 9 b M 5 z t d o K 9 z u 5 x 0 D 7 Y R / N a R 6 / R G / Q O B e g 9 O k B d d I R 6 i K A x + o q + o e + t X 8 6 G s + 2 8 m q u t t b u Z b b R U z s 4 f X C E q d g = = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " a x f Y 0 i W U l s X U b q Z O B D U B x B + B + u U = " > A A A C 7 n i c b Z L P S + Q w F M c z X X W 1 6 8 / 1 K E h x W P A 0 t H r Q m 4 I e P O r q 6 O B 0 K G m a a j B J Q 5 K u d k q O 3r z q b d n r / g m e / D M E / w b / C T M z w t j q g 8 C X 7 + c 9 3 n t J Y k G J 0 r 7 / 0 n C + T U x O f Z + e c X / M z s 0 v L C 7 9 P F V Z L h F u o 4 x m s h N D h S n h u K 2 J p r g j J I Y s p v g s v t o b 8 L M / W C q S 8 R N d C N x j 8 I K T l C C o r f U 7 7 E e L T b / l D 8 P 7 L I J 3 0 d x 5 e j x 6 v V 1 9 P I y W G s 9 h k q G c Y a 4 R h U p 1 A 1 / o X g m l J o h i 4 4 a 5 w g K i K 3 i B u 1 Z y y L D q l c N R j f f L O o m X Z t I e r r 2 h + 7 G i h E y p g s U 2 k 0 F 9 q e p s Y H 7 F u r l O t 3 s l 4 S L X m K N R o z S n n s 6 8 w d 5 e Q i R G m h Z W Q C S J n d V D l 1 B C p O 3 t u C H H 1 y h j D P K k D P u m D A c N 4 r T s G 1 N l N 2 N 2 U 2 e F H s P C R C d 1 r C r 4 u I b 7 4 m P b S N S x q m B l 3 C o / H + P z + m C d M e u Y e q F I T C m i f e P a n x D U 3 / 2 z O N 1 o B Z u t j a O g u b s N R j E N V s A a W A c B 2 A K 7 4 A A c g j Z A I A V 3 4 B 4 8 O M J 5 c P 4 6 / 0 a p T u O 9 Z h l U w v n / B v s r + 4 0 = < / l a t e x i t >

a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " a x f Y 0 i W U l s X U b q Z O B D U B x B + B + u U = " > A A A C 7 n i c b Z L P S + Q w F M c z X X W 1 6 8 / 1 K E h x W P A 0 t H r Q m 4 I e P O r q 6 O B 0 K G m a a j B J Q 5 K u d k q O 3 r z q b d n r / g m e / D M E / w b / C T M z w t j q g 8 C X 7 + c 9 3 n t J Y k G J 0 r 7 / 0 n C + T Ux O f Z + e c X / M z s 0 v L C 7 9 P F V Z L h F u o 4 x m s h N D h S n h u K 2 J p r g j J I Y s p v g s v t o b 8 L M / W C q S 8 R N d C N x j 8 I K T l C C o r f U 7 7 E e L T b / l D 8 P 7 L I J 3 0 d x 5 e j x 6 v V 1 9 P I y W G s 9 h k q G c Y a 4 R h U p 1 A 1 / o X g m l J o h i 4 4 a 5 w g K i K 3 i B u 1 Z y y L D q l c N R j f f L O o m X Z t I e rr 2 h + 7 G i h E y p g s U 2 k 0 F 9 q e p s Y H 7 F u r l O t 3 s l 4 S L X m K N R o z S n n s 6 8 w d 5 e Q i R G m h Z W Q C S J n d V D l 1 B C p O 3 t u C H H 1 y h j D P K k D P u m D A c N 4 r T s G 1 N l N 2 N 2 U 2 e F H s P C R C d 1 r C r 4 u I b 7 4 m P b S N S x q m B l 3 C o / H + P z + m C d M e u Y e q F I T C m i f e P a n x D U 3 / 2 z O N 1 o B Z u t j a O g u b s N R j E N V s A a W A c B 2 A K 7 4 A A c g j Z A I A V 3 4 B 4 8 O M J 5 c P 4 6 / 0 a p T u O 9 Z h l U w v n / B v s r + 4 0 = < / l a t e x i t > z < l a t e x i t s h a 1 _ b a s e 6 4 = " a x f Y 0 i W U l s X U b q Z O B D U B x B + B + u U = " > A A A C 7 n i c b Z L P S + Q w F M c z X X W 1 6 8 / 1 K E h x W P A 0 t H r Q m 4 I e P O r q 6 O B 0 K G m a a j B J Q 5 K u d k q O 3 r z q b d n r / g m e / D M E / w b / C T M z w t j q g 8 C X 7 + c 9 3 n

Figure 2: Adding conditional dependencies to the variational posterior. Shaded nodes indicate observed variabes. [left] The true generative model. [center] Variational posterior missing conditional dependencies present in the true posterior. [right] Variational posterior that matches the form of the true posterior.

Figure 3: Hierarchical decomposition of the latents. Shaded nodes indicate observed variables. [left] The true generative model. [right] Variational posterior that matches the form of the true posterior.

Figure 4: Comparing same-text transfer (STT), inter-text transfer (ITT), and prior samples (Prior) for variational models with and without text dependencies in the variational posterior (Var+Txt and Var, respectively). Error bars show 95% confidence intervals for the subjective evaluations.

Figure 5: MCD-DTW reference distance and inter-sample distance for hierarchical latents when transferring via z H and z L .

EVALUATION DETAILS Subjective evaluation Details for the subjective reference similarity and MOS naturalness evaluations are provided in Figures A.1 and A.2. To evaluate reference similarity, we use the AXY side-by-side template in Figure A.1, where A is the reference utterance, and X and Y are outputs from the model being tested and the baseline model.

Figure B.1, we augment the reconstruction loss plots from Figure 1 with additional rate/distortion plots (Alemi et al., 2018) and vary the KL weight, β, and as well as C. Optimization Examples Figure B.2 shows examples of how the KL weight, β, and KL term, R, evolve during training for different capacity limits, C. The curves shown are from the single speaker Var+Txt models discussed in Section 4.3. The target KL is achieved very quickly and then maintained throughout training via continual updates to β that are interleaved with updates to the model parameters. Single-speaker similarity and naturalness results Tables B.1 and B.2 list the raw numbers used in the single-speaker reference similarity and MOS naturalness plots shown in Figure 4 in the main paper. Also shown is MCD-DTW reference distance alongside subjective reference similarity.Hierarchical latents resultsThe similarity and inter-sample variability results for hierarchical latents from Figure5are shown in table format in Table B.3.

Figure A.1: Evaluation template for AXY prosodic reference similarity side-by-side evaluations.A human rater is presented with three stimuli: a reference speech sample (A), and two competing samples (X and Y) to evaluate. The rater is asked to rate whether the prosody of X or Y is closer to that of the reference on a 7-point scale. The scale ranges from "X is much closer" to "Both are about the same distance" to "Y is much closer", and can naturally be mapped on the integers from -3 to 3. Prior to collecting any ratings, we provide the raters with 4 examples of prosodic attributes to evaluate (intonation, stress, speaking rate, and pauses), and explicitly instruct the raters to ignore audio quality or pronunciation differences. For each triplet (A, X, Y) evaluated, we collect 1 rating, and no rater is used for more than 6 items in a single evaluation. To analyze the data from these subjective tests, we average the scores and compute 95% confidence intervals.

Figure A.2: Evaluation template for mean opinion score (MOS) naturalness ratings.A human rater is presented with a single speech sample and is asked to rate perceived naturalness on a scale of 1-5, where 1 is "Bad" and 5 is "Excellent". For each sample, we collect 1 rating, and no rater is used for more than 6 items in a single evaluation. To analyze the data from these subjective tests, we average the scores and compute 95% confidence intervals. Natural human speech is typically rated around 4.5.

Figure B.1: Figure 1 with additional plots showing reconstruction loss vs. the average KL term.In these plots we can see how R AVG varies with embedding dimensionality for constant KL weight, β, while using the KL limit, C, from the optimization problem in eq. 9 achieves constant R AVG .

(a) Transfer via zH.

Figure B.2: Evolution of the KL weight, β, and KL term, R, for different capacity limits, C. The values for the KL term are computed using single training batches. The top plots show the first 10,000training steps, while the bottom plots show the entire 300,000-step training run. The target KL is achieved within the first 1,000 steps and then maintained throughout training via continual updates to β that are interleaved with updates to the model parameters. The value of β is initialized at 1.0 and ends up at around 9.7, 2.6, 1.0, and 0.14 for C = 10, 50, 100, and 300, respectively.

BOUNDING MUTUAL INFORMATION The model with hierarchical latents shown in Figure C.1 gives us the following:p(z) = p(z H , z L ) = p(z L |z H )p(z H )

Inter-speaker same-text prosody transfer results for C = 150 with and without speaker dependencies in the variational posterior(Var+Txt+Spk and Var+Txt, respectively). SpkID denotes the fraction of the time the target speaker was chosen by the speaker classifier. For reference, we provide MOS and SpkID numbers for the baseline model and ground truth audio (though neither are "prior" samples). We can also see that for a fixed C H , increasing C L results in a larger amount of sampleto-sample variation (average MCD-DTW between samples) when inferring a single z ref H from the variational posterior and then sampling z L ∼ p(z L |z ref H

Table content: <table><tr><td/><td colspan="2">Inter-Speaker Transfer</td><td/><td colspan="2">Prior Samples</td></tr><tr><td>Model</td><td>AXY Ref. Similarity</td><td>MOS</td><td>SpkID</td><td>MOS</td><td>SpID</td></tr><tr><td>Var+Txt Var+Txt+Spk Baseline Ground Truth</td><td>0.364 ± 0.104 0.439 ± 0.087 --</td><td colspan="2">3.994 ± 0.066 80.1% 4.099 ± 0.061 95.8% ----</td><td colspan="2">3.674 ± 0.077 78.0% 3.906 ± 0.066 94.9% 4.086 ± 0.060 95.7% 4.535 ± 0.044 96.9%</td></tr><tr><td>the reference.</td><td/><td/><td/><td/><td/></tr></table>

1: Detailed subjective reference similarity scores and objective MCD-DTW reference distance for single speaker models at different capacity limits, C, and with and without text conditioning in the variational posterior(Var+Txt and Var, respectively). Notice how subjective reference similarity for same-text transfer is strongly negatively correlated with MCD-DTW.

Table content: <table><tr><td/><td colspan="2">Ref. Similarity</td><td>MCD-DTW</td></tr><tr><td>Model</td><td>Same-TT</td><td>Inter-TT</td><td>Same-TT</td></tr><tr><td>Var(C=10) Var(C=50) Var(C=100) Var(C=300)</td><td colspan="2">0.192 ± 0.093 0.182 ± 0.097 0.970 ± 0.102 0.509 ± 0.118 1.203 ± 0.102 -0.275 ± 0.139 1.625 ± 0.092 -0.502 ± 0.143</td><td>5.67 5.13 5.04 4.81</td></tr><tr><td colspan="3">Var+Txt(C=10) Var+Txt(C=50) Var+Txt(C=100) 1.346 ± 0.096 1.177 ± 0.103 0.138 ± 0.097 0.065 ± 0.095 1.014 ± 0.102 0.942 ± 0.104 Var+Txt(C=300) 1.514 ± 0.095 1.167 ± 0.110</td><td>5.68 5.11 4.94 4.83</td></tr></table>

2: MOS naturalness scores for single speaker models at different capacity limits, C, with and without text conditioning in the variational posterior (Var+Txt and Var, respectively). Scores are shown for prior samples (Prior), same-text transfer (Same-TT), and inter-text transfer (Inter-TT). These results are visualized in Figure 4 in the main paper. ± 0.056 4.444 ± 0.052 4.312 ± 0.053 Var+Txt(C=50) 4.360 ± 0.053 4.433 ± 0.052 4.326 ± 0.054 Var+Txt(C=100) 4.309 ± 0.056 4.447 ± 0.048 4.270 ± 0.055 Var+Txt(C=300) 3.805 ± 0.076 4.430 ± 0.050 4.162 ± 0.062

Table content: <table><tr><td>MOS score Same-TT 4.438 ± 0.049 4.366 ± 0.053 4.396 ± 0.049 Prior Inter-TT 4.582 ± 0.041 4.492 ± 0.048 4.035 ± 0.066 4.460 ± 0.051 4.029 ± 0.067 3.404 ± 0.093 4.388 ± 0.055 3.249 ± 0.095 2.343 ± 0.098 4.369 ± 0.054 2.733 ± 0.099 4.358 Reconstruction Loss ( Model Ground Truth Base Var(C=10) Var(C=50) Var(C=100) Var(C=300) Var+Txt(C=10)</td></tr></table>

H ) dz H dz L (45) ≤ q(z H |z L )q(z L ) log q(z H |z L ) p(z H ) dz H dz L (46) = p D (x)q(z H |z L )q(z L |x) log q(z H |z L ) p(z H ) dz H dz L dx (47) = p D (x)q(z L |x)D KL (q(z H |z L ) p(z H ))dz L dx(48)

Table content: <table><tr><td>≡ R AVG H</td><td>(49)</td></tr></table>

Peer review (iclr2020):
In this work authors present a regularized, variational autoencoder method for speech synthesis. To endow the latent space with more capacity, the authors employ a modified variational autoencoder objective, which uses a learnable Lagrange multiplier to impose a capacity limit on KL divergence between latent posterior and prior. The authors furthermore propose to decompose the latent embedding space into a two-level hierarchical representation to give generative process more control over style transfer and sample-to-sample variance. They extend earlier theoretical results providing upper bounds on the mutual information between data and its latent embedding to their hierarchical latent representation. In numerical experiments the authors evaluate their approach on a number of speech synthesis tasks involving same-text prosody transfer, inter-text style transfer, inter-speaker prosody transfer. They also analyze speech samples generated from latent samples drawn from the prior. The paper is well-written and easy to follow, but the significance of the main contribution of the paper remains unclear to me. The authors propose to use an augmented standard VAE loss with a capacity hyperparameter and a Lagrange multiplier, but such modifications have been used before and it is not clear to me where is the novelty in there? Moreover, the authors treat the Lagrange multiplier as a learnable parameter, which brings up the question how does it effect the learning dynamics. For instance if the KL-divergence reaches its capacity, the Lagrange multiplier may be pushed down to zero, which in turn can allow the posterior to diverge unchecked to possibly a point mass distribution? The authors provide no details on how beta (the Lagrange multiplier) evolves in their experience and how it effects the stochastic nature of the model. I am not sure why the authors call non-stochastic latent variable models heuristic (instead of deterministic) methods? In Figure 1, how are the loss values computed for variational methods? Can we see how the error bars look for different C values and embedding dimensions? Can the authors be more clear about why for the tasks they consider, a standard (deep) VAE architecture with non-hierarchical latents does not sufficiently capture variations in the data? Can the authors quantify the differences? I am unfamiliar with prior work in this application area, but maybe the work is novel with respect to the application of the regularized VAE framework to speech synthesis. However, the application alone is in my opinion not a contribution that is significant enough for publication.