Effective Use of Variational Embedding Capacity in Expressive End-to-End Speech Synthesis
Authors: Eric Battenberg
Soroosh Mariooryad
Daisy Stanton
RJ Skerry-Ryan
Matt Shannon
David Kao
Tom Bagby
Venue:  (2020)
Keywords: Speech Synthesis
Deep Generative Models
Latent Variable Models
Unsupervised Representation Learning

Abstract: Recent work has explored sequence-to-sequence latent variable models for expressive speech synthesis (supporting control and transfer of prosody and style), but has not presented a coherent framework for understanding the trade-offs between the competing methods. In this paper, we propose embedding capacity (the amount of information the embedding contains about the data) as a unified method of analyzing the behavior of latent variable models of speech, comparing existing heuristic (non-variational) methods to variational methods that are able to explicitly constrain capacity using an upper bound on representational mutual information. In our proposed model (Capacitron), we show that by adding conditional dependencies to the variational posterior such that it matches the form of the true posterior, the same model can be used for high-precision prosody transfer, text-agnostic style transfer, and generation of natural-sounding prior samples. For multi-speaker models, Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior. Lastly, we introduce a method for decomposing embedding capacity hierarchically across two sets of latents, allowing a portion of the latent variability to be specified and the remaining variability sampled from a learned prior. Audio examples are available on the web.

Under review as a conference paper at ICLR 2020
EFFECTIVE USE OF VARIATIONAL EMBEDDING CAPACITY IN EXPRESSIVE END-TO-END SPEECH
SYNTHESIS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has explored Requence-to-sequenee latent variable models for ex- pressive speech synthesis (supporting control and transfer of prosody and style), but has not presented coherent framework for understanding the trade-off's be- tween the competing methods. In this paper, we propose embedding capacity (the amount of information the embedding contains about the data) as unified method of analyzing the behavior of latent variable models of speech, comparing existing heuristic (non-variational) methods variational methods that able to explicitly constrain capacity using an upper bound on representational mutual information. our proposed model (Capacitron), tional dependencies to the variational posterior such that it matches the form of the true posterior, the same model can be used for text-agnostic style transfer, and generation of natural-sounding prior samples. For multi-speaker models, Capacitron is able to preserve target speaker identity during inter-speaker prosody transfer and when drawing samples from the latent prior. Lastly, we introduce a method for decomposing embedding capacity hierarchically across two sets of latents, allowing portion of the latent variability to be specified and the remaining variability sampled from learned prior. Audio examples are available on the weh
show that by adding condi-
prosody transfer,
INTRODUCTION
The synthesis of realistic human speech is a challenging problem that is important for natural human- computer interaction. End-to-end neural network-bard based approaches have seen significant progress in recent years (Wang et al. 2017 Taigman et al. 2018 Ping et al.) 2018 Sotelo et al. 2017). even matching human performance for short assistant- like utterances (Shen et al., 2018 However, these neural models are sometimes viewed as less interpretable or controllable than more traditional models composed of multiple stages of processing that each operate on reified linguistic or phonetic representations.
Text-to-speech (TTS) an underdetermined problem, meaning the same text input has an infinite number of reasonable spoken realizations. In addition to speaker and channel characteristics, im- portant sources of variability in TTS include intonation, stress, and rhythm (collcctively referred to prosody). These attributes convey linguistic, semantic, and emotional meaning beyond what is present in the lexical representation (i.e., the text) (Wagner & Watson 2010). Recent end-to-end TTS research has aimed to model and/or directly control the remaining variability in the output.
Eierry-Ryan et al. (2018) augment Tacotron-like model (Wang et al., 2017) with a deterministic encoder that projects reference speech into a learned embedding space. The system can be used for prosody transfer between speakers ("say like this"), but does not work for transfer between unrelated sentences, and does not preserve the pitch range of the target speaker. [ee & Kim (2019) partially address the pitch range problem by centering the learned embeddings using speaker-wise means.
http:://err.tttooo..aaaaaaaanoop demos/

---

Under review as a conference paper at ICLR 2020
Other work targets style transfer, text-agnostic variation prosody transfer. The Global Style Token (GST) system (Wang et al. 2018) uses a modified attention- reference encoder to transfer global style properties to arbitrary text, and Ma et al. (2019) use an adversarial objective to disentangle style from text.
Hsu et al. 2019) and Zhang et al. (2019) use variational approach (Kingma & Welling 2014) to tackle the style task. Advantages of this approach include its ability to generate style samples via the accompanying prior and the potential for better disentangling between latent style factors (Burgess)
iet al. 2018). Additionally, Hssu et al. (2019) use Gaussian mixture prior over the latents, which (when interpreting the mixture component index as a high-level discrete latent) allows form of hierarchical control.
This work extends the above approaches by providing the following contributions:
1. We propose a unified approach for analyzing the characteristics of TTS latent variable models, independent of architecture, using the capacity of the learned embeddings (i.c., the representational mutual information between the embedding and the data).
2. We target specific capacities for our proposed model using Lagrange multiplier- based opti- mization scheme, and show that capacity is correlated with perceptual reference similarity.
3. We show that modifying the variational posterior to match the form of the true posterior enables style and prosody transfer in the same model, helps preserve target speaker identity during inter-speaker transfer, and leads to natural-sounding prior samples even at high embedding capacities.
4. We introduce a method for controlling what fraction of the variation represented in an embedding is specified, allowing the remaining variation to be sampled from the model.
2 MEASURING REFERENCE EMBEDDING CAPACITY
2.1 LEARNING A REFERENCE EMBEDDING SPACE
Existing heuristic (non-variational) end-to-end approaches to prosody and style transfer (Skerry-Ryan
ett al. 2018 Wang et al. 2018 Lee & Kim 2019 Henter et al. 2018) typically start with the teacher- forced reconstruction loss, used train Tacotron-like squeenc-tt--equenee models and simply augment the model with a deterministic reference encoder, 9e (x), as shown in eq. (2).
L(x, Ys) = logr(x(yr, [yor x||1 K L'(x, yT) ys) = llgg ys. (x)) I1 (yT, Vs. (x)) x||1 K
where x is an audio spectrogram, Yr is the input text, ys is the target speaker (if training a multi- speaker model), fo(.) is deterministic function that maps the inputs to spectrogram predictions, and K is a normalization constant. Teacher-forcing implies that fa(-) is dependent on X when predicting spectrogram frame Xg. In practice, f-(-) serves as the greedy deterministic output of the model, and transfer is accomplished by pairing the embedding computed by the reference encoder with different text or speakers during synthesis.
In these heuristic models, the architecture chosen for the reference encoder determines the transfer characteristics of the model. This decision affects the information capacity of the embedding and allows the model to target specific trade-off between transfer precision (how closely the output resembles the reference) and generality (how well an embedding works when paired with arbitrary text). Higher capacity embeddings prioritize precision and are better suited for prosody transfer to similar text, while lower capacity embeddings prioritize generality and are better suited for text-agnostic style transfer.
The variational extensions from Hsu et al. 2019) and Zhang et al. (2019) augment the reconstruction loss in eq. (2) with KL divergence term. This encourages a stochastic reference encoder (variational posterior), qf (z)x),, to align well with a prior, p(z) (eq. (3). The overall loss is then equivalent to the negative evidence lower bound (ELBO) of the marginal likelihood of the data (Kingma & Welling 2014).
LELBO(X, ys) Exng(e(>)) (xwp.p//p/./../...... (
(3)
-log (exppxxyy,ss LELBO(X, yr.
(4)
2

---

Under review as a conference paper at ICLR 2020
Controlling embedding capacity in variational models can be accomplished more directly by manipu- lating the KL term in (). Recent work has shown that the KL term provides an upper bound on the mutual information between the data, x, and the latent embedding, q(z(x) (Hoffman & Johnson 2016
Makhzani et al. 2015 Alemi et al. 2018).
RAvG (x)) ( 1 (X; Z) = (x) (
R DRL (1(2)\))]l((()
(5)
g(z) (x-
(6)
RAv = 1,(X; Z) + PKL (7(2) (z))
(7)
- (X: Z) - R
(8)
where pD (x) is the data distribution, is the the KL term (3), RAVG the KL term averaged over the data distribution, (X; Z) is the representational mutual information (the capacity of 2), and ((2) is the aggregated posterior. This brief derivation expanded in Appendix C.1
The bound in (8) follows from 7 and the non-negativity of the KL divergence, and shows that the slack on the bound DxI. (q (Z) [l)))) the aggregate KL. In addition to providing a tighter bound, having low aggregate KL is desirable when sampling from the model via the prior, because then the samples of z that the decoder sees during training will very similar to samples from the prior.
Various approaches to controlling the KI. term have been proposed, including varying a weight on the KL term, 3 (Higgins et al. 2017), and penalizing its deviation from a target value (Alemi et al. 2018: Burgess et al. 2018). Because we would like to smoothly optimize for specific bound on the embedding capacity, we adapt the Lagrange multiplier- optimization approach of Rezende & Viola (2018) by applying to the KL term rather than the reconstruction term.
min max { Egzggo>>/o]== log po (x/2. ys)] + 8 #20
(9)
where 0 are the model parameters, 3 serves as an automatically-tunet weight on the KL term, C' the capacity limit, and updates to 0 and B are interleaved during training. We constrain 3 to be non-negative by passing an unconstrained parameter through softplus non-linearity, which makes the capacity constraint a limit rather than a target. This approach is less tedious than tuning 3 by hand and leads more consistent behavior from run-to-run. It also allows more stable optimization than directly penalizing the & deviation from the target KL.
2.2 ESTIMATING EMBEDDING CAPACITY
Estimating heuristic embedding capacity Unfortunately, the heuristic methods do not come packaged with easy way to estimate embedding capacity. We can estimate effective capacity ordering, however, by measuring the test-time reconstruction loss when using the reference encoder from each method. In Figure [ we show how the reconstruction loss varies with embedding dimensionality for the tanh- based prosody transfer (PT) and softmax-based global style token (GST) bottlenecks (Skerry-Ryan
al.) 2018; Wang et al. 2018) and for variational models (Var.) with different capacity limits, C. We also compare to a baseline Tacotron model without reference encoder. For this preliminary comparison, we use the expressive single-speaker dataset and training setup described in Section 4.2 Looking at the heuristic methods in Figure I we see that the GST bottleneck much more restrictive than the PT bottleneck, which hurts transter precision but allows sufficient embedding generality for text-agnostic style transfer.
Bounding variational embedding capacity We saw in (8 that the KL term is an upper bound on embedding capacity, so we can directly target a specific capacity limit by constraining the KL term using the objective in eq. ( For the three values of in Figure [ we can see that the reconstruction loss flattens out once the embedding reaches certain dimensionality. This gives us consistent way to control embedding capacity as it only requires using a reference encoder architecture with sufficient structural capacity (at least C) to achieve the desired representational capacity in the variational embedding. Because of this, we use 228-dimensional embeddings in all of our experiments, which should be sufficient for the range of capacities target.
3

---

Under review as a conference paper at ICLR 2020
Reconstruction Loss vs. Embedding Dim
Reconstrnction Loss (81)
8000
7900
7500
16
175
Embodding Dinensiomllit
Figure 1: Reconstruction loss embedding dimensionality for variety of heuristic and variational models. For the variational model (Var.), we vary the capacity limit, C. Notice how the reconstruction loss flattens out lower values for higher values of C. The heuristic models denoted by PT and GST for Prosody Transfer and Global Style Tokens. Figure B.1 in the appendix shows how the KL term changes when varying C well as the KL weight, 3.
3 MAKING EFFECTIVE USE OF EMBEDDING CAPACITY
3.1 MATCHING THE FORM OF THE TRUE POSTERIOR
In previous work (Hsu et 2019 Zhang et al. 2019), the variational posterior has the form (z)(x), which matches the form of the true posterior for a simple generative model p(x[z)p(z). However, for the conditional generative model used in TTS, p(x|z. yr. Vs p(z), is missing conditional dependencies present in the true posterior, p z x, Vr. Ys). Figure 2 shows this visually. In order to
yr
ys
Yr
ys
p(x v. Vs)p(z)
2(2(x)
g(2)x, yr.ys)
Figure 2: Adding conditional dependencies to the variational posterior. Shaded nodes indicate observed variabes. [left] The true generative model. [center] Variational posterior missing conditional dependencies present in the true posterior. [right] Variational posterior that matches the form of the true posterior.
match the form of the true posterior, we inject information about the text and the speaker into the network that predicts the parameters of the variational posterior. Speaker information represented as learned speaker-wise embedding vectors, while the text information is summarized into a vector by passing the output of the Tacotron text encoder through unidirectional RNN as done by Stanton et al. (2018). Appendix A. gives additional details.
For this work, we use simple diagonal Gaussian for the approximate posterior, q(z)x, Yr. Ys) and standard normal distribution for the prior, p(z). We use these distributions for simplicity and efficiency, but using more powerful distributions such as Gaussian mixtures or normalizing flows (Rezende Mohamed 2015) should decrease the aggregate KL, leading to better prior samples.
Because we are learning a conditional generative model, p(xlyr.ys) we could have used learned conditional prior, p(z|yr, ys), in order improve the quality of the output generated when sampling via the prior. However, in this work we focus on the transfer use case where we infer z (f yfrf rff from reference utterance and use it to re-synthesize speech using different text speaker inputs, x' "f yr. yy). Using fixed prior allows to share high probability region across all text and speakers so that an embedding inferred from one utterance is likely to lead non-degenerate output when being used with any other text speaker.
4

---

Under review as a conference paper at ICLR 2020
3.2 DECOMPOSING EMBEDDING CAPACITY HIERARCHICALLY
In inter-text style transfer uses cases, we infer znd from a reference utterance and then use it to generate a new utterance with the same style but different text. One problem with this approach is that zt completely specifies all variation that the latent embedding is capable of conveying to the decoder, p(x 1 YT, Ys). So, even though there are many possible realizations of an utterance with given style, this approach produce only ond
To address this issue, we decompose the latents, z, hierarchically (Sønderby et al., 2016) into high- level latents, Zh, and low-level latents, Zt,, shown in Figure 3 This differs from the hierarchical interpretation the Gaussian mixture prior used by [ssu et al. (2019) in that here the high-level latents continuous vectors rather than single categorical variable. Factorizing continuous latents in this way allows us to specify how the joint capacity, (X: [Z11, Zr.).. is divided between ž and ZL. This approach can also be extended to additional levels of latents, each containing prescribed proportion of the overall joint capacity.
ZII
X11
YT
Z
ys
ZL
ys
p(xlyT.ys)p(z,pp ws)p((z//2))pae
q(zn (ZL yr.ys)
Figure 3: Hierarchical decomposition of the latents. Shaded nodes indicate observed variables. [left]
The true generative model. [right] Variational posterior that matches the form of the true posterior.
As shown in eq. (8). the KL term, R is an upper bound on (X; Z). We can also derive similar bounds for I ZH) and I (X; Zr.). Derivations of these bounds are provided in Appendix C.2
T Z S n Exx-potx
pl) zm)p(()))
(10)
1 X: Zn) RAVG Ex-po(x) [x) [)]Dx (x (g(zzlzz))l((znn)
(11)
we define Rg RAVG R latents:
RAVG we end up with the following capacity limits for the hierarchical
1,(X RA
LX ZL)
DAVG
R
(12)
The negative ELBO for this model can be written
LELBO (x, ys. - mggin loog YT. ys)l + RH + Rr.
(13)
where Rn and Rg, are single data point estimates of RAVO and R computed from x. In order to specify how the joint capacity is distributed between the latents, we extend (9) to have two Lagrange multipliers and capacity targets.
min max 11 94.922
(Ex po + Su(RH CH) + ( CL) (14)
Cn limits the information capacity of zи, and C.. limits how much capacity 2,. has in excess of zh (i.e., the total capacity of ZL is capped at CH + Ct). This allows us to infer z (22 yref ygst) from a reference utterance and use it to sample multiple realizations, x' p(x|Zz. Nr, (s)p((z (z.e). Intuitively, the higher Cи is, the more the output will resemble the reference, and the higher Ct. is, the more variation we would expect from sample to sample when fixing 2 and sampling from p( 2
If the decoder were truly stochastic (not greedy), could actually sample multiple realizations given the ze but, high embedding capacities the variations would likely he very similar perceptually.
5

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
4.1 MODEL ARCHITECTURE AND TRAINING
Model architecture The baseline model we start with is Tacotron-based system (Wang al. 2017) that incorporates modifications from Akerry-Ryan et al. 2018), including phoneme inputs instead of characters, GMM attention (Graves] 2013), and a WaveNet neural vocoder (van den Oord ct al. 2016) to convert the output mel spectrograms into audio samples (Shen et al. 2018). The decoder RNN timestep. We use the CBHG text encoder from Wang et al. (2018) and the GMMv2b attention mechanism from Battenberg et (019))
a reduction factor of meaning that it produces two spectrogram frames per
For the heuristic models compared in Section 2.2 we augment the baseline Tacotron with the reference encoders described by Skerry-Ryan al. (2018) and Wang al. (2018). For the variational models that we compare in the following experiments, we start with the reference encoder from [Skerry-yyaa al. (2018) and replace the tanh bottleneck layer with an MLP that predicts the parameters of the variationall posterior. When used, the additional conditional dependencies (text and speaker) are fed into the MLP as well.
Training To train the models, the primary optimizer is run synchronously across 10 GPU workers (2 of them backup workers) for 200,000 training steps with an effective batch size of 256. uses the Adam algorithm (Kingma & Ba, 2015) with a learning rate that is annealed from 10-3 to 5 x 10-5 over 200,000 training steps. The optimizer for 3 is run asychronously on the workers and uses SGD with momentum 0.9 and a fixed learning rate of 10 The updates for these two optimizers interleaved, allowing to converge to a steady state value that achieves the target value for the KL term, as demonstrated in Figure B.2 in the appendix. Additional architectural and training details provided in Appendix A
4.2 EXPERIMENT SETUP
Datasets For single-speaker models, use an expressive English language audiobook dataset consisting of 50,086 training utterances (36.5 hours) and 912 test utterances spoken by Catherine Byers, the speaker from the 2013 Blizzard Challenge. Multi-speaker models are trained using high- quality English data from 58 voice assistant-like speakers, consisting of 419,966 training utterances (327 hours). We evaluate on a 9-speaker subset of the multi-speaker test data which contains 1808 utterances (comprising US, UK, Australian, and Indian speakers).
Tasks The tasks that we explore include same-text prosody transfer, inter-text style transfer, and inter-speaker prosody transfer. We also evaluate the quality of samples produced when sampling via the prior. For these tasks, we compare performance when using variational models with and without the additional conditional dependencies in the variational posterior at number of different capacity limits. For models with hierarchical latents, we demonstrate the effect of varying Cи and CL for same-text prosody transfer when inferring An and sampling 2 or when inferring 24. directly.
Evaluation We use croucesourced native speakers to collect two types of subjective evaluations. First, mean opinion score (MOS) rates naturalness use the AXY side-by-side comparison proposed by /Skerry-Ryan et al. (2018) to measure subjective similarity reference utterance relative the baseline model on a scale of [-3,3]. For example, score of 3 would mean that, compared to the baseline model, the model being tested produces samples much more perceptually similar the ground truth reference. We also use an objective similarity metric that uses dynamic time warping to find the minimum mel cepstral distortion (Kubichek) 1993) between two sequences (MCD-DTW). Lastly, for inter-speaker transfer, we follow Enterry-Ryan et al. (2018) and use simple speaker classifier to measure how well speaker identity is preserved. Additional details on evaluation methodologies are provided in Appendix A
scale of 1-5, 5 being the best. Second,
4.3 RESULTS
Single speaker For single-speaker models, compare the performance same and inter-text transfer and the quality of samples generated via the prior for models with and without text condi- tioning in the variational posterior (Var+ Txt and Var, respectively) different capacity limits, C.

---

Under review as a conference paper at ICLR 2020
Reference Similarity
MOS (Naturalness)
AXY Score (Moslel vs. Boseline)
1.5
4.5
1.0
MOS Seore
Var STT
0.5
Var ITT
3.3
VareTxt STT
VareTxi
0.0
3.0
Var Prior
Vare Txt Prior
0.5
Var STT
Var+Txa STT
Var ITT
Var+ Txt ITT
Baseline
2.0
1.0
Capacity Limit, C [nats]]
Capacity Limit, C[[nas]]
Figure 4: Comparing same-text transfer (STT), inter-text transfer (ITT), and prior samples (Prior) for variational models with and without text dependencies in the variational posterior (Var+ War+Txt and Var, respectively). Error bars show 95% confidence intervals for the subjective evaluations.
Similarity results for the transfer task are shown on the left side of Figure and demonstrate increas- ing reference similarity as C' is increased, with the exception the model without text conditioning on the inter-text transfer task. Looking the MOS naturalness results the right side of Figure 4 we see that both inter-text transfer and prior sampling take a serious hit as capacity is increased for the Var model, while the Var+Txt model able to maintain respectable performance even at very high capacities on all tasks.
Listening the audio examples, we can hear that the Var model produces degenerate output high capacities when attempting to transfer the style from a short utterance to long utterance. This indicates that the decoder probably hasn't 't seen similar embeddings paired with long utterances during training, which suggests that z is improperly correlated with text length. Similar behavior is also observed when generating prior samples using shorter or longer text. This means that an arbitrary (sampled from the prior or inferred from reference) is unlikely to pair well with text of an arbitrary length.
Multi-speaker For multi-speaker models, we compare inter-speaker same-text transfer perfor- mance and prior sample quality with and without speaker conditioning in the variational posterior (Var+Tx+SSp and Vxr+Txt, respectively) that both models are able to preserve characteristics of the reference utterance during transfer (AXY Ref. Similarity column), while the Var+Txt+Spk model has an edge in MOS for both inter-speaker transfer and prior samples (almost matching the MOS the deterministic baseline model even at high embedding capacity).
fixed capacity limit of 150 nats. In Table [ see
Similar to the utterance length argument in the single speaker section above, it is likely that adding speaker dependencies to the posterior allows the model (meaning z is not correlated with speaker identity), thereby forcing the decoder to learn to map all plausible points in the latent space to -souuranlouuningggggo utterances that preserve the target speaker's pitch range. The speaker classifier results show that the Var+Txt+Spk model preserves target speaker identity about as well as the baseline model and ground truth data (-5% of the time the classifier chooses a speaker other than the target speaker), whereas for the Var+Txt model this happens about 22% of the time. Though 22% seems like large speaker error rate, much lower than the 79% figure presented by (Skerry-Ryan et al. (2018) for a heuristic prosody transfer model. This demonstrates that even with a weakly conditioned posterior, the capacity limiting properties of variational models lead to better transfer generality and robustness.
the entire latent space for each speaker
Hierarchical latents To evaluate hierarchical decomposition of capacity in a single speaker setting, the MCD-DTW distance to quantify reference similarity and same-reference inter-sample variability. As shown Table B.I in the appendix, MCD-DTW strongly (negatively) correlates with subjective similarity.
The left side of Figure 5 shows results for samples generated using high-level latents, Zμ, inferred from the reference. As On is increased, strong downward trend in the average distance
7

---

Under review as a conference paper at ICLR 2020
Table I: Inter-speaker same-text prosody transfer results for C' 150 with and without speaker dependencies in the variational posterior (Var+Txt+Spk and Var+Txt, respectively). SpkID denotes the fraction of the time the target speaker was chosen by the speaker classifier. For reference, we provide MOS and SpkID numbers for the baseline model and ground truth audio (though neither are "prior" samples).
Inter- Inter-Speaker Transfer
Prior Samples
Model
AXY Ref. Similarity
MOS
SpkID
MOS
SpID
Var+Txt
0.364 0.104
3.994 0.066 4.099 + 0.061
80.1%
3.674 + 0.077
78.0%
Var+Txt+Spk
0.439 0.087
95.8%
3.906 + 0.066
94.9%
Baseline
4.086 + 0.060
95.7%
Ground Truth
4.535 + 0.044
96.9%
the reference. We also that for fixed Cn. increasing Ct. results in larger amount of sample- to-sample variation (average MCD-DTW between samples) when inferring single z from the variational posterior and then sampling zz. ~ p(z from the prior to use in the reconstructions.
The right side of Figure the same metrics but for samples generated using low-level latents, z inferred from the variational posterior. In this case, we see a slight downward trend in the reference distance as the total capacity limit, C, is increased (the trend less dramatic because the capacity is already fairly high). We also see significantly lower samer- distance because the variation modeled by the latents is completely specified by z In this case, we sample multiple z|!l's from (Zz |x y!r for the same x because using the same ž would lead identical output from the deterministic decoder.
Using Capacitron with hierarchical latents increases the model's versatility for transfer tasks. By inferring just the high-level latents, ZH, from reference, we can sample multiple realizations of an utterance that are similar to the reference, with the level of similarity controlled by CH. and the amount of comple-to-samppe variation controlled by Ct. The same model can also be used for higher fidelity, lower variability transfer by inferring the low-level latents, ZL, from a reference, with the level of similarity controlled by C' Cn + CL. As mentioned before, this idea could also be extended to use additional levels of latents, thereby increasing transfer and sampling flexibility.
Distaaee From Reference Transfer via
Transfer via zz
Distance Frous Reference
55
Inte-Sample Distance
Inter-Sumple Distance
MCD-DIW
4.5
Cp 20
Ca 20
Cy
Ca 50
Cy - 100
Cg 100
Cy 20
Gy
Gp 20
Ga 50
Cp
100
Cy 100
50
GL 100
50
100
G. 50
100
G 50
-050
- 100
G. 100
G. 50
G. 1081
C- 70
C 120
C- 100
150
C 150
C-200
C- 70
100
C- 120
C- 150
C 150
-00
Figure 5: MCD-DTW reference distance and inter-sample distance for hierarchical latents when
transferring via Zμ and Z1.
To appreciate the results fully, it is strongly recommended to listen to the audio examples available on the w
http:///ww....o.i.....oaooooaaoa demos/
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
We have proposed embedding capacity (i.e., representational mutual information) as a useful frame- work for comparing and configuring latent variable models of speech. Our proposed model, Capac- itron, demonstrates that including text and speaker dependencies in the variational posterior allows single model to be used successfully for variety of transfer and sampling tasks. Motivated by the multi-faceted variability of natural human speech, we also showed that embedding capacity can be decomposed hierarchically in order to enable the model to control a trade-off between transfer fidelity and sumpletoosamppe variation.
There are many directions for future work, including adapting the fixed-length variational embeddings to be variable-length and synchronous with either the text or audio, using more powerful distributions like normalizing flows, and replacing the deterministic decoder with a proper likelihood distribution. For transfer and control uses cases, the ability to distribute certain speech characteristics across specific subsets of the hierarchical latents would allow more fine-grained control different aspects of the output speech. And for purely generative, non-transfer use cases, using more powerful conditional priors could improve sample quality.
REFERENCES
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy, Fixing a broken elbo. In International Conference on Machine Learning, pp. 159-168, 2018.
Eric Battenberg, RJ Skerry-Ryan Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relatiee attention mechanisms for robust long-fore for- speech synthesis. arXiv preprini arXiv: 1910. 1910.1088,, 2019.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling 3-vae. arXiv preprint arXiv:/804.03399, 2018.
Alex Graves, Generating sequences with recurrent neural networks. arXiv preprint arXiv: 2013.
Gustav Eje Henter, Jaime Lorenzo-Trueba, Xin Wang, and Junichi Yamagishi. Deep encoder- decoder models for unsupervised learning of controllable speech synthesis. arXiv preprint 1171v.1807.11470. 2018
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
Matthew D Hoffiman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, 2016.
Wei-Ning Hsu, Yu Zhang, Ron Weiss, Heiga Zen, Yonghui Wu, Yuan Cao, and Yuxuan Wang. Hicrarchical generative modeling for controllable speech synthesis. In International Conference on Learning Representations, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference for Learning Representations, 2015.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
R Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Communica- tions, Computers and Signal Processing, 1993., IEEE Pacific Rim Conference on, volume 1, pp. 125-128. IEHE, 1993.
Younggun Lee and Taesu Kim. Robust and fine-graing prosody control of end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5911-5915. IEEE, 2019.

---

Under review as a conference paper at ICLR 2020
Shuang Ma, Daniel Mcduff, and Yale Song. A generative adversarial network for style modeling in text-to-speech system. In International Conference on Learning Representations, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitl,, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint 1rXiv:5511005441 2015.
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang. Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference Learning Representations, 2018.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pp. 1530-1538, 2015.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. arXiv preprint arXiv: 2018.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
RJ Skerry-RR Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalge, Spren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, pp. 3738-3746,
2016.
Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. Char2wav: End-to-end speech synthesis. In International Conference on Learning Representations, Workshop Track, 2017.
Daisy Stanton, Yuxuan Wang, and RJ Skerry- Skerry-Ryan. Predicting expressive speaking style from text in end-to-end speech synthesis. In 2018 IEEE Spoken Language Technology Workshop (SLT), pp. 595-602. IEEE, 2018.
Yaniv Taigman, Lior Wolf, Adam Polyak, and Eliya Nachmani. Voiceloop: Voice fitting and synthesis via a phonological loop. In International Conference Learning Representations, 2018.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop, 2016.
Michael Wagner and Duane G Watson. Experimental and theoretical advances in prosody: A review. Language and cognitive processes, 25(7-9):905-99) 9959990099999999999 2010.
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In Proceedings of Interspeech, August 2017.
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning, pp. 5167-5176, 2018
Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen-Hua Ling. Learning latent representations for style control and transfer in end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International IEEE, 2019.
Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
10

---

Under review as a conference paper at ICLR 2020
A
EXPERIMENT DETAILS
A.1 ARCHITECTURE DETAILS
Baseline Tacotron The baseline Tacotron we start with (which serves as (.) in eq. ① is similar to the original sequence-to-sequenc model described by Wang et al. (2017) but uses some modifications introduced by Skerry-Ryan et al. (2018). Input to the model consists of sequences of phonemes produced by text normalization pipeline rather than character inputs. The CBHG text encoder from Wang et al. 2017) is used to convert the input phonemes into a sequence of text embeddings. Before being fed the CBHG encoder, the phoneme inputs converted to learned 256-dimensional emheddings and passed through a pre-net composed of two fully connected ReL.U layers (with 256 and 128 units, respectively), with dropout of 0.5 applied to the output of each layer. For multi-speaker models, a learned embedding for the target speaker is broadcast-concatenate to the output of the text encoder.
The attention module uses single LSTM layer with 256 units and zoneout 0.1 0.1 followed by an MLP with 128 tanh hidden units compute parameters for the monotonic 5-component GMM attention window. Instead of using the exponential function to compute the shift and scale parameters of the GMM components as in (Graves) 2013). we use the softplus function, which we found leads to faster alignment and more stable optimization.
The autoregressive decoder module consists of LSTM layers each with 256 units, zoneout of 0.1, and residual connections between the layers. The spectrogram output is produced using linear layer on top the 2 LSTM layers, and we use a reduction factor of 2, meaning we predict two spectrogram frames for each decoder step. The decoder is fed the last frame of its most recent prediction (or the previous ground truth frame during training) and the current context as computed by the attention module. Before being fed to the decoder, the previous prediction is passed through the same pre-net used before the text encoder above.
Mel spectrograms The mel spectrograms the model predicts are computed from 24kHz audio using a frame size of 50ms, a hop size 12.5ms, an FFT size of 2048, and a Hann window. From the FFT energies, we compute 80 mel bins distributed between 80Hz and 12kHz.
Reference encoder The common reference encoder we use to compute reference embeddings starts with the mel spectrogram from the reference and passes it through stack of convolutional layers, each using ReI.U non-linearities, 3x3 filters, 2x2 stride, and batch normalization. The 6 layers have 32, 32, 64, 64, 128, and 128 filters, respectively. The output of this convolution stack is fed into unidirectional LSTM with 128 units, and the final output of the LSTM serves as the output of our basic reference encoder.
To replicate the prosody transfer model from Skerry- Ryan et al. 2018)) we pass the reference encoder output through an additional tanh bottleneck layer to compute the embedding. For the Style Tokens model in Wang et al. (2018), we pass the output through the Style Tokens bottleneck described in the paper. For the approximate posterior in our variational models, we pass the output of the reference encoder (and potentially vectors describing the text and/or speaker) through an MLP with 128 tanh hidden units to produce the parameters of the diagonal Gaussian posterior which we sample from to produce reference embedding. For all models with reference encoders, the resulting reference embedding is broadcast-concatenn
to the output of the text encoder.
Conditional inputs When providing information about the text to the variational posterior, pass the sequence of text embeddings produced by the text encoder to a unidirectional RÑN with 128 units and use its final output as fixed-length text summary that is passed to the posterior ML.. Speaker information passed to the posterior MI.P via a learned speaker embedding.
A.2 TRAINING DETAILS
For the optimization problems shown in eqs. p and (4). we use two separate optimizers. The first minimizes the objective with respect to the model parameters using the SynnCeppliasOptimmzze from Tensorflow with 10 workers (2 of them backup workers) and an effective batch size of 256. We
https: : http///wwwweeoflll///.../oo http:///www.....eoo//oooo//////a
Synckep1 imizer

---

Under review as a conference paper at ICLR 2020
also gradient clipping with a threshold of 5. This optimizer uses the Adam algorithm (Kingma &
Ba 2015) with 31 0.9, B2 = 0.999, < <= 10 and a learning rate that is set to 10 33 5.5 x 10
3 x 10 10 4, and 5 10 at 50k, 100k, 150k, and 200k steps, respectively. Training is run for
300k steps total.
The optimizer that maximizes the objective with respect to the Lagrange multiplier is asyn- chronously across the 10 workers (meaning each worker computes an independent update using its 32-example sub-batch) and uses SGD with a momentum of 0.9 and learning rate of 10-5, The Lagrange multiplier is computed by passing an unconstrained parameter through the softplus function in order enforee non noggetiaiii The initial value of the parameter is chosen such that the Lagrange multiplier equals 1 at the start of training.
A.3 EVALUATION DETAILS
Subjective evaluation Details for the subjective reference similarity and MOS naturalness eval- uations are provided in Figures I. and A.2 To evaluate reference similarity, we use the AXY side-by-side template in Figure A. where Á is the reference utterance, and X and Y are outputs from the model being tested and the baseline model.
MCD-DTW We evaluate the models with hierarchical latents using the MCD-DTW distance to quantify reference similarity and the amount of inter-sample variation. To compute mel cepstral distortion (MCD) (Kubichek (993). we use the same mel spectrogram parameters described in A. and take the DCT to compute the first 13 MFCCs (not including the Oth coefficient). The MCD between two frames the Euclidean distance between their MFCC vectors. Then we use the dynamic time warping (DTW) algorithm (Müller) 2007) (with a warp penalty of 1.0) to find an alignment between two spectrograms that produces the minimum MCD cost (including the total warp penalty). We report the average per-frame MCD-DTW.
To evaluate reference similarity, we simply compute the MCD-DTW between the synthesized audio and the reference audio (a lower MCD-DTW indicates higher similarity). The strong (negative) correlation between MCD-DTW and subjective similarity is demonstrated Table quantify inter-sample variation, we compute 5 output samples using the same reference and compute the average MCD-DTW between the first sample and each subsequent sample.
B ADDITIONAL RESULTS
Rate-distortion plots Figure B.I we augment the reconstruction loss plots from Figure with additional rate/distortion plots (Alemi et al. 2018) and vary the KL weight, B, and as well C.
Optimization Examples Figure B.2 shows examples of how the KL weight, B, and KL term, R. evolve during training for different capacity limits, C. The curves shown are from the single speaker Var+Txt models discussed Section 4.3 The target KL is achieved very quickly and then maintained throughout training via continual updates to that are interleaved with updates to the model parameters.
Single-speaker similarity and naturalness results Tables B.1 and B.2 list the raw numbers used in the single-speake reference similarity and MOS naturalness plots shown in Figure | in the main paper. Also shown is MCD-DTW reference distance alongside subjective reference similarity.
Hierarchical latents results The similarity and inter-sample variability results for hierarchical latents from Figure are shown in table format in Table B.3
12

---

Under review as a conference paper at ICLR 2020
IMPORTANT:
you lsten somp using honds envirommntt
you heaekphome
think you have good lestning abil any resson, you cantt the mudio sarpples
For this usik you will be given a reforence ap speech and will be asked to ducide whihh two speech sumples more ciosely marzhes is prosedy d.e. intonation stress fow)
You thatt presentte win audio a neteeence ce sceecht sampis. isten audio for the reterence Next, bareen two diterent speech samples and decce prosody.
Instructions
spoise speaser same speech samples, sut may that reference speech comple.
After nang het speech, plaase wast Imaut seconds befooe lakering to the right spwach frew to re-isten neleurce sarppée befors intaring (ollowiing
promunciation iesues
quality diffrencce. Pleose only prosody, which could
any
pltch, dises falls the sacech
Stress put on each word eyllatle (e.g. loudrese or pitch changee). Spewking new, changess throuhhout the speech surrple.
your qualiaaive
you articuate
Haadphanes, with naise the background, am Bntering spench sempiee uning neadphomee them rotse around me (people telling ett..
How are you intening
music playing, dondrdriiieers levell with
background. sm bewwing the spaech bendfporwss and lownloowll around (people taking, music soying loner, and fina, ete.).
samples?
Audio speakers or other (Please refease the task).
The reference for
rating
reference
Reforence
the speach
speech samples you
Paa Speech
Play Right Speech
Are there any problemm [check applicable)
sample?
There with let jplesse comment)
There problem wht the speach cample
Mightty vienar
Abeat same
Appply couarr
closer reference?
What makes your
Figure A.I: Evaluation template for AXY prosodic reference similarity side-by-side evaluations. A human rater is presented with three stimuli: reference speech sample (A), and two competing samples (X and Y) to evaluate. The rater asked to rate whether the prosody of X or is closer to that of the reference a 7-point scale. The scale ranges from "X is much closer" "Both are about the same distance" to "Y is much closer", and can naturally be mapped on the integers from -3 to 3. Prior collecting any ratings, provide the raters with 4 examples of prosodic attributes to evaluate (intonation, stress, speaking rate, and pauses), and explicitly instruct the raters to ignore audio quality or pronunciation differences. For each triplet (A, X. Y) evaluated, we collect I rating, and no rater is used for more than 6 items in single evaluation. To analyze the data from these subjective tests, we average the scores and compute 95% % confidence intervals.
Table B. Detailed subjective reference similarity scores and objective MCD-DTW reference distance for single speaker models at different capacity limits, C, and with and without text conditioning in the variational posterior (Var+Txt and Var, respectively). Notice how subjective reference similarity for same-text transfer is strongly negatively correlated with MCD-DTW.
Ref. Similarity
MCD-DTW
Model
Same-TT
Inter-TT
Same-TT
Van(C=10)
0.192 + 0.093
0.182 + 0.097
5.67
Var(C=50)
0.970 0.102
0.509 = 0.118
5.13
V=n(C=
1.203 0.102
-0.275 + 0.139
5.04
Var(C=300)
1.625 0.092
-0.502 + 0.143
4.81
Var+Tx((C=10)
0.138 0.097
0.065 0.095
5,68
Va55)
1.014 0.102
0,942 + 0.104
5.11
Var+Txi(C=100)
1.346 + 0.096
1.177 + 0.103
4.94
Var+Txt(C==00)
1.514 + 0.095
1.167 + 0.110
4.83
13

---

Under review as a conference paper at ICLR 2020
nstruction
IMPORTANT:
In thes projett you' listen au samp tes, Please relense this lask any the following mue:
21 You do have good luszn ubility
3) There consi badigiound notoo (stieet nolso loud fontbinocnd foner, open TViradio, people tallingg ess).
4) any
you
There DEVICE Heaadphones):
headptones.
than
preferned dosed headphones, headphones,
If you ane not sure which type you havn, pliane new the oitredda aticis
2]
soi volume your ausio dovizo confortutie ava..
in this task, we would like you laatnn ao speech rerrnce and then choose soore for the audio sample you vo just heard. This score should reffect your osliio of how natural or unnatural
Plasse:
1) ssenn to each sample loost tifoe, with at loaet one see break between ther.
eamopls for sach leval. Hacnew the buble lasken all nafenences. Important note: need lattnn the reforences it have itatened them bettrre
pption
Please speech serppee falls bettveen two inweds.
allowed "n-beewenn" example.
"Excoller Goodfyy Feol you
Nakura Kruas Scadlar
How are you lidening the spesch sample?
Headphones, notes the bxchground, am apeach sample haadpfanes and thane
ai-cord toners
and lane, esc.)
Headphonee some low-lavll notee background. am belnring to the spanch samplle using headphoress and some loweloval rotee around (peeple bekkng muskc playing sonditoness fins,
other
Please rate naturainsss spench sample:
Soore
Naluraa neaa
Descriptio
50
Excellent
Complenely natural speech
4.0
Good
Mosty notural spench
Fqually natural speech
Poor
1.0
Bad
Completely unnatuaal speech
Figure A.2: Evaluation template for mean opinion score (MOS) naturalness ratings, human rater is presented with single speech sample and is asked to rate perceived naturalness scale of 1-5, where "Bad" and is "Excellent". For each sample, we collect rating, and no rater is used for more than 6 items in single evaluation. To analyze the data from these subjective tests, average the scores and compute 95% confidence intervals. Natural human speech is typically rated around 4.5.
Table B.2: MOS naturalness scores for single speaker models different capacity limits, C, with and without text conditioning in the variational posterior (Var+Txt and Var, respectively). Scores are shown for prior samples (Prior), same-text transfer (Same-TT), and inter-text transfer (Inter-TT). These results are visualized in Figure in the main paper.
MOS score
Model
Prior
Same-TT
Inter-TT
Ground Truth
4.582 + 0.041
Base
4.492 + 0.048
Can
4.438 + 0.049
4.366 + 0.053
4,396 + 0.049
V=r(C=50)
4.035 + 0.066
4,460 + 0.051
4.029 = 0.067
C=r(C=100)
3,404 0.093
4.388
0.055
3.249 = 0.095
V3n(C==000)
2.343 0.098
4.369 0.054
2.733 + 0.099
Var+ Txi(C=10)
4.358 0.056
1.444
4.312 = 0.053
Var+ Var+Ix((==0)
4,360 + 0.053
4.433 - 0.052
4.326 + 0.054
Var+ Txt[C=1===0)
4.309 + 0.056
4.447 + 0.048
4.270 + 0.055
Var+ Mar+Tx(C==30)
3.805 + 0.076
4.430 + 0.050
4.162 + 0.062
14

---

Under review as a conference paper at ICLR 2020
Reconstruction Loss vs. Embedding Dim Varying F
Reconstruction Loss vs. R Virying B
Recinsirccton Lows (f1)
Baseline
8000
0900
3400
Var. - 0.1
Var. B
PT
CNT
Var. 10
Bascline
2230
Vaa. 100
Varying
Varying
8900
Var, Vir. Ci- 10
Bascline
Resinstrrction Lows (E.)
7800
7000
Via: 150
PT
Var. C-500 50
GST
7230
Va: 10
Brscline
Emhedding Dimensiomality
8000 Average KL [nais]
Figure B.1: Figure I with additional plots showing reconstruction loss vs. the average KL term. In these plots we can see how Rava varies with embedding dimensionality for constant KL weight, B, while using the KL limit, C, from the optimization problem in eq. 9 aachieves constant Ravo
Table B.3: Transfer using hierarchical latents. "Ref." is the the average MCD-DTW distance from the reference, and "a-smmp." is the average inter-sample MCD-DTW. These are the numbers used in the plots in Figure
(a) Transfer via Zn-
(b) Transfer Z1.-
Capacity Limits
MCD-DTW
Capacity Limits
MCD-DTW
Cи
Ct
C
Ref.
X-samp.
Ch
Ct
C
Ref.
X-samp.
0
0
6.054
0
0
0
6.054
20
50
70
5.517
4.638
20
50
70
4.991
3.899
20
100
120
5.453
4.670
50
50
100
4.916
3.876
50
50
100
5.172
4.245
20
100
120
4.882
3.847
50
100
150
5.166
4.332
100
50
150
4.834
3.830
100
50
150
4.952
3.999
50
100
150
4.797
3.832
100
100
200
5.000
4.147
100
100
200
4.852
3.858
15

---

Under review as a conference paper at ICLR 2020
Evolution Weight.
Evolution KL Term, R
www.
12.5
300
250
10.0
200
C= 10
R
150
100
100
2.5
10k
10k
Training steps
Training steps
Evolution KI. Weight, B
Evolution KI. 'eemm R
C- 10
C-50 50
C-
7.5
150
100
300
100
50
0.0
Training steps
Training steps
Figure B.2: Evolution of the KL weight, B, and KL term, R, for different capacity limits, C. The values for the KL term are computed using single training batches. The top plots show the first 10,000 training steps, while the bottom plots show the entire 300,000-step training run. The target KI. is achieved within the first 1,000 steps and then maintained throughout training via continual updates to that are interleaved with updates the model parameters. The value of is initialized at 1.0 and ends up at around 9.7, 2.6, 1.0, and 0.14 for C 10, 50, 100, and 300, respectively.
16

---

Under review as a conference paper at ICLR 2020
C
DERIVATIONS
C.I BOUNDING REPRESENTATIONAL MUTUAL INFORMATION
Definitions:
R= / ((()x) log g(z|x) p(2)
dz
(KL term)
(15)
RAvG |/ pD (x(x)g(z/x)) log
q( ((2/x) dxdz
(Average KL term)
(16)
p(z)
/| PD pD(x)g(z/x)) log
g(2(x) dxdz
(X:
(Representational mutual information)
(17)
((2)
(()
(x(x)g(2/x)dxx
(Aggregated posterior)
(18)
KL non-negativity:
I ((x)) log dx
g(x) p(a)
(19)
logg())
g(x) log p(z)dz
(20)
Mutual information upper bounded by the average KL (Alemi et al. 2018
/] px(x)g(z log
g(2)x) dxdz
(21)
q(z)
// pD(x)g(z()) log (x)g((())xxdd
pD(x)g(Z|x)) log (z))xxdz
(22)
pD(())(z)x) log (z)
((z) log log)
(23)
$ pD(x)q(Z x) log ( dxdz
((z) log p(z)(z
(24)
|/ pD(x))(z ( dxdz || (D())g(Z|x)) log logp(zz)dxc dxxdz
(25)
(x(x)a(Zix) log ((((x) p(z)
dxdz
(26)
RAVG
(27)
I,(: Z) AAVG
(28)
where the inequality in 24 follows from 20
The difference between the average KL and the mutual information is the aggregate KL:
/ʃ PD (x log g(z)
Rava I,(X;
dxdz
(29)
p(z)
/ q(z) log q(2) dz p(z)
(30)
D (g(zz)[((zz))
(Aggregate KL)
(31)
C.2 HIERARCHICALLY BOUNDING MUTUAL INFORMATION
The model with hierarchical latents shown in Figure C. gives us the following:
p( z)) = p(Zz. 2L) = p(zz Zz (
(32)
q x) 9(4.. 2 xx) q(z (x) (zn
(33)
17

---

Under review as a conference paper at ICLR 2020
KH
ZH
YT
ZL
ys
(z yr-
Figure C.I: Hierarchical decomposition of the latents. Shaded nodes indicate observed variables. [left] The true generative model. [right] Variational posterior that matches the form of the true posterior.
The conditional dependencies on yr and ys are omitted for compactness.
Define marginal aggregated posteriors:
g(ZL) = / pD (x())
(34)
((2))= / a(z))g(an)z)) 2z)
(35)
We can write the average joint KL term and mutual information as follows:
Rava /
(36)
/ [ntt:)/DDolltaaaaab//////////aad
1x(:; [Zn, Zr.)
(37)
Next we show that 1,(X; IZH ZL)) = (X(:
[/] PD ()))(((H. 2x log ((4n ) dxdzzdzz
(X; IZA Z.])
(38)
((2 (2) (z(1)
|/] pn(x)q(Zn 21 log ( x)
( dxd2u1R.
(39)
|| (D(x)g(ZLIx) log g(2) (x)
dxdzı.
(40)
g(z).)
(X;
(41)
Bound (X; Z.):
T,(X; Z1i ZT.) Ix(X;
(42)
(X; [ZH, Zu]) < RAVG
(43)
(X; Zr. RAVG
(44)
where (3) was shown in eq. 28
18

---

Under review as a conference paper at ICLR 2020
Again, using the non-negativity of the KL, can bound (Z ZL):
/ʃ (ZznlZ))((z) log ((H122)) dzudzz
(Z ZL)
(45)
q(zH)
// ((ZH(ZL)q(Z)) log 9 (Z dzzdzz
(46)
p(zH)
//] p log p(zu)
g(zn (1.) dzndzz,d dx
(47)
(48)
Rg
(49)
1,(X; 7n) - (Z,.; Zn)
(50)
1g (X; ZH) - Ri
(51)
where (50) can be demonstrated by applying the data processing inequality reversed version of the Markov chain, X + Z - Zμ
Define RL:
RL R - Ru
(52)
/ʃ [(2x (z)) log <<<,)x) dzndzz
(53)
p(zL|zH)
Giving the following bounds Lx; ZL. and (X(; ZH):
[x( ZH) & Rnr [X(: ZL) & DAYG + R
(54)
19


Reviews:
[{"review_id": "Hkg7mpxQ9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work authors present a regularized, variational autoencoder method for speech synthesis. To endow the latent space with more capacity, the authors employ a modified variational autoencoder objective, which uses a learnable Lagrange multiplier to impose a capacity limit on KL divergence between latent posterior and prior. The authors furthermore propose to decompose the latent embedding space into a two-level hierarchical representation to give generative process more control over style transfer and sample-to-sample variance. They extend earlier theoretical results providing upper bounds on the mutual information between data and its latent embedding to their hierarchical latent representation. In numerical experiments the authors evaluate their approach on a number of speech synthesis tasks involving same-text prosody transfer, inter-text style transfer, inter-speaker prosody transfer. They also analyze speech samples generated from latent samples drawn from the prior.\n\nThe paper is well-written and easy to follow, but the significance of the main contribution of the paper remains unclear to me. The authors propose to use an augmented standard VAE loss with a capacity hyperparameter and a Lagrange multiplier, but such modifications have been used before and it is not clear to me where is the novelty in there?\n\nMoreover, the authors treat the Lagrange multiplier as a learnable parameter, which brings up the question how does it effect the learning dynamics. For instance if the KL-divergence reaches its capacity, the Lagrange multiplier may be pushed down to zero, which in turn can allow the posterior to diverge unchecked to possibly a point mass distribution? The authors provide no details on how beta (the Lagrange multiplier) evolves in their experience and how it effects the stochastic nature of the model. \n\nI am not sure why the authors call non-stochastic latent variable models heuristic (instead of deterministic) methods?\n\nIn Figure 1, how are  the loss values computed for variational methods? Can we see how the error bars look for different C values and embedding dimensions?\n\nCan the authors be more clear about why for the tasks they consider, a standard (deep) VAE architecture with non-hierarchical latents does not sufficiently capture variations in the data? Can the authors quantify the differences? \n\nI am unfamiliar with prior work in this application area, but maybe the work is novel with respect to the application of the regularized VAE framework to speech synthesis. However, the application alone is in my opinion not a contribution that is significant enough for publication.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgtYejpYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new embedding method in expressive TTS by focusing on the capacity of hidden variables introduced in variational autoencoder. Such a method is supported by the KL divergence and lower bound theory and the paper well formulates/describes the capacity concept. In the experiments, the proposed method is compared with other conventional methods with well-designed subjective and objective metrics and shows the effectiveness in terms of the style transfer. The paper is well written in general.\n\nOne of the difficulties in this paper is reproducibility. The paper does not seem to provide source code and part of the data used in this paper also does not seem to be public data like libriTTS (I would be wrong but I could read that the multispeaker training data are not public), although I appreciate the authors' efforts to provide the implementation and evaluation details as much as possible in the appendix.\n\nAnother discussion of this paper is that it is not explicit to provide the meaning of the latent variables (this is a general issue in VAE) and I could not be fully convinced by the discussion and analysis based on the latent variable. There would be several semi-supervised studies to explicitly connect some (elements) of latent variables with actual attributes and I'd like to ask the authors to consider such direction to make the latent variable discussion in the paper more plausible.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJxsoyDqYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. Summary: This paper proposes Capacitron, a conditional variational latent variable model for TTS which allow for controllable latent variable capacity. They optimize the Lagrangian dual of the ELBO and restrict the capacity of the rate-term through a learnable, non-negative multiplier. They demonstrate the effectiveness of their approach on a range of TTS tasks such as same-text prosody transfer and inter-text style transfer, and provide extensive analyses on their latent variable capacity (in addition to comparisons to non-variational approaches based on Tacotron).\n\n2. Decision: Weak accept. I recommend this paper for acceptance due to its strong empirical results and clear presentation of the approach/unification of existing methods.\n\n3. Supporting arguments: The extension of Capacitron to existing methods such as [Hsu et al., 2019 and Zhang et al., 2019] is simple (basically adopting the beta-VAE approach), as the conditional generative model in both the vanilla and hierarchical forms exist already. But the authors do a thorough job evaluating the strengths and weaknesses of their method through ablation studies on latent variable capacity and comparing to existing baselines in their experiments. The results are also convincing, as demonstrated through human listening tests and the samples provided in the supplement. \n\n4. Feedback:\n- The authors mention in Section 3.1 that in previous work, the variational posterior has the form q(z|x). While this is true for [Zhang et. al, 2019], I believe that [Hsu et. al, 2019] also uses the conditional generative model as described in Figure 3 -- it would be helpful to provide a more clear distinction between the two works in the exposition. From what I understood, this work’s contribution is not so much the introduction of the conditional generative model but identifying the effects of controlling the rate term in the ELBO decomposition.\n- In Section 3.2, there is a lot of notation and several terms that make it difficult to parse Eq. 14 at first glance. For example, (1) R_L is never explicitly written out, (2) and is R == R_avg? It would be helpful to clean up this section so that eyeballing Eq. 14 would be easier for the reader (e.g. having all the relevant terms organized).\n\n5. Questions:\n- In the “Single speaker” section of Section 4.3, you mentioned that the “model has to divide the latent space into regions that correspond to different utterance lengths.” I’m curious -- is this something that was observed empirically?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates variational models of speech for synthesis, and in particular ways of making them more controllable for a variety of synthesis tasks (e.g. prosody transfer, style transfer).  They propose to do this via a modified VAE objective that imposes a learnable weight on the KL term, as well as using a hierarchical decomposition of latent variables.  The paper shows promising results and includes a good amount of analysis, and should be very interesting for speech synthesis researchers.  However, there is not much novelty from a machine learning perspective.  Therefore, I think the paper is not a great fit for ICLR and is better suited for a speech conference/journal.
Decision: Reject