Title: HIERARCHICAL GENERATIVE MODELING FOR CONTROLLABLE SPEECH SYNTHESIS

Abstract: This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model based on the variational autoencoder (VAE) framework, with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, we train a high-quality controllable TTS model on real found data, which is capable of inferring speaker and style attributes from a noisy utterance and use it to synthesize clean speech with controllable speaking style.

1. INTRODUCTION:

Recent development of neural sequence-to-sequence TTS models has shown promising results in generating high fidelity speech without the need of handcrafted linguistic features (Sotelo et al., 2017; Wang et al., 2017; Arık et al., 2017; Shen et al., 2018) . These models rely heavily on a encoderdecoder neural network structure (Sutskever et al., 2014; Bahdanau et al., 2015) that maps a text sequence to a sequence of speech frames. Extensions to these models have shown that attributes such as speaker identity can be controlled by conditioning the decoder on additional attribute labels (Arik et al., 2017; 2018; Jia et al., 2018) .

There are many speech attributes aside from speaker identity that are difficult to annotate, such as speaking style, prosody, recording channel, and noise levels. Skerry-Ryan et al. (2018) ; Wang et al. (2018) model such latent attributes through conditional auto-encoding, by extending the decoder inputs to include a vector inferred from the target speech which aims to capture the residual attributes that are not specified by other input streams, in addition to text and a speaker label. These models have shown convincing results in synthesizing speech that resembles the prosody or the noise conditions of the reference speech, which may not have the same text or speaker identity as the target speech.

Nevertheless, the presence of multiple latent attributes is common in crowdsourced data such as (Panayotov et al., 2015) , in which prosody, speaker, and noise conditions all vary simultaneously. Using such data, simply copying the latent attributes from a reference is insufficient if one desires to synthesize speech that mimics the prosody of the reference, but is in the same noise condition as another. If the latent representation were disentangled, these generating factors could be controlled independently. Furthermore, it is can useful to construct a systematic method for synthesizing speech with random latent attributes, which would facilitate data augmentation (Tjandra et al., 2017; 2018; Hsu et al., 2017b; 2018; Hayashi et al., 2018) by generating diverse examples. These properties were not explicitly addressed in the previous studies, which model variation of a single latent attribute.

Motivated by the applications of sampling, inferring, and independently controlling individual attributes, we build off of Skerry-Ryan et al. (2018) and extend Tacotron 2 (Shen et al., 2018) to model two separate latent spaces: one for labeled (i.e. related to speaker identity) and another for unlabeled attributes. Each latent variable is modeled in a variational autoencoding (Kingma & Welling, 2014) framework using Gaussian mixture priors. The resulting latent spaces (1) learn disentangled attribute representations, where each dimension controls a different generating factor; (2) discover a set of interpretable clusters, each of which corresponds to a representative mode in the training data (e.g., one cluster for clean speech and another for noisy speech); and (3) provide a systematic sampling mechanism from the learned prior. The proposed model is extensively evaluated on four datasets with subjective and objective quantitative metrics, as well as comprehensive qualitative studies. Experiments confirm that the proposed model is capable of controlling speaker, noise, and style independently, even when variation of all attributes is present but unannotated in the train set.

Our main contributions are as follows:

• We propose a principled probabilistic hierarchical generative model, which improves (1) sampling stability and disentangled attribute control compared to e.g. the GST model of Wang et al. (2018) , and (2) interpretability and quality compared to e.g. Akuzawa et al. (2018) . • The model formulation explicitly factors the latent encoding by using two mixture distributions to separately model supervised speaker attributes and latent attributes in a disentangled fashion. This makes it straightforward to condition the model output on speaker and latent encodings inferred from different reference utterances. • To the best of our knowledge, this work is the first to train a high-quality controllable textto-speech system on real found data containing significant variation in recording condition, speaker identity, as well as prosody and style. Previous results on similar data focused on speaker modeling (Ping et al., 2018; Nachmani et al., 2018; Arik et al., 2018; Jia et al., 2018) , and did not explicitly address modeling of prosody and background noise.

Leveraging disentangled speaker and latent attribute encodings, the proposed model is capable of inferring the speaker attribute representation from a noisy utterance spoken by a previously unseen speaker, and using it to synthesize high-quality clean speech that approximates the voice of that speaker.

2. MODEL:

Tacotron-like TTS systems take a text sequence Y t and an optional observed categorical label (e.g. speaker identity) y o as input, and use an autoregressive decoder to predict a sequence of acoustic features X frame by frame. Training such a system to minimize a mean squared error reconstruction loss can be regarded as fitting a probabilistic model p(X | Y t , y o ) = n p(x n | x 1 , x 2 , . . . , x n-1 , Y t , y o ) that maximizes the likelihood of generating the training data, where the conditional distribution of each frame x n is modeled as fixed-variance isotropic Gaussian whose mean is predicted by the decoder at step n. Such a model effectively integrates out other unlabeled latent attributes like prosody, and produces a conditional distribution with higher variance. As a result, the model would opaquely produce speech with unpredictable latent attributes. To enable control of those attributes, we adopt a graphical model with hierarchical latent variables, which captures such attributes. Below we explain how the formulation leads to interpretability and disentanglement, supports sampling, and propose efficient inference and training methods.

2.1. CONDITIONAL GENERATIVE MODEL WITH HIERARCHICAL LATENT VARIABLES:

Two latent variables y l and z l are introduced in addition to the observed variables, X, Y t , and y o , as shown in the graphical model in the left of Figure 1 . y l is a K-way categorical discrete variable, named latent attribute class, and z l is a D-dimensional continuous variable, named latent attribute representation. Throughout the paper, we use y * and z * to denote discrete and continuous variables, respectively. To generate speech X conditioned on the text Y t and observed attribute y o , y l is first sampled from its prior, p(y l ), then a latent attribute representation z l is sampled from the conditional distribution p(z l | y l ). Finally, a sequence of speech frames is drawn from p(X | Y t , y o , z l ), parameterized by the synthesizer neural network. The joint probability can be written as: Specifically, it is assumed that p(y l ) = K -1 to be a non-informative prior to encourage every component to be used, and p(z l | y l ) = N (µ y l , diag(σ y l )) to be diagonal-covariance Gaussian with learnable means and variances. As a result, the marginal prior of z l becomes a GMM with diagonal covariances and equal mixture weights. We hope this GMM latent model can better capture the complexity of unseen attributes. Furthermore, in the presence of natural clusters of unseen attributes, the proposed model can achieve interpretability by learning to assign instances from different clusters to different mixture components. The covariance matrix of each mixture component is constrained to be diagonal to encourage each dimension to capture a statistically uncorrelated factor.

EQUATION

)

2.2. VARIATIONAL INFERENCE AND TRAINING:

The conditional output distribution p(X | Y t , y o , z l ) is parameterized with a neural network. Following the VAE framework of Kingma & Welling (2014), a variational distribution q(y l | X) q(z l | X) is used to approximate the posterior p(y l , z l | X, Y t , y o ), which assumes that the posterior of unseen attributes is independent of the text and observed attributes. The approximated posterior for z l , q(z l | X), is modeled as a Gaussian distribution with diagonal covariance matrix, whose mean and variance are parameterized by a neural network. For q(y l | X), instead of introducing another neural network, we configure it to be an approximation of p(y l | X) that reuses q(z l | X) as follows:

p(y l |X) = z l p(y l | z l ) p(z l |X) dz l = E p(z l |X) [p(y l | z l )] ≈ E q(z l |X) [p(y l | z l )] := q(y l |X) (2)

which enjoys the closed-form solution of Gaussian mixture posteriors, p(y l | z l ). Similar to VAE, the model is trained by maximizing its evidence lower bound (ELBO), as follows:

L(p, q; X, Y t , y o ) = E q(z l |X) [log p(X | Y t , y o , z l )] -E q(y l |X) [D KL (q(z l | X) || p(z l | y l ))] -D KL (q(y l | X) || p(y l )) (3)

where q(z l | X) is estimated via Monte Carlo sampling, and all components are differentiable thanks to reparameterization. Details can be found in Appendix A.

2.3. A CONTINUOUS ATTRIBUTE SPACE FOR CATEGORICAL OBSERVED LABELS:

Categorical observed labels, such as speaker identity, can often be seen as a categorization from a continuous attribute space, which for example could model a speaker's characteristic F 0 range and vocal tract shape. Given an observed label, there may still be some variation of these attributes. We are interested in learning this continuous attribute space for modeling within-class variation and inferring a representation from an instance of an unseen class for one-shot learning.

To achieve this, a continuous latent variable, z o , named the observed attribute representation, is introduced between the categorical observed label y o and speech X, as shown on the right of Figure 1 . The model is comprised of three modules: a synthesizer, a latent encoder, and an observed encoder.

L o (p, q; X, Y t , y o ) = E q(zo|X)q(z l |X) [log p(X | Y t , z o , z l )] -D KL (q(z o | X) || p(z o | y o )) -E q(y l |X) [D KL (q(z l | X) || p(z l | y l ))] -D KL (q(y l | X) || p(y l )). (4)

To encourage z o to disentangle observed attributes from latent attributes, the variances of p(z o | y o ) are initialized to be smaller than those of p(z l | y l ). The intuition is that this space should capture variation of attributes that are highly correlated with the observed labels, so the conditional distribution of all dimensions should have relatively small variance for each mixture component. Experimental results verify the effectiveness, and similar design is used in Hsu et al. (2017a) . In the extreme case where the variance is fixed and approaches zero, this formulation converges to using an lookup table.

2.4. NEURAL NETWORK ARCHITECTURE:

We parameterize three distributions:

p(X|Y t , z o , z l ) (or p(X | Y t , z o , z l )), q(z o |X)

, and q(z l |X) with neural networks, referred to in Figure 2 as the synthesizer, observed encoder, and latent encoder, respectively. The synthesizer is based on the Tacotron 2 architecture (Shen et al., 2018) , which consists of a text encoder and an autoregressive speech decoder. The former maps Y t to a sequence of text encodings Z t , and the latter predicts the mean of p(

x n | x 1 , . . . , x n-1 , Z t , y o , z l ) at each step n.

We inject the latent variables z l and y o (or z o ) into the decoder by concatenating them to the decoder input at each step. Text Y t and speech X are represented as a sequence of phonemes and a sequence of mel-scale filterbank coefficients, respectively. To speed up inference, we use a WaveRNN-based neural vocoder (Kalchbrenner et al., 2018) instead of WaveNet (van den Oord et al., 2016) , to invert the predicted mel-spectrogram to a time-domain waveform.

The two posteriors, q(z l | X) and q(z o | X), are both parameterized by a recurrent encoder that maps a variable-length mel-spectrogram to two fixed-dimensional vectors, corresponding to the posterior mean and log variance, respectively. Full architecture details can be found in Appendix B.

3. RELATED WORK:

The proposed GMVAE-Tacotron model is most related to Skerry-Ryan et al. (2018) , Wang et al. (2018) , Henter et al. (2018) , which introduce a reference embedding to model prosody or noise.

The first uses an autoencoder to extract a prosody embedding from a reference speech spectrogram.

The second Global Style Token (GST) model constrains a reference embedding to be a weighted combination of a fixed set of learned vectors, while the third further restricts the weights to be one-hot, and is built on a conventional parametric speech synthesizer (Zen et al., 2009) . The main focus of these approaches was style transfer from a reference audio example. They provide neither a systematic sampling mechanism nor disentangled representations as we show in Section 4.3.1.

Similar to these approaches, Akuzawa et al. (2018) extend VoiceLoop (Taigman et al., 2018) with a latent reference embedding generated by a VAE, using a centered fixed-variance isotropic Gaussian prior for the latent attributes. This provides a principled mechanism for sampling from the latent distribution, but does not provide interpretability. In contrast, GMVAE-Tacotron models latent attributes using a mixture distribution, which allows automatic discovery of latent attribute clusters. This structure makes it easier to interpret the underlying latent space. Specifically, we show in Section 4.1 that the mixture parameters can be analyzed to understand what each component corresponds to, similar to GST. In addition, the most distinctive dimensions of the latent space can be identified using an inter-/intra-component variance ratio, which e.g. can identify the dimension controlling the background noise level as shown in Section 4.2.2.

Finally, the extension described in Section 2.3 adds a second mixture distribution to additionally models speaker attributes. This formulation learns disentangled speaker and latent attribute represen-tations, which can be used to approximate the voice of speakers previously unseen during training. This speaker model is related to Arik et al. (2018) , which controls the output speaker identity using speaker embeddings, and trains a separate regression model to predict them from the audio. This can be regarded as a special case of the proposed model where the variance of z o is set to be almost zero, such that a speaker always generates a fixed representation; meanwhile, the posterior model q(z o | X) corresponds to their embedding predictor, because it now aims to predict a fixed embedding for each speaker.

Using a mixture distribution for latent variables in a VAE was explored in Dilokthanakul et al. (2016) ; Nalisnick et al. (2016), and Jiang et al. (2017) for unconditional image generation and text topic modeling. These models correspond to the sub-graph y l → z l → X in Figure 1 

4. EXPERIMENTS:

The proposed GMVAE-Tacotron was evaluated on four datasets, spanning a wide degree of variations in speaker, recording channel conditions, background noise, prosody, and speaking styles. For all experiments, y o was an observed categorical variable whose cardinality is the number of speakers in the training set if used, y l was configured to be a 10-way categorical variable (K = 10), and z l and z o (if used) were configured to be 16-dimensional variables (D = 16). Tacotron 2 (Shen et al., 2018) with a speaker embedding table was used as the baseline for all experiments. For all other variants (e.g., GST), the reference encoder follows Wang et al. (2018) . Each model was trained for at least 200k steps to maximize the ELBO in equation 3 or equation 4 using the Adam optimizer. A list of detailed hyperparameter settings can be found in Appendix C. Quantitative subjective evaluations relied on crowd-sourced mean opinion scores (MOS) rating the naturalness of the synthesized speech by native speakers using headphones, with scores ranging from 1 to 5 in increments of 0.5. For single speaker datasets each sample was rated by 6 raters, while for other datasets each sample was rated by a single rater. We strongly encourage readers to listen to the samples on the demo page.1 

4.1. MULTI-SPEAKER ENGLISH CORPUS:

To evaluate the ability of GMVAE-Tacotron to model speaker variation and discover meaningful speaker clusters, we used a proprietary dataset of 385 hours of high-quality English speech from 84 professional voice talents with accents from the United States (US), Great Britain (GB), Australia (AU), and Singapore (SG). Speaker labels were not seen during training (y o and z o were unused), and were only used for evaluation.

Figure 3 : Assignment distribution over y l for each gender (upper) and for each accent (lower).

To probe the interpretability of the model, we computed the distribution of mixture components y l for utterances of a particular accent or gender. Specifically, we collected at most 100 utterances from each of the 44 speakers with at least 20 test utterances (2,332 in total), and assigned each utterance to the component with the highest posterior probability: arg max y l q(y l |X). We also explored what each dimension of z l controlled by decoding with different values of the target dimension, keeping all other factors fixed. We discovered that there were individual dimensions which controlled F 0 , speaking rate, accent, length of starting silence, etc., demonstrating the disentangled nature of the learned latent attribute representation. Appendix E contains visualization of attribute control and additional quantitative evaluation of using z l for gender/accent/speaker classification.

4.2. NOISY MULTI-SPEAKER ENGLISH CORPUS:

High quality data can be both expensive and time consuming to record. Vast amounts of rich real-life expressive speech are often noisy and difficult to label. In this section we demonstrate that our model can synthesize clean speech directly from noisy data by disentangling the background noise level from other attributes, allowing it to be controlled independently. As a first experiment, we artificially generated training sets using a room simulator (Kim et al., 2017) to add background noise and reverberation to clean speech from the multi-speaker English corpus used in the previous section. We used music and ambient noise sampled from YouTube and recordings of "daily life" environments as noise signals, mixed at signal-to-noise ratios (SNRs) ranging from 5-25dB. The reverberation time varied between 100 and 900ms. Noise was added to a random selection of 50% of utterances by each speaker, holding out two speakers (one male and one female) for whom noise was added to all of their utterances. This construction was used to evaluate the ability of the model to synthesize clean speech for speakers whose training utterances were all corrupted by noise. In this experiment, we provided speaker labels y o as input to the decoder, and only expect the latent attribute representations z l to capture the acoustic condition of each utterance.

4.2.1. IDENTIFYING MIXTURE COMPONENTS THAT GENERATE CLEAN/NOISY SPEECH:

Unlike clustering speakers, we expected that latent attributes would naturally divide into two categories: clean and noisy. To verify this hypothesis, we plotted the Euclidean distance between means of each pair of components on the left of Figure 4 , which clearly form two distinct clusters. The right two plots in Figure 4 show the mel-spectrograms of two synthesized utterances of the same text and speaker, conditioned on the means of two different components, one from each group. It clearly presents the samples (in fact, all the samples) drawn from components in group one were noisy, while the samples drawn from the other components were clean. See Appendix F for more examples.

4.2.2. CONTROL OF THE BACKGROUND NOISE LEVEL:

We next explored if the level of noise was dominated by a single latent dimension, and whether we could determine such a dimension automatically. For this purpose, we adopted a per-dimension LDA, which computed a between and within-mixture scattering ratio:

r d = K y l =1 p(y l )(µ y l ,d -μl,d ) 2 / K y l =1 p(y l )σ 2 y l ,d

, where µ y l ,d and σ y l ,d are the d-th dimension mean and variance of mixture component y l , and μl,d is the d-th dimension mean of the marginal distribution p(z l ) = K y l =1 p(y l ) p(z l | y l ). This is a scale-invariant metric of the degree of separation between components in each latent dimension.

We discovered that the most discriminative dimension had a scattering ratio r 13 = 21.5, far larger than the second largest r 11 = 0.6. Drawing samples and traversing values along the target dimension while keeping others fixed demonstrates that dimension's effect on the output. To determine the effective range of the target dimension, we approximate the multimodal distribution as a Gaussian and evaluate values spanning four standard deviations σl,d around the mean. To quantify the effect on noise level, we estimate the SNR without a reference clean signal following Kim & Stern (2008) .

Figure 5 plots the average estimated SNR over 200 utterances from two speakers as a function of the value in each latent dimension, where values for other dimensions are the mean of a clean component (left) or that of a noisy component (right). The results show that the noise level was clearly controlled by manipulating the 13th dimension, and remains nearly constant as the other dimensions vary, verifying that control has been isolated to the identified dimension. The small degree of variation, e.g. in dimensions 2 and 4, occurs because some of those dimensions control attributes which directly affect the synthesized noise, such as type of noise (musical/white noise) and initial background noise offset, and therefore also affect the estimated SNR. Appendix F.2 contains an additional spectrogram demonstration of noise level control by manipulating the identified dimension.

4.2.3. SYNTHESIZING CLEAN SPEECH FOR NOISY SPEAKERS:

In this section, we evaluated synthesis quality for the two held out noisy speakers. Evaluation metrics included subjective naturalness MOS ratings and an objective SNR metric. Table 1 compares the proposed model with a baseline, a 16-token GST, and a VAE variant which replaces the GMM prior with an isotropic Gaussian. To encourage synthesis of clean audio under each model we manually selected the cleanest token (weight=0.15) for GST, used the Gaussian prior mean (i.e. a zero vector) for VAE, and the mean of a clean component for GMVAE. For the VAE model, the mean captured the average condition, which still exhibited a moderate level of noise, resulting in a lower SNR and MOS. The generated speech from the GST was cleaner, however raters sometimes found its prosody to be unnatural. Note that it is possible that another token would obtain a different trade-off between prosody and SNR, and using multiple tokens could improve both. Finally, the proposed model synthesized both natural and high-quality speech, with the highest MOS and SNR.

4.3. SINGLE-SPEAKER AUDIOBOOK CORPUS:

Prosody and speaking style is another important factor for human speech other than speaker and noise. Control of these aspects of the synthesize speech is essential to building an expressive TTS system. In this section, we evaluated the ability of the proposed model to sample and control speaking styles. A single speaker US English audiobook dataset of 147 hours, recorded by professional speaker, Catherine Byers, from the 2013 Blizzard Challenge (King & Karaiskos, 2013) is used for training. The data incorporated a wide range of prosody variation. We used an evaluation set of 150 audiobook sentences, including many long phrases. Table 2 shows the naturalness MOS between baseline and proposed model conditioning on the same z l , set to the mean of a selected y l , for all utterances. The results show that the prior already captured a common prosody, which could be used to synthesize more naturally sounding speech with a lower variance compared to the baseline. F 0 tracks and spectrograms from GMVAE-Tacotron using different values for the "speed" dimension.

4.3.1. STYLE SAMPLING AND DISENTANGLED CONTROL:

Compared to GST, one primary advantage of the proposed model is that it supports random sampling of natural speech from the prior. Figure 6 illustrates such samples, where the same text is synthesized with wide variation in speaking rate, rhythm, and F 0 . In contrast, the GST model does not define a prior for normalized token weights, requiring weights to be chosen heuristically or by fitting a distribution after training. Empirically we found that the GST weight simplex was not fully exploited during training and that careful tuning was required to find a stable sampling region.

An additional advantage of GMVAE-Tacotron is that it learns a representation which disentangles these attributes, enabling them to be controlled independently. Specifically, latent dimensions in the proposed model are conditionally independent, while token weights of GST are in fact correlated. , where the F 0 also changed while controlling speed. Appendix G contains a quantitative analysis of disentangled latent attribute control, and additional evaluation of style transfer, demonstrating the ability of the proposed the model to synthesize speech that resembles the prosody of a reference utterance.

4.4. CROWD-SOURCED AUDIOBOOK CORPUS:

We used an audiobook dataset2 derived from the same subset of LibriVox audiobooks used for the LibriSpeech corpus (Panayotov et al., 2015) , but sampled at 24kHz and segmented differently, making it appropriate for TTS instead of speech recognition. The corpus contains recordings from thousands of speakers, with wide variation in recording conditions and speaking style. Speaker identity is often highly correlated with the recording channel and background noise level, since many speakers tended to use the same microphone in a consistent recording environment. The ability to disentangle and control these attributes independently is essential to synthesizing high-quality speech for all speakers. We demonstrate the ability of GMVAE-Tacotron to consistently generate high-quality speech by conditioning on a value of z l associated with clean output. We considered two approaches: (1) using the mean of the identified clean component, which can be seen as a preset configuration with a fixed channel and style;

(2) inferring a latent attribute representation z l from reference speech and denoising it by modifying dimensions3 associated with the noise level to predetermined values.

We evaluated a set of eight "seen clean" (SC) speakers and a set of nine "seen noisy" (SN) speakers from the training set, a set of ten "unseen noisy" (UN) speakers from a held-out set with no overlapping speakers, and the set of ten unseen speakers used in Jia et al. (2018) , denoted as "unseen clean" (UC).

For consistency, we always used an inferred z o from an utterance from the target speaker, regardless of whether that speaker was seen or unseen. As a baseline we used a Tacotron model conditioned on a 128-dimensional speaker embedding learned for each speaker seen during training.

Table 3 shows the SNR of the original audio, audio synthesized by the baseline, and by the GMVAE-Tacotron using the two proposed approaches, denoted as mean and latent-dn, respectively, on all speaker sets whenever possible. In addition, to see the effectiveness of the denoising operation, the table also includes the results of using inferred z l directly, denoted as latent. The results show that the inferred z l followed the same SNR trend as the original audio, indicating that z l captured the variation in acoustic condition. The high SNR values of mean and latent-dn verifies the effectiveness of using a preset and denoising arbitrary inferred latent features, both of which outperformed the baseline by a large margin, and produced better quality than the original noisy audio.

Table 4 compares the proposed model using denoised z l to the baseline in a subjective side-byside preference test. Table 5 further compares subjective naturalness MOS of the proposed model using the mean of the clean component to the baseline on the two seen speaker sets, and to the d-vector model (Jia et al., 2018) on the two unseen speaker sets. Specifically, we consider another stronger baseline model to compare on the SN set, which is trained on denoised data using spectral subtraction (Boll, 1979) , denoted as "+ denoise." Both results indicate that raters preferred the proposed model to the baselines. Moreover, the MOS evaluation shows that the proposed model delivered similar level of naturalness under all conditions, seen or unseen, clean or noisy. We evaluate whether the synthesized speech resembles the identity of the reference speaker, by pairing each synthesized utterance with the reference utterance for subjective MOS evaluation of speaker similarity, following Jia et al. (2018) .

Table 6 compares the proposed model using denoised latent attribute representations to baseline systems on the two seen speaker sets, and to d-vector systems on the unseen clean speaker set. The d-vector systems used a separately trained speaker encoder model to extract speaker representations for TTS conditioning as in Jia et al. (2018) . We considered two speaker encoder models, one trained on the same train-clean partition as the proposed model, and another trained on a larger scale dataset containing 18K speakers. We denote these two systems as d-vector and d-vector (large).

On the seen clean speaker set, the proposed model achieved similar speaker similarity scores to the baseline. However, on the seen noisy speaker set, both the proposed model and the baseline trained on denoised speech performed significantly worse than the baseline. We hypothesize that similarity of the acoustic conditions between the paired utterances biased the speaker similarity ratings. To confirm this hypothesis, we additionally evaluated speaker similarity of the ground truth utterances from a speaker whose recordings contained significant variation in acoustic conditions. As shown in Table 6 , these ground truth utterances were also rated with a significantly lower MOS than the baseline, but were close to the proposed model and the denoised baseline. This result implies that this subjective speaker similarity test may not be reliable in the presence of noise and channel variation, requiring additional work to design a speaker similarity test that is unbiased to such nuisance factors.

Finally, on the unseen clean speaker set, the proposed model achieved significantly better speaker similarity scores than the d-vector system whose speaker representation extractor was trained on the same set as the proposed model, but worse than the d-vector (large) system which was trained on over 15 times more speakers. However, we emphasize that: (1) this is not a fair comparison as the two models are trained on datasets of different sizes, and (2) our proposed model is complementary to d-vector systems. Incorporating the high quality speaker transfer from the d-vector model with the strong controllability of the GMVAE is a promising direction for future work.

5. CONCLUSION:

We describe GMVAE-Tacotron, a TTS model which learns an interpretable and disentangled latent representation to enable fine-grained control of latent attributes and provides a systematic sampling scheme for them. If speaker labels are available, we demonstrate an extension of the model that learns a continuous space that captures speaker attributes, along with an inference model which enables one-shot learning of speaker attributes from unseen reference utterances.

The proposed model was extensively evaluated on tasks spanning a wide range of signal variation. We demonstrated that it can independently control many latent attributes, and is able to cluster them without supervision. In particular, we verified using both subjective and objective tests that the model could synthesize high-quality clean speech for a target speaker even if the quality of data for that speaker does not meet high standard. These experimental results demonstrated the effectiveness of the model for training high-quality controllable TTS systems on large scale training data with rich styles by learning to factorize and independently control latent attributes underlying the speech signal.

A DERIVATION OF REPARAMETERIZED TRAINING OBJECTIVES:

This section gives detailed derivation of the evidence lower bound (ELBO) estimation used for training. We first present a differentiable Monte Carlo estimation of the posterior q(y l | X), and then derive an ELBO for each of the graphical models in Figure 1 , which differ in whether an additional observed attribute representation z o is used.

A.1 MONTE CARLO ESTIMATION OF THE REPARAMETERZIED CATEGORICAL POSTERIOR:

As shown in equation 2, we approximate the posterior over latent attribute class y l with

EQUATION

where q(z l | X) is a diagonal-covariance Gaussian, and p(y l | z l ) is the probability of z l being drawn from the y l -th Gaussian mixture component. We first denote the mean vector and the diagonal elements of the covariance matrix of the y l -th component as µ l,y l and σ 2 l,y l , and write the posterior over mixture components given a latent attribute representation, p(y l | z l ):

EQUATION

with

EQUATION

)

where D is the dimensionality of z l , and K is the number of classes for y l .

Finally, we denote the posterior mean and variance of q(z l | X) by μl and σ2 l , and compute a Monte Carlo estimate of the expectation in equation 5 after reparameterization:

EQUATION

where z(n) l = μl + σl (n) is a random sample, drawn from a standard Gaussian distribution using (n) ∼ N (0, I), and N is the number of samples used for the Monte Carlo estimation. The resulting estimate q(y l | X) is differentiable w.r.t. the parameters of p(z l | y l ) and q(z l | X).

A.2 DIFFERENTIABLE TRAINING OBJECTIVE:

We next derive the ELBO L(p, q; X, Y t , y o ) and rewrite it as a Monte Carlo estimate used for training: n ) ∼ N (0, I), and L(p, q; X, Y t , y o ) is the estimator used for training. Similarly, N is the number of samples used for the Monte Carlo estimate.

EQUATION

EQUATION

EQUATION

A.3 DIFFERENTIABLE TRAINING OBJECTIVE WITH OBSERVED ATTRIBUTE REPRESENTATION:

In this section, we derive the ELBO L o (p, q; X, Y t , y o ) when using an additional observed attribute representation, z o , as described in Section 2.3, and rewrite it with a Monte Carlo estimation used for training. As before, we denote the posterior mean and variance of q(z o | X) by μo and σ2 o .

EQUATION

:= Lo (p, q; X, y y , y o ),

where the continuous latent variables are reparameterized as z(n

) o = μo + σo (n ) o and z(n ) l = μl + σl (n ) l

, with auxiliary noise variables

(n ) o , (n ) l

∼ N (0, I). The estimator Lo (p, q; X, Y t , y o ) is used for training. N and N are the numbers of samples used for the Monte Carlo estimate.

B NEURAL NETWORK ARCHITECTURE DETAILS B.1 SYNTHESIZER:

The synthesizer is an attention-based sequence-to-sequence network which generates a mel spectrogram as a function of an input text sequence and conditioning signal generated by the auxiliary encoder networks. It closely follows the network architecture of Tacotron 2 (Shen et al., 2018) . The input text sequence is encoded by three convolutional layers, which contains 512 filters with shape 5 × 1, followed by a bidirectional long short-term memory (LSTM) of 256 units for each direction.

The resulting text encodings are accessed by the decoder through a location sensitive attention mechanism (Chorowski et al., 2015) , which takes attention history into account when computing a normalized weight vector for aggregation.

The base Tacotron 2 autoregressive decoder network takes as input the attention-aggregated text encoding, and the bottlenecked previous frame (processed by a pre-net comprised of two fullyconnected layers of 256 units) at each step. In this work, to condition the output on additional attribute representations, the decoder is extended to consume z l and z o (or y o ) by concatenating them with the original decoder input at each step. The concatenated vector forms the new decoder input, which is passed through a stack of two uni-directional LSTM layers with 1024 units. The output from the stacked LSTM is concatenated with the new decoder input (as a residual connection), and linearly projected to predict the mel spectrum of the current frame, as well as an end-of-sentence token. Finally, the predicted spectrogram frames are passed to a post-net, which predicts a residual that is added to the initial decoded sequence of spectrogram frames, to better model detail in the spectrogram and reduce the overall mean squared error.

Similar to Tacotron 2, we separately train a neural vocoder to invert a mel spectrograms to a timedomain waveform. In contrast to that work, we replace the WaveNet (van den Oord et al., 2016) vocoder with one based on the recently proposed WaveRNN (Kalchbrenner et al., 2018) architecture, which is more efficient during inference.

B.2 LATENT ENCODER AND OBSERVED ENCODER:

Both the latent encoder and the observed encoder map a mel spectrogram from a reference speech utterance to two vectors of the same dimension, representing the posterior mean and log variance of the corresponding latent variable. We design both encoders to have exactly the same architecture, whose outputs are conditioned by the decoder in a symmetric way. Disentangling of latent attributes and observed attributes is therefore achieved by optimizing different KL-divergence objectives.

For each encoder, a mel spectrogram is first passed through two convolutional layers, which contains 512 filters with shape 3 × 1. The output of these convolutional layers is then fed to a stack of two bidirectional LSTM layers with 256 cells at each direction. A mean pooling layer is used to summarize the LSTM outputs across time, followed by a linear projection layer to predict the posterior mean and log variance.

C DETAILED EXPERIMENTAL SETUP:

The network is trained using the Adam optimizer (Kingma & Ba, 2015) , configured with an initial learning rate 10 -3 , and an exponential decay that halved the learning rate every 12.5k steps, beginning after 50k steps. Parameters of the network are initialized using Xavier initialization (Glorot & Bengio, 2010) . A batch size of 256 is used for all experiments. Following the common practice in the VAE literature (Kingma & Welling, 2014), we set the number of samples used for the Monte Carlo estimate to 1, since we train the model with a large batch size.

Table 7 details the list of prior hyperparameters used for each of the four datasets described in Section 4: multi-speaker English data (multi-spk), noisified multi-speaker English data (noisy-multispk), single-speaker story-telling data (audiobooks), and crowd-sourced audiobook data (crowdsourced). To ensure numerical stability we set a minimum value allowed for the standard deviation of the conditional distribution p(z l | y l ). We initially set the lower bound to e -1 ; however, with the exception of the multi-speaker English data, the trained standard deviation reached the lower bound for all mixture components for all dimensions. We therefore lowered the minimum standard deviation to e -2 , and found that it left sufficient range to capture the amount of variation.

As shown in Table 11 , we found that increasing the dimensionality of z l from 16 to 32 improves reconstruction quality; however, it also increases the difficulty of interpreting each dimension. On the other hand, reducing the dimensionality too much can result in insufficient modeling capacity for latent attributes, however we have not carefully explored this lower bound.

Empirically, we found 16-dimensional z l to be appropriate for capturing the salient attributes one would like to control in the four datasets we experimented with. When evaluating the meaning of each dimensions, we find the majority of the dimensions to be interpretable, and the number of dummy dimensions which do not affect the model output varied across datasets, as each of them inherently has variation across a different number of unlabeled attributes. For example, the model trained on the multi-speaker English corpus (Section 4.1) has four dummy dimensions of z l that do not affect the output. In contrast, for the model trained on the crowd-sourced audio book corpus (Section 4.4), which contains considerably more variation in style and prosody, we found only one dummy dimension of z l . 

e 0 e -1 e -1 e -1 minimum σ l e -1 e -2 e -2 e -2 dim(y o ) N/A 84 N/A 1,172 dim(z o ) N/A N/A N/A 16 initial σ o N/A N/A N/A e -2 minimum σ o N/A N/A N/A e -4

D POSTERIOR COLLAPSE:

There are two latent variables in our graphical model as shown in Figure 1 (left): the latent attribute class y l (discrete) and latent attribute representation z l (continuous). We discuss the potential for posterior collapse for each of them separately.

D.1 POSTERIOR COLLAPSE OF LATENT ATTRIBUTE REPRESENTATION:

The continuous latent variable z l is used to directly condition the generation of X, along with two other observed variables, y o and Y t . In our experiments, we observed that the latent variable z l is always used, i.e. the KL-divergence of z l never drops to zero, without applying any tricks such as KL-annealing (Bowman et al., 2016) .

Previous studies report posterior-collapse of directly conditioned latent variables when using strong models (e.g. auto-regressive networks) to parameterize the conditional distribution of text (Bowman et al., 2016; Yang et al., 2017; Kim et al., 2018) . This phenomenon arises from the competition between (1) increasing reconstruction performance by utilizing information provided by the latent variable, and (2) decreasing the KL-divergence by making the latent variable uninformative. Auto-regressive models are more likely to converge to the second case during training because the improvement in reconstruction from utilizing the latent variable can be smaller than the increase in KL-divergence.

However, this does not always happen, because the amount of improvement resulted from utilizing the information provided by the latent variable depends on the type of data. The reason that the posterior-collapse does not occur in our experiments is likely a consequence of the complexity of the speech sequence distribution, compared to text. Even though we use an auto-regressive decoder, 

E.2 QUANTITATIVE ANALYSIS OF THE US FEMALE COMPONENTS:

To quantify the difference between the three components that model US female speakers (3, 5, and 6), we draw 20 latent attribute encodings z l from each of the three components and decode the same set of 25 text sequences for each one. Table 8 shows the average F 0 computed over 500 synthesized utterances for each component, demonstrating that each component models a different F 0 range. To objectively evaluate the ability of the proposed GMVAE-Tacotron model to control individual attributes, we compute two metrics: (1) the average F 0 (fundamental frequency) in voiced frames, computed using YIN (De Cheveigné & Kawahara, 2002) , and (2) the average speech duration. These correspond to rough measures of the speaking rate and degree of pitch variation, respectively.

We randomly draw 10 samples of seed z l from the prior, deterministically set the target dimension to μl,d -3 σl,d , μl,d , and μl,d + 3 σl,d to construct modified z * l , where μl,d and σl,d are the mean and standard deviation of the marginal distribution of the target dimension d. We then synthesize a set of the same 25 text sequences for each of the 30 resulting values of z * l . For each value of the target dimension, we compute an average metric over 250 synthesized utterances (10 seed z l × 25 text inputs). Results are shown in Table 10 . For the "pitch" dimension, we can see that the measured F 0 varies substantially, while the speech remains constant. Similarly, as the value in the "speaking rate" dimension varies, the measured duration varies substantially. However, there is also a smaller inverse effect on the pitch, i.e. the pitch slightly increases with the speaking rate, which is consistent with natural speaking behavior (Apple et al., 1979; Black, 1961) . These results are an indication that manipulating individual dimensions primarily controls the corresponding attribute.

G.2 PARALLEL STYLE TRANSFER:

We evaluated the ability of the proposed model to synthesize speech that resembled the prosody or style of a given reference utterance, by conditioning on a latent attribute representation inferred from the reference. We adopted two metrics from Skerry-Ryan et al. ( 2018) to quantify style transfer performance: the mel-cepstral distortion(MCD 13 ), measuring the phonetic and timbral distortion, and F 0 frame error (FFE), which combines voicing decision error and F 0 error metrics to capture how well F 0 information, which encompasses much of the prosodic content, is retained. Both metrics assume that the generated speech and the reference speech are frame aligned. We therefore synthesized the same text content as the reference for this evaluation. (Vaswani et al., 2017) , it effectively has 60 degrees of freedom, which might explain why it performs better in replicating the reference style. By increasing the dimension of z l to 32 (D = 32), the gap to the GST model is greatly reduced. Note that the number of degrees of freedom of the latent space as well as the total number of parameters is still smaller than in the GST model. As noted in Skerry-Ryan et al. (2018) , the ability of a model to reconstruct the reference speech frame-by-frame is highly correlated with the latent space capacity, and we can expect our proposed model to outperform GST on these two metrics if we further increase the dimensionality of z l ; however, this would likely reduce interpretability and generalization of the latent attribute control.

G.3 NON-PARALLEL STYLE TRANSFER:

"By water in the midst of water!" "And she began fancying the sort of thing that would happen: Miss Alice!" "She tasted a bite, and she read a word or two, and she sipped the amber wine and wiggled her toes in the silk stockings."

Reference style 1 Reference style 2 Reference style 3 In each row we varied the value for a single dimension while holding other dimensions fixed. In the top row, we controlled the F 0 by traversing dimension zero. Note that the speaker identity did not change while traversing this dimension. In the second row, the F 0 contours did change while traversing this dimension; however, it can be seen from the spectrograms that the leftmost one attenuated the energy in low-frequency bands, and the rightmost one attenuated energy in high-frequency bands. This dimension appears to control the shape of a linear filter applied to the signal, perhaps corresponding to variation in microphone frequency response in the training data. In the third row, the F 0 contours did not change, either. However, the background noise level does vary while traversing this dimension, which can be heard on the demo page. In the bottom row, variation in the speaking rate can be seen while other attributes remain constant. Audio samples can be found at https://google.github.io/tacotron/publications/gmvae_ controllable_tts#crowdsourced_audiobook.control.

:

https://google.github.io/tacotron/publications/gmvae_controllable_tts

This dataset will be open-sourced soon.

We found two relevant dimensions, controlling 1) low frequency, narrowband, and 2) wideband noise levels.

controllable_tts#multispk_en.control.

io/tacotron/publications/gmvae_controllable_tts#noisy_multispk_en.sample

Figures and Tables:

Figure 1: Graphical model representation of the proposed models. Observed class often corresponds to the speaker label. The left illustrates equation 1, and the right illustrates the extension from Section 2.3. The grey and white nodes correspond to observed and latent variables.

Figure 2: Training configuration of the GMVAE-Tacotron model. Dashed lines denotes sampling. The model is comprised of three modules: a synthesizer, a latent encoder, and an observed encoder.

Figure3plots the assignment distributions for each gender and accent in this set. Most components were only used to model speakers from one gender. Each component which modeled both genders (0, 2, and 9) only represented a subset of accents (US, US, and AU/GB, respectively). We also found that the several components which modeled US female speakers (3, 5, and 6) actually modeled groups of speakers with distinct characteristics, e.g. different F 0 ranges as shown in Appendix E. To quantify the association between speaker and mixture components, we computed the assignment consistency w.r.t. speaker:

Figure 4: Left: Euclidean distance between the means of each mixture component pair. Right: Decoding the same text conditioned on the mean of a noisy (center) and a clean component (right).1 M N i=1 Ni j=1 1 yij =ŷi where M is the number of utterances, y ij is the component assignment of utterance j from speaker i, and ŷi is the mode of {y ij } Ni j=1 . The resulting consistency was 92.9%, suggesting that the components group utterances by speaker and group speakers by gender or accent.

Figure 6: Mel-spectrograms of three samples with the same text, "We must burn the house down! said the Rabbit's voice." drawn from the proposed model, showing variation in speed, F 0 , and pause duration.

Figure 7: (a) Mel-spectrograms of two unnatural GST samples when setting the weight for one token -0.1: first with tremolo at the end, and second with abnormally long duration for the first syllable. (b) F 0 tracks and spectrograms from GMVAE-Tacotron using different values for the "speed" dimension.

Figure 7(b) contains an example of the proposed model traversing the "speed" dimension with three values: μl,d -2 σl,d , μl,d , μl,d + 2 σl,d , plotted accordingly from left to right, where μl,d and σl,dare the marginal distribution mean and standard deviation, respectively, of that dimension. Their F 0 tracks, obtained using the YIN(De Cheveigné & Kawahara, 2002) F 0 tracker, are shown on the left. From these we can observe that the shape of the F 0 contours did not change much. They were simply stretched horizontally, indicating that only the speed was manipulated. In contrast, the style control of GST is more entangled, as shown inWang et al. (2018, Figure 3(a))

ADDITIONAL RESULTS ON THE MULTI-SPEAKER ENGLISH CORPUS E.1 RANDOM SAMPLES BY MIXTURE COMPONENT Component 1: male Component 3: US female (low-pitched) Component 5: US female (high-pitched) Component 4: GB/AU female Component 8: US/SG female Component 8: US/SG male

Figure 8: Mel-spectrograms and F 0 tracks of three random samples drawn from each of six selected mixture components. Each component represents certain gender and accent group. The input text is "The fake lawyer from New Orleans is caught again." which emphasizes the difference between British and US accents. As mentioned in the paper, although samples from component 3 and 5 both capture US female voices, each component captures specific speakers with different F 0 ranges. The former ranges from 100 to 250 Hz, and the latter ranges from 200 to 350 Hz. Audio samples can be found at https://google.github.io/tacotron/publications/gmvae_ controllable_tts#multispk_en.sample

Figure 9: Mel-spectrograms and F 0 tracks of the synthesized samples demonstratoing independent control of several latent attributes. Each row traverses one dimension with three different values, keeping all other dimensions fixed.. All examples use the same input text: "The fake lawyer from New Orleans is caught again."The plots for dimension 0 (top row) and dimension 2 (second row) mainly show variation along the time axis. The underlying F 0 contour values do not change, however dimension 0 controls the duration of the initial pause before the speech begins, and dimension 2 controls the overall speaking rate, with the F 0 track stretching in time (i.e. slowing down) when moving from the left column to the right. Dimension nine (bottom row) mainly controls the degree of F 0 variation while maintaining the speed and starting offset. Finally, we note that differences in accent controlled by dimension 3 (third row) are easier to recognize by listening to audio samples, which can be found at https://google.github.io/tacotron/publications/gmvae_

Figure 10: Mel-spectrograms of random samples drawn from a noisy (left) and a clean (right) mixture component. Samples within each row are conditioned on the same speaker. Likewise, samples within each column are conditioned on the same latent attribute representation z l . For all samples, the input text is "This model is trained on multi-speaker English data." Samples drawn from the clean component are all clean, while samples drawn from the noisy component all contain obvious background noise. Finally, note that samples within each column contain similar types of noise since they are conditioned on the same z l . Audio samples can be found at https://google.github.

Figure 11: Mel-spectrograms of the synthesized samples demonstrating control of the background noise level by varying the value of dimension 13. Each row conditions on a seed z l drawn from a mixture component, where all values except for dimension 13 are fixed. The embedding used in row 1 and row 3 are drawn from a noisy component, and used in row 2 and row 4 are drawn from a clean component. In addition, we condition the decoding on the same speaker for the first two rows, and the same held-out speaker for the last two rows. The value of dimension 13 used in each column is shown at the bottom, and the input text is "Traversing the noise level dimension." In all rows, samples on the right are cleaner than those on the left, with the background noise gradually fading away as the value for dimension 13 increases. Audio samples can be found at https://google.github.io/ tacotron/publications/gmvae_controllable_tts#noisy_multispk_en.control

Figure 13: Mel-spectrograms of reference and synthesized style transfer utterances. The four reference utterances are shown on the top, and the four synthesized style transfer samples are shown below, where each row uses the same input text (shown above the spectrograms), and each column is conditioned on the z l inferred from the reference in the top row. From left to right, the voices of the three reference utterances can be described as (1) tremulous and high-pitched, (2) rough, low-pitched, and terrifying, and (3) deep and masculine. In all cases, the synthesized samples resemble the prosody and the speaking style of the reference. For example, samples in the first column have the highest F 0 (positively correlated to the spacing between horizontal stripes) and more tremulous (vertical fluctuations), and spectrograms in the middle column are more blurred, related to roughness of a voice. Audio samples can be found at https://google.github.io/tacotron/ publications/gmvae_controllable_tts#singlespk_audiobook.transfer

Figure15: Synthesized mel-spectrograms demonstrating independent control of speaking style and prosody. The same input text is used for all samples: "He waited a little, in the vain hope that she would relent: she turned away from him." In the top row, F 0 is controlled by setting different values for dimension eight. F 0 tracks show that the F 0 range increases from left to right, while other attributes such as speed and rhythm do not change. In the second row, the duration of pause before the phrase "she turned away from him." (red boxes) is varied. The three spectrograms are very similar, except for the width of the red boxes, indicating that only the pause duration changed. In the bottom row, the "roughness" of the voice is varied. The same region of spectrograms is zoomed-in for clarity, where the spectrograms became less blurry and the harmonics becomes better defined from left to right. Audio samples can be found at https://google.github.io/tacotron/ publications/gmvae_controllable_tts#singlespk_audiobook.control.

Figure16: Synthesized mel-spectrograms and F 0 tracks demonstrating independent control of attributes related to style, recording channel, and noise-condition. The same text input was used for all the samples: '"Are you Italian?" asked Uncle John, regarding the young man critically.' In each row we varied the value for a single dimension while holding other dimensions fixed. In the top row, we controlled the F 0 by traversing dimension zero. Note that the speaker identity did not change while traversing this dimension. In the second row, the F 0 contours did change while traversing this dimension; however, it can be seen from the spectrograms that the leftmost one attenuated the energy in low-frequency bands, and the rightmost one attenuated energy in high-frequency bands. This dimension appears to control the shape of a linear filter applied to the signal, perhaps corresponding to variation in microphone frequency response in the training data. In the third row, the F 0 contours did not change, either. However, the background noise level does vary while traversing this dimension, which can be heard on the demo page. In the bottom row, variation in the speaking rate can be seen while other attributes remain constant. Audio samples can be found at https://google.github.io/tacotron/publications/gmvae_

MOS comparison of the original audio, baseline and GMVAE.

Table content: <table><tr><td>Model</td><td>MOS</td></tr><tr><td>Original</td><td>4.67 ± 0.04</td></tr><tr><td colspan="2">Baseline 4.29 ± 0.11</td></tr><tr><td colspan="2">Proposed 4.67 ± 0.07</td></tr></table>

SNR of original audio, baseline, and the proposed models with different conditioned z l , on different speakers.

Table content: <table><tr><td colspan="3">Set Original Baseline</td><td colspan="3">Proposed mean latent latent-dn</td></tr><tr><td>SC</td><td>18.61</td><td colspan="3">14.33 15.90 16.28</td><td>17.94</td></tr><tr><td>SN</td><td>11.80</td><td colspan="2">9.69 15.82</td><td>6.78</td><td>18.94</td></tr><tr><td>UC</td><td>20.39</td><td colspan="3">N/A 15.70 16.40</td><td>18.83</td></tr><tr><td>UN</td><td>10.92</td><td colspan="2">N/A 15.27</td><td>4.81</td><td>16.89</td></tr></table>

Subjective preference (%) between baseline and proposed model with denoised z l on the set of "seen noisy" (SN) speakers.We augmented the model with the z o layer described in Section 2.3 to learn a continuous speaker representation and an inference model for it. The train-clean-{100,360} partitions were used for training, which spans 1,172 unique speakers and, despite the name, includes many noisy recordings. As in previous experiments, by traversing each dimension of z l we found that different latent dimensions independently control different attributes of the generated speech. Moreover, this representation was disentangled from speaker identity, i.e. modifying z l did not affect the generated speaker identity if z o was fixed. In addition, we discovered that the mean of one mixture component corresponded to a narrative speaking style in a clean recording condition. Demonstrations of latent attribute control are shown in Appendix H and the demo page.

Table content: <table><tr><td colspan="3">Baseline Neutral Proposed</td></tr><tr><td>4.0</td><td>10.5</td><td>85.5</td></tr></table>

Naturalness MOS of original audio, baseline, and proposed model with the clean component mean.

Table content: <table><tr><td colspan="2">Set Model</td><td>MOS</td></tr><tr><td/><td>Original</td><td>4.60 ± 0.07</td></tr><tr><td>SC</td><td colspan="2">Baseline Proposed 4.18 ± 0.06 4.17 ± 0.07</td></tr><tr><td/><td>Original</td><td>4.45 ± 0.08</td></tr><tr><td/><td>Baseline</td><td>3.64 ± 0.10</td></tr><tr><td>SN</td><td colspan="2">+ denoise 3.84 ± 0.10</td></tr><tr><td/><td colspan="2">Proposed 4.09 ± 0.08</td></tr><tr><td/><td>Original</td><td>4.54 ± 0.08</td></tr><tr><td>UC</td><td colspan="2">d-vector Proposed 4.26 ± 0.05 4.10 ± 0.06</td></tr><tr><td/><td>Original</td><td>4.34 ± 0.07</td></tr><tr><td>UN</td><td colspan="2">d-vector Proposed 4.20 ± 0.08 3.76 ± 0.12</td></tr></table>

Speaker similarity MOS.

Table content: <table><tr><td colspan="2">Set Model</td><td>MOS</td></tr><tr><td>SC</td><td>Baseline Proposed</td><td>3.54 ± 0.09 3.60 ± 0.09</td></tr><tr><td/><td>Original (different channels)</td><td>3.30 ± 0.27</td></tr><tr><td>SN</td><td>Baseline</td><td>3.83 ± 0.08</td></tr><tr><td/><td colspan="2">Baseline + denoise 3.23 ± 0.20</td></tr><tr><td/><td>Proposed</td><td>3.11 ± 0.08</td></tr><tr><td/><td>d-vector</td><td>2.23 ± 0.08</td></tr><tr><td>UC</td><td>d-vector (large)</td><td>3.03 ± 0.09</td></tr><tr><td/><td>Proposed</td><td>2.79 ± 0.08</td></tr></table>

Prior hyperparameters for each dataset used in Section 4.

Table content: <table><tr><td/><td>multi-spk</td><td colspan="3">noisy-multi-spk audiobooks crowd-sourced</td></tr><tr><td/><td>(Section 4.1)</td><td>(Section 4.2)</td><td>(Section 4.3)</td><td>(Section 4.4)</td></tr><tr><td>dim(y l )</td><td>10</td><td>10</td><td>10</td><td>10</td></tr><tr><td>dim(z l )</td><td>16</td><td>16</td><td>16</td><td>16</td></tr><tr><td>initial σ l</td><td/><td/><td/><td/></tr></table>

F 0 distribution for the three US female components.

Table content: <table><tr><td>Component</td><td>3</td><td>5</td><td>6</td></tr><tr><td>F 0 (Hz)</td><td colspan="3">169.1 ± 10.1 214.8 ± 8.4 192.2 ± 21.4</td></tr></table>

Accuracy (%) of linear classifiers trained on z l .To quantify how well the learned representation captures useful speaker information, we experimented with training classifiers for speaker attributes on the latent features. The test utterances were partitioned in a 9:1 ratio for training and evaluation, which contain 2,098 and 234 utterances, respectively. Three linear discriminant analysis (LDA) classifiers were trained on the latent attribute representations z l to predict speaker identity, gender and accent. Table9shows the classification results. The high accuracies in the table demonstrate the potential of applying the learned low-dimensional representations to tasks of predicting other unseen attributes.

Table content: <table><tr><td/><td colspan="3">Gender Accent Speaker Identity</td></tr><tr><td colspan="2">Train 100.00</td><td>98.76</td><td>97.66</td></tr><tr><td>Eval</td><td>98.72</td><td>98.72</td><td>95.39</td></tr></table>

Quantitative evaluation of disentangled attribute control of GMVAE-Tacotron.

Table content: <table><tr><td>Target dim</td><td>Attribute</td><td>μl,d -3 σl,d</td><td>μl,d</td><td>μl,d + 3 σl,d</td></tr><tr><td>Pitch dimension (d = 8)</td><td>Pitch (Hz) Duration (seconds)</td><td>178.3 3.5</td><td>187.1 3.5</td><td>204.4 3.5</td></tr><tr><td>Speaking rate dimension (d = 7)</td><td>Pitch (Hz) Duration (seconds)</td><td>197.1 3.0</td><td>189.7 3.5</td><td>182.5 4.0</td></tr></table>

Quantitative evaluation for parallel style transfer. Lower is better for both metrics.

Table content: <table><tr><td>Model</td><td>MCD 13</td><td>FFE</td></tr><tr><td>Baseline</td><td colspan="2">17.91 64.1%</td></tr><tr><td>GST</td><td colspan="2">14.34 41.0%</td></tr><tr><td>Proposed (16)</td><td colspan="2">15.78 51.4%</td></tr><tr><td>Proposed (32)</td><td colspan="2">14.42 42.5%</td></tr></table>

Peer review (iclr2019):
 The authors describe the conditioned GAN model to generate speaker conditioned Mel spectra. They augment the z-space corresponding to the identification with latent variables that allow a richer set of produced audio. In a way this is like a partially conditioned model that has "extra" degrees of freedom. It looks that the "latent" variables are just concaneted to the "original" set of z-values (altough with particular conditions to maximize independence). The conditioning of the z-space has originality in it and may provide interesting to the audience. Ultimately one coud think about z-space direction being totally mapped to specific features of the produced signal. Also, I am curious to know how the Mel spectra are used to produce the actual sound wave - as the phase information is not present if utilizing only the spectral amplitude. Very often this leads to suboptimal generation, and the remedy is to use the time domain like in ( https://arxiv.org/ftp/arxiv/papers/1810/1810.05319.pdf). However, in this case the audio samples show a pretty nice generation of sound. However, it is not really end to end. The manuscript has some curious decisios in its concepts - I do not see the architecture really hiearchial, nor end to end. I would prefer modifications on the paper that concentrate on the truly novel features. The paper is clear, well written and done with high ambition, from data set utilization to novel architetures to human quality panels. Results are good and interesting. NEW: The authors have addressed the concerns I had with the manuscript.