Semi-Supervised Generative Modeling for Controllable Speech Synthesis
Authors: Raza Habib
Soroosh Mariooryad
Matt Shannon
Eric Battenberg
RJ Skerry-Ryan
Daisy Stanton
David Kao
Tom Bagby
Venue:  (2020)
Keywords: TTS
Speech Synthesis
Semi-supervised Models
VAE
disentanglement

Abstract: We present a novel generative model that combines state-of-the-art neural text- to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn’t been possible with purely unsupervised methods. We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 1% (30 minutes) supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline. We will release audio samples at https://google.github.io/tacotron/publications/semisupervised_generative_modeling_for_controllable_speech_synthesis/.

Published as a conference paper at ICLR 2020
SEMI-SUPERVISED GENERATIVE MODELING FOR CONTROLLABLE SPEECH SYNTHESIS
Raza Habib Soroosh Mariooryad
Matt Shannon?
Eric Battenberg
R. Skerry-Ryan'
Daisy Stanton" David Kao² Tom Bagby²
'University College London (UCL)
"Google Research.
raza.bab barabbbbbees... www.uccc...... (soroosh, mattshannon, ebattenberg, rjryan, daisy, davidkao, tombagby / @google.com
ABSTRACT
We present a novel generative model that combines state-of-the-art neural text- to-speech (TTS) with semi-supervised probabilistic latent variable models. By providing partial supervision to some of the latent variables, we are able to force them to take consistent and interpretable purposes, which previously hasn't been possible with purely unsupervised TTS models. We demonstrate that our model is able reliably discover and control important but rarely labelled at- tributes of speech, such as affect and speaking rate, with as little as 30 minutes supervision. Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of of-the-art baseline. Audio samples are available on the w
INTRODUCTION
The ability to reliably control high level attributes of speech, such as emotional expression (affect) or speaking rate, is often desirable in speech synthesis applications. Achieving this control however made difficult by the necessity of acquiring a large quantity high quality labels. In this paper we show that semi-supervised latent variable models can take us significant step closer towards solving this problem.
Combining state-of-the-ard neural text-to-speech (TTS) systems with probabilistic latent variable models provides natural framework for discovering aspects of speech that are rarely labelled or even difficult to describe. Both inferring the latent prosody and generating samples with sufficient variety requires reasoning about uncertainty and thus a natural fit for deep generative models.
There has been recent progress in applying stochastic gradient variational Bayes (SGVB) (Kingmal & Welling 2013) Rezende et al. 2014) to training probabilistic neural TTS models. Battenberg et al. (2019) and Hsu al. (2018) have shown that is possible to use latent variable models discover teatures such as speaking style, speaking rate, arousal, gender and even the quality of the recording environment.
However, these models are formally non-identifiable (Hyvärinen & Pajunen 1999 Locatello et al., 2019) and this implies that repeated training runs will not reliably discover the same latent attributes. stage is necessary to identify what the model has learned on any given training run. In order be of practical use for control, it not enough for the models to discover latent attributes, they need to do so reliably and in way that is robust to random initialization and to changes in the model. We demonstrate that the addition of even modest amounts of supervision can be sufficient to achieve this reliability.
Even if they did, lengthy human post-processie
By augmenting state-of-the the art neural TTS with supevisee deep generative models within the VAE framework (Kingma et al.) 2014 Narayanaswamy al., 2017), show that it is possible to not only discover latent attributes speech but to do so reliable and controllable manner. In particular able to achieve reliable control over affect, speaking rate and FO variation (FO is the
Work performed while interning at Google Research.
attps: http:///ggggggg//eeeee gichub. io/ http/ccttro//chlleeeeeeoooo seat and www. wtwp:/iww...eeee..eeeeee generat ive iodeeooeeeee_eeoeeeeeeeoo ling for control Table_ee wwwpspc_eeethheeeeeee

---

Published as a conference paper at ICLR 2020
2_
broodee LoNas on otends
Higway layera
CB-1
Aterrton
Ale in apSted
Tue prriina Seeenle
Vierped
Dense Diopou!
OuenD
Decoder
embidddinge
Encader
(a) CBHG block
(b) Seq-to-seq network
Figure 1: Schematic showing how we parameterize the conditional likelihood p(z|y, Zu: 2,). Left: block of 1-D convolutions and RNNs originally introduced by Wang et al. (2017) and described in detail in the appendix. Right: Schematic of the sequence-to-seque seqquence network that outputs the means of our auto-regressive distribution. At each decoder time step, the network outputs the means for the next two spectrogram frames.
fundamental frequency oscillation of the vocal folds). Further, we provide demonstrations that is possible to transfer controllability to speakers for whom we have no labels. Our core contributions are:
To combine semi-supervised latent variable models with Neural TTS systems, producing system that can reliably discover attributes of speech we wish control.
To demonstrate that as little as 30 minutes supervision can be sufficient to improve prosody and allow control over speaking rate, fundamental frequency (FO) variation and affect, problem of interest to the speech community for well over two decades (Schröder 2001).
To imbue TTS models with control over affect, FO and speaking rate whilst still maintaining prosodic variation when sampling.
GENERATIVE MODEL.
Our generative model, shown in figures ! and 2a consists of an autoregressive distribution over sequence of acoustic features, Ti...; that generated conditioned sequence of text, y1.... and on two latent variables, 2, and 2,. The latent variables can be discrete or continuous. 2, represents the variations in prosody that seek to control and semi-supervised. zu fully unobserved and represents latent variations in prosody (intonation, rhythm, stress) that we wish to model but not explicitly control. Once trained, our model can be used to synthesize acoustic features from text. Similar to Tacotron 2 (Shen et al. (2018), we then generate waveforms by training second network such as WaveNet (van den Oord al. 2016) WaveRNN (Kalchbrenner al. 2018 act vocoder. In our case we use WaveRNN.
We parameterize our likelihood p( k...... Zuz 2g, 0)) by a sequence-to-sequene neural network with attention (Shen et al. 22188, Graves 2013 Bahdanau al. 2014) that is shown schematically in figure Details largely follow Tacotron (Wang et al. 2017) and are given in appendix [A] At each time step model a mel-spectrogram frame with fixed variance isotropic Laplace distribution whose mean output by the neural network. We condition each of the latent variables by concate- nating the vectors 2n and 2g to the representation of the text-encoder, before the application of the attention mechanism. the case of continuous & we use a standard normal prior and the case of discrete uniform categorical prior with one-hot encoding.
2

---

Published as a conference paper at ICLR 2020
Text and speaker
Unupervival Semi-uupervisal
Unsupervised
Ussuperiise Semi-superiied
Sa
Zu
Zu
supervised
spectrogmum
Mel spectroram
Mel spectoorra
(a) Generative model
(h) Unsupervised posterior
(c) Supervised posterior
Figure 2: Left: The graphical model showing the conditional independence assumptions between
each of the stochastic variables. Centre: The structure of the variational distribution used approx-
imate the posterior for fully unsupervised data points and Right: supervised points.
2.1 SEMI-SUPERVISED TRAINING
Following Kingma et al. (2014); Narayanaswamy et al. (2017), train model via stochastic gradient variational Bayes (SGVB). That is we approximately maximize the log-likelihood of our training data by maximizing variational lower bound using stochastic gradient ascent. Since we training with semi-supervision we fact need two lower bounds: one for the data points for which 2g is observed; one for the case where 2g is unobserved. In models the fully latent variable 2u is always continuous but the semi-supervised latent 2, can be continuous or discrete. The conditional
independence structure of variational distributions is shown in figures and 2c On supervised data, the per-datapoint bound is:
log p(z 0) log / p(2r, Zus 2 Zaly, 0) dz..
2 E Fete.le log p(zlu. Zu: 20, ()p((z))pppp)) q(zu Jr, 2s. v) (e. [log p(xlyl zus 20, 0)] + logp(ze) DKL (a(2aal v. 2. ()(((()))
(0, Φ3 2,
Where (1 y, 2s: Φ) is parametric variational distribution introduced approximately marginalize 2u- are the parameters the generative model and o are the parameters of the vari- ational distributions. The intractable integrals are approximated with reparameterized samples. For the cases where 2, is unobserved and discrete, the bound is:
logp(x/y,.0 0) log Ep(a,
p(a, Zus zoly,
Σ (0,
H
(2)
and when 2, is contimuous we replace the sum above with an integral and again approximate with reparameterized samples. The variational distributions are parameterized by neural network that takes as imput the text, spectrograms and other conditioning variables and outputs the parameters of the distribution. The exact structure of this network given in appendix A We have implicitly
assumed that g(zu. 2, Jz, Φ) may be factorized q(²u, 2, y. Φ) q(2 U, 2., (){(2.122 |z. y, Φ) with shared parameters between these two distributions (see appendix A). Optimizing the variational objective with respect to the parameters & encourages the variational distributions to match the posterior of the generative model p(2u: 22| y. 0). Unlike previous work (Hsu et al. 2018). we do not assume that the posterior exists in the model due to explaining away. That is to say that although the text and the latents are independent in our prior, observing the spectrogram correlates them in the posterior because they
the latents is independent of the text, this dependence likely
3

---

Published as a conference paper at ICLR 2020
Aroucal
Tanie
Dostted
H AhhhArcsss
Nageliiee
Angry
Delighos
Frusrase
Happy
Tourc
yowe Vollence
Dored
Reloxod
Lowe Accundl
Low Arousl Predita-Vvsletee
Negathv Vallence
TIreo
Caln
Figure 3: The circumplex model of emotion. Each possible emotion is represented in dimen- sional plane consisting of an arousal dimension and valence dimension. This figure is borrowed from Munoz-de Escalona & Canas (207))
both explain variation in the spectrogram. This has been shown be significant by Battenberg al. 2019).
we define:
((..11.)) = { [((2a z pr, Ø)
unsupervised
(4)
(70(2, Zssweves)
if supervised
then we can write the overall objective over both the supervised and unsupervised points succinctly a
C(0,0) x) = E.r,yy
Σ
0, 2.) + H((( 2 の))
(5)
where summation would again be replaced by integration for continuous 2, and (shown in equation is weighting factor that pre-multiplies the loss for any supervised point. This weighting was also used previous work such as Narayanaswamy et al. (2017). who showed to be beneficial at very low levels of supervision.
Writing the objective in this form allows an intuitive interpretation for the suniiviueeiiee training procedure. When supervision is provided, objective function is evaluated at the observed value of 2,, When supervision is not provided, we evaluate the objective function for every possible value of 2, and take (potentially infinite for continuous 2,) weighted average. The weighting in the average is given by q(2a |r, y, Φ), which is simultaneously trained to approximate the posterior p( z Jæ, y, 0). In other words, unsupervised utterances, we evaluate our objective for each possible value of the latent attribute and weight by the (approximate) posterior probability that this value of the latent was responsible for generating the utterance.
As q(2, |r, y, Φ) is trained to approximate p(z,|z, y, () we can expect it to become a reasonable classifierrregressor for the semi-supervised latent attribute as the model improves. For example when represents an affect label, p22./z. (z. 0) is the posterior probability, of the model, over affect given text and speech. By taking the most likely posterior class, this distribution can be used as an affect classifier. However, this variational distribution is only trained on unsupervised training points and so does not benefit directly from the supervised data. To overcome this problem we follow Kingma al. (2014) and add classification loss our objective. The overall objective becomes:
Cu L() aEz,y. [log q(2,) y, +)]
(6)
Where a is hyperparameter which adjusts the contribution of this term
We define the differential entropy of the delta function be
4

---

Published as a conference paper at ICLR 2020
3 DATA
We have used a proprictary high quality labeled data-set of 40 English speakers, The training set consists of 72,405 utterances with durations of at most 5 seconds (45 hours). The validation and test sets each contain 745 utterances or roughly 30 minutes of data. We vary the amount of super- vision in the experiments below. We also experimented with transferring controllability unlabeled data-set of audiobook recordings by Catherine Byers (the speaker from the 2013 Blizzard Challenge), which exhibits high variation in affect and prosody and to other speakers who were less expressive. We strongly encourage the reader to listen to the synthesized samples on our demo page
fully
In this work we chose to focus on learning to control affect with a discrete representation, as well as speaking rate and FO variation with continuous representation, as these are challenging aspects of prosody to control. Our method could be applied to other factors without modification.
3.1 AFFECT CONTROL
The best way to represent emotion an actively researched area and many models of affect exist. In this work we chose to follow the circumplex model of emotion (Russell 1980) which posits that most affective states can be represented a dimensional plane with one axis representing arousal and the other axis representing valence. Arousal measures the level of excitement valence measures positivity or negativity. Figure shows a chart of emotions plotted in the arousal- valence plane where
energy and
that, for example, high arousal and high valence corresponds to joy or happiness whereas high arousal and low valence might correspond to anger or frustration.
Our data-set recorded under studio conditions with trained voice actors who were prompted to read dialogues in one of three 2 -1, +2 and two arousal values:-2 (low), +2 (high). This was achieved by prompting the actors read dialogues in either a happy, sad angry voice two levels of arousal. This results in 6 possible affective states which we chose to model as discrete and use as our supervision labels.
3.2 SPEAKING RATE AND FO VARIATION CONTROL
In order to demonstrate that we can control continuous attributes we also created approximate real- valued labels for speaking rate and arousal for all of our data. We generate the approximate speaking rate number of syllables per second in each utterance. FO, also known as the fundamental fre- quency, measures the frequency of vibration of the vocal folds during voiced sounds. Variation in FO is highly correlated with arousal and roughly measures how expressive an utterance is. To create approximate arousal labels we extracted the F0 contour from each of our utterances, using the YIN algorithm (De Cheveigné & Kawahara, 2002), and measured its standard deviation. We then per- formed a whitening transform on these two approximate labels in order to match it to our standard normal prior.
These artificial labels would of course be cheap to obtain for the entire data-set and would not justify the use of semi-supervision in real applications. But, our objective here is to evaluate/demonntratte the efficacy of semi-supervision rather than to specifically control a particular attribute. We have chosen syllable rate and FO standard deviation, because they both correspond to subjectively distinct variations of interest, and they are more easily quantifiable than affect and so provide strong evidence of controllability. For the continuous latents we are not only able interpolate speaking-rates and FO variations but also to extrapolate outside of our training data. We provide examples on our demo page of samples with significantly greater/lower speed and FO variation than typically observed in natural speech.
4 EXPERIMENTS AND RESULTS
To evaluate the efficacy of semi sumeriuuerei latent variable models for controllable TTS we trained the model described in section on the above data-sets at varying levels of supervision as well as for 2
Sound demos are available at https:: / /googl github. ronppott loationn tona//en supervi sed.
generat innerattiv__aaae hwa_nodelloooooooooooe mag.noorrrrooooooooooo for_control lable, www.ppp_eenttiioeeeeeepe
5

---

Published as a conference paper at ICLR 2020
Valence
Arousal
baseline vs, angry
bascline vs. sad
baseline vs. happy
low vs. high
27 min (1%)
-0.20 + 0.10
-0.60 = 0.08
-0.43 + 0.09
-0.50 +: 0.09
preference
135 min (5%)
-0.74 +: 0.07
-0.83 0.06
-0.83 + 0.06
-0.57 + 0.08
score
270 min (10%)
-0.71 + 0.07
-0.86 = -0.95
-0.61 0.08
-0.59 + 0.08
Table I: Subjective metrics for affect control. Negative is a preference for the controlled model. +1 and indicate a preference for samples A and B, respectively. For valence, raters are told that a sample is intended to convey a particular emotion, e.g. happy, and then presented with sample from baseline without control (A), and controlled model (B), and asked to choose between them. For arousal, raters are told to choose the sample that is more vocally aroused, and presented with controlled samples in low (A) and high (B) arousal. To avoid bias, the orders are randomly altered during rating. We show preference score and 95% confidence intervals at multiple supervision levels.
varying settings of the hyperparameters: which controls the supervision loss and which over emphasizes supervised training points. We found that a value of = 1 was optimal for the discrete experiments and a 0 for the continuous experiments, which corresponds to simply optimizing the ELBO. For each experiment we report the results for the best r found, and 7 = 1. 7 corresponds to experiments with no ovighteig of the supervised points. All models were trained using the ADAM optimizer with learning rate of 10 -3 and run for 300, 000 training steps with a batch size of 256, distributed across 32 Google Cloud TPU chips. All models were implemented using tensorflow (Abadi et al., 2016).
Assessing the degree of control is challenging interpreting affect is subjective. We used two objective metrics of control as well as subjective evaluation from human raters and a third objective the test-set accuracy of 6-class affect classifier trained on the ground truth training data and applied to generated samples from the model (shown in figure 4a). The classifier is convolutional neural network whose structure mirrors the posterior network q( 2, Ir Φ) and its exact architecture is given in appendix A We also provide subjective metries of controllability, shown in table For speaking rate control, we measure the mean syllable rate eтrоr on a held out test-set. The syllable rate error is calculated as the absolute difference in syllable rate between the desired syllable rate and that measured from the synthesized sample. We calculate an analogous error rate for FO variation.
metric of overall quality. For affect, the first objective metric we introduced
Whilst the two metrics above measure controllability they don't don't tell us if this comes at the expense of degradation in synthesis quality. To probe quality we use three further metrics. The first was Mel- cepstral-Distortion Danooo--DDammiiimeeeee Warping (MCD-DTW) (Kubichek] 1993) on a held out test-set, shown in figure 4d MCD-DTW is a measure of the difference between the ground-truth spectro- gram and the synthesized mel spectrogram that is known to correlate well with human perception (Kubichek) 1993). The second metric of quality was crowd sourced mneanoopinion-score (MOS). The third metric of quality is speech recognition word error rate (WER) and character error rate (CER) audio samples. The MOS and speech recognition results are summarized in table
To demonstrate that semi-supervision by including unlabelled data is beneficial, we also provide MOS and speech recognition errors for fully supervised subsets of the data in table These show that at least close to 5 hours of data is required to train reasonable quality TTS model, far above the 30 minutes supervision needed to control prosodic aspects of speech.
We provide further details of all of these metrics in the appendix provided in appendix C
B and sample spectrograms are
DISCUSSION
The classification accuracy (see figure 4aa, subjective metrics (see table [ and error-rate results (see figure 4c) provide a clear demonstration that using semi-supervised latent variables, we are able to achieve control of both continuous and discrete attributes of speech. There is not a significant degradation in the overall quality and this is evidenced by the mean opinion scores which are above the baseline, Tacotron, and also speech recognition errors (see table 2). We also include baseline of our Tacotron model augmented only by the unsupervised latent 2., to aid comparison.

---

Published as a conference paper at ICLR 2020
Affect accuraty
0.8
0.45
y-10
0.7
0.6
0.35
0.5
Speaking
0.30
0--
0.25
y=1
0.20
100
8.81
0.05
Sup ervision fraction
Supervision fraction
(a) Affect classification accuracy
(b) Speaking-rate error
y=1
6.2
Affect y 100
y=10
Speaking rate y
13
FD stdsev 20
stddieu error
MCD-D1W
6.0
11
5.9
10
5.7
8di
0.05
6.
1.0
881
0.05
0.1
1.0
Supervision traction
Supervision fraction
(c) FO variation error
MCD-DTW
Figure 4: Objective evaluation metrics function of supervision fraction. 100% supervision
corresponds to 45 hours of supervised training data and 0% supervision corresponds to base tacotron. For MCD-DTW and error-rates lower is better.
Semi- Sunni-uuerriied (10% supervision)
continuous latent
discrete latent
ground truth
baseline
baseline with &n
FO
speaking-
affect
MOS
4.52+-0.07
4.09++0.9
4.24++0.08
4.28
4.16+00.08
4.17++0.09
WER
4.49
4.93
4.93
4.06
2.53
6.09
CER 2.11
2.38
2.22
1.80
1.14
3.12
Table 2: Metrics of overall quality: Mean Opinion Scores (MOS) alongside 95% confidence inter- vals, speech recognition word error rate (WER), and chararacter error rate (CER), The results show no degradation in performance compared the baseline.
27 min (1%)
54 min (2%) 108 min (4%)
135 min (5%)
270 min (10%)
45 hours (100%)
MOS
unintelligible 91.95
unintelligible
3.20:10113
3.52-10.11
4.03+10.08
4.08=10.99
WER
96.31
19.83
7.55
5.56
4.93
CER
74.57
78.9
12.54
4.46
2.79
2.22
Table 3: Metrics of overall quality for fully supervised data at varying data-set sizes, showing sig- nificant degradation below 270 minutes.
The MCD-DTW scores for variation and affect are improved at all levels of supervision (figure 4d Whilst the MCD-DTW is degraded for speaking rate. this is likely misleading metric when largeting changes in timing as the dynamic-timewwappin component of MCD-DTW changes exactly the aspect wish control. For speaking rate the combination of MOS and samples is better indication the overall quality. We are able to reduce the supervision level to levels as low as 1% 30 minutes and still have significant degree of control. We show
demo page that even
https::/ http///gll....hbb..b/e gichub. ront labla_speach, aeoecch_aynthatt
gow.rettgooggagoaoeoeoo ive. lig.mmoodlnnngn........ lingnfooo for

---

Published as a conference paper at ICLR 2020
at 3 minutes of supervision we can still achieve control of speaking rate and that able to extrapolate outside the range of values seen during training. On the affect data our classification accuracy doesn't 't degrade significantly until we reach 10% (300 minutes) supervision and remains significantly above chance down to levels as low as 1% (30 minutes), see figure 44 and table 1 Obtaining 30 minutes of supervised data is likely within reach of most teams constructing TTS systems. Unlike previous work on generative modelling for control (Hsu et al. 2018 Wang et al, 2018), we do not require a post-procsssing stage to determine what our latent variables control and we can pre-determine what aspects we wish to control through choice of data. By separating our latent variables into those that are partially supervised and those which are fully unsupervised we retain the ability to model other latent aspects of prosody; this means that we can still draw samples of varying prosody whilst holding constant the affect, speaking rate or FO variation.
We observe the greatest degree of affect control, as measured by classifier accuracy, when and 7 = 100. This means that to achieve the highest controllability we needed to 1) provide extra information to our approximate posterior (z,, V, () and 2) to over-represent the supervised data at low levels of supervision. Although both of these hyperparameters have been used in the literature before (Narayanaswamy et al.) 2017: Kingma et al. 2014) and shown to be either beneficial or necessary, they aren't strictly required by our probabilistic framework and so it is worth considering why they are needed. There are three potential sources of error in any generative model trained with SGVB: the model itself may be mis-specified such that the true data-generating distribution is not in the model class, the parametric family chosen to approximate the posterior may be overly restrictive and finally the optimization landscape may contain undesirable local minima. These problems have afflicted previous work with deep latent variable models trained with SGVB, resulting in models that don't 't use their latent variables unless trained with complex annealing schedules (Bowman et al., 2015). In our case we believe that the necessity to set a and y arises from a combination of model mis-specification and local minima. If cr is set to O, then at the start of training g(=, (z, y, σ) is trained only to approximate p(z,) Jr, y, 0o), which is randomly initialized. We found empirically that in our discrete-latent experiments this resulted in q(2a (z y, 6) collapsing early in training to point- mass on a single class. Having ended up in this undesirable local minimum the posterior distribution never recovered, despite this being an obviously poor approximation the model posterior later in training. The addition of the classification loss and supervision weighting were sufficient to overcome this collapse and allow q continue to model the posterior.
The optimization landscape is strongly affected by the relative size of the conditional likelihood and KL terms in our objective. These are in turn strongly affected by our choice of conditional inedpnn- dence assumptions and output-distributions. Thus, a natural direction for further work is to increase the expressivity of the conditional likelihood p(zly, 2z. Zu: Φ) to reduce model mis-specification. This could be done by learning the variance of the syplace-distribttion we currently use or by parameterizing more expressive output distributions that do not assume conditional independence across spectrogram channels. We conjecture that with more expressive output distributions, it may be possible to reduce the need for the a and y terms in the objective. In this work we chose to use quite simple unconditional diagonal Gaussian priors, our primary goal was to demonstrate the practicality of eemi-supervision. Another natural extension would be to use conditional-priors p(zly) and to more expressive priors such mixtures as was done in Hsu et al. (0018).
5.1 RELATED WORK
There has been enormous recent progress in neural TTS with numerous novel models proposed in recent years to synthesize speech directly from characters or phonemes (Shen et al.] 2018: Arik
et al. 2017 Gibiansky et al. 2017 Ping et al.) 2017 Vasquez & Lewis 2019 Differentiating factors between these models include the degree of parallelism, with some models using Transformer based architectures (Ren et al. 2019), the choice of conditional independence assumptions made [Vasquez& & Lewis, 2019) or the number of separately trained components (Gib-) jansky et al. 2017). Our work here is largely orthogonal to the exact structure of the conditional likelihood p [z y/, 2,, 2u) and could be combined with all of the above methods.
al.] 2017).
Much of the recent research focus has been on modeling latent aspects of prosody. Early attempts include Global Style Tokens (Wang et al.) 2018) which attempted to learn a trainable set of style- embeddings. Wang et al. (2018) condition the Tacotron decoder on a linear combination of em- bedding vectors whose weights during training are predicted from the ground-truth spectrogram.
8

---

Published as a conference paper at ICLR 2020
They were able to achieve prosodic control but there is no straightforward way sample utterances of varying prosody. More recently. attempts have also been made to combine probabilistic latent variable models trained using SGVB (Akuzawa et al. 2018 Wan et al. 2019). These models use a fully unsupervised and non-identifiable approach, which makes it difficult to disentangle or inter- pret their latent variables for control. (Hsu et al.] (20T8) attempt overcome this problem by using a Gaussian mixture as the latent prior and so perform clustering in the latent space. Battenberg ct al. (2019) introduce a hierarchical latent variable model to separate the modelling of style from prosody. However, all of these methods are fully unsupervised and this results in latents that can be hard to interpret or require complex posst-prcsssing...
The work most similar to ours is Wu et al. (209)) which also attempts to achieve affect control using semi-supervision with a heuristic approach based on Global Style Tokens (Wang et al., 2018). Wu et al. (2019) add cross-entropy objective to the weightings of the style-tokens that encourages them to be one-hot on points with supervision. Similar to our method, they are able to achieve control over affect but unlike our method they do not have principled probabilistic interpretation nor the ability to simultancously model aspects of prosody other than emotion. The result is that their method not able to draw samples of varying prosody for the same utterance with fixed emotion. Furthermore, whilst our method can be applied to both continuous and discrete controllable factors, its not clear how to extend the style-token based approach to handle continuous latent factors.
In the wider generative modelling literature, the combination of semi-supervision and deep latent variable models was first introduced in Kingma et al. (2014) who focus on using unlabelled data to improve classification accuracy. The potential to use the same technique for controllable genera- tion was recognized by Narayanaswamy et al. (0177) who also provided demonstrations on image synthesis tasks. Since that work, interest in learning disentangled latent variables has grown but generally pursued alternate directions such as re-weighting the ELBO (Higgins et al.j 2017), aug- menting the objective to encourage factorization (Kim & Mnih 2018) or using adversarial training (Mathieu et al., 2016). The ability to transfer controllability to speakers for whom we do not have supervision is referred to as domain transfer and our model bears similarities to that introduced by Isse et al. 2019) but they use a mixture in their latent space more similar to Hsu et al. (2018).
5.2 ETHICAL CONSIDERATIONS
As with many advances in speech synthesis, progress in controllability raises the prospect that bad actors may misuse the technology either for misinformation or to commit fraud. Improvements in data efficiency and realism increase these risks and, when publishing, a consideration has to be made as to whether the benefits of the developments outweigh the risks. It is the opinion of the authors in this case that, since the focus of this work is on improved prosody, with potential benefits to human-compute interfaces, the benefits likely outweigh the risks. We nonetheless urge the research community to take seriously the potential for misuse both of this work and broader advances in TTS.
CONCLUSION
We have shown that the combination of semi-supervised latent variable models with neural TTS presents a practical and principled path towards building speech synthesizers we can control. Un- like previous fully unsupervised methods, we are able to consistently and reliably learn to control predetermined aspects of prosody. Our method can be applied to any latent attribute of speech for which a modest amount of labelling can be obtained, whether it be continuous or discrete. In our experiments we found that 30 minutes of supervision was sufficient, a volume of data that is within the reach of most research teams. We are able to learn to control subtle characteristics of speech such as affect and for continuous attributes we have provided demonstrations of extrapolation to ranges never seen during training, and to speakers with no supervision. Augmenting existing state- of-the-art TTS systems with latent variables does not degrade synthesis quality and we evidence this with crowd sourced mean opinion scores. Unlike similar heuristic methods, our probabilistic formulation, allows us to draw samples of varying prosody whilst holding constant some attribute we wish to control.
9

---

Published as a conference paper at ICLR 2020
REFERENCES
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Is- ard, al. TensorFlow: system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265 (65-283, 2016.
K. Akuzawa, Y. Iwasawa, and Y. Matsuo. Expressive speech synthesis via modeling expressions with variational autoencoder. In Interspeech 2018, 2018.
S. Ó. Arik, M. Chrzanowski, A. Coates, G. Diamos, A. Gibiansky, Y. Kang, X. Li, J. Miller, A. Ng, and J. Raiman. Deep voice: Real-time neural text-to-speech. In Proceedings of the 34th Interna- tional Conference on Machine Learning- SoaninngVVoumme Volume 70, 2017.
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 1409,0473, 2014.
E. Battenberg, S. Mariooryad, D. Stanton, R. Skerry-Ryan M. Shannon, D. Kao, and T. Bagby. Effective use variational embedding capacity in expressive end-to-end speech synthesis. arXiv preprint arXiv: 03402, 2019.
E. Battenberg, R. Akerry-Ryan S. Mariooryad, D. Stanton, D. Kao, M. Shannon, and T. Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
Bowman, Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio. Generating sentences from continuous space. arXiv preprint arXiv:1511 1,06449, 2015.
A. De Cheveigné and H. Kawahara. YIN, fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America, pp. 1917-1930, 2002.
A. Gibiansky, S. Arik, G. Diamos, J. Miller, K. Peng, W. Ping, J. Raiman, and Y. Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In Advances in neural information processing systems, pp. 2962-2970, 2017.
A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. 3- V-VAE: Learning Basic Visual Concepts with Constrained Variational Framework. ICLR, 2017.
W. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang, Y. Cao, Y. Jia, Z. Chen, J. Shen, et al. Hierarchical generative modeling for controllable speech synthesis. International Conference On Learning Representations, 2018.
A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural Networks, 1999.
M. Ilse, J. M. Tomczak, C. Louizos, and M. Welling. DIVA: Domain invariant variational autoen- coders. arXiv preprint 17X1v:1905.10427. 2019.
N. Kalchbrenner, E. Elsen, K. Simonyan, S. Noury, N. Casagrande, E. Lockhart, F. Stimberg, A. Van den Oord, S. Dieleman, and K. Kavukcuoglu. Efficient neural audio synthesis. arXiv preprint
2018.
H. Kim and A. Mnih. Disentangling by factorising, arXiv preprint arXiv:1802.05983. 2018.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 2013.
D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling. Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
R. Kubichek. Mel-cepstral distance measure for objective speech quality assessment. Proceedings of IEEE Pacific Rim Conference Communications Computers and Signal Processing, 1993.
10

---

Published as a conference paper at ICLR 2020
F. Locatello, S. Bauer, M. Lucic, G. Ractsch, S. Gelly, B. Schölkopf, and O. Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. ings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 41 4124,,,, Long Beach, California, USA, 2019. PMIR.
Proceed-
M. F. Mathieu, J. Zhao. J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun. Disentangling factors of variation in deep representation using adversarial training. In Advances in Neural Information Processing Systems, pp. 5040-5048 2016.
E. Munoz-de Escalona and J. J. Canas. Online measuring of available resources. In H- W-Worklood 2017: The first international symposium human mental workload. Dublin Institute of Technol- ogy. 2017.
S. Narayanaswamy, B. TT. Paige, J. Van de Meent, A. Desmaison, N. Goodman, P. Kohli, F. Wood, and P. Torr. Learning disentangled representations with semi-supervised deep generative models. In Advances Neural Information Processing Systems, pp. 5925-5935, 2017.
W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, J. Raiman, and J. Miller. Deep voice Scaling text-to-speech with convolutional sequence learning. arXiv preprint 1rXiv:1710.076544 2017.
Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Zhao, and T. Liu. FastSpeech: Fast, robust and controllable text-to-speech. arXiv preprint arXiv: 2019.
D. Rezende, S. Mohamed, and D. Wierstra. Stochastie backpropagation and approximate infer- ence in deep generative models. In Proceedings of The International Conference on Machine Learning. 2014.
J. A. Russell, A circumplex model of affect. Journal of personality and social psychology, 1980.
Salimans, Karpathy, X. Chen, and D. P. Kingma. PixeICNN++: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv: 1701005777 2017.
M. Schröder. Emotional speech synthesis: A review. In Seventh European Conference on Speech Communication and Technology, 2001.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, and R. Skerrv-Ryan, Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
R. Kkerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang. D. Stanton, J. Shor, R. Weiss, R. Clark, and R. A. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with Tacotron. In Proceedings of the 35th International Conference Machine Learning, 2018.
Y. Taigman, L. Wolf, A. Polyak, and E. Nachmani. Voiceloop: Voice fitting and synthesis via phonological loop. arXiv preprint arXiv:1707.06588. 2017.
A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner. A. Senior, and K. Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03449, 2016.
S. Vasquez and M. Lewis. MelNet: A generative model for audio in the frequency domain. arXiv preprint
2019.
V. M. Velichko and N. G. Zagoruyko. Automatic recognition of 200 words. International Journal of Man Machine Studies, 2(3):223-22 22222, 1970.
V. Wan, C. Chan, T. Kenter, J. Vit, and R. Clark. CHiVE: Varying prosody in speech synthesis with a linguistically driven dynamic hierarchical conditional variational network. arXiv preprint arXiv: 190.00090 2019.
11

---

Published as a conference paper at ICLR 2020
Y. Wang. R. Skan,y-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10115, 2017.
Y. Wang, D. Stanton, Zhang, R. Skanry-Ryan, E. Battenberg, Shor, Y. Xiao, F. Ren, Y. Jia, and R. A. Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. arXiv preprint arXiv:1803.090 090177 2018.
Wu, Z. Ling, Liu, Jiang, H. Wu, and Dai. End-to-end emotional speech synthesis using style tokens and semi-supervised training, arXiv preprint arXiv: 1906. 10859, 2019.
H. Zen, V. Dang. R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu. LibriTTS: A corpus derived from LibriSpeech for text-to-speech. In Interspeech 2019, 2019.
12

---

Published as a conference paper at ICLR 2020
A
NEURAL NETWORK ARCHITECTURE
Module
hyperparameters
Input
Text normalized phonemes
Phoneme embedding
256-D
Pre-net
1-226--elu-Droopuu(O1.
7 12-128-Relu-Dropoutee
CBHG text encoder
ConvI ID bank: K=16, conv-k-128-RRe
Max pooling with stride=1 width=2
ConviD projections: conv-3-128-Ren
conv-3-128-Linees
Highway net: 4 layers of FC-128-Reih
Bidirectional GRU: 128 cells
Attention type Attention RNN
5 component GMM attention w/ softplus Graves 2013)
LSSTM-256-Zoneouut(.11 FC-128-tan
DecoderRNN
2- 2-layer residull-LSTMM-25550oooo
+ FC-80-Linnear
Frames-per-timestte (reduction factor) WaveRNN
2
5 layers DilatedCony ID-512
+ 2 layers TransposeConv + ReLu
7 GRU-768 conditioned on 5 previous samples FC-768-relu
+ 3 component MoL, 24kHz sample rate
Variational posterior
Spectrogram
6 Conv-layers 32-32-64-664-12--122
+ LSTM-128
FC-128-tanb
Optimizer
ADAM with learning rate 10-3, batch-size 256
Speaker embedding
64-D
Table 4: Summary of the hyperparameters described below.
Sequence-to-sequeecc model Our peruuence-to-equeeee network is modelled Tacotron (Wang) ett al. 2018) but uses some modifications introduced in (Skery--yyar et al. (2018). Input to the model consists of sequences of phonemes produced by inputs. The CBHG text encoder from Wang et al. (2017) is used to convert the input phonemes into a sequence of text embeddings. The phoneme inputs are converted learned 256-dimensional embeddings and passed through pre-net composed of two fully connected ReLU layers (with 256 and 128 units, respectively), with dropout of 0.5 applied to the output of each layer, before being fed to the encoder. For multi-speaker models, learned embedding for the target speaker is broadcast- concatenated to the output of the text encoder. The attention module uses a single LSTM layer with 256 units and zoncout of 0. followed by an MLP with 128 tanh hidden units compute parameters for the monotonic 5-component GMM attention window. We use GMMv2b attention mechanism described in (attteberg et al. (2020). Instead of using the exponential function to compute the shift and scale parameters of the GMM components as in Graves (2013). GMMv2vb uses the softplus function, and also adds initial bias to these parameters, which we found leads to faster alignment and more stable optimization. The attention weights predicted by the attention network are used to compute weighted sum of output of the text encoder, producing a context vector. The context vector is concatenated with the output of the attention LSTM layer before being passed to the first decoder LSTM layer. The autoregressive decoder module consists of 2 LSTT layers each with 256 units, zoneout of 0.1, and residual connections between the layers. The spectrogram output is produced using a linear layer on top of the LSTM layers, and we use a reduction factor of 2, meaning we predict two spectrogram frames for each decoder step. The decoder is fed the last frame of its most recent prediction (or the previous ground truth frame during training) and the current context as computed by the attention module. Before being fed to the decoder, the previous prediction is passed through a pre-net with the same same structure used before the text encoder above but its own parameters.
normalization pipeline rather than character
13

---

Published as a conference paper at ICLR 2020
CBHG text encoder We reuse the CHGB text encoder introduced in Wang al. (2018). The text encoder consists of a bank of I-D convolutional filters, followed by highway networks and a bidirectional gated recurrent unit (GRU) recurrent neural net (RNN). The input sequence is first convolved with sets of I-D convolutional filters, where the k-th set contains Cę filters of width k. The convolution outputs are stacked together and further the original paper we use a stride of I to preserve the original time resolution. We further pass the processed sequence to few fixed-width 1-D convolutions, whose outputs are added with the original input sequence via residual connections. Batch normalization is used for all convolutional layers. The convolution outputs are fed into a multi- multi-layer highway network to extract high-level features. Finally, we stack bidirectional GRU RNN on top to extract sequential features from both forward and backward context.
pooled, preserving time. As in
Variational posteriors The variational distributions g(za| x; y) and ((zu [2, y; 2.) are both struc- tured as diagonal Gaussian distributions whose mean and variance are parameterized by neural net- works. For discrete supervision we replace g(=g|>r, y) by a categorical distribution and use the same network to output just the mean. The imput to the distribution starts from the mel spectrogram z and passes it through a stack of 6 convolutional layers, each using ReLU non-linearities 3x3 filters, 2x2 stride, and batch normalization. The 6 layers have 32, 32, 64, 64, 128, and 128 filters, respectively. The output of this convolution stack is fed into a unidirectional LSTM with 128 units. We pass the final output of this LSTM (and potentially vectors describing the text and/or speaker) through an MLP with 128 tanh hidden units to produce the parameters of the diagonal Gaussian posterior which we sample from. All but the last linear layer of these networks is shared between the two to the
distributions g(z,|22, y) and q(zu|z, y. zz) The resulting sample is output of the text encoder. In our experiments zn always 32-dimensional and 2g either a one-hot vector across 6 classes or dimensional continuous value.
Conditional inputs When providing information about the text to the variational posterior, we pass the sequence of text embeddings produced by the text encoder to a unidirectional RNN with 128 units and use its final output as a fixed-length text summary that is passed to the posterior MLP. Speaker information is passed to the posterior MLP via a learned speaker embedding.
WaveRNN We used a WaveRNN model similar to that described in Kalchbrenner et al. (0018) 8) as vocoder. Our WaveRNN uses discretized mixture of logistics output described in Salimans ctt al. (201)) instead of the dual softmax from that paper, and conditions on 5 previous samples each step instead of only previous. We trained the network map from synthesized mel- spectrograms to waveforms, training on 900 sample windows. A conditioning stack of dilated con- volution and transpose convolutions is applied to the input spectrogram before tiling upsample to the audio sample rate.
B EVALUATION
Mel spectrograms The mel spectrograms the model predicts are computed from 24 kHz audio using a frame size of 50 ms, a hop size of 12.5 ms, an FFT size of 2048, and a Hann window. From the FFT energies, we compute 80 mel bins distributed between 80 Hz and 12 kHz.
MCD-DTW To compute mel cepstral distortion (MCD) (Kbbcchek) 1993), we use the same mel spectrogram parameters described above and take the discrete-cosine-transee 13 MFCCs (not including the Oth coefficient). The MCD between two frames is the Euclidean distance between their MFCC vectors. Then we use the dynamic time warping (DTW) algorithm (Velichko & Zagoruyko 1970) (with warp penalty of 1.0) to find an alignment between two spec- trograms that produces the minimum MCD cost (including the total warp penalty). We report the average fer-frame MCD-DTW.
to compute the first
Affect classifier The affect classifier has a very similar structure to the variational posterior. The input to the classifier starts from the mel spectrogram and passes it through a stack of convolu- tional layers, each using ReLU non- lincaritissss, 3x3 filters, 2x2 stride, and batch normalization. The 6 layers have 32, 32, 64, 64, 128, and 128 filters, respectively. The output of this convolution stack is
14

---

Published as a conference paper at ICLR 2020
fed into unidirectional LSTM with 128 units. The final output of the LSTM is then passed through a softmax non- non-linearity to get logits over the training classes. We use the same data splits described in section B to train and evaluate the classifier. The classifier is tuned on the validation set achieving 84.33% classification accuracy, generalizing well to the test set with 83.94% accuracy.
Mean opinion scores We use a human rating service similar to Amazon's 's Mechanical Turk, with a large pool of English speakers to collect MOS evaluations. The MOŞ template shown in figure 5 A human rater is presented with single speech sample and is asked to rate perceived naturalness on a scale male, and one female speaker in test set, totalling 371 utterances evaluate. For each sample, we collect rating, and no rater is used for more than items single evaluation set. total, 270 unique raters completed the evaluation sets presented in table Since raters randomly selected for each set, some raters have assessed multiple methods. Across the evaluation sets, the average and median of total number of ratings per rater 8.24, and 6, respectively. To analyze the data from these subjective tests, we average the scores and compute 95% confidence intervals. Natural human speech typically rated around 4.5. Samples used for MOS from our model were drawn using the mean of 2n, whilst sampling ž..
to 5, where is "Bad" and is "Excellent". We have selected the utterances one
Instruction
IMPORIANT::
priject
Please this lask fontain-ood Soner, Tohrado,, talking
(1 You do not have hesophones
21
da have good atality There
4] For resson, you con't hear ausio complee
AUIDO DEVICE (heedpphrree))
1) There aro many types heodshonas. you hawa more than one tyse, this profemed order (al dlosdd-back headphones, (b) open back headshones, 66) any other typa headphones. iifieedda contuttatte swsi.
type you
2| Please vnturn
This should
unnatural
the contends sou inded. You should not judge the gemmer or the content of the sentence just how at sounds.
('eesw
21 Uso the given 5 point coale rato the noturalneds the speach sample, The kollownng table provides a descritee of each naturaines weel the scale, as well one more referenne nnternccee. Important note: you do newd laten refermnces if you have listened them bettrr.
speech sovemp sea) for wach flestew the bette and
In In-Bewoon Ratinga: Flease nole that you are lowad to assign "In-i "i-Setwenn" hulir go (oor example, c ralling betwaan "Excullert and Gocdfy, Fool free use them you think the quality of the cample
Netcnshwas
Score
Natura move
2.
Comal odl meetun up.00n
Liner
43
Good
Moall maturi thwcc
12
Fai
Figally satari and unuaraai
Poez
Madt unak kpawch
Bac
Compldddy unmduaa sporoh
are you listening spooch sample?
Headphonee, nates the background. am indening speech compie uaing hewtthonee tham rokee around (pwoole talling musts plyyng alrered foowrss lin Headphones, low-level
background. Istering
hendppones low-lovel
an-oordlorers and fare,
Audio speakors or other.
Speech sample pplease Beten at least twice)
0:00/ 52
+
naturainess
Dherzpptin
Excellent
Compltedl natural speech
3.0
Fair
Equally natural and unnatura spoech
2.0
Poor
Mosily unnatural speech
Completely speech
Comments
Figure 5: Mean opinion score (MOS) evaluation template. For each utterance, the human raters assign a 1-5 score of the perceived naturalness, with being "Bad" and 5 being "Excellent".
Subjective affect control evaluation We use the same rater pool, and the same set 371 utter- ances used for the MOS evaluations. The A/B template is shown in figure 6 For each utterance, the human rater is presented with a pair of utterances to choose the one that better conveys the target emotion (e.g., happy in the figure). Both utterances generated with the text. To evaluate the
15

---

Published as a conference paper at ICLR 2020
control valence, present baseline (i.e, control) against utterances generated in specific valence category (angry, happy and sad). To evaluate the control over arousal, we present samples generated at low arousal against samples generated at high arousal, and ask the rater to choose the utterance that is more vocally aroused. We use mean of zu to generate all the samples.
Instructions
IMPORTANT:
taok requiros you
samples using heaophonees quict envirormnnt.
Please rolease this lask
There beckground nesas,
3. You trink you do not have goodd ladwningg atiility or
For any resson, you can'. hear the mucho sumpless.
this laok, your baden two diflereni sudio sumpihes containing speech: speach sumples are intended to convey a particuter speaking style, lest spodee plee sampliss corney style.
selecting raitig. you
dffeerenco,
Listening conditions
How are you lrtening to theoe spench comples?
Haadshones, with no notve the background
background
IPPORTNT: you don't have headphones, please release this task the reason, do not meet the uptront requirements for this task."
Tasks
instructions
Please lsten to both samples before selecting ratingg. if you can't tell the diference, make quick intulivv guess.
speech
Happy?
Speech samples
0:00 / 0:02
0:00 0:02
Which sidee sounds more Mappy??
Bateer
Belier
Figure 6: A/B evaluation affect control evaluation template. The emotion label (Happy in the figure) varies depending on the task.
C SAMPLE SPECTROGRAMS
Controlling affect Table shows the effect of varying the valence and arousal latent variable on the spectrogram and FO track. We can see that a low valence for sadness corresponds to the flattest FO track, and high arousal manifests in higher FO values and variations.
Controlling speaking rate and pitch variations Table 6 shows the effeett of varying speaking rate, and FO variation control variables on sample spectrogram and FO track. When controlling speaking rate (first column), the duration reduces while the FO variation remains stable. When controlling the FO variation (second column), the pitch dynamic range increases, while the duration remains constant, which demonstrates controllablity and also some degree of disentanglement.
increase the input speaking rate control,
D REPRODUCING RESULTS ON LIBRITTS PUBLIC DATASET
To verify the reproduciblity of our results on a public dataset, we trained models to control speaking rate and FO variation clean subset of LibriTTS dataset (Zen et al.) 2019). We only use the utterances below seconds, which 62 hours of data. We have done no tuning on this dataset, and directly used the hyperparameters used for internal dataset. Figures 7a and 7h show the
16

---

Published as a conference paper at ICLR 2020
Low Arousal
High Arousal
Angry
10
Sad
150
Happy
Table 5: Sample spectrogram and FO track plots, generated by varying affect labels, valence in y-axis, and arousal in x-axis.
200na
0.45
Speakngnate rat bo dawe
0.40
0.35
0.30
0.25
8.8
0.05
Superdsion
(a) Speaking-rad rate error function of supervision level.
19
70 sndser ener
18
15
4.8
0.5
0.1
Supertidin fractinnd
(b) FO variation error as a function of supervision level
Figure 7: Objective controllability of speaking rate and FO variation evaluation metrics presented
multiple supervision levels, on LibriTTS Zen et al. 2019) datasets. 100% supervision corresponds 62 hours of supervised data.
enrors of producing the desired speaking rate, and FO standard deviation, which generally go down as function of supervision level, with exception of 10% supervision for controlling FO variation $ Given, this is a lower quality dataset, with many more speakers and with much smaller data per speaker and also the fact that we have done zero hyperparameter tuning on this dataset, this result look very encouraging.
'We believe the result 10% he anomalous and likely due single had training run.
17

---

Published as a conference paper at ICLR 2020
Speaking Rate
2g FO Variation
1n
50
-30
330
-la
+la
+30
120
+50
3 500
Table 6: Sample spectrogram and FO track plots, generated by varying the speaking rate (first col- umn) and FO variation (second column). We use standard normal prior these factors and this table demos varying the control factor from -5o to 50, demonstrating the controllability. interpolation and extrapolation of conditional generation, and also disentanglement of these factors.
18


Reviews:
[{"review_id": "BygnsoO3Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use a semi-supervised VAE based text-to-speech (TTS) for expressive speech synthesis. The main contribution of this paper is that it can provide more explicit interpretation for the latent variable with the help of the supervised learning component. The formulations and implementations in the main body are quite high-level and we may not easily understand the technical/implementation details only with the main body but, in other words, the paper is well written to convey their main messages and of course some details are described in the appendix. The experiments show the effectiveness in terms of subjective (MOS) and objective (cepstral distance etc.) with a lot of audio examples on the demo page.\n\nMy concern for this paper is a lack of reproducibility. The paper uses the in-house data to perform their experiments and the code does not seem to be publically available. Also, the paper misses several detailed information (e.g., detailed configurations of the Wave RNN vocoder, what kind of neural network toolkits and libraries). The high computational cost (\"distributed across 32 Google Cloud TPU chips\") would also make the reproducibility difficult. I also would like to see whether this method can have some experimental comparisons with (Hsu et al., 2018) with their postprocessing to show the distinction in terms of the performance in addition to the functional difference. \n\nComments:\n- In general, the font size in the figures is too small\n- Figure 1: it's better to have an explanation of \"CBHG\". People outside the end-to-end TTS community cannot understand it.\n- Can you also control the noise level as shown in (Hsu et al., 2018) but more explicitly within this framework? Controlling the noise level is quite important for end-to-end TTS, and I think this method can fit this direction because we can easily obtain the noise attribute (supervision) by data simulation or annotate the noise. \n- Section 2, second paragraph y_{1...t} --> y_{1...k} (?)\n- equation (6), classification loss: I think this part requires more clarifications in this timing, e.g., by giving an example of classification tasks.\n- I think it's better to add what kind of (neural) vocoder is used in the main body (not in the appendix) to asses the sound quality for their experiments.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygP7mHAtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to do neural text to speech, conditioned on attributes such as valence and arousal and speech rate. A seq-to-seq network is trained using stochastic gradient variational Bayes.        \nThe idea is interesting and new.\n\nThe method section could be made clearer by giving first some intuition, explaining the formulas in the prose and introducing the terms used.\n\nRegarding the crowd sourced MOS: how were the ratings obtained? how many subjects were used for the rating? How were they selected? Was each rater presented with samples from each method? Or was each method assessed by different groups of raters?\n\nThe experimental setting is a little weak, relying mostly on the Mean Opinion Score. It would be useful to include more evalution, for instance:\n* run a speech recognition method on the generated speech and measure the error rate\n\n* examples could be included in the supplementary  (e.g. spectrograms)\n\n* For the evaluation of emotional speech to be meaningful, the proposed classifier should be tested, and more detailed given (hyper-parameters, training setting, validation/test splits?, etc). In particular, it would be useful to compare the proposed method for affect classification to an existing (state-of-the-art) methodology. A state-of-the-art method should be ideally be directly used to classify the generated sequences into emotional classes.\n\nThe authors collected data in studio conditions, as such it is hard to compare. Will the data be released? How much hdata was collected?\nIt seems that what the authors effectively do is condition on discrete emotion classes, not valence and arousal, which are continuous measures of affect.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkgb0FksYS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\n\nThis paper proposes to use semi-supervised learning to enforce interpretability on latent variables corresponding to properties like affect and speaking rate for text-to-speech synthesis. During training, only a few training items are annotated for these types of properties; for items where these labels are not given, the variables are marginalised out. TTS experiments are performed and the approach is evaluated objectively by training classifiers on top of the synthesised speech and subjectively in terms of mean opinion score.\n\nI should note that, although I am a speech researchers, I am not a TTS expert, and my review can be weighed accordingly.\n\nStrengths:\n\nThe proposed approach is interesting. I think it differs from standard semi-supervised training in that at test time we aren't explicitly interested in predicting labels from the semi-supervised labelled classes; rather, we feed in these labels as input to affect the generated model output. I agree that this is a principled way to impart interpretability on latent spaces which are obtained through unsupervised modelling aiming to disentangle properties like affect and speaking rate.\n\nWeaknesses:\n\nThis work misses some essential baselines, specifically a baseline that only makes use of the (small number of) labelled instances. In the experiments, the best performance is achieved when gamma is set very high, which (I think) correspond to the purely supervised case (I might be wrong). Nevertheless, I think a model that uses only the small amount of labelled data (i.e. without semi-supervised learning incorporating unlabelled data) should also be considered.\n\nAs a minor weakness, the evaluation seems lacking in that human evaluations are only performed on the audio quality, not any of the target properties that are being changed. For affect specifically, it would be helpful to know whether the changes can be perceived by humans. As a second minor weakness, some aspects of the paper's presentation can be improved (see below).\n\nOverall assessment:\n\nThe paper currently does not contain some very relevant baselines, and I therefore assign a \"weak reject\".\n\nQuestions, suggestions, typos, grammar and style:\n\n- p. 1: \"control high level attributes *of of* speech\"\n- p. 2: It would be more helpful to state the absolute amount of labelled data (since 1% is somewhat meaningless).\n- p. 2: I am not a TTS expert, but I believe the last of your contributions have already been achieved in other work.\n- Figure 2: It would be helpful if these figures are vectorised.\n- p. 4: \"*where* summation would again ...\"\n- Figure 4: Is there a reason for the gamma=1000 experiment, which performs best in (a), not to be included in (b) to (d)?\n- Section 5: Table 1 is not references in the text.\n- Section 5.1: \"P(x|y,z_s,z_u)\" -> \"p(x|y,z_s,z_u)\"\n- In a number of places, I think the paper meant to cite [1] but instead cited the older Kingma & Welling (2013) paper; for instance before equation (6) (this additional loss did not appear in the original VAE paper).\n\nReferences:\n\n[1] https://arxiv.org/abs/1406.5298\n\nEdit: Based on the author's response, I am changing my rating from a 'weak reject' to a 'weak accept'.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to enforce interpretability and controllability on latent variables, like affect and speaking rate, in a speech synthesis model by training in a semi-supervised way, with a small amount of labeled data with the variables of interest labeled.   The idea is sensible and the results are very encouraging, and the authors have addressed the initial concerns brought up by the reviewers.
Decision: Accept (Poster)