Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
Authors: Rafael Valle
Kevin J. Shih
Ryan Prenger
Bryan Catanzaro
Venue: ICLR 2021 Poster (2021)
Keywords: Text to speech synthesis
normalizing flows
deep learning

Abstract: In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis with style transfer and speech variation. Flowtron borrows insights from Autoregressive Flows and revamps Tacotron 2 in order to provide high-quality and expressive mel-spectrogram synthesis. Flowtron is optimized by maximizing the likelihood of the training data, which makes training simple and stable. Flowtron learns an invertible mapping of data to a latent space that can be used to modulate many aspects of speech synthesis (timbre, expressivity, accent). Our mean opinion scores (MOS) show that Flowtron matches state-of-the-art TTS models in terms of speech quality. We provide results on speech variation, interpolation over time between samples and style transfer between seen and unseen speakers. Code and pre-trained models are publicly available at \href{https://github.com/NVIDIA/flowtron}{https://github.com/NVIDIA/flowtron}.

Published as a conference paper at ICLR 2021
AN AUTOREGRESSIVE FLOW-BASED GENERATIVE NETWORK FOR TEXT-TO-SPEECH SYN- THESIS
FLOWTRON:
Rafael Valle, Kevin J. Shih, Ryan Prenger & Bryan Catanzaro NVIDIA
rafaeelvalleenvidia.coo
ABSTRACT
In this paper we propose Flowtron: an autoregressive flow-based generative network for text-to-speede synthesis with style transfer and speech variation. Flowtron borrows insights from Autoregressive Flows and revamps Tacotron 2 order to provide high-quality and expressive mel-spectrogram synthesis. Flowtron is optimized by maximizing the likelihood of the training data, which makes training simple and stable. Flowtron learns invertible mapping of data to a latent space that can be used to modulate many aspects of speech synthesis (timbre, expressivity, accent). Our mean opinion scores (MOS) show that Flowtron matches state-of- the-art TTS models in terms of speech quality. We provide results on speech variation, interpolation over time between samples and style transfer between seen and unseen speakers. Code and pre-trained models are publicly available at https:/ggitub.ooo//////////oto
INTRODUCTION
Current speech synthesis methods do not give the user enough control over how speech actually sounds. Automatically converting text to audio that successfully communicates the text was achieved a long time ago (Umeda et al.) 1968 Badham al.) 1983). However, communicating only the text information leaves out the acoustic properties of the voice that convey much of the meaning and human expressiveness.
spite of this, the typical speech synthesis problem is formulated as a text to speech (TTS) problem in which the user inputs only text since the 1960s. This work proposes normalizing flow model (Kingma & Dhariwal 2018 Huang et al. 2018) that learns an unsupervised mapping from non-textual information manipulable latent Gaussian distributions.
Taming the non-textual information in speech is difficult because the non-textual is unlabeled. A voice actor may speak the same text with different emphasis or emotion based context, but it is unclear how to label a particular reading. Without labels for the non-textual information, recent approaches (Shen al. 2017 Arik al.) 2017a b) PPig et al. 2017) have formulated speech synthesis as TTS problem wherein the non-textual information is implicitly learned. Despite their success in recreating non-textual information in the training set, the user has limited insight and control over
It is possible to formulate an unsupervised learning problem in such a way that the user can exploit the unlabeled characteristics of a data set. One way is formulate the problem such that the data is assumed to have a representation in some latent space, and have the model learn that representation. This latent space can then be investigated and manipulated to give the user more control over the generative model's 's output. Such approaches have been popular in image generation, allowing users to interpolate smoothly between images and to identify portions of the latent space that correlate with
various features (Radford et al. 2015 Kingma & Dhariwal 2018 Izmailov al. 2019).
Recent deep learning approaches to expressive speech synthesis have combined text and learned latent embeddings for non-textual information (Wang al. 2018 Skerry-Ryan et al. 2018 Hsu et al.) 2018 Habib et al. 2019; Sun et al. 2020). These approaches impose an undesirable paradox: they require making assumptions before hand about the dimensionality of the embeddings when the correct dimensionality can only be determined after the model is traine.. Even then, these embeddings not guaranteed to contain all the non-textual information it takes to reconstruct speech, often

---

Published as a conference paper at ICLR 2021
-0
(a) Time-averaged valuuess from multiple sam- ples from speakers with different timbres. + the centroid computed over samples from
(b) Time-averaged =-values from multi-
ple
samples from 123 LibriTTS speak-
ers. Each color represents speaker.
the same speaker.
+ is Male (lower Fo, third quadrant).
is Female (higher Fo, first quadrant).
Figure 1: T-SNE plot showing Flowtron partitioning the s-space according to acoustic characteristics.
resulting models with dummy uninterpretable latent dimensions and enough capacity, the appendices in Wang et al. 2018); Skerry-Ryan et al. (2018) Hsu et al. (2018) confirm.
Furthermore, most models
time due fixed- length embeddings. Their assumption is that variable-length embeddings are not robust to text and speaker perturbations (Skerry-Ryanl
able to manipulate speech characteristics
al. (2018), which we show not be the case. Finally, although VAES and GANs (Sun et al. 2020)] Habib et al. 2019] Hsu et al. 2018 Binkowski et al., 2019 Akuzawa al. 2018) provide a latent embedding that can be manipulated, they may difficult to train, are limited to approximate latent variable prediction, and rely on an implicit generative model
or ELBO estimate to perform MLE in the latent space (Kingma Dhariwal
2018
Lucic et al. 2018
Kingma et al. 2016).
In this paper we propose Flowtron: an autoregressive flow-based generative network for mel- spectrogram synthesis with style transfer over time and speech variation. Flowtron learns invertible function that maps a distribution over mel-spectrograms to a latent z-space parameterized by spherical Gaussian. Figure shows that acoustic characteristics like timbre and Fo correlate with portions of the z-space of Flowtron models trained without speaker embeddings.
With our formalization, can generate samples containing specific speech characteristics manifested in mel-space by finding and sampling the corresponding region in z-space (Gambardella et al.) 2019). Our formulation also allows to impose structure to the z-space and parametrize it with
Gaussian mixture, similar to Hsu et al. (2018). In our simplest setup, we generate samples with zero mean spherical Gaussian prior and control the amount of variation by adjusting its variance.
Compared to VAEs and GANs and their disadvantages enumerated in Kingma & Dhariwal (2018) manipulating latent prior in Flowtron comes at no cost in speech quality nor optimization challenges. Flowtron is able to generalize and produce sharp mel-spectrograms, even at high x² values, by simply maximizing the likelihood of the data while not requiring any additional Prenet or Postnet layer (Wang et al. 2017). nor compound loss functions required by most SOTA models (Shen et al. 2017 2017 Ryan al. 2018 Wang et al. 2018 Binkowski al. 2019).
Ping et
In summary, Flowtron is optimized by maximizing the exact likelihood of the training data, which makes training simple and stable. Using normalizing flows, learns an invertible mapping from data to latent space that can be manipulated to modulate many aspects of speech synthesis. Concurrent with this work are Glow-TTS (Kim et al. 2020) and Flow-TTS (Miao et al.) 2020). both of which incorporate normalizing flows into the TTS task. Our work differs from these two in that Flowtron an autoregressive architecture where we explore the use of flow to modulate speech and style variation. In contrast, Glow-TTS and Flow-TTS are parallel architectures that focus inference
2

---

Published as a conference paper at ICLR 2021
speed. Our mean opinion scores (MOS) show that Flowtron matches SOTA TTS models in terms of speech quality. Further, we provide results on speech variation, interpolation between samples and interpolation between styles over time, and style transfer between equal or different sentences. We hope this work, the first to show evidence that normalizing flows can be used for expressive text-to-speech synthesis and style transfer, will further stimulate developments in normalizing flows.
and unseen speakers with
2 FLOWTRON
Flowtron is an autoregressive flow that generates a sequence of mel-spectroggam frames. A normaliz- ing flow generates samples by first sampling latent variable from a known distribution p(z), and applying a series of invertible transformations to produce a sample from the target distribution p(æ). These invertible transformations f are known steps of flow:
f, 0... fz(z)
Because each transformation is invertible, we can directly evaluate the exact log-likelihood of the target distribution p(z) using the change of variables:
(x)) = log (z) + log det()
(2)
x = fi o f .... ......f' (x)
(3)
Where J is the Jacobian of the inverse transform Si (x)) By cleverly choosing the latent distribution p(z) and the invertible transformations, the exact log-likelihoon becomes simple and tractable.
2.1 LATENT DISTRIBUTIONS
We consider two simple distributions for the latent distribution zero mean spherical Gaussian and mixture of spherical Gaussians with fixed or learnable parameters.
~ ~(x;00 or x~ En Ex)
(4)
The zero-mean spherical Gaussian has simple lieliho.. The mixture of the spherical Gaussians, has inherent clusters that might result interesting aspects the audio information.
2.2 INVERTIBLE TRANSPORMATIONS
Normalizing flows are typically constructed using coupling layers (Dinh et al.) 2014 2016 Kingma Dhariwal 2018). In our case, we use an autoregressive affine coupling layer (Dinh et al. al, (0166).
The latent variable has the same number of dimensions and frames the resulting mel-spectrogram sample. The previous frames 11a-1 produce scale and bias terms, 8t and by respectively, that fffine-transfor the succeeding time step z1:
(log St, b.) N, speaker) f(x1) = (z: bi) + f (z,)
(5)
(6)
b:
(7)
Here, .N(() can be any autoregressive causal transformation (Shumway & Stoffer 2017). The affine coupling layer is reversible transformation, even though NN() itself need not be invertible. We use a O-vector for obtaining the scaling and bias terms what will affine transform 21. This O-vector constant also guarantees that the first is always known.

---

Published as a conference paper at ICLR 2021
x
Final x
Decoder
xK
Inverse Step Flow
Affine
Afinn
Xirm
Worm
Altention
Adention
Lanu
= cal(text emb, speaker emb)
Figure 2: Mapping data (x) to the latent dimension (z) for steps of flow. The right side shows a unrolled view of the NN() architecture. Text and speaker embeddings are channel-wise concatenated to produce context matrix c. At every time step i, a recurrent attention mechanism computes weighting distribution over the context matrix produce weighted-sum reduction over c, which is then passed through an LSTM-Come decoder architecture to generate affine parameters for transforming 2++1 2 As predicted parameters are always for the next time step, the first iteration is conditioned on pre-defined O-vector.
With an affine coupling layer, only the 8, term changes the volume of the mapping and adds a change of variables term to the loss. This term also penalizes the model for non-invertible affine mappings.
log [e (ooulingg (x))) log [s|
To evaluate the likelihood, we take the mel-spectrograms and pass them through the inverse steps of flow conditioned on the and optional speaker ids, adding the corresponding log [s] penalties, and evaluate the result based on the Gaussian likelihoods.
With this setup, it is also possible reverse the ordering of the mel-spectrogram frames in time without loss of generality. We reverse the order of frames on even steps of flow, defining a step of flow as full pass over the input sequence. This allows the model to learn dependencies both forward and backwards in time while remaining causal and invertible.
2.3 MODEL ARCHITECTURE
Our text encoder modifies the text encoder in Tacotron by replacing batch-norm with instance-norm. Our decoder and N architecture, depicted in Figure removes Prenet and Postnet layers from Tacotron previously thought to be essential (Shen al. 2017). Please compare Figure 2 describing our architecture and Figure A.4.4 descrriin Tacotron's 's architecture. We also provide model summary views in A.6
We use the content-basee tanh attention described in Vinyals et al. (2015), which can be easily modified to become also location sensitive. We use the Mel Encoder described in Hsu et al. (2018) to predict the parameters of the Gaussian Mixture. Following (Valle et al. 2019b), we use speaker- embeddings channel-wise concatenated with the encoder outputs at every token. We use single shared embedding for models not conditioned on speaker id.
The step of flow closest the latent variable has gating mechanism that prunes extra frames from the z-values provided to the model during inference. The length of z-values remains fixed on the next steps of flow.
2.4 INFERENCE
Inference, given trained model, simply matter of sampling z values from spherical Gaussian. or Gaussian Mixture, and running them through the network in the forward direction f. e.g. g. Eq. The parameters of the Gaussian mixture are either fixed or predicted by Flowtron. Training was
conducted with o² = 1, but we explore the effects of different values for o² in section 3.3 In general, we found that sampling z from a Gaussian with lower standard deviation than used during training resulted in better sounding mel-spectrograms, similarly concluded in Kingma & Dhariwal (2018) and Parrar et al. 2018). Our inference results use "² = 0.5 while sampling the prior and the posterior variance while sampling the posterior.
4

---

Published as a conference paper at ICLR 2021
2.5 POSTERIOR INFERENCE
Figure [ shows that several speech characteristics present in mel-spectrograms are clustered into regions of the z-space. Knowing this, we can treat the latent distribution prior g(z) = No, I) and obtain a posterior over the latent space of the flow model g(=|S1.m) conditioned on the evidence
Sim which are m data observations I; mapped to the latent space using C: f (x:). We can use Gaussian likelihood function with covariance matrix > compute the posterior above analytically, (2|( 1.mm = =N(Hpp Σp). Following the approach in Gambardella et al. (2019), defining Ç as the mean of Ç and using \ as a hyperparameter, we define the parameters of the posterior below. Please
see A.2 Algorithm and Gambardella et al. (2019) for implementation details and full derivation.
m
C Dp
(9)
m X
1
3 EXPERIMENTS
This section describes our training setup and provides quantitative and qualitative results, Our quantitative results show that Flowtron has mean opinion scores that are comparable to the state of the art. Our qualitative results demonstrate many features that are either impossible or inefficient to achieve using Tacotron, Tacotron GST and Tacotron GM-VAE. These features include variation control in speech, interpolation between samples, and style transfer over time.
We decode all mel-spectrograms into waveforms with WaveGlow (Prenger al. 2019) model available on github (Valle et al. 2019a).. This suggests that WaveGlow can be used as an universal decoder. In addition our illustrated and quantitative results, ask that the readers listen to Flowtron samples in our supplementary materials corresponding to our qualitative experiments.
3.1 TRAINING SETUP
We train Flowtron, Tacotron 2 and Tacotron GST models using dataset (LSH) that combines the LISpeech dataset (Ito et al. (017)) with two proprietary single speaker datasets with 20 and 10 hours each (Sally and Helen). We also train Flowtron model on the train-clean-106 100 subset of LibriTTS (Zen et al. (019) with 123 speakers and 25 minutes on average per speaker. Speakers with less than minutes of data and files that are larger than 10 seconds are filtered out. For each dataset, we use at least 180 samples for the validation set, and the remainder for the training set.
The models are trained on uniformly sampled normalized text and ARPAbet encodings obtained from the CMU Pronouncing Dictionary (Weide 1998). We do not perform any data augmentation. We adapt public Tacotron and Tacotron 2 GST repos to include speaker embeddings as described in Section 2] We use the same mel-speectrogram representation used in WaveGlow (Prenger et al.) 2019). We train Flowtron with a pre-trained text encoder, progressively adding steps of flow once the last step of flow has learned attend text. Flowtron models used in our experiments have 2 steps of flow. We forward readers to A.3 and A.4 for details on our training setup and ablation studies.
3.2 MEAN OPINION SCORE (MOS) COMPARISON
We use the LJS voice as a reference and compare MOS between real samples, samples from Flowtron with 2 steps of flow, and samples from Tacotron 2. Following guidelines in (Prenger et al., 2019). we crowd-sourced MOS tests on Amazon Mechanical Turk using 30 volume normalized utterances disjoint from the training set for evaluation, and randomly chose the utterances for each subject. The scores provided in (Prenger et al. 2019) are used for real samples.
The mean opinion scores are shown in Table with 95% confidence intervals computed over approximately 250 scores per source. The re- sults roughly match our subjective qualitative assessment. The larger advantage of Flowtron is in the control over the amount of speech vari- ation and the manipulation of the latent space.
Source
Flows
MOS
Real
4.27 - 0.13
Flowtron
2
3.66 - 0.16
Tacotron 2
3.52 = 0.17
Table 1: Mean Opinion Scores
5

---

Published as a conference paper at ICLR 2021
3.3 SAMPLING THE PRIOR
The simplest approach to generating samples with Flowtron is to sample from prior distribution (N of) and adjust a² control the amount of variation. Whereas o² 0 completely removes variation and produces outputs based on the model bias, mnrrasing o² will increase the amount of variation in speech.
3.3. SPEECH VARIATION
We illustrate the relationship between o² and control over variability by synthesizing Flowtron samples with o² E {0.0, 0.3, 1.0]. All samples are generated conditioned on the speaker Sally and the text "How much variation is there?". Despite the variability added by increasing o², all Flowtron-synthesized samples produce high quality speech.
Figure Bl shows that contrary to commonly held wisdom (Shen al., 2017 Arik et al. 2017a b Ping ct al. 2017; SSeeryy-Ryee et al. 2018:) Wang et al.) 2018 Binkowski et al. 2019). Flowtron generates
sharp harmonies and well resolved formants without compound loss Prenet Postnet layers.
(b) 0.5
Figure 3: Flowtron Mel-spectrograms illustrate increasing variability by using different o² and that
Flowtron is able produce sharp harmonies with high o² and without Prenet or Postnet layers.
Now we show that adjusting o² is a simple and valuable approach that provides more variation and control thereof than Tacotron, without sacrificing speech quality and despite of having a similar but simpler architecture. For this, we synthesize 10 samples with Tacotron 2 using different values for the Prenet dropout probability p E (0.45, 0.5, 0.55). scaling outputs accordingly. Samples computed on values of [0.3, 0.8] are not included because they sound unintelligible.
Figure provides plots of Fo contours extracted with the YIN algorithm (De Cheveigné Kawaharal 2002). with minimum Fo, maximum Fo, and harmonicity threshold equal to 80 Hz, 400 Hz and 0.3 respectively. Our results are similar to the previous sample duration analysis. expected, o² provides no variation in Fu contour"] while increasing o will increase variation in Fo contours.
Our results in Figure 4 also show that Flowtron samples are considerably less monotonous than the samples produced with Tacotron 2 at no cost and with similar but simpler architecture. Whereas increasing o² considerably increases variation Fg, modifying p barely produces any variation. This valuable because expressive speech is associated with non-monotonic Fo contours. In A.l we show similar results with respect to sentence duration.
3.3.2 INTERPOLATION BETWEEN SAMPLES
With Flowtron, can perform interpolation in z-space to achieve interpolation in mel-spectrogram space. This experiment evaluates Flowtron models with and without speaker embeddings. For the experiment with speaker embeddings, we choose the speaker Sally and the phrase "Il is well known that deep generative models have a rich latent space.", ". We generate spesttrcts by sampling (0, 0.8) twice and interpolating between them over 100 timesteps. For the experiment without speaker embeddings we interpolate between Sally and Helen using the phrase "We are testing this model.".
First, we perform inference by sampling ~ (O, 0.5) until we find values, and z,, that produce mel-spectrograms with Helen's and Sally's voice respectively. then generate samples by
Variatioos in a O are due different for WaveGlow.

---

Published as a conference paper at ICLR 2021
(a) Flowtron 0² =0
(b) Flowtron d 0.5
(c) Flowtron d²
(d) Tacotron 2 (0.45, 0.5, 0.55)
Figure 4: Fu contours obtained from samples generated by Flowtron and Tacotron 2 with different values for o² and Flowtron provides more variability and expressivity than Tacotron 2.
performing inference while linearly interpolating between 2n and 2s. Our same speaker interpolation samples show that Flowtron is able to interpolate between multiple samples while producing correct alignment maps. Our different speaker interpolation samples show that Flowtron is able to gradually and smoothly morph one voice into another.
3.4 SAMPLING THE POSTERIOR (STYLE TRANSFER)
We generate samples with Flowtron by sampling posterior distribution conditioned on the evidence containing speech characteristics of interest, as described in [2.5 and Gambardella et al. (019)). Tacotron 2 GST GSng et al. (2018) has equivalent posterior sampling approach. During inference, the model is conditioned on a weighted sum of global style tokens (posterior) queried through an embedding of existing audio samples (evidence). We evaluate Tacotron GST using single sample to query style token, or multiple samples to compute an average style token. For complete results, please refer audio samples in the supplemental material corresponding the following sections.
3.4. SEEN SPEAKER
In this section high variance in pitch, which we use as a proxy for comparing expressivity speech; the second (High Pitch), uses samples with high average pitch. In these experiments, provide comparisons between Pitch Mean and Pitch Standard Deviation from the Reference samples providing the style, Flowtron Baseline and after style transfer using Flowtron Posterior and Tacotron 2 GST.
two style transfer experiments: the first one (Expressive) samples with
Our experiments show that by sampling from the posterior or interpolating between the posterior and the Gaussian prior over time, Flowtron makes monotonic speaker gradually sound more expressive. Architectures similar to Tacotron GST with fixed-latent embeddings are not able to perform gradual changes in style over time. Table provides pitch summary statisties computed over phrases and 10 takes each and shows that Flowtron is overall closer to the reference providing the style than Tacotron GST. Our supplemental materials also show that Tacotron GST sentences are repetitive and contain vocal-fry like distortions.
Pitch Mean
Pitch Standard Deviation
Style
Expressive
High Pitch
Surprised
Expressive
High Pitch
Surprised
Model
Reference
53.6
55.2
58.2
4.5
2.3
1.0
FTA Posterior
53.4
53.3
55.5
2.5
2.3
3.0
FTA Baseline
53.1
52.6
52.8
2.2
1.9
1.9
Tacotron 2 GST
51.7
53.6
51.6
2.0
2.4
1.7
Table 2: Values closer to the Reference are better. Comparison between pitch (MIDI number) summary statistics from Reference providing the style, Flowtron with standard Gaussian prior (FTA Baseline) and samples after style transfer with Flowtron (FTA Posterior) and Tacotron GST. Our results show that FTA Posterior is overall more effective than Tacotron 2 GST in emulating the Reference by better matching its pitch summary statistics.
7

---

Published as a conference paper at ICLR 2021
3.4.2 SEEN SPEAKER WITH UNSEEN STYLE
We compare samples generated with Flowtron and Tacotron GST evaluate their ability emulate speaking style unseen during training of a speaker seen during training. While Sally's A data used during training consists of news article readings. the evaluation samples contain Sally's s interpretation of the somber and vampiresque novel, Born of Darkness (BOD).
Our samples show that while Tacotron GST fails to emulate the somber timbre in Born of Darkness, Flowtron succeeds in transferring not only the somber timbre, but also the low Fo and the long pauses associated with the narrative style.
3.4.3 UNSEEN SPEAKER
In this experiment we compare Flowtron and Tacotron GST samples to evaluate their ability to emulate the speaking style of speaker not seen during training. The styles comes from speaker ID 24 and her "surprised" samples in RAVDESS (Livingstone Russo 2018 a dataset with emotion labels. Table 2 shows that while the samples generated with Tacotron 2 GST are not able to emulate the high-pitched style from RAVDESS, Flowtron able to make Sally sound high-pitched as the "surprised" style.
3.5 INTERPOLATION BETWEEN STYLES (PRIOR AND POSTERIOR)
In this experiment illustrate how to control the speaking style at inference time by adjusting the parameter À in Equation 9 to interpolate between a baseline style (prior) and a target style (posterior). We model trained speaker) Born of Darkness dataset as evidence providing the target style. We synthesize posterior samples generated with Flowtron with À E (0.1, 0. 0.666, 1.0, 2.0). F the interpolation in style as interpolation in spectral profiles. Our supplemental materials aurally reflect similar interpolation in other non-textual characteristics.
LibriTTS and
single sample from Sally's 's (unseen
10
40
Flowtron Baseline
(b) Flowtron Posterior À
(c) Flowtron Posterior
43
(d) Flowtron Posterior À (c) Flowtron Posterior À 0.666
(f) Target Style
Figure 5: Spectral profiles from the targer style, from the Flowtron baseline generated using the prior, and from Flowtron samples generated using the posterior with different values for X. These images show that by decreasing the value of we gradually move the spectral profile from the baseline style (prior) the target style (posterior).
3.6 SAMPLING THE GAUSSIAN MIXTURE
In this last section we provide samples from Flowtron Gaussian Mixture (GM) and visualizations. We replicate the experiments in Tacotron GM-VAE (Hsu al. 2018) to visualize how speakers assigned to mixture components and provide samples in which we modulate speech characteristics by translating one of the dimensions of an individual mixture component.
For these experiments, Flowtron CM-LibriTTS is trained on LibriTTS without speaker embeddings and Gaussian mixture with 8 component with predicted mean, covariance and component assignment
8

---

Published as a conference paper at ICLR 2021
probabilities; Flowtron GM-LSH is trained on LSH with speaker embeddings and Gaussian Mixture with components, fixed mean and covariances and predicted component assignment probabilities.
3.6.1 VISUALIZING ASSIGNMENTS
We evaluate model interpretability on subset LibriTTS with 123 speakers and 1410 utterances, 180 of which come from the validation set. Following Hsu et al. (2018), each utterance is assigned
to the component with the highest posterior probability arg 4 p(ox x). We obtain posterior probabilities per utterance by using the Mel Encoder described in Section [2.3 and averaging the predicted component assignment probabilities over time. Figure suggests that information in each
component of Flowtron GM-LibriTTS is gender dependent.
We quantify the association between gender and mixture components with the metric described in Hsu N al. (2018). The assignment consistency with respect to gender is defined as M Ex=1 Σソ= 18
ye. where IV is the number of utterances, V/j is the component assignment of utterance from speaker i, and ie is the mode of { The assignment consistency in Flowtron GM-LibriTTS 82.4%, suggesting that the components group utterances by speaker and group speakers by gender. We
provide visualizations in Figure
Assignment distribution over component and gender
0.6
Probabilitt
Female
Male
0.2
7
Component
Figure 6: Component assignments suggest that information in each component is gender dependent.
3.6.2 TRANSLATING DIMENSIONS
We use the model Flowtron GM-LSH and focus on translating one of the dimensions of a single mixture component by adding offset. The samples in our supplementary material show that able to modulate specific speech characteristics like pitch and word duration. Although the samples generated by translating one the dimensions associated with pitch height have different pitch contours, they have the same duration. Similarly, our samples show that translating the dimension associated with length of the first word does modulate the pitch of the first word. We provide visualizations
in Figure in A.5
4 CONCLUSION
We propose new text to spectrrorog synthesis model based on autoregressive flows that is optimized by maximizing the likelihood and allows for speech variation and style transfer. Our results show that samples generated with Flowtron achieve mean opinion scores similar to SOTA TTS models. We demonstrate that our model learns a latent space that stores non-textual information, supervised using only MLE. Flowtron is able to produce high quality speech with high variability by adjusting o²
Our results show that the latent space over non-textual features that can be investigated and manipu- lated to give the user more control over the generative model's output. We provide many examples that showcase this, including transferring the style from speakers seen and unseen during training to another speaker using sentences with similar or different text, and making a monotonic speaker sound more expressive. For future work, we interested in using normalizing flows for few-shot speech synthesis, speech compression and in semi-supervised settings to exploit datasets with limited labels.

---

Published as a conference paper at ICLR 2021
REFERENCES
Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Expressive speech synthesis via modeling expressions with variational autoencoder. arXiv preprint wrXiv:180.0021355 2018.
Sercan Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. arXiv preprint arXiv: 170..0/70,.07255, 2017a.
Sercan Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. arXiv preprint arXiv:1705.08947, 2017b.
John Badham, Lawrence Lasker, Walter F Parkes, Arthur B Rubinstein, Matthew Broderick, Dabney Coleman, and John Wood. Wargames, 1983.
Mikotaj Binkowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. arXiv preprint arXiv: 1909. / 1646, 2019.
Alain De Cheveigné and Hideki Kawahara. Yin, fundamental frequency estimator for speech and music. The Journal of the Acousticul Society of America, 111(4)1117--1300, 2002.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 14X.v:1410.8516 2014.
Laurent Dinh, Jascha Dick-Dick,,,,,, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.0880; 2016.
Andrew Gambardella, Atilmm Günes Baydin, and Philip HS Torr. Transflow learning: Repurposing flow models without retraining. arXiv preprint arXiv:1911.13270, 2019.
Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, RJ Skerry-yaa,, Daisy Stanton, David Kao, and Tom Bagby. Semi-supervised generative modeling for controllable speech synthesis. arXiv preprint 1910:079,, 2019.
Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen. Jonathan Shen, et al. Hierarchical generative modeling for controllable speech synthesis. arXiv preprint arXiv:1810.07211 2018.
Chin-Wer Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. arXiv preprint vrXiv:1804.00779 2018.
Keith Ito et al. The LJ speech dataset, 2017.
Pavel Izmailov, Polina Kirichenko, Marc Finzi, and Andrew Gordon Wilson. Semi-supervised learning with normalizing flows. arXiv preprint arXiv:1912.13022 2019.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for
text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005.11129, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. arXiv preprint
2018.
Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp. 4743-4751 2016.
Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5), 2018.
10

---

Published as a conference paper at ICLR 2021
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systems, pp. 700-709, 2018.
C. Miao, S. Liang, M. Chen, J. Ma, S. Wang, and J. Xiao. Flow-tts: A non-autoregrressive network for text speech based on flow. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7209-7213, 2020.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. arXiv preprint arXiv:1802.0575 2018.
Wei Ping, Kainan Peng, Andrew Gibiansky. Sercan Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. arXiv preprint
arXiv:1710.07654, 2017.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 3617-3621. IEEE, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.06434 2015.
Jonathan Shen, Ruoming Pang. Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. RJ Skerry-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. arXiv preprint arXiv:1712.05580 2017.
Robert H Shumway and David S Stoffer. Time series analysis and its applications: with R examples. Springer, 2017.
RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron J Weiss, Rob Clark, and Rif A Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. arXiv preprint 1rXivv/803... 09047, 2018.
Guangzhi Sun, Yu Zhang, Ron Weiss, Yuan Cao, Heiga Zen, and Yonghui Wu. Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6264-6268. IEEE, 2020.
Noriko Umeda, E Matsui, Torazo Suzuki, and Hiroshi Omura. Synthesis of fairy tales using an analog vocal tract. In Proceedings of 6th International Congress on Acoustics, pp. B159-162, 1968.
Rafael Valle, Jason Li, Ryan Prenger, and Bryan Catanzaro. Mellotron github repo, 2019a. URL hhttps://gitmii
com/ NVID IA/ /e llotron.
Rafael Valle. Jason Li, Ryan Prenger, and Bryan Catanzaro. Mellotron: Multispeaker expres- sive voice synthesis by conditioning on rhythm, pitch and global style tokens. arXiv preprint arXiv: 1997, 2019b.
Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton, Grammar as foreign language. In Advances in neural information processing systems, pp. 2773-2781, 2015.
Yuxuan Wang, Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: A fully end-to-end text-to-speech synthesis model. arXiv preprint arXiv: 1703. 10135, 2017.
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ Skerry-Ryan Eric Battenberg, Joel Shor, Ying Xiao, Fei Ren, Ye Jia, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. arXiv preprint arXiv:/803.09m) 2018.
Robert L. Weide. The CMU pronouncing dictionary. URL: http://wwww.ppcch.........e...e. edd///// bin/cmudict, 1998.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss. Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904. 02882, 2019.
11

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 SPEECH VARIATION
Figure provides plots from sample durations in seconds. Our results show that larger values of o² produces samples with more variation in duration, whereas o² = 0 is fully deterministic. These results demonstrate that our latent space able to model duration, which is a critical non-textual component to expressiveness in speech.
10
0.6
0.5
000000
00 oocod Hogo
0.
ProeTron Tecotron
16
17
19
2.0
21
Figure 7:
Sample duration given o² and p show that Flowtron provides more variation in sample duration than Tacotron.
A.2 POSTERIOR INFERENCE
generate posterior samples with Flowtron by sampling a posterior distribution conditioned on evidence containing speech characteristics of interest, as described in (Gambardella et al. (019). We collect the evidence by performing a forward pass with Flowtron using with a speaker embed- ding, (s (N, I)), the observed mel-spectrogram, and the text from set of samples with the speech characteristics of interest. We use a specific speaker embedding when we want to factor out information about specific speaker from Ç.
Next, we compute Ç by averaging S,, over batch (i) or over batch and time (i, k) and use Equation compute the parameters of the posterior. When averaging over batch, repeat the z-values over the time dimension until they reach the desired length. We find in our experiments that averaging over batch is more efficient for transfering the style than averaging over batch and time. all experiments, we select the best performing samples given À values between m 0.1 and m * 4, where m is the number of samples in the evidence. While small À values move the mean of the posterior closer to the evidence and decreases its variance, large À values move the mean of the posterior closer to the prior and increase the variance.
Once the parameters of the posterior distribution are computed, we can sample the posterior distribu- tion and perform inference with the desired text and speaker. Algorithm | provides a description of posterior inference with Flowtron.
A.3 TRAINING DETAILS
We use the ADAM (Kingma & Ba 2014) optimizer with default parameters, 1e-4 learning rate andle-6 weight decay for Flowtron and 1e-3 learning rate and le-5 weight decay for the other following Wang et al. (2017). We anneal the learning rate once the generalization error starts toplateau and stop training once the the generalization enror stops significantly decreasing or startsincreasing. Flowtron models with 2 steps of flow were trained on the LSH dataset for approximately 1000 epochs, then tuned LibriTTS for 500 epochs. Tacotron and Tacotron GST are trained for approximately 500 epochs. Each model is trained on a single NVIDIA DGX-1 with 8 GPUs.
12

---

Published as a conference paper at ICLR 2021
Algorithm 1: Flowtron Posterior inference
Input:
Trained Flowtron model f, evidence audio samples 21.m Posterior sample
Output:
For each meli,k ; lexti, speaker; #1:mm
Si,k + f (meli, lerl;, speaker;)
if average over batch then
repeat each Ç& over the time dimension until target length is achieved
C f- Compute Si, average over batch k
else
Č Compute Gr, average over batch and time A:
end
Mp: Ep + Compute posterior parameters using Equation Initialize Zp ~ (H Ep)
Sample Zp from Zp
11 Perform inference with Flowtron using 2p, text and speaker
A.4 ABLATION STUDIES
A.4.1 COMPOSING FLOWS
We evaluated Flowtron models with 2, 3 and 6 steps of flow and found that more steps of flow have better likelihood but no significant qualitative improvement, while increasing inference time significantly. Hence, chose to report results on Flowtron models with steps of flow.
A.4.2 BIDIRECTIONAL PROCESSING
We compared the bidirectional (reversing the ordering of the mel-spectrogram frames in time on even numbered steps of flows) and unidirectional processing and found that bidirectional processing provides better likelihood and audio quality. Hence, use bidirectional processing in all our Flowtron models.
A.4.3 ADDITIVE VS AFFINE TRANSFORMATIONS
The Tacotron baseline without the postnet layer can be interpreted as additive single step autoregres- sive normalizing flow (ASSANF), By comparing Flowtron with Tacotron 2, we're comparing with model that is better than an (ASSANF), as Tacotron sans Postnet does not have sharp harmonics. Hence, we prefer affine over additive transformations.
A.4.4 COMPARISON WITH TACOTRON 2
The Tacotron 2 baseline without the postnet layer can be interpreted as additive single step autoregres- sive normalizing flow (ASSANF). By comparing Flowtron with Tacotron 2, we're comparing with model that is better than an (ASSANF), as Tacotron sans Postnet does not have sharp harmonics. Hence, we prefer affine over additive transformations.
13

---

Published as a conference paper at ICLR 2021
Postnot
Postnet
Postnot
Decoder
Decoder
Decoder
Tacotron Decoder
LaIW
Atlention
X
cat(text emb, speaker emb)
Figure 8: Visualization of the decoder in Tacotron 2 during training. Unlike Flowtron, Tacotron requires Prenet and Postnet layers to learn attention and produce sharp harmonics.
A.5 TRANSLATING DIMENSIONS
65.0
Timels)
(a) GMM Dimension Translation Altered Pitch Fixed Duration
50
1.30
(b) GMM Dimension Translation Altered Duration Fixed Pitch
Figure 9: (a) shows that by translating one of the dimensions of z we are able alter the pitch contour of the sentence while keeping the length fixed. (b) shows that by translation one of the dimensions of we able alter the length of the sentence while keeping similar pitch contour.
14

---

Published as a conference paper at ICLR 2021
A.6 FLOWTRON AND TACOTRON SUMMARY VIEW
#lowtron
Brbbedinn (3, 128)
tembedding:: Babedding
(flows) at
(0) AR_Step(
(conv): Convid(1024, 160, kenneLeizze=(.)) stride=(1,))
[1stm]: LS:M/1654, 1024, num_layerss
lattention_lstml: 1,8M188,, 1024)
tattention_layen Actentionn
[softmax): Roftmax (din=2)
(query): LinearNerm(
(Linear_layec): Sineertti__eestuuee--0222 out_ bias=Talse)
(key): Limeactiom
(1inear_layer) (inear((I_festurre out vut_features=640, biag=False)
(value): Linearlera(
(Linear_layer): Linear (1__testure--611) fet_reauues=-5 bias-Palae)
(v): Lineartorm!
(Linear_layer): (hearrtt_featuree-t out out_featuresel, blas=False)
Idense_layez): (enselayer(
(layers): ModuleList
(0) Lineartiorm(
(linear_layen Linear Linear(in (i0tturr11, f02turssses biss=Trus)
binearHormt
(Linser_layer) Lineer (in_ (e out fet_feeeures-1021, bias=True)
(1): AR_Back_Stee
(as_step): AR_Step(
(conv): convld(1024, 160, (L)),,,,, stride=(1,)) (1stm) 18TM(1664, 1024, num_layers=2)
(attention_lstm) 18TM(88, 1024) (attentior_layer) Attenticn(
(8oftmax): Softmax (dim=))
(queryl: LinescNozm( (Linear_layez): Lineer (ie_fetrrre,e, fut__eetures=600, bias=False)
(key): MinearMrmm
(Linear_layer): Linear (en_teetue,--66,, out_featurea-640, biaa-Falae)
(valuer: LinearNorm(
(Linear_layez): Lineer
out_[eatuuee=611, biss-False)
(v): timeartorm
(linear_layer): Linear
out_featurea=l, blag=False)
(denae_layer): DenseLayez
(layers): Modulelist(
(0): LinearMerm(
(1:mear_layer), tinear (Innfotinnttt fut_features=1124, bias=True)
(1): LinearNera( (Linear_layer) (in_aatiin_ out aut_"eatures-1224, blas=True)
gate__ayer]: MinearMorm
layee): Linear 164,,res116,, fea_ureaer,s-l, bias-True)
(encader): : Eneader
(convolutions): ModuleList
:0): Sequential(
(0): Conviorm(
(conv): (512 512, ke:nel_ stride=11, padding= (2,)
(1): (51, apa=1e-05, momentun=o.1,, affine=True, trock_rumnigg_tttteee
15

---

Published as a conference paper at ICLR 2021
Sequential
(0) Conviorm
(conv): (S12, 512, keznel_size=(55)) stride=11yl padding- [2,))
(1): Lnstancetiorm1d($12,,, eps=le-05, momentumo0.1, affinemtrue, trock_romming_sttttsevll..
12): Sequential
(0): Convtorm(
(cone): Conv 512, ke stride-11,1, padding (2,))
:1): (nstancellormdd(512,, eps-le-05, mom affine-True, track_running_steet stniing_tttts-Palsee
(1stm): 18TM(512, 256, Match_tirat-Trug rid
TacoLzon2
(embedding) (18ddiing
(encoder): Bacoder
(convelutions): ModuleList
(0) Sequential
(0) Convtorm
(conv): Convid(512, ke:nel_stie=-55)) stride-11,1, padding=(2,)) (2,))
(1) #atchNeemid(112, epsele-05, momentunn0.1, affine=irue, srack_ouniing_cctteereee
(1): Sequential
(0): Convtorm
(conv): (512 512, ke-nel_s'ze- (5,), atride-11,1, padding (2,)
(1): BatchNormld eps=le-05, momentum00.1, affine=True, seack_ouningg_.stteeeee
(2) Sequential
(0): Convtoomm(
(conv): (S12 kernel_aize= stridg=11,1, padding= (2,))
:1): (atchNcemldd$12,, eps-le momentun=0.1,, affine=Trwe, s
(1stm): 1582, 256, batch_fiiaae rirtt-rue, bid' ieru
(decoder): becoder
[preneL): Prenet
:layers): XoduleList(
(0): LinearNorm!
(linear_layer) tinear(in_fer (hnarr(t__eatturee-8) out_features=255, biag=False)
(1) LinearNorm!
(Linear_aaeer) Linear fenttreette,,,,,,, out_reatures=256, biag=False)
(attention_rnm 18THCe11 (896, 1024)
(ayye): ALteation!
lquery_layar), LdnearMorm(
(linear_layect: (iCfeeetett,,,, out_Lestures-125, bias-false)
Imenory layer) LinearNerm(
(1inear_layer1) Linear (hnear(h__oatuuraaa66 fot_ffaturss=128, blas=False)
(v): LinearNozm
[linear_layer1: Linear (inerrii__eatureee out tet_tteauree=,, blas-Palse)
(location_ LocationLayer! (location_eonss Convioorm
(cone): (2,vid 32, kernal_aize= (tride=(1,), (155) b'as=False)
(location_dense)) LinearNorm( (Lyyea) (inearr(n_ out_Lestures-128, bias-Palse)
[decoder_rnni: 187MCe11 (1664, 1024, bias-1)
(Linesr_projection) LinearNormm
(linear_layes): Linesrtin_Fatuusa-11G. out_featuues=60, biaseTrus)
(gate_layez): LineazNorm!
(linear_layer), Linsar iin features=1664, out_featureasl, bias=True)
16

---

Published as a conference paper at ICLR 2021
(postnet): Postnet (
(convolutions): ModuleList
(0) Sequential
( Convilorm(
(conv) Convid(80, 512, (5, strida=(1,), padding= (2,))
: BatchNeemld(512, ops=1s-05, momentune0.1, affineswirue, ssaik_runding_tttserrres
(1): Sequential
(0): Convtlorm(
(conv): (o1vld 512, sixe stride=11,l, paddings (2,))
: BatchMormtd($12, eps-le-U5, momentun=0.1,, aftine-True, =
(2) Sequential
(0): ConvNorm(
(cenv): Conv1d 512, keznelrrize (5,), stridem(1,1, padding (2,))
(1) RatchMormid(512, eps-1e-05, momentumeD.1, affine=True, troo__uoningg_satt--ruu
(3) Sequential
(0) ConvNorm(
(cenv): (S12 512, kezeel stride-(1,l, padding- (2,))
(1) RatchMormtd(112, eps=1e-05, momentuns0.1, affine=True,
(4) : Sequential
(0) Convborm(
(conv): Conv1d 80, kernel_size-(S,) (5,), (Le)de-(1,), padding=(2,))
(1): BatchMormId(60, epa=ie-05, momentum=C.1,, affinesTrue, bacck_raning_etatte-tnn
(spesker_embedding) (3bedding 128)
17


Reviews:
[{"review_id": "xbEC1xYTsid", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "FLOWTRON\n\n1.\nThe authors present Flowttron, an autoregressive text-to-speech network that is\na merging of the well-established Tacotron architecture along with flow-based\ngenerative models. The benefits to Flowtron are its simplicity to train and\ncomparable MOS to other state of the art models. Moreover, the model is able to\nproduce various styles of speech depending on how the latent variables are\nsampled, as well style transfer between seen and unseen speakers.\n\n2.\nThe introduction and discussion in the paper are strong and review the\nnecessary background information. The experiments are comprehensive and\nconvincing. Although there are artifacts in the audio samples (repeated\nphrases, distortion), the variation of speech is impressive.\n\nThe overall description of the model lacks technical thoroughness. The\ndescription of the flow networks is sufficient. However, how precisely\ntacotron2 is modified to incorporate the flow network is severely lacking.\nFigure 2 is confusing. I also don't understand what the authors mean by \"it is\nalso possible to reverse the ordering of the mel-spectrogram frames in time\nwithout loss of generality\"\n\nIt is not clear to me how the values in Table 2 are calculated.\n\nOn page 4, the second paragraph, what is the purpose of synthesizing samples\nwith Tacotron2 while varying the pre-net dropout probabilities.\n\n3/4. \nI would recommend an accept, given the authors can address the major technical\nproblems with the paper. Specifically, the description of the system needs to\nbe much more clear. The work is interesting, the experiments are sufficient,\nand the audio samples are convincing.\n\n\n6. There are many typos that should be fixed.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iiaBa54FPW_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a specific approach to the hot topic of  auto-regressive Flow-based speech synthesis. The fundamentals of coupled flow layers are exposed pretty well, but I quickly got lost in the details of the work. I have a number of specific comments to address that. Nonetheless, I find the paper is still a good (if confusing) read addressing a very interesting area.\n\n\"Their assumption is that variable-length embeddings are not robust to text and speaker perturbations, which we show not to be the case\" -- in what specific aspects do those works you refer make that assumption?\n\nRe: Figure 1, \"Figure 1 shows that acoustic characteristics like timbre and F0 correlate with portions of the z-space of Flowtron models trained without speaker embeddings.\" Label the axes of Figure 1?\n\nEq. (7): I can guess, but can you confirm the meaning of the circle-with-dot operator?\n\n\"Normalizing flows are typically constructed using coupling layers\": I nearly, but don't completely, understand the concept of the coupling layers. AIUI, the trick is that they are forward layers with two components that are inverses of each other.  I guess that Eqs. 5-7 in fact define just that. However, how is z_t computed? f(z_t) and its inverse defined in terms of s_t and b_t, which are functions of z_{1:t-1}. So where does z_t come from?\n\n\"we define z1 as a prependded 0 vector\" : (only one \"d\" in prepended). For z1, what is the 0 vector component prepended to? Or is z1 itself a zero vector?\n\n\"To evaluate the likelihood, we take the mel-spectrograms and pass them through the inverse steps of\nflow conditioned on the text and optional speaker ids, adding the corresponding log |s| penalties, and\nevaluate the result based on the Gaussian likelihoods.\"\n\nFor clarity, I suggest:\n- Define s.\n- Define the complete log-likelihood criterion.\n\n\"Our text encoder modifies Tacotron’s 2\": spelling.\n\n\"Figure 1 shows evidence that speech several characteristics present in mel-spectrograms\": check grammar.\n\n\"Section 2.5, Posterior Inference\": I suggest stating at this point how posterior inference will be used to generate actual samples.\n\nSection 3.2: The inference method has not yet been explained.\n\nTable 1: what happens if you use more Flows? Do the MOS scores improve?\n\n\"3.3.1 SPEECH VARIATION\": i agree that FlowTron seems to be able to generate more varied speech, but this is only a good thing if the samples are still natural. I don't see a metric or assessment of the naturalness of those more-varied samples (maybe i missed it).\n\nSection 3.3.2, \"Our different speaker interpolation samples show that Flowtron is able to gradually\nand smoothly morph one voice into another.\" Where is this measured/reported in the paper?\n\n\"3.4.2 SEEN SPEAKER WITH UNSEEN STYLE... We compare samples generated with Flowtron and Tacotron 2 GST to evaluate their ability to emulate a speaking style unseen during training of a speaker seen during training.\": where is this comparison shown in the paper?\n\n\nTable 2: what are the units?\n\n\"Table 2 shows that while the samples generated with Tacotron 2 GST are not able to emulate\nthe high-pitched style from RAVDESS, Flowtron is able to make Sally sound high-pitched as in the\n“surprised\" style.\" I find it a bit hard to interpret Table 2. Is the goal to match the reference mean and standard deviation as closely possible?\n\n\"3.5 SAMPLING THE GAUSSIAN MIXTURE In this last section we provide visualizations ...\": where do you provide it?\n\nre: 3.5.1 VISUALIZING ASSIGNMENTS and 3.5.2 TRANSLATING DIMENSIONS, same comment as before, where are the results provided?\n\n\n\nReferences:\n\"Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo\nKang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. Deep voice: Real-time neural\ntext-to-speech. arXiv preprint arXiv:1702.07825, 2017b.\" --> remove superfluous \"O\" for first author? Or add missing \"O\" in the previous reference?", "extra_scores": {}, "extra_text": {}}, {"review_id": "RHN50cYZFEb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper introduces a novel application of normalizing flows to speech synthesis, allowing direct optimization of spectrogram log-likelihoods which results in more natural variation at inference compared to L1/L2 losses that model the mean. This setup also allows more control over non-textual information and interpolation between samples and styles.\n\n\n### Recommendation\n\n**Accept**\nThe idea is good and experiments are good. There are some concerns about the clarity of the paper but those can be worked on.\n\n### Positives\n\n1. The paper introduces a novel architecture and demonstrates improved output variation and more controllability, which is an important current issue for TTS research.\n1. Many experiments investigating controllability are described, as well as some ablation studies for the model architecture in the appendix.\n\n\n### Negatives\n\n1. **Figure 1**: Please provide more informative captions. In the text, timbre and F0 are mentioned but it is not clear how that is relevant in the images. It is not clear what the colors mean in 1b, and if 1b is supposed to show a separation between male and female speakers the colors make it worse. Finally, are the points in 1b cluster centers or a single random sample per speaker?\n\n1. **Figure 2**: Would it be possible to make it clearer that there is an autoregressive dependency in the Attention and Decoder blocks due to the LSTM cell memory? The way the figure is currently drawn makes it seem as if each attention/decoder block can be computed in parallel from only the inputs from the previous flow iteration. \n\n1. In section 2.2 NN and f are described as acting on the latent variable frames $z_t$, but Figure 2 applies the NN and flow on the mel spectrogram frames x in order to produce z. Similarly, there is text saying \"we take the mel-spectrograms and pass them through the inverse steps of flow\" and \"Inference, [...], is simply a matter of sampling z values [...] and running them through the network\" but Figure 2 marks the block as \"Step of Flow\" rather than \"Inverse Step of Flow\". It would be good to smooth out the consistency.\n\n1. More discussion about the end of sequence prediction would be appreciated. As the entire sequence of z must be used for inference, I assume there is some constant max inference length z used to obtain a final x, and the end of sequence prediction only happens to the final x rather than at each step of the flow? How well does this model adapt to inference samples that are much longer than any of the inputs seen during training?\n\n1. **3.3.2 Interpolation Between Samples**: I don't understand why there was a need to sample z to find z_h and z_s. Is it not possible to take z_h and z_s from a random training example for the speaker? Or does that mean there is a correlation between the latent space z and the __content__ that is being spoken? If the latter, I would like to see more discussion on that.\n\n1. **Table 2**: I assume bold means closest to ground truth? It's really hard to know how to interpret this table and I don't think it supports saying that FTA Posterior is more effective. FTA did not capture the increase in pitch mean from expressive -> high pitch, nor the decrease in std from expressive -> surprised. A better visualization may be to express this table as a bar graph (3 separate groups of 3 bars each) and conclude FTA Posterior is able to produce much more variation than Tacotron 2 GST?\n\n### Misc\n\n#### Abstract\n\n1. \"varation\" -> \"variation\"\n1. spell out IAF\n1. \"We provide results on speech variation\" etc. sounds weak. eg. \"Flowtron produces output with far more natural variation compared with Tacotron 2 and enables interpolation over time between samples and style transfer between seen and unseen speakers in ways that are either impossible or inefficient to achieve with prior works.\"\n\n#### 1 Introduction\n\n1. \"Their assumption is that variable-length embeddings are not robust to text and speaker perturbations\" citation needed\n1. \"Flowtron learns an invertible function that maps a distribution over mel-spectrograms to a latent z-space parameterized by a spherical Gaussian.\" mention IAF and cite Kingma et al., 2016 here instead of the previous paragraph. \n1. \"Finally, although VAEs and GANs provide a latent embedding that can be manipulated, they may be difficult to train, are limited to approximate latent variable prediction\" While the IAF approach allows for direct optimization of log-likelihood, the latent variable encoding part is still approximate, just like in a VAE. The original Kingma paper even states that it is an approximate posterior. \n\n#### 2.2 Invertible Transformations\n\n1. $f^{-1}$ wouldn't be applied to $z_t$. Maybe $f^{-1}(x_t)$ or $f^{-1}\\left(f(z_t)\\right)$.\n\n#### 3.1. Training Setup\n\n1. \"progressively adding steps of flow on the last step of flow has learned to attend to text\" on->once?\n\n#### 3.4.2 Seen Speaker with Unseen Style\n\n1. \"Flowtron succeeds in transferring not only the somber timbre, the low F0 and the long pauses\nassociated with the narrative style\" -> \"not only the somber timbre, but also\"\n\n#### Appendix\n\n1. IAF is known to be quite inefficient. Is there any noticeable impacts on training or inference time? Or is this not a problem due to only using 2 layers of flow? It would be great if the ablation study in the appendix regarding layers of flow also covers this, but this is quite optional.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cx0PlzMI9Ng", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a text-to-speech synthesis system, called Flowtron which uses a normalizing flow to generate a sequence of mel-spectrogram frames. The difference between the proposed Flowtron and the previously prosed flow-based methods is, the authors argue as the main contributions, its ability to produce more diverse and expressive speech samples of specific speech attributes by sampling from the latent space. Evaluation is done using the two public datasets, and a number of experiments are performed to show that the proposed method not only achieves a good MOS score in general but also generates speech samples with variation, including the style transfer.\n\nOverall, the paper is fairly well organized with figures and tables in appropriate positions. However, it is sometimes difficult to follow, sections 3.3.2 - 3.5.2 in particular, because there are too many short subsections without supporting figures or tables. Furthermore, it is also hard to navigate through the folders and files provided in the supplementary material to listen to the speech samples.\n\nEvaluation is well done. The proposed method yields a comparable MOS score to a reference (Tacotron 2) in terms of speech quality. However, focus in experiments is on the many features that are not feasible in other approaches, which include variation control such as pitch or duration, interpolation between samples, and style transfer. But it still remains unclear that how a user can explicitly control these speech features or attributes in order to generate speech with the desired non-textual information, which requires much more than simple and random variation in pitch and/or duration.\n\nMinor comments: \n- There are several typos (even in the abstract) and grammatical errors throughout the text.\n- Again, it is quite difficult to find and compare the speech samples that correspond to features describe in the text. It would be better to embed the samples on the website for easier navigation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an autoregressive flow-based network, Flowtron, for TTS with style transfer. It integrates the Tacotron architecture with the flow-based generative model.  Extensive experiments are carried out in a controlled manner and the results show that the proposed Flowtron framework can achieve comparable MOS scores to the SOTA TTS models and is good at generating speech with different styles. All reviewers consider the work interesting.  There are concerns raised on technical details which mostly have been cleared by the authors' rebuttal. The exposition also has been greatly improved based on the reviewers' suggestions and questions.  Overall, this is an interesting paper and I would recommend acceptance. 
Decision: Accept (Poster)