Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis

1 INTRODUCTION . Current speech synthesis methods do not give the user enough control over how speech actually sounds . Automatically converting text to audio that successfully communicates the text was achieved a long time ago ( Umeda et al. , 1968 ; Badham et al. , 1983 ) . However , communicating only the text information leaves out the acoustic properties of the voice that convey much of the meaning and human expressiveness . In spite of this , the typical speech synthesis problem is formulated as a text to speech ( TTS ) problem in which the user inputs only text since the 1960s . This work proposes a normalizing flow model ( Kingma & Dhariwal , 2018 ; Huang et al. , 2018 ) that learns an unsupervised mapping from non-textual information to manipulable latent Gaussian distributions . Taming the non-textual information in speech is difficult because the non-textual is unlabeled . A voice actor may speak the same text with different emphasis or emotion based on context , but it is unclear how to label a particular reading . Without labels for the non-textual information , recent approaches ( Shen et al. , 2017 ; Arik et al. , 2017a ; b ; Ping et al. , 2017 ) have formulated speech synthesis as a TTS problem wherein the non-textual information is implicitly learned . Despite their success in recreating non-textual information in the training set , the user has limited insight and control over it . It is possible to formulate an unsupervised learning problem in such a way that the user can exploit the unlabeled characteristics of a data set . One way is to formulate the problem such that the data is assumed to have a representation in some latent space , and have the model learn that representation . This latent space can then be investigated and manipulated to give the user more control over the generative model ’ s output . Such approaches have been popular in image generation , allowing users to interpolate smoothly between images and to identify portions of the latent space that correlate with various features ( Radford et al. , 2015 ; Kingma & Dhariwal , 2018 ; Izmailov et al. , 2019 ) . Recent deep learning approaches to expressive speech synthesis have combined text and learned latent embeddings for non-textual information ( Wang et al. , 2018 ; Skerry-Ryan et al. , 2018 ; Hsu et al. , 2018 ; Habib et al. , 2019 ; Sun et al. , 2020 ) . These approaches impose an undesirable paradox : they require making assumptions before hand about the dimensionality of the embeddings when the correct dimensionality can only be determined after the model is trained . Even then , these embeddings are not guaranteed to contain all the non-textual information it takes to reconstruct speech , often resulting in models with dummy or uninterpretable latent dimensions and not enough capacity , as the appendices in Wang et al . ( 2018 ) ; Skerry-Ryan et al . ( 2018 ) ; Hsu et al . ( 2018 ) confirm . Furthermore , most models are not able to manipulate speech characteristics over time due to fixedlength embeddings . Their assumption is that variable-length embeddings are not robust to text and speaker perturbations ( Skerry-Ryan et al. , 2018 ) , which we show not to be the case . Finally , although VAEs and GANs ( Sun et al. , 2020 ; Habib et al. , 2019 ; Hsu et al. , 2018 ; Bińkowski et al. , 2019 ; Akuzawa et al. , 2018 ) provide a latent embedding that can be manipulated , they may be difficult to train , are limited to approximate latent variable prediction , and rely on an implicit generative model or ELBO estimate to perform MLE in the latent space ( Kingma & Dhariwal , 2018 ; Lucic et al. , 2018 ; Kingma et al. , 2016 ) . In this paper we propose Flowtron : an autoregressive flow-based generative network for melspectrogram synthesis with style transfer over time and speech variation . Flowtron learns an invertible function that maps a distribution over mel-spectrograms to a latent z-space parameterized by a spherical Gaussian . Figure 1 shows that acoustic characteristics like timbre and F0 correlate with portions of the z-space of Flowtron models trained without speaker embeddings . With our formalization , we can generate samples containing specific speech characteristics manifested in mel-space by finding and sampling the corresponding region in z-space ( Gambardella et al. , 2019 ) . Our formulation also allows us to impose a structure to the z-space and to parametrize it with a Gaussian mixture , similar to Hsu et al . ( 2018 ) . In our simplest setup , we generate samples with a zero mean spherical Gaussian prior and control the amount of variation by adjusting its variance . Compared to VAEs and GANs and their disadvantages enumerated in Kingma & Dhariwal ( 2018 ) , manipulating a latent prior in Flowtron comes at no cost in speech quality nor optimization challenges . Flowtron is able to generalize and produce sharp mel-spectrograms , even at high σ2 values , by simply maximizing the likelihood of the data while not requiring any additional Prenet or Postnet layer ( Wang et al. , 2017 ) , nor compound loss functions required by most SOTA models ( Shen et al. , 2017 ; Ping et al. , 2017 ; Skerry-Ryan et al. , 2018 ; Wang et al. , 2018 ; Bińkowski et al. , 2019 ) . In summary , Flowtron is optimized by maximizing the exact likelihood of the training data , which makes training simple and stable . Using normalizing flows , it learns an invertible mapping from data to latent space that can be manipulated to modulate many aspects of speech synthesis . Concurrent with this work are Glow-TTS ( Kim et al. , 2020 ) and Flow-TTS ( Miao et al. , 2020 ) , both of which incorporate normalizing flows into the TTS task . Our work differs from these two in that Flowtron is an autoregressive architecture where we explore the use of flow to modulate speech and style variation . In contrast , Glow-TTS and Flow-TTS are parallel architectures that focus on inference speed . Our mean opinion scores ( MOS ) show that Flowtron matches SOTA TTS models in terms of speech quality . Further , we provide results on speech variation , interpolation between samples and interpolation between styles over time , and style transfer between seen and unseen speakers with equal or different sentences . We hope this work , the first to show evidence that normalizing flows can be used for expressive text-to-speech synthesis and style transfer , will further stimulate developments in normalizing flows . 2 FLOWTRON . Flowtron is an autoregressive flow that generates a sequence of mel-spectrogram frames . A normalizing flow generates samples by first sampling a latent variable from a known distribution p ( z ) , and applying a series of invertible transformations to produce a sample from the target distribution p ( x ) . These invertible transformations f are known as steps of flow : x = f1 ◦ f2 ◦ . . .fk ( z ) ( 1 ) Because each transformation is invertible , we can directly evaluate the exact log-likelihood of the target distribution p ( x ) using the change of variables : log pθ ( x ) = log pθ ( z ) + k∑ i=1 log |det ( J ( f−1i ( x ) ) ) | ( 2 ) z = f−1k ◦ f −1 k−1 ◦ . . .f −1 1 ( x ) ( 3 ) Where J is the Jacobian of the inverse transform f−1i ( x ) . By cleverly choosing the latent distribution p ( z ) and the invertible transformations , the exact log-likelihood becomes simple and tractable . 2.1 LATENT DISTRIBUTIONS . We consider two simple distributions for the latent distribution z : a zero-mean spherical Gaussian and a mixture of spherical Gaussians with fixed or learnable parameters . z ∼ N ( z ; 0 , I ) or z ∼ ∑ k φ̂kN ( z ; µ̂k , Σ̂k ) ( 4 ) The zero-mean spherical Gaussian has a simple log-likelihood . The mixture of the spherical Gaussians , has inherent clusters that might result in interesting aspects of the audio information . 2.2 INVERTIBLE TRANSFORMATIONS . Normalizing flows are typically constructed using coupling layers ( Dinh et al. , 2014 ; 2016 ; Kingma & Dhariwal , 2018 ) . In our case , we use an autoregressive affine coupling layer ( Dinh et al. , 2016 ) . The latent variable z has the same number of dimensions and frames as the resulting mel-spectrogram sample . The previous frames z1 : t−1 produce scale and bias terms , st and bt respectively , that affine-transform the succeeding time step zt : ( log st , bt ) = NN ( z1 : t−1 , text , speaker ) ( 5 ) f ( zt ) = ( zt − bt ) ÷ st ( 6 ) f−1 ( zt ) = st zt + bt ( 7 ) Here , NN ( ) can be any autoregressive causal transformation ( Shumway & Stoffer , 2017 ) . The affine coupling layer is a reversible transformation , even though NN ( ) itself need not be invertible . We use a 0-vector for obtaining the scaling and bias terms what will affine transform z1 . This 0-vector constant also guarantees that the first z is always known . With an affine coupling layer , only the st term changes the volume of the mapping and adds a change of variables term to the loss . This term also penalizes the model for non-invertible affine mappings . log |det ( J ( f−1coupling ( x ) ) ) | = log |s| ( 8 ) To evaluate the likelihood , we take the mel-spectrograms and pass them through the inverse steps of flow conditioned on the text and optional speaker ids , adding the corresponding log |s| penalties , and evaluate the result based on the Gaussian likelihoods . With this setup , it is also possible to reverse the ordering of the mel-spectrogram frames in time without loss of generality . We reverse the order of frames on even steps of flow , defining a step of flow as a full pass over the input sequence . This allows the model to learn dependencies both forward and backwards in time while remaining causal and invertible . 2.3 MODEL ARCHITECTURE . Our text encoder modifies the text encoder in Tacotron 2 by replacing batch-norm with instance-norm . Our decoder and NN architecture , depicted in Figure 2 , removes the Prenet and Postnet layers from Tacotron previously thought to be essential ( Shen et al. , 2017 ) . Please compare Figure 2 describing our architecture and Figure 8 in A.4.4 describing Tacotron ’ s architecture . We also provide model summary views in A.6 We use the content-based tanh attention described in Vinyals et al . ( 2015 ) , which can be easily modified to become also location sensitive . We use the Mel Encoder described in Hsu et al . ( 2018 ) to predict the parameters of the Gaussian Mixture . Following ( Valle et al. , 2019b ) , we use speakerembeddings channel-wise concatenated with the encoder outputs at every token . We use a single shared embedding for models not conditioned on speaker id . The step of flow closest to the latent variable z has a gating mechanism that prunes extra frames from the z-values provided to the model during inference . The length of z-values remains fixed on the next steps of flow .

Summary: This paper presents a text-to-speech synthesis system, called Flowtron which uses a normalizing flow to generate a sequence of mel-spectrogram frames. The difference between the proposed Flowtron and the previously prosed flow-based methods is, the authors argue as the main contributions, its ability to produce more diverse and expressive speech samples of specific speech attributes by sampling from the latent space. Evaluation is done using the two public datasets, and a number of experiments are performed to show that the proposed method not only achieves a good MOS score in general but also generates speech samples with variation, including the style transfer.