PortaSpeech: Portable and High-Quality Generative Text-to-Speech

1 Introduction . Recently , deep learning-based text-to-speech ( TTS ) has attracted a lot of attention in speech community [ 2 , 15 , 20 , 22 , 24 , 25 , 29 , 35 ] . Among neural network-based TTS systems , some of them generate mel-spectrograms autoregressively from text [ 15 , 22 , 29 , 35 ] and suffer from slow inference speed and robustness ( word skipping and repeating ) problems [ 25 ] , while others [ 8 , 12 , 16 , 19 , 24 , 25 ] generate mel-spectrograms in parallel with comparable quality using non-autoregressive architecture , called NAR-TTS , which enjoys fast inference and avoids robustness issues in the meanwhile . In general , modern TTS models aim to achieve the following goals : • Fast : to reduce the cost of computational resources and apply the model to real-time applications , the inference speed of TTS model should be fast . ∗Equal contribution . †Corresponding author 3Audio samples are available at https : //portaspeech.github.io/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . • Lightweight : to deploy the model to mobile or edge devices , the model size should be small and the runtime memory footprint should be low . • High-quality : to improve the naturalness of synthesized speech , the model should capture the details ( frequency bins between two adjacent harmonics , unvoiced frames and high-frequency parts ) in natural speech . • Expressive : to generate expressive and dynamic speech , the model should use powerful prosody modeling methods to accurately model the fundamental frequency and duration of speech . • Diverse : to prevent the synthesized speech from being too dull and tedious when generating long speech , the model should be able to generate diverse speech samples with different intonations given one text input sequence . To achieve the above goals , in this work , we propose PortaSpeech , a portable and high-quality generative text-to-speech model , which generates mel-spectrograms with natural details and expressive prosody using a lightweight architecture . Specifically , • Through some preliminary experiments ( see Section 4.2 ) , we find that VAE is good at capturing the long-range semantics features ( e.g. , prosody ) , while normalizing flow is good at reconstructing the frequency bin-wise details . Based on these observations , we adopt VAE with an enhanced prior followed by a flow-based post-net as the main model architecture of PortaSpeech , which helps PortaSpeech generate high-quality and expressive results . In addition , PortaSpeech can generate diverse speech by sampling latent variables from the prior of VAE and post-net . • Through the experiments , we also find that even when the model is very small , VAE is still good at capturing the prosody , making it possible for PortaSpeech to reduce its model size using a lightweight VAE . Besides , we introduce the grouped parameter sharing mechanism to the post-net to compress its model size . By doing these , PortaSpeech can be very lightweight and fast at a small performance cost . • To model the prosody better and generate more expressive speech , we introduce a linguistic encoder with mixture alignment , which combines hard word-level alignment and soft phonemelevel alignment . Our proposed linguistic encoder also reduces the dependence on fine-grained ( phoneme-level ) alignment and alleviates the burden of the speech-to-text aligner . Experiments on the LJSpeech [ 7 ] dataset show that PortaSpeech outperforms other state-of-the-art TTS models with comparable model parameters in voice quality and prosody in terms of both subjective and objective evaluation metrics . When compressing the model size , our PortaSpeech shows only a slight performance degradation but enjoys the benefits of a much smaller number of model parameters ( about 4x model size reduction ) and lower memory footprints ( about 3x memory reduction ) compared with FastSpeech 2 . The main contributions of this work are summarized as follows : • We analyze the characteristics of VAE and normalizing flow when applied to TTS and combines the advantages of VAE and normalizing flow to generate mel-spectrograms with rich details and expressive prosody . • We propose mixture alignment in the linguistic encoder , which improves the prosody and reduces the dependence on fine-grained ( phoneme-level ) hard alignment . • Using lightweight VAE and introducing the grouped parameter sharing mechanism to the post-net , PortaSpeech can generate high-quality speech with a small number of model parameters and small runtime memory footprints . 2 Background . In this section , we describe the background of TTS and the basic knowledge of VAE and normalizing flow . We also review the existing applications of VAE and normalizing flow in non-autoregressive TTS and analyze their advantages and disadvantages . Text-to-Speech Text-to-speech ( TTS ) models convert input text or phoneme sequence into melspectrogram ( e.g. , Tacotron [ 35 ] , FastSpeech [ 25 ] ) , which is then transformed to waveform using vocoder ( e.g. , WaveNet [ 33 ] ) , or directly generate waveform from text ( e.g. , FastSpeech 2s [ 24 ] and EATS [ 5 ] ) . End-to-end text-to-speech models have gradually developed from autoregressive to non-autoregressive architecture : early autoregressive text-to-speech models [ 29 , 35 ] generate each mel-spectrogram frame conditioned on previous ones , resulting in high inference latency and low robustness . Recently , several non-autoregressive TTS works have been proposed , which generate melspectrogram frames in parallel . FastSpeech [ 25 ] and ParaNet [ 21 ] are the first non-autoregressive TTS models , which use pre-trained autoregressive TTS teacher models to extract text-to-spectrogram alignments from the training data to bridge the length gap between text and speech for non-autoregressive student model . FastSpeech 2 [ 24 ] introduces more variation information of speech , including pitch and energy , to alleviate the one-to-many mapping problem in TTS . While these methods need external text-to-spectrogram alignment models or tools , Glow-TTS [ 8 ] directly searches for the most probable monotonic alignment between text and the latent representation of speech using normalizing flows and dynamic programming . In addition to improving the performance of non-autoregressive models , some works focus on lightweight and portable model designs : SpeedySpeech [ 32 ] replaces the self-attention layers with fully convolutional blocks to reduce the computational complexity . LightSpeech [ 17 ] leverages neural architecture search ( NAS ) to automatically design more lightweight models , while the training of NAS consumes huge resources . In this work , we save the model parameters by taking advantage of the characteristics of VAE and normalizing flow and introducing the grouped parameter sharing mechanism . VAE The VAE is a generative model in the form of pθ ( x , z ) = p ( z ) pθ ( x|z ) , where p ( z ) is a prior distribution over latent variables z and pθ ( x|z ) is the likelihood function that generates data x given latent variables z which can be considered as a decoder . It is parameterized by a neural network θ . Since the true posterior pθ ( x , z ) over the latent variables of a VAE is usually analytically intractable , we approximate it with a variational distribution qφ ( z|x ) , which can be viewed as an encoder . The parameters θ and φ can be optimized by maximizing the evidence lower bound ( ELBO ) : log pθ ( x ) ≥Eqφ ( z|x ) [ log pθ ( x , z ) qφ ( z|x ) ] = Ez∼qφ ( z|x ) [ log pθ ( x|z ) − log qφ ( z|x ) pθ ( z ) ] =Ez∼qφ ( z|x ) [ log pθ ( x|z ) ] −KL ( qφ ( z|x ) ‖pθ ( z ) ) ≡ L ( θ , φ ) . Recently , some works successfully apply VAE to TTS . One of them is BVAE-TTS [ 14 ] , which adopts a bidirectional-inference variational autoencoder that learns hierarchical latent representations using both bottom-up and top-down paths to increase its expressiveness . Thanks to the hierarchical structure and latent modeling , BVAE-TTS can capture the dynamism and variability of ground-truth prosody . However , its generated mel-spectrograms are very blurry and over-smoothing , resulting in unnatural sounds , due to the posterior collapse [ 1 , 6 ] and the reconstruction loss term used in BVAE-TTS , which has independency assumption of generated frequency bins given latent variables . Normalizing Flow Normalizing flow is a kind of generative models [ 3 , 4 ] which has several advantages including exact log-likelihood evaluation and fully-parallel sampling . In generation , normalizing flows [ 3 , 4 ] transform the latent variable z into a datapoint x through a composition of invertible functions f = f1 ◦ f2 ◦ · · · ◦ fK and we assume a tractable prior pθ ( z ) over latent variable z sampled from a simple distribution ( e.g. , a Gaussian distribution ) . In training , the log-likelihood of a datapoint x can be computed exactly using the change of variables rule : log pθ ( x ) = log pθ ( z ) + K∑ i=1 log |det ( dhi/dhi−1 ) | , ( 1 ) where h0 = x , hi = fi ( hi−1 ) , hK = z and |det ( dhi/dhi−1 ) | is the Jacobian determinant . We learn the parameters of f1 . . . fK by maximizing Equation ( 1 ) over the training data . Given g = f−1 , we can now generate a sample x̂ by sampling z ∼ pθ ( z ) and computing x̂ = g ( z ) . There are several normalizing flow-based non-autoregressive TTS methods : Flow-TTS [ 19 ] is an early flow-based TTS method , which replaces the decoder in FastSpeech with Glow [ 9 ] and jointly learns the alignment and mel-spectrogram generation through a single network . Then Glow-TTS [ 8 ] is proposed , which combines the normalizing flow and dynamic programming-based monotonic alignment to enable fast , diverse and controllable speech synthesis . These methods handle the blurry mel-spectrogram problems well due to the nature of the normalizing flow . However , according to our experiments ( see Section 4.2 ) , flow-based NAR-TTS model usually requires a huge model capacity to achieve good performance , and the performance can drop notably when reducing the number of model parameters . 3 PortaSpeech . Considering the characteristics of VAE and normalizing flow mentioned in Section 2 , to build a TTS system that can meet the goals described in Section 1 , we propose PortaSpeech , which combines the advantages of VAE and normalizing flows and overcomes their deficiencies . As shown in Figure 1a , PortaSpeech is composed of a linguistic encoder with mixture alignment , a variational generator with enhanced prior and a flow-based post-net with the grouped parameter sharing mechanism . First , the text sequence with word-level boundary is fed into the linguistic encoder to extract the linguistic features in both phoneme and word level . Secondly , to model the expressiveness and variability of speech with lightweight architecture , we train the VAE-based variational generator to maximize the ELBO over the ground-truth mel-spectrograms conditioned on the linguistic features , whose prior distribution is modeled by a small volume-preserving normalizing flow . Finally , to refine and enhance the natural speech details in the generated mel-spectrograms , we train the post-net by maximizing the likelihood of ground-truth mel-spectrograms conditioned on both the linguistic features and the outputs of the variational generator . During inference , the text is transformed to mel-spectrograms by successively passing through the linguistic encoder , the decoder of the variational generator and the reversed flow-based post-net . We describe these designs and the training and inference procedures in detail in the following subsections . We put more details in Appendix A .

Summary: To overcome the shortcomings that autoregressive text-to-speech (AR TTS) models have (i.e. high inference latency, low robustness), non-AR TTS models such as Glow-TTS and BVAE-TTS have recently been proposed, which show competitive performance compared to AR TTS models.      In this paper, a non-autoregressive (non-AR) TTS model called PortaSpseech is proposed, which effectively reduces the number of its parameters by combining VAE and normalizing flow, while showing better performance than Glow-TTS and BVAE-TTS. First, by conducting preliminary analyses on VAE and Flow, it shows which model has which advantages over the other: VAE is good at capturing prosody even with small size and Flow is good at reconstructing the frequency bin-wise details. Then, to fully utilize the advantage of each method, this paper proposes novel ways to combine the models. To help model capture prosody, lightweight VAE equipped with enhanced prior is used in decoder of the main model, and normalizing flow with grouped parameter sharing mechanism is used as a post-net, which is helpful to improve fidelity while maintaining a small number of parameters. In addition to that, it also proposes a linguistic encoder with a mixture alignment mechanism, which is effective in learning prosody compared to the phoneme-level hard alignment.  Through various experiments, this paper supports its claim that PortaSpeech can generate high-quality speech with a small number of parameters. In preliminary analyses, by measuring MOS in terms of prosody and quality separately, it shows the characteristic that each VAE and Flow model has. Then, it compares PortaSpeech with previous non-AR TTS models and shows its superiority. Plus, ablation studies also show that each architectural component is really effective to improve the performance of PortaSpeech.