Bidirectional Variational Inference for Non-Autoregressive Text-to-Speech

1 INTRODUCTION . End-to-end text-to-speech ( TTS ) systems have recently attracted much attention , as neural TTS models began to generate high-quality speech that is very similar to the human voice ( Sotelo et al. , 2017 ; Wang et al. , 2017 ; Shen et al. , 2018 ; Ping et al. , 2018 ; Li et al. , 2019 ) . Typically , those TTS systems first generate a mel-spectrogram from a text using a sequence-to-sequence ( seq2seq ) model ( Sutskever et al. , 2014 ) and then synthesize speech from the mel-spectrogram using a neural vocoder like WaveGlow ( Prenger et al. , 2019 ) . Early neural TTS systems have used an autoregressive ( AR ) architecture to generate a melspectrogram mainly because of its two benefits . First , the AR generation eases the difficulty of modeling mel-spectrogram distribution by factorizing the distribution into the product of homogeneous conditional factors in sequential order . Second , the seq2seq based AR architecture helps the model predict the length of the target mel-spectrogram from an input text , which is a non-trivial task because there are no pre-defined rules between the lengths of text and mel-spectrogram . Although they facilitate high-quality speech synthesis , AR TTS models have several shortcomings . First , they can not generate a mel-spectrogram in parallel , so the inference time increases linearly with mel-spectrogram time steps . Second , the AR-based generation suffers from accumulated prediction error , resulting in being vulnerable to the out-of-domain data , e.g . very long input text , or text patterns not existing in the training dataset . In this work , we present a novel non-AR TTS model called BVAE-TTS that achieves fast and robust high-quality speech synthesis . BVAE-TTS generates a mel-spectrogram in parallel by adopting a bidirectional-inference variational autoencoder ( BVAE ) ( Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Maaløe et al. , 2019 ; Vahdat & Kautz , 2020 ) consisting of 1-D convolutional networks . For the high-quality speech synthesis , BVAE-TTS learns mel-spectrogram distribution jointly with hierarchical latent variables in a bidirectional manner , where BVAE uses both bottom-up and top-down paths . Furthermore , to match the length of the target mel-spectrogram at inference , BVAE-TTS has an additional module called duration predictor , which predicts how many steps of a mel-spectrogram will be generated from each phoneme . To train the duration predictor , we employ an attention mechanism in BVAE-TTS to make BVAE-TTS utilize the text while learning attention maps between the text and the mel-spectrogram , where the mapping information is used for duration labels . Our BVAE-TTS has advantages over the previous non-AR TTS models as follows : • It has a simpler training process compared to the previous non-AR TTS models such as ParaNet ( Peng et al. , 2020 ) and FastSpeech ( Ren et al. , 2019 ) . In the previous TTS models , well-trained AR teacher models are needed for duration labels or knowledge-distillation . Although FastSpeech 2 ( Ren et al. , 2020 ) removes the dependency on the teacher model , it still requires additional duration labels and acoustic features prepared in advance using other speech analysis methods . In contrast , BVAE-TTS requires only the text-speech paired dataset without any helps from the teacher model . • It is more flexible in designing its architecture compared to the previous flow-based non-AR TTS models such as Flow-TTS ( Miao et al. , 2020 ) and Glow-TTS ( Kim et al. , 2020 ) . The flow-based models have architectural constraints caused by their bijective transformation property , which leads to deeper models with a lot of parameters . On the contrary , the VAE-based model is free from the architectural constraints . In experiments , we compare our BVAE-TTS with Tacotron 2 and Glow-TTS in terms of speech quality , inference speed , and model size . The results show that our model achieves 27 times speed improvement over Tacotron 2 in generating a mel-spectrogram with similar speech quality . Furthermore , BVAE-TTS outperforms the state-of-the-art non-AR TTS model , Glow-TTS , in both speech quality and inference time , while having 58 % fewer model parameters . Additionally , we analyze how the latent representations are learned by BVAE-TTS . In this analysis , we confirm that the bottom part of BVAE-TTS captures the variation of mel-spectrograms that can occur from a text . Related work : In the meantime , several TTS systems have utilized VAE to relax the one-to-many mapping nature in TTS , so improve the naturalness and the controllability of the systems . For example , ( Hsu et al. , 2018 ) and ( Zhang et al. , 2019 ) incorporate VAE to Tacotron 2 to learn the style or prosody of the input speech . However , previous uses of VAE have been limited to an auxiliary network in TTS based on the main AR TTS model . To the best of our knowledge , our BVAE-TTS is the first parallel TTS model that directly uses the VAE architecture to the task of TTS . More discussions about other related works on the previous non-AR TTS models are in Section 5 . 2 BACKGROUND . 2.1 BIDIRECTIONAL-INFERENCE VARIATIONAL AUTOENCODER Variational autoencoder ( VAE ) is a neural network generative model pθ ( x , z ) parameterized by θ , where x is an observed data and z is a latent vector . In practice , since we only have a dataset X = { x1 , ... , xN } without the knowledge about z , θ is typically optimized by maximizing the likelihood : max θ logpθ ( X ) = max θ N∑ i=1 log ∫ z pθ ( xi , z ) dz . ( 1 ) However , the integral over z is intractable to compute . Therefore , the VAE introduces an approximate posterior qφ ( z|x ) and does variational inference while maximizing the evidence lower bound ( ELBO ) : logpθ ( x ) ≥ Eqφ ( z|x ) [ logpθ ( x|z ) ] −DKL [ qφ ( z|x ) ||p ( z ) ] . ( 2 ) In practice , for easy sampling and easy computation of the KL-divergence , each of the prior p ( z ) and the approximate posterior qφ ( z|x ) is usually modeled as a multivariate normal distribution with a diagonal covariance matrix . For a more expressive model , the latent vector z can be factorized into { z1 , ... , zK } with hierarchical dependency , where K is the number of hierarchies . Then , each of the prior and the approximate posterior is represented as pθ ( z ) = Πkpθ ( zk|z < k ) and qφ ( z|x ) = Πkqφ ( zk|z < k , x ) , respectively . In ( Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Vahdat & Kautz , 2020 ) , the variational inference is designed in a bidirectional way based on bottom-up path and top-down path , while letting the inference network ( left ) and generative network ( right ) share their parameters as shown in Figure 1 . First , along the bottom-up path , BVAE extracts hierarchical features from x and stores them inside of it . Then , along the top-down path , BVAE does the variational inference and reconstructs the input data considering the stored hierarchical features together . This architecture helps the model effectively learn the hierarchies between the latent variables , and equation ( 2 ) is changed as follows : logpθ ( x ) ≥ Eqφ ( z|x ) [ logpθ ( x|z ) ] − K∑ k=1 Eqφ ( z < k|x ) [ DKL [ qφ ( zk|x , z < k ) ||p ( zk|z < k ) ] ] . ( 3 ) 2.2 DURATION PREDICTOR IN NON-AUTOREGRESSIVE TEXT-TO-SPEECH . To achieve the non-autoregressive ( non-AR ) text-to-speech ( TTS ) model , the model needs to predict the length of the target mel-spectrogram from a text . This is because there is no way to access to the length of the target mel-spectrogram at inference . However , this is a challenging task considering that there are no pre-defined rules between the lengths of text and mel-spectrogram . Recently , several non-AR TTS models ( Ren et al. , 2019 ; Zeng et al. , 2020 ; Kim et al. , 2020 ) resolved the issue by introducing a module called duration predictor . The duration predictor is a module that predicts how many mel-spectrogram steps will be generated from each phoneme . First , using the duration predictor , the non-AR TTS models compute durations D̂ = { d̂1 , ... , d̂S } corresponding to each phoneme based on phoneme representations H = { h1 , ... , hS } , where each d̂i is a positive integer that is rounded off from a positive real number , and S is the number of phonemes . Then , H is expanded to the length of the target mel-spectrogram T , by repeating each hi as many steps as d̂i . Finally , the non-AR TTS models generate a mel-spectrogram in parallel by decoding the expanded phoneme representations . In practice , since there are no ground-truth duration labels for the training of the duration predictor , the non-AR models obtain the duration labels using various methods , and we adopt a method used in FastSpeech ( Ren et al. , 2019 ) . From well-aligned attention maps , the duration labels are obtained according to di = ∑t=T t=1 [ argmaxs as , t == i ] , where as , t represents an attention weight given from the t-th mel-spectrogram step to the s-th phoneme . 3 METHODOLOGY . In this section , we explain a novel non-autoregressive ( non-AR ) TTS model , BVAE-TTS , which is based on the bidirectional-inference variational autoencoder ( BVAE ) . As shown in Figure 2- ( a ) , during training , BVAE-TTS is given a mel-spectrogram with a phoneme sequence , and it is trained to reconstruct the mel-spectrogram while maximizing the ELBO . Here , the duration predictor is jointly trained using the attention maps that BVAE-TTS generates during training . As shown in Figure 2- ( c ) , at inference BVAE-TTS generates a mel-spectrogram from a phoneme sequence using the duration predictor as described in Section 2.2 , while using its top-down path for decoding the expanded phoneme representations . In Appendix A.1 , pseudo-codes for the training and inference of BVAE-TTS are contained for detailed descriptions . The other aspects of BVAE-TTS are described in the following sub-sections in more detail . 3.1 USING BVAE FOR TEXT-TO-SPEECH . Unlike the previous BVAE models ( Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Vahdat & Kautz , 2020 ) are trained to generate natural images , our model should learn to generate a mel-spectrogram that is not only natural but also corresponding to the input text . To this end , we add a dot-product attention network ( Bahdanau et al. , 2015 ) on top of the BVAE , which is a channel for BVAE-TTS to learn how to utilize the text properly . First , using a text encoder , key ( K ) and value ( V ) are obtained from a phoneme sequence , and from the bottom-up path , query ( Q ) is obtained . Here , obtaining Q is different from the bottom-up paths of the previous BVAE studies used in the image domain , where only the parameters for posterior approximation are obtained . Second , based on the dot-product attention with Q , K , and V , the V are expanded to Vexp to fit the length of the top-down path , and then the Vexp is inputted into the top-down path of BVAE-TTS . Lastly , the BVAE-TTS does both the variational inference and mel-spectrogram reconstruction along the top-down path using the expanded text representations with the following objectives : Lrecon = −Eqφ ( z|x , y ) [ logpθ ( x|z , y ) ] , ( 4 ) LKL = K∑ k=1 Eqφ ( z < k|x , y ) [ DKL [ qφ ( zk|x , z < k , y ) ||p ( zk|z < k , y ) ] ] , ( 5 ) where x represents mel-spectrogram , y represents text , z represents latent representation , and mean absolute error ( MAE ) loss is used for the Lrecon . In addition to that , a duration predictor is jointly trained to predict durations corresponding to each phoneme in the logarithmic domain using mean square error ( MSE ) loss , Ldur = E [ ( log di − log d̂i ) 2 ] , where di and d̂i are obtained as described in Section 2.2 . The duration predictor takes as input the V obtained from the text encoder , and here the V is detached from the computational graph to prevent it from affecting the BVAE training .

Summary: Neural models that autoregressively generate mel spectrograms from text (or phonemes), such as Tacotron, have been used to generate high quality synthetic speech. However, they suffer from slow inference speed due to their autoregressive nature. To alleviate this, non-autoregressive models have been proposed, such as FastSpeech and Glow-TTS. The proposed model, BVAE-TTS, is yet another non-autoregressive speech synthesis model (outputting spectrograms), with two key advantages over the aforementioned models: (a) no autoregressive teacher model is required, as in FastSpeech, which simplifies training, and (b) fewer parameters are needed than in Glow-TTS, since there is no bijectivity constraint (allowing a more expressive architecture to be used). Models are compared with inference speed and MOS, and BVAE-TTS compares favorably on both both metrics when compared to Glow-TTS.