Bidirectional Variational Inference for Non-Autoregressive Text-to-Speech

1 INTRODUCTION . End-to-end text-to-speech ( TTS ) systems have recently attracted much attention , as neural TTS models began to generate high-quality speech that is very similar to the human voice ( Sotelo et al. , 2017 ; Wang et al. , 2017 ; Shen et al. , 2018 ; Ping et al. , 2018 ; Li et al. , 2019 ) . Typically , those TTS systems first generate a mel-spectrogram from a text using a sequence-to-sequence ( seq2seq ) model ( Sutskever et al. , 2014 ) and then synthesize speech from the mel-spectrogram using a neural vocoder like WaveGlow ( Prenger et al. , 2019 ) . Early neural TTS systems have used an autoregressive ( AR ) architecture to generate a melspectrogram mainly because of its two benefits . First , the AR generation eases the difficulty of modeling mel-spectrogram distribution by factorizing the distribution into the product of homogeneous conditional factors in sequential order . Second , the seq2seq based AR architecture helps the model predict the length of the target mel-spectrogram from an input text , which is a non-trivial task because there are no pre-defined rules between the lengths of text and mel-spectrogram . Although they facilitate high-quality speech synthesis , AR TTS models have several shortcomings . First , they can not generate a mel-spectrogram in parallel , so the inference time increases linearly with mel-spectrogram time steps . Second , the AR-based generation suffers from accumulated prediction error , resulting in being vulnerable to the out-of-domain data , e.g . very long input text , or text patterns not existing in the training dataset . In this work , we present a novel non-AR TTS model called BVAE-TTS that achieves fast and robust high-quality speech synthesis . BVAE-TTS generates a mel-spectrogram in parallel by adopting a bidirectional-inference variational autoencoder ( BVAE ) ( Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Maaløe et al. , 2019 ; Vahdat & Kautz , 2020 ) consisting of 1-D convolutional networks . For the high-quality speech synthesis , BVAE-TTS learns mel-spectrogram distribution jointly with hierarchical latent variables in a bidirectional manner , where BVAE uses both bottom-up and top-down paths . Furthermore , to match the length of the target mel-spectrogram at inference , BVAE-TTS has an additional module called duration predictor , which predicts how many steps of a mel-spectrogram will be generated from each phoneme . To train the duration predictor , we employ an attention mechanism in BVAE-TTS to make BVAE-TTS utilize the text while learning attention maps between the text and the mel-spectrogram , where the mapping information is used for duration labels . Our BVAE-TTS has advantages over the previous non-AR TTS models as follows : • It has a simpler training process compared to the previous non-AR TTS models such as ParaNet ( Peng et al. , 2020 ) and FastSpeech ( Ren et al. , 2019 ) . In the previous TTS models , well-trained AR teacher models are needed for duration labels or knowledge-distillation . Although FastSpeech 2 ( Ren et al. , 2020 ) removes the dependency on the teacher model , it still requires additional duration labels and acoustic features prepared in advance using other speech analysis methods . In contrast , BVAE-TTS requires only the text-speech paired dataset without any helps from the teacher model . • It is more flexible in designing its architecture compared to the previous flow-based non-AR TTS models such as Flow-TTS ( Miao et al. , 2020 ) and Glow-TTS ( Kim et al. , 2020 ) . The flow-based models have architectural constraints caused by their bijective transformation property , which leads to deeper models with a lot of parameters . On the contrary , the VAE-based model is free from the architectural constraints . In experiments , we compare our BVAE-TTS with Tacotron 2 and Glow-TTS in terms of speech quality , inference speed , and model size . The results show that our model achieves 27 times speed improvement over Tacotron 2 in generating a mel-spectrogram with similar speech quality . Furthermore , BVAE-TTS outperforms the state-of-the-art non-AR TTS model , Glow-TTS , in both speech quality and inference time , while having 58 % fewer model parameters . Additionally , we analyze how the latent representations are learned by BVAE-TTS . In this analysis , we confirm that the bottom part of BVAE-TTS captures the variation of mel-spectrograms that can occur from a text . Related work : In the meantime , several TTS systems have utilized VAE to relax the one-to-many mapping nature in TTS , so improve the naturalness and the controllability of the systems . For example , ( Hsu et al. , 2018 ) and ( Zhang et al. , 2019 ) incorporate VAE to Tacotron 2 to learn the style or prosody of the input speech . However , previous uses of VAE have been limited to an auxiliary network in TTS based on the main AR TTS model . To the best of our knowledge , our BVAE-TTS is the first parallel TTS model that directly uses the VAE architecture to the task of TTS . More discussions about other related works on the previous non-AR TTS models are in Section 5 . 2 BACKGROUND . 2.1 BIDIRECTIONAL-INFERENCE VARIATIONAL AUTOENCODER Variational autoencoder ( VAE ) is a neural network generative model pθ ( x , z ) parameterized by θ , where x is an observed data and z is a latent vector . In practice , since we only have a dataset X = { x1 , ... , xN } without the knowledge about z , θ is typically optimized by maximizing the likelihood : max θ logpθ ( X ) = max θ N∑ i=1 log ∫ z pθ ( xi , z ) dz . ( 1 ) However , the integral over z is intractable to compute . Therefore , the VAE introduces an approximate posterior qφ ( z|x ) and does variational inference while maximizing the evidence lower bound ( ELBO ) : logpθ ( x ) ≥ Eqφ ( z|x ) [ logpθ ( x|z ) ] −DKL [ qφ ( z|x ) ||p ( z ) ] . ( 2 ) In practice , for easy sampling and easy computation of the KL-divergence , each of the prior p ( z ) and the approximate posterior qφ ( z|x ) is usually modeled as a multivariate normal distribution with a diagonal covariance matrix . For a more expressive model , the latent vector z can be factorized into { z1 , ... , zK } with hierarchical dependency , where K is the number of hierarchies . Then , each of the prior and the approximate posterior is represented as pθ ( z ) = Πkpθ ( zk|z < k ) and qφ ( z|x ) = Πkqφ ( zk|z < k , x ) , respectively . In ( Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Vahdat & Kautz , 2020 ) , the variational inference is designed in a bidirectional way based on bottom-up path and top-down path , while letting the inference network ( left ) and generative network ( right ) share their parameters as shown in Figure 1 . First , along the bottom-up path , BVAE extracts hierarchical features from x and stores them inside of it . Then , along the top-down path , BVAE does the variational inference and reconstructs the input data considering the stored hierarchical features together . This architecture helps the model effectively learn the hierarchies between the latent variables , and equation ( 2 ) is changed as follows : logpθ ( x ) ≥ Eqφ ( z|x ) [ logpθ ( x|z ) ] − K∑ k=1 Eqφ ( z < k|x ) [ DKL [ qφ ( zk|x , z < k ) ||p ( zk|z < k ) ] ] . ( 3 ) 2.2 DURATION PREDICTOR IN NON-AUTOREGRESSIVE TEXT-TO-SPEECH . To achieve the non-autoregressive ( non-AR ) text-to-speech ( TTS ) model , the model needs to predict the length of the target mel-spectrogram from a text . This is because there is no way to access to the length of the target mel-spectrogram at inference . However , this is a challenging task considering that there are no pre-defined rules between the lengths of text and mel-spectrogram . Recently , several non-AR TTS models ( Ren et al. , 2019 ; Zeng et al. , 2020 ; Kim et al. , 2020 ) resolved the issue by introducing a module called duration predictor . The duration predictor is a module that predicts how many mel-spectrogram steps will be generated from each phoneme . First , using the duration predictor , the non-AR TTS models compute durations D̂ = { d̂1 , ... , d̂S } corresponding to each phoneme based on phoneme representations H = { h1 , ... , hS } , where each d̂i is a positive integer that is rounded off from a positive real number , and S is the number of phonemes . Then , H is expanded to the length of the target mel-spectrogram T , by repeating each hi as many steps as d̂i . Finally , the non-AR TTS models generate a mel-spectrogram in parallel by decoding the expanded phoneme representations . In practice , since there are no ground-truth duration labels for the training of the duration predictor , the non-AR models obtain the duration labels using various methods , and we adopt a method used in FastSpeech ( Ren et al. , 2019 ) . From well-aligned attention maps , the duration labels are obtained according to di = ∑t=T t=1 [ argmaxs as , t == i ] , where as , t represents an attention weight given from the t-th mel-spectrogram step to the s-th phoneme . 3 METHODOLOGY . In this section , we explain a novel non-autoregressive ( non-AR ) TTS model , BVAE-TTS , which is based on the bidirectional-inference variational autoencoder ( BVAE ) . As shown in Figure 2- ( a ) , during training , BVAE-TTS is given a mel-spectrogram with a phoneme sequence , and it is trained to reconstruct the mel-spectrogram while maximizing the ELBO . Here , the duration predictor is jointly trained using the attention maps that BVAE-TTS generates during training . As shown in Figure 2- ( c ) , at inference BVAE-TTS generates a mel-spectrogram from a phoneme sequence using the duration predictor as described in Section 2.2 , while using its top-down path for decoding the expanded phoneme representations . In Appendix A.1 , pseudo-codes for the training and inference of BVAE-TTS are contained for detailed descriptions . The other aspects of BVAE-TTS are described in the following sub-sections in more detail . 3.1 USING BVAE FOR TEXT-TO-SPEECH . Unlike the previous BVAE models ( Sønderby et al. , 2016 ; Kingma et al. , 2016 ; Vahdat & Kautz , 2020 ) are trained to generate natural images , our model should learn to generate a mel-spectrogram that is not only natural but also corresponding to the input text . To this end , we add a dot-product attention network ( Bahdanau et al. , 2015 ) on top of the BVAE , which is a channel for BVAE-TTS to learn how to utilize the text properly . First , using a text encoder , key ( K ) and value ( V ) are obtained from a phoneme sequence , and from the bottom-up path , query ( Q ) is obtained . Here , obtaining Q is different from the bottom-up paths of the previous BVAE studies used in the image domain , where only the parameters for posterior approximation are obtained . Second , based on the dot-product attention with Q , K , and V , the V are expanded to Vexp to fit the length of the top-down path , and then the Vexp is inputted into the top-down path of BVAE-TTS . Lastly , the BVAE-TTS does both the variational inference and mel-spectrogram reconstruction along the top-down path using the expanded text representations with the following objectives : Lrecon = −Eqφ ( z|x , y ) [ logpθ ( x|z , y ) ] , ( 4 ) LKL = K∑ k=1 Eqφ ( z < k|x , y ) [ DKL [ qφ ( zk|x , z < k , y ) ||p ( zk|z < k , y ) ] ] , ( 5 ) where x represents mel-spectrogram , y represents text , z represents latent representation , and mean absolute error ( MAE ) loss is used for the Lrecon . In addition to that , a duration predictor is jointly trained to predict durations corresponding to each phoneme in the logarithmic domain using mean square error ( MSE ) loss , Ldur = E [ ( log di − log d̂i ) 2 ] , where di and d̂i are obtained as described in Section 2.2 . The duration predictor takes as input the V obtained from the text encoder , and here the V is detached from the computational graph to prevent it from affecting the BVAE training .

Summary: This paper presents BVAE-TTS, which applies hierarchical VAEs (using an approach motivated by NVAE and Ladder VAEs) to the problem of parallel TTS.  The main components of the system are a dot product-based attention mechanism that is used during training to produce phoneme duration targets for the parallel duration predictor (that is used during synthesis) and the hierarchical VAE that converts duration-replicated phoneme features into mel spectrogram frames (which are converted to waveform samples using a pre-trained WaveGlow vocoder).  The system is compared to Glow-TTS (a similar parallel system that uses flows instead of VAEs) and Tacotron 2 (a non-parallel autoregressive system) in terms of MOS naturalness, synthesis speed, and parameter efficiency.