Parallel Neural Text-to-Speech

1 INTRODUCTION . Text-to-speech ( TTS ) , also called speech synthesis , has long been a vital tool in a variety of applications , such as human-computer interactions , virtual assistant , and content creation . Traditional TTS systems are based on multi-stage hand-engineered pipelines ( Taylor , 2009 ) . In recent years , deep neural networks based autoregressive models have attained state-of-the-art results , including highfidelity audio synthesis ( van den Oord et al. , 2016 ) , and much simpler sequence-to-sequence ( seq2seq ) pipelines ( Sotelo et al. , 2017 ; Wang et al. , 2017 ; Ping et al. , 2018b ) . In particular , one of the most popular neural TTS pipelines consists of two components ( Ping et al. , 2018b ; Shen et al. , 2018 ) : ( i ) an autoregressive seq2seq model that generates mel spectrogram from text , and ( ii ) an autoregressive neural vocoder ( e.g. , WaveNet ) that generates raw waveform from mel spectrogram . This pipeline requires much less expert knowledge and only needs pairs of audio and transcript as training data . The autoregressive nature of these components leads to slow synthesis , because they operate sequentially at a high temporal resolution of waveform samples and spectrogram . Most recently , Parallel WaveNet ( van den Oord et al. , 2018 ) and ClariNet ( Ping et al. , 2018a ) were proposed for parallel waveform synthesis by distilling an inverse autoregressive flow ( IAF ) ( Kingma et al. , 2016 ) from a pretrained WaveNet . However , they still rely on autoregressive or recurrent components to predict the frame-level acoustic features ( e.g. , 100 frames per second ) , which can be slow at synthesis on modern hardware optimized for parallel execution . In this work , we introduce a fully parallel neural TTS system by proposing a non-autoregressive text-to-spectrogram model . Our major contributions are as follows : 1 . We propose ParaNet , the first non-autoregressive attention-based architecture for TTS , which is fully convolutional and converts text to mel spectrogram . We compare the nonautoregressive ParaNet with its autoregressive counterpart ( Ping et al. , 2018b ) in terms of speech quality , synthesis speed and attention stability . It achieves ∼ 46.7 times speed-up over Deep Voice 3 ( DV3 ) at synthesis , while maintaining comparable speech quality using a WaveNet vocoder . ParaNet iteratively refines the attention alignment between text and spectrogram in a layer-by-layer manner , and it can produce stable attentions on the challenging test sentences as DV3 with attention masking . 2 . Based on ParaNet , we build the first fully parallel neural TTS system by combining ParaNet with parallel neural vocoders . Our system can generate speech from text through a single feed-forward pass . We investigate several parallel vocoders within the parallel TTS system , including the distilled IAF vocoder ( Ping et al. , 2018a ) and WaveGlow ( Prenger et al. , 2019 ) . 1Synthesized speech samples are in : https : //parallel-neural-tts-demo.github.io/ . To explore the possibility of training IAF vocoder without distillation in this parallel system , we also propose an alternative approach , WaveVAE , which can be trained from scratch within the variational autoencoder ( VAE ) framework ( Kingma and Welling , 2014 ) . We organize the rest of this paper as follows . Section 2 discusses related work . We introduce the nonautoregressive text-to-spectrogram architecture in Section 3 , and discuss the parallel neural vocoders in Section 4 . We report experimental results in Section 5 , and conclude the paper in Section 6 . 2 RELATED WORK . Neural speech synthesis has obtained the state-of-the-art results and gained a lot of attention . Several neural TTS systems were proposed , including WaveNet ( van den Oord et al. , 2016 ) , Deep Voice ( Arık et al. , 2017a ) , Deep Voice 2 ( Arık et al. , 2017b ) , Deep Voice 3 ( Ping et al. , 2018b ) , Tacotron ( Wang et al. , 2017 ) , Tacotron 2 ( Shen et al. , 2018 ) , Char2Wav ( Sotelo et al. , 2017 ) , VoiceLoop ( Taigman et al. , 2018 ) , WaveRNN ( Kalchbrenner et al. , 2018 ) , Transformer TTS ( Li et al. , 2019 ) and ClariNet ( Ping et al. , 2018a ) . In particular , Tacotron and Deep Voice 3 employ seq2seq framework with the attention mechanism ( Bahdanau et al. , 2015 ) , yielding much simpler pipeline compared to traditional multistage pipeline . Their excellent extensibility leads to promising results for several challenging tasks , such as voice cloning ( Arik et al. , 2018 ; Nachmani et al. , 2018 ; Jia et al. , 2018 ; Chen et al. , 2019 ) . All of these state-of-the-art TTS systems are based on autoregressive models . RNN-based autoregressive models , such as Tacotron and WaveRNN ( Kalchbrenner et al. , 2018 ) , lack parallelism at both training and synthesis . CNN-based autoregressive models , such as Deep Voice 3 and WaveNet , enable parallel processing at training , but they still operate sequentially at synthesis since each output element must be generated before it can be passed in as input at the next time step . Recently , there are some non-autoregressive models proposed for neural machine translation . For example , Gu et al . ( 2018 ) trains a feed-forward neural network conditioned on fertility values , which is obtained from an external alignment system . Kaiser et al . ( 2018 ) proposes a latent variable model for fast decoding , while it remains autoregressive between latent variables . Lee et al . ( 2018 ) iteratively refines the output sequence through a denoising autoencoder framework . Arguably , non-autoregressive model plays a more important role in text-to-speech , where the output speech spectrogram consists of hundreds of time steps for a short text with a few words . To the best of our knowledge , our work is the first non-autoregressive seq2seq model for TTS and provides as much as 46.7 times speed-up at synthesis over its convolutional autoregressive counterpart . Flow-based generative models ( Rezende and Mohamed , 2015 ; Kingma et al. , 2016 ; Dinh et al. , 2017 ; Kingma and Dhariwal , 2018 ) transform a simple initial distribution into a more complex one by applying a series of invertible transformations . In previous work , flow-based models have obtained state-of-the-art results for parallel waveform synthesis ( van den Oord et al. , 2018 ; Ping et al. , 2018a ; Prenger et al. , 2019 ; Kim et al. , 2019 ; Yamamoto et al. , 2019 ) . Variational autoencoder ( VAE ) ( Kingma and Welling , 2014 ; Rezende et al. , 2014 ) has been applied for representation learning of natural speech for years . It models either the generative process of waveform samples ( Chung et al. , 2015 ; van den Oord et al. , 2017 ) , or spectrograms ( Hsu et al. , 2019 ) . In previous work , autoregressive or recurrent neural networks are employed as the decoder of VAE ( Chung et al. , 2015 ; van den Oord et al. , 2017 ) , but they can be quite slow at synthesis . 3 TEXT-TO-SPECTROGRAM MODEL . Our parallel TTS system has two components : 1 ) a feed-forward text-to-spectrogram model , and 2 ) a parallel waveform synthesizer conditioned on spectrogram . In this section , we first present an autoregressive text-to-spectrogram model derived from Deep Voice 3 ( DV3 ) ( Ping et al. , 2018b ) . We then introduce ParaNet , a non-autoregressive text-to-spectrogram model ( see Figure 1 ) . 3.1 AUTOREGRESSIVE ARCHITECTURE . Our autoregressive model is based on DV3 , a convolutional text-to-spectrogram model , which consists of three components : • Encoder : A convolutional encoder , which encodes text into internal hidden representation . • Decoder : A causal convolutional decoder , which decodes the encoder representation with an attention mechanism to log-mel spectragrams in an autoregressive manner with an ` 1 loss . It starts with 1× 1 convolutions to preprocess the input log-mel spectrograms . • Converter : A non-causal convolutional post-processing network , which processes the hidden representation from the decoder using both past and future context information and predicts the log-linear spectrograms with an ` 1 loss . It enables bidirectional processing . All these components use the same 1-D convolution with a gated linear unit as in DV3 . The major difference between our model and DV3 is the decoder architecture . The decoder of DV3 has multiple attention-based layers , where each layer consists of a causal convolution block followed by an attention block . To simplify the attention distillation described in Section 3.3.1 , our autoregressive decoder has only one attention block at its first layer . We find that reducing the number of attention blocks does not hurt the generated speech quality in general . 3.2 NON-AUTOREGRESSIVE ARCHITECTURE . Our ParaNet ( see Figure 2 ) uses the same encoder architecture as the autoregressive model . The decoder of ParaNet , conditioned solely on the hidden representation from the encoder , predicts the entire sequence of log-mel spectrograms in a feed-forward manner . As a result , both its training and synthesis can be performed in parallel . Specially , we make the following major architecture modifications from the autoregressive seq2seq model to the non-autoregressive model : 1 . Non-autoregressive decoder : Without the autoregressive generative constraint , the decoder can use non-causal convolution blocks to take advantage of future context information and to improve model performance . In addition to log-mel spectrograms , it also predicts log-linear spectrograms with an ` 1 loss for slightly better performance . 2 . No converter : Non-autoregressive model removes the non-causal converter since it already employs a non-causal decoder . Note that , the major motivation of introducing non-causal converter in DV3 is to refine the decoder predictions based on bidirectional context information provided by non-causal convolutions ( Ping et al. , 2018b ) . 3.3 PARALLEL ATTENTION MECHANISM . It is challenging for the non-autoregressive model to learn the accurate alignment between the input text and output spectrogram . Note that , we need the full parallelism within the attention mechanism . For example , the location-sensitive attention ( Chorowski et al. , 2015 ; Shen et al. , 2018 ) improves attention stability , but it performs sequentially at both training and synthesis , because it uses the cumulative attention weights from previous decoder time steps as an additional feature . Previous non-autoregressive decoders rely on an external alignment system ( Gu et al. , 2018 ) , or an autoregressive latent variable model ( Kaiser et al. , 2018 ) . In this work , we present several simple & effective techniques , which could obtain accurate and stable alignment with the multi-step attention ( Gehring et al. , 2017 ) . In particular , our non-autoregressive decoder can iteratively refine the attention alignment between text and mel spectrogram in a layer-by-layer manner as illustrated in Figure 3 . Specially , the decoder adopts a dot-product attention mechanism and consists of K attention blocks ( see Figure 2 ) , where each attention block uses the per-time-step query vectors from convolution block and per-time-step key vectors from encoder to compute the attention weights ( Ping et al. , 2018b ) . The attention block computes context vectors as the weighted average of the value vectors from the encoder . The non-autoregressive decoder starts with an attention block , in which the query vectors are solely positional encoding ( see Section 3.3.2 for details ) . The first attention block then provides the input for the convolution block at the next attention-based layer .

Summary: This submission belongs to the field of text-to-speech synthesis. In particular, it looks at the problem of efficient training and inference. Many current state-of-the-art approaches in the area employ an autoregressive component at one or another part of text-to-speech model making training and/or inference complicated/slow. The main idea behind this paper is to remove autoregressive components.