Non-Attentive Tacotron: Robust and controllable neural TTS synthesis including unsupervised duration modeling

This paper presents Non-Attentive Tacotron based on the Tacotron 2 text-to-speech model , replacing the attention mechanism with an explicit duration predictor . This improves robustness significantly as measured by unaligned duration ratio and word deletion rate , two metrics introduced in this paper for large-scale robustness evaluation using a pre-trained speech recognition model . With the use of Gaussian upsampling , Non-Attentive Tacotron achieves a 5-scale mean opinion score for naturalness of 4.41 , slightly outperforming Tacotron 2 . The duration predictor enables both utterance-wide and per-phoneme control of duration at inference time . When accurate target durations are scarce or unavailable in the training data , we propose a method using a fine-grained variational auto-encoder to train the duration predictor in a semi-supervised or unsupervised manner , with results almost as good as supervised training . 1 INTRODUCTION . Autoregressive neural text-to-speech ( TTS ) models using an attention mechanism are known to be able to generate speech with naturalness on par with recorded human speech . However , these types of models are known to be less robust than traditional approaches ( He et al. , 2019 ; Zheng et al. , 2019 ; Guo et al. , 2019 ; Battenberg et al. , 2020 ) . These autoregressive networks that predict the output one frame at a time are trained to decide whether to stop at each frame , and thus a misprediction on a single frame can result in serious failures such as early cut-off . Meanwhile , there are little to no hard constraints imposed on the attention mechanism to prevent problems such as repetition , skipping , long pause or babbling . To exacerbate the issue , these failures are rare and are thus often not represented in small test sets , such as those used in subjective listening tests . However , in customer-facing products , even a one-in-a-million chance of such problems can severely degrade the user experience . There have been various works aimed at improving the robustness of autoregressive attention-based neural TTS models . Some of them investigated reducing the effect of the exposure bias on the autoregressive decoder , using adversarial training ( Guo et al. , 2019 ) or adding regularization to encourage the forward and backward attention to be consistent ( Zheng et al. , 2019 ) . Others utilized or designed alternative attention mechanisms , such as Gaussian mixture model ( GMM ) attention ( Graves , 2013 ; Skerry-Ryan et al. , 2018 ) , forward attention ( Zhang et al. , 2018 ) , stepwise monotonic attention ( He et al. , 2019 ) , or dynamic convolution attention ( Battenberg et al. , 2020 ) . Nonetheless , these approaches do not fundamentally solve the robustness issue . Recently , there has been a surge in the use of non-autoregressive models for TTS . Rather than predicting whether to stop on each frame , non-autoregressive models need to determine the output length ahead of time , and one way to do so is with an explicit prediction of the duration for each input token . A side benefit of such a duration predictor is that it is significantly more resilient to the failures afflicting the attention mechanism . However , one-to-many regression problems like TTS can benefit from an autoregressive decoder as the previous mel-spectrogram frames provides context to disambiguate between multi-modal outputs . In this paper , we propose , Non-Attentive Tacotron1 , a neural TTS model that combines the robust duration predictor with the autoregressive decoder of Tacotron 2 ( Shen et al. , 2018 ) . Our work is similar to DurIAN ( Yu et al. , 2019 ; Zhang et al. , 2020 ) , which incorporates the duration predictor with an autoregressive decoder . But besides the differences in architecture , we also introduce a couple of novel features in our model . The key contributions of this paper are summarized as follows : 1 . Replacing the attention mechanism in Tacotron 2 with duration prediction and upsampling modules leading to better robustness with the naturalness matching recorded natural speech ; 2 . Introduction of Gaussian upsampling significantly improving the naturalness compared to vanilla upsampling through repetition ; 3 . Global and fine-grained controlling of durations at inference time ; 4 . Semi-supervised and unsupervised duration modeling of Non-Attentive Tacotron , allowing the model to be trained with few to no duration annotations ; and 5 . More reliable evaluation metrics for measuring robustness of TTS models , as well as comparing Non-Attentive Tacotron with Tacotron 2 with respect to those metrics . 2 RELATED WORKS . In the past decade , model-based TTS synthesis has evolved from hidden Markov model ( HMM ) -based approaches ( Zen et al. , 2009 ) to using deep neural networks . Over this period , the concept of using an explicit representation of token ( phoneme ) durations has not been foreign . Early neural parametric synthesis models ( Zen et al. , 2013 ) require explicit alignments between input and target and include durations as part of the bag of features used to generate vocoder parameters . Explicit durations continue to be used with the advent of the end-to-end neural vocoder WaveNet ( Oord et al. , 2016 ) in works such as Deep Voice ( Arik et al. , 2017 ; Gibiansky et al. , 2017 ) and CHiVE ( Kenter et al. , 2019 ) . As general focus turned towards end-to-end approaches , the autoregressive sequence-to-sequence model with attention mechanism used in neural machine translation ( NMT ) ( Bahdanau et al. , 2015 ) and automatic speech recognition ( ASR ) ( Chan et al. , 2016 ) became an attractive option , removing the need to represent durations explicitly . This led to works such as Char2Wav ( Sotelo et al. , 2017 ) , Tacotron ( Wang et al. , 2017 ; Shen et al. , 2018 ) , Deep Voice 3 ( Ping et al. , 2018 ) , and Transformer TTS ( Li et al. , 2019 ) . Similar models have been used for more complicated problems , like direct speech-to-speech translation ( Jia et al. , 2019 ) , speech conversion ( Biadsy et al. , 2019 ) , and speech enhancement ( Ding et al. , 2020 ) . Tacotron 2 , on which our work is based , is one such model . It connects a character-level encoder and an autoregressive decoder producing mel spectrogram frames with the use of a location-sensitive attention mechanism ( Chorowski et al. , 2015 ) . Recently , there has been a surge of non-autoregressive models , bringing back the use of explicit duration prediction . This approach initially surfaced in NMT ( Gu et al. , 2017 ) , then made its way into TTS with models such as FastSpeech ( Ren et al. , 2019 ; 2020 ) , AlignTTS ( Zeng et al. , 2020 ) , TalkNet ( Beliaev et al. , 2020 ) , and JDI-T ( Lim et al. , 2020 ) . See Appendix C for a rough categorization of these models . To train the duration predictor , FastSpeech uses target durations extracted from a pre-trained autoregressive model in teacher forcing mode , while JDI-T also extracts target durations from a separate autoregressive model but co-trains it with the feed-forward model . TalkNet uses a CTC-based ASR model to extract target durations , while CHiVE , FastSpeech 2 , and DurIAN use target durations from an external aligner module utilizing forced alignment . Finally , AlignTTS forgoes target durations completely and uses a specialized alignment loss inspired by the Baum-Welch algorithm to train a mixture density network for alignment . 1Audio samples available in supplemental materials . 3 MODEL . Modern neural TTS models typically consist of two separate networks : ( 1 ) a feature generation network that transforms input tokens ( e.g. , grapheme or phoneme ids ) into acoustic features ( e.g. , mel-spectrogram ) , and ( 2 ) a vocoder network that transforms the acoustic features into a time-domain audio waveform . This paper focuses on the feature generation network , and can be used with any vocoder network , e.g. , WaveNet ( Oord et al. , 2016 ) , WaveRNN ( Kalchbrenner et al. , 2018 ) , WaveGlow ( Prenger et al. , 2019 ) , MelGAN ( Kumar et al. , 2019 ) , or WaveGrad ( Chen et al. , 2020 ) . The architecture of Non-Attentive Tacotron is illustrated in Figure 1a . See Appendix A for specific parameter value settings . The model follows that of Tacotron 2 ( Shen et al. , 2018 ) , transforming input idsX = ( x1 , . . . , xN ) of length N into mel-spectrogram predictions Y = ( y1 , . . . , yT ) of size T ×K . Phonemes are used as inputs , and include a silence token at word boundaries as well as an end-of-sequence token . The ids are used to index into a learned embedding and is then passed through an encoder consisting of 3 × ( dropout , batch normalization , convolution ) layers followed by a single bi-directional LSTM with ZoneOut to generate a 2-dimensional output of length N . This output is concatenated with a speaker embedding vector to produce the final encoder outputH = ( h1 , ... , hN ) . The autoregressive decoder also follows Tacotron 2 , and predicts mel-spectrograms one frame at a time . At training time , teacher forcing ( Williams & Zipser , 1989 ) is employed and the previous groundtruth mel-spectrogram frame is used as input , while at inference time the previous predicted mel-spectrogram frame is used . This previous frame is passed through a pre-net containing two fully-connected layers of ReLU units with dropout , then concatenated with an upsampled ( aligned ) encoder output corresponding to the current frame . The upsampled encoder outputs for future frames are not visible to the decoder at the current frame . In Tacotron 2 , this upsampling or alignment is achieved using a location-sensitive attention mechanism ( Chorowski et al. , 2015 ) , while in this work the attention mechanism is not used and a separate upsampling mechanism described later is used in its stead . The result is then passed through two uni-directional LSTM layers with ZoneOut . The LSTM output is concatenated with the upsampled encoder output yet again then projected to the mel-spectrogram dimension as frames of a preliminary predicted spectrogram Y ′ . Once all the mel-spectrogram frames have been predicted , they are passed through a 5-layer batch normalized convolutional post-net with tanh activation on all except the last layer . The post-net predicts a residual to add to the prediction Y ′ to obtain the final prediction Y . In place of the attention mechanism used in Tacotron 2 , duration-based models upsample the encoder outputs using per-token duration information . This can be done by simply repeating each encoder output by its duration as in FastSpeech ( Ren et al. , 2019 ) , but instead we adopt a different process we call Gaussian upsampling , which is described in subsection 3.1 . Note that while durations in seconds are used for loss computation , they are converted to durations in integer frames for upsampling . For Gaussian upsampling , a duration and a range parameter must be predicted for each token . The range parameter is called thus because it controls the range of a token ’ s influence . The duration predictor passes the encoder output through two bi-directional LSTM layers followed by a projection layer to predict the numeric duration d = ( d1 , . . . , dN ) for each input token . During training , these predicted durations are only used for loss computation , and the target durations are used instead in the upcoming steps2 . The range parameter predictor passes the encoder output concatenated with durations through two bi-directional LSTM layers followed by a projection layer and a SoftPlus activation to predict a positive range parameter σ for each input token . After the encoder outputs are upsampled , a Transformer-style sinusoidal positional embedding ( Vaswani et al. , 2017 ) is concatenated . The positional embedding tracks the index of each upsampled frame within each token ; if the duration values are [ 2 , 1 , 3 ] , the indices for the positional embedding would be [ 1 , 2 , 1 , 1 , 2 , 3 ] . The model is trained using a combination of duration prediction loss and mel-spectrogram reconstruction loss . The duration prediction loss is the L2 loss between predicted and target durations in seconds , and the mel-spectrogram reconstruction loss is a L1 + L2 loss between the predicted and the groundtruth mel-spectrogram both before and after the post-net ( following Jia et al . ( 2018 ) ) . L = Lspec + λdurLdur ( 1 ) Ldur = 1 N ‖d− d∗‖22 ( 2 ) Lspec = 1 TK T∑ t=1 ( ‖y′t − y∗t ‖1 + ‖y ′ t − y∗t ‖ 2 2 + ‖yt − y ∗ t ‖1 + ‖yt − y ∗ t ‖ 2 2 ) ( 3 )

Summary: In this paper, the authors introduce a text-to-speech model based on Tacotron 2, called Non-Attentive Tacotron. Instead of an attention mechanism, a duration predictor is utilized to improve robustness, which is evaluated by two metrics, unaligned duration ratio (UDR) and word deletion rate(WDR). The authors propose semi-supervised and unsupervised duration modeling with a fine-grained variational auto-encoder (FVAE).