Speech-T: Transducer for Text to Speech and Beyond

1 Introduction . Transducer [ 7 ] is a sequence-to-sequence model widely used in automatic speech recognition ( ASR ) [ 19 , 17 , 36 , 28 ] . ASR Transducer consists of a speech encoder that converts an input speech sequence to hidden representations , a text encoder that processes already generated text tokens autoregressively , and a joint network that predicts the next text token . Transducer networks are trained to maximize the alignment probability between speech and text sequences , with the help of a forward algorithm on the output probability lattice ( see Figure 1 for more details ) . The alignments 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learned in the output probability lattice of the Transducer are strictly monotonic , thus well modeling monotonic alignments between speech and text sequences in ASR . By restricting the speech encoder only to see the previous speech frames , Transducer naturally supports streaming inputs and is the one of the most popular solutions for streaming ASR [ 36 , 12 , 33 , 3 ] . Although Transducer is originally designed for ASR , clearly , its advantages of modeling monotonic alignments between inputs and outputs and supporting streaming inputs perfectly match the requirements of text to speech ( TTS ) synthesis [ 26 , 30 , 22 , 1 , 6 , 16 , 13 , 20 , 21 , 11 ] . First , learning alignments between text and speech is important in TTS [ 26 ] , and different approaches such as attention mechanisms [ 22 , 8 , 25 , 13 ] or duration prediction [ 20 , 21 , 11 , 4 ] have been leveraged to model the alignments . However , attention mechanisms suffer from unstable alignments with word skipping and repeating issues [ 16 , 20 , 8 ] , and duration prediction suffers from additional complication because of the training of duration predictor and training-inference mismatch caused by the predictor [ 20 , 21 , 11 ] . Thus , the inherent monotonic alignments in Transducer can be a quite competitive solution for TTS over previous methods . Second , streaming TTS ( or incremental TTS ) [ 14 , 5 , 23 ] , which aims to support streaming text inputs , can greatly save the synthesis latency and can be widely used in online scenarios such as conversations . Unfortunately , existing solutions for streaming TTS [ 14 ] mostly rely on the argmax operation of the encoder-decoder attention values to decide the end of speech frame corresponding to an input word , which is not accurate and robust . Transducer can naturally support streaming inputs and can be a better solution compared with previous methods for streaming TTS . Therefore , we explore the possibility of applying Transducer for TTS in this work . However , applying Transducer to TTS is challenging due to the distinctive characteristics of speech ( e.g. , mel-spectrograms ) generation in TTS compared with the text token generation in ASR . First , there are two actions in the output probability lattice of Transducer [ 7 , 32 ] : emission that predicts a text token and transition that predicts a blank token to indicate null outputs in current step and the transition to the next input speech frame [ 7 ] . In ASR Transducer , the blank token is added into the token vocabulary , which means that the emission ( token prediction ) and transition ( blank prediction ) can be modeled in a unified probability distribution through the softmax on all the vocabulary tokens . However , mel-spectrograms in TTS are continuous , which makes it hard to trade off the two probabilities when calculating the output probability lattice in TTS Transducer . As shown in our experiments , ill tradeoff between the two probabilities causes the failure of alignment learning . Second , while predicting current mel-spectrogram , previous mel-spectrograms can provide enough information as they are very similar due to the continuity between consecutive mel-spectrograms [ 22 , 2 ] , which may lead to copying the previous frame instead of learning information from the text encoder and thus harms alignment learning . Moreover , Transducer learns the appropriate alignment path from huge candidate paths in the probability lattice , which further causes difficulties in alignment learning . In this work , we propose a Transformer [ 29 ] based Transducer model , named SpeechTransducer ( Speech-T for short ) , for TTS . First , we design a lazy forward algorithm that separates transition prediction from mel-spectrogram prediction when deriving the loss function of Transducer . Specifically , this lazy forward algorithm on the probability lattice only calculates the probability of transition/nontransition ( using a binary classification ) , without the probability of mel-spectrogram prediction . Second , we introduce a diagonal constraint [ 25 , 16 , 2 ] in the probability lattice to assist alignment learning , which ignores the alignment paths that deviate too much from the diagonal . The intuitive idea behind this design is that the alignments between text and speech should be monotonic , and thus lie in the diagonal region in the probability lattice . Third , by adjusting the text encoder of Speech-T to look the whole context or only the previous context , it can support full-sentence TTS or streaming TTS . Last but not the least , we further extend Speech-T to support both TTS and ASR together for the first time , which enjoys several advantages including fewer parameters as well as streaming synthesis and recognition in a single model . Experiment results on LJSpeech datasets demonstrate that Speech-T 1 ) is more robust than attention based autoregressive TTS models due to its inherent monotonic alignments between text and speech , 2 ) can naturally support streaming TTS with good voice quality , and 3 ) enjoys the benefit of joint modeling TTS and ASR in a single network . The main contributions of this work are summarized as follows : • To leverage the advantages of Transducer ( i.e. , monotonic alignment modeling and streaming input supporting ) , we propose SpeechTransducer for TTS . To the best of our knowledge , we are the first to achieve competitive voice quality in TTS and support streaming TTS using Transducer . • We design a lazy forward algorithm to trade off the two probabilities of mel-spectrogram prediction and transition prediction , and design a diagonal constraint to ensure the alignment learning between text and speech , to ensure the quality of SpeechTransducer . • We further extend SpeechTransducer to support both TTS and ASR at the same time , which enjoys several advantages including compact modeling ( fewer parameters ) , streaming synthesis and recognition , self-ranking , etc . To the best of our knowledge , we are the first to successfully perform the two tasks with a single model . 2 Background . In this section , we first introduce the basic formulation of neural Transducer , and then describe the application of Transducer on several sequence-to-sequence tasks . Formulation of Transducer Neural Transducer is originally proposed to model the sequence transduction between input speech and output text sequences in ASR [ 7 ] . It learns strictly monotonic alignments between speech ( e.g. , mel-spectrogram ) sequence X = { x1 , x2 , ... , xU } and output text ( e.g. , phoneme or character ) sequence Y = { y1 , y2 , ... , yT } , where U and T are the length of speech and text sequences respectively ( U is short for utterance , T is short for text ) . A Transducer contains 1 ) a speech encoder which extracts acoustic representations from the speech sequence X , 2 ) a text encoder works autoregressively to extract hidden representations from the text sequence Y , and 3 ) a joint network to predict the next token in Y based on the combination between the output hidden representations of the speech encoder and text encoder . 3 1 2 3 4 5 6 p ( 1,0 ) ∅ ( 1,1 ) ∅ ( 2,1 ) p ( 3,1 ) p ( 5,2 ) ∅ ( 3,2 ) ∅ ( 4,2 ) ∅ ( 5,3 ) ∅ ( 6,3 ) Speech ( U=6 ) Te xt ( T =3 ) 2 1 0 right in the lattice represents a possible alignment between the speech and text sequence . During the training of Transducer , it marginalizes all the alignments ( paths ) A between X and Y over the probability lattice shown in Figure 1 and minimizes the negative log-probability of a conditional distribution : Lasr = logP ( Y |X ) = log X ↵2F 1 ( y ) P ( ↵|X ) , ( 1 ) where F 1 is the inverse function of F that removes the blank token from each possible alignment ↵ between X and Y . Each ↵ consists of U blanks ( ) and T tokens ( y1 , y2 , y3 , . . . , yT ) . For example , we refer ↵ = ( y1 , y2 , , , y3 , , ) as a possible alignment between X = ( x1 , x2 , x3 , x4 ) and Y = ( y1 , y2 , y3 ) , then F ( ↵ ) becomes Y . Computing Equation ( 1 ) by summing all possible alignments naively is computationally intractable . Therefore , Graves [ 7 ] introduced an efficient forward algorithm to handle this problem . In brief , it defines a forward variable ↵ ( u , t ) to represent the probability of outputting Y1 : t given X1 : u . ↵ ( u , t ) can be obtained by recursive calculation : ↵ ( u , t ) = ↵ ( u 1 , t ) ( u 1 , t ) + ↵ ( u , t 1 ) p ( u , t 1 ) , ( 2 ) where ( u 1 , t ) represents the transition probability ( predicting blank token given xu 1 and yt ) , p ( u , t 1 ) represents the emission probability ( predicting token yt given xu and yt 1 ) , and the initial condition is ↵ ( 1 , 0 ) = 1 . Application of Transducer Neural Transducer network has several advantages : 1 ) Monotonic alignment modeling . The alignment path in the output probability lattice is inherent monotonic from bottom left to top right , which makes it suitable for the tasks with monotonic alignments between the input and output sequences , e.g. , ASR . 2 ) Streaming input support . Transducer can generate text based on streaming speech input , by making the speech encoder only to see the previous speech context and using the transition ( blank token ) to determine the finish of text generation based on current input speech segment . Due to these advantages , Transducer has been widely used in ASR for streaming scenarios [ 19 , 17 , 9 ] . RNN was used as the backbone of speech encoder and text encoder in early ASR Transducer ( i.e. , RNN-T ) [ 7 ] , and then Transformer [ 29 ] was introduced into Transducer ( i.e. , Transformer-T [ 36 , 28 ] ) to enhance the ability of modeling long-term dependency . Besides ASR , Transducer can also be applied on other sequence-to-sequence tasks such as text summarization [ 34 , 35 ] , morphological inflection [ 34 ] , and machine translation [ 35 ] . The application of Transducer on TTS is still under-exploited . Yasuda et al . [ 32 ] leveraged the monotonic alignment of Transducer in TTS , but generated speech with poor voice quality . The reasons include : 1 ) It is hard to trade off the probability of transition and emission ( mel-spectrogram prediction ) when calculating the output probability lattice , since mel-spectrogram and transition predictions are modeled separately ( mel-spectrogram prediction using regression , while transition prediction using binary classification ) . 2 ) The alignments between speech and text in TTS are more difficult to learn than those in ASR . The voice generated by Yasuda et al . [ 32 ] usually has incorrect alignments such as extremely long duration for a certain phoneme . In our work , we apply Transducer to TTS by reformulating the forward algorithm with a lazy probability lattice to avoid the troublesome probability tradeoff , and adding a diagonal alignment constraint on the loss to facilitate the alignment learning . We also adjust the Transudcer model to support streaming TTS and further support both TTS and ASR in a single Transducer model , which can bring many interesting applications to explore .

Summary: This is a very interesting application of the Neural Transducer framework that is quite popular for end-to-end ASR to the problem of TTS. The authors adopt the dynamic programming sum over many possible alignments of text and audio to minimize a TTS-oriented L1 frame prediction loss. The details they propose are reasonable; though not the only way forward, the proposal is to decouple the alignment path probability from frame prediction probabilities, while nonetheless multiplying alignment path probability by the L1 prediction loss. Good results are presented compared to a strong Transformer TTS baseline, though I am not sure whether that baseline is itself as strong as the best WaveNet or Tacotron results. Furthermore, a unified TTS/ASR system is described sharing parameters of the acoustic/text encoders, predicting both feature frames for TTS and next symbols for ASR.