Speech-T: Transducer for Text to Speech and Beyond

1 Introduction . Transducer [ 7 ] is a sequence-to-sequence model widely used in automatic speech recognition ( ASR ) [ 19 , 17 , 36 , 28 ] . ASR Transducer consists of a speech encoder that converts an input speech sequence to hidden representations , a text encoder that processes already generated text tokens autoregressively , and a joint network that predicts the next text token . Transducer networks are trained to maximize the alignment probability between speech and text sequences , with the help of a forward algorithm on the output probability lattice ( see Figure 1 for more details ) . The alignments 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learned in the output probability lattice of the Transducer are strictly monotonic , thus well modeling monotonic alignments between speech and text sequences in ASR . By restricting the speech encoder only to see the previous speech frames , Transducer naturally supports streaming inputs and is the one of the most popular solutions for streaming ASR [ 36 , 12 , 33 , 3 ] . Although Transducer is originally designed for ASR , clearly , its advantages of modeling monotonic alignments between inputs and outputs and supporting streaming inputs perfectly match the requirements of text to speech ( TTS ) synthesis [ 26 , 30 , 22 , 1 , 6 , 16 , 13 , 20 , 21 , 11 ] . First , learning alignments between text and speech is important in TTS [ 26 ] , and different approaches such as attention mechanisms [ 22 , 8 , 25 , 13 ] or duration prediction [ 20 , 21 , 11 , 4 ] have been leveraged to model the alignments . However , attention mechanisms suffer from unstable alignments with word skipping and repeating issues [ 16 , 20 , 8 ] , and duration prediction suffers from additional complication because of the training of duration predictor and training-inference mismatch caused by the predictor [ 20 , 21 , 11 ] . Thus , the inherent monotonic alignments in Transducer can be a quite competitive solution for TTS over previous methods . Second , streaming TTS ( or incremental TTS ) [ 14 , 5 , 23 ] , which aims to support streaming text inputs , can greatly save the synthesis latency and can be widely used in online scenarios such as conversations . Unfortunately , existing solutions for streaming TTS [ 14 ] mostly rely on the argmax operation of the encoder-decoder attention values to decide the end of speech frame corresponding to an input word , which is not accurate and robust . Transducer can naturally support streaming inputs and can be a better solution compared with previous methods for streaming TTS . Therefore , we explore the possibility of applying Transducer for TTS in this work . However , applying Transducer to TTS is challenging due to the distinctive characteristics of speech ( e.g. , mel-spectrograms ) generation in TTS compared with the text token generation in ASR . First , there are two actions in the output probability lattice of Transducer [ 7 , 32 ] : emission that predicts a text token and transition that predicts a blank token to indicate null outputs in current step and the transition to the next input speech frame [ 7 ] . In ASR Transducer , the blank token is added into the token vocabulary , which means that the emission ( token prediction ) and transition ( blank prediction ) can be modeled in a unified probability distribution through the softmax on all the vocabulary tokens . However , mel-spectrograms in TTS are continuous , which makes it hard to trade off the two probabilities when calculating the output probability lattice in TTS Transducer . As shown in our experiments , ill tradeoff between the two probabilities causes the failure of alignment learning . Second , while predicting current mel-spectrogram , previous mel-spectrograms can provide enough information as they are very similar due to the continuity between consecutive mel-spectrograms [ 22 , 2 ] , which may lead to copying the previous frame instead of learning information from the text encoder and thus harms alignment learning . Moreover , Transducer learns the appropriate alignment path from huge candidate paths in the probability lattice , which further causes difficulties in alignment learning . In this work , we propose a Transformer [ 29 ] based Transducer model , named SpeechTransducer ( Speech-T for short ) , for TTS . First , we design a lazy forward algorithm that separates transition prediction from mel-spectrogram prediction when deriving the loss function of Transducer . Specifically , this lazy forward algorithm on the probability lattice only calculates the probability of transition/nontransition ( using a binary classification ) , without the probability of mel-spectrogram prediction . Second , we introduce a diagonal constraint [ 25 , 16 , 2 ] in the probability lattice to assist alignment learning , which ignores the alignment paths that deviate too much from the diagonal . The intuitive idea behind this design is that the alignments between text and speech should be monotonic , and thus lie in the diagonal region in the probability lattice . Third , by adjusting the text encoder of Speech-T to look the whole context or only the previous context , it can support full-sentence TTS or streaming TTS . Last but not the least , we further extend Speech-T to support both TTS and ASR together for the first time , which enjoys several advantages including fewer parameters as well as streaming synthesis and recognition in a single model . Experiment results on LJSpeech datasets demonstrate that Speech-T 1 ) is more robust than attention based autoregressive TTS models due to its inherent monotonic alignments between text and speech , 2 ) can naturally support streaming TTS with good voice quality , and 3 ) enjoys the benefit of joint modeling TTS and ASR in a single network . The main contributions of this work are summarized as follows : • To leverage the advantages of Transducer ( i.e. , monotonic alignment modeling and streaming input supporting ) , we propose SpeechTransducer for TTS . To the best of our knowledge , we are the first to achieve competitive voice quality in TTS and support streaming TTS using Transducer . • We design a lazy forward algorithm to trade off the two probabilities of mel-spectrogram prediction and transition prediction , and design a diagonal constraint to ensure the alignment learning between text and speech , to ensure the quality of SpeechTransducer . • We further extend SpeechTransducer to support both TTS and ASR at the same time , which enjoys several advantages including compact modeling ( fewer parameters ) , streaming synthesis and recognition , self-ranking , etc . To the best of our knowledge , we are the first to successfully perform the two tasks with a single model . 2 Background . In this section , we first introduce the basic formulation of neural Transducer , and then describe the application of Transducer on several sequence-to-sequence tasks . Formulation of Transducer Neural Transducer is originally proposed to model the sequence transduction between input speech and output text sequences in ASR [ 7 ] . It learns strictly monotonic alignments between speech ( e.g. , mel-spectrogram ) sequence X = { x1 , x2 , ... , xU } and output text ( e.g. , phoneme or character ) sequence Y = { y1 , y2 , ... , yT } , where U and T are the length of speech and text sequences respectively ( U is short for utterance , T is short for text ) . A Transducer contains 1 ) a speech encoder which extracts acoustic representations from the speech sequence X , 2 ) a text encoder works autoregressively to extract hidden representations from the text sequence Y , and 3 ) a joint network to predict the next token in Y based on the combination between the output hidden representations of the speech encoder and text encoder . 3 1 2 3 4 5 6 p ( 1,0 ) ∅ ( 1,1 ) ∅ ( 2,1 ) p ( 3,1 ) p ( 5,2 ) ∅ ( 3,2 ) ∅ ( 4,2 ) ∅ ( 5,3 ) ∅ ( 6,3 ) Speech ( U=6 ) Te xt ( T =3 ) 2 1 0 right in the lattice represents a possible alignment between the speech and text sequence . During the training of Transducer , it marginalizes all the alignments ( paths ) A between X and Y over the probability lattice shown in Figure 1 and minimizes the negative log-probability of a conditional distribution : Lasr = logP ( Y |X ) = log X ↵2F 1 ( y ) P ( ↵|X ) , ( 1 ) where F 1 is the inverse function of F that removes the blank token from each possible alignment ↵ between X and Y . Each ↵ consists of U blanks ( ) and T tokens ( y1 , y2 , y3 , . . . , yT ) . For example , we refer ↵ = ( y1 , y2 , , , y3 , , ) as a possible alignment between X = ( x1 , x2 , x3 , x4 ) and Y = ( y1 , y2 , y3 ) , then F ( ↵ ) becomes Y . Computing Equation ( 1 ) by summing all possible alignments naively is computationally intractable . Therefore , Graves [ 7 ] introduced an efficient forward algorithm to handle this problem . In brief , it defines a forward variable ↵ ( u , t ) to represent the probability of outputting Y1 : t given X1 : u . ↵ ( u , t ) can be obtained by recursive calculation : ↵ ( u , t ) = ↵ ( u 1 , t ) ( u 1 , t ) + ↵ ( u , t 1 ) p ( u , t 1 ) , ( 2 ) where ( u 1 , t ) represents the transition probability ( predicting blank token given xu 1 and yt ) , p ( u , t 1 ) represents the emission probability ( predicting token yt given xu and yt 1 ) , and the initial condition is ↵ ( 1 , 0 ) = 1 . Application of Transducer Neural Transducer network has several advantages : 1 ) Monotonic alignment modeling . The alignment path in the output probability lattice is inherent monotonic from bottom left to top right , which makes it suitable for the tasks with monotonic alignments between the input and output sequences , e.g. , ASR . 2 ) Streaming input support . Transducer can generate text based on streaming speech input , by making the speech encoder only to see the previous speech context and using the transition ( blank token ) to determine the finish of text generation based on current input speech segment . Due to these advantages , Transducer has been widely used in ASR for streaming scenarios [ 19 , 17 , 9 ] . RNN was used as the backbone of speech encoder and text encoder in early ASR Transducer ( i.e. , RNN-T ) [ 7 ] , and then Transformer [ 29 ] was introduced into Transducer ( i.e. , Transformer-T [ 36 , 28 ] ) to enhance the ability of modeling long-term dependency . Besides ASR , Transducer can also be applied on other sequence-to-sequence tasks such as text summarization [ 34 , 35 ] , morphological inflection [ 34 ] , and machine translation [ 35 ] . The application of Transducer on TTS is still under-exploited . Yasuda et al . [ 32 ] leveraged the monotonic alignment of Transducer in TTS , but generated speech with poor voice quality . The reasons include : 1 ) It is hard to trade off the probability of transition and emission ( mel-spectrogram prediction ) when calculating the output probability lattice , since mel-spectrogram and transition predictions are modeled separately ( mel-spectrogram prediction using regression , while transition prediction using binary classification ) . 2 ) The alignments between speech and text in TTS are more difficult to learn than those in ASR . The voice generated by Yasuda et al . [ 32 ] usually has incorrect alignments such as extremely long duration for a certain phoneme . In our work , we apply Transducer to TTS by reformulating the forward algorithm with a lazy probability lattice to avoid the troublesome probability tradeoff , and adding a diagonal alignment constraint on the loss to facilitate the alignment learning . We also adjust the Transudcer model to support streaming TTS and further support both TTS and ASR in a single Transducer model , which can bring many interesting applications to explore .

Summary: Motivation: Transducer for TTS, streaming  Model: Uses convolution + self-attention on text input (either causal/auto-regressive for streaming, or full context). Convolution + auto-regressive self-attention for mel-spectogram (audio features).  Very nice idea to use transducer for TTS, to separate blank from the emissions (but not new).  Also, as one further extension: Single combined model for both ASR + TTS.