Speech-T: Transducer for Text to Speech and Beyond
Authors: Jiawei Chen
Xu Tan
Yichong Leng
Jin Xu
Guihua Wen
Tao Qin
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: transducer
text to speech
streaming
automatic speech recognition
alignment learning

Abstract: Neural Transducer (e.g., RNN-T) has been widely used in automatic speech recognition (ASR) due to its capabilities of efficiently modeling monotonic alignments between input and output sequences and naturally supporting streaming inputs. Considering that monotonic alignments are also critical to text to speech (TTS) synthesis and streaming TTS is also an important application scenario, in this work, we explore the possibility of applying Transducer to TTS and more. However, it is challenging because it is difficult to trade off the emission (continuous mel-spectrogram prediction) probability and transition (ASR Transducer predicts blank token to indicate transition to next input) probability when calculating the output probability lattice in Transducer, and it is not easy to learn the alignments between text and speech through the output probability lattice. We propose SpeechTransducer (Speech-T for short), a Transformer based Transducer model that 1) uses a new forward algorithm to separate the transition prediction from the continuous mel-spectrogram prediction when calculating the output probability lattice, and uses a diagonal constraint in the probability lattice to help the alignment learning; 2) supports both full-sentence or streaming TTS by adjusting the look-ahead context; and 3) further supports both TTS and ASR together for the first time, which enjoys several advantages including fewer parameters as well as streaming synthesis and recognition in a single model. Experiments on LJSpeech datasets demonstrate that Speech-T 1) is more robust than the attention based autoregressive TTS model due to its inherent monotonic alignments between text and speech; 2) naturally supports streaming TTS with good voice quality; and 3) enjoys the benefit of joint modeling TTS and ASR in a single network.

Speech-T: Transducer for Text to Speech and Beyond
Jiawei Chen
Xu Tan Microsoft Research Asia xuta @ microsoft.com
South China University of Technology esjiaweichen @ mail.scut.edu.co
Yichong Leng
Jin Xu
University of Science and Technology of China 1yc123go @ mail. ust..edstcedu
Tsinghua University
j-xul 88 @ munill.singhua.cddu.nn
Guihua Wen
Tao Qin
Tie- Yan Liu
South China University of Technology crghwen@ scut.enucut.
Microsoft Research Asia taoqin @ microsoft.com
Microsoft Research Asia tyliu @ microsoft.com
Abstract
Neural Transducer (e.g.. RNN-T) has been widely used automatic speech recog- nition (ASR) due its capabilities of efficiently modeling monotonic alignments between input and output sequences and naturally supporting streaming inputs. Considering that monotonic alignments are also critical to text to speech (TTS) synthesis and streaming TTS is also an important application scenario, in this work, we explore the possibility of applying Transducer it is challenging because it is difficult to trade off the emission (continuous mel- spectrogram prediction) probability and transition (ASR Transducer predicts blank token to indicate transition to next input) probability when calculating the output probability lattice in Transducer, and is not easy learn the alignments between text and speech through the output probability lattice. We propose Speech Trans- ducer (Speech-T for short), a Transformer based Transducer model that 1) uses a new forward algorithm to separate the transition prediction from the continuous mel-spectrogram prediction when calculating the output probability lattice, and uses a diagonal constraint in the probability lattice to help the alignment learning; 2) supports both full-sentence or streaming TTS by adjusting the look-aheaa context; and 3) further supports both TTS and ASR together for the first time, which enjoys several advantages including fewer parameters as well as streaming synthesis and recognition in single model. Experiments Speech-T 1) more robust than the attention based autoregressive TTS model due to its inherent monotonic alignments between text and speech; 2) naturally supports streaming TTS with good voice quality; ; and 3) enjoys the benefit of joint modeling TTS and ASR in single network.
TTS and more. However,
LJSpeech datasets demonstrate that
Introduction
seuuence-to-seqenne model widely used in automatic speech recognition ASR Transducer consists of a speech encoder that converts an input speech sequence to hidden representations. a text encoder that processes already generated text tokens autoregressively, and a joint network that predicts the next text token. Transducer networks are trained maximize the alignment probability between speech and text sequences, with the help of forward algorithm on the output probability lattice (see Figure for more details). The alignments
Transducer (ASR) 99
7
36
28
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

21 I.. First, learning learned in the output probability lattice of the Transducer are strictly monotonic, thus well modeling monotomic alignments between speech and text sequences in ASR. By restricting the speech encoder only to see the previous speech frames, Transducer naturally supports streaming inputs and is the one of the most popular solutions for streaming ASR [36 12
Although Transducer is originally designed for ASR, clearly, its advantages of modeling mono- tonic alignments between inputs and outputs and supporting streaming inputs perfectly match the requirements of text to speech (TTS) synthesis [26 300
22
6
[6]
aligmments between text and speech is important in TTS 26 and different approaches such as atten-
4 have been leveraged to model the aligmments. However, attention mechanisms suffer from unstable alignments with word skipping and
tion mechanisms [22 g 23 [
or duration prediction 20
repeating issues [o 20
R and duration prediction suffers from additional complication because of
the training of duration predictor and training-infereet
mismatch caused by the predictor (20] [1.
Thus, the inherent monotonic alignments in Transducer can be quite competitive solution for TTS
over previous methods. Second, streaming TTS (or incremental TTS) [4 support streaming text inputs, can greatly save the synthesis latency and can be widely used in online scenarios such as conversations. Unfortunately, existing solutions for streaming TTS [EI mostly rely on the argmax operation of the encoder-decoder attention values to decide the end of speech frame corresponding to an input word, which is not accurate and robust. Transducer can naturally support streaming inputs and can be better solution compared with previous methods for streaming TTS. Therefore, we explore the possibility of applying Transducer for TTS in this work.
5 2 J which aims to
However, applying Transducer to TTS is challenging due to the distinctive characteristics of speech (e.g., mel-spectrograms) generation in TTS compared with the text token generation in ASR. First, there are two actions in the output probability lattice of Transducer 7 32 emission that predicts a text token and transition that predicts a blank token to indicate null outputs in current step and the transition to the next input speech frame 77.. In ASR Transducer, the blank token is added into the token vocabulary, which means that the emission (token prediction) and transition (blank prediction) can be modeled in unified probability distribution through the softmax tokens. However, mel- speessspctooramms in TTS are continuous, which makes it hard to trade off the two probabilities when calculating the output probability lattice in TTS Transducer. As shown in our experiments, ill tradcoff between the two probabilities causes the failure of alignment learning. Second, while predicting current mel-spectrogram, previous spe-srrccss can provide enough information as they are very similar due to the continuity between consecutive mel-spectrograms [22 21 which may lead to copying the previous frame instead of learning information from the text encoder and thus harms alignment learning. Moreover, Transducer learns the appropriate alignment path from huge candidate paths in the probability lattice, which further causes difficulties in alignment learning.
all the vocabulary
In this work, propose Transformer (Speech-T for short), for TTS. First, we design a lazy forward algorithm that separates transition pre- diction from mel-spectrogram prediction when deriving the loss function of Transducer. Specifically, this lazy forward algorithm on the probability lattice only calculates the probability of transition/non- transition (using a binary classification), without the probability of mel-spectrogram prediction. Second, we introduce a diagonal constraint 2 16 2
20
based Transducer model, named SpeechTransduce
in the probability lattice to assist alignment learning, which ignores the alignment paths that deviate too much from the diagonal. The intuitive idea behind this design is that the alignments between text and speech should be monotonic, and thus lie in the diagonal region in the probability lattice. Third, by adjusting the text encoder of Speech-T to look the whole context or only the previous context, it can support full-sentence TTS or streaming TTS. Last but not the least, we further extend Speech-T support both TTS and ASR together for the first time, which enjoys several ad advantages including fewer parameters as well as streaming synthesis and recognition that Speech-T 1) is more robust than attention based autoregressive TTS models due to its inherent monotonic alignments between text and speech, 2) can naturally support streaming TTS with good voice quality, and 3) enjoys the benefit of joint modeling TTS and ASR in a single network.
single model. Experiment results on LJSpeech datasets demonstrate
The main contributions of this work are summarized as follows:
To leverage the advantages of Transducer (i.e., monotonic alignment modeling and streaming input supporting), we propose
for TTS. To the best of our knowledge, we are the first to achieve competitive voice quality in TTS and support streaming TTS using Transducer.

---

We design a lazy forward algorithm to trade off the two probabilities of mel--pectrogrann prediction and transition prediction, and design diagonal constraint to ensure the alignment learning between text and speech, to ensure the quality of Speech SpeechTranssuue Transducer.
We further extend SpeechTransduce to support both TTS and ASR at the same time, which enjoys several advantages including compact modeling (fewer parameters), streaming synthesis and recognition, self-ranking, etc. To the best of our knowledge, we are the first to successfully perform the tasks with single model.
2 Background
In this section, we first introduce the basic formulation of neural Transducer, and then describe the application of Transducer several sequence-ter requence-tossqquene to-sequence tasks.
Formulation of Transducer Neural Transducer is originally proposed to model the sequence transduction between input speech and output text sequences in ASR 77. It learns strictly mono- tonic alignments between speech (e.g., mel-spectroggram) sequence X {Ii, 22,,.., Ty } and output text (e.g., phoneme character) sequence Y {1/1 V/2 ........ yr where U and T the length of speech and text sequences respectively (U is short for utterance. T is short for text). A Transducer contains 1) speech encoder which extracts acoustic representations from the speech sequence X, 2) a text encoder works autoregressively to extract hidden representations from the text sequence Y, and 3) a joint network predict the next token in Y based the combination between the output hidden representations of the speech encoder and text encoder. This combination is similar Cartesian product, where each hidden representation of speech is con- catenated with each hidden representation of text to get a output probability lattice with a shape of (U, T). as shown in Figure [ Each node (m, n) in the output probability lattice represents that speech sequence X1m is aligned to text sequence Yim- Each vertical edge in the lattice represents an emis- sion (i.e., a normal token prediction) and the prob- ability of this edge is the corresponding probability of token prediction. Each horizontal edge repre- sents transition to the next input element (i.e., blank token added into the output vocabulary to represents null output) and the probability of this edge is the corresponding probability of blank token prediction. Each path from the bottom left to top right the lattice represents a possible alignment between the speech and text sequence.
(5,3)
066.3
003.21
014
P(5.2)
(T-3
Tean 0.2.1)
(PG.))
2
Speech (U=6)
Figure 1: The output probability lattice of ASR Transducer. The red path is a possible align- ment between the speech and text sequences.
During the training of Transducer, it marginalizes all the alignments (paths) A between X and Y over the probability lattice shown in Figure | distribution:
and minimizes the negative log- probaabbbyl of conditional
East
log X(Y -log
Σ P(a[X),
aEF-i(y)
where F-1 is the inverse function of F that removes the blank token from each possible alignment a between X and Y. Each a consists of U blanks (()) and T tokens (y1, y2, J3...., ..r) For example, (x1, 32, T3, x11 and
we refer Y = (11, 92, 93). then F(a) becomes Y.
(y1 32, o, d. U3. ó, Φ) a possible alignment between
Computing Equation by summing all possible alignments naively is computationally intractable. Therefore, Graves introduced an efficient forward algorithm to handle this problem. In brief, defines a forward variable au, to represent the probability of outputting Yı.t given X1.u- a(u, can be obtained by recursive calculation:
(2) where suu 1, represents the transition probability (predicting blank token given Tn-1 1 and yc). p(u, (uu, 1) represents the emission probability (predicting token yr given x. and 9/-1),, and the initial condition is a11, 0) 1.
a t) = a(u - 1, t)t) 1, + a(u, t 1)p(u, t 1),

---

Application of Transducer Neural Transducer network has several advantages: 1) Monotonic aligmment modeling. The alignment path in the oulput probability lattice is inherent monotonic from bottom left to top right, which makes it suitable for the tasks with monotonic alignments between the input and output eeuueneeueeeee sequences, e.g., ASR. 2) Streaming input support. Transducer can generate text based on streaming speech input, by making the speech encoder only to see the previous speech context and using the transition (blank token) to determine the finish of text generation based on current input speech segment. Due to these advantages, Transducer has been widely used in ASR for streaming scenarios 19 17 21. RNN was used as the backbone of speech encoder and text encoder in early ASR Transducer (i.e., RNN-T) 77, and then Transformer [29] was introduced into Transducer (i.e.. Transformer-T 36 28 to enhance the ability of modeling long-term dependency.
Besides ASR, Transducer can also be applied
other sequence-tor -eeeuuence tasks such as text
summarization 64 35
morphological inflection 34), and machine translation (35
The application of Transducer on TTS is still under-exploited. Yasuda et al. 22 leveraged the monotonic alignment of Transducer in TTS, but generated speech with poor voice quality. The reasons include: 1) It is hard to trade off the probability of transition and emission (mel-spectrogram prediction) when calculating the output probability lattice, since mel-spectrogram and transition predictions are modeled separately (mel-spectrogram prediction using regression, while transition prediction using binary classification). 2) The alignments between speech and text in TTS are more difficult to learn than those in ASR. The voice generated by Yasuda et al. [2 usually has incorrect alignments such as extremely long duration for a certain phoneme. In our work, we apply Transducer to TTS by reformulating the forward algorithm with a lazy probability lattice to avoid the troublesome probability tradeoff, and adding a diagonal alignment constraint on the loss to facilitate the alignment learning. We also adjust the Transudcer model support streaming TTS and further support both TTS and ASR in a single Transducer model, which can bring many interesting applications to explore.
3 Speech Transdueee
In this section, we first introduce the overall model architecture of our proposed Speech (Speech-T for short) in Section
3.1 Then we introduce our proposed lazy forward algorithm on the probability lattice in Section 3.2 Next, we propose diagonal constraint on the alignment paths to help the alignment learning in Section [333] Based on the proposed lazy forward algorithm and aligmment constraint, we derive our loss function of Speech-T in Section 3.4 Moreover, we describe the design of Speech-T T for streaming TTS in Section 3.5
At last, we extend Speech-T to support
both TTS and ASR in a single model in Section 3.6
3.1 Overall Model Architecture
As shown Figure a)) Speech-T consists of a text encoder that converts an input text (e.g., phoneme) sequence to hidden representations, a speech encoder that processes already generated speech (e.g.. mel-spectrogram) frames autoregressively. and a joint network that processes the combination of text and speech hidden representations to predict the next speech frame. Each token in text sequence X with length of T is converted into embedding while the speech sequence Y with length of L is processed by a Pre-Net with several convolutional layers, and then taken as input to the text and speech encoder respectively. As shown in Figure (b) and (c). both the text and speech encoders are based on Transformer 22). which consists of several blocks where each block contains self-attentin attentonn network and a ID convolutional network 20), both followed by a residual connection and a layer normalization. A difference between the two encoders is that the speech encoder adopts causal self-attention and convolution to only see the previous context to support the autoregressive prediction in joint network. As shown in Figure dd) the text and speech hidden representations with length of I' and U respectively are processed by linear layer and then concatenated together in a way of Cartesian product, resulting
shape of (I' U). which are further processed by several linear layers to get the output probability lattice to predict speech frames during training. We describe the specific designs to train Speech-T in the following subsections.
In both TTS and ASR, we use I' and U to represent the length of text and speech sequence respectively, and X and Y represent the input and output sequence respectively.

---

Yu+1,
Linear
Add Norm
Add Norm
Joint Net
N) Add & Norm
Conv
Causal Conv
N2
Lincar
Text
Speech Encoder
Add & Norm
Add & Norm
Encoder
Concat
Causal Self Attention
Self Atlention
Linear
Linear
Embedding
Pre-Net
Position Embedding
Position Embedding
Yu
Speech-T
(b) Text Encoder
(c) Speech Encoder
(d) Joint Net
Figure 2: (a) The overall model structure of Speech-T. (b) Text Encoder. (c) Speech Encoder. (d) Joint Net, whose output dimension is 81, where the first 80 dimension is for mel-spectrogram prediction and the remaining dimension is for transition prediction.
3.2 Lazy Forward Algorithm
In ASR Transducer (e.g., RNN-T), it is easy to trade off the probability of emission (text token prediction) and transition (blank token prediction) when calculating the Transducer loss since the blank token is added into the token vocabulary and all tokens modeled in unified probability distribution through a single softmax operation. However, mel-spectrograms in TTS are continuous and modeled regression problem while transition prediction is modeled as classification problem, which makes it hard to trade off the two probabilities in TTS Transducer. Thus, we design lazy forward algorithm on the probability lattice to separate transition prediction from mel-spectrogram prediction when deriving the loss function of Transducer. Specifically, we only calculate the probability of transition/non-transte transsttionn (by using a binary classification) to get the forward variable at,, u), without taking the probability of mel-spectrogram prediction into account:
(t,, u) = a(t 1, u) u) + att, 1)(1 - ptt, u - 1)),
where 1, u) the probability of transition at node (t (t-1,u) (i.c., the combination of the hidden representations corresponding text token æt-1 and speech frame yu). The forward algorithm called lazy because we do not eagerly calculate the exact emission probability in Equation (3) Instead, we use probability 1 o(t, not care about the exact probability of the specific mel-spectrogram prediction.
1) to represent the overall probability of emission, but do
binary classification similar way (Equation 13 and 14 Raffel al. (X).. However, there are several differences: 1) The formulation in Raffel et al. [18 is used to ensure the encoder-decoder attention (e.g., in automatic speech recognition, text summarization, and neural machine translation) monotonie, and does not handle the specific/detailed emission probabilities, i.c., the emission probability of each target token like we do; 2) Our scenario different from theirs and they can not address the problems we handle in text speech. We also noticed blog introducing similar formulation during paper rebuttal. However, there are several differences: 1) The motivation of proposing our lazy forward algorithm is that we found it is hard to trade off the two probabilities if using traditional Transducer formulation in our preliminary experiments. Although we tried different weights to balance the two loss terms, we fail to find stable one, which motivates transition and emission prediction in single joint network but not separate networks, and we design alignment constraint the lazy forward algorithm to derive the final loss function, since we found it hard to converge if not using any constraint on the calculation of the probability lattice, and we also formulate our Transducer in a streaming way and extend it model unified TTS/ASR in a single Transducer model; 3) We conduct comprehensive experimental studies to verify the effectiveness of the proposed method, but not pure brainstorming formulation.
There are some previous works [ also modeling transition/non-trannn
to propose some alternative methods; 2) We model the
3.3 Alignment Constraint
Ideally, the transition prediction can learned well through the lazy forward algorithm and thus good alignments can be obtained through the learned transition. However, it is challenging for the
5

---

aligmment learning (transition prediction) in TTS, due to several reasons: 1) While predicting current mel-spectrogram, previous spe-stpecss can provide enough information as they are very similar due to the continuity between consecutive mel-sppettsgs,sss, which leads to copying the previous frame instead of learning information from the text encoder and thus harms alignment learning. Taking the lattice in Figure B as an example, due to copy, the learned alignment path may first go up to reach the top and then go right to reach the end, or first go right to reach the rightmost and then go up to reach the end, both of which are unreasonable and deviate too much from the diagonal. 2) Transducer is trained to learn the appropriate alignment path by marginalizing the huge candidate paths over the probability lattice and thus causes difficulties in alignment learning.
Based on prior knowledge, the alignment path should be mono- tonic and lie in the diagonal region in the probability lattice. which could be good inductive bias to help the alignment learning. Thus, we introduce a diagonal constraint in the proba- bility lattice assist alignment learning, as shown in Figure 5 which ignores the alignment paths that deviate too much from the diagonal. Our alignment constraint works as following steps: (1) We construct an alignment path, as shown in the
Speech
red path in Figure 3 by aligning the text and speech sequence in training data with some alignment tools (e.g., MFA (I5]). (2) To ensure the flexibility in the constraint, we extend the alignment path by frames in both the left and right along the horizontal axis to form a diagonal and banded region, as shown in the green area in Figure (3) We only take the path within this diagonal and banded region into account for loss calculation, and thus our Speech-T can learn the alignments (transition prediction) better.
Text(T)
Figure 3: The diagonal and banded
region for alignment constraint. We take 2 as an example.
3.4 Loss Function
Based on the proposed lazy forward algorithm and alignment constraint, derive the loss function our proposed Speech-T for TTS
Lus ΣΣ (t u) E band)a u) o(t, u))l f(t, u))
(4)
where i[(t, u) E band] is an indicator function to indicate whether the index (t, u) is in the constraint banded region of the probability lattice or not. We will only calculate the loss from the indexes that are in the banded region. at, u) is the forward variable calculated in Equation B
Y1113
Speech (13 frames)
Y9:10
V/88
and 1 o(t, u) the overall probability of emission node (t, u). The L1 loss |yu+1 ftt u) (which is calcu- lated between the ground-truti mel-spectrogram frame ya+1 and the predicted mpe-sppecroggram [t, u)) is weighted by att, u1 étt, u)), where the weight means the probabil- ity of emission at node (t, u) by taking the forward variable into account. In this way, we do not need to trade off the probability of transition and certain mel-spectrogram pre- diction. In our implementation, the joint network outputs vector with 81 dimension, where the first 80 dimension for -peestpctoogram prediction and the last dimension followed by sigmoid function to predict the transition.
Y3:7
Y/44
Y1.2
X1:1 X2.3 X4.5 X6.6 X7.9 X10:1 Text (6 tokens, phonemes)
Figure 4: k look-ahead mechanism, where we take k = 1 as an example. The green part means that the speech frames can see the context of the corre- sponding text tokens.
3.5 Streaming TTS with Speech-T
Due to the advantage Transducer architecture used Speech-T, it can naturally support streaming TTS [4 5 23
When text token arrives, Speech-T predicts mel-spectrogram autoregressively, until transition predicted, which indicates the end of generation for this text token. After that, the Speech-T will wait for the next text token in the

---

X2e1
Add & Norm
Softmax
Linear
X.N)
Joint Net
Causal Conv
Add & Norm
Add & Norm
Ne
Text Encoder
Speech Encoder
& Norm
Linear
Linear
Causal Self Attention
Concat
Embedding
Pre-Net
Position Embedding
Linear
Lineur
Xx
Yu
(a) Speech-T T (Unified ITS/ASR)
(b) Text/ Encoder
(c) Joint Net
Figure 5: (a) The overall model structure of Speech-T for unified TTS/ASR. (b) Text/Speech Encoder.
(c) Joint Net, where the output dimension is 81 (same with that in Figure for TTS, and vocabulary size plus (blank token) for ASR.
streaming inputs. We restrict each token in the encoder to only see the previous tokens. Alternatively. to trade off the latency and accuracy in streaming TTS, we can allow the token in the encoder to see future tokens (k look-ahead, where context information. In practice, a word that contains multiple phoneme tokens is coming as a whole during streaming inputs. Thus, we allow each phoneme token can see all the phoneme tokens in the current word, and use k to represent the number of words instead of phonemes. We illustrate the k look-ahead mechanism in Figure 4
degenerates to only see previous tokens) extract more
3.6 Unified TTS/ASR with Speech-T
Considering that the text and speech encoder in Transducer can serve for both TTS and ASR, in this section, we further extend our proposed Speech-T as a unified model to support both TTS and ASR at the same time. We illustrate the model architecture of Speech-T for unified TTS/ASR in Figure 5 TTS and ASR share the same text encoder, speech encoder, and also joint network, except that there are some task specific branchs in the joint network for TTS and ASR individually, as shown in Figure Cic.. The model structure the text/speech encoder is shown in Figure ()), both restricting the current token/frame to only the previous tokens/frames to support the autoregressive prediction of both speech and text. In this way, both TTS and ASR can support streaming inputs. The unified TTS/ASR based on Speech-T enjoys several advantages, including compact modeling (with fewer parameters), streaming synthesis and recognition, self-ranking and speech chain [27] in single model. We demonstrate some preliminary experiment results in this paper and leave more studies on these advantages as future work.
4 Experiments and Results
In this section, we first introduce the experimental setup, and then show the results of our Speech-T for TTS in terms of audio quality and robustness, and further conduct some studies evaluate the effectiveness of the designs in Speech-T. T. At last, we show the experiment results on unified TTS/ASR with Speech-T and discuss its potential benefits.
4.1 Experimental Setup
Dataset We conduct experiments on LJSpeech 10], a public speech dataset consisting of 13,100 English audio clips and corresponding text transcripts. The total length of the audio is approximately 24 hours. We randomly split the dataset into three parts: 12500 samples for training, 300 samples for validation and 300 samples for test. We use 22.05kHz sampling rate, extract the 80-dimensional mel-spectrogram with 50 ms window size and 12.5 ms hop size, and normalize mel-spectrograms to have zero mean and unit variance. We convert the text sequence into phoneme sequence with saspheme-to-phoneee conversion 24
7

---

Implementation Details We use TransformerTTS [3 as the baseline system, which is an au- toregressive tetterrtateetiondeeee
model using Transformer [29] as the basic model structure. Both the encoder and decoder consist of 6 Transformer blocks, each with a self-attention network and feed-forward network with convolution 20 The number of self-attention head set to 2, the dimension of embedding size and hidden size are both 256, and the inner dimension of feed-forward network 1024. We adopt the encoder in TransformerTTS as our speech encoder and text encoder in Speech-T. According the requirements in full-sentence TTS and steaming TTS, these encoders can utilize causal or non-causal self-attention and convolution. We train our model on 8 Tesla V100 GPUs with batch size of 6 sentences each GPU. We use the Adam optimizer with B1 0.9, 32 = 0.98, é = 10 and follow the same learning rate schedule in [29]. It takes about 3 days for training. The predicted mne-stpectsog
transformed into audio waveform using well-traines Parallel WaveGAN vocoder [31]. More details of the model configurations and training details are presented in supplemental material (Section A.
4.2 Results of Speech-T
Audio Quality We first evaluate the perceptual qual- ity of Speech-T by mean opinion score (MOS) test, where 20 native English speakers are involved to give quality score for the synthesized speech. We compare Speech-T with other systems: 1) Recording, the ground- truth speech; 2) GT+ where the recording is converted to mel-spectrograms first and then converted back to speech by vocoder; 3) TransformerTTS. [3], which is a Transformer based autoregressive TTS model with high voice quality. All the systems (including
Table 1: The MOS results with 95% confi- dence intervals.
Method
MOS
Recording
4.00 0.07
Vocoder
3.85 + 0.08
TransformerTTS [k]
3.77 + 0.07
Speech-T
3.74 = 0.07
Speech-T) Parallel WaveGAN [1] the vocoder for fair comparison. The MOS results
shown in Table T It can be seen that our proposed Speech-T can achieve similar voice quality with TransformerTTT'. demonstrating that we successfully introduce Transducer into TTS with high voice quality
Table 2:
The robustness results of TransformerTTS and Speech-T. The number is calculated in sentence level (e.g., a sentence that has multiple skipping errors will be counted only once). # Errors means the number of sentences with either or both repeating and skipping errors.
Method
# Repeat
# Skip
# Errors
Error Rate
TransformerTTSS
6
15
16
32%
Speech-T
0
0
Robustness Since Speech-T can learn monotonic alignments between text and speech, which can gen- erate more robust speech compared with the TTS models (e.g., TransformerI'SS) using attention mechanism for alignment learning 30 [3]
Table 3: The MOS results of streaming TTS with 95% confidence intervals.
We conduct the robust test between Speech-T and TransformerTTS using 50 partic- ularly hard sentences 20 to measure the ratio of word skipping and repeating e The results are shown in Table 2 It can be seen that TransformerTTS. based on attention mechanism for alignment learning, has much higher error ratio. Benefiting from the monotonic align- ment in Transducer, Speech-T can generate all sentences with no repeating and skipping errors, achieving highly robust speech synthesis.
Method
MOS
Recording
4.02 + 0.12
Speech-T (k = xc)
3.76 + 0.11
Speech-T (k 2)
3.68 + 0.13
Speech-T (f: 1)
3
Specch-T T (k 0)
3.14
'The audio samples generated by Speech-T and the baseline systems be found in https:// speechresearch. github. wtt wt/speechttanssuucer/ 'These sentences include spellings, repeated numbers, single letters, and long sentences. We attach the 50 sentences in the supplemental material. (Section A.4)

---

Streaming TTS We evaluate our Speech-T for streaming TTS (designed in Section 3.5 We compare the voice quality the k look-ahead mechanism with different k. The results are shown in Table 3 It can be seen that the voice quality increases with the increase of k, at a cost of increased latency. k- 00 achieves the best quality, which is the full-sentence oon-straming TTS as we study in Section The voice quality of the streaming Speech-T (k 2) nearly catches up with that of the non-streaming Speech-T T (k 00).
4.3 Analysis of Speech-T
In this subsection, we conduct some analyses to verify the effectiveness of the designs in Speech-T, including the lazy forward algorithm and the alignment constraint.
Lazy Forward Algorithm We compare our lazy forward algorithm with two baselines: 1) the de- fault forward algorithm in Transducer ASR (sim-
Table 4: The CMOS results of different for- ward algorithms.
ilar to Equation 2 with swapped u and t), where the emission probability p is obtained from the mel- spectrogram ly. f f(t, u 1)] loss. 2) the forward al- gorithm in Yasuda al.
Method
CMOS
Lazy algorithm in Speech-T
0
22 which uses Gaussian
Default algorithm GMM based
N/A -1.54
mixture model (GMM) to calculate the probability of spe-sppetoo prediction and uses modified forward algorithm that still cannot address the prob-
ability tradeoff issue. Note that except for the forward algorithm, all settings are the same to ensure We can have several observations: 1) the
fair comparison. The CMOS results are shown in Table E default forward algorithm in Transducer ASR cannot generate any reasonable speech (we denote the CMOS as N/A), due to the intractable tradcoff between the probability of transition and emission in TTS. Our lazy algorithm can avoid the tradeoff issue and generate voice with good quality. 2) The voice quality by our lazy algorithm is much higher than that by GMM based 32 demonstrating the effectiveness of our algorithm.
Alignment Constraint We further study the effectiveness of alignment constraint in Speech-T. We explore two different settings: 1) removing our proposed alignment constraint Speech-T: 2) replacing our alignment constraint with a handcrafted alignment constraint, which only differs with our alignment constraint in step (1) as described in the second paragraph of Section 3.3 Instead of obtaining accurate alignment from external tool, in the handcrafted constraint, we first calculate the ratio between the number of mel-spectrogram frames and number of phoneme tokens T for every text-speech pair, and use U/T (after rounding) as the duration of each phoneme to get the alignment. The CMOS results are shown in Ta- 5
It can be seen that 1) removing the alignment constraint in Speech- T cannot generate any reasonable speech (we denote the CMOS as N/A), and 2) replacing the align- ment constraint with handcrafted constraint can make the voice qual- ity much worse, demonstrating the effectiveness of our alignment constraint in Speech-T.
Table 5: The CMOS results on alignment constraint.
Method
CMOS
Speech-T
0
Speech-T w/o alignment constraint
N/A
Speech-T with handcrafted diagonal constraint
-1.33
4.4 Results of Unified TTS/ASR
We conduct experiments to evaluate our extended Speech-T for unified TTS/ASR. The experimental setup follows that in Sec- tion 4.1 We compare the performance of both TTS and ASR between our unified Speech-T and the separate TTS and ASR models. Since our Speech-T for unified TTS/ASR works streaming manner, the separate models also work in streaming manner (using causal self-attention and convolution both in
Table 6: Comparison between unified ASR/TTS model and separate TTS and ASR models.
Method
CMOS
PER
Speech-T (Unified TTS/ASR) Speech-T (TTS only)
0
9.0%
-0.09
/
Speech-T (ASR only)
9.7%

---

the text and speech encoder) for comparison. We measure phoneme eтror rate (PER) for ASR and MOS for TTS. and show the results Table 6 can be seen that the unified TTS/ASR achieves slightly better CMOS and PER compared with the separate streaming models, showing that Speech-T can not only enjoy compact model parameter by unifying TTS and ASR, but also boost the per- formance by joint training in a single model. We discuss other benefits of unified TTS/ASR with Speech-T in supplemental material (Section A.2) due to space limitation.
Conclusions and Future Work
Motivated by the advantages of Transducer such efficiently modeling monotonic alignments between input and output sequences and naturally supporting streaming inputs, in this work, we proposed Srasucrr, with several designs address the challenges when applying Transducer to TTS. Experiment results demonstrate that Speech Transducer achieves much better robustness due the advantages of monotonic alignments while enjoying good voice quality. The extension of SpeechTransduue to unified TTS/ASR demonstrates the flexibility of Transducer based architecture and its potential for many applications, such as streaming TTS/ASR, self-reranking, and TTS/ASR co-adaption for target speakers in single model. A limitation of Trascurrr is that it suffers from slow inference speed compared with con-autoregressive TTS models. Thus, designing fast SpeechTransducee Transducer models is an interesting direction. Applying SpeechTransdiee tasks such voice conversion and speech enhancement will also be interesting areas to explore. We hope that our SpeechTransducer Transducer can inspire the community to explore more possibilities.
to more speech
10

---

References
[1] Sercan Ô Ank, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yong- guo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, al. Deep voice: Real-time neural text-to-speech. In International Conference on Machine Learning, pages 195-204. PMLR, 2017.
[2] Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, and Tao Qin. Multispeech: Multi-speaker text to speech with transformer. In INTERSPEECH, pages 4024-4028, 2020.
[3] Xie Chen, Yu Wu, Zhenghao Wang, Shujie Liu, and Jinyu Li. Developing real-time streaming transformer transducer for speech recognition on large-scale dataset. In ICASSP 2021-202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3904-5908. IEEE, 2021.
[4] Jeff Donahue, Sander Dieleman, Mikolaj Binkkwski, Erich Elsen, and Karen Simonyan. End- to-end adversarial text-to-speech. In ICLR, 2021.
[5] Nikolaos Ellinas, Georgios Vamvoukakis, Konstantinos Markopoulos. Aimilios Chalamandaris, Georgia Maniati, Panos Kakoulidis, Spyros Raptis, June Sig Sung. Hyoungmin Park, and Pirros Tsiakoulis. High quality streaming speech synthesis with low, gentenne-lenghh-indepeedddd latency. Proc. INTERSPEECH 2020, pages 2022-2026, 2026, 2020.
[6] Andrew Gibiansky, Sercan Ömer Arik, Gregory Frederick Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In NIPS, 2017.
[7] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint 1211 2012.
[8] Mutian He, Yan Deng, and Lei He. Robust sequence-to--equenee acoustic modeling with stepwise monotonic attention for neural tts. pages 1293-1297, 2019.
[9] Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, et al. Streaming end-to-end speech recognition for mobile devices. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6381-6383 IEBE, 2019.
[10] Keith Ito. The lj speech dataset. https:///keithiiio.como/// Datasppeeccc--aasee/// Dataset/ 2017.
[11] Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33, 2020.
[12] Gakuto Kurata and George Saon. Knowledge distillation from offline to streaming rnn transducer for end-to-end speech recognition. Proc. INTERSPEECH 2020, pages 2117-2121, 2020.
[13] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6706-6713, 2019.
[14] Mingbo Ma, Baigong Zheng, Kaibo Liu, Renjie Zheng, Hairong Liu, Kainan Peng, Kenneth Church, and Liang Huang. Incremental text-to-speech synthesis with prefix-to-prefix framework. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3886-3896. 2020.
[15] Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In INTERSPEECH. volume 2017, pages 498-502, 2017.
[16] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 2000-speaker neural text-to-speech. Proc. ICLR, pages 214-217, 2018.

---

[17] Rohit Prabhavalkar, Kanishka Rao, Tara N Sainath, Bo Li, Leif Johnson, and Navdeep Jaitly, A comparison of sequence-to-seee -o--euueceee models for speech recognition. In INTERSPEECH, pages 939-944 039-943,220 2017.
[18] Colin Raffel, Minh- W Luong, Peter J Liu, Ron Weiss, and Douglas Eck. Online and linear- time attention by enforcing monotonic alignments. In International Conference on Machine Learning, pages 2837-2846. PMLR, 2017.
[19] Kanishka Rao, Haşim Sak, and Rohit Prabhavalkar. Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 193-199. IEEE, 2017.
[20] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yar Liu. Fastspeech: Fast, robust and controllable text speech. In NeurIPS, 2019.
[21] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021. URL httpps://ooperreiiee http:p/s/pppparreeeeeeeeeeeorere meet/forrumSid-iilPPootttt
[22] Jonathan Shen, Ruoming Pang. Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783, IEEE, 2018.
[23] Brooke Stephenson, Laurent Besacier, Laurent Girin, and Thomas Hueber. What the future brings: Investigating the impact of lookahead for incremental neural tts. Proc. INTERSPEECH 2020, pages 215-219, 2020.
[24] Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie- Yan Liu. Token- level ensemble distillation for grapheme-tt--phoneme conversion. In INTERSPEECH, 2019.
[25] Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara. Efficiently trainable text-to- speech system based deep convolutional networks with guided attention. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4784- 4788. IEEE, 2018
[26] Xu Tan, Tao Qin, Frank Soong, and Yan Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2 15561, 2021.
[27] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Listening while speaking: Speech chain by deep learning. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 301-308. IEEE, 2017.
[28] Anshuman Tripathi, Jacyoung Kim. Qian Zhang, Han Lu, and Hasim Sak. Transformer transducer: One model unifying streaming and non-streaming speech recognition. arXiv preprint arXiv:2010.03/92 2020.
[29] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 6000-6010, 2017.
[30] Yuxuan Wang, RJ Ekerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. Proc. INTERSPEECH 2017, pages 4006-4010, 2017.
[31] Ryuichi Yamamoto, Eunwoo Song, and Jae-Me Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEFF International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.
[32] Yusuke Yasuda, Xin Wang, and Junichi Yamagishi. Initial investigation of an encoder-decoder end-to-end tts framework using marginalization of monotonic hard latent alignments. In arXiv preprint arXiv: 1153,, 2019.
12

---

[33] Jiahui Yu, Chung-Cheng Chiu, Bo Li, Shuo-yin Chang, Tara N Sainath, Yanzhang He, Arun Narayanan, Wei Han, Anmol Gulati. Yonghui Wu, et al. Fastemit: Low-latency streaming asr with sequence-level emission regularization. In ICASSP 202 2021-1021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6004-6008. IEEE, 2021.
[34] Lei Yu, Jan Buys, and Phil Blunsom. Online segment to segment neural transduction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1307-1316, 2016.
[35] Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomas Kocisky. The neural noisy channel. In ICLR, 2017.
[36] Qian Zhang, Han Lu, Hasim Sak, Anshuman Tripathi, Erik McDermott, Stephen Koo, and Shankar Kumar. Transformer transducer: streamable speech recognition model with trans- former encoders and mn-t loss. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7829-7833. IEEE, 2020.
13


Reviews:
[{"review_id": "QPRKXfSXs3", "reviewer": "Reviewer_kkqQ", "summary": "Motivation: Transducer for TTS, streaming\n\nModel:\nUses convolution + self-attention on text input (either causal/auto-regressive for streaming, or full context).\nConvolution + auto-regressive self-attention for mel-spectogram (audio features).\n\nVery nice idea to use transducer for TTS, to separate blank from the emissions (but not new).\n\nAlso, as one further extension: Single combined model for both ASR + TTS.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The application of the transducer to TTS is novel (as far as I know).\n\n> For the first challenge, we design a lazy forward algorithm that separates transition prediction from mel-spectrogram prediction ...\n\nThis separation of blank (separation of transition + emission) is also done in [\"Hybrid Autoregressive Transducer (HAT)\"](https://arxiv.org/abs/2003.07705) and [\"A New Training Pipeline for an Improved Neural Transducer\"](https://arxiv.org/abs/2005.09319).\nSo this separation is not really new (as it sounds from the text). But the application of this idea to TTS is new. And using it as an approximation for the dynamic programming (ignoring the emission prob) is also new (as far as I know). However, I think for the idea of the separation itself (transition + emission), the corresponding earlier work should be cited, which did the same kind of separation.\n\nThere are some problems in this work. Esp there are potential flaws with the loss function. The dependence on an external alignment is another problem. See below for more details.\n\nThe code should be published as well.\n\nThis work is unfortunately not quite complete, and seems more work-in-progress (see below for further comments on this).\n\nThere are some open questions, so the clarity could be improved. (See below.)\n\nThe overall organization could be better. It seems as if the work first started with the non-streaming model. And then in a later section, the streaming model is introduced. And later, the combined ASR+TTS model. The earlier sections are written in a way that completely ignores the later sections.\n\nThis work can be very significant. However, it should publish the code to better allow others to build on this. And the current work should be completed further.\n\n---\n\nSome random further comments:\n\n> ... existing solutions for streaming TTS [14] mostly rely on the argmax operation of the encoder-decoder attention values to decide the end of speech frame corresponding to an input word, which is not accurate and robust. Transducer can naturally support streaming inputs ...\n\nBut the Transducer would also use argmax during inference time for the transition? I don't exactly understand the argument about the argmax.\n\n\n> In both TTS and ASR, we use T and U to represent the length of text and speech sequence respectively, and use X and Y to represent the input and output sequence respectively.\n\nT and U is clear then. But X and Y is unclear. Input is speech in ASR, and input is text in TTS. Or does X always mean speech?\n\n\n> As shwon in Figure ...\n\nTypo.\n\n\n> A difference between the two encoders is that the speech encoder adopts causal self-attention and convolution to only see the previous context to support the autoregressive prediction in joint network. \n\nI thought the text encoder also needs to be auto-regressive to support streaming? Otherwise you must know the whole text in advance, and this is not streaming.Or I assume this streaming aspect comes later? It is a bit confusing to read this here, as this sounds as if it would apply in general, not only for the initial non-streaming experiments.\n\n\n> Loss function of our proposed Speech-TTS\n\nThe probability is not in log space? Why? This is inconsistent to the original transducer loss.\n\nThe formula is also wrong in that the loss should only use alpha(T,U) (because that recursively includes the whole probability). This is strange, and wrong, and again not the same as in the original transducer. Was it used in this incorrect way?\n\nThe approach of the \"lazy\" calculation using the emission binary prob only is fine, and actually very nice. But maybe it is applied incorrectly (as argued above).\n\nFor the comparison to the original loss (Section 4.3), I'm not sure if the emission probability is used correctly. The mel-spectogram prediction can be formulated in a probabilistic way as in the mixture density network approach. This would use a simple Gaussian, not even a mixture, not even prediction the variance, just the mean, and then you actually almost have that, except that the loss would be different (not L1 but similar as scaled L2).\n\n> Figure 3\n\nThis figure is strange in that $T > U$ (i.e. num text labels is longer than num speech frames), which is usually not the case, but you have $U >> T$.\n\n> Section 3.3, Alignment Constraint\n\nSo it turns out this needs good external alignments, from other external models, and otherwise does not work so well (Table 5). This is a big shortcoming. The model should be able to produce good alignments by itself.\n\nThe question is why. Maybe because the loss is actually flawed (as argued before)?\n\nIf it needs existing good alignments, a simpler framewise loss along the path could be used, to not calculate the joint network for the whole lattice. Similar as in [\"A New Training Pipeline for an Improved Neural Transducer\"](https://arxiv.org/abs/2005.09319).\n\n---\n\n> Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A] The quality of a speech synthesis model is relatively stable. Besides, MOS test by multiple judgers on multiple sentences can provide mean and variance of voice quality.\n\nIn my experience, it is not stable at all (at least attention-based models). The MOS test for a single model but multiple sentences does not tell anything about the variance due to different random seeds. I think error bars via different random seeds would be an interesting addition here.\n\n> Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See Section 4.1.\n\nThis seems to be incorrect. Section 4.1 just explains what the code was based on, and the modifications made. It seems that the new code will not be included. This is a serious problem. This code must be published alongside this work.\n\n---\n\nThere is a lot of potential in this work. The approach and idea of using the transducer for TTS, or even ASR and TTS combined, is very nice.\n\nUnfortunately, it is still somewhat incomplete. There are many open questions. There seem to be some errors (e.g. flaws in the loss function). There is the shortcoming of needing an external alignment, which is unexpected, and should be studied and understood better, why this is needed (maybe because of the flaws in the loss function). The new proposed loss also should be compared to the standard transducer loss, which is missing, but important for analysis and better understanding. Also, the new code must be published.\n\nIf you want to make this idea / proposal of your model public right away, I would suggest to publish it on Arxiv now anyway, and maybe submit it to some workshop, where such work-in-progress papers are more appropriate.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Does not apply.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Hdq1WI1Cmei", "reviewer": "Reviewer_fXgK", "summary": "This is a very interesting application of the Neural Transducer framework that is quite popular for end-to-end ASR to the problem of TTS. The authors adopt the dynamic programming sum over many possible alignments of text and audio to minimize a TTS-oriented L1 frame prediction loss. The details they propose are reasonable; though not the only way forward, the proposal is to decouple the alignment path probability from frame prediction probabilities, while nonetheless multiplying alignment path probability by the L1 prediction loss. Good results are presented compared to a strong Transformer TTS baseline, though I am not sure whether that baseline is itself as strong as the best WaveNet or Tacotron results. Furthermore, a unified TTS/ASR system is described sharing parameters of the acoustic/text encoders, predicting both feature frames for TTS and next symbols for ASR. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a reasonable proposal whose time has come; it is the natural TTS counterpart of the plethora of ASR work done with Neural Transducers.\n\nOverall, though the presentation is on the whole clear, the work suffers from some poor writing and grammatical errors, and from a slight lack of historical context.\n\nThe authors might want to frame the work in a more standard framework for auto-regressive generative modeling of speech. The proposal is very much the successor of HMM-based speech synthesis, given the independence assumptions of the Neural Transducer that allow the use of the forward summation of path scores. In particular, they seem to struggle with the concept of the path alignment probability given their use of an L1 loss, but that can be seen as a special case of a single gaussian probability model with shared radial variance. Eq (4) uses path probabilities to weight the L1 prediction loss, but the path probabilities themselves are only dependent on a binary transition probability model. So Eq. (4) is a hybrid loss with no clear probabilistic interpretation other than \"weighted L1 loss\". Instead, the loss could be formulated along the lines of HMM-based TTS as a generative model of the audio, and indeed as a deep generative model along the lines proposed e.g. by\n\nM. Schuster , \"On supervised learning from sequential data with applications for speech recognition\", PhD Thesis, 1999.\n\nH. Zen and A. Senior, \"Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis\", 2014.\n\nvan den Oord et al. \"Wavenet\", 2016\n\nH. Zen, \"Generative Model-Based Text-to-Speech Synthesis\", 2018\n\nGiven the Neural Transducer model's simplifying independence assumptions, a generative model along those lines can then employ the highly convenient/efficient forward path summation (= dynamic programming).\n\nSpecific comments:\n\nRe: Section 2, background: it is confusing how they have copied the Alex Graves RNN-T framework, but inverted some aspects. They use T to denote the length of the text sequence and U the length of the audio utterance, while it is the other way around in the original Graves work. Thus Eq. (2) is confusing in its non-standard sense of what is a transition to a new symbol and what is remaining in the current symbol.\n\nRe: \"Computing Equation (1) by summing all possible alignments naively is computationally intractable. Therefore, Graves [7] introduced an efficient forward algorithm to handle this problem\": the forward algorithm was proposed decades before Graves' work and used in ASR for 20-25 years before Graves, see e.g.\n\nKai Fu Lee, Automatic Speech Recognition -- The Development of the SPHINX System, 1989.\n\nL. R. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition, PTR Prentice-Hall, 1993.\n\nRe: phrasing of \"Application of Transducer\", in many places. \"Transducer\" is a general term that is particular well known in the speech and language community from the Weighted Finite State Transducer work by Riley & Mohri, and others. I think it should be \"Neural Transducer\" -- as an abbreviation for the specific model proposed by Alex Graves.\n\nRe: \"1) It is hard to trade off the probability of transition and emission (mel-spectrogram prediction) when calculating the output probability lattice, since mel-spectrogram and transition predictions are modeled separately (mel-spectrogram prediction using regression, while transition prediction using binary classification)\" : see my previous comments\n\nRe: \"2) The alignments between speech and text in TTS are more difficult to learn than those in ASR\" : how is the alignment problem fundamentally different between TTS and ASR?\n\nRe: constrained alignments and monotonic RNN-T, see the specifically monotonic constraints (no purely vertical transitions) proposed in https://ieeexplore.ieee.org/document/9003822, \"Monotonic Recurrent Neural Network Transducer and Decoding Strategies\" by  Anshuman Tripathi; Han Lu; Hasim Sak; Hagen Soltau, 2019.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any issues here that stand out in this regard.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "D9Pc9l1rfj", "reviewer": "Reviewer_GhAx", "summary": "This paper proposed the use of transformer based transducers for the text-to-speech (TTS) task. Different from transducer based automatic speech recognition (ASR) systems which utilize a blank symbol at the output, MFCC prediction in TTS cannot accommodate for that. To solve this problem, the authors propose to use a lazy forward algorithm to compute the likelihood of the output. In addition, they add a diagonal constraint to achieve text-to-speech alignments that lie close to the diagonal of the alignment lattice. Even though the MOS score is not better than the baseline transformerTTS model, the results show reduction in the sentence errors. The paper also shows preliminary results on combining ASR and TTS in the same Speech-T model and shows slight reduction in the phoneme error rate with the joint model as compared to ASR-only model. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper introduces tricks such as the lazy forward algorithm and the diagonal constraints so that transducers can be used for the TTS task successfully. The paper refers to a previous study on the use of transducers for TTS and tries to address the weaknesses of that model in this paper. \n  \nQuality: The method seems technically sound. Although qualitative results not significantly better than the baseline method, quantitative performance of Speech-T is better than the baseline model. Section 4.3 provides a limited but sufficient analysis of the effect of individual components. \nOne comment regarding the joint ASR and TTS experiment is that the ASR output is not performing ASR in the conventional sense, it rather performs a next phoneme prediction task. It is true that it takes speech and outputs a phoneme label but is slightly different than the sequence to sequence speech-to-text task.  \n\nClarity: The paper is clearly written, except a few typos. \nLine 120: nagative, Line 328: accruate, Line 338: strcutrue, Line 339: reusults\n\n\nSignificance: The paper provides practical methods for the use of transducers for the TTS task which would provide a tool for the TTS community. The joint ASR+TTS training could be interesting to the ASR community as well. \n\n+++ Updates after author-rebuttal:\nI have read the reviews by other reviewers and the author-rebuttal. A few comments on the novelty regarding the use of by Reviewer kkqQ make me re-evaluate the novelty score of the paper. This slightly lowers my original score. Also, the background information needs an update as mentioned by Reviewer fXgK. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "MOS scores in Table 1 make the quality of the output speech slightly questionable. However, quantitative analysis show that it is better than the baseline. Some questions to further explore might be if we can by-pass external modules such as grapheme-to-phoneme conversion and vocoder and achieve grapheme to time-domain speech signal synthesis.    \n", "review_text": "", "extra_scores": {"ethical_concerns": "From the paper, it was not clear how the speaker variability will be handled in the model. Is there an analysis of how male vs female speech are generated and which ones have better quality?", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to apply Neural Transducers (popular for end-to-end ASR) to the problem of TTS, as well as TTS+ASR, which is agreed to be novel enough for NeurIPS by reviewers. The paper in its current form suffers however from its presentation, which raised many questions (mostly addressed by the rebuttal, though). In addition, it was noted that the experimental work could have been more convincing, as currently the quantitive results are no better than the baseline. Overall, the paper is thus very borderline, trending accept assuming the authors will heavily work on improving the presentation.
Decision: Accept (Poster)