Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This is a very interesting application of the Neural Transducer framework that is quite popular for end-to-end ASR to the problem of TTS. The authors adopt the dynamic programming sum over many possible alignments of text and audio to minimize a TTS-oriented L1 frame prediction loss. The details they propose are reasonable; though not the only way forward, the proposal is to decouple the alignment path probability from frame prediction probabilities, while nonetheless multiplying alignment path probability by the L1 prediction loss. Good results are presented compared to a strong Transformer TTS baseline, though I am not sure whether that baseline is itself as strong as the best WaveNet or Tacotron results. Furthermore, a unified TTS/ASR system is described sharing parameters of the acoustic/text encoders, predicting both feature frames for TTS and next symbols for ASR. 
main_review: This is a reasonable proposal whose time has come; it is the natural TTS counterpart of the plethora of ASR work done with Neural Transducers.

Overall, though the presentation is on the whole clear, the work suffers from some poor writing and grammatical errors, and from a slight lack of historical context.

The authors might want to frame the work in a more standard framework for auto-regressive generative modeling of speech. The proposal is very much the successor of HMM-based speech synthesis, given the independence assumptions of the Neural Transducer that allow the use of the forward summation of path scores. In particular, they seem to struggle with the concept of the path alignment probability given their use of an L1 loss, but that can be seen as a special case of a single gaussian probability model with shared radial variance. Eq (4) uses path probabilities to weight the L1 prediction loss, but the path probabilities themselves are only dependent on a binary transition probability model. So Eq. (4) is a hybrid loss with no clear probabilistic interpretation other than "weighted L1 loss". Instead, the loss could be formulated along the lines of HMM-based TTS as a generative model of the audio, and indeed as a deep generative model along the lines proposed e.g. by

M. Schuster , "On supervised learning from sequential data with applications for speech recognition", PhD Thesis, 1999.

H. Zen and A. Senior, "Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis", 2014.

van den Oord et al. "Wavenet", 2016

H. Zen, "Generative Model-Based Text-to-Speech Synthesis", 2018

Given the Neural Transducer model's simplifying independence assumptions, a generative model along those lines can then employ the highly convenient/efficient forward path summation (= dynamic programming).

Specific comments:

Re: Section 2, background: it is confusing how they have copied the Alex Graves RNN-T framework, but inverted some aspects. They use T to denote the length of the text sequence and U the length of the audio utterance, while it is the other way around in the original Graves work. Thus Eq. (2) is confusing in its non-standard sense of what is a transition to a new symbol and what is remaining in the current symbol.

Re: "Computing Equation (1) by summing all possible alignments naively is computationally intractable. Therefore, Graves [7] introduced an efficient forward algorithm to handle this problem": the forward algorithm was proposed decades before Graves' work and used in ASR for 20-25 years before Graves, see e.g.

Kai Fu Lee, Automatic Speech Recognition -- The Development of the SPHINX System, 1989.

L. R. Rabiner and B.-H. Juang, Fundamentals of Speech Recognition, PTR Prentice-Hall, 1993.

Re: phrasing of "Application of Transducer", in many places. "Transducer" is a general term that is particular well known in the speech and language community from the Weighted Finite State Transducer work by Riley & Mohri, and others. I think it should be "Neural Transducer" -- as an abbreviation for the specific model proposed by Alex Graves.

Re: "1) It is hard to trade off the probability of transition and emission (mel-spectrogram prediction) when calculating the output probability lattice, since mel-spectrogram and transition predictions are modeled separately (mel-spectrogram prediction using regression, while transition prediction using binary classification)" : see my previous comments

Re: "2) The alignments between speech and text in TTS are more difficult to learn than those in ASR" : how is the alignment problem fundamentally different between TTS and ASR?

Re: constrained alignments and monotonic RNN-T, see the specifically monotonic constraints (no purely vertical transitions) proposed in https://ieeexplore.ieee.org/document/9003822, "Monotonic Recurrent Neural Network Transducer and Decoding Strategies" by  Anshuman Tripathi; Han Lu; Hasim Sak; Hagen Soltau, 2019.


limitations_and_societal_impact: I don't see any issues here that stand out in this regard.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed the use of transformer based transducers for the text-to-speech (TTS) task. Different from transducer based automatic speech recognition (ASR) systems which utilize a blank symbol at the output, MFCC prediction in TTS cannot accommodate for that. To solve this problem, the authors propose to use a lazy forward algorithm to compute the likelihood of the output. In addition, they add a diagonal constraint to achieve text-to-speech alignments that lie close to the diagonal of the alignment lattice. Even though the MOS score is not better than the baseline transformerTTS model, the results show reduction in the sentence errors. The paper also shows preliminary results on combining ASR and TTS in the same Speech-T model and shows slight reduction in the phoneme error rate with the joint model as compared to ASR-only model. 
main_review: Originality: This paper introduces tricks such as the lazy forward algorithm and the diagonal constraints so that transducers can be used for the TTS task successfully. The paper refers to a previous study on the use of transducers for TTS and tries to address the weaknesses of that model in this paper. 
  
Quality: The method seems technically sound. Although qualitative results not significantly better than the baseline method, quantitative performance of Speech-T is better than the baseline model. Section 4.3 provides a limited but sufficient analysis of the effect of individual components. 
One comment regarding the joint ASR and TTS experiment is that the ASR output is not performing ASR in the conventional sense, it rather performs a next phoneme prediction task. It is true that it takes speech and outputs a phoneme label but is slightly different than the sequence to sequence speech-to-text task.  

Clarity: The paper is clearly written, except a few typos. 
Line 120: nagative, Line 328: accruate, Line 338: strcutrue, Line 339: reusults


Significance: The paper provides practical methods for the use of transducers for the TTS task which would provide a tool for the TTS community. The joint ASR+TTS training could be interesting to the ASR community as well. 

+++ Updates after author-rebuttal:
I have read the reviews by other reviewers and the author-rebuttal. A few comments on the novelty regarding the use of by Reviewer kkqQ make me re-evaluate the novelty score of the paper. This slightly lowers my original score. Also, the background information needs an update as mentioned by Reviewer fXgK. 
limitations_and_societal_impact: MOS scores in Table 1 make the quality of the output speech slightly questionable. However, quantitative analysis show that it is better than the baseline. Some questions to further explore might be if we can by-pass external modules such as grapheme-to-phoneme conversion and vocoder and achieve grapheme to time-domain speech signal synthesis.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Motivation: Transducer for TTS, streaming

Model:
Uses convolution + self-attention on text input (either causal/auto-regressive for streaming, or full context).
Convolution + auto-regressive self-attention for mel-spectogram (audio features).

Very nice idea to use transducer for TTS, to separate blank from the emissions (but not new).

Also, as one further extension: Single combined model for both ASR + TTS.

main_review: The application of the transducer to TTS is novel (as far as I know).

> For the first challenge, we design a lazy forward algorithm that separates transition prediction from mel-spectrogram prediction ...

This separation of blank (separation of transition + emission) is also done in ["Hybrid Autoregressive Transducer (HAT)"](https://arxiv.org/abs/2003.07705) and ["A New Training Pipeline for an Improved Neural Transducer"](https://arxiv.org/abs/2005.09319).
So this separation is not really new (as it sounds from the text). But the application of this idea to TTS is new. And using it as an approximation for the dynamic programming (ignoring the emission prob) is also new (as far as I know). However, I think for the idea of the separation itself (transition + emission), the corresponding earlier work should be cited, which did the same kind of separation.

There are some problems in this work. Esp there are potential flaws with the loss function. The dependence on an external alignment is another problem. See below for more details.

The code should be published as well.

This work is unfortunately not quite complete, and seems more work-in-progress (see below for further comments on this).

There are some open questions, so the clarity could be improved. (See below.)

The overall organization could be better. It seems as if the work first started with the non-streaming model. And then in a later section, the streaming model is introduced. And later, the combined ASR+TTS model. The earlier sections are written in a way that completely ignores the later sections.

This work can be very significant. However, it should publish the code to better allow others to build on this. And the current work should be completed further.

---

Some random further comments:

> ... existing solutions for streaming TTS [14] mostly rely on the argmax operation of the encoder-decoder attention values to decide the end of speech frame corresponding to an input word, which is not accurate and robust. Transducer can naturally support streaming inputs ...

But the Transducer would also use argmax during inference time for the transition? I don't exactly understand the argument about the argmax.


> In both TTS and ASR, we use T and U to represent the length of text and speech sequence respectively, and use X and Y to represent the input and output sequence respectively.

T and U is clear then. But X and Y is unclear. Input is speech in ASR, and input is text in TTS. Or does X always mean speech?


> As shwon in Figure ...

Typo.


> A difference between the two encoders is that the speech encoder adopts causal self-attention and convolution to only see the previous context to support the autoregressive prediction in joint network. 

I thought the text encoder also needs to be auto-regressive to support streaming? Otherwise you must know the whole text in advance, and this is not streaming.Or I assume this streaming aspect comes later? It is a bit confusing to read this here, as this sounds as if it would apply in general, not only for the initial non-streaming experiments.


> Loss function of our proposed Speech-TTS

The probability is not in log space? Why? This is inconsistent to the original transducer loss.

The formula is also wrong in that the loss should only use alpha(T,U) (because that recursively includes the whole probability). This is strange, and wrong, and again not the same as in the original transducer. Was it used in this incorrect way?

The approach of the "lazy" calculation using the emission binary prob only is fine, and actually very nice. But maybe it is applied incorrectly (as argued above).

For the comparison to the original loss (Section 4.3), I'm not sure if the emission probability is used correctly. The mel-spectogram prediction can be formulated in a probabilistic way as in the mixture density network approach. This would use a simple Gaussian, not even a mixture, not even prediction the variance, just the mean, and then you actually almost have that, except that the loss would be different (not L1 but similar as scaled L2).

> Figure 3

This figure is strange in that $T > U$ (i.e. num text labels is longer than num speech frames), which is usually not the case, but you have $U >> T$.

> Section 3.3, Alignment Constraint

So it turns out this needs good external alignments, from other external models, and otherwise does not work so well (Table 5). This is a big shortcoming. The model should be able to produce good alignments by itself.

The question is why. Maybe because the loss is actually flawed (as argued before)?

If it needs existing good alignments, a simpler framewise loss along the path could be used, to not calculate the joint network for the whole lattice. Similar as in ["A New Training Pipeline for an Improved Neural Transducer"](https://arxiv.org/abs/2005.09319).

---

> Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [N/A] The quality of a speech synthesis model is relatively stable. Besides, MOS test by multiple judgers on multiple sentences can provide mean and variance of voice quality.

In my experience, it is not stable at all (at least attention-based models). The MOS test for a single model but multiple sentences does not tell anything about the variance due to different random seeds. I think error bars via different random seeds would be an interesting addition here.

> Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See Section 4.1.

This seems to be incorrect. Section 4.1 just explains what the code was based on, and the modifications made. It seems that the new code will not be included. This is a serious problem. This code must be published alongside this work.

---

There is a lot of potential in this work. The approach and idea of using the transducer for TTS, or even ASR and TTS combined, is very nice.

Unfortunately, it is still somewhat incomplete. There are many open questions. There seem to be some errors (e.g. flaws in the loss function). There is the shortcoming of needing an external alignment, which is unexpected, and should be studied and understood better, why this is needed (maybe because of the flaws in the loss function). The new proposed loss also should be compared to the standard transducer loss, which is missing, but important for analysis and better understanding. Also, the new code must be published.

If you want to make this idea / proposal of your model public right away, I would suggest to publish it on Arxiv now anyway, and maybe submit it to some workshop, where such work-in-progress papers are more appropriate.

limitations_and_societal_impact: Does not apply.
review_title: null

Meta-review: The paper proposes to apply Neural Transducers (popular for end-to-end ASR) to the problem of TTS, as well as TTS+ASR, which is agreed to be novel enough for NeurIPS by reviewers. The paper in its current form suffers however from its presentation, which raised many questions (mostly addressed by the rebuttal, though). In addition, it was noted that the experimental work could have been more convincing, as currently the quantitive results are no better than the baseline. Overall, the paper is thus very borderline, trending accept assuming the authors will heavily work on improving the presentation.
Decision: Accept (Poster)