Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents an approach for end-to-end speech synthesis, where every step is learned jointly with the others. Specifically,  the proposed model takes a character sequence as input and outputs an audio signal directly. The model is trained using an combination of losses, including an adversarial loss. In the experimental section, the proposed approach is compared against strong baselines, and ablation studies are presented.

Pros:
- The proposed approach is novel and a significant step toward competitive end-to-end models.
- The related work is thorough as far as I can tell.
- The experiments are insightful, showing the impact of each part of the system.

Cons:
- The performance are promising, but still below the baselines.
- The end-to-end claim is a bit misleading as the character-based model is not performing well, and the phoneme-based model is not really end-to-end, as the g2p part is not trained jointly.
- The paper is sometime not easy to follow.

Detailed comments:
- The reason behind using an adversarial loss is not really explained in the paper. A few lines before section 2.1 would help clarify that.
- The order of the sub-sections (2.1-2.7) in Section 2 is not intuitive and seems a bit random, making the section slightly hard to follow.  
- Section 2.7: "inconsistent spelling rules of the English language" -> It's not about inconsistent rules: every language has inconsistent rules, several dialects and variations in the pronunciation of words. Please rephrase.
- It's not clear which dataset was used in the experiment. If it is a private dataset, please state it clearly.

Overall, despite the paper's two main weaknesses (not fully end-to-end and lower performance),  I think it is a significant step towards fully end-to-end model and should be accepted. 
review_title: Review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a fully end-to-end TTS system with adversarial training. The proposed method has three main advantages: 1) a simple, end-to-end pipeline with only two submodules, with a fully differentiable and efficient architecture; 2) a flexible dynamic time warping to compute the loss between the predicted (generated) and true spectrograms; 3) a good MOS performance compared with the other state-of-the-art systems with more supervision. The evaluation is done very thoroughly with a variety of ablation studies.

Overall, I vote for accepting the paper. First of all, the paper is very well organized and easy to follow. Related works are well summarized, while focusing on the main differences on the proposed method. Method is explained in great detail with easy-to-follow descriptions, proper equations, and very appropriate figures. Solid evaluation is performed, and experimental results and speech samples are convincing.

Pros:
+ The structure of the paper allows an easy read.
+ The main contributions are clearly stated and supported by the experiments.
+ Major works on the similar topic are widely covered and referenced.
+ Evaluation is thorough enough to support the arguments with in-depth ablation studies.
+ Appendices provide useful, supplementary information.

Cons:
- No comparison over the computational cost nor model size is presented. It is of particular interest because the proposed model is non-autoregressive, and thus may be capable of a causal, real-time inference.
- No use of widely accepted benchmark datasets. More direct comparison would be of interest.

Minor comments/questions:
- If I understood correctly, the training sample has no phoneme-level alignment. Instead, only a sentence-speech pair is provided. If so, how do you select the corresponding text snippet from a sentence when you randomly sample 2 seconds of audio from the training examples whose length varies from 1 to 20 seconds?
- Section 2.5 on DTW is rather lengthy. DTW is a quite well-known algorithm for alignment between the two sequences, the detailed explanation on the algorithm may be omitted without the loss of readability, in my opinion.
- In Section 2.1, T is used to denote the total number of output times steps of the aligner, while T in Section 2.4 denotes the number of mel-frequency frames. Are these T's identical?
- The proposed aligner module doesn't seem to be very useful compared with the attention-based aligner as seen in the ablation study (Table 1): very small improvement from 3.551 to 3.559 MOS. Can you provide more explanations?
review_title: A fully end-to-end text-to-speech model with limited supervision that generates speech with high quality.

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a novel TTS system that 1) relies on almost no intermediate representation; and 2) is entirely feedforward instead of autoregressive. There are two major strengths in the paper. First, several modules in the proposed system are novel and smart, including the aligner and the dynamic programming loss, and it is, to my knowledge, the first feedforward text-to-speech system that does not rely on the intermediate representation. Second, the experiment result is convincing. There are, however, some room for improvement and questions for the author to clarify.

First, the elegance in the architecture is overshadowed by the complicated training algorithm. The training loss is like the superposition of common loss terms in the speech synthesis community, making the method look a bit heuristic. The complicated training algorithm also makes the proposed method harder to reproduce. It would be helpful if the authors can provide brief guidelines for readers trying to reimplement EATS, such as how to tune the hyperparameters.

Second, notice that EATS performs slightly worse than GAN-TTS, which does not quite show the benefit of end-to-end training (unlike ClariNet). I understand that there are a lot of challenges in training EATS, and the authors have briefly discussed this in Section 5. However, it is worthwhile to expand the discussion a bit by showing further experiments that demonstrate the potential benefit of end-to-end training.

Finally, although end-to-end comes with the (potential) merits of improved data efficiency and improved quality, it also has its downsides. Without a clearly interpretable hidden representation, it is harder to have direct control over prosody. How would prosody control be possible under the end-to-end framework?

Despite the weaknesses, this paper makes sufficiently novel contributions in TTS, making it above the acceptance threshold. I would look forward to further justifications of the EATS paradigm.
review_title: Elegant architectural design but complicated training

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 
## Summary

The authors propose EATS, a method for TTS from unaligned audio and text data, directly to the waveform. Previous work either use aligned phonetic features, or output spectrograms that are later converted to a waveform by a deep vocoder model.

In order to achieve this, the authors had to use several tricks, some already existing, for instance taken from the GAN TTS architecture, and some novel. The three key novelties are:
- differentiable monotonic attention with gaussian kernel and length prediction.
- dynamic time warping for the spectrogram loss.
- using both spectrogram and waveform domain discriminators

The authors provide a comprehensive ablation study with MOS score, although their model is under the state of the art by a significant margin.

## Review

This paper builds on GAN TTS, and tries to make it trainable end-to-end without aligned features.

The two main contributions, namely dynamic time warping and monotonic attention with gaussian kernel are both elegant, and can likely be used for many other applications related to time series with heterogeneous time scales. In particular, the time warping loss allows to accomodate both for the natural irregularities in spoken speech, as well as providing sufficient signal for the monotonic attention to work.

The rest of the architecture is very similar to GAN TTS except for the spectrogram domain discriminator that was added.

While the model is under the state of the art for TTS, the samples are already quite convincing. The authors conduct a thorough ablation study, both with MOS and audio samples.

Overall I think this is a really good paper, that is likely to prove quite useful for the development of end to end speech synthesis solution. As I already mentioned, I also believe that the approach of using dynamic time warping and monotonic attention can be used for other kind of time series.


## Remarks and questions to the authors

- Table 1, MOS for Tacotron 2 would be very informative. All the baselines are trained on aligned data while Tacotron is a legitimate contender for EATS as it can be trained on the same data. The point of the authors is that their methods is simpler because the training is in one stage. However, given the large number of losses and components in their model, with their respective hyper-parameters to tune, I'm not entirely sold on the simplicity argument. The tacotron 2 paper reports a MOS of 4.5 but on a private dataset.
- Section 3, [1] used the same simple L1 + log spectrogram loss as used here.
- I was surprised by the bad performance of the transformer attention, in particular in the audio samples, the output for this model is garbage towards the end of the signal. Any clue on why this would happen?
- It would be interesting to have a benchmark, in particular, can the model generate speech in real time on GPU and on CPU?

[1] SING: Symbol-to-Instrument Neural Generator, Defossez et al. Neurips 2018.
review_title: Review of End-to-end adversarial text-to-speech

Meta-review: This paper investigates a speech synthesis approach that directly generates raw audios from text or phoneme inputs in an end-to-end fashion.  The approach first maps the input texts/phonemes into a representation sequence that is aligned with the output at a lower sampling frequency by a differentiable aligner and then upsamples the representation sequence to the full audio frequency by a decoder.  A number of techniques including adversarial training and soft DTW are applied to improve the training.  The experimental results are good. There are raised concerns from the reviewers which are mostly cleared by the rebuttal of the authors.  After the rebuttal and discussion, all reviewers are supportive on accepting the paper. 
Decision: Accept (Oral)