Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper describes work on Parallel Neural Text-To-Speech.  

There are a number of efforts in this direction going on simultaneously.  FastSpeech is one that was noted in public comment.  I have seen other work under submission as well.  I don't believe this undermines the fundamentals of the work, but it will color claims of being "first" by each submission.

WaveVAE is an interesting approach to a parallel neural vocoding.  However it performs works than the IAF distilled ClariNet parallel vocoder with the same amount of parameters.  It is not clear what advantages WaveVAE has over ClariNet.

More problematic, ParaNet is described as providing a speedup over Deep Voice while maintaining quality.  However, this claim is somewhat misleading.  The maintained quality is only achieved when using an autoregressive wavenet vocoder.  It appears as though the inference speed up is only measured on the feature to mel conversion, while the wall-clock inference time is (likely) dominated by wavenet inference (a notoriously computationally intensive process).  When evaluated with parallel vocoders (ClariNet, WaveVAE or WaveGlow) ParaNet performs quite poorly when compared to Deep Voice 3. (This is especially true when using WaveGlow -- the best performing vocoder for DV3 MOS=3.96, ParaNet MOS=3.21).  While there are fewer attention errors (cf. Table 2) this doesn't seem to impact MOS to a great degree.

Attention Distillation is somewhat equivalent to using an external alignment or duration model as is done in classical TTS.  Requiring this external resource somewhat undermines the claim of being an "end-to-end" TTS system.  It is worth comparing attention distillation to, say, training a traditional alignment model based on forced alignment and feeding the target durations as a conditioning feature along with the text.  This is what ParallelWaveNet (https://arxiv.org/pdf/1711.10433.pdf) does.  From that perspective it is not clear why ParaNet+neural vocoder should be considered as "more parallel" than ParallelWaveNet or other IAF vocoding approaches paired with a traditional front end.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes improvements to the TTS architecture in DeepVoice3. 

For the most part (as far as I am aware), neural architectures for TTS are encoder-decoder based. The backbone can either be an RNN or a wavenet style model (autoregressive). Synthesizing audio from the feature representation (mel spectrogram) by a neural vocoder is usually an autoregressive model from the wavenet family. 

In the current work, the main novelties seem to be the following:

1) Replace the autoregressive component in the decoder (seemingly inspired by DeepVoice3) with a non-autoregressive model. This could be very advantageous because synthesis in autoregressive models can be quite slow owing to the sample level generations (and to overcome this defect, faster sampling with inverse autoregressive flows has been used in parallel wavenet, clarinet, etc. and probability density distillation). Here, if I can interpret the paper correctly, the authors use a trained attention model (autoregressive), to distill attention for the non-autoregressive setup used, which would otherwise have difficulties learning alignment. The paper claims speed up of ~50X over DeepVoice 3 which is very significant.

2) In the mel to audio converter (vocoder), the proposal is a VAE Wavenet, with appropriate modifications for the sequence modeling. The authors mention that there are similarities to the approach used in Clarinet (this has a closed form KLD between the distilled distributions, which makes things easy). 

Experiments: It is mentioned that poor attention alignments are the cause of many issues in these architectures (repeats, mispronunciations, skipped words, etc.), and they go on to show that their architecture fares well as regards these metrics, while maintaining a comparable MOS score with DeepVoice3+Wavenet. 

Evaluations: I am reasonably convinced by the audio demos provided. 

My thoughts: 
The paper is generally a good addition to the TTS literature. These are difficult to implement, brittle setups, and a practitioner could spend a lot of time debugging their broken attention curves. To that end, I feel that any technique that throws light into the modeling process would be useful for the practitioner. It is suggested that we can use a non-autoregressive model with speedups. Likewise, they also use a wavenet VAE, which they claim can be trained without distillation (could the authors please clarify this point?)

I do, however, feel that the paper has a few drawbacks. 

1) The presentation is not at all clear. This is a very subjective comment, but I feel that this work might be unreadable to someone who hasn't studied the literature (starting from Tacotron, DeepVoice 1, 2, 3, wavenet, parallel wavenet, transformer, distillation, etc.). Furthermore, the paper does not seem to make things any clearer with succinct architecture diagrams. Just as a comparison, I would like to draw attention to Tacotron (1, 2) in which I think the details can actually be worked out with some effort. 

2) Why are we using the WaveVAE - is this just to do away with the probability distillation in wavenet? Are we also using the  IAF setup as described in Kingma's work?

3) I really feel that we need more architecture diagrams for this work to be useful. That being said, the authors do provide a list of hyperparameters for potential use. 

4) Distilling attention seems to require a previously trained autoregressive attention model. What if we don't have one ready at hand?

In summary, while I see that the work will definitely be useful to the Speech Synthesis practitioner, the clarity of the paper could be improved and we need a few more diagrams (maybe even code) to make it implementable. 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This submission belongs to the field of text-to-speech synthesis. In particular, it looks at the problem of efficient training and inference. Many current state-of-the-art approaches in the area employ an autoregressive component at one or another part of text-to-speech model making training and/or inference complicated/slow. The main idea behind this paper is to remove autoregressive components. 

I believe there is a great deal of interest for efficient and fast text-to-speech. I believe stirring more interest towards non-autoregressive approaches is the right direction. I believe that the approach proposed does indeed accomplish the task of removing autoregressive components from the text-to-speech model. I find however that the results with parallel neural vocoders for the proposed approach to lag behind (autoregressive) Deep Voice 3 with the same set of vocoders even though when an autoregressive vocoder is used the situation is different. 

I find that the presentation of this work to be lacking a balance. This work makes 2 contributions 1) non-autoregressive text-to-spectrogram and 2) non-autoregressive spectrogram-to-speech model. 

 The first contribution is presented in a completely verbal manner to describe the complex process of text-to-speech mapping. For instance, I cannot find a proper technical description of 1) anywhere in the introduction, only statements that it is non-autoregressive and parallel. 

The second contribution is squashed into a single page. Due to complexity involved I believe each of these contributions needs to be written (and assessed) separately. For instance, on page 6 you have a) encoder, b) decoder, c) VAE objective, d) STFT loss all discussed in very short details. 

Other comments:
1) The Related work section would benefit from a bit more connected story than a list of RNN, flow and VAE based approaches. 
2) Block diagrams are helpful but not having a mathematical description makes them more ambiguous than they should be.
3) "residual channel 256 conditioned", "miserable attention errors"
4) What is significance between 6 and 8 attention errors made by ParaNet and DV3?


review_title: Official Blind Review #1

Meta-review: The paper proposed a non-autoregressive attention based encoder-decoder model for text-to-sepectrogram using attention distillation. It is shown to bring good speedup to conventional autoregressive ones. The paper further adopted VAE for the vocoder training which trains from scratch although performs worse than existing method (e.g. ClariNet). 

The main concerns for this paper come from the unclear presentation:
* As the reviewer pointed out, there're some misleading claims that the speedup gains was obtained without the consideration of the full context (i.e. not including the whole inference time).
* The paper failed to clear present the architectures developed/used in the paper and the differences from those used in the literature. The reviewers suggested the use of diagram to aid the presentation.
* The two contributions are unbalanced presented. Due to the complexities involved, it's better to explain things in more details. 
The authors acknowledged the reviewers comments during rebuttal, but did not make any changes to the paper.
Decision: Reject