Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Post rebuttal and discussion
========================
Several reviewers have pointed out that the paper needs more comparisons/ablations with existing models (e.g. Paranet/Fastnet). To this end, I think we at least need a comparison with Paranet, which is a 'comparable' non-autoregressive CNN based VAE based model with a few other components such as attention distillation. 

There are components in the paper that could do with more ablation studies 
- argmax with straight through estimator
- some guidelines on BVAE blocks and tuning

In light of these points, together with the fact that we don't have any theoretical novelties in this paper, I reduce my score to 6. Even so, I feel that the paper would be a valuable contribution because 
a) A generative model (GAN/VAE/VQVAE/Flow based models/score matching based models) might add extra benefit in the synthesis problem, as compared with a supervised model without a similar generative component such as Tacotron. The NVAE has been shown to significantly outperform the regular VAE in image generation tasks. It stands to reason that it would do well in speech generation also. 
b) Speed, robustness and ease of implementation (although this remains to be demonstrated).
 
Initial Review
===========

This paper proposes a non-autoregressive (non AR) way to perform text to speech synthesis. It uses a VAE based setup - adapted from the recent image paper NVAE to build two stacks of hierarchical VAE blocks  (in priors), one going bottom up and the other, top down. The key claims are that it results in improved speed, and reduced model footprint from using a non AR architecture, with excellent quality comparable to the best autoregressive/recurrent methods in Tacotron2 [2] and non AR glow-TTS[3].

The work contains many interesting ideas for TTS, and I am very interested in seeing how this work pans out in practical speech synthesis applications.

Key ideas:
1. The bidirectional stack, which they call BVAE is adapted from the recent NVAE work which has produced stellar image generations. The model uses 1D convolutions under the hood, in contrast with the fashionable, but slow autoregressive flows or recurrent models. If one can get such a model to work, it could be advantageous in effecting savings in computational time and model size. 

During training, at the top of the bottom-up stack, text features are inflated to the size of the mel spectrogram features, and reconstructed with the top down BVAE stack. For inference, text is inflated to an expanded text matching audio mels, and then sent down the top-down stack to give a mel sample.

2. Attention modeling: An important consideration here is to align text and mel, commonly done with an attention mechanism. In this work, the attention alignment shows up as a duration model, which is rather interesting, and seemingly gives additional flexibility. After aligning text and mel (using dot product), the alignment can be reinterpreted as a duration model by comparing phoneme and mel frame alignments. Furthermore, they use a discrete match with argmax rather than a sum over all attention alignments as is generally done. This also necessitates the use of the straight-through estimator while backpropagating since the durations are rounded entities. This type of modeling seems also to be used in the Glow-TTS  work but with alignments determined through dynamic programming.

I found the result that the model is not very sensitive to alignment mismatches to be quite remarkable.

3. Fittings for robustness during inference: They use several instructive ideas - jittering text, adding positional embeddings, diagonal penalty (since alignment is mostly diagonal) and KLD annealing. 

4. Analyses - ablations to see which of the VAE blocks affect the result by varying temperature (from Glow [3]).

My thoughts:
Generally, the paper made for fascinating reading. Having worked with Tacotron, I have always felt that adding a VAE to that (RNN based) setup would improve its generative capabilities by giving it additional regularization qualities, among other things. That we can see the model perform better when we add jitter and can also respond to the duration specified seems to corroborate that in a loose way (figure 10). 

- Could the authors clarify how the duration modeling results in 'monotonic' alignments? As far as I can see, the argmax guarantees a unique match, but is monotonicity necessary?

From section 5.3.2:
"Since the text is forced to be used monotonically in the duration-based generation, it makes the model more robust to the attentionerrors while making fewer pronouncing mistakes."

- A comparison with an equivalent soft attention implementation might be insightful. 

- Multi Speaker TTS: I am wondering how this model would perform in a multispeaker dataset, say libritts. One aspect that the paper does not touch in detail is in its capabilities as a generative model. It would be interesting, for instance, to see if this model can in any way separate speaker style from content with a multispeaker model.

Overall, I think this paper would be a good addition to the body of speech synthesis work, and recommend that it is accepted.


[1] NVAE: https://arxiv.org/pdf/2007.03898.pdf
[2]: Tacotron2: https://arxiv.org/pdf/1712.05884.pdf
[3] Glow-TTS: https://arxiv.org/pdf/2005.11129.pdf
[4]: Glow: https://arxiv.org/pdf/1807.03039.pdf



review_title: Novel, fast architecture with many insightful ideas - accept

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper presents BVAE-TTS, which applies hierarchical VAEs (using an approach motivated by NVAE and Ladder VAEs) to the problem of parallel TTS.  The main components of the system are a dot product-based attention mechanism that is used during training to produce phoneme duration targets for the parallel duration predictor (that is used during synthesis) and the hierarchical VAE that converts duration-replicated phoneme features into mel spectrogram frames (which are converted to waveform samples using a pre-trained WaveGlow vocoder).  The system is compared to Glow-TTS (a similar parallel system that uses flows instead of VAEs) and Tacotron 2 (a non-parallel autoregressive system) in terms of MOS naturalness, synthesis speed, and parameter efficiency. 


Reasons for score:
Overall, I think the system presented in this paper could be a valuable contribution to the field of end-to-end TTS; however, from a machine learning perspective, the contributions are incremental and quite specific to TTS.  In addition, I have some slight concerns about the clarity of the presentation that made it harder to understand the (fairly simple) approach and its motivation than I’d expect from an ICLR paper.  Finally, the quality of the speech produced by the system is only evaluated on a single dataset and uses only 50 synthesized examples in the subjective ratings.  For these reasons, I feel this paper would be a better fit for a speech conference or journal after addressing the evaluation and presentation issues, but I would still support acceptance if other reviewers push for it and my concerns are addressed. 


High-level Comments:
* The speed, parameter efficiency, and MOS results are quite promising.  However, when considering the Glow-TTS paper (which this seems like a direct followup to), the system improvements seem quite incremental (replace flows with HVAEs and replace the monotonic alignment search with soft attention plus argmax).  
* Incremental system improvements are great if they result in significant improvements that are demonstrated through rigorous experiments, however, compared to Glow-TTS, the experiments are not nearly as comprehensive and convincing. Listening to a few of the audio examples provided in the supplemental materials, I don’t get the sense that the audio quality is significantly better than that of Glow-TTS as is suggested by the MOS numbers (BVAE-TTS sounds a bit muffled to my ears relative to Glow-TTS). 
* Since this system uses the same deterministic duration prediction paradigm as Glow-TTS (and other parallel TTS systems), it suffers from the same duration averaging effects and inability to sample from the full distribution of prosodic realizations.  
* The motivation would be made clearer if you were more specific early on about the potential advantage of VAE's relative to flows however you want to describe it (parameter efficiency, more flexible layer architectures, more powerful transformations per layer, etc.).  
* I'd recommend providing similar motivation for using dot-product soft attention plus straight-through argmax instead of Glow-TTS's alignment search or other competing approaches.  Is it because it's a superior approach or just because it's different from existing approaches? 

Detailed Comments:
* Section 2:  I don’t believe Tacotron is actually the *first* end-to-end TTS system.  Maybe it was the first to gain widespread attention, but I know that char2wav (if you count that as e2e TTS) preceded it chronologically in terms of first arxiv submission date.
* Section 2: The Related Work section is fairly redundant with information that is already presented in the introduction.  It might be worth combining the two sections.  This should free up space for additional experiments, explanations, or analysis. 
* Section 4.1: The first paragraph here was quite confusing upon a first reading.  I had to read the second sentence (“Via the attention network…”) many times to understand what was being described.
* Section 5.2: I’m curious how you arrived at a sample temperature of 0.333.  Was this empirically tuned for BVAE-TTS or in response to Glow-TTS’s findings? 
* Section 5.2, “Inference Time”: It seems important to include details about the hardware platform used to gather the speed results. 
* There are minor English style and grammar issues throughout the paper that make the paper slightly more difficult to read.  Please have the paper proofread to improve readability. 

Update (Nov 24, 2020):
After reading through the author responses and the updated version of the paper, I feel like a sufficient number of my concerns have been addressed to increase my score to 6.  Specifically, the motivation has been made clearer, the related work section is no longer redundant with the intro, and the authors gave an adequate explanation about the necessity of their attention-based alignment method.  
review_title: Potentially valuable contribution to parallel TTS, with some concerns. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper combined fastspeech with a hierarchical VAE (or ladder VAE? in their paper it called bidirectional VAE) to achieve parallel and high quality text-to-mel syntheisis. 

The paper claims these contributions: (1) Introducing an online fashion for duration prediction, instead of distillation in FastSpeech and ParaNet. So the model is more e2e. (2) Introducing an BVAE, which extract features hierarchically to better capture prosody (overcome one-to-many) problem. During inference, can use the prior directly. This is directly than previous VAE application in TTS, which is only use to capture residual information. (3) it's faster and with same quality as autoregressive Tacotron and with better quality than other published non-autoregressive model.

The key strength of this paper is the architecture is new. I think using a hierarchical VAE here is reasonable. 

My concerns mostly from the conclusion and experiments.
(1) The paper claims compare to previous non-autoregressive model, they are more e2e, since both FastSpeech (also use duration predictor) and ParaNet (without VAE) rely on distillation. However, there is another paper called FastSpeech 2 (https://arxiv.org/abs/2006.04558, published on June 8th), the model also claim " 1) removing the teacher-student distillation to simplify the training pipeline".  Can the author explain the difference? Also i think need to cite that paper because it published in June and very related.
(2) As mentioned in (1), ParaNet and FastSpeech1/2 are very related to this paper. But why only compare with waveglow?
(3) The paper has an ablation study section, but it missing couple very simple baseline. 1) remove VAE, purely predict mel-features based on duration and phoneme embeddings. 2) using a simple VAE instead of hierachical one. How it affect the performance.
(4) One key claim of this paper is that it is as good as Tacotron 2. However, for the in-domain test, the 0.2 behind. By listening the audio samples provided by the author, it indeed significantly worse. The out of domain looks better, I suspect the reason is Tacotron 2 has some attention failures due to it not robust as duration based model. A proper baseline here, is a FastSpeech model. Could you also provide OOD samples? It's really hard to believe such prosody gap can be filled by switch domain.
(5) Back to the original motivation, why we need non-autoregressive model for TTS? For neural based TTS system, most of time is in vocoder. Even we assume the speed for mel-to-spec is important, I don't think measure speed with batch size = 1 is important, because non-autoregressive model can not be streaming. A proper comparison is measure FLOPS and throughput. This might make more sense for offline TTS. This is a minor concern, as long as the quality are good enough.
(6) The paper claims their model is more compact, but there is no comparison for a smaller Taco2 model or other non-autoregressive model. 

In summary, based on my understanding, this paper proposed a new non-autoregressive based text-to-mel model with quality regression but possible better robustness. My opinion is that it's a borderline for ICLR, since the importance of the proposed VAE was not well justified, and the quality was not as good as autoregressive model. 
review_title: An interesting paper with some nonsolid claims

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: Neural models that autoregressively generate mel spectrograms from text (or phonemes), such as Tacotron, have been used to generate high quality synthetic speech. However, they suffer from slow inference speed due to their autoregressive nature. To alleviate this, non-autoregressive models have been proposed, such as FastSpeech and Glow-TTS. The proposed model, BVAE-TTS, is yet another non-autoregressive speech synthesis model (outputting spectrograms), with two key advantages over the aforementioned models: (a) no autoregressive teacher model is required, as in FastSpeech, which simplifies training, and (b) fewer parameters are needed than in Glow-TTS, since there is no bijectivity constraint (allowing a more expressive architecture to be used). Models are compared with inference speed and MOS, and BVAE-TTS compares favorably on both both metrics when compared to Glow-TTS.

Pros:

1. The evaluation of the model is done well, in a clear way. LJSpeech is used, a dataset which is commonly used and easily accessible. MOS and inference speech are provided, and error bars are provided for MOS values. BVAE-TTS is compared to Glow-TTS and Tacotron 2 (one other non-autoregressive model, and one well-known AR baseline), and hyperparameters are provided. A single vocoder (pretrained WaveGlow) is used on all models, isolating the effect of the spectrogram prediction model used.

2. Section 4.3, pertaining to using attention distributions to learn a duration predictor, is interesting and novel. Using positional encodings is standard and using a loss guide is unsurprising. However, while jitter and straight-through estimators are not uncommon, all of these things together make a compelling and novel approach to using attention to infer discretized durations and compensate for that train-test mismatch well. I believe that a similar technique could be used in other models as well.

3. The model is an application of similar ideas from image synthesis, which is interesting, in that it demonstrates that some of those techniques work equally well for spectrogram synthesis. This sort of cross-modal result points to the strength of the method being used, which is a valuable data point for the research community.

Cons:

1. The biggest weakness of this paper, in my view, is that deciphering the model itself is quite difficult. Although the model bears resemblance to NVAE (for which code is released), understanding the fine details is tricky, and the paper does little to aid in that effort. 

In particular, understanding the exact layer inputs and outputs and parameters of the normal distributions being used is difficult, and I believe the paper would benefit significantly from a pseudocode explanation of the network. For example, I did not understand why the generative model produced both $\mu_l$ and $\Delta \mu_l$, and whether $\mu_l$ was predicted with a dense layer or was the accumulation of the prior BVAE stacks' $\Delta \mu_l$ values (and similar for $\Sigma$). 

I also wonder why the output of the attention layer is not provided to the encoder; perhaps there is a fundamental reason for this which I am missing, or perhaps this is simply an architecture choice.

A very clear explanation of the method itself, perhaps as psuedocode for where the means and variances come from and which features they interact with and what it sampled when, would in my view make this among the top papers.

Recommendation:  Accept. The paper is well written and results are strong, although I would prefer if the method itself were explained more clearly.
review_title: Great results and thorough evaluation with a well-motivated model, but presentation could be better

Meta-review: Non autoregressive modelling for text to speech (TTS) is an important and challenging problem. This paper proposes a deep VAE approach and show promising results. Both the reviewers and the authors have engaged in a constructive discussion on the merits and claims of the paper. This paper will not be the final VAE contribution to TTS but represents a significant enough contribution to the field to warrant publication. It is highly recommended that the authors take into account the reviewers' comments.
Decision: Accept (Poster)