Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose PortaSpeech, a non-autoregressive model that can sythesize high-quality speech with reduced model size. Based on the experiment results, PortaSpeech slightly outperforms counterpart models in speech quality and shows only a slight performance degradtaion when reducing the model size.
main_review: Strength:
The proposed mixture alignment is novel.
PortaSpeech has good MOS ratings even with reduced model size.

Weekness:
The comparsion with [1] is missing since [1] also leverages a variational mel-spectragram encoder.
The comparsion of proposed mixture alignment with gaussian alignment [2][3][4] is missing
The model requires external aligner for training while other NAR models such as EATS [2], Glow-TTS, EfficientTTS [4] can be trained end-to-end.   
The robustness evalutaion is missing.

[1] Isaac Elias, et al. Parallel Tacotron: Non-Autoregressive and Controllable TTS. ICASSP 2021

[2] Jeff Donahue, et al. End-to-end adversarial text-to-speech. In ICLR, 2021.

[3] Jonathan Shen, et al. Non-attentive Tacotron: Robust and controllable neural TTS synthesis including unsupervised duration modeling. arXiv:2010.04301, 2020.

[4] Chenfeng Miao, et al. EfficientTTS: An efficient and high-quality text-to-speech architecture. ICML 2021

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper combines the high quality generation abilities of a flow network with the prosodic variation of a VAE, successfully avoiding the drawbacks of each architecture, to produce a single-speaker TTS model that can be scaled down significantly while still achieving close to state-of-the-art results.
main_review: The idea is well motivated and the paper is well written. My biased subjective opinion is that there's nothing groundbreaking so this would probably not be one of the top papers but it is solid.


3.2 Variational Generator with Enhanced Prior

It seems to me that instead of treating the complex distribution as the prior, this setup should be equivalent to continue using a gaussian prior, but adding the reverse flow and flow to the VAE encoder and decoder respectively?

Which makes me think that a gaussian prior with a sufficiently large encoder/decoder should still work just as well. The ablation experiment in the paper only removed the flow, but did not increase the size of the encoder/decoder by the size of the flow, so it is not exactly a fair comparison. It is very possible that the parameter coupling happening in the flow is key to making it work well though, which would be an interesting conclusion.


3.3 Flow-based Post-Net

This is outside of the scope of this work, but we know flow-based methods can generate waveforms directly ala WaveGlow. Have you considered doing so instead of generating mel spectrograms, which should further simplify the mobile TTS story?


4.1 Experimental Setup

In my opinion, LJSpeech is not a great dataset. It is single speaker and does not contain a lot of prosody variation. It would have been better to see results on something like LibriTTS.


4.3 Performance

I find the comparison of pitch accuracy a bit strange. For models with VAE it would make sense to use reference posterior latents and check that pitch contours match. But otherwise, it's expected that given a piece of text there can be many valid ways to vocalize it and so a higher pitch accuracy isn't necessarily always a positive result for a TTS model.


Mixture Alignment

Can the mixture alignment be used to specify absolute durations for individual words at inference time? What about for silences? If it gives this capability then this can be an important feature and is superior to the alignment in FastSpeech which can only do relative scaling.


# Originality
VAE-based and flow-based non-autoregressive TTS models are not new. I imagine there have been plenty of experiments combining the two as well. The use of a volume-preserving flow inside the VAE is novel. The ability to reduce the model size while maintaining quality with parameter sharing in the flow might be novel.


# Quality
The submission is high quality and includes relevant experiments. Some experiments on a tougher dataset would make it a better submission.


# Clarity
The submission is clear and well organized.


# Significance
The results can be a good step towards high quality mobile TTS, though it is hard to say how much traction this can gain compared to approaches like sparsifying/compressing more well known models. If the flow could be altered to generate waveforms directly while matching the quality of FastSpeech2+Parallel WaveGAN, this could immediately become a top paper in TTS.
limitations_and_societal_impact: The authors mention the potential negative societal impact, but do not offer any suggestions.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper summarizes a text-to-speech system which uses a VAE with a normalizing flow-based prior to generate spectrograms, and then uses a GAN model to convert the synthesized spectrogram to a waveform. 
  
main_review: Results indicate a very slight improvement in terms of quantitative metrics. In terms of qualitative synthesis quality to me the proposed model does not sound perceptibly better. 

The `small' version of the proposed model seems to reduce the number of parameters and the memory usage, and this might be useful in edge device applications. I am not exactly sure if the existing TTS models do not have more efficient variants though. 

Overall, the paper seems to propose a valid model, and although not ground breaking, the model seems to be doing a decent job in terms of generation quality, and being memory efficient. 
limitations_and_societal_impact: The authors do not discuss this.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: To overcome the shortcomings that autoregressive text-to-speech (AR TTS) models have (i.e. high inference latency, low robustness), non-AR TTS models such as Glow-TTS and BVAE-TTS have recently been proposed, which show competitive performance compared to AR TTS models.  
  
In this paper, a non-autoregressive (non-AR) TTS model called PortaSpseech is proposed, which effectively reduces the number of its parameters by combining VAE and normalizing flow, while showing better performance than Glow-TTS and BVAE-TTS. First, by conducting preliminary analyses on VAE and Flow, it shows which model has which advantages over the other: VAE is good at capturing prosody even with small size and Flow is good at reconstructing the frequency bin-wise details. Then, to fully utilize the advantage of each method, this paper proposes novel ways to combine the models. To help model capture prosody, lightweight VAE equipped with enhanced prior is used in decoder of the main model, and normalizing flow with grouped parameter sharing mechanism is used as a post-net, which is helpful to improve fidelity while maintaining a small number of parameters. In addition to that, it also proposes a linguistic encoder with a mixture alignment mechanism, which is effective in learning prosody compared to the phoneme-level hard alignment.

Through various experiments, this paper supports its claim that PortaSpeech can generate high-quality speech with a small number of parameters. In preliminary analyses, by measuring MOS in terms of prosody and quality separately, it shows the characteristic that each VAE and Flow model has. Then, it compares PortaSpeech with previous non-AR TTS models and shows its superiority. Plus, ablation studies also show that each architectural component is really effective to improve the performance of PortaSpeech.
main_review: This paper gives useful guidance about, in which part, each of the generative model frameworks, VAE and Flow, is helpful for TTS models to have better performance. Especially, speaking of prosody, since it is not a general concept applied to other general generative models, I think it is quite remarkable and can give a good intuition for other researchers to develop other TTS models.  
  
However, I think it would be better if there were deeper analyses because it seems that most claims are supported by comparing MOS-P/Q. Personally, it sounds sensible that Flow is helpful in generating high-quality speech, but it sounds a little weak that VAE has an advantage in capturing prosody. 
  
In terms of clarity, when I read this paper, I am a little confused about the follows:  
* For me, the word 'inter-word alignment' makes me think there is an attention mechanism between words as in Machine Translation.  
* The paper says that Lightweghit VAE is used in the decoder, but I cannot figure out how the VAE is lightweight. I think it could be better if the number of parameters of each module is separately written in the appendix.  
* I am a little confused when I am reading about enhanced prior. When I see figure 3-(c), the dotted line means residual connection? What original prior is used? How the prior is enhanced? I think the explanation about it could be added, and I suppose that there would be a reference of a paper "Improved Variational Inference with Inverse Autoregressive Flow, kingma et al."  
* In section 3.4, are the coarse-grained mel-spectrograms just the mel-spectrograms before post-net? I understand in that way, but the word 'coarse-grained' makes me think that it is a down-sampled one or something.  

Overall, I think this paper gives a valuable guide to other researchers to develop TTS models using the VAE and Flow generative models and I think this paper could be better if there were much more analytic experiments.
limitations_and_societal_impact: As mentioned in the main review, I think it would have more analytic experiments.

Related to the preliminary study, when comparing VAE and Flow architectures, since the architectures share the same encoder and length regulator, I wonder about the performance gap of capturing prosody beyond the durations.  I think it could be better if there is an analytic experiment showing the reason how VAE can help model learn prosody.

As mentioned in Section 2, there have been many models trying to reduce the size of TTS models, so it could be better if PortaSpeech is compared with the models such as SpeedySpeech and LightSpeech. (although I am aware that the models approach TTS in different ways such as teacher-student model or NAS)

I hope the idea of combining VAE and flow makes it possible to develop an end-to-end TTS model such as VITS in "Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech, Kim et al".
review_title: null

Meta-review: The paper presents an interesting blend of VAE and flow based approach for TTS. The reviewers raised several points about the comparisons -- including that some of the baselines are possibly not as good as the original work, since original implementations were not released and third party implementations had to be used for comparison. The authors address a lot of these concerns in the discussions and added analyses and clarification that I hope will make it to the final submission as they strengthen the presentation significantly. Thanks, to the reviewers for their constructive suggestions. 
Decision: Accept (Poster)