Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper introduces a novel approach for prosody control in text-to-speech models. A sequence of prosody representations are introduced as latent variables in variational training. Different from prior work, where prosody information are explicitly represented, the propose method learns general and implicit ones. The general representation, on the other hand, enables better modeling of prosody information, which is crucial for on-to-many mapping problem in TTS systems. The paper also provides numerical results compared with recent works, showing a significant improvements on naturalness. The authors also provides system outputs for baseline and proposed model, which shows advantages of the proposed model.
strengths_and_weaknesses: Strengths
1. The paper in general is well written. The introduction of methodology and motivation is straightforward.
2. The methods, including prosody modeling and dual parallel auto-encoder, are fairly novel.
3. The numerical results and system output support the advantage of proposed model.
4. The analysis and ablation study are thorough.

Weaknesses 
I don't see significant weaknesses in the paper. However, there are some minor issue I will later address in the question section.
questions: 1. Could you add variable notations in Figure 1 (e.g. where are x, c, z etc)
2. What do you mean by "severe linguistic information leakages" in line 118. Could you illustrate more?
3. How do you get the hard alignment matrix A in equation (2)? I found a term called hard alignment prosody extractor in section 4.5.3, could you illustrate more?
4. In equation (4), what is normalizing flow f_theta? Is there a reference to such design of priori? If not, could you illustrate more?
5. There are six weights in loss functions and the values are pretty diverse according to appendix. And only one set of values are provided. Is the model performance sensitive to weight values?
6. What does CMOS stands for on line 276?
7. In section 4.5.2, Prosody Variation, you mentioned a high variance of from standard normal distribution sampling. I was wondering if you have considered the mismatch between training and inference (mode(b) in this case). During the training time it seems that the priori is not standard.
8. This is not question but personal opinion, which is not related to the quality of the paper. While I understand the authors would like to choose a good abbreviation, (CLONE in the paper), the paper title is not super related to the novelty and strength of model, which is prosody modeling. (controllable and lossless sound very generic)
limitations: The authors address their concern on limitations and potential negative societal impact.
As to the limitations, I would suggest author provide more training details, especially on hyperparameter tuning, which can be super useful for future research to replicate the work.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors propose a controllable and lossless method of Non-Autoregressive End-to-End Text-to-Speech synthesis.
strengths_and_weaknesses: * Weaknesses
1. The authors claim that the proposed method generates lossless speech, but comparing the human recording and synthesized audio presented by the authors clearly shows that there is a quality loss. Also, considering that it is a generative model based on neural networks, there is an error in the authors' claim.
1. The authors claim that the proposed method achieves high controllability, but I would like to ask what can be controlled. There is no evidence in the paper that the desired speech audio can be synthesized by controlling the model proposed in this work.
1. The authors claim the following:
”Unlike the mel-spectrogram used in two-stage methods, the intermediate representation in single-stage methods is predicted by the model without training supervision, subject to prediction errors and over-smoothness.”\
It is not clear which intermediate features in single stage models are claimed, and the authors should provide scientific analysis of the problem of intermediate features pointed out by the authors.
1. The authors claim the following:
“we carefully study the problem of unsatisfactory high-frequency information generation in single-stage end-to-end speech synthesis, which is caused by the lack of the supervision of ground truth acoustic features during training.”\
Considering that other models are also supervised using mel-spectrogram loss similar to the proposed method, it is difficult to see that other models lack the supervision of the ground truth, and the method proposed is fundamentally the same as the previous work in the point of view.
1. It is necessary to clarify whether the inability to synthesize high frequencies with high quality is a common problem that occurs with other models or a problem that only occurs with the proposed model. In addition, in order to claim that DPA is a general method to increase the quality of high frequency bands, it must be shown that it is effective even when applied to other models.
1. The authors did not specify how the comparative models were trained. Referring to the quality of audio samples, the performance of some models differs in quality from the audio samples presented in the original work or best-performing open source implementations. Given that the human recording samples presented by the authors show uniform quality than the widely used public dataset, some models appear to be compared without convergence. All comparisons should be made between sufficiently converged models, and the authors should specify the training settings of the comparison models, and if they did not converge to the level suggested in the original work, the authors should re-experiment and revise the comparison results.
1. Most recent work use widely used public datasets and author's official implementations and pre-trained weights for fair and unambiguous comparisons. The models used by the authors for comparison are models with well-performing public implementations and pre-trained weights. By training the proposed model with the public dataset(eg, LJSpeech), authors can easily compare it with other models. This is the fairest and most reliable comparison method that can claim that the proposed method shows state-of-the-art performance.
1. Some figures need to be improved, such as font size.
questions: Included in what was described above.
limitations: The authors mentioned limitations in the appendix.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper introduces a text-to-speech model that combines several generative models such as VAE, GAN, and Normalizing Flow. The authors propose to use phoneme-level latent variable to generate prosodic latent variable. Normalizing Flow was used to make more complex probability distributions for the latent space of VAE. The evaluation on internal dataset shows that the proposed method works better than recent end-to-end SOTA model, VITS.
strengths_and_weaknesses: - strengths
    - The authors try to disentangle latent information that was entangled in a single random variable in VITS.
    
- weakness
    - more thorough subjective evaluations are required to evaluate how good the prosody modeling of the proposed method is. Simple MOS is not enough.
    - Some technical details seem wrong. See Questions.
questions: - line 121, “We use the obtained duration of phonemes to construct the hard alignment matrix” → The process of obtaining the duration of phonemes was not mentioned before this.
- How do the authors safely assume that the ouput of prosody predictor is actually prosodic latent variable?
- Why do text information included in both reference and source text the same? This is not a practical scenario and not a fair way to test the actual performance.
- Why is approximate posterior distribution denoted as q(z|x,c)? Isn’t it q(z|x)?
- It seems not technically correct to call the transformed prior with normalizing flow “a conditional prior (p(z|c)) because no condition was actually given. It makes sense in the paper VITS because the conditional prior distribution is constructed by feeding a text information to the network.
- Why do the authors use TFGAN for fastspeech2 and tacotron2 baselines when there’s HiFi-GAN vocoder that is used very often in many recent works? I don’t think I have seen TTS works that use TFGAN for the vocoder.
- Why do the authors evaluate the proposed model using internal datasets?
limitations: Both the limitations and potential negative societal impact are written very briefly in Appendix.
review_title: null

Meta-review: metareview: I am in agreement with the last 2 reviewers.

1) there are many concerns about the technical correctness of the paper that can be improved
2) more thorough evaluations and experiments are needed

i'm marking this as reject and i encourage the authors to address reviewer comments and resubmit.
confidence: Certain
Decision: Reject