Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: * Offers an alternative differentiable monotonic alignment generator over EfficientTTS, VITS, FastSpeech 2, non-attentive Tacotron, Parallel Tacotron 2, etc.
* Using variational inference, learns a 2-level hierarchical latent representation, which can be used to influence the diversity of speech samples.
* Joint-trains a vocoder (Hifi-GAN derived) from the latent representations.
* Presents results on TTS and voice conversion, demonstrating improvement over:
  * TTS: EFTS-CNN, VITS and ablations of EFTS2
  * VC: YourTTS

strength_and_weaknesses: # Strengths:
* Making the alignment procedure fully differentiable is potentially useful.
* Using a hierarchical VAE for the acoustic model appears to be beneficial.

# Weaknesses:
* Variational Alignment Predictor
  * The variational alignment predictor’s loss has independence assumptions in its reconstruction loss that do not optimize the overall sequence probability of durations, which is a common weakness in duration prediction models.
  * The only direct apples-to-apples comparison to other duration models in the paper is “EFTS 2 (DAP)” which does not have details provided that I could find. Since the primary contribution of the paper is the differentiable variational alignment predictor, I think a much deeper comparison to other methods is warranted (e.g. EFTS 2 with the alignment model replaced with that of VITS, EFTS, FastSpeech 2, or Parallel Tacotron 2, or EFTS 2 with a stop gradient appropriately placed to show that the differentiability of the alignment model is important)
* The comparison with EFTS does not appear fair as EFTS does not use phonemes as input. 
* Does not present CMOS tests for some crucial comparisons. The VITS and EFTS2 results have the same mean MOS. The results would be stronger with a direct SxS comparison between VITS, EFTS, and EFTS 2, as MOS is not a sensitive enough metric to differentiate between systems that are substantially similar in performance.
* The claim of being an “end-to-end” TTS model is weakened by the use of phonemes as the input (which eliminates broad swathes of problems in end-to-end TTS around verbalization, pronunciation learning, and more). I suggest removing all references of being an “end-to-end” TTS model to avoid confusion.
* The use of hand-designed representations (linear-scale and mel-scale spectrograms) is a further downside in the claim that this is an end-to-end TTS model. While it’s true that hand-designed intermediate representations are not present in this model (e.g. as they are in FastSpeech), the use of hand-designed losses that guide e.g. the latent z_1 and z_2 towards representing all of the information contained within a mel spectrogram, is very similar. 
* The control features demonstrated do not seem particularly beneficial for real world use cases, as playing with latent variables does not provide many affordances. A more compelling argument would be results (e.g. in the samples page) showing how to use control to achieve a desired curation result – e.g. emotion control, copying style from a seed sample, changing speaking rate or pitch range or word level emphasis, etc. As stated, the latent variables seem to show some measure of interpretability but it's hard to get a sense how useful that is without deeper study / explication.


Nits:
* Equation 14 appears incorrect unless $z_1$ and $z_2$ are independent. Should it be $q(z|y) = q(z_1|y_{lin}) q(z_2|z_1,y_{lin})$ ?



clarity,_quality,_novelty_and_reproducibility: The manuscript is well written and clearly described.

The differentiable monotonic alignment mechanism which predicts offsets and spans appears novel, however the overall novelty of the work is limited as it is heavily derived from existing published work -- EfficientTTS, VITS, and HiFiGAN.

The reproducibility is excellent given the content of the prose and the code in the supplemental material.
summary_of_the_review: I appreciate the hard work of the authors on this manuscript. It was a well-written and enjoyable read.

My feeling is that the main contribution of the work is in the alignment network, as that is one of the large unsolved problems in TTS. Unfortunately, I do not think that with a dataset like LJSpeech we have enough variation of durations to say one way or the other that this problem is solved. As I pointed out in the strengths/weaknesses, I believe the conditional independence assumptions between phoneme timesteps are still present (we are not optimizing the sequence loss of the durations), and that the work does not directly demonstrate what is gained by making the alignments differentiable, or make careful comparison to other alignment mechanisms as the original EFTS paper did in comparing the impact of changing just the alignment mechanism between the current popular alternatives in the same overall model.

The main aspect for improvement in the experiments section is in comparison to other works. I would like to see a more apples-to-apples comparison to EFTS (which I believe uses character inputs), and a CMOS comparison to VITS -- which has the same MOS score and so cannot be compared against easily. 

In my opinion, the LJSpeech dataset is far too plain in the variation of durations in the data to say whether this method will extend well to more diverse datasets. In some sense LJ is the MNIST of TTS and has outlasted its usefulness for pushing SOTA in TTS. A more challenging dataset to test this on would be the Blizzard 2013 challenge set, or the LibriTTS dataset.

I have a quibble with the use of the term "end to end TTS", as I mention in the strengths/weaknesses section. I think phonemes as input and heavy use of spectrograms in the training objective detract from the claim of this being end-to-end TTS. I would like to reserve the term for people who are working on the difficult challenges in mapping written-domain text to speech (e.g. learning verbalization in addition to pronunciation).

Finally, the technical novelty is diminished since it is building heavily upon VITS, EFTS, and HifiGAN. Since the results are not a massive improvement over existing / prior art, and I question whether the evaluation results were a fair comparison in some cases, I think it will be better to either refine the work until it is clearly a step function in quality improvement (so the results cannot be doubted) or to find a speech-specific venue. The relevance of the alignment mechanism is already pretty specific to TTS and VC, so it may be of less general interest to the ICLR audience.

I think a speech-specific conference such as ICASSP would be a better venue.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a hierarchical VAE-based end-to-end TTS model. EfficientTTS 2 replaces the flow-based prior in VITS with convolution-based hierarchical VAE priors and introduces a fully differentiable aligner for duration modeling. EFTS 2 shows a comparable result to VITS and Your-TTS with faster inference speed.
strength_and_weaknesses: Strengths

* More efficient than VITS and YourTTS by using a hierarchical VAE prior rather than affine coupling blocks
- Propose a fully differentiable aligner
- The sample quality of EFTS2 is comparable to that of VITS. Also, the EFTS2-VC is slightly better than YourTTS with faster inference speed.
- Provide the implementation of EfficientTTS 2

Weaknesses
* Marginal improvement in parameter efficiency (0.87x inference params of VITS) and inference speed (1.5x faster than VITS)
* The motivation of a differentiable aligner is not well addressed in the result section. 
* In Figure 3, the F0 contour of EFTS2 does not seem stable. Please provide the samples with diverse $t_1$ and $t_2$. 
* Why not provide multi-speaker TTS results for EFTS 2-VC even though it uses a text encoder? 

Comments
* Since VITS is significantly faster than real-time (68x), 1.5 times faster than VITS doesn't seem great improvement. For the great improvement on the efficiency of the end-to-end TTS models, it is essential to consider both the modules for the prior distribution of the VAE and the waveform generator.
* One of the main contributions is to propose a differentiable aligner and the paper argues that it is an advantage over the existing non-differentiable aligner. It will be helpful for readers to show that the proposed differentiable aligner is better in terms of training efficiency and performance compared to the existing non-differentiable aligner such as Monotonic Alignment Search (MAS).
* Did you use $t_1=t_2=1.0$ for evaluation? Figure 3 only demonstrates the F0 contour when $t_2=0.8$. What is the effect of varying $t_2$? More analysis and samples for diverse $t_1$ and $t_2$ will help understand the role of each prior distribution. (Minor) What is $t_A$ in Fig 3? Scaling factor on alignment?
* VITS works well for both TTS and VC on the multi-speaker dataset (VCTK). This work shows only the voice conversion result of EFTS 2-VC for the VCTK dataset. As EFTS 2-VC uses a text encoder for voice conversion, it seems that EFTS 2-VC can also be used as a multi-speaker TTS model. Why not provide the multispeaker TTS results?
clarity,_quality,_novelty_and_reproducibility: The method is well written. The proposed method is 1.5 times faster than the VITS w/ comparable quality. The authors provide the implementation in the supplementary material (Reproducible)
summary_of_the_review: The proposed method has good sample quality similar to VITS, which is the basic variational framework, and some efficiency was obtained by replacing the flow-based prior distribution model with a hierarchical prior. Some experimental results are required to show the importance of the contributions of EFTS 2.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors designed and experimented a one-stage text-to-speech model using a fully differenctiable method.
strength_and_weaknesses: [Strength]
* Inference efficiency is higher than the previous work the authors compared.

[Weaknesses]
* Synthesis efficiency improved, but the quality seems to have degraded.

clarity,_quality,_novelty_and_reproducibility: - The authors claim that:
"VITS simply repeats each hidden text representation
by its corresponding duration. This repetition operation is non-differentiable thus hurting the quality of generated speech."
Although the authors use a differentiable method unlike VITS, the quality is not better than VITS, which does not sufficiently support the authors' claim. Experimental results and comparative results will be needed to support the authors' claims. For example, quality changes when the differentiable method claimed by the authors are applied to VITS must be presented.

- Since the authors did not provide samples of VITS used for MOS comparison on the demo page, I synthesized and compared those samples using the official implementation and pretrained weights of VITS. In my subjective evaluation, it is observed that VITS clearly synthesizes better quality audio, and contrary to the MOS evaluation results, the quality of the proposed model is lower. Therefore, I would like to raise a question about the MOS results; the confidence intervals of the presented scores seem to differ significantly, which means that a very small number of evaluation samples were used, or the variances of the evaluation results are very large. I would like the authors to present the full audio samples used in the evaluation and the raw data of the evaluation results.

- The authors seem to focus on the problems of the previous work (VITS), and since the proposed method is a fully differentiable model, comparison with NaturalSpeech mentioned in the manuscript is more appropriate than VITS, and it is omitted.

- The authors claim that:
"the EfficientTTS 2 (EFTS2), that
overcomes the above issues of current one-stage models with competitive model performance and higher efficiency"
Judging from the content of the paper, no issues other than "efficiency" (eg, speech quality) have been addressed. The authors should make corrections other than to improve the synthesis efficiency of the previous work.
summary_of_the_review: - There are doubts about the results presented by the authors. In addition, authors will have to diversify their comparative models.
- According to the results, there are no improvements other than efficiency, so some parts need to be corrected.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: (a) Summarize the scientific claims and findings of the paper based on your own reading and characterizations from the reviewers. 
This paper proposes a hierarchical VAE-based end-to-end TTS model: EfficientTTS 2. The proposed model uses a convolution-based hierarchical VAE to build the acoustic model and adopts a fully differentiable aligner. EfficientTTS 2 achieves comparable performance to VITS and Your-TTS with faster inference speed.
(b) What are the strengths of the paper? 
1. The alignment procedure fully differentiable is novel and potentially useful.
2. The use of the hierarchical VAE for acoustic modeling has been shown to provide better performance than existing methods. 
(c) What are the weaknesses of the paper? 
1. More comparisons with other existing TTS methods should be provided to confirm the effectiveness of the proposed approach. 
2. The overall novelty of the work is limited because it is heavily derived from existing published work -- EfficientTTS, VITS, and HiFiGAN.
3. The claimed improvements in parameter efficiency and inference speed are somehow margional.


What might be missing in the submission?
1. More comparisons with other existing TTS methods should be provided to confirm the effectiveness of the proposed approach. 
2. The paper has some degree of scientific novelty, but may not meet ICLR's very high standards.
justification_for_why_not_higher_score: 1. More comparisons with other existing TTS methods should be provided to confirm the effectiveness of the proposed approach. 
2. The paper has some degree of scientific novelty, but may not meet ICLR's very high standards.
justification_for_why_not_lower_score: 1. The proposed new alignment procedure is novel and potentially useful.
2. The proposed method indeed improves the parameter efficiency and inference speed.
Decision: Reject