Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a novel speech synthesis model which utilizes untranscribed speech data to improve the representation of the linguistic information. The authors combine the representation of wave2vec (self-supervision) as an input to a linguistic encoder which is used as a prior for an acoustic VAE. The authors augments the VAE usual ELBO with multiple terms to improve the audio quality, and also add KL divergence term between the acoustic prior and the linguistic prior (which is conditioned on text) to support speech synthesis. 
The authors perform multiple experiments to demonstrate the effectiveness of their method, and ablation studies to justify their design choices. The authors also demonstrate how the proposed method allows untranscribed text-to-speech by fine tuning the acoustic encoder over audio-only data.
strengths_and_weaknesses: Strengths:
* The audio quality of the provided samples is very good compared to competing methods.
* The proposed model demonstrates consistent improvement over existing methods.
* The proposed model supports untranscribed text-to-speech.

Weaknesses:
* The formulation of the acoustic VAE is misleading, since the "weights sharing" between the acoustic prior and the linguistic posterior is critical, but is not part of Eq. 2 (the ELBO)
* The system overview lacks clarity, and adding all the loss terms (instead of a single KL term)  could help.

questions: * Lines 85-92 :This paragraph repeats the previous paragraph?

* Line 121: NVAE is not a general hierarchical VAE but rather a particular hierarchical VAE for images

* Line 127: Eq. 2 does not make sense. As it currently stands, the rightmost term has no effect on the generative process or the acoustic posterior.
In practice, this effect is established via the weight sharing of p_theta_a(z_a|x_w2v) = q_phi_l(z_lx_w2v) but this critical information is rather easy to miss. Please fix it by adapting equation 2 to explicitly show the effect of the weight sharing.

I add here an example of formulating Eq. 2 with explicit weight -sharing: replace q_phi_l(z_l|w_w2v) with  p_theta_a(z_a|x_w2v) and separate Eq. (2) into two terms, ELBO + and KL( p(z_a|x_wv2) || p(z_a|c) )
This formulation explicitly shows that the conditional prior p(z_a|x_w2v) is regularized to match the linguistic conditional prior p(z_l |c)

* Line 128: It could be useful to define A here

* Line 283: extra "The"

* Line 306: Which KL divergence problem did you encounter? Posterior collapse? Other problems?
limitations: The authors discussed the limitations of the proposed method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In order to address the challenge in TTS that it is hard to infer both the linguistic and acoustic attributes from the text directly, the paper proposes HierSpeech, which is a high-quality end-to-end TTS system based on a hierarchical conditional variational autoencoder (VAE) utilizing self-supervised speech representations. The pipeline firstly extracts self-supervised linguistic representation from text, and then converts it to acoustic representations, and finally generates waveform.
strengths_and_weaknesses: Strengths:
1. The paper introduces self-supervised linguistic representation from wav2vec to help the TTS.
2. The experiments verify the effectiveness of proposed pipeline. The paper also discusses the potential of HierSpeech on few-shot TTS and voice conversion.
3. The paper is well-written and easy to understand.

Weaknesses:
There is no obvious weakness.

questions: I think the author can discuss the difference of propose method with method in [1], which also makes use of self-supervised linguistic representation. Whether the only difference in the pipeline is that the method in [1] leverages a spectrum synthesizer and hifi-gan but the proposed method incorporates hifi-gin into HierSpeech pipeline (since the HierSpeech also utilizes GAN loss for waveform synthesis)?


[1] VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies end-to-end text-to-speech synthesis and few-shot speaker adaptation using untranscribed speech. In particular, the authors extend from the VITS model and leverage self-supervised speech representations as the intermediate latent variable to bridge the gap between text and acoustic features. An additional phoneme prediction loss is used to refine self-supervised representation, making it closer to flow-transformed acoustic latent z_a. In addition, the injection of the intermediate latent inferred from self-supervised model enables speaker adaptation using untranscribed data.
strengths_and_weaknesses: Strengths
1. An interesting approach to integrate linguistic representations learned by SSL models into the end-to-end TTS pipeline
2. Provides a neat way of speaker adaptation with untranscribed data
3. Phoneme prediction loss seems to help not only HierSpeech but also VITS
4. Good ablation studies comparing different layers of wav2vec representations for HierSpeech and favorable results compared to the baselines (VITS).

Weaknesses
1. Some details are missing. See the question section.

questions: 1. Adding PP to the VITS posterior encoder is helpful. The posterior encoder in VITS is analogous to the acoustic encoder in HierSpeech. Have the authors tried adding PP loss to z_a also?
2. What is “sliced” z_a?
3. What is the “global conditioning” in Section 2.3 and how is it used? Is it different from speaker embedding? If it is different, does training on transcribed data require it?
4. What is the speaker encoder and how is it trained?

limitations: Yes, the authors discussed it. It is nicely addressed and mitigation strategies are presented.
review_title: null

Meta-review: metareview: all reviewers agree
* the paper is interesting and novel
* the proposed method has solid experiments and good results
* paper is well written

this paper should be accepted to the conference.
confidence: Certain
Decision: Accept