Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a denoising diffusion probabilistic model (DDPM) to learn to produce natural spectrograms from noise without a condition. This enables them to train a generative model on unlabelled speech data. They show the effectiveness of their approach by inpainting masked out parts of a spectrogram and by showing off audio samples of unconditioned spectrogram babble, that has been vocoded to a waveform. They further propose a phoneme classification module that serves as conditioning signal for the DDPM during sampling in order to generate spectrograms that match a given phoneme sequence, turning the unconditional DDPM into a text-to-speech model, which the authors call Guided-TTS. This allows for all components of the model to be trained individually on different datasets, alleviating the need for large labelled datasets for TTS.
main_review: Strengths of the Paper:
- Proposes a solution to one of the biggest bottlenecks in TTS.
- The idea of using feedback from a separate model as conditioning signal during sampling has been done for image generation, however the application of a phoneme classifier for this in the speech domain is a very elegant solution.
- Experiments of transferring the knowledge of the phoneme-guidance module to an unseen corpus show promising results for training on speech completely without transcription.
- In general, convincing experiments and results

Weaknesses of the Paper:
- As stated in section 5.1, the model needs to be quite large. This probably also means expensive computation during both training and inference. There are no mentions of training time, training hardware or real-time-factor and hardware during inference. The latter of which is quite important for TTS.
- The phoneme recognizer still needs paired data in order to be trained. And while it seems to work well cross-speaker and even cross-accent and cross-gender as the demo page shows, I assume it would work much less well across e.g. languages or highly expressive domains. So paired data would still be needed for those applications, which are the ones that suffer the most from low-resource.

Comments:
- The weakness of paired data being required for the phoneme classifier in challenging domains could potentially be fixed by auto-alignment frameworks, that learn alignments self- or semi-supervised. Those tend to deliver less accurate results however.  Are the accurate alignments of an aligner such as the MFA used in the paper required? I would be interested in the quality-drop-off given less accurate alignments for training the phoneme recognizer, since the guided sampling seems quite complex. In other words, how precise does the guidance of the phoneme classifier have to be?
- I find the level of control that the use of a duration predictor offers intriguing. I think it would be interesting to see how well the unconditional model can handle unnatural conditions, such as holding the same phoneme for multiple seconds. 
summary_of_the_review: This paper presents a novel method for TTS with convincing experiments and results. Although it has some weaknesses, I tend to see it marginally above the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents Guided-TTS which is a TTS model that learns to generate speech from untranscribed speech data. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier to guide the generative
process of mel-spectrograms from the conditional distribution given transcript.
main_review: The proposed Guided-TTS model utilizes both diffusion probabilistic model with the gradient signals from phoneme classifier, duration predictor and speaker encoder. The overall architecture is more like pretraining + finetuning or multitask learning (if jointly trained), which is not exactly 'generate speech from untranscribed speech data', since both untranscribed and transcribed data are used in the process.

Besides that, one of the main benefits of using untranscribed speech data, which is to reduce the amount of annotated transcribed data required, is not showed in the paper. In the experiment results, the pattern of data annotation reduction is not illustrated and discussed.
summary_of_the_review: Overall, this paper proposes a new architecture of utilizing both untranscribed and transcribed speech data to generate speech with good quality.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper borrows the backbone from Grad-TTS [1], building an unconditional diffusion based model on speech input. With a phone classifier and duration predictor trained on transcribed data, and a speaker encoder trained on labeled speaker dataset, the model was able to synthesize speech on given text. Experimental results shows it's comparable with Grad-TTS [1] trained on transcribed data (by conditioning on the text during training). Ablation shows it can generalize to a diversified dataset. 


[1] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS:
A Diffusion Probabilistic Model for Text-to-Speech. In Proceedings of the 38th International
Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of
Proceedings of Machine Learning Research, pp. 8599–8608. PMLR, 2021a.
main_review: Strengths:

(1) I really like the direction, doing TTS by decoupling audio modeling, text (phoneme) modeling and speaker modeling. This could potentially  utilize large amount data and respect to each task's data requirement, for example, audio modeling could use much more noisy data.
(2) The way it adding gradients from phoneme classifier to guide unconditional model is new to speech application. The paper also propose practical trick to make it works better as norm based guidance.

Weakness:

(1) The paper doesn't really prove having additional (more diversified dataset) is better. From Table 1, using LibriTTS and LJSpeech looks about the same. To train the phoenme classifier it still need the whole LJSpeech dataset. Table 2 trying to emphasis the model can decouple generative model training and the classifier. However, it doesn't compare with a proper baseline, for example, you can also train a vocoder and mel-predictor plus speaker emb on different dataset.

(2) The paper doesn't answer how stable of such a model. One potential issue of decoupled model is the TTS cannot always follow the given text. It's better to report the ASR metric, so it would make sure the MOS wasn't misleading, for example. it could generate human sound, but with lots of pronunciation error and still get good MOS score.

(3) Following on (1), it's unclear to me for table 2 what data used to train classifier and what data for DPM.

(4) In the related section, it's better to compare with non e2e model, given the fact this model use phoneme classifier, aligner (to get duration) and a vocoder like DPM. As a comparison, traditional tts (before tacotron) model will first train a model to get linguistic features and duration and then use a vocoder on top of it. 


Other comments:
(1) wihtout -> without in introduction. 
(2) Table 1 and 2 was not very clear to me. Can you explain "In addition, we show that we can generate high quality samples using untranscribed speech of speaker", does it mea your phoneme classifier is trained on different data? It's better to mark it in the table.
summary_of_the_review: The biggest advantage of the proposed framework is it was able to use large amount speech data so it potentially could modeling wider range of prosody or other aspect of speech. However, the paper's experimental results cannot well-support it. I would happy to change the score if the author can improve follows:

(1) Adding evidence such guidance framework is relative stable, e.g. faithfully reflect the text it been given. 
(2) Demonstrate this model is better than a simpler baseline: a text-to-mel model + speaker verification network + vocoder (trained on diversified mel). My bet is if here is a win, DPM model might capture better prosody. If showing prosody win is hard, at least demonstrate this model give better results on Blizzard which i assume the training data is more noisy?
(3) The readability of Table 1 and 2 can be improved. E.g. adding what data been trained for the classifier, which speaker been used etc.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Combines an unconditional denoising diffusion probabilistic model (DDPM) with a separately trained phoneme classifier to be able to guide the generation of mel spectogram towards a given aligned phoneme sequence, thus effectively having a conditional generative model. 

This is based on the same principles which were applied before on images (Song et al. (2021b)).
main_review: Strengths:

- The application of the method from Song et al to TTS is novel.
- The demonstrated method allows the use of lots of untranscribed data for TTS, which is very useful.

Weaknesses:

- The novelty is limited. It is just a direct application of a previously proposed method (Song et al) on TTS.
- The experiments and analysis are weak.
- Even with the use of more data, the method is not able to outperform existing models.
- No published code.

Unconditional DDPM can be trained unsupervisedly, yes.
But not the phoneme classifier?

For training framewise phoneme classifier, we align transcript and speech using a forced alignment tool, Montreal Forced Aligner (MFA).
Pretrained MFA models? So implicitly making use of additional data? This should be made very clear. A statement like "Guided-TTS only makes use of LJSpeech for this experiment" is factually incorrect.

"the norm of the unconditional score"
Is this the norm of the gradient or of the log prob, or what exactly?

tts -> TTS. Other English grammar errors (missing articles etc).

HiFi-GAN vocoder, where is it from? pretrained? public?

As I understand, Table 1 uses only LJSpeech (fully transcribed) for all models, including Guided-TTS, except the last row of Guided-TTS, which uses LibriTTS for the phoneme classifier?
Grad-TTS performs better (Table 1).

Table 2 uses only the specified other corpora (e.g. HiFi TTS) for training but not LJSpeech in that case?
Why does it perform worse, even with more data? Because of mismatched conditions?
What would happen when LJSpeech + HiFi-TTS is combined, or other combinations? It should get better overall, also over the LJSpeech-only case, right? This should be verified.

Only those two tables with experiments?
This is way too little. A lot of further things should be analyzed. E.g.:

- What is the effect of the norm-based guidance and the gradient scale s?
- What is the effect of different types of classifiers, e.g. making the classifier stronger or weaker?
- Can we actually improve by the use of more (untranscribed) data? Isn't this the main motivation behind this? That was my understanding. But this is actually not shown, or not even tested. What is the point then in being able to use more data?
- Etc

Where is the code?

summary_of_the_review: While the novelty is limited, this is still a very interesting direction to investigate. However, in its current form, it provides way too little experiments and analysis.
review_title: null

Meta-review: The paper proposes using unlabelled speech data for TTS by decoupling parts of the model.
However, all reveiwers agree that the technique is already known and the experimental results are not strong enough to make advantage of training on more data.
A reject.
Decision: Reject