Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper puts forth adversarial architectures for TTS. Currently, there aren't many examples (e.g. Donahue et al,  Engel et al. referenced in paper) of GANs being used successfully in TTS, so this papers in this area are significant. 

The architectures proposed are convolutional (in the manner of Yu and Koltun), with increasing receptive field sizes taking into account the long term dependency structure inherent in speech signals. The input to the generator are linguistic and pitch signals - extracted externally, and noise. In that sense, we are working with a conditional GAN. 

I found the discriminator design very interesting. As the comment below notes, it is a sort of patch GAN discriminator (See pix2pix, and this comment from Philip Isola - https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/issues/39) and that is could be quite significant in that it classifies at different scales. In the image world, having a single discriminator for the whole model would not take into account local structure of the images. Likewise, perhaps we can imagine something similar in the case of audio at varying scales - in fact, audio dependencies are even more long range. That might be one reason why the variable window sizes work here. 

The paper also presents to image analogues for metrics based on FID and the KID, with the features being taken from DeepSpeech2. 

I found the speech sample presented very convincing. In general, the architectures are also presented quite clearly, so it seems that we might be able to reproduce these experiments in our own practice. It is also promising that producing good speech could be achieved by a non-autoregressive or attention based architecture.

The authors mention that they hardly encounter any issues with training stability and mode collapse. Is that because of the design of the multiple discriminator architecture?

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes to enable GAN based TTS in the time domain with the careful designs of the (non-autoregressive) generator and discriminator. There have been various trials of GAN-TTS but not so many success and I'm glad to hear that the proposed method seems to enable GAN-TTS with fast inference thanks to the non-autoregressive property. The method also proposes new objective measures inspired by the image recognition network based on the high-level features generated by end-to-end ASR, which is also another important contribution of this paper. 

My concern for this paper is reproducibility. Although I really appreciate the authors' efforts on providing implementational details in the appendix, the code and data do not seem to be publicly available, and I'm expecting that the implementation of this technique is relatively hard due to their complex designs of the generator and discriminator. Apart from that, the paper is well written overall by well describing the trend of GAN studies in the image processing and the application of such image processing oriented GAN techniques to TTS.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: I want thank the authors for solving this long-standing GAN challenge in raw waveform synthesis. With all due respect, previous GAN trials for audio synthesis are inspiring, but their audio qualities are far away from the state-of-the-art results. Although the speech fidelity of GAN-TTS is still worse than WaveNet and Parallel WaveNet from the posted sample, it has begun to close the significant performance gap that has existed between autoregressive models and GANs for raw audios. Overall, this is a very good paper with significant contributions to the filed.

Detailed comment:

1, In WaveNet, the conditional features (linguistic / mel-spectrogram) are added as bias terms in the convolutional layers. Did the authors tried this alternative architecture for the generator, which uses the white noisy z as network input (similar as flow-based models, e.g., Parallel WaveNet) and the conditional features as bias term in the convolutional layers? 

2, Could the authors comment the importance of serval architecture choices in this work? From Table 1, it seems to me that the ensemble of random window discriminators is the most important (perhaps the only important) contributing factor for the success. For example, the MOS score was boosted from 1.889 to 4.213 by replacing a single full discriminator to the ensemble of RWDs.

3, The notations in Eq. (1) and (2) are messy. Although I can figure their meaning from the context, one may clarify certain notations if they appear at the first time. 

4, The stable training (NO model collapses) is pretty impressive. Could the authors shed some light on the potential reason? Does the ensemble of RWD regularizes the training? What's your experience for training FullD (does not have random window ) and cRWD_1 (only has one random window discriminator)? Are they still very stable? Also, could the authors comment on the importance of large batch size -- 1024 for stable training of GAN-TTS? 

5, Although there is a notable difference, one may properly mention previous work Yamamoto et al. (2019), which uses GAN as an auxiliary loss within ClariNet and obtains high-fidelity speech ( https://r9y9.github.io/demos/projects/interspeech2019/ ).   

Yamamoto et al. Probability Density Distillation with Generative Adversarial Networks for High-Quality Parallel Waveform Generation. 2019.


=== update === 

Thank you for the detailed response.  
2,  Thanks for the elaboration.    
4,  It would be very interesting to see an analysis of model stability with smaller batch sizes. 

review_title: Official Blind Review #3

Meta-review: The authors design a GAN-based text-to-speech synthesis model that performs competitively with state-of-the-art synthesizers.  The reviewers and I agree that this appears to be the first really successful effort at GAN-based synthesis.  Additional positives are that the model is designed to be highly parallelisable, and that the authors also propose several automatic measures of performance in addition to reporting human mean opinion scores.  The automatic measures correlate well (though far from perfectly) with human judgments, and in any case are a nice contribution to the area of evaluation of generative models.  It would be even more convincing if the authors presented human A/B forced-choice test results (in addition to the mean opinion scores), which are often included in speech synthesis evaluation, but this is a minor quibble.
Decision: Accept (Talk)