Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors propose to do neural text to speech, conditioned on attributes such as valence and arousal and speech rate. A seq-to-seq network is trained using stochastic gradient variational Bayes.        
The idea is interesting and new.

The method section could be made clearer by giving first some intuition, explaining the formulas in the prose and introducing the terms used.

Regarding the crowd sourced MOS: how were the ratings obtained? how many subjects were used for the rating? How were they selected? Was each rater presented with samples from each method? Or was each method assessed by different groups of raters?

The experimental setting is a little weak, relying mostly on the Mean Opinion Score. It would be useful to include more evalution, for instance:
* run a speech recognition method on the generated speech and measure the error rate

* examples could be included in the supplementary  (e.g. spectrograms)

* For the evaluation of emotional speech to be meaningful, the proposed classifier should be tested, and more detailed given (hyper-parameters, training setting, validation/test splits?, etc). In particular, it would be useful to compare the proposed method for affect classification to an existing (state-of-the-art) methodology. A state-of-the-art method should be ideally be directly used to classify the generated sequences into emotional classes.

The authors collected data in studio conditions, as such it is hard to compare. Will the data be released? How much hdata was collected?
It seems that what the authors effectively do is condition on discrete emotion classes, not valence and arousal, which are continuous measures of affect.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposes to use a semi-supervised VAE based text-to-speech (TTS) for expressive speech synthesis. The main contribution of this paper is that it can provide more explicit interpretation for the latent variable with the help of the supervised learning component. The formulations and implementations in the main body are quite high-level and we may not easily understand the technical/implementation details only with the main body but, in other words, the paper is well written to convey their main messages and of course some details are described in the appendix. The experiments show the effectiveness in terms of subjective (MOS) and objective (cepstral distance etc.) with a lot of audio examples on the demo page.

My concern for this paper is a lack of reproducibility. The paper uses the in-house data to perform their experiments and the code does not seem to be publically available. Also, the paper misses several detailed information (e.g., detailed configurations of the Wave RNN vocoder, what kind of neural network toolkits and libraries). The high computational cost ("distributed across 32 Google Cloud TPU chips") would also make the reproducibility difficult. I also would like to see whether this method can have some experimental comparisons with (Hsu et al., 2018) with their postprocessing to show the distinction in terms of the performance in addition to the functional difference. 

Comments:
- In general, the font size in the figures is too small
- Figure 1: it's better to have an explanation of "CBHG". People outside the end-to-end TTS community cannot understand it.
- Can you also control the noise level as shown in (Hsu et al., 2018) but more explicitly within this framework? Controlling the noise level is quite important for end-to-end TTS, and I think this method can fit this direction because we can easily obtain the noise attribute (supervision) by data simulation or annotate the noise. 
- Section 2, second paragraph y_{1...t} --> y_{1...k} (?)
- equation (6), classification loss: I think this part requires more clarifications in this timing, e.g., by giving an example of classification tasks.
- I think it's better to add what kind of (neural) vocoder is used in the main body (not in the appendix) to asses the sound quality for their experiments.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Overview:

This paper proposes to use semi-supervised learning to enforce interpretability on latent variables corresponding to properties like affect and speaking rate for text-to-speech synthesis. During training, only a few training items are annotated for these types of properties; for items where these labels are not given, the variables are marginalised out. TTS experiments are performed and the approach is evaluated objectively by training classifiers on top of the synthesised speech and subjectively in terms of mean opinion score.

I should note that, although I am a speech researchers, I am not a TTS expert, and my review can be weighed accordingly.

Strengths:

The proposed approach is interesting. I think it differs from standard semi-supervised training in that at test time we aren't explicitly interested in predicting labels from the semi-supervised labelled classes; rather, we feed in these labels as input to affect the generated model output. I agree that this is a principled way to impart interpretability on latent spaces which are obtained through unsupervised modelling aiming to disentangle properties like affect and speaking rate.

Weaknesses:

This work misses some essential baselines, specifically a baseline that only makes use of the (small number of) labelled instances. In the experiments, the best performance is achieved when gamma is set very high, which (I think) correspond to the purely supervised case (I might be wrong). Nevertheless, I think a model that uses only the small amount of labelled data (i.e. without semi-supervised learning incorporating unlabelled data) should also be considered.

As a minor weakness, the evaluation seems lacking in that human evaluations are only performed on the audio quality, not any of the target properties that are being changed. For affect specifically, it would be helpful to know whether the changes can be perceived by humans. As a second minor weakness, some aspects of the paper's presentation can be improved (see below).

Overall assessment:

The paper currently does not contain some very relevant baselines, and I therefore assign a "weak reject".

Questions, suggestions, typos, grammar and style:

- p. 1: "control high level attributes *of of* speech"
- p. 2: It would be more helpful to state the absolute amount of labelled data (since 1% is somewhat meaningless).
- p. 2: I am not a TTS expert, but I believe the last of your contributions have already been achieved in other work.
- Figure 2: It would be helpful if these figures are vectorised.
- p. 4: "*where* summation would again ..."
- Figure 4: Is there a reason for the gamma=1000 experiment, which performs best in (a), not to be included in (b) to (d)?
- Section 5: Table 1 is not references in the text.
- Section 5.1: "P(x|y,z_s,z_u)" -> "p(x|y,z_s,z_u)"
- In a number of places, I think the paper meant to cite [1] but instead cited the older Kingma & Welling (2013) paper; for instance before equation (6) (this additional loss did not appear in the original VAE paper).

References:

[1] https://arxiv.org/abs/1406.5298

Edit: Based on the author's response, I am changing my rating from a 'weak reject' to a 'weak accept'.
review_title: Official Blind Review #4

Meta-review: The authors propose to enforce interpretability and controllability on latent variables, like affect and speaking rate, in a speech synthesis model by training in a semi-supervised way, with a small amount of labeled data with the variables of interest labeled.   The idea is sensible and the results are very encouraging, and the authors have addressed the initial concerns brought up by the reviewers.
Decision: Accept (Poster)