Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this work authors present a regularized, variational autoencoder method for speech synthesis. To endow the latent space with more capacity, the authors employ a modified variational autoencoder objective, which uses a learnable Lagrange multiplier to impose a capacity limit on KL divergence between latent posterior and prior. The authors furthermore propose to decompose the latent embedding space into a two-level hierarchical representation to give generative process more control over style transfer and sample-to-sample variance. They extend earlier theoretical results providing upper bounds on the mutual information between data and its latent embedding to their hierarchical latent representation. In numerical experiments the authors evaluate their approach on a number of speech synthesis tasks involving same-text prosody transfer, inter-text style transfer, inter-speaker prosody transfer. They also analyze speech samples generated from latent samples drawn from the prior.

The paper is well-written and easy to follow, but the significance of the main contribution of the paper remains unclear to me. The authors propose to use an augmented standard VAE loss with a capacity hyperparameter and a Lagrange multiplier, but such modifications have been used before and it is not clear to me where is the novelty in there?

Moreover, the authors treat the Lagrange multiplier as a learnable parameter, which brings up the question how does it effect the learning dynamics. For instance if the KL-divergence reaches its capacity, the Lagrange multiplier may be pushed down to zero, which in turn can allow the posterior to diverge unchecked to possibly a point mass distribution? The authors provide no details on how beta (the Lagrange multiplier) evolves in their experience and how it effects the stochastic nature of the model. 

I am not sure why the authors call non-stochastic latent variable models heuristic (instead of deterministic) methods?

In Figure 1, how are  the loss values computed for variational methods? Can we see how the error bars look for different C values and embedding dimensions?

Can the authors be more clear about why for the tasks they consider, a standard (deep) VAE architecture with non-hierarchical latents does not sufficiently capture variations in the data? Can the authors quantify the differences? 

I am unfamiliar with prior work in this application area, but maybe the work is novel with respect to the application of the regularized VAE framework to speech synthesis. However, the application alone is in my opinion not a contribution that is significant enough for publication.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper introduces a new embedding method in expressive TTS by focusing on the capacity of hidden variables introduced in variational autoencoder. Such a method is supported by the KL divergence and lower bound theory and the paper well formulates/describes the capacity concept. In the experiments, the proposed method is compared with other conventional methods with well-designed subjective and objective metrics and shows the effectiveness in terms of the style transfer. The paper is well written in general.

One of the difficulties in this paper is reproducibility. The paper does not seem to provide source code and part of the data used in this paper also does not seem to be public data like libriTTS (I would be wrong but I could read that the multispeaker training data are not public), although I appreciate the authors' efforts to provide the implementation and evaluation details as much as possible in the appendix.

Another discussion of this paper is that it is not explicit to provide the meaning of the latent variables (this is a general issue in VAE) and I could not be fully convinced by the discussion and analysis based on the latent variable. There would be several semi-supervised studies to explicitly connect some (elements) of latent variables with actual attributes and I'd like to ask the authors to consider such direction to make the latent variable discussion in the paper more plausible.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: 1. Summary: This paper proposes Capacitron, a conditional variational latent variable model for TTS which allow for controllable latent variable capacity. They optimize the Lagrangian dual of the ELBO and restrict the capacity of the rate-term through a learnable, non-negative multiplier. They demonstrate the effectiveness of their approach on a range of TTS tasks such as same-text prosody transfer and inter-text style transfer, and provide extensive analyses on their latent variable capacity (in addition to comparisons to non-variational approaches based on Tacotron).

2. Decision: Weak accept. I recommend this paper for acceptance due to its strong empirical results and clear presentation of the approach/unification of existing methods.

3. Supporting arguments: The extension of Capacitron to existing methods such as [Hsu et al., 2019 and Zhang et al., 2019] is simple (basically adopting the beta-VAE approach), as the conditional generative model in both the vanilla and hierarchical forms exist already. But the authors do a thorough job evaluating the strengths and weaknesses of their method through ablation studies on latent variable capacity and comparing to existing baselines in their experiments. The results are also convincing, as demonstrated through human listening tests and the samples provided in the supplement. 

4. Feedback:
- The authors mention in Section 3.1 that in previous work, the variational posterior has the form q(z|x). While this is true for [Zhang et. al, 2019], I believe that [Hsu et. al, 2019] also uses the conditional generative model as described in Figure 3 -- it would be helpful to provide a more clear distinction between the two works in the exposition. From what I understood, this work’s contribution is not so much the introduction of the conditional generative model but identifying the effects of controlling the rate term in the ELBO decomposition.
- In Section 3.2, there is a lot of notation and several terms that make it difficult to parse Eq. 14 at first glance. For example, (1) R_L is never explicitly written out, (2) and is R == R_avg? It would be helpful to clean up this section so that eyeballing Eq. 14 would be easier for the reader (e.g. having all the relevant terms organized).

5. Questions:
- In the “Single speaker” section of Section 4.3, you mentioned that the “model has to divide the latent space into regions that correspond to different utterance lengths.” I’m curious -- is this something that was observed empirically?
review_title: Official Blind Review #2

Meta-review: This paper investigates variational models of speech for synthesis, and in particular ways of making them more controllable for a variety of synthesis tasks (e.g. prosody transfer, style transfer).  They propose to do this via a modified VAE objective that imposes a learnable weight on the KL term, as well as using a hierarchical decomposition of latent variables.  The paper shows promising results and includes a good amount of analysis, and should be very interesting for speech synthesis researchers.  However, there is not much novelty from a machine learning perspective.  Therefore, I think the paper is not a great fit for ICLR and is better suited for a speech conference/journal.
Decision: Reject