Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: FLOWTRON

1.
The authors present Flowttron, an autoregressive text-to-speech network that is
a merging of the well-established Tacotron architecture along with flow-based
generative models. The benefits to Flowtron are its simplicity to train and
comparable MOS to other state of the art models. Moreover, the model is able to
produce various styles of speech depending on how the latent variables are
sampled, as well style transfer between seen and unseen speakers.

2.
The introduction and discussion in the paper are strong and review the
necessary background information. The experiments are comprehensive and
convincing. Although there are artifacts in the audio samples (repeated
phrases, distortion), the variation of speech is impressive.

The overall description of the model lacks technical thoroughness. The
description of the flow networks is sufficient. However, how precisely
tacotron2 is modified to incorporate the flow network is severely lacking.
Figure 2 is confusing. I also don't understand what the authors mean by "it is
also possible to reverse the ordering of the mel-spectrogram frames in time
without loss of generality"

It is not clear to me how the values in Table 2 are calculated.

On page 4, the second paragraph, what is the purpose of synthesizing samples
with Tacotron2 while varying the pre-net dropout probabilities.

3/4. 
I would recommend an accept, given the authors can address the major technical
problems with the paper. Specifically, the description of the system needs to
be much more clear. The work is interesting, the experiments are sufficient,
and the audio samples are convincing.


6. There are many typos that should be fixed.

review_title: FLOWTRON

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes a specific approach to the hot topic of  auto-regressive Flow-based speech synthesis. The fundamentals of coupled flow layers are exposed pretty well, but I quickly got lost in the details of the work. I have a number of specific comments to address that. Nonetheless, I find the paper is still a good (if confusing) read addressing a very interesting area.

"Their assumption is that variable-length embeddings are not robust to text and speaker perturbations, which we show not to be the case" -- in what specific aspects do those works you refer make that assumption?

Re: Figure 1, "Figure 1 shows that acoustic characteristics like timbre and F0 correlate with portions of the z-space of Flowtron models trained without speaker embeddings." Label the axes of Figure 1?

Eq. (7): I can guess, but can you confirm the meaning of the circle-with-dot operator?

"Normalizing flows are typically constructed using coupling layers": I nearly, but don't completely, understand the concept of the coupling layers. AIUI, the trick is that they are forward layers with two components that are inverses of each other.  I guess that Eqs. 5-7 in fact define just that. However, how is z_t computed? f(z_t) and its inverse defined in terms of s_t and b_t, which are functions of z_{1:t-1}. So where does z_t come from?

"we define z1 as a prependded 0 vector" : (only one "d" in prepended). For z1, what is the 0 vector component prepended to? Or is z1 itself a zero vector?

"To evaluate the likelihood, we take the mel-spectrograms and pass them through the inverse steps of
flow conditioned on the text and optional speaker ids, adding the corresponding log |s| penalties, and
evaluate the result based on the Gaussian likelihoods."

For clarity, I suggest:
- Define s.
- Define the complete log-likelihood criterion.

"Our text encoder modifies Tacotron’s 2": spelling.

"Figure 1 shows evidence that speech several characteristics present in mel-spectrograms": check grammar.

"Section 2.5, Posterior Inference": I suggest stating at this point how posterior inference will be used to generate actual samples.

Section 3.2: The inference method has not yet been explained.

Table 1: what happens if you use more Flows? Do the MOS scores improve?

"3.3.1 SPEECH VARIATION": i agree that FlowTron seems to be able to generate more varied speech, but this is only a good thing if the samples are still natural. I don't see a metric or assessment of the naturalness of those more-varied samples (maybe i missed it).

Section 3.3.2, "Our different speaker interpolation samples show that Flowtron is able to gradually
and smoothly morph one voice into another." Where is this measured/reported in the paper?

"3.4.2 SEEN SPEAKER WITH UNSEEN STYLE... We compare samples generated with Flowtron and Tacotron 2 GST to evaluate their ability to emulate a speaking style unseen during training of a speaker seen during training.": where is this comparison shown in the paper?


Table 2: what are the units?

"Table 2 shows that while the samples generated with Tacotron 2 GST are not able to emulate
the high-pitched style from RAVDESS, Flowtron is able to make Sally sound high-pitched as in the
“surprised" style." I find it a bit hard to interpret Table 2. Is the goal to match the reference mean and standard deviation as closely possible?

"3.5 SAMPLING THE GAUSSIAN MIXTURE In this last section we provide visualizations ...": where do you provide it?

re: 3.5.1 VISUALIZING ASSIGNMENTS and 3.5.2 TRANSLATING DIMENSIONS, same comment as before, where are the results provided?



References:
"Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo
Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, et al. Deep voice: Real-time neural
text-to-speech. arXiv preprint arXiv:1702.07825, 2017b." --> remove superfluous "O" for first author? Or add missing "O" in the previous reference?
review_title: Very interesting paper that I found rather hard to follow.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a text-to-speech synthesis system, called Flowtron which uses a normalizing flow to generate a sequence of mel-spectrogram frames. The difference between the proposed Flowtron and the previously prosed flow-based methods is, the authors argue as the main contributions, its ability to produce more diverse and expressive speech samples of specific speech attributes by sampling from the latent space. Evaluation is done using the two public datasets, and a number of experiments are performed to show that the proposed method not only achieves a good MOS score in general but also generates speech samples with variation, including the style transfer.

Overall, the paper is fairly well organized with figures and tables in appropriate positions. However, it is sometimes difficult to follow, sections 3.3.2 - 3.5.2 in particular, because there are too many short subsections without supporting figures or tables. Furthermore, it is also hard to navigate through the folders and files provided in the supplementary material to listen to the speech samples.

Evaluation is well done. The proposed method yields a comparable MOS score to a reference (Tacotron 2) in terms of speech quality. However, focus in experiments is on the many features that are not feasible in other approaches, which include variation control such as pitch or duration, interpolation between samples, and style transfer. But it still remains unclear that how a user can explicitly control these speech features or attributes in order to generate speech with the desired non-textual information, which requires much more than simple and random variation in pitch and/or duration.

Minor comments: 
- There are several typos (even in the abstract) and grammatical errors throughout the text.
- Again, it is quite difficult to find and compare the speech samples that correspond to features describe in the text. It would be better to embed the samples on the website for easier navigation.
review_title: Proposes a flow-based TTS with high expressivity and style variation.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ### Summary

This paper introduces a novel application of normalizing flows to speech synthesis, allowing direct optimization of spectrogram log-likelihoods which results in more natural variation at inference compared to L1/L2 losses that model the mean. This setup also allows more control over non-textual information and interpolation between samples and styles.


### Recommendation

**Accept**
The idea is good and experiments are good. There are some concerns about the clarity of the paper but those can be worked on.

### Positives

1. The paper introduces a novel architecture and demonstrates improved output variation and more controllability, which is an important current issue for TTS research.
1. Many experiments investigating controllability are described, as well as some ablation studies for the model architecture in the appendix.


### Negatives

1. **Figure 1**: Please provide more informative captions. In the text, timbre and F0 are mentioned but it is not clear how that is relevant in the images. It is not clear what the colors mean in 1b, and if 1b is supposed to show a separation between male and female speakers the colors make it worse. Finally, are the points in 1b cluster centers or a single random sample per speaker?

1. **Figure 2**: Would it be possible to make it clearer that there is an autoregressive dependency in the Attention and Decoder blocks due to the LSTM cell memory? The way the figure is currently drawn makes it seem as if each attention/decoder block can be computed in parallel from only the inputs from the previous flow iteration. 

1. In section 2.2 NN and f are described as acting on the latent variable frames $z_t$, but Figure 2 applies the NN and flow on the mel spectrogram frames x in order to produce z. Similarly, there is text saying "we take the mel-spectrograms and pass them through the inverse steps of flow" and "Inference, [...], is simply a matter of sampling z values [...] and running them through the network" but Figure 2 marks the block as "Step of Flow" rather than "Inverse Step of Flow". It would be good to smooth out the consistency.

1. More discussion about the end of sequence prediction would be appreciated. As the entire sequence of z must be used for inference, I assume there is some constant max inference length z used to obtain a final x, and the end of sequence prediction only happens to the final x rather than at each step of the flow? How well does this model adapt to inference samples that are much longer than any of the inputs seen during training?

1. **3.3.2 Interpolation Between Samples**: I don't understand why there was a need to sample z to find z_h and z_s. Is it not possible to take z_h and z_s from a random training example for the speaker? Or does that mean there is a correlation between the latent space z and the __content__ that is being spoken? If the latter, I would like to see more discussion on that.

1. **Table 2**: I assume bold means closest to ground truth? It's really hard to know how to interpret this table and I don't think it supports saying that FTA Posterior is more effective. FTA did not capture the increase in pitch mean from expressive -> high pitch, nor the decrease in std from expressive -> surprised. A better visualization may be to express this table as a bar graph (3 separate groups of 3 bars each) and conclude FTA Posterior is able to produce much more variation than Tacotron 2 GST?

### Misc

#### Abstract

1. "varation" -> "variation"
1. spell out IAF
1. "We provide results on speech variation" etc. sounds weak. eg. "Flowtron produces output with far more natural variation compared with Tacotron 2 and enables interpolation over time between samples and style transfer between seen and unseen speakers in ways that are either impossible or inefficient to achieve with prior works."

#### 1 Introduction

1. "Their assumption is that variable-length embeddings are not robust to text and speaker perturbations" citation needed
1. "Flowtron learns an invertible function that maps a distribution over mel-spectrograms to a latent z-space parameterized by a spherical Gaussian." mention IAF and cite Kingma et al., 2016 here instead of the previous paragraph. 
1. "Finally, although VAEs and GANs provide a latent embedding that can be manipulated, they may be difficult to train, are limited to approximate latent variable prediction" While the IAF approach allows for direct optimization of log-likelihood, the latent variable encoding part is still approximate, just like in a VAE. The original Kingma paper even states that it is an approximate posterior. 

#### 2.2 Invertible Transformations

1. $f^{-1}$ wouldn't be applied to $z_t$. Maybe $f^{-1}(x_t)$ or $f^{-1}\left(f(z_t)\right)$.

#### 3.1. Training Setup

1. "progressively adding steps of flow on the last step of flow has learned to attend to text" on->once?

#### 3.4.2 Seen Speaker with Unseen Style

1. "Flowtron succeeds in transferring not only the somber timbre, the low F0 and the long pauses
associated with the narrative style" -> "not only the somber timbre, but also"

#### Appendix

1. IAF is known to be quite inefficient. Is there any noticeable impacts on training or inference time? Or is this not a problem due to only using 2 layers of flow? It would be great if the ablation study in the appendix regarding layers of flow also covers this, but this is quite optional.
review_title: Accept. The idea is good and experiments are good. There are some concerns about the clarity of the paper but those can be worked on.

Meta-review: This paper proposes an autoregressive flow-based network, Flowtron, for TTS with style transfer. It integrates the Tacotron architecture with the flow-based generative model.  Extensive experiments are carried out in a controlled manner and the results show that the proposed Flowtron framework can achieve comparable MOS scores to the SOTA TTS models and is good at generating speech with different styles. All reviewers consider the work interesting.  There are concerns raised on technical details which mostly have been cleared by the authors' rebuttal. The exposition also has been greatly improved based on the reviewers' suggestions and questions.  Overall, this is an interesting paper and I would recommend acceptance. 
Decision: Accept (Poster)