Bidirectional Variational Inference for Non-Autoregressive Text-to-Speech
Authors: Yoonhyung Lee
Joongbo Shin
Kyomin Jung
Venue: ICLR 2021 Poster (2021)
Keywords: text-to-speech
speech synthesis
non-autoregressive
VAE

Abstract: Although early text-to-speech (TTS) models such as Tacotron 2 have succeeded in generating human-like speech, their autoregressive architectures have several limitations: (1) They require a lot of time to generate a mel-spectrogram consisting of hundreds of steps. (2) The autoregressive speech generation shows a lack of robustness due to its error propagation property. In this paper, we propose a novel non-autoregressive TTS model called BVAE-TTS, which eliminates the architectural limitations and generates a mel-spectrogram in parallel. BVAE-TTS adopts a bidirectional-inference variational autoencoder (BVAE) that learns hierarchical latent representations using both bottom-up and top-down paths to increase its expressiveness. To apply BVAE to TTS, we design our model to utilize text information via an attention mechanism. By using attention maps that BVAE-TTS generates, we train a duration predictor so that the model uses the predicted duration of each phoneme at inference. In experiments conducted on LJSpeech dataset, we show that our model generates a mel-spectrogram 27 times faster than Tacotron 2 with similar speech quality. Furthermore, our BVAE-TTS outperforms Glow-TTS, which is one of the state-of-the-art non-autoregressive TTS models, in terms of both speech quality and inference speed while having 58% fewer parameters.

Published as a conference paper at ICLR 2021
BIDIRECTIONAL VARIATIONAL INFERENCE FOR NON- CON--AUTOREGRESSVVE TEXT-TO-SPEECH
Yoonhyung Lee, Joongbo Shin, Kyomin Jung
Department of Electrical and Computer Engineering Scoul National University
Seoul, South Korea {cpi1234, jbshin, k kjung)esnu. ac kr
ABSTRACT
Although early text-to-speech (TTS) models such as Tacotron 2 have succeeded generating human-like speech, their autoregressive architectures have several limitations: (1) They require a lot of time to generate a mel-spectrogram con- sisting of hundreds of steps.
(2) The autoregressive speech generation lacks ro- bustness due to its error propagation property. In this paper, propose a novel non-autoregressive TTS model called BVAE-TTS, which eliminates the architec- tural limitations and generates mel-spectrogram in parallel. BVAE-TTS adopts a indirectionalinferenn variational autoencoder (BVAE) that Iearns hicrarchical latent representations using both bottom-up and top-down paths increase its expressiveness. To apply BVAE to TTS, we design our model to utilize text infor- mation via attention mechanism. By using attention maps that BVAE-TTS gen- erates, we train a duration predictor so that the model uses the predicted duration of each phoneme at inference. In experiments conducted on LJSpeech dataset, we show that our model generates a mel-spectrogram 27 times faster than Tacotron 2 with similar speech quality. Furthermore, our VVAE-TTS outperforms Glow- TTS, which is one of the flate-of-hhe-a non-autoregressive TTS models, in terms of both speech quality and inference speed while having 58% fewer parameters.
INTRODUCTION
End-to-end text-to-speech (TTS) systems have recently attracted much attention, as neural TTS models began generate high-quality speech that is very similar the human voice (Sotelo al.) 2017 Wang et al. 2017; [Shen et al. 2018 Ping et al., 2018; Li et al.) 2019). Typically, those (seq2seq) model (Sutskever et al. 2014) and then synthesize speech from the mel-spectrogram using a neural vocoder like WaveGlow (Prenger et al. 2019).
TTS systems first generate mel-spectrogram from text using
Early neural TTS systems have used an autoregressive (AR) architecture to generate a mel- spectrogram mainly because of its benefits. First, the AR generation eases the difficulty of modeling mel-spectrogram distribution by factorizing the distribution into the product of homoge- neous conditional factors in sequential order. Second, the seq2seq based AR architecture helps the model predict the length of the target mel-spectrogram from an input text, which is non-trivial task because there are no pre-defined rules between the lengths of text and mel-spectrogram.
Although they facilitate high-quality speech synthesis, AR TTS models have several shortcomings. First, they cannot generate a mel-spectrogram in parallel, so the inference time increases linearly with mel-spectrogram time steps. Second, the AR-based generation suffers from accumulated pre- diction error, resulting in being vulnerable to the out-of-domain data, e.g. very long input text, or text patterns not existing in the training dataset.
In this work, we present a novel non-AR TTS model called BVAE- TTS that achieves fast and ro- bust high-quality speech synthesis. 3VA--TTS generates meesstoctoo in parallel by adopting a ceneettionaa-nffeeenee variational autoencoder (BVAE) (Sønderby et al., 2016; Kingma et al.
2016 Maaløe al. 2019 Vahdat & Kautz 2020 consisting of 1-D convolutional networks. For

---

Published as a conference paper at ICLR 2021
the high-quality speech synthesis, BVAE-TTS learns mel-spectrogram distribution jointly with hier- archical latent variables in a bidirectional manner, where BVAE uses both bottom-up and top-down paths. Furthermore, to match the length of the target mel-spectrogram at inference, B BVAE-TTS has an additional module called duration predictor, which predicts how many steps of a mel-spectrogram will be generated from each phoneme. To train the duration predictor, we employ an attention mech- anism in B VVA--TSS to make BVAE-TTS utilize the text while learning attention maps between the text and the mel-spectrogram, where the mapping information is used for duration labels.
Our B VAE-TTS has advantages over the previous non-AR TTS models as follows:
It has simpler training process compared to the previous non-AR TTS models such as ParaNet (Peng et al.] (220)) and FastSpeech (Ren et al. 2019). In the previous TTS models, well-trained AR teacher models are needed for duration labels knowledge-dissillatinn Although FastSpeech (Ren et al.] 2020) removes the dependency on the teacher model, it still requires additional duration labels and acoustic features prepared in advance using other speech analysis methods. In contrast, B EVAE-TTS requires only the text-speech paired dataset without any helps from the teacher model.
is more flexible in designing its architecture compared to the previous flow-based non-AR TTS models such as Flow-TTS (Miao et al. (020) and Glow-TTS (Kim et al., 2020). The flow-based models have architectural constraints caused by their bijective transformation property, which leads to deeper models with a lot of parameters. On the contrary, the VAE-based model is free from the architectural constraints.
In experiments, we compare our BVAE-TTS with Tacotron 2 and Glow-TTS in terms of speech quality, inference speed, and model size. The results show that our model achieves 27 times speed improvement over Tacotron 2 in generating a mel-spectrogran with similar speech quality. Further- more, BVAE-TTS outperforms the state-of-the-a art non-AR TTS model, Glow-TTS, in both speech quality and inference time, while having 58% fewer model parameters. Additionally, we analyze how the latent representations are learned by BVAB-TTS. In this analysis, we confirm that the bot- tom part of B BVAE-TTS captures the variation of mel-spectrograms that can occur from a text.
Related work: In the meantime, several TTS systems have utilized VAE relax the one-to-many mapping nature in TTS, so improve the naturalness and the controllability of the systems. For example, (Hsu et al., 2018) and (Zhang al. 2019) incorporate VAE to Tacotron 2 learn the style or prosody of the input speech. However, previous uses of VAE have been limited to an auxiliary network in TTS based on the main AR TTS model. To the best of our knowledge, our BVAE-TTS is the first parallel TTS model that directly uses the VAE architecture to the task of TTS.
More discussions about other related works on the previous non-AR TTS models are Section 5
BACKGROUND
2.1 BIDIRECTIONAL VARIATIONAL. AUTORNCODER
Variational autoencoder (VAE) is a neural network genera- tive model po (x, 2) parameterized by 0, where is ob- served data and z is a latent vector. In practice, since we only have a dataset X {x.,..., XN] without the knowl- edge about z, 0 is typically optimized by maximizing the likelihood:
x
z1
Bettom-up
Top- po-ddowm
shared
22
i2
unio
N
maxlogpe = max
log
px po(xx,2)dz. dz.
(I)
However, the integral over z is intractable to compute. Therefore, the VAE introduces an approximate posterior
Figure 1: Schematic diagram of the iidirectional-infereen variational au- toencoder. Samplings of latent vari- ables occur in the layers expressed as circles.
qo (z)x) and does variational inference while maximizing the evidence lower bound (ELBO):
lxgpe(x)= 2 Eyg (x/x) [x|z) - DKL igo (z)))))
2

---

Published as a conference paper at ICLR 2021
In practice, for easy sampling and easy computation of the KL-divergence, each of the prior p(z) and the approximate posterior qo (z|x) is usually modeled as a multivariate normal distribution with diagonal covariance matrix.
For a more expressive model, the latent vector z can be factorized into {zi...., 2k} with hicrarchical dependency, where K is the number of hierarchies. Then, each of the prior and the approximate posterior is represented as (z) = & Zg (zck) and go (2(xx) = [a Zzk, x), respectively. In (Sønderby et al. 2016; Kingma et al., 2016 Vahdat & Kautz (020), the variational inference is designed in a bidirectional way based on bottom-up path and top-down path, while letting the inference network (left) and generative network (right) share their parameters as shown in Figure T First, along the bottom-up path, B VAE extracts hierarchical features from x and stores them inside of it. Then, along the top-down path, BVAE does the variational inference and reconstructs the input data considering the stored hierarchical features together. This architecture helps the model effectively learn the hierarchies between the latent variables, and equation (2) is changed as follows:
K
logpo(x) 2 (m/x)
(m/x) [logPe (x|z)]
Egglacs (x) [D (g0(Zz z<.) (p( (3)
2.2 DURATION PREDICTOR IN NON-AUTOREGRESSII TEXT-TO-SPEECH
To achieve the non-autoregressive (non-AR) text-to-speech (TTS) model, the model needs to predict the length of the target mel-speetrogram from text. This is because there is no way to access to the length of the target mel-spectrogram inference. However, this is challenging task considering that there are no pre-defined rules between the lengths of text and mel-spectrogram. Recently, several non-AR TTS models (Ren et al. 2019 Zeng et al., 22200] Kim et al. (0200) resolved the issue by introducing a module called duration predictor. The duration predictor is a module that predicts how many mel-spectrogram steps will be generated from each phoneme.
First, using the duration predictor, the non-AR TTS models compute durations D {di, ..., ds corresponding to each phoneme based on phoneme representations H = {hi...., hs). where each d, is positive integer that is rounded off from a positive real number, and S is the number of phonemes. Then, H is expanded to the length of the target mel-spectrogram T, by repeating each h, as many steps as d, Finally, the non-AR TTS models generate a mel-spectrogram parallel by decoding the expanded phoneme representations.
In practice, since there are no ground-tu truth duration labels for the training of the duration predictor, the non-AR models obtain the duration labels using various methods, and we adopt method used in FastSpeech (Ren et al. 2019). From well-aligned attention maps, the duration labels are obtained according to d; = E Jargmax, a,,t = i]. where a,,t represents an attention weight given from the 1-th mel-spectrogram step to the s-th phoneme.
3 METHODOLOGY
In this section, we explain a novel non-autoregressive (non-AR) TTS model, BVAE-TTS, which based on the bidirectional-inferend variational autoencoder (BVAE). As shown in Figure 2 (a), during training, B VAE-TTS is given a mel-spectrogram with phoneme sequence, and it is tramed to reconstruct the mel-spectrogram while maximizing the ELBO. Here, the duration predictor is jointly trained using the attention maps that BVAE-TTS generates during training. As shown in Figure (c), at inference BVAE-TTS generates a mel-spectrogram from a phoneme sequence using the duration predictor as described in Section 2.2 while using its top-down path for decoding the expanded phoneme representations. In Appendix pseudo-codes for the training and inference of B VVA--TTS contained for detailed descriptions. The other aspects of BVAE-TTS are described in the following sub-sections in more detail.
3.1
USING BVAE FOR TEXT-TO-SPEECH
Unlike the previous BVAE models (Sønderby et al. 2016 Kingma et al. 2016 Vahdat & Kautz trained generate natural images, our model should learn generate mel-spectrogram
2020)
3

---

Published as a conference paper at ICLR 2021
Phonemes
Phonemes
loga
Duration
Duration
Encoder
Encoder
MSE
S0
Vesp
Vexp
Get. (AM A
attention
logd
PC
dim
dim &
Black
COIV
cooV
BVAE Layer
BVAE Layer
Layer
comv
ViLA
Layer
BVAE Layer
BVAE Layer
Downsample
Upsample
conv
Upsample
CoIN
ComV
Pre-net
Projection
Projection
(Apta Alx,)
(a) Training
(b) BVAE Layer
(c) Inference
Figure 2: (a) The training procedure of BVA--TTS. The scissors represent to detach the signal from the computational graph to block the gradient signal in backpropagation. The Downsample and Upsample layers are included in even-numbered B VAE blocks. V Vexp represents the expanded V fit the length of the top-down path input. The f()) represents the Straight-Through argmax with jitter, and V C represents the gradient signal. (b) B BVAE Layer. The dotted arrow represents sampling. The red lines indicate the parameters of prior and approximate posterior normal distributions. (c)
The inference procedure of BVAE-TTS that uses the top-down path only.
that is not only natural but also corresponding to the input text. To this end, we add a dot-product attention network (Bahdanau al. 2015) top of the BVAE, which is channel for BAA-TTTS to learn how to utilize the text properly. First, using a text encoder, key (K) and value (V) are obtained from a phoneme sequence, and from the bottom-up path, query (Q) is obtained. Here, obtaining Q is different from the bottom-up paths of the previous BVAE studies used in the image domain, where only the parameters for posterior approximation are obtained. Second, based the dot-product attention with Q. K, and V, the V are expanded to Vexp to fit the length of the top-down path, and then the V Vexp is inputted into the top-down path of BVA--TTS. Lastly, the BVAE-TTS does both the variational inference and mel-spectrogram reconstruction along the top-down path using the expanded text representations with the following objectives:
Lrecon -Ego (ogpo(x|z,))
(4)
E (xcleeexx))) (z Zick LKL Σ
where x represents mel-spectrogram, represents text, represents latent representation, and mean absolute error (MAE) loss is used for the Lrecon.
In addition to that, a duration predictor is jointly trained to predict durations corresponding to each phoneme in the logarithmic domain using mean square error (MSE) loss, Laur = E[(lo d; log 2. where d and d; are obtained as described Section 2.2 The duration predictor takes as input the V obtained from the text encoder, and here the V is detached from the computational graph
to prevent it from affecting the BVAE training.
3.2 ARCHITECTURE OF BVAE-TTS
In this section, we describe the architecture of B BVAE-TTS that hierarchically learns the latent rep- resentations based BVAE blocks consisting of BVAE layers.
4

---

Published as a conference paper at ICLR 2021
BVAE block: As shown in Figure (a). the main part of BVAE-TTS is the stacked BVAE blocks, each consisting of BVAE layers. To guide the multi-scale tin-to-coars latent features to be con- tained in the latent hierarchies, the time dimension of imput mel-spectrogram is downsampled using bilinear downsampling operations (Zhang 2019) in even-numbered B VAE blocks along the bottom- up path. Here, the numbering of B blocks starts with one and increases from bottom top. On the contrary. its time dimension is upsampled again along the top-down path, by repeating the sig- nals in the BVAE blocks where the downsamplings have occurred. In odd-numbered BVAE-blocks, the channel dimension is decreased along the bottom-up path and is increased along the top-down path. It done the pre- or post-convolutional network of the first BVAE layer in the BVAE blocks shown in Figure (b).
The main element of the BVAE block is the BVAE layer. As shown in Figure (b), at each but and top-down path, the parameters of the prior and the approximate posterior distributions {kg; Σx); ) {Ayu,, AΣ ), {Apkaa AΣx;) } are obtained from 1-D convolutional networks. Then, the prior distribution pz(za (ZZR) and the approximate posterior distribution qs (z Zeks x, y) are defined as follow:
BVAE layer:
Po Po(2i|ReR.N) == NNWL Σ&),
Zck, = N(p ApE, + Apt2: Σg. AEE, A
(7)
where diagonal covariance matrices Σ are used after applying softplus function to guarantee that they are positive. This parameterization follows (Vahdat & Kautz, (020), where the approximate posterior (o(Za|Acc y is relative to the prior Po [2x (0(2//zeRR y). With this parameterization, when the prior moves, the approximate posterior moves accordingly while making the BVAE training eas- ier and more stable. During training, the latent representation Zę is sampled from (ZA Z<ck, x),, and sampled from ps (z& &<cky y) at inference. Other details on the BVAE-TTS architecture such as text encoder duration predictor are in Appendix A.2
3.3 BRIDGE THE GAP BETWEEN TRAINING AND INFERENCE
When BVAE-TTS reconstructs a spectreccto during training, text representations are expanded via the attention network. In contrast, text representations are expanded via the duration predictor at inference. Therefore, to bridge the gap between the attention -based mel-spectrogram generation and the duration-based mel-speetrogram generation, use the following techniques in this work.
Frraight-Througee argmax: In the duration- based generation, the predicted duration of each phoneme is used after it is rounded the nearest integer. means that there is corresponding phoneme for every time step of a mel-spectrogram. Therefore, during training, we use a trick called Sraight-TTrouuee (ST) argmax, where the phoneme representation given the largest attention weight from each query time step, which is computed using arg max operation, is passed to the top-down path instead of the weighted sum in the attention mechanism. However, during backpropagation, the parameter update is conducted as if the signal was result of the weighted sum.
Jitter: To make the model more robust to the errors of the duration predictor, we apply jitter to the text representations, where each of the text representations obtained from the ST-argmax is replaced with a text representation attended by one of the neighboring queries with each probability of 25% during training. We also experimentally observe that applying jitter makes the learning of the attention maps more stable, so the attention maps are not defused throughout the training and stay diagonal.
Positional encoding biasing & Guided attention: In order to reduce the gap between the attention- based generation and the duration-based generation, it is important for the learned attention maps to have diagonal shapes. Therefore, we use two additional techniques to directly help B BVA--TT learn the diagonal attention maps. First, we add positional encoding vectors with different angular speeds to query and key as an inductive bias following (Ping et al., 2018). Second, we use an additional guided attention loss Cyuide that gives penalties for attention weights deviating from the diagonal following (Tachibana et al. 2018). For more details on the techniques in this section, see Appendix
With the above techniques, BVAE-TTS is trained with the following objective:
Liotal Lrecon + a LKL Ldur + Lguide,
(8)
5

---

Published as a conference paper at ICLR 2021
where the is warm-up constant that linearly increases from 0 to over the first 20% of training. This technique is proposed in (Sønderby et al. 2016) to weaken the variational regularization the early stages of training.
EXPERIMENTS
In this section, describe the experimental setup and the results obtained from the quantitative and qualitative experiments that are conducted to evaluate our BVAE-TTS") For comparison, we use two state-of-the-ar TTS models: Tacotron for AR TTS model and Glow-TTS non-AR TTS model. Here, we use the pre-trained weights of the models that are publicly available.
4.1 EXPERIMENTAL SETUP
In the experiments, we mainly use the LJSpeech dataset (Ito & Johnson 2017) consisting of 12500 / 100 / 500 samples for training / validation / test splits, respectively. For speech data, convert raw waveforms into -sgeee--peectoogras with 1024 window length and 256 hop length and use them target sequences of our VAE-TTTS model. For text data, convert raw texts into phoneme sequences using grapheme-to to---to-ppphee to-phoneme library (Park 2019) and use them as input sequences of BVAE-TTS.
We train the BVAE-TTS consisting of 4 BVAE blocks for 300K iterations with a batch size of 128. For optimizer, using the learning rate scheduling used (Vaswani et al. [2017] where initial learning rate of le- 3 and warm-up step of 4000 are used. Training of BVAE-TTS takes about 48 hours Intel(R) Xeon(R) Gold 5120 CPU (2.2GHz) and NVIDIA V100 GPU on the Pytorch 1.16.0 library with Python 3.6. 10 over the Ubuntu 16.04 LTS. For more details on the hyperparameters, see Appendix A.4
the Adamax Optimizer (Kingma & Ba 2015) with 81 0.9, /3 0.999
4.2
EXPERIMENTAL RESULTS
In this section, we compare BVAE-TTS with Tacotron and Glow-TT TTS terms of speech quality, inference time, and model size. For the quality evaluation, vocoder that converts mel-spectrograms to waveforms. When we sample latent representations in the temperature of 0.333 for the models for better speech quality.
pre-trained WaveGlow El
Glow-TTS and BVAE-TTS, (Kngmma & Dhariwal 2018
Table 1: Experimental results. The MOS-ID and MOS-OOD are written with 95% confidence intervals. The number in parentheses represents the number of parameters of BVAE-TTS that used at inference.
Method
MOS-ID
MOS-OOD
Inference Time (ms)
# of Parameters
GT Audio
4.68 + 0.06
GT Mel-spectrogram
4.41 + 0.07
Tacotron 2
4.35 + 0.07
4.16 + 0.07
658.5
28.200 28.2M
Glow-TTS
3.96 + 0.08
3.89 + 0.10
43.07
28.6M
BVAE-TTS
4.14 = 0.07
4.21 + 0.07
24.20
16.0M (12.0M)
Speech quality: In this experiment, we measure the Slaan-Opinion-Scom (MOS) for audio samples generated by each TTS model using fifty sentences randomly sampled from the in-domain LJSpeech test set (MOS-ID). In addition, measure another MOS on fifty sentences randomly sampled from the test-clean set of LibriTTS (Zen al. 2019) to see the generalization ability on the out-of-domain

---

Published as a conference paper at ICLR 2021
text data (MOS-OOD) Via Amazon Mechanical Turk (AMT), assign five testers living in the United States to each audio sample, and ask them to listen to the audio sample and give a score between 1 to in 9-scale based on its naturalness.
The MOS results shown in Table T demonstrate the superiority our BVAE-TTS, where it out- performs the state-of-the-art non-AR TTS model, Glow-TTS, both MOS-ID and MOS-OOD. Although BVAE-TTS does not surpass Tacotron 2 in MOS-ID, our model achieves better results on MOS-OOD. shows its robustness the out-of-domain text over the autoregressive TTS model which suffers from the accumulated prediction error. For a better understanding of the speech qual- ity generated by the models, we strongly encourage the readers to listen to the audio samples in the supplementary materia on the demo page f0
Inference time: We measure inference times taken to generate a mel-spectrogram from text on the 500 sentences of LJSpeech test set in GPU environment. The average inference time of each TTS model is shown in [ As can be seen in the table, our BVAE-TTS shows 27.2 times faster inference speed average compared with Tacotron 2, and it is also 1.78 times faster than Glow- TTS. Besides, due to the sequential generation property of AR TTS model, the gap between the inference speed of VAB-TTTS and Tacotron 2 increases with longer length of an input text. See Appendix B for more details.
Model size: As can be seen in the last column of Table I BVAE-TTS has the smallest number of parameters of 16.0M while maintaining high-quality speech synthesis. Furthermore, BVAE-TTS gets smaller (to 12.0M) inference because the layers belonging to the bottom-up path are not used to generate a mel-spectrogram, where the number of parameters is 58% fewer parameters compared Glow-TTS. This shows that the training principle of BVAE-TTS that hierarchically learns the latent features while adjusting hidden dimensions enables our model to have small parameters. is contrary the flow-based TTS models such as Flow-TTS (Miao al. 2020) and Glow-TTS (Kim et al. 2020), where many parameters are used due to its architectural constraint.
4.3 MODEL ANALYSIS
As our BVAE-TTS the first VAE-based parallel TTS model, we conduct several analyses on it. First, analyze BVAE-TTS to see how the hierarchies contained in the latent representations and how the variance in mel-spectrograms learned. Then, we verify the effectiveness of the techniques used in BVAE-TTS such Straight-Through (ST) argmax and jitter through ablation studies.
4.3.1 ANALYSIS ON HIERARCHY
this experiment, we conduct an analysis on the hierarchical latent representation learn- ing of BVAE-TTS. To see how the latent fea- tures of the mel-spectrograms the hierarchies, we observe the variations of the mel-spectrograms sampled from the same text, while using different temperatures for dif- ferent BVAE blocks. Specifically, we set tar- get BVAE block among the four BVAE blocks and increase the variance of the target BVAE block, by using temperature of 1.0 or 2.0 or 5.0 for the samplings occured the BVAE lay- ers belonging to the target BVAE block. On the contrary, we lower the variance of the non- target BVAE blocks using temperature of 0.333. Then, we sample 100 different mel- spectrograms each from the same text, while varying the target B BAA block and tempera- ture.
temperature: 1.0
temperature:
learned in
tomperature: 5.0
Standard deviation on
1.0-
0.6
0.2
Block1
Block2
Block4
Target Block
Figure 3: Averages of pixel-by-pixel standard de- viations measured on randomly sampled 100 mel- spectrograms.

---

Published as a conference paper at ICLR 2021
Figure shows the averages of sxxel-by-pixel standard deviations measured the randomly sam- pled 100 mel-spectrograms from the same text. The block numbers in the figure are given from one observe that the variance
to four starting from the BVE block at the bottom. In this experiment, of the speech is mostly contained in the latent representations of B VAE blocks 1, 2, which are close the mel-spectrogram. However, there is not much variance in the generated mel-spectrograms when we increase the temperature of BVE blocks 3, 4, which are close to the text representations. Therefore, we can conclude that the global content is mostly contained in the expanded text rep- resentations obtained using the text encoder and the duration predictor, and the BVAE blocks 3, focus on building the content rather than its style. Note that while Figure standard deviations measured using one examplar sentence, "One, Two, Three.", the tendency is consistent regardless of the input sentence. Mel-spectrogram samples obtained in this experiment are Appendix C
4.3.2 ABLATION STUDY
Training
Inference
(a) BVAE-TTS
(b) No jitter
(c) No ST-argmax
Figure 4: Examples ablations. Each row represents (1) learned attention map, (2) reconstructed mel-spectrogram, (3) diagonally stacked predieted durations, (4) generated mel-spectrogram from a text. (1), (2) are obtained during training, and (3), are obtained at inference. The red lines represent the section of the durations corresponding "comma' and 'whitespace" in text.
We conduct ablation studies to see the effects of applying jitter and Straight- Thrguuhh (ST) argmax to the soft attention mechanism in BVAE-TTS, and the results shown in Figure Here, since jitter is included in ST-argmax (jitter is applied to the output of the arg max), the ablation study of not using ST-argmax represents when BVAE-TTS is trained using a normal soft attention mechanism.
The most noticeable differences appear in the attention maps that they learn. As shown in the first row of Figure (a), (b), applying jitter shows the effectiveness for BVAE-TTS to learn well-aligned attention maps. It results in using more accurate duration labels to train the duration predictor, which leads more natural speech. We observe that BVAE-TTS without applying jitter still generates clear speech even though it is little unnatural, where it obtains 3.68 MOS on the I.SSeeech dataset. shown in bottom mel-spectrogram of Figure (c), the BVAE-TTS without ST- argmax technique just generates stuttering sound.
As shown Figure (a), although the BVAE-TTS also does not learn the perfect attention map, BVAE-TTS successfully generates mel-spectrogram inference. Since the text is forced to be used monotonically the dtrationn-basee generation, it makes the model more robust to the attention errors while making fewer pronouncing mistakes. In addition, when using the duration predictor, is also possible to locally control the speed of speech by adjusting the predicted durations. The experiment the speed control is included in Appendix D
8

---

Published as a conference paper at ICLR 2021
5 DISCUSSION
To overcome the limitations that the autoregressive (AR) TTS models have, various non-AR archi- tectures have been recently proposed. On one hand, there have been feed-forward neural networks such as ParaNet (Peng et al. 2020), and FastSpeech 1, 2 (Ren et al. 2019; 2020). which use knowl- edge distillation or additional duration labels and acoustic features. Although they succeeded in enabling their models to predict the lengths of the target mel-spectrograms, the feed-forwar archi- tectures did not fit the one-to-many mapping problems in TTS. Therefore, FastSpeech (Ren et al.) 2019) uses targets mel-sppetroggams that the AR teacher model generates. This is because much of the diversity in original mel-spectrograms has been eliminated in the -peettreess generated by the AR teacher model. Besides, FastSpeech 2 (Ren et al.) 2020)) even directly uses additional pre-obtained acoustic features such as pitch and energy to relax the one-to-many mapping nature in TTS. In contrast to the models, it can be seen that BVAE-TTS is asked to solve one-to-one map- ping problems during training because there is only one possible target for the reconstruction task. As result, BVAE-TTS can generate natural and diverse samples while learning latent features in mel-spectrograms..
On the other hand, there have been generative flow-based non-AR TTS models such Flow-TTS (Miao et al. 2020) and Glow-TTS (Kim et al. 2020). While their speech quality is comparable to that of AR TTS models, flow-based generative models usually have a problem that they require lot of model parameters. In the models. the dimensions of the hidden representations in the flow networks should be the same through the whole network, and their bipartite flow requires many layers and larger hidden size because of its lack of expressiveness (Ping et al., 2019 Lee et al.) 2020). Contrary to flow-based TTS models, our BVAE-TTS is free from the above issue. this work, by designing BVAE-TTS in hierarchical architecture with varying hidden dimensions, we can outperform the flow-based TTS model, Glow-TTS in both speech quality and speed, while having a much smaller model size.
CONCLUSION
In this work, we propose BVAE-TTS, which is the first VAE-based non-AR TTS model that gen- erates a mel-spectrogram from a text in parallel. To use the BVAE architecture in text-to-speech, combine BVAE with attention mechanism to utilize a text, and to extract duration labels for the training of the duration predictor. In experiments. BVAE-TTS achieves to generate speech 27x faster than Tacotron with similar speech quality, and also outperforms Glow-TTS in terms of both speech quality and inference time with 58% fewer parameters. Since our VAE-based TTS model shows competitive performance and has many advantages over the previous non-AR TTS models, we hope it becomes good starting point for future VAE-based TTS research.
ACKNOWLEDGMENTS
K. Jung is with ASRI, Scoul National University, Korea. This work was supported by the Ministry of Trade, Industry & Energy (MOTIE, Korea) under the Industrial Technology Innovation Program (No. 10073144). This research was results of study on the "HPC Support" Project, supported by the 'Ministry of Science and ICT' and NIPA.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxix or.. abs 409.00.00 0473
Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, et al. Hierarchical generative modeling for controllable speech synthesis. In International Conference on Learning Representations, 2018.
Keith Ito and Linda Johnson.
The lj speech dataset.
bttps://keithito: com/
L.J- S-3-Specch-Dataseet 2017.

---

Published as a conference paper at ICLR 2021
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005. 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR (Poster),
2015. http::t:://axtt org/ abs / 1412. 6980
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in neural information processing systems, pp- 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ily Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743 4443-4751,20 2016.
Sang-gil Lee, Sungwon Kim, and Sungroh Yoon. Nanoflow: Scalable normalizing flows with sub- linear parameter complexity, arXiv preprint arXiv:2006.062: 2020.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with trans- former network. In Proceedings the AAAI Conference on Artificial Intelligence, volume 33, pp- 6706-6713. 2019.
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. In Advances neural information processing systems, pp. 6551-6562, 2019.
Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flow- tts: A non-autoregressive network for text to speech based on flow. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7209-7213.
IEEE, 2020.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Jongseok Park, Kyubyong & Kim. g2pe.
2019.
Kainan Peng. Wei Ping, Zhao Song, and Kexin Zhao. Non-autoregressive neural text-to-speech. In Proceedings of the 37th International Conference on Machine Leurning, pp. 10192-10204. PMLR, 2020.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 1000-speaker neural text-to-speech. Proc. ICLR, pp. 214-217, 2018.
Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. arXiv preprint arXiv: 1912.01219 2019.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing ( (ICASSP), pp. 3617-3621. IEEE, 2019.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Sys- tems, pp. 3171-3180. 2019.
Yi Ren, Chenxu Hu, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Ya Liu. Fastspeech 2: Fast and high-quality end-to-end text-to-speech. arXiv preprint arXiv:2006.045: 2020.
Jonathan Shen, Ruoming Pang. Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. 20/8 FEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779 IEEE, 2018.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-3746, 2016.
10

---

Published as a conference paper at ICLR 2021
Jose Sotelo, Soroush Mehri, Kundan Kumar, João Felipe Santos, Kyle Kastner, Aaron C. Courville, and Yoshua Bengio. Char2wav: End-to-end speech synthesis. In 5th International Confer- Learning Representations. ICLR 2017. Toulon, France, April 24-26. 2017, Workshop Track Proceedings. OpenReview.net,201. 2017. URL htt.htttss///ppneeeeee eww.ewwwwe.... net. forum? farrmm?co i=
B B1VWyySKX
Iyy Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara. Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4784-4785 IEEE, 2018.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. arXiv preprint arXiv:2007.03898, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 6008, 2017.
Yuxuan Wang, R.J. Skerry- Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgian- nakis, Rob Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In Proc. Interspeech 2017, pp. 4010, 2017. doi: 00214437/meerppechh200---- 2tt//100.111177 21437 hnt.eeeeeeeetttteeeeeeeec.. 2017
URL
http://dxx doi
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-t-Speechh. In Proc. Interspeech 2019, pp. 1526-1530, 2019. doi: 10.21457//meerrpeech..00 2019-244.11921 URL http:://dx doi. .ooiioorr org/
10. 10.21437/ wtte1//////tterppecchhhh. 2019 2441
Zhen Zeng, Jianzong Wang, Ning Cheng, Tian Xia, and Jing Xiao. Aligntts: Efficient feed-forward text-to-speech system without explicit alignment. In ICASSP 2020-2020 IEEE International Con- ference Acoustics, Speech and Signal Processing (ICASSP), pp- 6714 6718. IEEE, 2020.
Richard Zhang. Making convolutional networks shift-invariant again. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7324-7334 PMLR, 2019. UR//sttp.//prcced.ngg ml press /ttess/////// v77 zhang 9a. html
Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen-Hua Ling. Learning latent representations for style control and transfer end-to-end speech synthesis. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6945-6949. IEEE, 2019.

---

Published as a conference paper at ICLR 2021
A DETAILS ON BVAE-TTS
A.1 ALGORITHMS
Algorithm 1: Pseudo-code of BVAE-TTT TTS training
Data:
X, y: a mel-spectrogram, a phoneme sequence
Q, K, V. V Vexp: query, key, value, and expanded value matrices
h, heep: hidden representations, expanded hidden representations
BVAE-TTS I]: an 1-th BVAE layer in b-th BVAE block.
P Pauery: P positional encoding vectors for query and key
A: soft attention map obtained before applying the ST-argmax technique
D: phoneme durations extracted from the attention map
D: phoneme durations predicted by the duration predictor
a: a warm-up constant
Result:
x: a reconstructed mel-spectrogram
Liotal, Crecon, LKL, Cour, Guudde: total loss, and losses that make up the total loss
K, V + Text Encoder(y)
h P PreNet
for b 0 B do
if' 6% = then
h Downsample (h);
end for
< 0 L do h. A AΣE, < BVAE-TTS(6, [ Botttomppp ; BVAE-TTSI b,, Ape, + A
BVAE-TTS b, A Softplus(ASi,
end
end
Q IncresseDimenssoon (h);
V exp: A + Attention (2 PPP K - Pkkey V):
heap DeecreaseDimension (V exp);
CKL + 0; to do
for b b+ B
for f- L to do
heap: klloss +- BVAE-TTS|b, Toppoon (heap);
LKL + LKL + kl loss
end
1%2 = then
heep Upsample (hezp);
end
end
x Projection(tenn):
D + ExtractDurations (A);
D DurationPrecicttor (V);
Lrecon + MXE(X x):
Ldur + MSE (log( D). log D)):
Lguide + Gui dedAttenttteen onLoss (A);
Liotal &- Lrecon t LKL + Ldur + Liuides
12

---

Published as a conference paper at ICLR 2021
Algorithm 2: Pseudo-code of BVAE-TTS inference
Data:
X, y: a mel-spectrogram, a phoneme sequence
K, V, V exp' key, value, and expanded value matrices
hezp: expanded hidden representations
AVAE-TS// I]: an 1-th VAE layer in b-th BVAE block.
D: phoneme durations predicted by the duration predictor
Result:
x: a reconstructed mel-spectrogram
K, V TextEncoder(y)):
Đ + DuratiionPredictor (V):
V exp ExpandBeepresentttiooss...V hezp - Deeereasseiieessin (Vep));
for b b- B - to 0 do
for f
to do heap:- BVAE-TTS(6, TopDown (heap);
end
if 6%62 = then Meap + Upsample (heap);
end
end
x + Projection(teep::
A.2 ARCHITECTURE DETAILS
Text encoder: To obtain key (K) and value (V) used in the attention mechanism, we use a text encoder used in Deep 3 (Ping al. 2018), which consists of series of 1-D convolutional network with gated linear unit and residual connection; y v0.5 ( convi (ar) a (conv2(1)))- The last hidden representations of the text encoder are used as K, and is computed according to V= (O.((K + E), where represents phoneme embedding vectors.
Duration predictor: We duration predictor used in FastSpeech (Ren al. 2019), which consists of two Conv Le---LL-LLLyeeeee LaLLLLLLLLeNNoNo--Doooee Then, take exponential of the output values and add ones to them guarantee that the predicted durations are larger than one.
blocks, followed by a linear projection layer.
Pre-net & Projection layer & Dimension matching layers: To adjust dimensions properly, there are four additional layers; Pre-net, Projection layer, and two additional dimension matching layers. Pre-net consists of two Conv Id-Dropout ELU blocks, and Projection layer consists of two Conv Id- Saaaeeeeoom--Dppooo blocks followed by Conv ld layer with sigmoid activation. For the dimension matching layers, we use Conv 1d layer with kernel width increase the dimension, and use linear layer to decrease the dimension.
To increase the receptive field of the model, the pre- post- convolutional networks use dilated convolution with the dilation factors 1, 2, 4 for each BVAE layer in each B BVAE block. Every convolutional network except for the networks that output the parameters of the prior and approximate posterior distributions are followed by an ELU layer. When the residual path is merged with the original path, we combine the signals according to y v0.5(x f(x:) ) to stabilize training. Following (Vahdat & Kautz] 2020). we apply the spectral normalization (Miyato et al. 2018) to the convolutional networks located the residual path to stabilize training.
BVAE block layer:
A.3 ADDITIONAL TECHNIQUES
Straight- Thrguuh argmax: Let A {a,,}} be the attention matrix obtained from the dot dro-put operation in the attention mechanism. Then, the original attention mechanism expands the value V
13

---

Published as a conference paper at ICLR 2021
to Vex Vexp according Vexp AT V. However, with the ST-argmax, another one-hot attention matrix Aonehot {a',t) firstly obtained, where for and t, a',, if as = (ax,(ar,) else 0. Then V exp obtained as follow:
Vexp = (A + (Aonhhot A dettacch)))
where by using the detach function, we let the gradient flow to A instead of Aonchot during back- propagation.
Jitter: To apply jitter to the hidden states, we apply jitter to Aonehot obtained from ST-argmax, of which example shown in Figure By using function 'random.choice' that returns an element chosen from a list according to probabilities applying jitter to Aonehot can be implemented according a',t random.choices((((..- a' d' l p [0.25, 0.5, 0.25)).
20-
15
10
30
10
50
Figure 5: An example of applying jitter to ideal diagonal attention matrix Aonehat.
Positional encoding biasing: We add positional encoding vectors with different angular speed w, to query Q and key K to introduce inductive bias of diagonal attention:
Ppopso.21) sin W pos
(10)
10000 W2 * pos
(popo.21t11 cos(
(11)
1000024/2
where pos is the position, i is the dimension, and d is the dimension of Q and K.
For Q, w. = 1, and for K, w, = T/S, where T and S are the numbers of time steps of Q and K.
Then, the attention matrix A is obtained according to A (K + PE (Q P.query)
Guided attention: directly lead the attention map to be formed diagonal, we additionally
introduce Lguide follow:
Lyuide Bla,,t (1 (/1))///////////'
where g is set to 0.2.
14

---

Published as a conference paper at ICLR 2021
A.4 HYPER PARAMETERS
Table 2: Hyperparameters of BVAE-TIL ATAS.
Hyperparameter
BVAE-TTS
Phoneme Embedding Dimension
256
Text Encoder Layers
7
Text Encoder Hidden Dimension
256
Text Encoder Conv ID Kernel Width
5
Text Encoder Conv II) Filter Size
256
Text Encoder Dropout
0.1
Pre-net Layers
2
Pre-net Dropout
0.5
Pre-net Hidden Dimension
256
Downsampling Conv ID kernel
[0.25, 0.5, 0.25]
Projection Layers
Projection Dropout
0.5
Projection ConvID Kernel Width
5
Projection ConviD Filter Size
256
Duration Predictor Conv ID Kernel Width
3
Duration Predictor Conv ID Filter Size
256
Duration Predictor Dropout
0.1
BVAE Blocks
4
BVAE Layers per block
3
BVAE ConviD Kernel Width
Hidden Dimensions of BVAE blocks
128, 128, 64, 64
Total Number of Parameters
16.0M (12.0M)
B PARALLEL SYNTHESIS
Tacotron2
12
GLOW-TTS
BVAE-TTS
6.8
6.6
644
0.2
100
The number of characters
Figure 6: Inference time measured on 500 sentences of I.Sppeech test set.
To the benefit of parallel mel-spectrogram synthesis, 2, Glow-TTS, and BVAE-TTS on the 500 sentences of LJSpeecc test set in GPU environment. Figure 6 shows that the inference times of the non-AR TTS models almost constant even if the length of the input text gets longer. On the contrary, the inference time of Tacotron2 linearly increases.
the inference time of Tacotron
15

---

Published as a conference paper at ICLR 2021
C ANALYSIS ON HIERARCHY
While changing the target BVAE block and its temperature as described in 4.3.1 we observe the generated mel-speetrogram samples. As shown in Figure 7 the variance of the mel-spectrograms is clear when we increase the temperature of the two bottom VAE blocks. On the contrary, the mel- spectrograms are almost the same when we increase the temperature of the top BVAE blocks. Especially, when we set the temperature of 'BVAE block 2 to 5.0, the mel-spectrograms are the most diverse with good speech quality.
Temperature: 1.0
block 1
block 2
block 3
block 4
Temperature: 2.0
block
block 2
block 3
block 4
Temperature: 5.0
block 1
block 2
block 3
block 4
Figure 7: Mel-spoctrossams generated from the same text, "One, Two, Three.".
16

---

Published as a conference paper at ICLR 2021
Temperature: 1.0
block 1
block 2
block 3
block 4
Temperature: 2.0
block 1
block 2
block 3
block 4
Temperature: 5.0
block 1
block 2
block 3
block 4
Figure Mel-spectrograms generated from the same text, "Hello, my friends.".
17

---

Published as a conference paper at ICLR 2021
Temperature: 1.0
block 1
block 2
block 3
block 4
Temperature: 2.0
block 1
block 2
block 3
block 4
Temperature: 5.0
block 1
block 2
block 3
block 4
Figure 9: Mel-spectrograms generated from the same text, "Trick or treat!".
18

---

Published as a conference paper at ICLR 2021
D SPEED CONTROL
60
0
20
40
60
80
100
40
20
60
20
0
20
40
60
a0
180
Mel-spectrogram steps
Figure 10: Speed controlled mel-spectrograms generated from the same text "One, Two, Three."
While AR TTS models generally suffer from the lack of controllability, BVAE-TTS control the fine-grained speed of speech by multiplying positive constant to the durations predicted by the duration predictor. Figure 10 shows three mel-spectrograms produced by BVAE-TTS using the same sentence, "One, Two, Three.". While changing the target word from "One" to "Three", we multiply 2.0 to the durations of phonemes belonging to the target word, and multiply 0.7 to the durations of phonemes belonging to the non-target words. In this experiment, we observe that BVAE-TTS successfully generates speech while varying the pronouncing speed of each word in single sentence. Interestingly, we observe that our model intensifies the pronunciation of the target word, showing the capability of appropriately adjusting the prosody according to the speed.
19


Reviews:
[{"review_id": "pkC866oRd-8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: Neural models that autoregressively generate mel spectrograms from text (or phonemes), such as Tacotron, have been used to generate high quality synthetic speech. However, they suffer from slow inference speed due to their autoregressive nature. To alleviate this, non-autoregressive models have been proposed, such as FastSpeech and Glow-TTS. The proposed model, BVAE-TTS, is yet another non-autoregressive speech synthesis model (outputting spectrograms), with two key advantages over the aforementioned models: (a) no autoregressive teacher model is required, as in FastSpeech, which simplifies training, and (b) fewer parameters are needed than in Glow-TTS, since there is no bijectivity constraint (allowing a more expressive architecture to be used). Models are compared with inference speed and MOS, and BVAE-TTS compares favorably on both both metrics when compared to Glow-TTS.\n\nPros:\n\n1. The evaluation of the model is done well, in a clear way. LJSpeech is used, a dataset which is commonly used and easily accessible. MOS and inference speech are provided, and error bars are provided for MOS values. BVAE-TTS is compared to Glow-TTS and Tacotron 2 (one other non-autoregressive model, and one well-known AR baseline), and hyperparameters are provided. A single vocoder (pretrained WaveGlow) is used on all models, isolating the effect of the spectrogram prediction model used.\n\n2. Section 4.3, pertaining to using attention distributions to learn a duration predictor, is interesting and novel. Using positional encodings is standard and using a loss guide is unsurprising. However, while jitter and straight-through estimators are not uncommon, all of these things together make a compelling and novel approach to using attention to infer discretized durations and compensate for that train-test mismatch well. I believe that a similar technique could be used in other models as well.\n\n3. The model is an application of similar ideas from image synthesis, which is interesting, in that it demonstrates that some of those techniques work equally well for spectrogram synthesis. This sort of cross-modal result points to the strength of the method being used, which is a valuable data point for the research community.\n\nCons:\n\n1. The biggest weakness of this paper, in my view, is that deciphering the model itself is quite difficult. Although the model bears resemblance to NVAE (for which code is released), understanding the fine details is tricky, and the paper does little to aid in that effort. \n\nIn particular, understanding the exact layer inputs and outputs and parameters of the normal distributions being used is difficult, and I believe the paper would benefit significantly from a pseudocode explanation of the network. For example, I did not understand why the generative model produced both $\\mu_l$ and $\\Delta \\mu_l$, and whether $\\mu_l$ was predicted with a dense layer or was the accumulation of the prior BVAE stacks' $\\Delta \\mu_l$ values (and similar for $\\Sigma$). \n\nI also wonder why the output of the attention layer is not provided to the encoder; perhaps there is a fundamental reason for this which I am missing, or perhaps this is simply an architecture choice.\n\nA very clear explanation of the method itself, perhaps as psuedocode for where the means and variances come from and which features they interact with and what it sampled when, would in my view make this among the top papers.\n\nRecommendation:  Accept. The paper is well written and results are strong, although I would prefer if the method itself were explained more clearly.", "extra_scores": {}, "extra_text": {}}, {"review_id": "K-8KCHnVmK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post rebuttal and discussion\n========================\nSeveral reviewers have pointed out that the paper needs more comparisons/ablations with existing models (e.g. Paranet/Fastnet). To this end, I think we at least need a comparison with Paranet, which is a 'comparable' non-autoregressive CNN based VAE based model with a few other components such as attention distillation. \n\nThere are components in the paper that could do with more ablation studies \n- argmax with straight through estimator\n- some guidelines on BVAE blocks and tuning\n\nIn light of these points, together with the fact that we don't have any theoretical novelties in this paper, I reduce my score to 6. Even so, I feel that the paper would be a valuable contribution because \na) A generative model (GAN/VAE/VQVAE/Flow based models/score matching based models) might add extra benefit in the synthesis problem, as compared with a supervised model without a similar generative component such as Tacotron. The NVAE has been shown to significantly outperform the regular VAE in image generation tasks. It stands to reason that it would do well in speech generation also. \nb) Speed, robustness and ease of implementation (although this remains to be demonstrated).\n \nInitial Review\n===========\n\nThis paper proposes a non-autoregressive (non AR) way to perform text to speech synthesis. It uses a VAE based setup - adapted from the recent image paper NVAE to build two stacks of hierarchical VAE blocks  (in priors), one going bottom up and the other, top down. The key claims are that it results in improved speed, and reduced model footprint from using a non AR architecture, with excellent quality comparable to the best autoregressive/recurrent methods in Tacotron2 [2] and non AR glow-TTS[3].\n\nThe work contains many interesting ideas for TTS, and I am very interested in seeing how this work pans out in practical speech synthesis applications.\n\nKey ideas:\n1. The bidirectional stack, which they call BVAE is adapted from the recent NVAE work which has produced stellar image generations. The model uses 1D convolutions under the hood, in contrast with the fashionable, but slow autoregressive flows or recurrent models. If one can get such a model to work, it could be advantageous in effecting savings in computational time and model size. \n\nDuring training, at the top of the bottom-up stack, text features are inflated to the size of the mel spectrogram features, and reconstructed with the top down BVAE stack. For inference, text is inflated to an expanded text matching audio mels, and then sent down the top-down stack to give a mel sample.\n\n2. Attention modeling: An important consideration here is to align text and mel, commonly done with an attention mechanism. In this work, the attention alignment shows up as a duration model, which is rather interesting, and seemingly gives additional flexibility. After aligning text and mel (using dot product), the alignment can be reinterpreted as a duration model by comparing phoneme and mel frame alignments. Furthermore, they use a discrete match with argmax rather than a sum over all attention alignments as is generally done. This also necessitates the use of the straight-through estimator while backpropagating since the durations are rounded entities. This type of modeling seems also to be used in the Glow-TTS  work but with alignments determined through dynamic programming.\n\nI found the result that the model is not very sensitive to alignment mismatches to be quite remarkable.\n\n3. Fittings for robustness during inference: They use several instructive ideas - jittering text, adding positional embeddings, diagonal penalty (since alignment is mostly diagonal) and KLD annealing. \n\n4. Analyses - ablations to see which of the VAE blocks affect the result by varying temperature (from Glow [3]).\n\nMy thoughts:\nGenerally, the paper made for fascinating reading. Having worked with Tacotron, I have always felt that adding a VAE to that (RNN based) setup would improve its generative capabilities by giving it additional regularization qualities, among other things. That we can see the model perform better when we add jitter and can also respond to the duration specified seems to corroborate that in a loose way (figure 10). \n\n- Could the authors clarify how the duration modeling results in 'monotonic' alignments? As far as I can see, the argmax guarantees a unique match, but is monotonicity necessary?\n\nFrom section 5.3.2:\n\"Since the text is forced to be used monotonically in the duration-based generation, it makes the model more robust to the attentionerrors while making fewer pronouncing mistakes.\"\n\n- A comparison with an equivalent soft attention implementation might be insightful. \n\n- Multi Speaker TTS: I am wondering how this model would perform in a multispeaker dataset, say libritts. One aspect that the paper does not touch in detail is in its capabilities as a generative model. It would be interesting, for instance, to see if this model can in any way separate speaker style from content with a multispeaker model.\n\nOverall, I think this paper would be a good addition to the body of speech synthesis work, and recommend that it is accepted.\n\n\n[1] NVAE: https://arxiv.org/pdf/2007.03898.pdf\n[2]: Tacotron2: https://arxiv.org/pdf/1712.05884.pdf\n[3] Glow-TTS: https://arxiv.org/pdf/2005.11129.pdf\n[4]: Glow: https://arxiv.org/pdf/1807.03039.pdf\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5QX06tv9adT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents BVAE-TTS, which applies hierarchical VAEs (using an approach motivated by NVAE and Ladder VAEs) to the problem of parallel TTS.  The main components of the system are a dot product-based attention mechanism that is used during training to produce phoneme duration targets for the parallel duration predictor (that is used during synthesis) and the hierarchical VAE that converts duration-replicated phoneme features into mel spectrogram frames (which are converted to waveform samples using a pre-trained WaveGlow vocoder).  The system is compared to Glow-TTS (a similar parallel system that uses flows instead of VAEs) and Tacotron 2 (a non-parallel autoregressive system) in terms of MOS naturalness, synthesis speed, and parameter efficiency. \n\n\nReasons for score:\nOverall, I think the system presented in this paper could be a valuable contribution to the field of end-to-end TTS; however, from a machine learning perspective, the contributions are incremental and quite specific to TTS.  In addition, I have some slight concerns about the clarity of the presentation that made it harder to understand the (fairly simple) approach and its motivation than I’d expect from an ICLR paper.  Finally, the quality of the speech produced by the system is only evaluated on a single dataset and uses only 50 synthesized examples in the subjective ratings.  For these reasons, I feel this paper would be a better fit for a speech conference or journal after addressing the evaluation and presentation issues, but I would still support acceptance if other reviewers push for it and my concerns are addressed. \n\n\nHigh-level Comments:\n* The speed, parameter efficiency, and MOS results are quite promising.  However, when considering the Glow-TTS paper (which this seems like a direct followup to), the system improvements seem quite incremental (replace flows with HVAEs and replace the monotonic alignment search with soft attention plus argmax).  \n* Incremental system improvements are great if they result in significant improvements that are demonstrated through rigorous experiments, however, compared to Glow-TTS, the experiments are not nearly as comprehensive and convincing. Listening to a few of the audio examples provided in the supplemental materials, I don’t get the sense that the audio quality is significantly better than that of Glow-TTS as is suggested by the MOS numbers (BVAE-TTS sounds a bit muffled to my ears relative to Glow-TTS). \n* Since this system uses the same deterministic duration prediction paradigm as Glow-TTS (and other parallel TTS systems), it suffers from the same duration averaging effects and inability to sample from the full distribution of prosodic realizations.  \n* The motivation would be made clearer if you were more specific early on about the potential advantage of VAE's relative to flows however you want to describe it (parameter efficiency, more flexible layer architectures, more powerful transformations per layer, etc.).  \n* I'd recommend providing similar motivation for using dot-product soft attention plus straight-through argmax instead of Glow-TTS's alignment search or other competing approaches.  Is it because it's a superior approach or just because it's different from existing approaches? \n\nDetailed Comments:\n* Section 2:  I don’t believe Tacotron is actually the *first* end-to-end TTS system.  Maybe it was the first to gain widespread attention, but I know that char2wav (if you count that as e2e TTS) preceded it chronologically in terms of first arxiv submission date.\n* Section 2: The Related Work section is fairly redundant with information that is already presented in the introduction.  It might be worth combining the two sections.  This should free up space for additional experiments, explanations, or analysis. \n* Section 4.1: The first paragraph here was quite confusing upon a first reading.  I had to read the second sentence (“Via the attention network…”) many times to understand what was being described.\n* Section 5.2: I’m curious how you arrived at a sample temperature of 0.333.  Was this empirically tuned for BVAE-TTS or in response to Glow-TTS’s findings? \n* Section 5.2, “Inference Time”: It seems important to include details about the hardware platform used to gather the speed results. \n* There are minor English style and grammar issues throughout the paper that make the paper slightly more difficult to read.  Please have the paper proofread to improve readability. \n\nUpdate (Nov 24, 2020):\nAfter reading through the author responses and the updated version of the paper, I feel like a sufficient number of my concerns have been addressed to increase my score to 6.  Specifically, the motivation has been made clearer, the related work section is no longer redundant with the intro, and the authors gave an adequate explanation about the necessity of their attention-based alignment method.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4bygvQ1cRVB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combined fastspeech with a hierarchical VAE (or ladder VAE? in their paper it called bidirectional VAE) to achieve parallel and high quality text-to-mel syntheisis. \n\nThe paper claims these contributions: (1) Introducing an online fashion for duration prediction, instead of distillation in FastSpeech and ParaNet. So the model is more e2e. (2) Introducing an BVAE, which extract features hierarchically to better capture prosody (overcome one-to-many) problem. During inference, can use the prior directly. This is directly than previous VAE application in TTS, which is only use to capture residual information. (3) it's faster and with same quality as autoregressive Tacotron and with better quality than other published non-autoregressive model.\n\nThe key strength of this paper is the architecture is new. I think using a hierarchical VAE here is reasonable. \n\nMy concerns mostly from the conclusion and experiments.\n(1) The paper claims compare to previous non-autoregressive model, they are more e2e, since both FastSpeech (also use duration predictor) and ParaNet (without VAE) rely on distillation. However, there is another paper called FastSpeech 2 (https://arxiv.org/abs/2006.04558, published on June 8th), the model also claim \" 1) removing the teacher-student distillation to simplify the training pipeline\".  Can the author explain the difference? Also i think need to cite that paper because it published in June and very related.\n(2) As mentioned in (1), ParaNet and FastSpeech1/2 are very related to this paper. But why only compare with waveglow?\n(3) The paper has an ablation study section, but it missing couple very simple baseline. 1) remove VAE, purely predict mel-features based on duration and phoneme embeddings. 2) using a simple VAE instead of hierachical one. How it affect the performance.\n(4) One key claim of this paper is that it is as good as Tacotron 2. However, for the in-domain test, the 0.2 behind. By listening the audio samples provided by the author, it indeed significantly worse. The out of domain looks better, I suspect the reason is Tacotron 2 has some attention failures due to it not robust as duration based model. A proper baseline here, is a FastSpeech model. Could you also provide OOD samples? It's really hard to believe such prosody gap can be filled by switch domain.\n(5) Back to the original motivation, why we need non-autoregressive model for TTS? For neural based TTS system, most of time is in vocoder. Even we assume the speed for mel-to-spec is important, I don't think measure speed with batch size = 1 is important, because non-autoregressive model can not be streaming. A proper comparison is measure FLOPS and throughput. This might make more sense for offline TTS. This is a minor concern, as long as the quality are good enough.\n(6) The paper claims their model is more compact, but there is no comparison for a smaller Taco2 model or other non-autoregressive model. \n\nIn summary, based on my understanding, this paper proposed a new non-autoregressive based text-to-mel model with quality regression but possible better robustness. My opinion is that it's a borderline for ICLR, since the importance of the proposed VAE was not well justified, and the quality was not as good as autoregressive model. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Non autoregressive modelling for text to speech (TTS) is an important and challenging problem. This paper proposes a deep VAE approach and show promising results. Both the reviewers and the authors have engaged in a constructive discussion on the merits and claims of the paper. This paper will not be the final VAE contribution to TTS but represents a significant enough contribution to the field to warrant publication. It is highly recommended that the authors take into account the reviewers' comments.
Decision: Accept (Poster)