PortaSpeech: Portable and High-Quality Generative Text-to-Speech
Authors: Yi Ren
Jinglin Liu
Zhou Zhao
Venue: NeurIPS 2021 Poster (2021)
Keywords: text to speech
speech synthesis
lightweight architecture
non-autoregressive

Abstract: Non-autoregressive text-to-speech (NAR-TTS) models such as FastSpeech 2 and Glow-TTS can synthesize high-quality speech from the given text in parallel. After analyzing two kinds of generative NAR-TTS models (VAE and normalizing flow), we find that: VAE is good at capturing the long-range semantics features (e.g., prosody) even with small model size but suffers from blurry and unnatural results; and normalizing flow is good at reconstructing the frequency bin-wise details but performs poorly when the number of model parameters is limited. Inspired by these observations, to generate diverse speech with natural details and rich prosody using a lightweight architecture, we propose PortaSpeech, a portable and high-quality generative text-to-speech model. Specifically, 1) to model both the prosody and mel-spectrogram details accurately, we adopt a lightweight VAE with an enhanced prior followed by a flow-based post-net with strong conditional inputs as the main architecture. 2) To further compress the model size and memory footprint, we introduce the grouped parameter sharing mechanism to the affine coupling layers in the post-net. 3) To improve the expressiveness of synthesized speech and reduce the dependency on accurate fine-grained alignment between text and speech, we propose a linguistic encoder with mixture alignment combining hard word-level alignment and soft phoneme-level alignment, which explicitly extracts word-level semantic information.  Experimental results show that PortaSpeech outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metrics, and shows only a slight performance degradation when reducing the model parameters to 6.7M (about 4x model size and 3x runtime memory compression ratio compared with FastSpeech 2). Our extensive ablation studies demonstrate that each design in PortaSpeech is effective.

PortaSpeech: Portable and High-Quality Generative Text-to-Speech
Ren" Zhejiang University rayerenśzju. edu.
Jinglin Liu" Zhejiang University jinglinliuOzju. edu. en
Zhou Zhao Zhejiang University
zhaozhou@zju. edu cn
Abstract
Non-autoregressive text-to-speech (NAR-TTS) models such FastSpeech 2 [24] and Glow-TTS { can synthesize high-quality speech from the given text in parallel. After analyzing two kinds of generative NAR-TTS models (VAE and normalizing flow), we find that: VAE is good at capturing the long-range semantics features (e.g.. prosody) even with small model size but suffers from blurry and unnatural results; and normalizing flow is good at reconstructing the frequency bin-wise details but performs poorly when the number of model parameters is limited. Inspired by these observations, to generate diverse speech with natural details and rich prosody using lightweight architecture, generative -o---eee-eehhhhh model. Specifically, 1) to model both the prosody and mel-spectrogram details accurately, adopt lightweight VAE with enhanced prior followed by a flow-based post-net with strong conditional inputs as the main architecture. 2) To further compress the model size and memory footprint, introduce the grouped parameter sharing mechanism to the affine coupling layers in the post-net. 3) To improve the expressiveness of synthesized speech and reduce the dependency on accurate fine-grained alignment between text and speech, we propose a linguistic encoder with mixture alignment combining hard word-level alignment and soft phoneme-leve alignment, which explicitly extracts word-level semantic information. Experimental results show that PortaSpeech outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metries, and shows only a slight performance degradation when reducing the model parameters to 6.7M (about 4x model size and 3x runtime memory compression ratio compared with FastSpeech 2). Our extensive ablation studies demonstrate that each design in PortaSpeech is effective
propose PortaSpeech, a portable and high-quality
Introduction
Recently, deep learning- based text-to-speech (TTS) has attracted
attention in speech commu- 29 35 Among neural network-based -ased TTS systems, some of them generate
nity [5] 20 22 24
25
mel-spectroggams autoregressively from text [15] 22 29 35) and suffer from slow inference speed
and robustness (word skipping and repeating) problems (25). while others [8] [2 [6]
[9]
24 25
generate mee-spectrograms in parallel with comparable quality using non-autoregressive architecture, called NAR-TTS, which enjoys fast inference and avoids robustness issues in the meanwhile. In general, modern TTS models aim to achieve the following goals:
Fast: to reduce the cost of computational resources and apply the model to real-time applications, the inference speed of TTS model should be fast.
"Equal contribution.
1 "orresponndig author
3 Audio samples are available at https::/ppottappeed. githhi..itttu..... io/
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Lightweight: to deploy the model to mobile or edge devices, the model size should be small and the runtime memory footprint should be low.
High-quality: improve the naturalness of synthesized speech, the model should capture the details (frequency bins between two adjacent harmonics, unvoiced frames and high-frequency parts) in natural speech.
Expressive: to generate expressive and dynamic speech, the model should use powerful prosody modeling methods to accurately model the fundamental frequency and duration of speech.
Diverse: prevent the synthesized speech from being too dull and tedious when generating long speech, the model should be able to generate diverse speech samples with different intonations given text input sequence.
To achieve the above goals, in this work, we propose PortaSpeech, portable and high-quality gener- ative text-to-speech model, which generates mel-speetrograms with natural details and expressive prosody using a lightweight architecture. Specifically,
Through some preliminary experiments (see Section 4.2), find that VAE is good at capturing the long-range semantics features (e.g., prosody), while normalizing flow is good at reconstructing the frequency bin-wise details. Based on these observations, we adopt VAE with enhanced prior followed by a flow-based post-net as the main model architecture of PortaSpeech, which helps PortaSpeech generate high-quality and expressive results. In addition, PortaSpeech can generate diverse speech by sampling latent variables from the prior of VAE and post-net.
Through the experiments, we also find that even when the model is very small, VAE is still good at capturing the prosody, making possible for PortaSpeech to reduce its model size using lightweight VAE. Besides, we introduce the grouped parameter sharing mechanism to the post-net to compress its model size. By doing these, PortaSpeech can be very lightweight and fast a small performance cost.
To model the prosody better and generate more expressive speech, we introduce a linguistic encoder with mixture alignment, which combines hard word-level alignment and soft phoneme- level aligmment. Our proposed linguistic encoder also reduces the dependence on fine- grained (phoneme-level) alignment and alleviates the burden of the speech-to to-text aligner.
dataset show that PortaSpeech outperforms other state-of-the ff-tth--rt TTS models with comparable model parameters in voice quality and prosody in terms of both subjective and objective evaluation metrics. When compressing the model size, our PortaSpeech shows only slight performance degradation but enjoys the benefits of much smaller number of model parameters (about 4x model size reduction) and lower memory footprints (about 3x memory reduction) compared with FastSpeech 2. The main contributions of this work are summarized as follows:
Experiments on the LISpeech 7
We analyze the characteristics of VAE and normalizing flow when applied to TTS and combines the advantages of VAE and normalizing flow to generate mel-spectrograms with rich details and expressive prosody.
We propose mixture alignment in the linguistic encoder, which improves the prosody and reduces the dependence ine-grained (phoneme-level) hard alignment.
Using lightweight VAE and introducing the grouped parameter sharing mechanism to the post-net, PortaSpeech can generate high-quality speech with small number of model parameters and small runtime memory footprints.
2 Background
In this section, we describe the background of TTS and the basic knowledge of VAE and normalizing flow. We also review the existing applications of VAE and normalizing flow in non-autoregressive TTS and analyze their advantages and disadvantages.
Text-to-Speech Text-to-speech (TTS) models convert input text or phoneme sequence into mel- spectrogram (e.g., Tacotron (35), FastSpeech (25).. which then transformed to waveform using vocoder (e.g., WaveNet [33)) directly generate waveform from text (e.g., FastSpeech 2s [24

---

and EATS (5). End-to-end rext-to-speech models have gradually developed from autoregressive to non-autoregrressvve architecture: early autoregressive text-to-speech models (2 35 generate each mel-spectrogram frame conditioned on previous ones, resulting in high inference latency and low robustness. Recently, several non-autoregressive TTS works have been proposed, which generate mel- spectrogram frames in parallel. FastSpeech [25] and ParaNet [2] are the first non-autoregressive TTS models, which use pre-trained autoregressive TTS teacher models to extract (ext-to-spectrogrra align- ments from the training data to bridge the length gap between text and speech for non-autoregressive student model. FastSpeech 2 24 introduces more variation information of speech, including pitch and energy, to alleviate the one-to-many mapping problem in TTS. While these methods need external text-to-spece -op-ttt--pe alignment models or tools, Glow-TTS g directly searches for the most probable monotonic alignment between text and the latent representation of speech using normalizing flows and dynamic programming. In addition to improving the performance of hon-autoregressive models, some works focus on lightweight and portable model designs: SpeedySpeech [32 replaces the self-attention layers with fully convolutional blocks to reduce the computational complexity. LightSpeech [17] leverages neural architecture search (NAS) to automatically design more lightweight models, while the training of NAS consumes huge resources. In this work, we save the model parameters by taking advantage of the characteristics of VAE and normalizing flow and introducing the grouped parameter sharing mechanism.
VAE The VAE is generative model in the form of px z) (x( where p(z) is prior distribution over latent variables z and (x(2)) ) is the likelihood function that generates data x given latent variables z which can be considered as a decoder. It is parameterized by neural network A. Since the true posterior po(x. z) over the latent variables of a VAE is usually analytically intractable, approximate it with a variational distribution (z(z)x) which can be viewed an encoder. The parameters B and á can be optimized by maximizing the evidence lower bound (ELBO):
Po z)
= Egngg (a/x) log po (x|z)) - -log qoi [z) xx)
log (x) (k)((1)))) log
[z(xx
(z)
-F [log po
po [x]]] - KL (1o (20(2)x))lpo(())
(),
Recently, some works successfully apply VAE to TTS. One of them is BVAE-TTS 14, which adopts a bidirectional-inferrne variational autoencoder that learns hierarchical latent representations using both bottom-up and top-down paths increase its expressiveness. Thanks the hierarchical structure and latent modeling, B BVAE-TTS can capture the dynamism and variability of ground-trude prosody. However, its generated mee-spectssgrams are very blurry and over-smoothing, resulting in unnatural sounds, due to the posterior collapse [I] 6 and the reconstruction loss term used in BVAE-TTS, which has independency assumption of generated frequency bins given latent variables.
Normalizing Flow Normalizing flow is a kind of generative models 3 4 which has several advantages including exact log-likelln likelihood evaluation and fully-parallel sampling. In generation, normalizing flows 3 transform the latent variable z into a datapoint x through a composition of invertible functions f o f2 D f and assume tractable prior po (z) over latent variable z sampled from a simple distribution (e.g.. a Gaussian distribution). In training, the log-likelihood of a datapoint x can be computed exactly using the change of variables rule:
K
(x) = loz)o(z) +
Σ log (db;
where hn = x, h, = (h; lx = 2 and det (dh,
is the Jacobian determinant. We learn the parameters of fj ... fx by maximizing Equation ①1) over the training data. Given g = f-1, we can now generate a sample x by sampling z ~ (z) and computing x = g()).
There are several normalizing flow-based non-autoregressive TTS methods: Flow- NWS-TTS 19 is an early flow-based TTS method, which replaces the decoder in FastSpeech with Glow and jointly learns the alignment and mel-spectrogram generation through single network. Then Glow-TTS g proposed, which combines the normalizing flow and dynamic programming-bor based monotonic alignment to enable fast, diverse and controllable speech synthesis. These methods handle the blurry mel-spectrogram problems well due to the nature of the normalizing flow. However, according to our experiments (see (Section 4.2). flow-based NAR-TTS model usually requires huge model capacity to achieve good performance, and the performance can drop notably when reducing the number of model parameters.
3

---

Word-to-Phoneme
Attention
HiRiGAN
Decoder
Squeezel UnSqueeze
Word-level
L.R
Duration Predictor
Duration
VP-Flow
Affine Coupling Layer
Flow-based Post-Net
Hor Zo
Inv 1xl Convid
Word Encoder
Encoder
Variational Generator
ActNorm
WI
Phoneme Encoder
N(.1)
Linguistic Encoder
Phoneme, Word Boundary
Phoneme
Boundary
(a) PortaSpeech
(b) Linguistic Encoder
(c) Variational Generator
(d) Flow-based Post-Net
Figure The overall architecture for PortaSpeech. subfigure (b), "WP" denotes the word-level pooling operation, "LR" denotes the length regulator proposed in FastSpeech and "sinusoidal-lig symbol" denotes the positional encoding. In subfigure (c), "VP-Flow" denotes the polume-preserving normalizing flow. In subfigure (c) and (d), the operations denoted with dotted lines are only used in the training procedure.
3 PortaSpeech
Considering the characteristics of VAE and normalizing flow mentioned in Section 2 to build TTS system that can meet the goals described in [Section we propose PortaSpeech, which combines the advantages of VAE and normalizing flows and overcomes their deficiencies. As shown in Figure la PortaSpeech is composed of linguistic encoder with mixture aligmment, variational generator with enhanced prior and flow-based post-net with the grouped parameter sharing mechanism. First, the text sequence with word-level boundary fed into the linguistic encoder extract the linguistic features in both phoneme and word level. Secondly, to model the expressiveness and variability of speech with lightweight architecture, we train the VAE-based variational generator to maximize the ELBO over the ground-truth mel-spectrograns conditioned the linguistic features, whose prior distribution is modeled by a small polume-ppesertve normalizing flow. Finally, to refine and enhance train the post-net by maximizing
the natural speech details in the generated mel-spectrog,,,ss the likelihood ground-trute mel-spectrograms conditioned on both the linguistic features and the outputs of the variational generator. During inference, the text is transformed to mel-spectrograms by successively passing through the linguistic encoder, the decoder of the variational generator and the reversed flow-based post-net. We describe these designs and the training and inference procedures in detail in the following subsections. We put more details in Appendix A
3.1 Linguistic Encoder with Mixture Alignment
To expand the lengths of linguistic features (outputs of the linguistic encoder), previous autoregressive TTS models introduce a duration predictor to predict the number of frames of each phoneme (phoneme duration) and the ground-truth phoneme duration (hard alignment) is obtained by external models/tools (e.g., FastSpeech (25) and FastSpeech (24) or jointly monotonic alignment training (e.g., Glow-TTS [ and BVAE-TTS [14). However, phoneme-level hard alignment has several issues: since some of the boundaries between two phonemes are naturally uncertain) Rl it challenging for the alignment model to obtain very accurate phoneme-level boundaries, which inevitably introduces errors and noises. Further, these alignment errors and noises can affect the training of duration predictor, which hurts the prosody of the generated speech in inference. To tackle
'It could be difficult to determine the exact boundary between two phonemes in millisecond level even for manually labeling.

---

these problems, we introduce mixture alignment to the linguistic encoder, which uses soft alignment in phoneme level and keeps hard alignment in word level.
As shown in Figure lb, our linguistic encoder consists of a phoneme encoder, a word encoder, duration predictor and word-to-phoneme attention module and detailed architecture of these modules are put in Appendix [ Suppose we have an input phoneme sequence together with the word boundary (for example, "HH AEI Z N EHI V ERO", where "I" denotes the word boundary in phoneme sequence). First, we encode the phoneme sequence into phoneme hidden states Hp. Then we apply word-level pooling on Hp to obtain the input representation of the word encoder, which averages the phoneme hidden states inside each word according to the word boundary. The word encoder then encodes the word-level hidden states into word-level hidden states and expanded them to match the length of the target mel-speetrogram (denoted as H.w)) using length regulator with the word-level duration. Finally, to add fine-grained linguistic information, we introduce a attention module, which takes Hi as the query and Hp as the key and the value. In addition, due to the monotonic nature of -ettto--peetrogrrm alignment, to encourage the attention to be close to the diagonal, we add word-level relative positional encoding embedding to both Hp and H. before they are fed into the attention module. To predict the word-level duration, we use the duration predictor which takes Hp input and then sums the predicted duration of the phonemes in each word as the word-level d Our mixture alignment mechanism avoids the uncertain and noisy phoneme-level alignment extraction and duration prediction while keeping fine-grained, soft and close-to-diagonal -preetr-aee alignment.
3.2 Variational Generator with Enhanced Prior
To achieve expressive and diverse speech generation with lightweight architecture, we introduce VAE as the -pe-spectro generator, called variational generator. However, traditional VAE uses simple distribution (e-g., Gaussian distribution) as the prior, which results in strong constraints on the posterior: optimizing with Gaussian prior pushes the posterior distribution towards the mean, limiting diversity and hurting the generative power [18] 31. To enhance the prior distribution, inspired by [10 18 26 22), we introduce a small volume-presse preeeerreeeeinggg normalizing flow! which transforms complex distributions through a series of K invertible mappings (a stack of WaveNet residual blocks with dilation 1). Then we take the complex distributions as the prior of the VAE. When introducing normalizing flow-based enhanced prior, the optimization objective of the mel-spectrogram generator becomes:
simple distributions (e.g., Gaussian distribution)
logp(x|c)) 2 (xlx po (x|z, c)] (L e) Ppp(z))) L(ò, 0),
where Φb and denote the model parameters of VAE encoder, VAE decoder and the normalizing flow-based enhanced prior, respectively; e denotes the outputs of linguistic encoder. Due to the introduction of normalizing flows, the KL term in Equation (2) no longer offers a simple closed-form solution. So estimate the expectation w.r.t. 46 c) via Monte-Carlo method by modifying the KL term:
(T(gg(Z)xx, e) po(2)c) (xlx,e) [log go (2)xx))) log (z)c)))
As shown in Figure 1c. in training, the posterior distribution N(u o,,) is encoded by the encoder of the variational generator. Then 2g is sampled from the posterior distribution using reparameterization and is passed to the decoder of the variational generator (the right dotted line). the meanwhile, the posterior distribution is fed into the VP-Flow to convert it to standard normal distribution (the middle dotted line). In inference, VP-Flow converts a sample in the standard normal distribution into sample 2p in the prior distribution of the variational generator and we pass the Zp to the decoder of the variational generator.
3.3 Flow-based Post-Net
To generate high-quality mel-spectrograms, normalizing flows 88
19 have been widely proved to be
effective. Unlike simple loss-based (L. or MSF-based) VAE-based methods that often generate
'In training. the ground truth word-level duration can be obtained by external forced alignment tools or autoregressive TTS models.
'For simplicity and convenience, we use volume- preserving flow (VP-Flow (lo-Flow), which does not need to consider the Jacobian term when calculating the data log- likelihokoood We find that volume- preserving is powerful enough for modeling the prior.
5

---

blurry outputs, flow-based models can overcome the over-smoothing problem and generate more realistic oulputs. To model rich details in ground-truth, mel-spectrograms, we introduce a flow-based post-net with strong condition inputs to refine the outputs of the variational generator. As shown in Figure 1d] the architecture of the post-net adopts Glow [9] and is conditioned on the outputs of the variational generator and the linguistic encoder. In training, the post-net transforms the mel- spectrogram samples into latent prior distribution (isotropic multivariate Gaussian) and calculates the exact log-likelihood of the data using the change of variables. In inference, we sample the latent variables from the latent prior distribution and pass them into the post-net reversely to generate the high-quality mel-spectrogram.
However, flo-bbased models suffer from large model footprints. Since the conditional inputs contain the text and prosody information, our post-net only focuses on modeling the details in mel-spectrograms, greatly reducing requirements for model capacity. To further reduce the model size and keep the modeling power, we introduce the grouped parameter sharing mechanism to the affine coupling layer, which shares some model parameters among different flow steps (f,, f.+1....
shared
Ia +
Xe]
unshared
16.0 Projection
G). As shown in Figure 2| we divide all flow steps (f,, f2, .... fx) into several groups and share the model parameters of NN (WaveNet-like network, see Appendix [A.3] in the coupling layers among flow steps in a group. Our grouped parameter sharing mechanism is similar to the shared neural density estimator proposed in [[3 with some differences that: 1) we simplify the model by removing the flow indication embedding since the unshared conditional projection layer in different flow steps can help the model to indicate the position of the step; 2) instead of sharing the parameters among all flow steps, we generalize the sharing mechanism by sharing the parameters among flow steps in group, making it easier adjust the number of trainable model parameters without changing the model architecture.
NN
Conditional Projection
[x]
Figure 2: Affine coupling layer with grouped param- eter sharing. Green block means sharing the model pa- rameters of this block among flow layers in group.
3.4 Training and Inference
In training, the final loss of PortaSpeech consists of the following loss terms: 1) duration prediction loss Laur: MSE between the predieted and the ground-truth word-level duration in log scale; 2) reconstruction loss of variational generator Lvg: MAE between the ground-truth mel-spectogram
and that generated by the variational generator: 3) the KL-divergence of variational generator Lкд log 9,(2) [z)xx c) log pa (z/c). where z ~ 9a (z|x, c). according to Equation (3); and 4) the negative nigllikel of the post-net L.N. In inference, the linguistic encoder first encodes the text sequence, predicts the word-level duration and expand the hidden states via mixture alignment to obtain the linguistic hidden states Hr. Secondly, we sample from the enhanced prior, and then the decoder of the variational generator generates the coarse-grained mel-spectrograms Mc (the output mel- spectrograms before post-net) conditioned on the linguistic hidden states Hr, Thirdly, the post-net converts randomly sampled latent into fine-grained mel-spectrograms M, conditioned on H., and M Finally, M is transformed to waveform using a pre-trained vocoder. Since we use hard word-level alignment in PortaSpeech, absolute durations for individual words can also be specified inference time like FastSpeech [25]. As for silences, we add a word boundary symbol as an extra special word such as "SIL" between two words in training. In this way, we can adjust the duration of silences via modifying the duration of the special word "SIL".
4 Experiments
4.1
Experimental Setup
Datasets We evaluate PortaSpeech on LJSpeech dataset 77. which contains 13100 English audio clips and corresponding text transcripts. Following FastSpeech (2), we split LJSpeech dataset into three subsets: 12229 samples for training, 348 samples (with document title LJ003) for validation and 523 samples (with document title LJ001 and LJ002) for testing. We randomly choose 50 samples in the test set for subjective evaluation and use all testing samples for objective evaluation. We convert with an open-source grapheme-to-
the text sequence to the phoneme sequence 2 25
29
30
35

---

phoneme too We transform the raw waveform with the sampling rate 22050 into mel selespects with the frame size 1024 and the hop size 256.
following 25 29
Model Configuration Our PortaSpeech consists of an encoder, variational generator and a post- net. The encoder consists of multiple feed-forward Transformer blocks [25] with relative position encoding [28] following Glow-TTS [8]. The encoder and decoder in variational generator are 2D- convolution networks. The post-net adopts the architecture of Glow [9]. We conduct experiments
on two settings with different model sizes: PortaSpeech (normal) and PortaSpeech (small). We add
more detailed model configurations of these two settings in Appendix B
Training and Evaluation We train the PortaSpeech 1 NVIDIA 2080Ti GPU, with batch size of 64 sentences on each GPU. We use the Adam optimizer with 31 = 0.9, 32 0.98, é = 10-9 and follow the same learning rate schedule in [4]. It takes 320k steps for training until convergence. The output mel-sppetroggams of our model are transformed into audio samples using HiFi-GAN [ trained in advance. We conduct the MOS (mean opinion score) and CMOS (comparative mean opinion score) evaluation on the test set to measure the audio quality via Amazon Mechanical Turk. We keep the text content consistent among different models exclude other interference factors, only examining the audio quality or prosody. Each audio is listened by at least 20 testers. We analyze the MOS and CMOS in two aspects: prosody (naturalness of pitch, energy and duration) and audio quality (clarity, high-frequency and original timbre reconstruction), and score MOS-P/CMOS-P and MOS-Q/CMOS--Q corresponding to the MOS/CMOS of prosody and audio quality. We tell the tester to focus on one aspect and ignore the other aspect when scoring MOS/CMOS of this aspect. We put more information about the subjective evaluation in Appendix B.2
4.2 Preliminary Analyses on VAE and Flow
In image generation tasks, VAE is good capturing the overall image structure information (low- frequency parts) while discarding small sharp textures/details mel-spectrograms, low-frequency parts correspond to the shape of harmonics, which determines the pitch and prosody of speech. Thus we can intuitively infer that VAE is good at modeling the prosody while not good at modeling the details in speech. While flow-hased models generate high-quality images at the cost of very large model size and huge computation complexity and we may infer that flow-based models can model the details in speech well with large model size.
parts). Similarly, in
Table 1:
The audio performance comparisons among different NAR-TTS models with different numbers of model parameters (#Params.). GT (voc.) denotes the waveform reconstructed from ground-truth mel-spectrograas using HiFi-GAN [1].
Methods
Configs
MOS-P
MOS-Q
#Params
GT (voc.)
4.49 1 0.07
4.16 1 0.06
big middle
3.71 1 0.06
3.96 1 0.07
41.2M
Flow-based
3 0.07
3.54 0.12
10.2M
small
3.21 0.12
3.42 0.14
4.5M
big
3.81 + 0.07
3.75 0.08
43.2M
VAE-based
middle
3.79 + 0.08
3.69 + 0.09
9.3M
small
3.72 + 0.08
3.51 + 0.11
4.4M
To verify our hypothesis and explore the characteristic of VAE and flow-based models in TTS, we conduct audio quality (MOS-Q) and prosody (MOS-P) comparisons among several VAE and flow-based NAR-TTS models with different model sizes: 1) big: more than 40M model parameters; 2) middle: about 10M model parameters; and 3) small: about 5M model parameters. We keep the architecture of the encoders in three models consistent. The detailed model architecture and configurations are put in Appendix [A.4] The results are shown in Table I From the table, we see that 1) when reducing the model capacities, the prosody quality of flow-based models drops significantly. In contrast, that of VAB-based model only drops slightly, according to MOS-P. This
7

---

(a) GT
(b) TransformerTTS
(c) FastSpeech
(d) FastSpeech 2
(e) Glow-TTS
(1) BVAE-TTS
(g) PortaSpeech (normal)
(h) PortaSpeech (small)
Figure 3: Visualizations of the ground-truth and generated mel mel-spectrograms by different TTS models. The corresponding text is "In being comparatively modern".
phenomenon inspires us to apply VAE-based mel-spectrogram decoder (variational generator) to our lightweight TTS model. 2) Compared with flow-based models, VAE-based model has poorer audio quality upper bound according to MOS-Q Q, which motivates us to make up for shortcomings of VAE by introducing a flow-based post-net refine the meesspectss generated by VAE.
4.3 Performance
Table 2: The audio performance (MOS-Q and MOS-P), inference latency, peak memory (Peak Mem.) and number of model parameters (#Params.) comparisons. The evaluation is conducted on a server with NVIDIA 2080Ti GPU and batch size 1. The mel-speettograms are converted to waveforms using Hif--GAN (V1) [T] RTF denotes the real-time factor, that the seconds required for the system (together with Hifi-GAN vocoder) to synthesize one-second audio.
Method
MOS-P
MOS-
RTF
Peak Mem.
#Params.
GT
4.52 + 0.07
4.41 + 0.06
/
GT (voc.)
4.48 +
4.15 + 0.07
Tacotron
3.85 0.07
3.80 + 0.08
0.115
61.78MB
28.2M
Transformer TTS 115)
3.87 + 0.06
3.82 + 0.07
0.955
118.66MB
24.2M
FastSpeech 25
3.63 + 0.08
3.72 + 0.08
0.0198
115.2MB
23.5M
FastSpeech [24]
3.72 + 0.07
3.83 + 0.06
0.0200
124.8MB
27.0M
Glow-TTS [8]
3.61 + 0.07
3.88 + 0.08
0,0196
116.4MB
28.6M
BVAE-TTS (14)
3.80 0.06
3.72 1 0.06
0.0169
IM.
12.0M
PortaSpeech (normal)
3.89 + 0.06
3.92 + 0.06
0.0216
83.6MB
21.8M
PortaSpeech (small)
3.82 + 0.06
3.86 + 0.06
0.0208
39.3MB
6.7M
We compare the quality of generated audio samples, inference latency, model size and memory footprin 10 of our PortaSpecch (normal and small model size) with other systems, including 1) GT, the ground truth audio; 2) GT' (Mel HiFi-GAN), where first convert the ground truth audio into mmelsppectrograms, and then convert the mel-spectrograms back to audio using HiFi-GAN: 3) Tacotron
2 [29]; 4) Transformer TTS [3]] 5) FastSpeech (25); ; 6) FastSpeech 2 (44; 7) Glow-77S 8 and 8)
BVAE-TTS [4 [1] The results are shown in Table 2 We have the following observations:
9hhe model parameters do not include the encoder of VAE in BVAE-TTS and PortaSpeech. We profile the peak GPU memory using MemReporter in pytorch memlab
com/ Stonesj tu/ tttptttttpphhtpeeeeeee pyytorch__mmlab/ and find maximum "active_bytes" as the peak memory during inference. "We fail in reproducing the performance of SVAE-TTS reported in the original paper, use hard text-to-speech alignment in their model and obtain reasonable results.

---

For audio quality, PortaSpeech (normal) outperforms previous TTS models in both audio quality (MOS- Q) and prosody (MOS-P), and only has slight performance degradation when reducing the model size, which shows the superiority of our proposed method.
For model size and memory foolprint, PortaSpeech (small) has the smallest model size and memory footprint. Compared with FastSpeech 2, PortaSpeech (small) achieves 4x model size and 3x memory footprint compression ratios.
For inference speed, PortaSpeech (small) speeds up the end-to-end speech generation by 5.5x and 45.9x compared with Tacotron and Transformer and achieves similar RTF with other NAR-TTS models.
Besides, we conduct some experiments on the multi-speaker dataset and draw similar conclusions (see Appendix C]. We also conduct robustness evaluation on both single-speaker and multi- speaker dataset Appendix [ and find that PortaSpeech achieves comparable robustness performance with state-of-the-ar NAR-TTS models.
4.4 Visualizations
We then visualize the mel-spectrograms generated by the above systems in Figure 3 PortaSpeech can generate mel-spectrograms with rich details frequency bins between two adjacent harmonics, unvoiced frames and sighfrreuency parts, which results in natural sounds. Besides, visualize the diverse mel-spectrograms generated by PortaSpeech in Appendix conclusion, our experiments demonstrate that PortaSpeech achieves the goals described in Section [fast, lightweight, high-quality, expressive and diverse).
We can see that
4.5
Ablation Studies
We conduct ablation studies to demonstrate the effectiveness of designs in PortaSpeech, including the enhanced prior, our post-net and the mixture alignment. We put more analyses the grouped parameter sharing mechanism in Appendix G We conduct CMOS evaluation for these ablation studies. The results are shown in Table 3
Enhanced Prior To demon- strate the effectiveness of en- hanced normalizing flow-based prior, compare models with those with simple Gaussian prior the original VAE. The sults are shown in row 2 in TTa- ble 3 We can see that CMOS- P drops when removing the en- hanced prior, indicating that the enhanced prior can improve the prosody. Since the prosody mainly modeled by VAE, com- pared with simple Gaussian prior, the enhanced prior has weaker sumptions and restrictions on the shape of the VAE prior distribution.
Table 3: Audio prosody and quality comparisons for ablation study. MA denotes mixture alignment the linguistic encoder; PN denotes the flow-based post-net; EP denotes the enhanced prior in the variational generator; Cour denotes convolutional post-net used in Tacotron 2 [29]
normal
small
Setting
CMOS-P
CMOS-Q
CMOS-P
CMOS-Q
PortaSpeech
0.000
0.000
0.000
0.000
EP
-0.194
-0.014
-0.212
-0.098
PN
-0.012
-0.458
-0.007
-0.162
PN Conv
-0,010
-0,441
-0.005
-0.148
.MA
-0.241
-0.127
-0.312
-0.157
Post-Net To demonstrate the effectiveness and necessity of flow-based post-net, we compare PortaSpeech with that without the post-net and that with convolutional post-net, which is widely used in previous TTS models, such Tacotron 2 [29]. The results are shown in row and row 4 in Table 3 From row 3, it can be seen that CMOS-Q drops significantly when removing our post-net, demonstrating that our post-net can improve the audio quality of the gen- erated specrspctoosssss.. From row 4, we can see that our flow-based post-net outperforms the commonly used convolutional post-net.
Table 4: Average absolute duration error comparisons in word and sentence level test set for PortaSpeech (small).
Settings
Word (ms)
Sentences (s)
w/ MA w/o MA
96.3
1.40
136.7
1.84

---

Mixture Alignment To demonstrate the effectiveness of mixture alignment, we replace the mix- ture alignment in the linguistic encoder with the phoneme-level hard alignment proposed in Fast- Speech 25 The results are shown in row 5 in Table 3 We can see that PortaSpeech with mixture alignment outperforms that with phoneme-level hard alignment in terms of both CMOS-P and CMOS- Q. These results demonstrate that 1) mixture alignment can improve the prosody, which may benefit from more accurate duration extraction and prediction; 2) mixture alignment can also improve the generated voice quality since the soft alignment helps the end-to-end model optimization. Then calculate the average absolute duration error in word and sentence level on the test set for PortaSpeech (small) with and without mixture alignment. The results are shown in Table 4 It can be that the linguistic encoder with mixture alignment predicts more accurate duration, also demonstrating the effectiveness of the mixture alignment. We visualize the attention alignments generated by our linguistic encoder in Appendix A. showing that PortaSpeech can create reasonable alignments which is close the diagonal.
5 Conclusion
In this paper, we proposed PortaSpeech, a portable and high-quality generative text-to-speech model. PortaSpeech uses a variational generator with an enhanced prior followed by flow-based post-net with grouped parameter sharing mechanism as the main model architecture. We also proposed a new linguistic encoder with mixture alignment to improve the prosody and reduce the dependence on the hard fine-grained alignment, which combines the hard word-level and soft phoneme-level alignments. Our experimental results show that PortaSpeech outperforms other TTS models in voice quality and prosody and shows only slight performance degradation when reducing the model size. We also conduct comprehensive ablation studies to verify the effectiveness of each component in PortaSpeech. However, to take advantage of the merits of VAE and normalizing flow, we sacrifice at the cost of more complicated model designs than previous NAR-TTS models: the overall architecture, which cascades linguistic encoder, VAE and post-net, is somewhat complicated. In the future, we will verify the effectiveness of PortaSpeech on multi-speaker and multilingual scenarios. We will also try to tap its potential on other tasks, such as voice conversion and end-to-end waavefo- generation.
Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant No. N0.20220FF0882505, National Natural Science Foundation of China under Grant No.61836002. No.62072397, Zhejiang Natural Science Foundation under Grant LR19F020006 and Baidu Scholar- ship Program.
References
[1] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif Saurous, and Kevin Murphy. Fixing a broken elbo. In International Conference on Machine Learning, pages 159-168. PMLR, 2018
[2] Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yong- guo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, al. Deep voice: Real-time neural text-to-speech. arXiv preprint arXiv: 2017.
[3] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 160iv:4400.8516, 2014.
[4] Laurent Dinh, Jascha Gohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08800 2016.
[5] Jeff Donahue, Sander Dieleman, Mikolaj Binkkoski,, Erich Elsen, and Karen Simonyan. End- to-end adversarial text-to-speech. arXiv preprint arXiv:2006.03575 2020.
[6] Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg- Kirkptiika....ck. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv: 1955,05534 2019.
10

---

[7] Keith Ito. The Ij speech dataset.
2017.
[8] Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: generative flow for text-to-speech via monotonic alignment search. arXiv preprint urXiv:2005. / /29 2020.
[9] Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible convolutions. In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.
[10] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743-44777 47513,4751,200 2016.
[11] Jungil Kong. Jachyeon Kim, and Jackyoung Bac. Hifi-gam: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33, 2020.
[12] Adrian taaccuki.. Fastpitch: Parallel text-to-speech with pitch prediction. arXiv preprint arXiv:2006.06873 2020.
[13] Sang-gil Lee, Sungwon Kim, and Sungroh Yoon. Nanoflow: Scalable normalizing flows with sublinear parameter complexity. arXiv preprint arXiv:2006.06280) 2020.
[14] Yoonhyung Lee, Joongho Shin, and Kyomin Jung. Bidirectional variational inference for non-autoregressive text-to-speech. International Conference on Learning Representations, 2020.
[15] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 6706-6713, 2019.
[16] Dan Lim, Won Jang, Hyeyeong Park, Bongwan Kim, Jesam Yoon, et al. Jdi-t: Jointly trained duration informed transformer for text-to-speech without explicit alignment. arXiv preprint arXiv:2005.07799, 2020.
[17] Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Jinzhu Li, Sheng Zhao, Enhong Chen, and Tie- Yan Liu. Lightspeech: Lightweight and fast text to speech with neural architecture search. arXiv preprint arXiv:2102.044: 0400000 2021.
[18] Shweta Mahajan, Iryna Gurevych, and Stefan Roth. Latent normalizing flows for many-to-many cross-domain mappings. arXiv preprint arXiv:2002.0666) 2020.
[19] Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma, Shaojun Wang, and Jing Xiao. Flow- tts: A non-autoregressive network for text to speech based on flow. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 7209-7213. IEEE, 2020.
[20] Huaiping Ming, Dongyan Huang, Lei Xie, Jie Wu, Minghui Dong, and Haizhou Li. Deep bidirectional Istm modeling of timbre and prosody for emotional voice conversion. 2016.
[21] Kainan Peng, Wei Ping, Zhao Song. and Kexin Zhao. Non-autoreeressive neural text-to-speech. ICML. 2020.
[22] Wei Ping, Kainan Peng. Andrew Gibiansky, Serean O. Arik. Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference on Learning Representations, 2018.
[23] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617-3621. IEEF, 2019.
[24] Yi Ren, Chenxu Hu, Tao Qin, Sheng Zhao, Zhou Zhao and Tie- Yan Liu. Fastspeech Fast and high-quality end-to-end text-to-speech. arXiv preprint arX1v:2006.04558, 2020.
11

---

[25] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems, pages 3165-3174, 2019.
[26] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXi:: 2015.
[27] Hendra Setiawan, Matthias Sperber, Udhay Nallasamy, and Matthias Paulik. Variational neural machine translation with normalizing flows. arXiv preprint arXiv:2005. 2020.
[28] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv: 02155, 2018.
[29] Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Skerrv-Ryan, conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783. IEEE, 2018.
Natural tts synthesis by
[30] Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie- Yan Liu. Token- level ensemble distillation for waweeeemm-t---hneeme conversion. In INTERSPEECH. 2019.
[31] Jakub Tomezak and Max Welling. Vae with vampprior. In International Conference on Artificial Intelligence and Statistics, pages 1214-1223, PMLR, 2018.
[32] Jan Vainer and Ondrej Dušek. Speedyspeech: Efficient neural speech synthesis. arXiv preprint arXiv:2008.0333 033802 2020.
[33] Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW, 125, 2016.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 5998-6008, 2017.
[35] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint
10135, 2017.
[36] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram, In ICASSP 2020-2020 IEEF International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEFE, 2020.
12


Reviews:
[{"review_id": "voJduH4Uluz", "reviewer": "Reviewer_e88V", "summary": "This paper summarizes a text-to-speech system which uses a VAE with a normalizing flow-based prior to generate spectrograms, and then uses a GAN model to convert the synthesized spectrogram to a waveform. \n  ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Results indicate a very slight improvement in terms of quantitative metrics. In terms of qualitative synthesis quality to me the proposed model does not sound perceptibly better. \n\nThe `small' version of the proposed model seems to reduce the number of parameters and the memory usage, and this might be useful in edge device applications. I am not exactly sure if the existing TTS models do not have more efficient variants though. \n\nOverall, the paper seems to propose a valid model, and although not ground breaking, the model seems to be doing a decent job in terms of generation quality, and being memory efficient. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss this. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "TSbCpdBtHi", "reviewer": "Reviewer_t4mP", "summary": "To overcome the shortcomings that autoregressive text-to-speech (AR TTS) models have (i.e. high inference latency, low robustness), non-AR TTS models such as Glow-TTS and BVAE-TTS have recently been proposed, which show competitive performance compared to AR TTS models.  \n  \nIn this paper, a non-autoregressive (non-AR) TTS model called PortaSpseech is proposed, which effectively reduces the number of its parameters by combining VAE and normalizing flow, while showing better performance than Glow-TTS and BVAE-TTS. First, by conducting preliminary analyses on VAE and Flow, it shows which model has which advantages over the other: VAE is good at capturing prosody even with small size and Flow is good at reconstructing the frequency bin-wise details. Then, to fully utilize the advantage of each method, this paper proposes novel ways to combine the models. To help model capture prosody, lightweight VAE equipped with enhanced prior is used in decoder of the main model, and normalizing flow with grouped parameter sharing mechanism is used as a post-net, which is helpful to improve fidelity while maintaining a small number of parameters. In addition to that, it also proposes a linguistic encoder with a mixture alignment mechanism, which is effective in learning prosody compared to the phoneme-level hard alignment.\n\nThrough various experiments, this paper supports its claim that PortaSpeech can generate high-quality speech with a small number of parameters. In preliminary analyses, by measuring MOS in terms of prosody and quality separately, it shows the characteristic that each VAE and Flow model has. Then, it compares PortaSpeech with previous non-AR TTS models and shows its superiority. Plus, ablation studies also show that each architectural component is really effective to improve the performance of PortaSpeech.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper gives useful guidance about, in which part, each of the generative model frameworks, VAE and Flow, is helpful for TTS models to have better performance. Especially, speaking of prosody, since it is not a general concept applied to other general generative models, I think it is quite remarkable and can give a good intuition for other researchers to develop other TTS models.  \n  \nHowever, I think it would be better if there were deeper analyses because it seems that most claims are supported by comparing MOS-P/Q. Personally, it sounds sensible that Flow is helpful in generating high-quality speech, but it sounds a little weak that VAE has an advantage in capturing prosody. \n  \nIn terms of clarity, when I read this paper, I am a little confused about the follows:  \n* For me, the word 'inter-word alignment' makes me think there is an attention mechanism between words as in Machine Translation.  \n* The paper says that Lightweghit VAE is used in the decoder, but I cannot figure out how the VAE is lightweight. I think it could be better if the number of parameters of each module is separately written in the appendix.  \n* I am a little confused when I am reading about enhanced prior. When I see figure 3-(c), the dotted line means residual connection? What original prior is used? How the prior is enhanced? I think the explanation about it could be added, and I suppose that there would be a reference of a paper \"Improved Variational Inference with Inverse Autoregressive Flow, kingma et al.\"  \n* In section 3.4, are the coarse-grained mel-spectrograms just the mel-spectrograms before post-net? I understand in that way, but the word 'coarse-grained' makes me think that it is a down-sampled one or something.  \n\nOverall, I think this paper gives a valuable guide to other researchers to develop TTS models using the VAE and Flow generative models and I think this paper could be better if there were much more analytic experiments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As mentioned in the main review, I think it would have more analytic experiments.\n\nRelated to the preliminary study, when comparing VAE and Flow architectures, since the architectures share the same encoder and length regulator, I wonder about the performance gap of capturing prosody beyond the durations.  I think it could be better if there is an analytic experiment showing the reason how VAE can help model learn prosody.\n\nAs mentioned in Section 2, there have been many models trying to reduce the size of TTS models, so it could be better if PortaSpeech is compared with the models such as SpeedySpeech and LightSpeech. (although I am aware that the models approach TTS in different ways such as teacher-student model or NAS)\n\nI hope the idea of combining VAE and flow makes it possible to develop an end-to-end TTS model such as VITS in \"Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech, Kim et al\".\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "EYQwFe_Tbg7", "reviewer": "Reviewer_zUv7", "summary": "The authors propose PortaSpeech, a non-autoregressive model that can sythesize high-quality speech with reduced model size. Based on the experiment results, PortaSpeech slightly outperforms counterpart models in speech quality and shows only a slight performance degradtaion when reducing the model size.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe proposed mixture alignment is novel.\nPortaSpeech has good MOS ratings even with reduced model size.\n\nWeekness:\nThe comparsion with [1] is missing since [1] also leverages a variational mel-spectragram encoder.\nThe comparsion of proposed mixture alignment with gaussian alignment [2][3][4] is missing\nThe model requires external aligner for training while other NAR models such as EATS [2], Glow-TTS, EfficientTTS [4] can be trained end-to-end.   \nThe robustness evalutaion is missing.\n\n[1] Isaac Elias, et al. Parallel Tacotron: Non-Autoregressive and Controllable TTS. ICASSP 2021\n\n[2] Jeff Donahue, et al. End-to-end adversarial text-to-speech. In ICLR, 2021.\n\n[3] Jonathan Shen, et al. Non-attentive Tacotron: Robust and controllable neural TTS synthesis including unsupervised duration modeling. arXiv:2010.04301, 2020.\n\n[4] Chenfeng Miao, et al. EfficientTTS: An efficient and high-quality text-to-speech architecture. ICML 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}, {"review_id": "5sNdUQC9rK", "reviewer": "Reviewer_LAM4", "summary": "This paper combines the high quality generation abilities of a flow network with the prosodic variation of a VAE, successfully avoiding the drawbacks of each architecture, to produce a single-speaker TTS model that can be scaled down significantly while still achieving close to state-of-the-art results.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is well motivated and the paper is well written. My biased subjective opinion is that there's nothing groundbreaking so this would probably not be one of the top papers but it is solid.\n\n\n3.2 Variational Generator with Enhanced Prior\n\nIt seems to me that instead of treating the complex distribution as the prior, this setup should be equivalent to continue using a gaussian prior, but adding the reverse flow and flow to the VAE encoder and decoder respectively?\n\nWhich makes me think that a gaussian prior with a sufficiently large encoder/decoder should still work just as well. The ablation experiment in the paper only removed the flow, but did not increase the size of the encoder/decoder by the size of the flow, so it is not exactly a fair comparison. It is very possible that the parameter coupling happening in the flow is key to making it work well though, which would be an interesting conclusion.\n\n\n3.3 Flow-based Post-Net\n\nThis is outside of the scope of this work, but we know flow-based methods can generate waveforms directly ala WaveGlow. Have you considered doing so instead of generating mel spectrograms, which should further simplify the mobile TTS story?\n\n\n4.1 Experimental Setup\n\nIn my opinion, LJSpeech is not a great dataset. It is single speaker and does not contain a lot of prosody variation. It would have been better to see results on something like LibriTTS.\n\n\n4.3 Performance\n\nI find the comparison of pitch accuracy a bit strange. For models with VAE it would make sense to use reference posterior latents and check that pitch contours match. But otherwise, it's expected that given a piece of text there can be many valid ways to vocalize it and so a higher pitch accuracy isn't necessarily always a positive result for a TTS model.\n\n\nMixture Alignment\n\nCan the mixture alignment be used to specify absolute durations for individual words at inference time? What about for silences? If it gives this capability then this can be an important feature and is superior to the alignment in FastSpeech which can only do relative scaling.\n\n\n# Originality\nVAE-based and flow-based non-autoregressive TTS models are not new. I imagine there have been plenty of experiments combining the two as well. The use of a volume-preserving flow inside the VAE is novel. The ability to reduce the model size while maintaining quality with parameter sharing in the flow might be novel.\n\n\n# Quality\nThe submission is high quality and includes relevant experiments. Some experiments on a tougher dataset would make it a better submission.\n\n\n# Clarity\nThe submission is clear and well organized.\n\n\n# Significance\nThe results can be a good step towards high quality mobile TTS, though it is hard to say how much traction this can gain compared to approaches like sparsifying/compressing more well known models. If the flow could be altered to generate waveforms directly while matching the quality of FastSpeech2+Parallel WaveGAN, this could immediately become a top paper in TTS.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mention the potential negative societal impact, but do not offer any suggestions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an interesting blend of VAE and flow based approach for TTS. The reviewers raised several points about the comparisons -- including that some of the baselines are possibly not as good as the original work, since original implementations were not released and third party implementations had to be used for comparison. The authors address a lot of these concerns in the discussions and added analyses and clarification that I hope will make it to the final submission as they strengthen the presentation significantly. Thanks, to the reviewers for their constructive suggestions. 
Decision: Accept (Poster)