Controllable and Lossless Non-Autoregressive End-to-End Text-to-Speech
Authors: Zhengxi Liu
Qiao Tian
Chenxu Hu
Xudong Liu
Mengling Wu
Yuping Wang
Hang Zhao
Yuxuan Wang
Venue: NeurIPS 2022 Submitted (2022)
Keywords: text-to-speech
prosody modeling
VAE
flow-based model

Abstract: Some recent studies have demonstrated the feasibility of single-stage neural text-to-speech, which does not need to generate mel-spectrograms but generates the raw waveforms directly from the text. Single-stage text-to-speech often faces two problems: a) the one-to-many mapping problem due to multiple speech variations and b) insufficiency of high frequency reconstruction due to the lack of supervision of ground-truth acoustic features during training. To solve the a) problem and generate more expressive speech, we propose a novel phoneme-level prosody modeling method based on a variational autoencoder with normalizing flows to model underlying prosodic information in speech. We also use the prosody predictor to support end-to-end expressive speech synthesis. Furthermore, we propose the dual parallel autoencoder to introduce supervision of the ground-truth acoustic features during training to solve the b) problem enabling our model to generate high-quality speech. We compare the synthesis quality with state-of-the-art text-to-speech systems on an internal expressive English dataset. Both qualitative and quantitative evaluations demonstrate the superiority and robustness of our method for lossless speech generation while also showing a strong capability in prosody modeling.

Controllable and Lossless
Non-Autoregressive End-to-End Text-to-Speech
Anonymous Author(s)
Affiliation
Address email
Abstract
Some recent studies have demonstrated the feasibility of single-stage neural text- to-speech, which does not need to generate mel-spectroggrass but generates the raw waveforms directly from the text. Single-stage text-to-speech often faces two problems: a) the one-to mn-to mapping problem due to multiple speech variations and b) insufficiency of high frequency reconstruction due the lack of supervision of ground-truth acoustic features during training. To solve the a) problem and generate more expressive speech, we propose novel phoneme-level prosody modeling method based on a variational autoencoder with normalizing flows to model underlying prosodic information in speech. We also use the prosody predictor to support end-to-end expressive speech synthesis. Furthermore, we propose the dual parallel autoencoder to introduce supervision of the ground-truth acoustic features during training to solve the b) problem enabling our model generate high-quality speech. We compare the synthesis quality with state-of- the-art text-to-speech systems an internal expressive English dataset. Both qualitative and quantitative evaluations demonstrate the superiority and robustness our method for lossless speech generation while also showing strong capability in prosody modeling.
2
3
10
11
12
15
16
17
Introduction
18
19
With the rapid development of deep learning, neural text-to-speech (TTS) systems can generate natural and high-quality speech and thus have drawn much attention in the machine learning and speech community. TTS task that aims at synthesizing raw speech waveforms from the given
20
21
22 source text. Most previous neural TTS systems' pipelines are two-stage. The first stage to generate intermediate speech representations (e.g., mel-spectrograms) autoregressively [36 29
23
24 19 or
24
non-autorrereessivlly 27
26 14] from input text. The second stage synthesize speech waveforms
from the generated intermediate speech representations using vocoder [10 systems with two-stage pipelines can synthesize high-quality speech but still have trawbacks because they need sequential training or fine-tuning [15]. In addition, the use of predefined intermediate representations prevents further improvement in overall performance, as two system components can not be jointly trained and connected by learned intermediate representations.
22
25
38
33 These
26
27
28
29
Recently, several works (FastSpeech 2s 266 EATS 71. VITS (15) have proposed parallel end-to-end TTS models that generate raw waveforms directly from input text in a single stage. FastSpeech 2s introduces explicit pitch and energy as mel-spectrogram decoder conditions to alleviate the one-to-
31
32
33 many mapping problem in the TTS system. However, it needs to extract these handcrafted features in
Submitted to 36th Conference Neural Information Processing Systems (NeurIPS 2022). Do distribute.

---

34 advance, complicating the training pipeline. Moreover, FastSpeech 2s only models pitch and energy
35 but does not disentangle other prosody features from the speech. EATS and VITS can synthesize high-quality speech, but they do not disentangle prosody information from speech, so they can not achieve prosody modeling and control.
36
37
To solve the problem that previous single-stage stage parallel end-to-end TTS models do not model the general prosody, we propose the Controllable and LOssless Non-autoregressive End-to-end TTS (CLONE), which contains some carefully designed components to disentangle and model the general prosody from speech.
39
40
41
42
Firstly, to better solve the one-to-many mapping problem, we need to model the information variance other than text in speech. We propose an implicit phoneme-level prosody latent variable modeling instead of only explicitly modeling pitch and energy in FastSpeech 2s. The phoneme-level prosody latent variable models general prosody in the speech in a unified way without supervision. Specifically, we assume that prosody follows a normal distribution and use variational autoencoder [16] (VAE) with normalizing flows 28 [ to model it, which enhances the modeling ability of pure VAE and enables better modeling of prosody information that has extremely high variance. We propose a
44
45
46
47
48
49 prosody predictor to predict the prior distribution of phoneme-level prosody latent variable from the input phoneme, which enables end-to-end synthesis as a TTS system.
50
51 In addition, we carefully study the problem of unsatisfactory figh-frequency information generation
52 in single-stage end-to-end speech synthesis, which is caused by the lack of the supervision of ground- truth acoustic features during training. To enhance the learned intermediate representation, we propose 54 the dual parallel autoencoder (DPA) that consists of two parallel encoders (the acoustic encoder and the posterior wave encoder) and a wave decoder. DPA uses ground-truth linear spectrograms to regularize the learned intermediate representations for efficient learning. Besides, introduce the multi-band discriminator (MBD) that significantly speeds up model convergence and improves generation quality. DPA and MBD enable CLONE to synthesize high-quality speech at the lossless high sampling rate (48 kHz) with better high-frequency information.
53
55
58
59
80
We conduct experiments on our private speech datasets. The results of extensive evaluations show that CLONE outperforms SOTA two-stage and single-stage TTS models [29] 26 15) in terms of speech quality. In addition, CLONE can synthesize lossless speech at 48 kHz with better speech quality. Furthermore, we demonstrate that CL.NN can model and control prosody by the phoneme-level prosody latent variable and generate speech with appropriate prosodic inflections. We attach audio samples generated by CLONE at httttt://cooeeeeurrrere http:p/2222.........o github. io/CLONE/
61
63
64
65
2 Related Work
66
Text-to-Speech Text-to-Speech (TTS), which aims to synthesize intelligible and natural speech waveforms from the given text, has attracted much attention in recent years. Specifically, the neural network-based TTS models [36] 29 27 26 22 have achieved tremendous progress. The quality of the synthesized speech improved a lot and is close to that of the human counterpart. The previous prevalent methods are two-stage. The general pipeline of two-stage methods is: first, generate the 34 or non-
68
69
70
71
72
acoustic features (c.g., mel-spectrograms) from text autoregressively [36] 29
73
autoregressively (27
26 [4| 23), then synthesize the raw waveforms conditioned on the acoustic 25
features 100
22
18 Recently, several single-stage end-to-end TTS models 26 7
13 have been 15 outperforms the
75
proposed to generate raw waveform directly from the text. Among them, VITS
78
two-stage models due to the advantages of learned intermediate speech representations obtained by fully end-to-end training. However, these single-stage methods have poor controllability over the prosody of synthesized speech. Specifically, EATS 77 and VITS cannot control prosody. FastSpeech can only control some pre-defined prosodic features (i.e., pitch and energy), and these features are required to be extracted advance. Unlike the aforementioned single-stage models, by utilizing conditional VAE with normalizing flows, CLONE achieves high controllability over the general phoneme-level prosody of synthesized speech.
77
78
79
2s 26
80
2

---

Intter wid
Autecrodder
Wave Decoder
Accustic Enowler
Mylre M/uue
Linguisik Feature
Intere F
Pandly sene
Flow
-N((0.1)
Daraion Prokeke
Acoustic Encoder
Poserier Wave Linoddrr
Liqulakk Poatas
Repenasttiatiin
Rquunateiiaiin
Ma
Prepetton
As-
Prosudy tradicier
Posterice Encoder
Posterior Enccalrr
Form lood Reedy
Text Ensowder
Dilased CNN
Phoneen
Liner Spotrggee
(a) The overview of CLONE
(b) The overview of prosody modeling method
Figure 1: The overview of CLONE and prosody modeling method.
Prosody Modeling Many previous works have focused on learning underlying non-textual infor- mation (e.g., style and prosody) in speech. In particular, some works [30 [7 introduce a reference embedding to model style and prosody.
84
85
33 extracts prosody embedding from reference spec- trogram, and (37 models reference embedding as a weighted combination of a bank of learned embeddings. Some works []
86
87
39 use the variational autoencoder (VAE) to model latent representa- tions for styles and prosody of speech. Specifically, [32] uses multi-level VAE to model fine graiied prosody at the phoneme and word level in autoregressive way.
86
39 and 12 integrate VAE
with Tacotron for better style modeling. Some works [8
||] use GMM based mixture density network to model prosodic information at phoneme and word levels. Unlike the previous models, CLONE adopts a conditional VAE with normalizing flows for the phoneme-level prosody modeling
91
92
99 to a single-stage parallel TTS system. Inspired by [8 5 40 that improve the expressive capability of prior and posterior distribution with normalizing flows, we add normalizing flows to enhance
94
95 the representation power of our prior distribution for better prosody modeling. Furthermore, the
96 prosody predictor enables CLONE predict the prior distribution of prosody directly from the text and control the prosody of generated speech during inference without the need for manually adjusting the sampling points (39] or a reference speech 37 or other modality input (e.g., video [13
99
3
CLONE
100
3.1
Overview
101
The overall model structure of CLONE shown in Figure can be regarded as a conditional VAE. Firstly, the posterior encoder converts the input spectrograms to a sequence of phoneme-level prosody latent variable 2. Then, the acoustic encoder transforms the prosody latent variable into the learnable intermediate representations conditioning on linguistic features. Finally, the wave decoder predicts the
102
103
104
105
waveforms from the learnable intermediate representations. The objective of CLONE is to maximize the evidence lower bound (ELBO) of the intractable marginal lig-likelihood of data log, (æ cc:
ELBO = Ey æ.c) (o 22
Da (g xx c)llpo(= c)),
(1)
107
where and z denote the linguistic feature and the phoneme-level prosody latent variable respectively, gs(= c) is the approximate posterior distribution of z given a data point x and condition c,
108
pa( 2, is the likelihood function of x given and c, and po(2 is the prior distribution of
given

---

111
The training loss of CLONE is the negative ELBO, which consists of the reconstruction loss
112 (- (-Rgo( [logPe(x: 2, e)) and the KI. divergence (1s(( x e) ())(p c))). The details
113 of the reconstruction loss and the KL divergence are described in Section 3.4.2 and Section 3.2
114 respectively.
115
3.2
Prosody Modeling
116
To model the prosody of speech more appropriately, we determine to model the phoneme-level prosody rather than the frame-level prosody or the word-level prosody. Because the frame-level prosody causes severe linguistic information leakages during training, and the granularity of the word-level prosody is too large to reflect the details of the prosody well. Since the input linear spectrogram is at the frame level, we need to convert the frame-level prosody feature to the phoneme level. We the obtained duration of phonemes construct the hard alignment matrix representing the correspondence between phonemes and spectrogram frames and convert the frame-level prosody feature to the phoneme-level prosody feature by the matrix as follows:
117
118
119
120
122
PF phone diag(s) A PF frame:
where PF phpneee xd and PF frome R xd denote the phoneme-level prosody feature and the frame-level prosody feature, respectively, A E R TTT denotes the hard alignment matrix, and RTa denotes the inverse of the duration of phonemes (s; 1/ Ty Oig).
125
126
127
The mean and standard deviation of the approximate posterior distribution qo(2 x, c) are predicted from the obtained phoneme-leve prosody feature and linguistic feature. We obtain the linguistic feature e by expanding the output of the text encoder according to the phoneme duration. The length of the linguistic feature is the same as the number of frames in the spectrogram.
128
129
130
The formula for KL divergence is as follows:
Cu Das (qo(2 xz () c)) = (= (s(2 x2,)) log po(z |c]]
(3)
Unlike traditional VAE, assume that the approximate posterior distribution of the phoneme-level prosody latent variable is normal distribution rather than a standard normal distribution, i.e.,
133
134 de (z| Iz c) = '' (2; Pa (zz, c), ae (z, c)). As a TTS model, we want to control the phoneme-level prosody explicitly. If follows the standard normal distribution, the prosody variation is implicitly determined in the random sampling process, which not desired. Thus, a normal distribution with variable mean and variance is a better choice. enables the prosody variation to be contained in the mean and variance of normal distribution so that the corresponding prosody can be determined by predicting the mean and variance. In addition, compared with standard normal distribution, normal distribution is more complex, which increases the prosody modeling ability to obtain diverse prosody variation.
135
136
137
138
139
141
142
We also need to increase the expressiveness of the prior distribution to match the posterior distribution. Therefore, we apply normalizing flows, which enable an invertible transformation from a simple standard normal distribution into more complex prior distribution following the rule of change-of- variables:
143
144
(2 c) N (fo(2):0, det ff((2)
148
where fo denotes the normalizing flow. After the reparameterization of VAE, we get the phoneme- level prosody latent variable which represents the phoneme-level prosody of the speech. We need 148 to convert phoneme-level to frame-level variable PF frame to match the length of the linguistic
149
feature follows:
PF frame AT.
(5)
where A T is the transposed matrix of A.
150
151
3.3
Prosody Predictor
152 We propose a prosody predictor model the correspondence between phoneme and phoneme-level prosody. The prosody predictor can predict the mean and variance of qs (a x c) from the output

---

154
of the text encoder. Therefore, CLONE can predict phoneme-lever prosody from text input in the inference stage. During inference, there are three modes to generate highly natural speech with
155
156
suitable prosody (details in Section 4.4p. The optimization goal of the prosody predictor is to minimize
157
the KL divergence between the predicted normal distribution and the posterior distribution. Therefore,
158
the training loss of the prosody predictor is as follows:
Epp (N pp ((pe,
159
where Hyp and pp are the mean and standard deviation predicted by the prosody predictor. Compared with Fastspeech 2 to directly predict pitch and energy, we predict the distribution of phoneme-level prosody, avoiding the one-to-many mapping problem.
160
161
162
is worth noting that the duration predictor we used is an improved version of the duration predictor in FastSpeech 2 [26). Since prosodic information partly determines the phoneme duration, we use as the condition of the duration predictor besides the output of the text encoder, i.e., d D2(,, e RTP, where DP denotes the duration predictor, and /. is the output of the text encoder. In this way,
163
165
188
CLONE can generate more stable and natural phoneme durations.
187
3.4
Waveform Generation
3.4.1
Motivation
169
Some end-to-end TTS studies 226]] 15 focus on generating raw waveforms directly from phonemes recently. These single-stage TTS systems usually generate learned intermediate representations from phonemes and then synthesize raw waveforms from the learned intermediate representations. Unlike the mel-spectrogram used in two-stage methods, the intermediate representation in single-stage methods is predicted by the model without training supervision, subject to prediction errors and over-smoothness: The lack of supervision of intermediate representations during training expands the search space of the single-stage model, resulting in the model being more challenging to optimize, which is reflected in the poor modeling ability for high-frequency information in our experiments. To narrow the search space of waveform generation in single-stage models, we introduce ground-truth speech signals. We design an autoencoder architecture called Dual Parallel Autoencoder (DPA) to regularize the learned intermediate representation. In addition, to further enhance the quality of the generated waveforms, we propose new discriminator called Multi-Band Discriminator (MBD).
170
173
174
175
176
177
178
180
181
MBD divides the waveform into multiple bands so that our model can separately supervise the low-fre wwwfrreqeency and high- fegh-frequency parts of the audio, improving the overall quality of the synthesized
182
183
speech.
184
3.4.2
Dual Parallel Autoencoder
185
In the dual parallel autoencoder, two parallel encoders, namely the acoustic encoder and the posterior wave encoder, generate intermediate representation, and dual training method is used to optimize them. The acoustic encoder transforms into the predicted intermediate representations ir condi- tioning on linguistic features, and the posterior wave encoder transforms linear spectrograms into the intermediate representations ir. The wave decoder acts as the decoder of DPA and generates the waveform from both intermediate representations. In practice, we concatenate ir and ir in the batch dimension to get ér frooect and send ir coneaat into the wave decoder to produce the waveform w. For dual training, we compute the mean absolute error (MAE) Lir between and fr, so that ir gets the supervision of ground-truth acoustic features from ér. and is regularized by ir, which assists CLONE in learning intermediate representation efficiently. Please note that we only use the acoustic encoder without the posterior wave encoder during inference.
186
188
189
190
191
192
193
195
196
To calculate the reconstruction loss, we convert i to mel-spectrogram my and calculate MAE with truth mel-spectrogram mgt- To make ir and fr focus on the information at the human voice frequency band for better prosody modeling, we use a one-layer linear network predict the mel-spectrogram my from ir coneett- Therefore, the whole reconstruction loss Crecon is:
197
198
Lrecon MAE(mge, m1) ) + MAE(mgt, m2).
5

---

200
3.4.3
Discriminator
We use the popular adversarial training approach HiFi-GAN 17 improve the resolution of synthesized speech. The following equations describe the loss of adversarial training:
DadvD = ((( [((D(w) - -)2 + (D(1))*)
(8)
I.
LadoG Euw [(((1)) 1) + 8C B(,
Σ 1=1
1 Nr
MAE(DD(w). Dr)n)))))D,(()))))))))) D((U))
203
where DaduD, LadoG and fm denote the loss of the discriminator, the loss of the wave decoder, and the loss the feature map, respectively. denotes the total number of layers in the discriminators. Di and N₁ denote the features and the number of features in the 1-th layer of the discriminator, respectively. 3 is the coefficient of the feature map loss fym. and we set 0.1 our experiments.
204
205
206
We use two discriminators for joint adversarial training, namely MPD in HiFi-GAN and MBD. MBD uses the pseudo quadrature mirror filter bank (Pseudo-QMF) to divide the waveform into N sub- bands. These N sub-bands with one full-band are respectively sent to N + 1 scale discriminators in MelGAN [18]. By applying different discriminators on different frequency bands of the synthesized audio, MBD can significantly enhance the generation quality of high-frequency parts, allowing 211
209
210
212
CLONE generate high-fidelity and lossless audio at high sample rate. A similar idea is used in 21 However, our method differs in that send different frequency bands into different discriminators.
215
3.5
Loss Function
216
The total loss of CLONE can be expressed as follows:
f \1 Frecon A2 Lui + A * fir + 'a * Lpp Às Lodur + 7 * LadvG,
(10)
where As represent coefficients of different components of the total loss.
4
Experiment
218
219
4.1
Dataset
220
used proprietary English speech datasets, including a single-speaker dataset and multi-speaker dataset. The single-speaker dataset contains 11,176 utterances with total audio length of 10 hours
221
222
both kHz and 48 kHz. We use 9,000 utterances the training set, 100 utterances as the validation and the remaining data the test set. The multi-speaker speech data contains five English speakers (two males and three females) with total audio length of 22 hours. We evaluate the high-qualin generation capability of CLONE on the single-speaker dataset and the prosody transfer capability of CLONE on the multi-speaker dataset.
225
226
227
4.2
Data Preprocessing
228
We convert the sequences into phoneme sequences following 2 19 26 experiments, use o0-dimensional mel-spectrograms and linear spectrograms with 2048 filters. For the audio at 24 kHz, the hop size is 300, and the window size is 1200. For the audio 48 kHz, the hop size is 300, and the window size is 2048. All use the Hann window.
232
4.3
Model Configuration
233
Our text encoder uses a stack feed-forward transformer blocks [35,, and the prosody predictor consists of 4 WaveNet residual blocks (dilated CNN) (22]. which consists of layers of dilated
234
235
convolutions with gated activation unit and skip connection. The duration predictor consists of
2-layer ID convolutional network with ReLU activation, each followed by layer normalization B and the dropout layer (31), and an extra linear layer with ReLU activation to output scalar, which is

---

238
the predicted phoneme duration. The posterior encoder consists of 8 blocks of dilated CNN, and the normalizing flow is a stack of affine coupling layers [6] consisting of 4 blocks of dilated CNN. The acoustic encoder is composed of 8 blocks of dilated CNN. The posterior wave encoder consists of blocks of dilated CNN. The structure of the wave decoder is the same as HiFi-GAN VI. To match our hop size, we change the upsampling rate 5, 5, 4, 3 and change the upsampling kernel size to
239
240
241
242
15, 15, 12, 9. The detailed typer-parametten of CLONE are listed in the appendix.
244
4.4
Training and Inference
245
We train our model on NVIDIA Tesla V100 32G GPUs with the batch size of 16 on each GPU for
248
500k steps. We use the A (20) optimizer with B1 0.8 and 82 = 0.99. The learning rate of the same optimization settings, with fixed learning rate of le-4. As VITS, to reduce training time and GPU memory usage, we employ windowed generator for training, randomly sampling segments of intermediate representation with window size of 32 frames as input to the wave decoder.
CLONE is fixed at 2e-4. The discriminator
249
250
251
CLONE has three modes of inference, and the difference lies in how the prosody latent variable is calculated. (a) Use the prosody predictor predict prosody information based on text information, and the imput of the model is only text information, just like typical TTS model. (b) The prosody latent variable is obtained through the inverse flow transformation, where the inputs of CLONE are textual information and the sampling value of the standard normal distribution. (c) The prosody information is extracted by the posterior encoder from the input linear spectrogram. In this mode, the
252
254
255
256
257
model inputs are text information and the reference linear spectrogram. Besides, we test the inference
258
speed of CLONE in mode (a) on an NVIDIA Tesla V100 32G GPU and compare it to that of VITS. The RTF of CLONE and VITS are 0.012 and 0.017, respectively, indicating that model has comparable inference speed the SOTA single-stage parallel TTS model.
259
260
261
4.5
Evaluation
4.5.1
MOS Evaluation
263
We conduct the MOS (mean opinion score) evaluation to measure the synthesis quality of different models. We use each model to synthesize the same 30 utterances] and let 15 English native speakers evaluate them. We compare with the state-of-the-art (SOTA) autoregressive TTS model Tacotron 2 22 with GMM-based attention mechanisms [4) the SOTA non-autoregresssvve TTS model FastSpeech 2 [26). and the SOTA single-stage TTS model VITS [15]. The vocoder for Tacotron 2 and FastSpeech 2 is TFGAN 33 as TFGAN has better generation robustness than HiFi-GAN
264
265
267
268
269
in our experiments. The above models all generate audio at 24 kHz. We evaluate two kinds of
270
CLONE namely 24 kHz CLONE and 24 kHz CLONE without MBD, of which the discriminators are the same as those used in HiFi-GAN. The results of the MOS evaluation are shown in Table can be seen that CLONE surpasses other SOTA models, indicating that our phoneme-level prosody modeling method and the introduction of DPA enable the model to synthesize highly natural speech
271
272
273
with appropriate prosodic inflections. In addition, CLONE is better than CLONE without MBD, demonstrating the effectiveness of MBD for generating higher quality speech.
275
Table 1: The MOS result with 95% confidence intervals of different models.
Method
MOS
CI
Tacotron 2 + TFGAN
+0.0578 4.0983
4.0717
FastSpeech 2 TEGAN
#0.0579
VITS
4.0108
+0.0601
24 kHz CLONE w/o MBD
4.1000 4.1567
+0,060
24 kHz CLONE
+-0.0528
'To test the synthesis quality and robustness of the model and avoid data leakage, we use 30 general utterances outside the dataset for testing.
7

---

278
Furthermore, we conduct CMOS evaluations on 30 utterances with ground recording audio. We compare 24 kHz VITS, 24 kHz CLONE, and 48 kHz CLONE with 48 kHz ground-truth audio, respectively, as shown Table We find that 48 kHz CLONE can generate the audio close to the 48
277
278
279
kHz ground-truth audio and outperform both the 24 kHz CLONE and kHz VITS, demonstrating that CLONE can synthesize high-fidelity 48 kHz audio.
280
Table 2: The CMOS comparison to evaluate speech generation quality at high sample rates. The audio synthesized by 48 kHz CLONE, 24 kHz CLONE, and 24 kHz VITS is compared with 48 kHz ground-truth audio, respectively.
Method
CMOS
48 kHz Ground-truth Audio
0
48 kHz CLONE
-0.1712
24 kHz CLONE
-0.2393
24 kHz VITS
-0.3621
281
4.5.2
Prosody Modeling
00 Plcck con cor.leuu of gounrad Sea-1 and Oo af strekgua prosady taai
10 Plac www.ilggiaaa strac creilaia
15 Eastgy noveisaatin qpatiedd
Figure 2: Visualization of prosody modeling. (a) shows the variation in the pitch of the audio generated from all -1 and all sampling, respectively. (b) shows the pitch comparison of the reference audio, transfer result, and end-to-end synthesis result. (c) and (d) show the pitch and energy of the reconstruction result and reference audio, respectively.
Prosody Variation We infer CLONE in mode (b), i.c., sampling values in the standard normal distribution and obtaining the phoneme-level prosody latent variable through the inverse flow trans- formation. We find that the prosody of the generated speech has very high variance by sampling different values in the standard normal distribution, which further proves the prosody modeling capability of CLONE. Figure 2 visualizes the variation in the pitch of the audio generated by setting sample values to all 1 and all 1, respectively. can be seen that a significant prosody variation can be achieved by adjusting sample values in the standard normal distribution of the inverse flow transformation.
284
285
286
287
288
Prosody Transfer We test the prosody transfer performance of CLONE. Firstly, train multi- speaker CLONE Then we infer CLONE in mode (c). We input the reference speech of speaker to the posterior encoder and use the speaker embedding of speaker 2 and the duration of the reference speech to synthesize the transfer result. The goal is to use the timbre of speaker to synthesize audio with the same prosody as the reference audio of speaker 1. We also use the end-to-end synthesis result with the timbre of speaker for comparison. In Figure Pp, the pitch envelope of the transfer result is very similar that of the reference audio and quite different from that of the end-to-end synthesis result. Besides, we calculate the MAE on pitch and energy of the prosody transfer result and the end-to-end synthesis result (both are synthesized using the duration of the reference audio) with the reference audio as ground truth, as shown in Table 3 It can be seen that the MAE of the prosody transfer result is significantly smaller than that of the end-to-end synthesis result. Above two experiments prove the effectiveness prosody transfer.
291
292
293
294
295
296
298
289
300
301
TTo implement multi-speaker TTS, we add speaker embedding to prosody predictor, duration predictor. posterior encoder, and acoustic encoder.
8

---

Table 3: The mean absolute error of pitch and energy of different methods.
Method
Pitch MAE
Energy MAE
end-to-end synthesis
79.38
35.91
prosody transfer synthesis
41.44
31.92
reconstruction synthesis by CLONE with HAPE
32.56
16.34
reconstruction synthesis by CLONE with SAPE
64.83
23.70
302
Prosody Reconstruction CLONE can extract the prosody of the reference audio through the
303
posterior encoder and then reconstruct the audio as conditional VAF. We draw the pitch and energy
curves of the reference audio and the reconstruction result, as shown in Figure 2 and Figure 2 We find that the two curves in each figure are very similar, indicating that CLONE can accurately extract and reconstruct the prosody (e.g-, pitch and energy) of the reference audio.
304
305
307
4.5.3
Ablation Study
(a) CLONE without DPA
(b) CLONE without MBD
(c) CLONE
Figure 3: The spectrograms generated by three different CLONE, (a) CLONE without DPA, (b) CLONE without MBD, and (c) complete CLONE.
Waveform Generation To further investigate the improvement of our method waveform gen- eration, we conduct ablation experiments. Firstly, we conduct MOS evaluation on the audio synthesized by the CLONE with and without MBD, as shown in Table [ We find that MBD enhances the quality of synthesized audio. In addition, we plot the spectrograms of the synthesized audio, as shown in Figure 3 Figure 33 shows that without the DPA, the synthesized audio suffers obvious svor-smos at high frequency, and Figure B shows that without MBD, the tigh-frequency details of synthesized audio are insufficient. These demonstrate that DPA significantly weakens the over- smottshmoss of the high frequency. and MBD further enhances the high- gighfrreuency details.
310
311
312
313
Prosody Extraction CLONE uses hard alignment prosody extractor (HAPE) to extract prosody, which improves the accuracy of prosody extraction. To verify this, we compare HAPE with the soft attention prosody extractor (SAPE) 9 32
317
318
where the text encoder output to query the ground-truth linear spectrogram by soft attention. We compute the MAE on pitch and energy of the above two methods Table It can be seen that the MAE HAPE is smaller than that of SAPE, indicating that HAPE extract prosody more accurately.
319
320
Conclusion
322
In this work, we propose single-stage TTS system called CLONE that can directly generate lossless waveforms from the text in parallel. Specifically, we design a phoneme-level prosody modeling method based on a variational autoencoder with normalizing flows and a prosody predictor to solve the
324
325
326
one-to- to-taty mapping problem better and support end-to-end expressive speech synthesis. Besides,
327
the dual parallel autoencoder introduced solve the problem of lacking supervision of ground- --tthth acoustic features during training, which allows the single-stage model generate lossless speech.
328
Our experiments demonstrate that CLONE outperforms existing SOTA single-stag stage and two-stage
TTS models in speech quality while performing strong controllability prosody.

---

References
331
332
[1] Kei Akuzawa, Yusuke Iwasawa, and Yutaka Matsuo. Expressive speech synthesis via modeling expressions with variational autoencoder. Proc. Interspeech 2018, pages 3067-3071, 2018.
333
[2] Sercan Ô Ank, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yong- guo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, al. Deep voice: Real-time neural text-to-speech. In International Conference Machine Learning, pages 195-204. PMLR, 2017.
334
335
336
338
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton, Layer normalization. arXiv preprint arXiv:1607.064500 2016.
339
[4] Eric Battenberg, R.J. Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6194-6198, 2020.
341
342
343
[5] Xi Chen, Diederik P Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, Ilya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv: (6l:/6/1..077/ 2016.
348
347
[6]
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
348
350
[7] Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End- to-end adversarial text-to-speech. In International Conference on Learning Representations. 2020.
351
352
353
[8] Chenpeng Du and Kai Yu. Rich prosody diversity modelling with phone-level mixture density 2021.
354
network. arXiv preprint
355
[9] Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Ron J Weiss, and Yonghui Wu. Parallel tacotron: Non-autoregressive and controllable its. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5709-5713, IEEE, 2021.
356
359
[10] Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on acousties, speech, and signal processing, 32(2):236-22 243, 1984.
360
[1I] Yiwei Guo, Chenpeng Du, and Kai Yu. Unsupervised word-level prosody tagging for control- lable speech synthesis. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7597-7601. IEEE, 2022.
382
363
384
[12] Wei-Ning Hsu, Yu Zhang, Ron J Weiss, Heiga Zen, Yonghui Wu, Yuxuan Wang, Yuan Cao, Ye Jia, Zhifeng Chen, Jonathan Shen, et al. Hierarchical generative modeling for controllable speech synthesis. In International Conference on Learning Representationss 2018.
367
[13] Chenxu Hu, Qiao Tian, Tingle Li, Wang Yuping, Yuxuan Wang, and Hang Zhao Neural dubber: Dubbing for videos according to scripts. In Advances in Neural Information Processing Systems, 2021.
368
389
370
[14] Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005. 129, 2020.
371
372
[15] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference Machine Learning, pages 5530-5540. PMLR, 2021.
10

---

375
[16] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1r 14, 2013.
arXiv preprint
376
377
[17] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022- 17033, 2020.
379
380
[18] Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. arXiv preprint ar1vv.1910 2019.
383
384
[19] Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4706-6713, 2019.
385
387
[20] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
386
389
[21] Ahmed Mustafa, Nicola Pia, and Guillaume Fuchs. Stylemelgan: An efficient high-fidelity adversarial vocoder with temporal adaptive normalization. In ICASSP 2021-2021 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6034-6034 6038. IEEE, 2021.
390
391
392
393
[22] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 1609v00609.0444 2016.
394
395
[23]
Kainan Peng, Wei Ping, Zhao Song, and Kexin Zhao. Non-autoregressive neural text-to-speech. In International conference on machine learning, pages 7586 7598. PMLR, 2020.
397
398
[24] Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. Proc. ICLR, pages 217,217, 2018.
399
401
[25] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617-3621. IEEE, 2019.
402
403
404
[26] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao and Tie- Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021.
405
406
407
[27] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao and Tie- Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Systems, 2019.
408
409
[28] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
412
[29] Jonathan Shen, Ruoming Pang, Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang. Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, al. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779-4783 IEEE, 2018
413
415
[30] RJ Skerry-Raan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron 416
417
Weiss, Rob Clark, and Rif Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In international conference on machine learning, pages PMLR, 2018
11

---

420
[31] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958,, 2014.
421
422
[32] Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, and Yonghui Wu. Fully- hierarchical fine-grained prosody modeling for interpretable speech synthesis. In ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing (ICASSP).
425
426
pages 6264-6268. IEEE, 2020.
427
[33]
Qiao Tian, Yi Chen, Zewang Zhang, Heng Lu, Linghui Chen, Lei Xie, and Shan Liu. Tfgan: Time and frequency domain based generative adversarial network for high- fighfiidelity speech synthesis. arXiv preprini arXiv: 122006, 2020.
430
[34] Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis. International Conference on Learning Representations, 2020.
431
432
433
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint arXiv: 10l.::7766.362..
434
435
2017.
[36] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703. 2017.
438
439
[37] Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning, pages 51 3189-5189. PMLR, 2018
440
442
443
[38] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203 IEEE, 2020.
447
[39] Ya-Jic Zhang, Shifeng Pan, Lei He, and Zhen- Ling. Learning latent representations for style control and transfer end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6945-6949, IEEE, 2019.
448
449
450
451
[40] Zachary Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, pages 7673-7682, PMLR, 2019.
452
Checklist
453
1. For all authors...
455
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
456
contributions and scope? [Yes]
457
(b) Did you describe the limitations of your work? [Yes] See appendix.
458
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
459
appendix.
480
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
481
482
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
12

---

484
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
466
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] We give the demo website URL of this work, where you can find results generated by our proposed method. We also attach important code snippets in the supplementary material, which are helpful for reproducing the main experimental results.
468
469
470
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Training details see Section 4 hyperparameters appendix.
475
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
474
475
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.4
476
477
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
479
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
481
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We use internal datasets.
402
483
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
484
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [No]
487
488
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
489
490
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No]
491
13


Reviews:
[{"review_id": "bfpfXvp0_aq", "reviewer": "Reviewer_xNqf", "summary": "This paper introduces a text-to-speech model that combines several generative models such as VAE, GAN, and Normalizing Flow. The authors propose to use phoneme-level latent variable to generate prosodic latent variable. Normalizing Flow was used to make more complex probability distributions for the latent space of VAE. The evaluation on internal dataset shows that the proposed method works better than recent end-to-end SOTA model, VITS.", "questions": "- line 121, “We use the obtained duration of phonemes to construct the hard alignment matrix” → The process of obtaining the duration of phonemes was not mentioned before this.\n- How do the authors safely assume that the ouput of prosody predictor is actually prosodic latent variable?\n- Why do text information included in both reference and source text the same? This is not a practical scenario and not a fair way to test the actual performance.\n- Why is approximate posterior distribution denoted as q(z|x,c)? Isn’t it q(z|x)?\n- It seems not technically correct to call the transformed prior with normalizing flow “a conditional prior (p(z|c)) because no condition was actually given. It makes sense in the paper VITS because the conditional prior distribution is constructed by feeding a text information to the network.\n- Why do the authors use TFGAN for fastspeech2 and tacotron2 baselines when there’s HiFi-GAN vocoder that is used very often in many recent works? I don’t think I have seen TTS works that use TFGAN for the vocoder.\n- Why do the authors evaluate the proposed model using internal datasets?", "limitations": "Both the limitations and potential negative societal impact are written very briefly in Appendix.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- strengths\n    - The authors try to disentangle latent information that was entangled in a single random variable in VITS.\n    \n- weakness\n    - more thorough subjective evaluations are required to evaluate how good the prosody modeling of the proposed method is. Simple MOS is not enough.\n    - Some technical details seem wrong. See Questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IA7XbuUFB_H", "reviewer": "Reviewer_1KcR", "summary": "The authors propose a controllable and lossless method of Non-Autoregressive End-to-End Text-to-Speech synthesis.", "questions": "Included in what was described above.", "limitations": "The authors mentioned limitations in the appendix.", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Weaknesses\n1. The authors claim that the proposed method generates lossless speech, but comparing the human recording and synthesized audio presented by the authors clearly shows that there is a quality loss. Also, considering that it is a generative model based on neural networks, there is an error in the authors' claim.\n1. The authors claim that the proposed method achieves high controllability, but I would like to ask what can be controlled. There is no evidence in the paper that the desired speech audio can be synthesized by controlling the model proposed in this work.\n1. The authors claim the following:\n”Unlike the mel-spectrogram used in two-stage methods, the intermediate representation in single-stage methods is predicted by the model without training supervision, subject to prediction errors and over-smoothness.”\\\nIt is not clear which intermediate features in single stage models are claimed, and the authors should provide scientific analysis of the problem of intermediate features pointed out by the authors.\n1. The authors claim the following:\n“we carefully study the problem of unsatisfactory high-frequency information generation in single-stage end-to-end speech synthesis, which is caused by the lack of the supervision of ground truth acoustic features during training.”\\\nConsidering that other models are also supervised using mel-spectrogram loss similar to the proposed method, it is difficult to see that other models lack the supervision of the ground truth, and the method proposed is fundamentally the same as the previous work in the point of view.\n1. It is necessary to clarify whether the inability to synthesize high frequencies with high quality is a common problem that occurs with other models or a problem that only occurs with the proposed model. In addition, in order to claim that DPA is a general method to increase the quality of high frequency bands, it must be shown that it is effective even when applied to other models.\n1. The authors did not specify how the comparative models were trained. Referring to the quality of audio samples, the performance of some models differs in quality from the audio samples presented in the original work or best-performing open source implementations. Given that the human recording samples presented by the authors show uniform quality than the widely used public dataset, some models appear to be compared without convergence. All comparisons should be made between sufficiently converged models, and the authors should specify the training settings of the comparison models, and if they did not converge to the level suggested in the original work, the authors should re-experiment and revise the comparison results.\n1. Most recent work use widely used public datasets and author's official implementations and pre-trained weights for fair and unambiguous comparisons. The models used by the authors for comparison are models with well-performing public implementations and pre-trained weights. By training the proposed model with the public dataset(eg, LJSpeech), authors can easily compare it with other models. This is the fairest and most reliable comparison method that can claim that the proposed method shows state-of-the-art performance.\n1. Some figures need to be improved, such as font size.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "956tD4Lppgm", "reviewer": "Reviewer_Pq1m", "summary": "The paper introduces a novel approach for prosody control in text-to-speech models. A sequence of prosody representations are introduced as latent variables in variational training. Different from prior work, where prosody information are explicitly represented, the propose method learns general and implicit ones. The general representation, on the other hand, enables better modeling of prosody information, which is crucial for on-to-many mapping problem in TTS systems. The paper also provides numerical results compared with recent works, showing a significant improvements on naturalness. The authors also provides system outputs for baseline and proposed model, which shows advantages of the proposed model.", "questions": "1. Could you add variable notations in Figure 1 (e.g. where are x, c, z etc)\n2. What do you mean by \"severe linguistic information leakages\" in line 118. Could you illustrate more?\n3. How do you get the hard alignment matrix A in equation (2)? I found a term called hard alignment prosody extractor in section 4.5.3, could you illustrate more?\n4. In equation (4), what is normalizing flow f_theta? Is there a reference to such design of priori? If not, could you illustrate more?\n5. There are six weights in loss functions and the values are pretty diverse according to appendix. And only one set of values are provided. Is the model performance sensitive to weight values?\n6. What does CMOS stands for on line 276?\n7. In section 4.5.2, Prosody Variation, you mentioned a high variance of from standard normal distribution sampling. I was wondering if you have considered the mismatch between training and inference (mode(b) in this case). During the training time it seems that the priori is not standard.\n8. This is not question but personal opinion, which is not related to the quality of the paper. While I understand the authors would like to choose a good abbreviation, (CLONE in the paper), the paper title is not super related to the novelty and strength of model, which is prosody modeling. (controllable and lossless sound very generic)", "limitations": "The authors address their concern on limitations and potential negative societal impact.\nAs to the limitations, I would suggest author provide more training details, especially on hyperparameter tuning, which can be super useful for future research to replicate the work.\n\n ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The paper in general is well written. The introduction of methodology and motivation is straightforward.\n2. The methods, including prosody modeling and dual parallel auto-encoder, are fairly novel.\n3. The numerical results and system output support the advantage of proposed model.\n4. The analysis and ablation study are thorough.\n\nWeaknesses \nI don't see significant weaknesses in the paper. However, there are some minor issue I will later address in the question section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject