FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
Authors: Yi Ren
Chenxu Hu
Xu Tan
Tao Qin
Sheng Zhao
Zhou Zhao
Tie-Yan Liu
Venue: ICLR 2021 Poster (2021)
Keywords: text to speech
speech synthesis
non-autoregressive generation
one-to-many mapping
end-to-end

Abstract: Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.

Kumar et al. FASTSPEECH 2: FAST AND HIGH-QUALITY END-TO- END TEXT TO SPEECH
Yi Ren'; Chenxu ;u Xu Tan?, Tao Qin Sheng Zhao Zhou Zhao Tie-Yan Liu?
"Zhejiang University
frayeren, chenxuhu, zhaozhou edu.
Miirosoff Research Asia
{xuta, taogin, tyliuj wmiccuoottcoooocccoommmo ttw.ciii..yy/llmeerrroooooooooo
3 Miicrosoft Azure Speech
Sheng. Zha cchho@miirooo cheoomicroooottccmmm
ABSTRACT
Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al., 2019) can synthesize speech significantly faster than previous autoregressive mod- els with comparable quality. The training of FastSpeech model relies on an au- toregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in out- put), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has sev- eral disadvantages: 1) the teacher-student distillation pipeline complicated and time-consuming, 2) the duration extracted from the teacher model is not accu- rate enough, and the target mel-spectrograms distilled from teacher model suf- fer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the is- sues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simpli- fied output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifi- cally, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves 3x train- ing speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and Fast- Speech 2 can even surpass autoregressive models. Audio samples available ththb. / wtwpthhww...offsttttooetoe fastspeech2/
INTRODUCTION
Neural network based text speech (TTS) has made rapid progress and attracted lot of attention in the machine learning and speech community in recent years (Wang et al., 2017 Shen et al,
2018 Ming al.] 2016 Arik et al.] 2017: Ping al.] 2018 Ren et al. 2019 Li al. 2019) Previous neural TTS models (Wang et al.] 2017 Shen et al. 2018 Ping et al. 2018 Li et al. 2019) first generate mel-spectrograms autoregressivvely from text and then synthesize speech from the generated mee-spectrograms using separately trained vocoder (Van Den Oord et al. 2016
Oord al. 2017
Prenger et al. 2019 Kim et al. 2018 Yamamoto
2020
Authors contribute equally to this work.
Corresponding author

---

2019).
They usually suffer from slow inference speed and robustness (word skipping and repeating)
issues (Ren et al. 2019: Chen et al., 2020 In recent years, non- -ant-rrrrrrrssivvee TTS models (Ren
et al. 2019 Lañcucki 2020 Kim et al. 2020 Lim et al. 2020 Miao et al.) 2020 Peng et al.) 2019) are designed to address these issues, which generate -eeesppecto with extremely fast speed and avoid robustness issues, while achieving comparable voice quality with previous autoregressive
models.
Among those non-autoregressivv TTS methods, FastSpeech (Ren al., 2019) is one of the most successful models. FastSpeech designs two ways to alleviate the one-to-many mapping problem: 1) Reducing data variance in the target side by using the generated mel-spectrogram from an au- toregressive teacher model as the training target (i.e., knowledge distillation). 2) Introducing the duration information (extracted from the attention map of the teacher model) to expand the text se- quence to match the length of the mel-spectrogram sequence. While these designs in FastSpeech ease the learning of the one-to-many mapping problem (see Section 2.1) in TTS, they also bring several disadvantages: 1) The two-stage teacher-student training pipeline makes the training process complicated. 2) The target mel-spectrograms generated from the teacher model have some informa- tion los compared with the ground-truth ones, since the quality of the audio synthesized from the generated mel-spectrograms is usually worse than that from the ground-truth truth ones. 3) The duration extracted from the attention map of teacher model is not accurate enough.
In this work, we propose FastSpeech 2 to address the issues in FastSpeech and better handle the one-to-many mapping problem in non-autoregressive TTS. To simplify the training pipeline and avoid the information loss due to data simplification in teacher-student distillation, we directly train the FastSpeech 2 model with ground-truth target instead of the simplified output from teacher. To reduce the information gap (input does not contain all the information to predict the target) be- tween the input (text sequence) and target output spectrooct) and alleviate the mn-yto-many mapping problem for non-autoregressive TTS model training, we introduce some variation infor- mation of speech including pitch, energy and more accurate duration into FastSpeech: in training, we extract duration, pitch and energy from the target speech waveform and directly take them as conditional inputs: in inference, we use values predicted by the predictors that are jointly trained with the FastSpeech 2 model. Considering the pitch is important for the prosody of speech and is also difficult to predict due to the large fluctuations along time, we convert the pitch contour into pitch spectrogram using continuous wavelet transform (Tueerr 1988, Grossmann & Morlet 1984) and predict the pitch in the frequency domain, which can improve the accuracy of predicted pitch. To further simplify the speech synthesis pipeline, we introduce FastSpeech 2s, which does not use mel-spectrograms as intermediate output and directly generates speech waveform from text in inference, enjoying low latency in inference. Experiments on the LJSpeech (Ito 2017) dataset show that 1) FastSpeech 2 enjoys much simpler training pipeline (3x training time reduc- tion) than FastSpeech while inherits its advantages of fast, robust and controllable (even more controllable in pitch and energy) speech synthesis, and FastSpeech 2s enjoys even faster infer- ence speed; 2) FastSpeech and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. We attach audio samples generated by FastSpeech 2 and 28 at ht
github. io/ wwstaattspesee spstsppech2/)
The main contributions of this work are summarized as follows:
FastSpeech achieves 3x training speed-up over FastSpeech by simplifying the training pipeline.
FastSpeech 2 alleviates the one-to-many mapping problem TTS and achieves better voice qual- ity.
FastSpeech 2s further simplifies the inference pipeline for speech synthesis while maintaining high voice quality, by directly generating speech waveform from text.
2 FASTSPEECH 2 AND 2s
In this section, we first describe the motivation of the design in FastSpeech 2, and then introduce the architecture of FastSpeech which aims improve FastSpeech better handle the one-to-
'The speech generated by the teacher model loses some variation information about pitch, energy, prosody, etc., and is much simpler and less diverse than the original recording in the training data.
2

---

SastS
1
Mcl-spectrogram Decoder
Waveform Decoder
Predictor
Linear Layer
ConvID
Enery
LN Dropout
Energy Predictor
Conv 1x1
Fositional
D
Franding
Pitch
Conv ReLU
Variance Adaptor
Pitch Predictor
Gated Activation
Encoder
Daration
LN + Dropout
Dilated ConvID
Poctiooaa Fucoding
Duration Predictor
ConvlD ReLU
Transposed ConvID
Phoneme Embedding
Phonome
(c) Duration/picch/enrrgy predictor
(d) Waveform decoder
(a) FastSpeech 2
(b) Variance adaptor
Figure 1:
The overall architecture for FastSpeech 2 and 2s. LR in subfigure (b) denotes the length
regulator proposed in FastSpeech. LN in subfigure (c) denotes layer normalization.
many mapping problem, with simpler training pipeline and higher voice quality, At last, we extend FastSpeech FastSpeech 2s for fully end-to-end wext-to-wavefomm synthesi
2.1
MOTIVATION
TTS is a typical one-to-many mapping problem (Wagg al., 2017 Zhu al. 2017 Jayne al. 2012 Gadermayr et al.] 2020 Chen et al.] 2021). since multiple possible speech sequences can correspond to text sequence due variations in speech, such as pitch, duration, sound volume and prosody. In -on-autoreeressive TTS, the only input information is text which is not enough to fully predict the variance in speech. In this case, the model is prone overfit to the variations of the target speech in the training set, resulting in poor generalization ability. As mentioned in Section I although FastSpeech designs two ways to alleviate the one-to-many mapping problem, they also bring about several issues including I) the complicated training pipeline; 2) information loss of target mel-spectrogram as analyzed in Table and 3) not accurate enough ground-truth duration as shown in Table 5a In the following subsection, introduce the detailed design of FastSpeech which aims to address these issues.
2.2 MODEL OVERVIEW
The overall model architecture of FastSpeech 2 is shown in Figure The encoder converts the phoneme embedding sequence into the phoneme hidden sequence, and then the variance adaptor adds different variance information such as duration, pitch and energy into the hidden sequence, finally the mel-spectrogram decoder converts the adapted hidden sequence into mel-spectrogram sequence in parallel. We use the feed-forward Transformer block, which a stack of self- attention (Vaswani et al. (017) layer and ID-convolution as in FastSpeech (Ren et al.] 2011). as the basic structure for the encoder and mel-spectrogram decoder. Different from FastSpeech that re- lies teacher-student distillation pipeline and the phoneme duration from teacher model, Fast- Speech makes several improvements. First, we remove the teacher-student distillation pipeline, and directly ground-truth -ee-speccss information loss in distilled mel-spectrograms and increase the upper bound of the voice quality. Second, variance adaptor consists of not only duration predictor but also pitch and energy predictors, where 1) the duration predictor uses the phoneme duration obtained by forced align- ment (McAuliffe et al. 2017) training target, which is more accurate than that extracted from the attention map of autoregressive teacher model as verified experimentally in Section 3.2.2 and 2) the
target for model training, which can avoid the
In this work, wave-for refers to phoneme-to-wwaeeorm,, while our method can also be appied to character-level sequence directly.
3

---

additional pitch and energy predictors can provide more variance information, which is important to ease the one-to-many mapping problem in TTS. Third, to further simplify the training pipeline and push it towards fully end-to-end system, we propose FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder). In the following subsections, we describe detailed designs of the variance adaptor and direct waveform generation in our method.
2.3 VARIANCE ADAPTOR
The variance adaptor aims to add variance information (c.g., duration, pitch, energy, etc.) to the phoneme hidden sequence, which can provide enough information predict variant speech for the one-to-many mapping problem in TTS. We briefly introduce the variance information as follows: 1) phoneme duration, which represents how long the speech voice sounds; 2) pitch, which is key feature to convey emotions and greatly affects the speech prosody; 3) energy, which indicates frame- level magnitude of mel-spectrograms and directly affects the volume and prosody of speech. More variance information can be added in the variance adaptor, such as emotion, style and speaker, and we leave it for future work. Correspondingly. the variance adaptor consists of 1) a duration predictor (i.e., the length regulator, as used in FastSpeech), 2) pitch predictor, and 3) an energy predictor, as shown in Figure [b] In training, we take the ground-truth value of duration, pitch and energy extracted from the recordings as input into the hidden sequence to predict the target speech. At the same time, we use the ground-truth duration, pitch and energy as targets to train the duration, pitch and energy predictors, which are used in inference to synthesize target speech. As shown in Figure Ic] the duration, pitch and energy predictors share similar model structure (but different model parameters), which consists of 2-layer iD-oovvoltiioall network with ReLU activation, each followed by the layer normalization and the dropout layer, and an extra linear layer to project the hidden states into the output sequence. In the following paragraphs, we describe the details of the three predictors respectively.
Duration Predictor The duration predictor takes the phoneme hidden sequence as input and pre- diets the duration of each phoneme, which represents how many mel frames correspond to this phoneme, and is converted into logarithmic domain for case of prediction. The duration predic- tor is optimized with mean square error (MSE) loss, taking the extracted duration training tar- get. Instead of extracting the phoneme duration using a pre-trained autoregressive TTS model in FastSpeech, we use Montreal forced alignment (MFA) (McAuliffe et al. (2017) too to extract the phoneme duration, in order to improve the alignment accuracy and thus reduce the information gap between the model input and output.
Pitch Predictor Previous neural network based TTS systems with pitch prediction Arik et al.) 2017 Gibiansky et al. 2017) often predict pitch contour directly. However, due to high variations of ground-truth pitch, the distribution of predicted pitch values is very different from ground-truth distribution, as analyzed in Section 3.2.2 To better predict the variations in pitch contour, we use continuous wavelet transform (CWT) decompose the continuous pitch series into pitch spectro- gram (Suni et al. 2013; Hirose & Tao 2015) and take the pitch spectrogram as the training target for the pitch predictor which is optimized with MSE loss. In inference, the pitch predictor predicts the pitch spectrogram, which is further converted back into pitch contour using inverse continuous wavelet transform (ICWT). We describe the details of pitch extraction, CWT, iCWT and pitch pre- dictor architecture in Appendix [ To take the pitch contour as input in both training and inference, we quantize pitch Fu (ground-truth/preed) truth/predicted value for train/ifference respectively) of each frame to 256 possible values in log-scale and further convert it into pitch embedding vector p and add it to the expanded hidden sequence sequence.
Energy Predictor We compute 1.2-norm of the amplitude of each short-time Fourier transform (STFT) frame as the energy. Then we quantize energy of each frame to 256 possible values uni- formly, encoded it into energy embedding and add it to the expanded hidden sequence similarly to
3 'MFA is an open-source system for speech-text alignment with good performance, which can be trained on paired tuxt-audio corpus without any manual alignment annotations. We train MFA on our training set only without other external dataset. We will work non-eutoreeresssive TTS without external alignment models in the future.
4

---

pitch. We an energy predictor to predict the original values of energy instead of the quantized values and optimize the energy predictor with MSE loss
2.4 FASTSPEECH 2s
To enable fully end-to-end text-to-waveform generation, in this subsection, extend FastSpeech to FastSpeech 2s, which directly generates waveform from text, without cascaded mel-spectrogram generation (acoustic model) and waveform generation (vocoder). As shown in Figure Ia FastSpeech 2s generates waveform conditioning on intermediate hidden, which makes it more compact in infer- ence by discarding mel-spectrogram decoder and achieve comparable performance with a cascaded system. We first discuss the challenges in non-autoregressive text-to- wavtf-wav generation, then describe details of FastSpeech 2s, including model structure and training and inference processes.
Challenges in Text-to- Waveform Generation When pushing TTS pipeline towards fully end- to-end framework, there are several challenges: 1) Since the waveform contains more variance information (e.g.. phase) than mel-spectrograms, the information gap between the input and output is larger than that in text-to-spectrogram generation. 2) It is difficult to train on the audio clip that corresponds to the full text sequence due the extremely long waveform samples and limited GPU memory. As result, we can only train on a short audio clip that corresponds to a partial text sequence which makes hard for the model capture the relationship among phonemes in different partial text sequences and thus harms the text feature extraction.
Our Method To tackle the challenges above, we make several designs in the waveform decoder: 1) Considering that the phase information is difficult to predict using variance predictor (Engel et al.) 2020), we introduce adversarial training the waveform decoder to force it to implicitly recover the phase information by itself (Yamamoto al. 2020). 2) We leverage the mel-spectrogram decoder of FastSpeech 2, which is trained on the full text sequence to help on the text feature extraction. As shown in Figure Id] the waveform decoder is based the structure of WaveNet (Van Den Oord iet al. al. 2016) including non-causal convolutions and gated activation (Van den Oord et al. 2016). The waveform decoder takes sliced hidden sequence corresponding to short audio clip as input and upsamples it with transposed 1 ID-convolution to match the length of audio clip. The discriminator in the adversarial training adopts the same structure in Parallel WaveGAN (Yamamoto et 2020) which consists of ten layers of non-causal dilated I-D convolutions with leaky ReLU activation function. The waveform decoder is optimized by the multi-resolution STFT loss and the LSGAN discriminator loss following Parallel WaveGAN, In inference, we discard the mel-spectrogram de- coder and only use the waveform decoder to synthesize speech audio.
2.5 DISCUSSIONS
In this subsection, we discuss how FastSpeech 2 and 2s differentiate from previous and concurrent works.
Compared with Deep Voice (Arik al. 2017), Deep Voice Giiiiassy et al. 2017) and other methods Fan et al. (2014); ; Ze et al. (2013) which generate waveform autoregressively and also pre- dict variance information such as duration and pitch, Fastspeech 2 and 2s adopt self-attentie based feed-forward network to generate mel-spectrograms or waveform in parallel. While some existing non-autoregressive acoustic models (Zeng al. 2020 Lim et al. 2020 Kim et al.) 2020) mostly focus on improving the duration accuracy, FastSpeech 2 and 2s provide more variation information (duration, pitch and energy) as inputs to reduce the information gap between the input and output. A concurrent work (Eancucki] 2020) employs pitch prediction in phoneme level, while FastSpeech 2 and 2s predict more fine-gre ene-grained pitch contour in frame level. In addition, to improve the prosody in synthesized speech, FastSpeech 2 and 2s further introduce continuous wavelet transform to model the variations in pitch.
While some text-to-ue -oa-efoor models such as ClariNet (Ping et al.) 2019) jointly train an autore- gressive acoustic model and a non-autoregressive vocoder, FastSpeech 2s embraces the fully non- autoregressive architecture for fast inference. A concurrent work called EATS (Donahue al. 2020
'We do not transform energy using CWT since energy is not as highly variable as pitch on LJSpeech dataset, and we do not observe gains when using it.
5

---

also employs non-autoregrressvv architecture and adversarial training convert text to waveform directly and mainly focuses on predicting the duration each phoneme end-to-end using a differen- tiable monotonic interpolation scheme. Compared with EATS, FastSpeech 2s additionally provides more variation information to ease the one-to-many mapping problem TTS.
Previous non-autoregressivve vocoders (Oord al. 2017 Prenger al. 2019 Yamamoto et al. 2020 Kumar et al. 2019) are not complete text-to-speech systems, since they convert time aligned linguistic features waveforms, and require separate linguistic model to convert input text to linguistic features or an acoustic model to convert input text to acoustic features (e.g., mel- spectrograms). FastSpeech 2s is the first attempt to directly generate waveform from phoneme sequence fully in parallel, instead of linguistic features or mel-spectrograms.
EXPERIMENTS AND RESULTS
3.1 EXPERIMENTAL SETUP
Datasets We evaluate FastSpeech 2 and 2s LJSpeech dataset (Ito 2017). LJSpeech contains 13, 100 English audio clips (about 24 hours) and corresponding text transcripts. We split the dataset into three sets: 12.228 samples for training. 349 samples (with document title LJ003) for validation and 523 samples (with document title IJ001 and IJ002)) for testing. For subjective evaluation, we randomly choose 100 samples in test set. To alleviate the mispronunciation problem, convert the text sequence into the phoneme sequence (Arik et al., 2017 Wang et al., [2017; Shen et al.) 2018 Sun al. 2019) with an open-source grapheme-to-phonene into mel mel--peetrograms following Shen et al. (2018) and set frame size and hop size to 1024 and 256 with respect to the sample rate 22050.
too We transform the raw waveform
Model Configuration Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al. 2019) in the encoder and the mel-spectrogram decoder. The output linear layer in the decoder converts the hidden states into 80-dimensional mee-sspectoggsmss and our model is optimized with mean absolute error (MAE). We add more detailed confi ooniiuurations of FastSpeech and 2s used in our experiments in Appendix A The details of training and inference are added in Appendix B
3.2
RESULTS
Method
MOS
GT
4,30 0.07
Method
CMOS
GT (Mel + PWG)
3.92 0.08
FastSpeech 2
0.000
Tacotron 2 Shen al. 2018 (Mel PWG)
3.70 0.08
Transformer TTS (Li et al.
20/9 (Mel + PWG)
3.72 0.07
FastSpeech
-0.885
Transformer TTS
-0.235
FastSpeech Ren al.
2019)
(Mel + PWG)
3.68 0.09
FastSpeech 2 (Mel PWG)
3.83 0.08
FastSpeech
3.71 0.09
(a) The MOS 95% confidence intervals.
(b) CMOS comparison.
Table 1: Audio quality comparison.
In this section, first evaluate the audio quality, training and inference speedup of FastSpeech and 2s. Then we conduct analyses and ablation studies of our method
3.2. MODEL PERFORMANCE
Audio Quality To evaluate the perceptual quality, we perform mean opinion score (MOS) (Chu & Peng 2006) evaluation on the test set. Twenty native English speakers are asked to make quality
htttps://tt thub. hatlbe..oo///an ccom/kyybyoogggggg
We put some audio samples in the supplementary materials and github. Coanub.o/faae /oa.ub..o//aattspecchh...

---

Method
Training Time (h)
Inference Speed (RTF)
Inference Speedup
Transformer TTS (Li et al. 2019)
38.64
9.32 x 10
FastSpeech (Ren et al. 2019)
53.12
1.92 x 10-2
48.5x
FastSpeech 2
17.02
1.95 10
47.8x
FastSpeech 2s
92.18
1.80 x 10
51.8x
Table 2: The comparison of training time and inference latency in waveform synthesis. The training time of FastSpeech includes teacher and student training. RTF denotes the real-time factor, that the time (in seconds) required for the system synthesize one second waveform. The training and inference latency tests are conducted on a server with 36 Intel Xeon CPUs, 256GB memory, NVIDIA V100 GPU and batch size of 48 for training and for inference. Besides, we do not include the time of GPU memory garbage collection and transferring input and output data between the CPU and the GPU. The speedup in waveform synthesis for FastSpeech is larger than that reported in Ren et al. 2019) since we use Parallel WaveGAN as the vocoder which is much faster than WaveGlow.
judgments about the synthesized speech samples. The text content keeps consistent among different systems so that all testers only examine the audio quality without other interference factors. We compare the MOS of the audio samples generated by FastSpeech 2 and FastSpeech 2x with other systems, including 1) GT, the ground-truth recordings; 2) GT (Mel + PWG), where we first con- vert the ground-trun trrth audio into mel-spectrogramss and then convert the mel-spectrograms back to audio using Parallel WaveGAN (Yamamoto et al. (2020) (PWG); 3) Tacotron (Shen et al. 2018) (Mel PWG); 4) Transformer TT' (Li et al. 2019) (Mel PWG); 5) FastSpeech (Ren et al. 2079) (Mel + PWG). All the systems in 3), 4) and 5) use Parallel WaveGAN as the vocoder for a fair comparison. The results are shown in Table [ It can be seen that FastSpeech 2 can surpass and FastSpeech 2s can match the voice quality of autoregressive models Transformer TTS and Tacotron Importantly, FastSpeech outperforms FastSpeech, which demonstrates the effectiveness of pro- viding variance information such as pitch, energy and more accurate duration and directly taking ground-truth speech as training target without using teacher-student distillation pipeline.
Training and Inference Speedup FastSpeech 2 simplifies the training pipeline of FastSpeech by removing the teacher-student distillation process, and thus reduces the training time. We list the total training time of Transformer TTS (the autoregressive teacher model), FastSpeech (including the training of Transformer TTS teacher model and FastSpeech student model) and FastSpeech 2 in Table 2 can be seen that FastSpeech 2 reduces the total training time by 3.12x compared with Peeth. Note that training time here only includes acoustic model training, without considering the vocoder training. Therefore, we do not compare the training time of FastSpeech 2s here. We then evaluate the inference latency of FastSpeech 2 and 2s compared with the autoregressive Transformer TTS model, which has the similar number of model parameters with FastSpeech 2 and 2s. We show the inference speedup for waveform generation in Table 2 It can be seen that compared with the Transformer TTS model, FastSpeech 2 and 2s speeds up the audio generation by 47.8x and 51.8x respectively in waveform synthesis. We can also see that FastSpeech 2s is faster than FastSpeech due to fully end-to-end generation.
3.2.2 ANALYSES ON VARIANCE INFORMATION
Method
7
K
DTW
GT
54.4
0.836
0.977
Tacotron 2
44.1
1.28
1.311
26.32
TransformerTTS
40.8
0.703
1.419
24.40
FastSpeech
50.8
0.724
-0.041
24.89
FastSpeech
54.1
0.881
0.996
24.39
FastSpeech 2 CWT
42.3
0.771
1.115
25.13
FastSpeech 2s
53.9
0.872
0.998
24.37
Table 3: Standard deviation (σ), skewness (7), kurtosis (K) and average DTW distances (DTW) of pitch in ground-truth and synthesized audio.
7

---

More Accurate Variance Information in Synthesized Speech In the paragraph, we measure if providing more variance information (e.g., pitch and energy) as input in FastSpeech and 2s can indeed synthesize speech with more accurate pitch and energy.
For pitch, we compute the moments (standard deviation (σ), skewness (?) and kurtosis ((C) (An- dreeva et al. 2014; Niebuhr & Skarmitzl 2019) and average dynamic time warping (DTW) Müller (2007) distance of the pitch distribution for the ground-truth speech and synthesized speech. The results are shown in Table 3 It can be seen that compared with FastSpeech, the moments (a, Y and K.) of generated audio of FastSpeech 2/2s are more close to the ground-truth audio and the av- erage DTW distances the ground-truth pitch are smaller than other methods, demonstrating that FastSpeech 2/2s can generate speech with more natural pitch contour (which can result in better prosody) than FastSpeech. We also conduct case study on generated pitch contours Appendix
D
Method
FastSpeech
FastSpeech 2
FaxtSpeech 2x
MAE
0.142
0.131
0.133
Table The mean absolute error (MAE) of the energy in synthesized speech audio.
For energy, we compute the mean absolute error (MAE) between the frame-wise energy extracted from the generated waveform and the ground-truth speech. To ensure that the numbers of frames in the synthesized and ground-truth speech are the same, we use the ground-truth duration extracted by MFA in both FastSpeech and FastSpeech 2. The results are shown in Table We can see that the MAE of the energy for FastSpeech 2/2s are smaller than that for FastSpeech, indicating that they both synthesize speech audio with more similar energy to the ground-truth audio.
More Accurate Duration for Model Training We then analyze the accuracy of the provided duration information train the duration predictor and the effectiveness of more accurate duration for better voice quality based on FastSpeech. We manually align 50 audio generated by the teacher model and the corresponding text in phoneme level and get the ground-truth phoneme-leve duration. We compute the average of absolute phoneme boundary differences (McAuliffe et al. (017)) using the duration from the teacher model of FastSpeech and from MFA as used in this paper respectively. The results are shown in Table 5a] We can see that MFA can generate more accurate duration than the teacher model of FastSpeech. Next, we replace the duration used in FastSpeech (from teacher model) with that extracted by MFA, and conduct the CMOS (Loizou] 2011) test to compare the voice quality between the two FastSpeech models trained with different durations The results are listed in Table and and it can be seen that more accurate duration information improves the voice quality of FastSpeech, which verifies the effectiveness of our improved duration from MFA.
Method
A (ms)
Setting
CMOS
Duration from teacher model Duration from MFA
19.68
FastSpeech Duration from teacher
0
12.47
FastSpeech + Duration from MFA
+0. +0.195
(a) Alignment accuracy comparison.
(b) CMOS comparison.
Table 5: The comparison of the duration from teacher model and MFA. A means the average of absolute boundary differences.
3.2.3 ABLATION STUDY
Pitch and Energy Input We conduct ablation studies to demonstrate the effectiveness of several variance information of FastSpeech and 2s, including pitch and energy We conduct CMOS eval- uation for these ablation studies. The results are shown in Table We find that removing the energy (Row in both subtables) in FastSpeech 2 and 2s results in performance drop in terms of voice quality (-0.040 and 160 CMOS respectively), indicating that energy is effective for FastSpeech 2 in improving the voice quality, and more effective for FastSpeech 2s. We also find that removing the
7 Both models are trained with mel-spectrograms generated hy the teacher model. We do not study duration information since duration is necessary for FastSpeech and FastSpeech 2. Besides, have already analyzed the effectiveness of our improved duration in the last paragraph.
8

---

pitch (Row 4 in both subtables) in FastSpeech 2 and 2s results in -0.245 and -1.130 CMOS respec- tively, which demonstrates the effectiveness of pitch. When we remove both pitch and energy (the last row in both subtables), the voice quality further drops, indicating that both pitch and energy can help improve the performance of FastSpeech and 2s.
Predicting Pitch in Frequency Domain To study the effectiveness of predicting pitch in fre- quency domain using continuous wavelet transform (CWT) as described in Section 2.3, we directly fit the pitch contour with mean square error like energy in FastSpeech and 2s. We conduct CMOS evaluation and get CMOS drops of 0.185 and 0.201 I for FastSpeech 2 and 2s respectively. We also compute the moments of pitch and average DTW distance to the ground-trun pitch as shown row 6 (denoeted as FastSpeech 2 CWT) in Table 3 The results demonstrate that CWT can help model the pitch better and improve the prosody of synthesized speech, and thus obtaining better CMOS score.
To verify the effectiveness of the mel-spectrogram decoder in FastSpeech 2s on text feature extraction as described in Section 2.4 we remove the mel-spectrogram decoder and conduct CMOS evaluation. causes a 0.285 CMOS drop, which demonstrates that the mel-spectrogram decoder is essential to high-quality waveform generation.
Mec--Spectrogram Decoder in FastSpeech 2s
Setting
CMOS
Setting
CMOS
FastSpeech 2
FastSpeech 2s
FastSpeech energy
-0.040
FastSpeech 2x energy
-0.160
FastSpeech 2 pitch
-0.245
FastSpeech 2s pitch
-1.130
FastSpeech 2 pitch energy
-0.370
FastSpeech 2s pitch energy
-1.355
(a) CMOS comparison for FastSpeech 2.
(h) CMOS comparison for FastSpeoch 2s.
Table CMOS comparison in the ablation studies.
CONCLUSION
In this work, we proposed FastSpeech fast and high-quality end-to-end TTS system, address the issues in FastSpeech and ease the one-to-many mapping problem: 1) we directly train the model with ground-trute -tunh mel- -pesspectsogs to simplify the training pipeline and also avoid information loss compared with FastSpeech; and 2) we improve the duration accuracy and introduce more vari- ance information including pitch and energy to ease the one-to-many mapping problem, and improve pitch prediction by introducing continuous wavelet transform. Moreover, based on FastSpeech 2, we further developed FastSpeech 2s, non-autoregressive eex-to-wavefoore generation model, which enjoys the benefit of fully end-to-end inference and achieves faster inference speed. Our experi- mental results show that FastSpeech and 2s outperform FastSpeech, and FastSpeech 2 can even surpass autoregressive models in terms of voice quality, with much simpler training pipeline while inheriting the advantages of fast, robust and controllable speech synthesis of FastSpeech.
High quality, fast and fully end-to-end training without any external libraries is definitely the ulti- mate goal of neural TTS and also a very challenging problem. To ensure high quality of FastSpeech 2, we an external perf-peeformance alignment tool and pitch extraction tools, which may seem a little complicated, but are very helpful for high-quality and fast speech synthesis. We believe there will be more simpler solutions to achieve this goal the future and we will certainly work on fully end-to-end TTS without external alignment models and tools. We will also consider more variance information (Zhang al. 2021 further improve the voice quality and speed up the inference with more
model [uo et al. 2021).
ACKNOWLEDGMENTS
This work was supported in part by the National Key R&D Program of China (Grant No.2018AAA01000003, National Natural Science Foundation of China (Grant No.62072397). Zhe- jiang Natural Science Foundation (Grant No.LR19F020006, National Natural Science Foundation of China (Grant No.6 61 (1836002) and X Lab, the Second Academy of CASIC, Beijing, 100854, China.

---

REFERENCES
Bistra Andreeva, Gražyna Demenko, Bernd Möbius, Frank Zimmerer, Jeanin Jügler, and Magdalena Oleskowicz-Popiel. Differences of pitch profiles in germanic and slavic languages. In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
Sercan O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. arXiv preprint arXiv: 000..7702277..1 2017.
Mingjian Chen, Xu Tan, Yi Ren, Jin Xu, Hao Sun, Sheng Zhao, and Tao Qin. Multispeech: Multi- speaker text to speech with transformer. INTERSPEECH, pp. 4024-4028, 2020.
Mingjian Chen, Xu Tan, Bohan Li, Yanqing Liu, Tao Qin, sheng zhao, and Tie- Liu. Adaspeech: Adaptive text to speech for custom voice. In International Conference Learning Representa- tions, 2021. URL
.nt httt//oormmnnddyyyttttt
Min Chu and Hu Peng. Objective measure for estimating mean opinion score of synthesized speech, April 4 2006. US Patent 7,024.362.
Jeff Donahue, Sander Dieleman, Mikotaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. arXiv preprint arXiv:2006.03575, 2020.
Jesse Engel, Lamtharn Hantrakul, Chenjje Gu, and Adam Roberts. Ddsp: Differentiable digital signal processing. arXiv preprinl arXiv:2001.04460 2020.
Yuchen Fan, Yao Qian, Feng-Long Xie, and Frank Soong. Tts synthesis with bidirectional Istm based recurrent neural networks. In Fifteenth Anmual Conference of the International Speech
Communication Association, 2014.
Michael Gadermayr, Maximilian Tschuchnig, Dorit Merhof, Nils Krämer, Daniel Truhn, and Burkhard Gess. An asymetric cycle-consistency loss for dealing with many-to-one mappings in image translation: A study on thigh mr seans. arXiv preprint arXiv:2004. 1001, 2020.
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping. Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In Advances in neural information processing systems, pp. 2962-2970, 2017.
Alexander Grossmann and Jean Morlet. Decomposition of hardy functions into square integrable wavelets of constant shape. SIAM journal on mathematical analysis, 15(4):723-736, 1984.
Keikichi Hirose and Jianhua Tao. Speech Prosody in Speech Synthesis: Modeling and generation of prosody for high quality and flexible speech synthesis. Springer, 2015.
Keith Ito. The lj speech dataset.
LJ- Lo--eeeecccc-S-cchcchhhc---- Speech-Dataset) / 2017.
Chrisina Jayne, Andreas Lanitis, and Chris Christodoulou. One-to-many neural network mapping techniques for face image synthesis. Expert Systems with Applications, 39(10)9778---787 2012.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: generative flow for text-to-speech via monotonic alignment search. arXiv preprint arXiv:2005.1120 129, 2020.
Sungwon Kim, Sang-gi gil Lee, Jongyoon Song, Jaehyeon Kim, and Sungroh Yoon. Flowavenet: A generative flow for raw audio. arXiv preprint 11Xiv:1811.02111 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing
Systems, pp. 14881-14892, 2019.
Adrian Lańcucki. Fastpitch: Parallel text-to-speech with pitch prediction. arXiv preprint arXiv:2006.0687. 2020.
10

---

Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with trans- former network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 6706-677 6711, 2019.
Dan Lim, Won Jang, Hyeyeong Park, Bongwan Kim, Jesam Yoon, et al. Jdi-t: Jointly trained duration informed transformer for text-to-speech without explicit alignment. arXiv preprint arXiv:2005.07799, 2020.
Philipos C Loizou. Speech quality assessment. In Multimedia analysis, processing and communi- cations, pp. 623-654. Springer, 2011.
Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Jinzhu Li, Sheng Zhao, Enhong Chen, and Tie- Yan Liu. Lightspeech: Lightweight and fast text to speech with neural architecture search. In 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, pp. 498- 502, 2017.
Chenfeng Miao, Shuang Liang, Minchuan Chen, Jun Ma. Shaojun Wang, and Jing Xiao. Flow- tts: A non-autoregressive network for text to speech based on flow. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7209-7213. IEEE, 2020.
Huaiping Ming, Dongyan Huang, Lei Xie, Jie Wu, Minghui Dong, and Haizhou Li. Deep bidirec- tional Istm modeling of timbre and prosody for emotional voice conversion. 2016.
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Oliver Niebuhr and Radek Skamittz.. Measuring a speaker's acoustic correlates of pitch- which? contrastive analysis based on perceived speaker charisma. In Proceedings of 19th International Congress of Phonetic Sciences, 2019.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, al.
Parallel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:1711.1043: 2017.
Kainan Peng, Wei Ping. Zhao Song, and Kexin Zhao. Parallel neural text-to-speech. arXiv preprint 1rXiv:1905.08459, 2019.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference on Learning Representations, 2018.
Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to- speech. International Conference on Learning Representations, 2019.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEEE, 2019.
Yi Ren, Yangiun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Sys- tems, pp. 3165-3174, 2019.
Harold Ryan. Ricker, ormsby: klander, bntterwo-a choice of wavelets, 1994.
Jonathan Shen, Ruoming Pang. Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural tts synthesis by con- ditioning wavenet on mel spectrogram predictions. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4779-4783. IEFE, 2018.

---

Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie- Yan Liu. Token-level ensemble distillation for grapheme-to-phoneme to-phoneme conversion. In INTERSPEECH, 2019.
Antti Santeri Suni, Daniel Aalto, Tuomo Raitio, Paavo Alku, Martti Vainio, et al. Wavelets for intonation modeling in hmm speech synthesis. In 8th ISCA Workshop Speech Synthesis, Pro- ceedings, Barcelona, August 31 31-September 2, 2013. ISCA, 2013.
Franz B Tuteur. Wavelet transformations in signal detection. IFAC Proceedings Volumes, 21(9):
1061-1065, 1988.
Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW, 125, 2016.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelenn decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Yuxuan Wang. RJ Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv: 10135, 2017.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199-6203. IEEE, 2020.
Heiga Ze, Andrew Senior, and Mike Schuster. Statistical parametric speech synthesis using deep neural networks. In 2013 ieee international conference on acoustics, speech and signal process- ing, pp. 7962-7966. IEEE, 2013.
Zhen Zeng, Jianzong Wang, Ning Cheng, Tian Xia, and Jing Xiao. Aligntts: Efficient feed-forward text-to-speech system without explicit aligmment. In ICASSP 2020-2020 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6714 IEEE, 2020.
Chen Zhang, Yi Ren, Xu Tan, Jinglin Liu, Kejun Zhang, Tao Qin, Sheng Zhao, and Tie- Yan Liu. De- noispeech: Denoising text to speech with frame-level noise modeling. In 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.
Yun Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in neural information processing systems, pp. 465-476, 2017.
12

---

A MODEL CONHIGURATION
Our FastSpeech 2 consists of 4 feed-forward Transformer (FFT) blocks (Ren et al. 2019) in the en- coder and the mel-spectrogram decoder. In each FFT block, the dimension of phoneme embeddings and the hidden size of the self-attention are set to 256. The number of attention heads is set to 2 and the kernel sizes of the ID-convolution in the 2-layer convolutional network after the self-attention layer are set to 9 and 1, with input/output size of 256/1024 for the first layer and 1024/256 in the second layer. The size of the phoneme vocabulary is 76, including punctuations. In the variance set to 3, with input/output sizes of 256/256 for
predictor, the kernel sizes of the Io-onvolutioon both layers and the dropout rate is set to 0.5. Our waveform decoder consists of 1-layer transposed ID-convolution with filter size 64 and 30 dilated residual convolution blocks, whose skip channel size and kernel size of ID-convolution are set to 64 and 3. The configurations of the discriminator in FastSpeech 28 are the same Parallel WaveGAN (Yamamoto al. 2020 We list hyperparameters
and contigurations of all models used in our experiments in Table 7
Hyperparameter
Transformer TTS
FastSpecch/FastSpeehh 2/2s
Phoneme Embedding Dimension
256
256
Pre-net Layers
3
Pre-net Hidden
256
Encoder Layers
4
4
Encoder Hidden
256
256
Encoder Conv ID Kernel
9
9
Encoder Conv ID Filter Size
1024
1024
Encoder Attention Heads
2
2
Mel-Spectrogram Decoder Layers
4
Mel-Spectrogram Decoder Hidden
256
256
Mel-Spectrogram Decoder ConvID Keree
Spee-rpectrormm Decoder ConvID Filter Size
1024
1024
Mel-Spectrogram Decoder Attention Headers
2
2
Encoder/Decoder Dropout
0.1
0.1
Variance Predictor Conv ID Kernel
3
Variance Predictor CD Filter Size
256
Variance Predictor Dropout
0.5
Waveform Decoder Convolution Blocks
30
Waveform Decoder Dilated Cony ID Kernel size
3
Waveform Decoder Transposed Conv ID Filter Size
Wavelorm Decoder Skip Channlel Size
64
Batch Size
48
48/48/12
Total Number of Parameters
24M
23M/27M/28M
Table 7: Hyperparameters of Transformer TTS, FastSpeech and FastSpeech 2/2s.
B TRAINING AND INFERENCE
We train FastSpeech 2 on NVIDIA V100 GPU, with batchsize of 48 sentences. We use the Adam
optimizer (Kingma & Ba 2014 with 3 = 0.9, B2 = 0.98, 10-9 and follow the same learning rate schedule in Vaswani et al. (017) It takes 160k steps for training until convergence. In the inference process, the output mel-spectrograms of our FastSpeech 2 are transformed into audio samples using pre-trained Parallel WaveGAN (Yamamoto et al. | For FastSpeech 2s, we train the model on NVIDIA V100 GPUs, with batchsize of 6 sentences on each GPU. The waveform decoder takes the sliced hidden states corresponding to 20,480 waveform sample clips as input. The optimizer and learning rate schedule for FastSpeech 2s are the same as FastSpeech The details of the adversarial training follow Parallel WaveGAN (Yamamoto et al. 2020 It takes 600k steps for training until convergence for FastSpeech 2s.
WaveGAN
13

---

C MODELING PITCH WITH CONTINUOUS WAVELET TRANSFORM
C.I CONTINUOUS WAVELET TRANSFORM
Given a continous pitch contour function Fa. we can convert it to pitch spectrogram WT,, using
continuous wavelet transform (Tueerr 1988 Grossmann & Morlet 1984):
toc
W = 1//2
Fo(I)v( da
where yi is the Mexican hat mother wavelet (Ryan 1994), Fo(x)) is the pitch value in position x; and are scale and position of wavelet respectively. The original pitch contour Fo can be recovered from the wavelet representation (TT, L) by inverse continuous wavelet transform (iCWT) using the following formula:
+00
+00
15-1) 7
Fo(t)
W (T, r 5/2
dndr
Suppose that we decompose the pitch contour Fo into 10 scales (Ming et al. 2016 Fo can be represented by 10 separate components given by:
(ttt) = W(2 To, t)
2.5)
(1)
where i = 1,... 10 and To 5mss which is originally proposed in Suni et al. (2013). Given 10 wavelet components Wi(1), we can recompose pitch contour F6 by the following formula (Ming
et al. 2016):
10
Po(1) = Σ (((((i 2.5) 5/2
(2)
C.2 IMPLEMENTATION DETAILS
First extract the pitch contour using Woooodd Since CWT is very sensitive to discontinuous signals, we preprocess the pitch con- tour follows: 1) use linear interpolation to fill the unvoiced frame in pitch contour; 2) we transform the resulting pitch contour to loga- rithmic scale; 3) we normalize it to zero mean and unit variance for each utterance, and we have to save the original utterance-level mean and vari- ance for pitch contour reconstruction; and 4) we convert the normalized pitch contour to pitch spectrogram using continuous wavelet transform following Equation] I
10
+
Predictor
Pitch Contour
CWT
JCWT
Pitch Spectrogram, Mean/Var
As shown in Figure pitch predictor consists of 2-layer ID- convolutional network with ReLU activation, each followed by the layer normalization and the dropout layer, and extra linear layer to project the hidden states into the pitch spectrogram. To predict the mean/variance of recovered pitch contour for each utterance, we average the hidden states output by the ID-convolutional network on the time dimension to global vector and project it to mean and variance using linear layer.
Pitch Prodictor
Duration
Duration Predictor
Figure 2: Details in pitch predictor. CWT and iCWT denote con- tinuous wavelet trans- form and inverse contin- uous wavelet transform respectively.
We train the pitch predictor with ground-truth pitch spectrogram and the mean/variance of pitch contour and optimize with mean square error. During inference, we predict the pitch spectrogram and the mean/variance of recovered pitch contour using pitch predictor, inverse the pitch spectrogram to pitch contour with inverse continuous wavelet transform (iCWT) following Equation 2 and finally denormalize it with the predicted mean/variance.
14

---

D CASE STUDY ON PITCH CONTOUR
In this section, we conduct the case study on pitch contours the audios generated by different methods. We randomly choose I utterance from the test set and plot the pitch countor of ground- truth audio samples and that generated by FastSpeech, FastSpeech 2, FastSpeech 2s in Figure We can see that FastSpeech 2 and 2s can capture the variations pitch better than FastSpeech thanks to taking pitch information as input.
300
Time (ftan))
300
|4,1+
250
230
4
100
200
200
M
(1
1he
100
200
300
500
600
100
200
300
400
500
00.0
700
(a) Ground Truth
(b) FastSpeech
400
350
M
350
300
Tis 0tamo)
300
4
hn W
MWN
250
250
hu41 /
100
200
300
400
300
600
700
100
200
500
4.00
500
600
700
(c) FastSpeech
(d) FastSpeech
Figure 3: Pitch contours extracted from generated and ground-truth audio samples. We only plot the voiced part of pitch contour. The input text is "The worst, which perhaps was the English, was terrible falling-off from the work of the earlier presses".
E VARIANCE CONTROL
FastSpeech and 2s introduce several variance information to ease the one-to-many mapping prob- lem in TTS. As byproduct, they also make the synthesized speech more controllable and can be used to manually control pitch, duration and energy (volume) of synthesized audio. As a demon- stration, manipulate pitch input control the pitch of synthesized speech in this subsubsection. We show the mel-spectrograms before and after the pitch manipulation Figure From the sam- ples, we can see that FastSpeech 2 generates high-quality mel-spectrograms after adjusting the Po from 0.75 1.50 times. Such manipulation can also be applied FastSpeech 2s and the results are put in the supplementary materials. We also put the audio samples controlled by other variance information supplementary materials.
(a) Fe Fi
(b) fo 0.75F
(c) F. 1.50.
Figure 4: The mel-spectrograms the voice with different fo. Fo is the fundamental frequency of original audio. The red curves denote Fg contours. The input text is "They discarded this for more completely Roman and far less beautiful letter."
15


Reviews:
[{"review_id": "jYLxKet3uYl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nIn FastSpeech 2 and 2s, the authors propose several changes to the non-autoregressive FastSpeech model. An autoregressive teacher model was used in FastSpeech to alleviate the one-to-many problem of TTS by providing phoneme durations estimated by attention mechanism, and predicted mel-spectrograms as soft targets to the final non-autoregressive model. In FastSpeech 2, the authors do away with the teacher-student knowledge distillation pipeline and instead utilize the following to alleviate the one-to-many problem while retaining a non-autoregressive architecture:\n- ground-truth mel spectrogram is used as the training targets\n- a force alignment tool is used to extract more accurate phoneme durations\n- pitch and energy information is also used as conditioning for the model as a means to introduce variance information\n\nThese changes boost the speech quality scores of the FastSpeech 2 architecture over autoregressive TTS architectures while reducing both training and inference times considerably. The authors also introduce an end-to-end text-to-waveform variant of the model: FastSpeech 2s. This model leverages the mel-spectrogram decoder, adversarial training, and a wavenet-like architecture to enable waveform prediction at a shorter time-scale. The Fastspeech 2s architecture performs at par with autoregressive TTS models. The authors further perform ablation studies that demonstrate the importance of utilizing accurate duration, pitch and energy information.\n\nPositives:\n1. The authors explain the model architecture clearly and perform thorough evaluation and ablation studies.\n2. The changes in FastSpeech lead to improvements in performance both in terms of speech quality and time (training and inference). The quality of the samples provided is pretty good and since the models having only been trained with a publicly available dataset which is not unreasonably large shows great potential.\n\nNegatives:\n1. The authors make several claims that need substantiating. Below are a few:\n    - 2.3: The authors claim that pitch is the key feature to convey emotion and affect speech prosody while energy affects speech volume. This is not entirely correct. Emotion can be conveyed by variation in pitch as well as energy and the contribution of each of these features towards speech prosody cannot be so easily disentangled. I would recommend adding that energy is also an important factor in speech prosody. \n    - footnote 3: The authors mention that energy is not as highly variable as speech. This may be an artifact of the dataset being utilized which consists only of neutral speech. This could be included in the text as well.\n    - 3.2.2: The authors claim that since the distribution of pitch extracted from FastSpeech 2 synthesized speech is closer to groung truth distribution, the pitch contours are more natural. This claim is easier made if a comparison of time-series pitch contours is made. Including a simple DTW distance would strengthen the argument. Additionally, there are metrics proposed in prosody transfer literature[1] that can be used here.\n2. FastSpeech 2 method section is not very clearly explained. While it is understandable that there are space constraints, each component requires at least one sentence explaining its purpose and how it works. The paragraph lacks any detail of the adversarial training method and how phase retrieval happens.\n\nMinor edits:\n\"challenges lie in\" -> challenges in\n\nReferences:\n[1] Skerry-Ryan, R. J., et al. \"Towards end-to-end prosody transfer for expressive speech synthesis with tacotron.\" arXiv preprint arXiv:1803.09047 (2018).", "extra_scores": {}, "extra_text": {}}, {"review_id": "dYqV0Nuxr3B", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is motivated by, and intends to address shortcomings of, the previously published FastSpeech paper, specifically problems with the internal duration prediction of that model, and its requirement for a teacher-student configuration. The submission makes clear what the problems with that original approach are, and the evaluation later confirms this. What is missing from this work, however, is a more candid statement about the trade-offs between the FastSpeech and FastSpeech2 architectures since the current implementation requires externally aligned datasets, or the resources to develop them.  \n\n\nIn other words, although the authors present this paper as a natural and organic extension of the original FastSpeech work, one could argue that in some ways it represents a step backwards with respect to the spirit of sequence-to-sequence models that solve the alignment problem without the need for external supervision. One may very well have the resources to afford such supervision, so there's definitely a place for this type of model, but the paper should be more forthcoming in the pros and cons of this new approach.\n\nHaving a duration-prediction model explicitly trained with externally-determined duration targets, in fact, brings this work closer to the more \"classical\" earlier-generation pipeline neural architectures (pre seq-to-seq), with the notable difference that everything here is trained jointly. I think the paper would be well served by a broader review of the literature that tries to situate it beyond its similarity with FastSpeech only. Some relevant parts of the literature seem uncited (I've included examples below). If space is an issue, some of the material in the Appendix could be shortened, or just referenced.\n\n\nThe paper is generally well written and easy to follow. Some comments and needed clarifications follow:\n\nThe paper says that the variance adaptor \"adds\" different variance information. Do you literally mean that you sum up the different embeddings (the use of a + in Fig. 1b would suggest as much), or do you concatenate/stack them? If it's the former, try to motivate this choice since summing up linguistic embeddings with acoustic embeddings doesn't necessarily seem an obvious choice.\n\nSec. 2.3 --  *Previous neural network based TTS systems with pitch prediction (Arik et al.,2017; Gibiansky et al., 2017) often predict pitch contour directly.* -- There is, in fact, an extensive (and relevant, but uncited) literature on direct neural prediction of f0 with external f0 targets as the authors do here. See, e.g,:\n- Statistical Parametric Speech Synthesis Using Deep Neural Networks (Zen et al., 2013)\n- Prosody Contour Prediction with Long Short-Term Memory, Bi-Directional, Deep Recurrent Neural Networks (Fernandez et al., 2014)\n- TTS Synthesis with Bidirectional LSTM based Recurrent Neural Networks (Fan et al., 2014)\n\nThe idea of conditioning seq-2-seq models on some acoustic properties of the output (in the spirit of the variance adaptor of FastSpeech2) is also not new, and has been proposed to either improve control or stability of such models. See. e.g.,\n\n- Sequence to Sequence Neural Speech Synthesis with Prosody Modification Capabilities (Shechtman et al., 2019)\nand more recently:\n-  Controllable Neural Text-to-Speech Synthesis Using Intuitive Prosodic Features (Raitio et al., 2020)\n\n\n Sec. 2.5: *Previous non-autoregressive vocoders (Oord et al., 2017; Prenger et al., 2019; Yamamoto et al.,2020; Kumar et al., 2019) are not complete text-to-speech systems, since they convert time aligned linguistic features to waveforms, and require a separate linguistic model to convert input text to linguistic features or an acoustic model to convert input text to acoustic features (e.g., mel-spectrograms). FastSpeech 2s is the first attempt to directly generate waveform from phoneme sequence, instead of linguistic features or mel-spectrograms.* -- A phoneme sequence is a type of linguistic information that needs to be extracted from the input text. It's hard to argue that something that requires a G2P is more of a full TTS system. I assumed that the input text was also normalized prior to the G2P, so by starting from a phoneme sequence, you're assuming access to both a normalizer and a G2P.\n\n\nEvaluation; What type of sentences were used in the evaluation? The paper mentions that the training of FastSpeech2s required using shorter clips to deal with memory issues. Do you restrict your listening test to also contain only simpler, shorter sentences? Systems that perform close to each other when synthesizing shorter prompts can behave very differently when attempting more complex inputs (questions; lists; sentences with syntactic embedding or relative clauses; etc.)\n \nThough the numbers cluster nicely, consider adding parwise significance tests to the results in Table 1.\n\nThe analysis/resynthesis condition (GT(Mel+PWG)) shows a drop of 0.38 MOS points with respect to the ground truth (4.3 vs 3.92). Can you comment on this? It seems large.\n\nSec. 2.2.3. Ablation studies -- Any reason why you didn't include similar tables to Table 6 for the other studies (2nd and 3rd paragraph in this section)?\nMinor edits:\n\n- since waveform contains --> since the waveform contains\n- discuss the challenges lie --> discuss the challenges that lie\n- pitch predictor consists --> the pitch predictor consists\n- A few acronyms incorrectly capitalized in the bibliography (tts --> TTS / flowavenet --> FloWaveNet / etc.)\n- It's diffcult to make sense out of the Ryan, 1994 reference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MqwwiU_a9sV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents several improvements over the original teacher-student framework in FastSpeech: 1) training the model with ground-truth target instead of the output from teacher, 2) extract phoneme duration, pitch and energy from speech and directly take them as conditional inputs in training and use predicted values in inference.  Importantly, it uses pretrained forced aligner (MFA) to extract the phoneme durations for training.  \n\nComments:\n\n1, Typo: In introduction, \"alleviate the one-to mapping\"\n\n2, The \"one-to-many mapping\" is not an issue in general. This setting widely exists in generative modeling, e.g., label conditioned image synthesis, mel spectrogram conditioned waveform synthesis. The problem of FastSpeech (and other non-autoregressive TTS models) is really due to the over-simplified output distributions, which assume conditional independence between frames and frequency bins for mel spectrogram. As a result, these models doesn't account for the variations in real data.\n\n3, The introduction, section 2.1, and section 2.2 contain too much duplicated information. One may shorten the text properly.\n\n4, The motivation & architecture of FastSpeech 2s (Figure 1 (a)) is similar to the previous text-to-waveform model clarinet. Both of them use mel prediction task to guide the training. The authors overclaim that \"FastSpeech 2s is the first attempt to directly generate waveform from phoneme sequence\".\n\n5, The model is similar to traditional TTS pipeline with separate duration model, pitch/F0 model etc. The CWT/iCWT-based pitch predictor is interesting. \n\n6, The MOS of autoregressive Tacotron 2 is relatively low. Which implementation did you use?\n\n7, One may also report the standard deviation, skewness, and kurtosis of pitch in synthesized audio from autoregressive model (Tacotron 2 and Transformer TTS). I assume their results would be closer to GT.\n\n8, In Table 5, the durations from teacher model are extracted by teacher forcing ground-truth mel spectrogram? One may mention that MFA is pretrained on a much larger dataset, thus may have better generalization than the teacher model trained on small LJSpeech dataset.\n\nPros:\n- Good sample quality.\n- Sufficient ablation study.\n\nCons:\n- The proposed pipeline is far more complicated than existing end-to-end TTS model.\n- Inaccurate & confusing claims (see my comments).\n- The novelty is rather limited. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "6uD_wDZlDBl", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper further extends the non-autoregressive end-to-end TTS model \"FastSpeech\".  The new work includes (1) removing the knowledge distillation from an AR model to the non-AR model which was required in FastSpeech, (2) introduced \"variance\" representations such as energy and F0 in addition to duration.  (1) made the training procedure simpler.  (2) improved the naturalness as well as controllability of synthesized speech.\n\nTechnical details and model architecture are available in the body of the paper, whereas additional details are also provided in the appendix.  The model was evaluated in the LJSpeech dataset, which is a publicly available dataset and often used for TTS model evaluations.  Audio samples are also provided in the demo page.\n\nThere are a few parts need to be addressed.  (1) made the training procedure simpler but it also introduces the necessity to rely on external aligner, which actually complicates the whole voice building process.  It is not well discussed in the paper.  (2) also introduces the dependency to external modules, such as F0 extractor and makes the model less \"end-to-end\".  The dependency to an external F0 extractor can introduce errors, as F0 extractors sometimes produce errors such as pitch halving/doubling.  The authors should also discuss it from this point.\n\nComments:\n- For pitch and energy 256-bin quantization was used.  Did the author observe any issue if continuous values were directly used?\n- It is unclear whether there is any difference between FastSpeech 2s waveform decoder and Parallel WaveGAN decoder.\n- Comparison with FastPitch: the authors claimed \"results in better prosody\" but I cannot see any evidence or experimental validation for this claim.  If you have any results, please include it.  Otherwise, this is too strong claim.\n- To demonstrate FastSpeech 2 comparable to AR models, having a side-by-side comparison between them is helpful.\n- Table 3 includes moments of pitch of natural and synthetic speech.  Can you include these numbers of FastSpeech 2 with F0 fit via MSE which you discussed in the ablation study? \n- If I understand correctly pitch mean/var are constant within an utterance, but pitch spectrogram is frame-by-frame.  In Figure 2 both of them are predicted from pitch predictor but the nature of these values are different (global / frame-by-frame).  Please include more explanation how they are predicted in the pitch predictor.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5GQv7o-ecwR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a number of improvements over the non-autoregressive FastSpeech TTS model.\n\nOne of the trends that TTS has seen since Tacotron was announced is a retreat from fully end-to-end modeling for TTS.  FastSpeech2 is another entry in that.  The text input is first converted to phones, and prosodic information can be inferred from text (in a unified system) or can be conditioned by a user.  All of these are valuable improvements for quality and customizability.  \n\nIn particular, FastSpeech takes an interesting modular approach to conditioning on pitch, duration and energy for its prosody component.  It relies on ground truth predictors during training, but learns to infer these via multitask training so they are not required during inference.   The use of a dense and continuous pitch representation as a pitch spectrogram is interesting as well.  This could be a limitation for customizability as a user would be more comfortable to provide a pitch target rather than specifying a pitch spectrogram.\n\nThis paper is clearly written and includes sufficient detail for reproducibility.  The evaluation is convincing especially as it relates to training and inference time.\n\nMinor comment: In Table 2 the FastSpeech2s training time number is omitted.  This is because it is not directly comparable to FastSpeech2 or the other training numbers -- it includes training a neural vocoder while the others do not.  However, since the discussion of training time as a valuable measure is already begun, I would suggest that the authors include this number, with the explanation that it is not comparable. (or include vocoder training time along with the other methods.)\n\nComment: The one-to-many TTS problem --- where one lexical utterance can be produced in many different correct ways -- is only partly addressed here for two reasons. 1) there is variation other than duration, energy and pitch that can vary between realizations of the same utterance, voice quality, background and channel noise, and speaker effects.  2) more significantly, the one-to-many problem is only approached during model training.  During inference, the process is still deterministic given the state of the model.    The inference side of the problem is important for objective model evaluation.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a number of techniques to improve the existing non-autoregressive end-to-end TTS model -- FastSpeech. These techniques include replacing the teacher forcing with ground truth targets and using a variation adaptor to introduce auxiliary information such as duration, energy and pitch.  The experiments show that the proposed Fastspeech 2 model is faster in training  compared to the existing FastSpeech model and meanwhile can still achieve high quality synthesized speech.  The work reported in the paper is essentially about system improvement over FastSpeech but has it value in the speech community given the current interest in non-autoregressive rapid TTS.  On the other hand, concerns are also raised regarding the complexity of the pipeline and the significance of the novelty. The authors' rebuttal is good and has addressed most of the concerns.  Overall, this is an interesting paper and can be accepted.  
Decision: Accept (Poster)