EfficientTTS 2: Variational End-to-End Text-to-Speech Synthesis and Voice Conversion
Authors: Chenfeng Miao
Qingying Zhu
Minchuan Chen
Jun Ma
Shaojun Wang
Jing Xiao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Text-to-Speech
Voice Conversion
End-to-End

Abstract: Text-to-speech (TTS) field is recently dominated by one-stage text-to-waveform models, in which the speech quality is significantly improved compared to two-stage models. However, the best-performing open-sourced one-stage model, the VITS, is not fully differentiable and suffers from relatively high computation costs. To address these issues, we propose EfficientTTS 2 (EFTS2), a fully differentiable end-to-end TTS framework that is highly efficient. Our method adopts an adversarial training process, with a differentiable aligner and a hierarchical-VAE-based waveform generator. The differentiable aligner is built upon the EfficientTTS. A hybrid attention mechanism and a variational alignment predictor are incorporated into our network to improve the expressiveness of the aligner. The use of the hierarchical-VAE-based waveform generator not only alleviates the one-to-many mapping problem in waveform generation but also allows the model to learn hierarchical and explainable latent variables that control different aspects of the generated speech. We also extend EFTS2 to the voice conversion (VC) task and propose EFTS2-VC, an end-to-end VC model that allows efficient and high-quality conversion. Experimental results suggest that the two proposed models match their strong counterparts in speech quality with a faster inference speed and smaller model size. 

Under review as a conference paper at ICLR 2023
EFFICIENTTTS 2: VARIATIONAL END-TO-END TEXT- TO-SPEECH S SYNTHESIS AND VOICE CONVERSION
Anonymous authors Paper under double-blind review
ABSTRACT
Text-to-speech (TTS) field is recently dominated by one-stage text-to- waveform models, in which the speech quality is significantly improved compared to two- stage models. However, the rest-performing open-sourced one-stage model, the VITS (Kim et al. (202T). D. is not fully differentiable and suffers from relatively high computation costs. To address these issues, we propose Efficien[TTS 2 (EFTS2), fully differentiable end-to-end TTS framework that is highly efficient. Our method adopts an adversarial training process, with a differentiable aligner and hierarchical VAE--AA-baeee waveform generator. The differentiable aligner is built upon the EfficientTTS (Miao al. 2021). A hybrid attention mechanism and a variational alignment predictor are incorporated into our network to improve the expressiveness of the aligner. The use of the hierarchical VAE-based wave- form generator not only alleviates the one-to-many mapping problem in waveform generation but also allows the model to learn hierarchical and explainable latent variables that control different aspects of the generated speech. We also extend EFTS2 to the voice conversion (VC) task and propose EFTS2-VC, an end-to-end VC model that allows efficient and high-quality conversion. Experimental results suggest that the two proposed models match their strong counterparts in speech quality with a faster inference speed and smaller model size.
INTRODUCTION
Text-to-speech (TTS) task aims at producing human-like synthetic speech signals from text inputs. In recent years, neural network systems have dominated the TTS field, sparked by the develop- ment of autoregressive (AR) models (Wang et al. 2017, Shen et al., 2018 Ping et al., 2018) and non-autoregressive (NAR) models (Miao et al.) 20200 Ren et al., 2019 2021). The conventional neural TTS systems cascade two separate models: an acoustic model that transforms the input text sequences into acoustic features (e.g. mel-spectrogramm (Wang et al., 2017 Ren et al. 2019), fol- lowed by a neural vocoder that transforms the acoustic Teatures into audio waveforms (Valin & Skoglund| 2019 Yamamoto et al. 2020). Although two-stage TTS systems have demonstrated the capability producing human-lite like speech, these systems come with several disadvantages. First of all, the acoustic model and the neural vocoder cannot be optimized jointly, which often hurts the quality of the generated speech. Moreover, the separate training pipeline not only complicates the training and deployment but also makes difficult for modeling downstream tasks.
Recently, in the TTS field, there is a growing interest in developing one-stage text-to-wavefome models that can be trained without the need for mel-spectrograms (Weiss et al. [222] I Donahue al.)
2021 Kim et al. 202T). Among all the open-sourced text-to-waveform models, VITS (Kim et al., 2021 achieves the best model performance and efficiency. However, it still has some drawbacks. Firstly, the MAS method (Kim et al. (020) used to learn sequence alignment in VITS is precluded in the standard pnok-propagttion process, thus affecting training efficiency. Secondly, in order to generate a time-aligned textual representation, VITS simply repeats each hidden text representation by its corresponding duration. This repetition operation is non-differentiable thus hurting the quality of generated speech. Thirdly, VITS utilizes bijective transformations, specifically affine coupling layers, to compute latent representations. However, for affine coupling layers, only half of the input data gets updated after each transformation. Therefore, one has to stack multiple affine coupling layers to generate meaningful latent representations, which increases the model size and further reduces the model's efficiency. A recent work NaturalSpeech (Tan al. 2022) improves upon

---

Under review as a conference paper at ICLR 2023
VITS by leveraging learnable differentiable aligner and bidirectional prior/posterior module. However, the training of the learnable differentiable aligner requires a warm-up stage, which is a pretraining process with the help of external aligners. Although the bidirectional prior/posterior module of NaturalSpeech can reduce the training and inference mismatch caused by the bijective flow module, it further increases the model's computational cost of training.
recent work EfficientTTS (EFTS) (Miao et al. 2021) proposed a NAR architecture with differen- tiable alignment modeling that is optimized jointly with the rest of the model. In EFTS, a family of text-to-mel- -er--o-mee-speetrograam models and a text-to-waveform model are developed. However, the per- formance of the text-to- tet-tttowvveoomm model is close but no better than two-stage models. Inspired by EFTS, we propose an end-to-end text-to-wave waveeorro TTS system, the EfficientTTS 2 (EFTS2), that overcomes the above issues of current one-stage models with competitive model performance and higher efficiency. The main contributions of this paper are as follows:
We improve upon the alignment framework of EFTS by proposing a hybrid attention mech- anism and a variational alignment predictor, empowering the model to learn expressive la- tent time-aligned representation and have controllable diversity in speech rhythms. (Section 2.2.1
We introduce 2-layer hierarchical-VAB. VAB-based waveform generator that not only produces high-quality outputs but also learns hierarchical and explainable latent variables that control different aspects of the generated speech. (Section 2.2.2)
We develop an end-to-end adversarial TTS model, EFTS2, that is fully differentiable and can be trained end-to-end. It matches the baseline VITS in naturalness and offers faster inference speed and a smaller model footprint. (Section 2.2
We extend EFTS2 to the voice conversion (VC) task and propose EFTS2-VG. -C, end-to- end VC model. The conversion performance of EFTS2-VC is comparable to a state-of-the art model (YourTTS, [asanova et al. (2022) ) while obtaining significantly faster inference speed and much more expressive speaker-independent latent representations. (Section 2.3
METHOD
Our goal is to build an ideal TTS model that enables end-to-end training and high-fidelity speech generation. To achieve this, we consider two major challenges designing the model:
(i) Differentiable aligner. The TTS datasets usually consist of thousands of audio files with corre- sponding text scripts that are, however, not time aligned with the audios. Many previous TTS works either use external aligners (Ren et al.) 2019; 202 Chen et al. (021)) non-differentiable internal aligners (Kim et al. 2020 2021] Popov et al. 2021) for alignment modeling, which complicates the training procedure and reduces the model's 's efficiency. An ideal TTS model requires internal differentiable aligner that can be optimized jointly with the rest of the network. Soft attention (Bah- danau et al., 2015) is mostly used in building an internal differentiable aligner. However, computing soft attention requires autoregressive decoding, which is inefficient for speech generation (Weiss) al. (222I. Donahue et al., proposes to use Gaussian upsampling and Dynamic Time Warping (DTW) for alignment learning, while training such a system is inefficient. To the best of our knowl- edge, EFTS is the only NAR framework that enables both differentiable alignment modeling and high-quality speech generation. Therefore, we integrate and extend it into the proposed models.
(ii) Generative modeling framework. The goal of a generative task, such as TTS. is estimate the probability distribution of the training data, which is usually intractable in practice. Multiple deep generative frameworks have been proposed to address this problem, including Auto- Regr-Reeresevve models
et al.] (2015)), Normalizing Flows (NFs, Kingma & Dhariwal] (2018)), De- noising Diffusion Probabilistic Models (DDPMs, Ho et al. (2020), Generative Adversarial Net- works (GANs, Goodfellww et al. (2014) and Variational Auto- Encoderss (VAEs, Kingma & Welling 2014)). However, AR models have linear growing generation steps; NFs use bijective transfor- mations and often suffer from large model footprints: DDPMs require many iterations to produce high-quality samples. In this work, we propose to use GAN structure with a hierarchical- VAE-based generator, which allows efficient training and high-fidelity generation.
2

---

Under review as a conference paper at ICLR 2023
The rest of this section is structured follows. In Section 2.1
discuss the background knowledge of the differentiable aligner of EFTS. In Section 2.2 we present the proposed TTS model EFTS2. In Section 2.3
we propose EFTS2-VC, voice conversion model built on EFTS2.
2.1 BACKGROUND: DIFFERENTIAL ALIGNER IN EFFICIENTTTS
In this part, we briefly describe the underlying previous work EFTS, upon which we build our model that simultaneously learns text-audio alignment and speech generation.
The architecture of EFTS is shown in Figure 1 A text-encoder encodes the text sequence a: E R into hidden vector 3₁ RT:.D, while a mel-encoder encodes the mel-spectrogram y Rm2. into vector 3/, E R RTD.D During training, the text-mel alignment is computed using scaled dot- product attention mechanism /Vaswani et al., 2017). as Eq. (T. which enables parallel computation.
yh 20h
a = Maxt
vD
However, is unavailable in the inference phase, making computing a intractable. To address this problem, Miao et al. introduce the idea of alignment vectors, which are used to reconstruct the attention matrix using series of non non-parametric differentiable transformations:
(2) where π E RT² and e E R are two alignment vectors and a' is the reconstructed align- ment matrix. A parametric alignment predictor with an output of é, the predicted e, is trained jointly given input Z and therefore allows tractable computation of a' in the inference phase based on é. The alignment vector æ is defined as the sum of the input index weighted by a:
O E R ,25 Eq.
π EER
R
Eq.
EE R
Eq. 6)
a' E RR
Σ
Jy
a.
Decoder
where 0 S i T - and 0 S T2 - are indexes of the input and output sequence respectively. Here, π can be considered as the expected location that each output timestep attends to over all possible input locations. According to the conclusion of EFTS, π should follow some monotonic constraints including:
X Kalign
To
0 < Am; 11, TT2-1 Ti -
A Aligument Predictor
a
where Ary π -- Therefore, additional transformations are employed to constraint to be monotonic (Miao et al. (2021). Eq. (8-10)).
X
Yn
Text- Eccoder
Mei- Encoder
It is worth noting that, the alignment vector π is representation of the alignment matrix with same length of the output. How- for
task with inconsistent ouput- length like TTS, it is more natural to have an input-level alignment vector during the inference phase. Thus, differentiable re-sampling method is proposed in EFTS. Let e denote the re-sampled alignment vector, then is computed follows:
X
Figure 1: Overall architec- ture of EFTS. The arrow with a broken line represents the computation in the infer- ence phase only.
exp o 2 ( 1)2)
T3
NH.
Mi
(5)
n exp (-0-2((T,)) 1)2)*
n=U
Here is a hyper-parameter. In EFTS, is called the aligned position vector. A Gaussian transfor- mation is used to calculate a' from e:
exp (-o le; ((:;-1)' j]
de TT-1 exp (-o 2 j)2)
(6)
In the training phase, e is used to construct a' In the inference phase, a' is derived from the predicted alignment vector é. As replacement of the original attention matrix a., the reconstructed attention matrix a' is further used to map the hidden vector D; to time aligned representation Talign and produce the outputs.
3

---

Under review as a conference paper at ICLR 2023
Hierarchical
Decoder
Decoder
Raw Wave
N(Ag (2)
Raw Wave f
Falign
HiFL-AN Generator
HIFL-GAN Generator
Differentiable Aligner
M( (3)
(2)
22
Attention Reconstruction
1.P
Prior 2
Posterior
Prior Network
WaveNet blocks
Loss
& (a,
WaveNe blocks
MSE+
(a.b)
N(4 (1)
(1)
21
Mi (1)
Na !0
VAP Decoder Net blocks
K1.
Prior Network
Loss
WaveNet blocks
A
Hee (a,
Hierarchical
Prior
Zolign:
Prior Network
N (Ps (
(1)
(A)
WaveNet blocks
Reconstruction
Falign
é, (a,
Differentiable Aligner
Mh.
VAP Encoder
T
I.P
I.P
WaveNet blucks
VAP Decoder
37h
WaveNet
I7
blocks
Phoneme Encoder
Encoder
KI. Loss
Scaled Dot-Product Attention
Phoneme Encoder
WaveNef blocks
Transformer
blocks
Mion
(( 1)
Transformer
blocks
Phoneme x
STFT
2A (0,, 1)
Sequence
Raw Wave
Sequence
(a) Training Phase
(b) Differentiable Aligner
(c) Inference Phase
Figure 2: Overall architecture of EFTS2's generator. LP refers to linear projection. The dotted lines refer to training objectives.
2.2 EPFICIENTTTS 2: VARIATIONAL END-TO-END TEXT-TO-SPEECH
To better describe our model, we divide the generator, shown Figure into two main blocks: (i) the differentiable aligner, which maps the input hidden state to time aligned hidden represen- tation Falign; and (ii) the hierarchical- VAE-based waveform generator, which produces the output waveform y from Ealign. More details will be discussed in the rest of this section.
2.2.1 DIFFERENTIABLE ALIGNER
Grounded on EFTS, we construct the differentiable aligner with two major improvements: (i) hybrid attention mechanism and (ii) a variational alignment predictor. The structure the differen- tiable aligner is shown in Figure 2b
Hybrid attention mechanism The performance of the aligner in EFTS heavily depends on the ex- pressiveness of the reconstructed attention matrix a', which is derived from the alignment vector e. Here, e can be considered as the expected aligned positions for each input token over all possi- ble output frames. However, in the TTS task, input token normally attends to multiple output frames. Therefore. is better to incorporate the boundary positions of each input token when con- structing the attention matrix. To this end, introduce hybrid attention mechanism that integrates two attention matrices: the first attention matrix (1) is derived from in EFTS (Eq 2 and the second attention matrix a (2) is derived from the token boundaries using the following transforma- tions:
0 E RT..I Eq.
T
Eq.
(a ( RT)
E Ti.T
where (a R. Th) are start and end boundaries of the input tokens. We call the process from the attention matrix a to boundary pairs (a, b) the Attention to Boundaries (A2B) transfor- mation and the process from boundary pairs (a, b) to the reconstructed attention matrix (?) the Boundaries Atlention (B2A) transformation. Inspired by Eq. (). the A2B transformation is
4

---

Under review as a conference paper at ICLR 2023
formulated using the following equations:
PGg = exple (-o
(TT - -1:)²)
[0, =0,
i=
where pi =
(8)
2n= exp (-00--20
(TT - p)))²)
-
0<i<
ai+1,
i< Ti -1
04 In. n, b
(9)
T2 1,
i T
In the meantime, the B2A transformation is designed as follows:
exp (energy
energy,, (lj a.l + 16, -1 (6. a:))²
(2) T1-1 exp (energym.))
(10)
m=0
As can be seen, for the ith input token with its corresponding boundaries (a,, b,), fenergy, و reaches the maximum value 0 only if the output position j falls into its boundaries, meaning as j S by. For an output position outside of the boundaries, the further it is away from the boundaries, the lower value of (energv.,)) it gets, resulting in less attention weight.
Note that the proposed B2A approach works for all those TTS models with explicit token durations, and is potentially better than the conventional approaches: (i) compared to the repetition operation (Ren al. 2019: 202T Kim et al.] 2020, 2021). the proposed approach differentiable and enables batch computation; (ii) compared to the popular Gaussian upsampling (Donahue et al. 2021 Shen ict al. 2020) that considers only the centralized position, the proposed approach employs boundary positions, which is more informative; (iii) compared to the learnable upsampling (Elias al. 2021
Tan et al. 2022), the proposed approach is monotonic and much easier to train.
In preliminary experiments, of a in Eq. (6) and Eq. (IO). In order to obtain better model performance, we use learnable a in this work. We further map the hidden representation æt time-aligned hidden representation Batign using a approach similar to the multi-head attention mechanism in Vaswani et al. (2017):
found out that the model performance is greatly influenced by choice
head = ( (x,,W())
Falign Concat head W
(11)
where (W(+), 8o are learnable linear transformations. The Fatign is then fed into the hierarchical- VAE-based waveform generator as input.
Variational alignment predictor NAR TTS models generate the entire output speech in parallel, thus alignment information is required in advance. To address this problem, many previous NAR models train duration predictor to predict the duration of each input token (Ren al. 2019 Kim eet al. 202T). Similarly, EFTS employs an aligned position predictor to predict the aligned position vector e, As opposed to a vanilla deterministic alignment predictor (DAP), in this work, we use variational alignment predictor (VAP) to predict the alignment vector e and the boundary positions and b. The main motivation behind this is consider the alignment prediction problem generative problem since one text input can be expressed with different rhythms. Specifically, the VAP encoder receives the relative distances e and b-a, a, and outputs latent posterior distribution (a(zale a, b a, 2 conditioned on FA. while the VAP decoder estimates the output distribution by inputting AA, and conditioned on 24. The prior distribution is a standard Gaussian distribution. For simplicity, both the encoder and the decoder of VAP are parameterized with non-causal WaveNet residual blocks. The training objective of the VAP is computed as:
Lalign =Xi((1a))) - log(e a + 1)l + ||dg (zA) log(b - a + +)
A2 DELL (c b, a. x.),), (A) (c b, b- a, xx)) [| O, (12)
where, dg and de are outputs of the VAP decoder, u and is a small value to avoid numerical instabilities. The first term in Eq. (12) is the reconstruction loss that computes the log-scale mean square error (MSE) between the predieted relative distances
(A) (A) and o are outputs of the VAP encoder,
and target relative distances. The second term is the KI. divergence between the posterior and prior distributions. In the inference phase, the alignment vector é and boundary positions à and are computed as:
b.
Σ (exp((de) mz)))m)) -e), à; m==
i= 0
é exp(((d))( - à (13)
> 0
5

---

Under review as a conference paper at ICLR 2023
where Z is sampled from the standard Gaussian distribution. A stop gradient operation is added to the inputs of the VAP encoder, which helps the model to learn a more accurate alignment in the training phase.
2.2.2 CERRARCHICAL-VAE--BEI
WAVEFORM GENERATOR
Producing high-quality waveforms from linguistic features (e.g. texts, phonemes, or hidden lin- guistic representation Dalign) is known as a particularly challenging problem. This is mainly be- cause linguistic features do not contain enough necessary information (e.g. pitch and energies) for waveform generation. A primary idea is to use a VAE-based generator that learns the waveform generation from a latent variable 2. This is sampled from informative posterior distribution (=(=))) parameterized by network with acoustic features as input. A prior estimator Pu (Xatinn)) with Palign input is also trained jointly. Training such a system is minimize the reconstruction emorr between the real and predicted waveform and the KI. divergence between the prior and the posterior distribution. However, the prior distribution contains no acoustic information while the learned posterior must be informative w.r.t. the acoustic information. The information gap makes it hard to minimize the KI. divergence between the prior and the posterior distribution. To tackle this problem, we introduce a 2-layer hierarchical- VAE structure that enables informative prior formula- tion. The hierarchical- -Aaiiiccal------Bae VAE-based waveform generator is composed
the following blocks: (i) posterior network which takes the linear spectrograms Ylin as input and outputs two latent Gaussian posterior yo [2]] (ssn); 4s (22 yru):: (ii) hierarchical prior network which consists of two stochastic layers: the first layer receives Falign and outputs latent Gaussian prior po(z) [2] [align); the sec- ond layer takes a latent variable 21 and formulates another latent Gaussian prior pe (22 (=1). where 21 is sampled from posterior distribution 2o ( in training phase and from prior distribution (z) (atign) in inference phase; (iii) a decoder which produces the waveform from the latent vari- able z2 where #2 is sampled from posterior distribution qo yrin) in training phase and from prior distribution pe (z2 æ1) in inference phase.
Therefore, the overall prior and posterior distributions are formulated as:
Po Falign)))) po (2) (oo
(0(=|y) = 4 (22)
(14)
The training objective is:
Lwar ( (log (o + + (z](((tton)))
Crecons + DKL
Bm.) (21 (aligg)) DKL (2o2lle)))) (Po
=>al|ymmet Imeill |1 + A4 (DRL (DRL(N (yii)) ox (1) (y))))
(2) N/ (1) (Zalign). ) (((atm)))))(() + DKL (N(z2; u (Vun), o (3)6))
N(22: pg (2) (z1). 08 (2) (z1))))
(15)
Here, the reconstruction loss is the L1 loss between the target mel-spectrogram ymee and the pre- dicted meelsppectrogamm &mel which is derived from the generated waveform ÿ. this work, the two prior estimators and the posterior estimator are all parameterized by stacks of non-causal WaveNet residual blocks with the estimated means and variations, while the decoder is inspired by the gen- erator of HiFi-GAN (Kong al. 2020). Similar to EFTS and VITS, the decoder part is trained on sliced latent variables with corresponding sliced audio segments for memory efficiency. There are some previous TTS models (Kim et al. 2021 Tan et al. 2022) that also incorporate the VAE frame- work in end-to-end waveform generation. EFTS2 differs from them in two aspects: (i) EFTS2 uses 2-layer hierarchical VAE while previous works use single-layer VAE; and (ii) in previous work, the KL. divergence between the prior and posterior distributions estimated between a latent variable (which is just sample from posterior distribution) and a multivariate Gaussian distribution, while EFTS2 computes the KL divergence between two multivariate Gaussian distributions which allows for more efficient training.
2.2.3 THE OVERALL MODEL ARCHITECTURE
The overall model architecture of EFTS2 is based GAN, which consists of generator and multi- ple discriminators. We follow Kong et al. (2020) in implementing the multiple discriminators whose performance is experimentally confirmed by many previous works (You et al., 2021 Kim et al.,

---

Under review as a conference paper at ICLR 2023
The feature matching loss Lim is also employed for training stability. In the training phase, a phoneme sequence z is passed through a phoneme encoder to produce the latent representation The while the corresponding linear spectrogram Yton is passed through spectrogram encoder to produce the latent representation yı. and two latent Gaussian posteriors. [ Same as EFTS and VITS, the phoneme encoder is parameterized by stack of feed-forward Transformer blocks. The proposed differentiable aligner receives the latent representation T6 and y/, and outputs the time-aligned latent representation Valign. Then Palign is further fed the hierarchical- -aabb-ed waveform generator to produce the output ÿ. The overall training objective of the proposed generator G is:
2021].
Liotal
LLwa Latign + Ladv(G) (Gm(G)
(16)
2.3 EFTS2-VCH END-TO-END VOICE CONVERSION
Voice conversion (VC) is task that modifies source speech signal with the target speaker's tim- bre while keeping the linguistic contents of the source speech unchanged. The proposed voice conversion model, EFTS2-VC (shown in Appendix A is built upon EFTS2 with several module differences:
The alignment predictor excluded in EFTS2-VC since there is no need to explicitly tell the text-spectrogram alignment in the inference phase.
Instead of using e or the token boundaries, the reconstructed attention matrix of is derived from (Eq. (17)). This not only simplifies the computation pipeline but also allows the network to obtain a more accurate text-spectrogram alignment. Similar to the TTS model, EFTS2-VC VC uses multiple reconstructed attentions by employing multiple learnable (a) |R: = 1...,,,) H).
exp (-0 (
2 1)
d,
(17)
2m=0 exp On (TT m)
The speaker embedding of the source waveform, which is extracted from trained speaker encoder, is introduced as conditions to the spectrogram encoder, the second prior network, and the HiFi-GAN generator.
Again, we consider the bierarchical-VAEE-basee framework discussed in Section 2.2.2 During train- ing, the prior distribution pe (z) (eligg) is estimated by stack of WaveNet blocks with Palign as the only input. Since aliggn is the time-aligned textual representation without any information about the speaker identity, we can easily draw the conclusion that the prior distribution po(21 [align)) ) con- tains only the textual information and does not contain any information about the speaker identity. The conclusion can be further extended to the posterior distribution Pz (1(t)))) since the network trained by minimizing the KI. divergence between the prior distribution and posterior distribu- tion. Therefore, the spectrogram encoder works speaker disentanglement network that strips the speaker identity while preserving the textual (or content) information. Then the second prior net- work and the variational decoder reconstruct the speech from the content information and the input speaker embeddings. During inference, the disentanglement network and the reconstruction net- work are conditioned on different speaker embeddings. Specifically, the disentanglement network receives the spectrogram and the speaker embedding from a source speaker, and outputs a latent dis- tribution (2) lyten.. Meanwhile, the reconstruction network produces the output waveform from the latent variable æ1 and the speaker embedding from target speaker. With these designs, EFTS2- VC performs an ideal conversion that preserves the content information of the source spectrogram while producing the output speech that matches the speaker characteristics of a target speaker.
3 EXPERIMENTS
Due to the paper page limitation, we put the dataset settings. model configurations, and training de- tails in Appendix We present the experimental results in the following subsections and Appendix
Ideally, end-to-end TTS system should operate on unnormalised text. We external tools to convert the unnormalised texts to phonemes in this work. We will explore some data-driven approaches the future to address this limitation.
Audio samples can be found htt.ees//aaoyyoooeeeeeaaagtttttt

---

Under review as a conference paper at ICLR 2023
Table 1: MOS results from baseline mod- els, the ablation studies and EFTS2 on LJ- Speech.
Table 2: MOS and sim-MOS for voice conversion periments on VCTK dataset.
Model
MOS
Ground Truth
4.39 = 0.12
Speaker
Model
MOS
sim-MOS
EFTS EFTS-CNN CNN VITS
HIFI-GAN
4.17 0.08
EFTS2-VC
4.22 1: 0.08
4.14 1 0.14
Seen
4.48 = 0.14
YourTTS
4.18 d:
4.14 - 0.12
EFTS2 (1-layer VAE)
4.12 0.12
EFTS2-VC
4.08 + 0.13
3.95 + 0.08
EFTS2 (Single Attention)
4.31 + 0.11
Unseen
YourTTS
4.06 +
3.98 + 0.13
EFTS2 (DAP)
4.44 + 0.08
EFTS2
4.48 + 0.11
Table 3: Comparison of the number of parameters and inference speed. All models are benchmarked using the same hardware.
Total Params
Inference Params
Inference Speed (kHz)
Model
Real-time
(M)
(M)
EFTS-CNN + HIFI-GAN
58.34
52.18
1768.32 1508.31
x80.20
VITS
34.64
27.74
x68.04
EFTS2
32.38
24.35
2247.52
x101.92
3.1
TTS SPEECH QUALITY EVALUATION
In this subsection, we compare the quality of audio samples generated by EFTS2 and the baseline models. The baseline models are the best-performing publicly-available model VITS (Kim et al.) 2021] and the best-performing model in the EFTS family, EFTS-CNN + HIFI-GAN, with our own implementation. The quality of audio samples is measured by the 5-scale mean opinion score (MOS) evaluation. Ablation studies are also conducted to validate our design choices. The MOS results on LJ- LJ-Speech dataset (Ito 2017) shown in Table I EFTS2 significantly outperforms 2-stage EFTS-CNN. In addition, we also observed that there is no significant difference between EFTS2 and VITS. Both models achieve comparable scores ground truth audios, which means that the speech quality of EFTS2 and VITS is very close to natural speech. Ablation studies confirm the importance of our design choices. Removing either the hierarchical- VAE structure (1-layer VAE) or the hybrid attention mechanism (Single Attention) leads to a significant MOS decrease. Although the deterministic alignment predictor (DAP) has a similar MOS score the variational alignment predictor, it lacks diversity which is very important for speech generation.
3.2 MODEL EFFICIENCY
The inductive biases of the proposed hierarchical- Serarchical-VAA--aaeee VAE-based generator make the overall model smaller and significantly faster than the baseline models. The model size and inference speed of EFTS2 along with the baseline models presented Table Since EFTS2's generator employs significantly smaller number of convolution blocks in comparison with VITS, the inference speed is greatly improved. Specifically, EFTS2 is capable of running frequency of 2247.52 kHz, which is 1.5x faster than VITS.
3.3
VOICE CONVERSION
The conversion performance of EFTS2-VC is evaluated on the VCTK dataset (Yamagishi et al.) 2019) with a comparison to the baseline model YourTTS (Casanova et al. 2022). Table presents the MOS scores and similarity scores. EFTS2-VC achieves slightly better MOS scores and simi- larity scores for seen speakers and comparable similarity scores for unseen speakers. Note that the conversion of YourTTS requires rumning the flow module bidirectionally, which results in slow conversion speed. On the other hand, EFTS2- VC significantly faster. It runs 2.15x x faster than YourTTS on Tesla V100 GPU.
8

---

Under review as a conference paper at ICLR 2023
400
t1-0.0, f--088 4-0.0
FD (H2))
200
0.00
0.25
0.30
0.75
1.00
1:25
1,30
1.75
400
4m0.5, 5m0.8. 4=0.0
FO (HZ2)
200
0.00
0.25
0.50
0.75
1.00
1.25
1.50
175
f=1.0, t.-0.8, 4-0000
FO (HZ))
200
0.00
0.25
0.50
0.75
1.25
1.50
1775
Figure 3: F0 contours obtained from the test samples generated by EFTS2 with different f1.
3.4 ANALYSIS OF THE LATENT HIERARCHY
One question from Section 2.2.2 is whether the hierarchical architecture the proposed generator empowers the model have controllable diversity in hierarchical and explainable latent variables. Analysis of two latent variables 21 and 22 confirms this statement. Figure 3 shows the scatter plots of the FO contours extracted from audios generated with 3 different sets of 21 and #2. All the audios are synthesized using the same phoneme sequence and the same alignment, which means the time- aligned text representation Falign are precisely the same for all waveforms. The t and by are two scaling factors applied on the variances of the latent distributions (o and po po respectively, For each pair l and /2. pairs of 2 and 22 are sampled and then used synthesize waveforms. As shown in Figure increasing l considerably increases the variation of FO, whereas large l2 barely produces any variation on FO when 1 = 0. This means essential acoustic features such as FO are mostly fixed after #1 is sampled. In other words, (0(21) is linguistic latent representation offering variations on the spectrogram domain acoustic information, while pe (=(2)) contains the spectrogram domain acoustic information and offers variations time domain information. This is important because though we did not explicitly give model any constrain, it still learns the hierarchical and explainable latent representations with controllable diversity.
4 CONCLUSION AND DISCUSSION
We presented EfficientTTS 2 (EFTS2), a novel end-to-end TTS model that adopts an adversarial training process, with generator composed of differentiable aligner and hierarchical- VAE- based speech generator. Compare to baseline models, EFTS2 is fully differentiable and enjoys a smaller model size and higher model efficiency, while still allowing high-fidelity speech generation with controllable diversity. Moreover, we extend EFTS2 to the VC task and propose VC model, EFTS2-VC, that capable of efficient and high-quality end-to-end voice conversion.
The primary goal of this work is to build competitive TTS model that allows for end-to-end high- quality speech generation. In the meantime, the proposed design choices can easily be incorporated into other TTS frameworks. Firstly, the proposed B2A approach could potentially be a handier replacement for conventional upsampling techniques nearly all NAR TTS models, given that is differentiable, informative, and computationally cheap. Secondly, the differenniable aligner may be superior alternative for any external aligner or non-differentiable aligner, as improves the uniformity of the model and makes the training process end-to-end. Thirdly, the 2-layer hierarchical- VAE-based waveform generator can potentially outperform the popular Row-VAE-bal --s-baded counterpart Kim et al.) 2021 Tan al. 2022) since is more efficient and offers more flexibility in network design. Lastly and most importantly, the entire architecture of EFTS2 could serve as practical solution sequence- to sequence tasks that have the nature of monotonic alignments. We leave these assumptions to future work while providing our implementations as a research basis for further exploration.

---

Under review as a conference paper at ICLR 2023
5 REPRODUCIBILITY STATEMENT
To encourage reproducibility, we attach the code of EFTS2 and EFTS2-VC in the supplemental ma- terials. Please refer to the README.md in the supplemental materials for the training and inference details of our code.
REFERENCES
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference Learning Representations, 2015.
Edresson Casanova, Julian Weber, Christopher D Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. YouTTSS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In International Conference Machine Learning. 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, Najim Dehak, and William Chan, WaveGrrd 2: Iterative Refinement for Text-to-Speech Synthesis. In Interspeech, 2021.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. VoxCeleb2: Interspeech, 2018.
Deep speaker recognition. In
Jeff Donahue, Sander Dieleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-Ene End Adversarial Text-to-Speech. In International Conference on Learning Representations, 2021.
Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang. Ye Jia, RJ Skerry-yyan and Yonghui Wu. Parallel Tacotron 2: A Non- Autorreeeerrei Neural TTS Model with Differentiable Duration Modeling. In Interspeech, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Infor- mation Processing Systems, 2014.
Hee Soo Heo, Bong-Jin Lee, Jaesung Huh, and Joon Son Chung. Clova baseline system for the voxceleb speaker recognition challenge 2020. In arXiv, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Neural Information Processing Systems, 2020.
K Ito. The lj speech dataset. 2017.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-TTS: Generative Flow for Texe-to-Speech via Monotonic Alignment Search. In Advances in Neural Information Processing Systems, 2020.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech. In International Conference Machine Learning, 2021.
D. P. Kingma and P Dhariwal. Glow: Generative Flow with Invertible Ixl Convolutions. In Ad- vances in Neural Information Processing Systems, 2018.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Confer- ence on Learning Representations, 2014.
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis In Advances in Neural Information Processing Systems, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In International Con- ference Learning Representations, 2019.
C. Miao, S. Liang, M. Chen, J. Ma. S. Wang, and J. Xiao. Flow-TTS: A Nun-Autoregresssvve Net- work for Text to Speech Based On Flow. ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP). 2020.
10

---

Under review as a conference paper at ICLR 2023
Chenfeng Miao, Liang Shuang, Zhengchen Liu, Chen Minchuan, Jun Ma, Shaojun Wang, and Jing Xiao. EfficientTTS: An Efficient and High -uallit Text-to- text-to-Speech Architecture. In International Conference Machine Learning, 2021.
W. Ping, K. Peng, A. Gibiansky, S. O. Arik, A. Kannan, S. Narang, Raiman, and Miller. Deep Voice 3: 2000-Speaker Neural Text-to-Speech. In International Conference on Learning Repre- sentations, 2018.
Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text-to- speech. In International Conference on Learning Representations, 2019.
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In International Conference on Machine Learning, 2021.
Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu. FastSpeech: Fast, Robust and Controllable Text to Speech. In Advances in Neural Information Processing Systems, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. FastSpeech 2: Fast and High-Quality End-to-End Text Speech In International Conference on Learning Representations, 2021.
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, and R. Skerry Ryan. Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predic- tions. In ICASSP 2018-2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu. Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling. In arXiv, 2020.
Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, Frank Soong. Tao Qin, Sheng Zhao, and Tie- Yan Liu. NaturalSpeech: End- to-End Text to Speech Synthesis with Human-Level Quality. In arXiv, 2022.
Jean-Mare Valin and Jan Skoglund. LPCNet: Improving neural speech synthesis through linear pre- diction. In ICASSP 2019-2019 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP). IEEE, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Y. Wang, R. Skerry-Raan, D. Stanton, J. Weiss Y. Wu, N. Jaitly, and Z. Yang. Tacotron: Towards End-to-End Speech Synthesis. In Interspeech, 2017.
Ron J Weiss, RJ Skerry-Ryan Eric Battenberg, Soroosh Mariooryad, and Diederik P Kingma. Wave- Tacotron: Spectrogram-Free Free End-to-End Texe-to-Speech Synthesis. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2021.
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English
Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2019.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel WaveGAN: A fast waveform gen- cration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020.
Jaeseong You, Dalhyun Kim, Gyuhyeon Nam, Geumbyeol Hwang, and Gyeongsu Chae. GAN Vocoder: Multi- Resolution Discriminator Is All You Need. In Interspeech, 2021.
11

---

Under review as a conference paper at ICLR 2023
Hicrarchical Prior
Decoder
Decoder
W K of
Raw Wave
Raw Wave
HIFEGAN Gengrator
DIFFGAN Generator
LP
22
Prior Network WaveiNe Itlecks
Illerarchical
Prior
Ni ())
A1
M(4 of
NG 19) (91))
LP
Taliss
Prior Network
Aligner
Aliention Reconstrwetion
WaveNeer bloclis
Posterior ( ())
-Egltgn
Differentaalle Aligner
IP
EP
Phoneme Encoder
Eneader WaveeNet
Transformer
Speaker Lacuder
Scaled Atteation
Speaker Eneader
Encader
Speaker Lacoder
blocks
Posterior STFT
Phoneme
STFT
Turget
Sequence
(a) Training Phase
(b) Differentiable Aligner
(c) Inference Phase
Figure 4: Overall model architecture of EFTS2-VC. LP refers linear projection. The dotted lines refer to the training objectives.
A OVERALL ARCHITECTURE OF EFTS2-VC GENERATOR
The overall architecture of EFTS2-VC's generator is shown in Figure
B EXPERIMENTAL SETUP
Datasets Two public datasets are used our experiments, the LJ Speech dataset (Ito 2017) and the VCTK dataset |Ymaagishi et al. 2019). The LJ Speech dataset is an English speech corpus consisting of 13,100 audio clips of single female speaker. Each audio file is single-channel 16-bit PCM with sampling rate of 22050 Hz. The VCTK dataset is multi-speaker English speech corpus that contains 44 hours of audio clips of 108 native speakers with various accents. The original audio format is 16-bit PCM with a sample rate of 44kHz. In our experiments, all audio clips are converted into bit and down- sampled to 22050 Hz. Both datasets are randomly split into a training set, a validation set, and test set.
Preprocessing The linear spectrograms of the original audio are used as the input of the spectrogram encoder. The FFT size, hop size, and window size used in Short- time Fourier transform (STFT) to obtain linear spectrograms are set to 1024, 256, and 1024 respectively. Before training, the text sequences converted phoneme sequences using open-sourced software phonemizer
Configurations The phoneme encoder of EFTS2 a stack of 6 Feed-Forward Transformer (FFT) blocks, where each FFT block consists of multi-head attention layer with 2 attention heads and convolutional feed-forward layer with a hidden size of 192. The HiFi-GAN generator consists of residual convolution blocks, where each block has a transpose convolution layer and 3 ID convo- lution layers following Kong et al. (202)). The rest of EFTS2 is composed of stacks of non-causal WaveNet residual blocks. Specifically. the number of convolution layers in the VAP encoder, the VAP decoder, the first prior network, the second prior network, and the spectrogram encoder are 3, 3, 3, 5, and 16 respectively. The kernel size and the dilation rate for all the WaveNet lay- ers. EFTS2-VG shares similar model configurations with EFTS2 except that the variational aligned predictor is excluded from EFTS2- VC. The scaling factor t t to are set to 0.7, 0.8, 0.3 respec- tively. Two linear projections that produce prior distributions (LP blocks in Hierarchical Prior block
https://ggi thwb. httpss//gggtthbb.co///oooh/hh//e zer
12

---

Under review as a conference paper at ICLR 2023
200
Erooder tmeetta
1,50
0.6
Ercedeer timestoo
250
0.4
50
50
100
200
300
400
500
600
100
200
300
400
500
600
Docoder timestop
Docoeer timesto
(a) a (1)
(b) a (2)
Figure Visualization of the attention matrices of EFTS2
in Figure 2a and two linear projections that produce posterior distributions (green LP blocks in Posterior block Figure 2a are initialized with zeros, such that both the prior and the posterior are initially standard Gaussian distributions, with KL divergence of zero. The deterministic align- ment predictor (DAP), which takes 24, input and outputs the alignment vectors (eg. à. b, ê), is parameterized by convolution layers and a linear mapping. Each convolution layer is followed by layer normalization and leaky ReLU activiation. The trained speaker encoder of EFTS2-VC is speaker recognition model (Heo et al. (2020) trained on the voxceleb2 (Chung et al., 2018) dataset. The pre-trained model is publicly available (Heo et 2020).. The baseline model EFTS2-VC is pre-trained model from YourTTS (Casanova et al. (022) trained on the VCTK dataset. For fair comparison, we down-sampled the generated audios of EFTS2-VG VC
kHz match the sample rate of YourTTS' generated audios during the evaluation process. The hyper-parameters of EFTS2 and EFTS2-VC are listed in Table
Training Both EFTS2 and EFTS-VC are trained on 4 Tesla V100 GPUs with 16G memory. The batch size on each GPU is set to 32. The AdamW optimizer (Loshchilov Hutter 2019) with 31 0.8, B2 0.99 is used to train the models. The initial learning rate set to 2 10 and decays at every training epoch with decay rate of 0.998. Both models converge at 5004 step.
MOS Evaluation We conducted the Mean Opinion Score (MOS) tests evaluate the model per- formance of EFTS2 and EFTS2-VE VC. 15 raters were asked to make naturalness judgments about the randomly selected audio samples from the test set, and then gave their rating scores 5-point Likert scale score (1 Bad; 2 = Poor; 3 = Fair; 4 = Good; 5 = Excellent) with rating increments of 0.5.
C ANALYSIS OF EFTS2
C.1 VISUALIZATION THE ATTENTION MATRICES
The attention matrices of EFTS2 visualized in Figure As can be seen, both (!)) and (?) are monotonic. While (2) learns clean boundaries for the input tokens, (1)) learns a more smooth alignment.
C.2 TRAINING LOSS COMPARISON
Comparisons of the model performance between a TTS model using a 2-layer hierarchical VAE- based generator and TTS model using a 1-layer VAE-based generator visualized in Figure As shown in the figure, with an additional variational structure, both the KL loss and mel-spectrogram reconstruction loss decrease significantly.
13

---

Under review as a conference paper at ICLR 2023
Table 4:
of EFTS2 and EFTS2-VC
Modules
EFTS2
EFTS2-VC
GimbeddingDimension 192,
FFTBlocks =
HiddenDimmnnsion = 512,
Phoneme Encoder
AttentionHeads = =2,
ConyFitterSize = 768,
SonvKernelSize = 33
WaveNet Layers 16,
kernelSize
Spectrogram Encoder
Dilation = 1.
FilterSize 192,
WaveNet Layers 3,
kernelSize = 3,
VAP Encoder
Dilation 1,
FilterSize 192.
WaveNet Layers =
kernelSize = 3,
VAP Decoder
Dilation
FilterSize 192,
AttentionHeads = 2,
Attention reconstruction
HiddenDimmension 384,
WaveNet Layers 3,
kerneliize 5,
Prior Network
Dilation = 1.
FilterSize 192,
WaveNet Layers =
kernelSize = 5,
Prior Network
Dilation
FilterSize 192,
ConvBlocks = 4,
UpsamplingRate = [8,8,2,2].
UpsamplingKermelSizees [16,16,4,4]
HiFi-GAN Generator
UpsamplelnittalChamnel 512
ConvLayers 3, ConyKernelSize [3,7.11],
ConyDiiation [1.3.5]
D COMPARISON OF DIFFERENT ALIGNERS
We conducted a 5-point bydi-b Comparative Mean Opinion Score (CMOS) evaluation to verify the effectiveness of the proposed aligner. We consider the following approaches for comparison:
Non-Differentiable approaches. peated upsampling (Ren et al. 2019 2021); and ND-MAS internal aligner using MAS (Kim et al. 2021).
ND-External- Reeeritton-e external aligner with re-
Differentiable approaches.
A.Esternal-Gaussian-CCeCaC
external aligner using the
upsamlping approach proposed by EATS (Donahue et al.) 2021 Shen et al., 2020): D- Caaermal-Gaussia--Boonneee
external aligner using proposed B2A approach following internal aligner with single learnable attention (Elias)
Eq. (10); D-Internal-Leanaale
et al. (021). The token boundaries are derived from π following Eq. ( D-Internal- Gaussian-e: internal aligner. The attention is derived from alignment vector (Miao al.,
[021]; Baseemaal-Gussian-BBunneee
internal aligner. The attention is derived from token boundaries following Eq. (T0); D-Hybrid: The proposed hybrid attention.
14

---

Under review as a conference paper at ICLR 2023
4.0
25
2-ayer VAE 1-ayer VAE
2-kayar VAE
L-byyr VAE
3.5
24
23
3.0
22
2.5
2.0
200
200
300
350
400
50
100
15:0
200
300
3500
400
(a) KT. loss
EFTS2
(b) Mel reconstruction loss curves of EFTS2
Figure 6: Loss curves of the ablations
Table 5: Comparson of different aligners.
Model
CMOS
D-Hybrid (ours)
0
-o-Extemal-Repetinee (Ren al.
2021
-0.25
ID-Lnermal--MA (Kim et al.
2021
-0.10
Eexternal-Gaussian
Donahue
2021
-0.23
D
(ours)
-0.18
D-Internal-- Learnable (Elias et al. 2021
does not converge
D--neeral-GGuusiian (Mao al
2021
-0.18
D-Ieeranal- Gaussian-
Internal-Gaussian.
Gaussian-Boundaries Boundaries (ours)
-0.02
All these models are built up using the 2-aaye-hiirarcchica--AAA based waveform generator and deterministic alignment predictor (DAP). For both the two non-differentiable models, the convolu- tional prior network is excluded. The first variational prior of the two oon-differentiable models formulated by firstly mapping the text hidden representation æ6 to an input level Gaussian prior distribution, and then expanding the Gaussian prior through repetition. The phoneme durations extracted using MAS (Kim et al. 2021 for all those models using external aligners.
The CMOS results are presented in in Table and we have the following observations: 1) the model centernal-Learnable does not converge at all while other models are
using learnable upsampling able produce reasonable results; 2) the models with internal aligners outperform those using external aligners even if facilitated with the same upsampling approach, which demonstrates the im- portance of jointly learning alignment and speech generation. The proposed approach D-Internal- that uses Gaussian attention derived from token boundaries, significantly outperforms other upsampling approaches. The best model D-Hybrid, that combines D-Internal- Gaussian-e and --anternal-Caussinn Bounsiar--B further boosts the model performance. 3) D- Hybrid achieves a performance gain of 0. over ND-Internal-MAS, verifying the significance of the proposed differentiable aligner over MAS. We also notice that ND-Internal-MAS performs worse than VITS because VITS achieves comparable speech quality of our best perfforeno model, while there is notable performance gap between ND-Internal-MAS and D-Hybrid. One assump- tion is that the repeated latent variable of ND-Internal-MAS has very similar local representations, which leads to the large receptive field size required for the decoder network.
Couusiaan-Boundadd
15

---

Under review as a conference paper at ICLR 2023
Table 6: CMOS comparison between EFTS2 and bascline models
Model
CMOS
EFTS2
0
VITS
-0.02
EFTS-CNN + HiFiGAN
-0.31
Table 7: Comparison with previous text-to-waveform models
Model
High Efficiency
High Quality
Differentiable
End-to-End Training
Clarinct (Ping al.
2019)
WaveTacotron (Weiss
2021
EATS (Donahue el al. 202
EFTS-way (Miao el al.
2021
FastSpeech2s (Ren el
2021
VITS (Kim al. 2021
NaturalSpeech lan al. 2022
EFTS2
E COMPARISONS WITH BASELINE MODELS AND OTHER TEXT-TO- WAVEEOOR MODELS
As EFTS2 achieves very similar MOS score to VITS, we further present the side-by-side CMOS results in Table 6 As be seen, EFTS2 slightly outperforms VITS with CMOS score increase of 0.02. In Table 7 we compare the advantages of EFTS2 with previous text-to-wavefone models in terms of training pipelines, differentiability, model performance, and model efficiency. EFTS2 is the only differentiable model that allows for end-to-end training, high-quality and high-efficiency generation.
16


Reviews:
[{"review_id": "PMM3hvo5vE", "reviewer": "Reviewer_eJWJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a hierarchical VAE-based end-to-end TTS model. EfficientTTS 2 replaces the flow-based prior in VITS with convolution-based hierarchical VAE priors and introduces a fully differentiable aligner for duration modeling. EFTS 2 shows a comparable result to VITS and Your-TTS with faster inference speed.", "summary_of_the_review": "The proposed method has good sample quality similar to VITS, which is the basic variational framework, and some efficiency was obtained by replacing the flow-based prior distribution model with a hierarchical prior. Some experimental results are required to show the importance of the contributions of EFTS 2.", "strength_and_weaknesses": "Strengths\n\n* More efficient than VITS and YourTTS by using a hierarchical VAE prior rather than affine coupling blocks\n- Propose a fully differentiable aligner\n- The sample quality of EFTS2 is comparable to that of VITS. Also, the EFTS2-VC is slightly better than YourTTS with faster inference speed.\n- Provide the implementation of EfficientTTS 2\n\nWeaknesses\n* Marginal improvement in parameter efficiency (0.87x inference params of VITS) and inference speed (1.5x faster than VITS)\n* The motivation of a differentiable aligner is not well addressed in the result section. \n* In Figure 3, the F0 contour of EFTS2 does not seem stable. Please provide the samples with diverse $t_1$ and $t_2$. \n* Why not provide multi-speaker TTS results for EFTS 2-VC even though it uses a text encoder? \n\nComments\n* Since VITS is significantly faster than real-time (68x), 1.5 times faster than VITS doesn't seem great improvement. For the great improvement on the efficiency of the end-to-end TTS models, it is essential to consider both the modules for the prior distribution of the VAE and the waveform generator.\n* One of the main contributions is to propose a differentiable aligner and the paper argues that it is an advantage over the existing non-differentiable aligner. It will be helpful for readers to show that the proposed differentiable aligner is better in terms of training efficiency and performance compared to the existing non-differentiable aligner such as Monotonic Alignment Search (MAS).\n* Did you use $t_1=t_2=1.0$ for evaluation? Figure 3 only demonstrates the F0 contour when $t_2=0.8$. What is the effect of varying $t_2$? More analysis and samples for diverse $t_1$ and $t_2$ will help understand the role of each prior distribution. (Minor) What is $t_A$ in Fig 3? Scaling factor on alignment?\n* VITS works well for both TTS and VC on the multi-speaker dataset (VCTK). This work shows only the voice conversion result of EFTS 2-VC for the VCTK dataset. As EFTS 2-VC uses a text encoder for voice conversion, it seems that EFTS 2-VC can also be used as a multi-speaker TTS model. Why not provide the multispeaker TTS results?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MBosvhk4NV", "reviewer": "Reviewer_m4Dz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "* Offers an alternative differentiable monotonic alignment generator over EfficientTTS, VITS, FastSpeech 2, non-attentive Tacotron, Parallel Tacotron 2, etc.\n* Using variational inference, learns a 2-level hierarchical latent representation, which can be used to influence the diversity of speech samples.\n* Joint-trains a vocoder (Hifi-GAN derived) from the latent representations.\n* Presents results on TTS and voice conversion, demonstrating improvement over:\n  * TTS: EFTS-CNN, VITS and ablations of EFTS2\n  * VC: YourTTS\n", "summary_of_the_review": "I appreciate the hard work of the authors on this manuscript. It was a well-written and enjoyable read.\n\nMy feeling is that the main contribution of the work is in the alignment network, as that is one of the large unsolved problems in TTS. Unfortunately, I do not think that with a dataset like LJSpeech we have enough variation of durations to say one way or the other that this problem is solved. As I pointed out in the strengths/weaknesses, I believe the conditional independence assumptions between phoneme timesteps are still present (we are not optimizing the sequence loss of the durations), and that the work does not directly demonstrate what is gained by making the alignments differentiable, or make careful comparison to other alignment mechanisms as the original EFTS paper did in comparing the impact of changing just the alignment mechanism between the current popular alternatives in the same overall model.\n\nThe main aspect for improvement in the experiments section is in comparison to other works. I would like to see a more apples-to-apples comparison to EFTS (which I believe uses character inputs), and a CMOS comparison to VITS -- which has the same MOS score and so cannot be compared against easily. \n\nIn my opinion, the LJSpeech dataset is far too plain in the variation of durations in the data to say whether this method will extend well to more diverse datasets. In some sense LJ is the MNIST of TTS and has outlasted its usefulness for pushing SOTA in TTS. A more challenging dataset to test this on would be the Blizzard 2013 challenge set, or the LibriTTS dataset.\n\nI have a quibble with the use of the term \"end to end TTS\", as I mention in the strengths/weaknesses section. I think phonemes as input and heavy use of spectrograms in the training objective detract from the claim of this being end-to-end TTS. I would like to reserve the term for people who are working on the difficult challenges in mapping written-domain text to speech (e.g. learning verbalization in addition to pronunciation).\n\nFinally, the technical novelty is diminished since it is building heavily upon VITS, EFTS, and HifiGAN. Since the results are not a massive improvement over existing / prior art, and I question whether the evaluation results were a fair comparison in some cases, I think it will be better to either refine the work until it is clearly a step function in quality improvement (so the results cannot be doubted) or to find a speech-specific venue. The relevance of the alignment mechanism is already pretty specific to TTS and VC, so it may be of less general interest to the ICLR audience.\n\nI think a speech-specific conference such as ICASSP would be a better venue.", "strength_and_weaknesses": "# Strengths:\n* Making the alignment procedure fully differentiable is potentially useful.\n* Using a hierarchical VAE for the acoustic model appears to be beneficial.\n\n# Weaknesses:\n* Variational Alignment Predictor\n  * The variational alignment predictor’s loss has independence assumptions in its reconstruction loss that do not optimize the overall sequence probability of durations, which is a common weakness in duration prediction models.\n  * The only direct apples-to-apples comparison to other duration models in the paper is “EFTS 2 (DAP)” which does not have details provided that I could find. Since the primary contribution of the paper is the differentiable variational alignment predictor, I think a much deeper comparison to other methods is warranted (e.g. EFTS 2 with the alignment model replaced with that of VITS, EFTS, FastSpeech 2, or Parallel Tacotron 2, or EFTS 2 with a stop gradient appropriately placed to show that the differentiability of the alignment model is important)\n* The comparison with EFTS does not appear fair as EFTS does not use phonemes as input. \n* Does not present CMOS tests for some crucial comparisons. The VITS and EFTS2 results have the same mean MOS. The results would be stronger with a direct SxS comparison between VITS, EFTS, and EFTS 2, as MOS is not a sensitive enough metric to differentiate between systems that are substantially similar in performance.\n* The claim of being an “end-to-end” TTS model is weakened by the use of phonemes as the input (which eliminates broad swathes of problems in end-to-end TTS around verbalization, pronunciation learning, and more). I suggest removing all references of being an “end-to-end” TTS model to avoid confusion.\n* The use of hand-designed representations (linear-scale and mel-scale spectrograms) is a further downside in the claim that this is an end-to-end TTS model. While it’s true that hand-designed intermediate representations are not present in this model (e.g. as they are in FastSpeech), the use of hand-designed losses that guide e.g. the latent z_1 and z_2 towards representing all of the information contained within a mel spectrogram, is very similar. \n* The control features demonstrated do not seem particularly beneficial for real world use cases, as playing with latent variables does not provide many affordances. A more compelling argument would be results (e.g. in the samples page) showing how to use control to achieve a desired curation result – e.g. emotion control, copying style from a seed sample, changing speaking rate or pitch range or word level emphasis, etc. As stated, the latent variables seem to show some measure of interpretability but it's hard to get a sense how useful that is without deeper study / explication.\n\n\nNits:\n* Equation 14 appears incorrect unless $z_1$ and $z_2$ are independent. Should it be $q(z|y) = q(z_1|y_{lin}) q(z_2|z_1,y_{lin})$ ?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BBZkJ1Daem", "reviewer": "Reviewer_EWgd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors designed and experimented a one-stage text-to-speech model using a fully differenctiable method.", "summary_of_the_review": "- There are doubts about the results presented by the authors. In addition, authors will have to diversify their comparative models.\n- According to the results, there are no improvements other than efficiency, so some parts need to be corrected.", "strength_and_weaknesses": "[Strength]\n* Inference efficiency is higher than the previous work the authors compared.\n\n[Weaknesses]\n* Synthesis efficiency improved, but the quality seems to have degraded.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject