HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis
Authors: Sang-Hoon Lee
Seung-Bin Kim
Ji-Hyun Lee
Eunwoo Song
Min-Jae Hwang
Seong-Whan Lee
Venue: NeurIPS 2022 Accept (2022)
Keywords: Speech Synthesis
Text-to-Speech
Self-supervised speech representation

Abstract: This paper presents HierSpeech, a high-quality end-to-end text-to-speech (TTS) system based on a hierarchical conditional variational autoencoder (VAE) utilizing self-supervised speech representations. Recently, single-stage TTS systems, which directly generate raw speech waveform from text, have been getting interest thanks to their ability in generating high-quality audio within a fully end-to-end training pipeline. However, there is still a room for improvement in the conventional TTS systems. Since it is challenging to infer both the linguistic and acoustic attributes from the text directly, missing the details of attributes, specifically linguistic information, is inevitable, which results in mispronunciation and over-smoothing problem in their synthetic speech. To address the aforementioned problem, we leverage self-supervised speech representations as additional linguistic representations to bridge an information gap between text and speech. Then, the hierarchical conditional VAE is adopted to connect these representations and to learn each attribute hierarchically by improving the linguistic capability in latent representations. Compared with the state-of-the-art TTS system, HierSpeech achieves +0.303 comparative mean opinion score, and reduces the phoneme error rate of synthesized speech from 9.16% to 5.78% on the VCTK dataset. Furthermore, we extend our model to HierSpeech-U, an untranscribed text-to-speech system. Specifically, HierSpeech-U can adapt to a novel speaker by utilizing self-supervised speech representations without text transcripts. The experimental results reveal that our method outperforms publicly available TTS models, and show the effectiveness of speaker adaptation with untranscribed speech.

HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis
Sang-Hoon Lee
Seung-Bin Kim?
Ji-Hyun Lee²
Eunwoo Song?
Min-Jae Hwang
Seong-Whan Lee
Department of Brain and Cognitive Engineering, Korea University, Scoul, Korea
Department of Artificial Intelligence, Korea University, Seoul, Korea
NAVER Corp., Seongnam, Korea
2 (sh lee, sb-kim, jihyun-lee, 1korea ac kr
3 (eunwoo. (eunwoo.song, song, min-jae haanjja..wanngeeee. wwwaann/dnvverrrpp.co com
Abstract
This paper presents HierSpeech, a high-quality end-to-end text-to-speech (TTS) system based on hierarchical conditional variational autoencoder (VAE) utilizing self-supervised speech representations. Recently, single-stage TTS systems, which directly generate raw speech waveform from text, have been getting interest thanks to their ability in generating high-quality audio within fully end-to-end training pipeline. However, there is still a room for improvement in the conventional TTS systems. Since it is challenging to infer both the linguistic and acoustic attributes from the text directly, missing the details attributes, specifically linguistic in- formation, is inevitable, which results in mispronunciation and over-smoothing problem in their synthetic speech. To address the aforementioned problem, we leverage self-supervised speech representations additional linguistic representa- tions to bridge an information gap between text and speech. Then, the hierarchical conditional VAE is adopted connect these representations and to learn each attribute hierarchically by improving the linguistic capability in latent representa- tions. Compared with the state-of-the-art TTS system, HierSpeech achieves +0.303 comparative mean opinion score, and reduces the phoneme eттоr rate of synthe- sized speech from 9.16% 5.78% on the VCTK dataset. Furthermore, we extend our model to HierSpeech- U, an untranscribed text-to-speech system. Specifically, HierSpeech-U can adapt to a novel speaker by utilizing self-supervised speech representations without text transcripts. The experimental results reveal that our method outperforms publicly available TTS models, and show the effectiveness of speaker adaptation with untranscribed speech.
1 Introduction
Text-to-speech (TTS) systems have undergone significant improvements in synthesizing high-quality speech from text sequence. Conventional TTS systems generally consist of two parts; an acoustic model and vocoder. First, acoustic models (Wang et al. 2017] Shen et al., (018) have shown the success synthesizing acoustic features (e.g., Mel-spectrogram) as an intermediate feature from
"Corresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

text sequence, and the vocoder (Oord et al. 2016) converts the acoustic features into raw waveforms consecutively.
However, previous TT'S models are subject to two limitations: 1) although speech consists of various attributes (e.g.. pronunciation, rhythm, intonation, and timbre) (Qian et al. 2020 Choi et al. 2021) most previous models synthesize acoustic features from the text sequence at once (Ren et al. 2019
which exacerbates the one-to-many mapping problem; and 2) in the two-stage pipeline, each part of the TTS system should be trained independently, which results in the degradation of the audio quality
(Ren et al. 2021a bj Lee et al. 2021b).
Recently, single-stage end-to-end TTS models, which directly generate a raw waveform from text, successfully reduce these limitations of the two-stage pipeline. For instance, VITS (Kim al. )021) adopts variational inference augmented with the normalizing flow (Kim et al.) 2020) and adversarial training (Kong et al.) 2020) to improve the expressiveness of the model, which can learn rich representations from speech data and synthesize waveforms directly from the text. However, despite efforts to reduce the information gap between text and speech, these models are subject to speech mispronunciation and over-smoothing problems. In the process of synthesizing speech, they still generate all the acoustic attributes from text sequence the same time. Therefore, missing the details of some attributes between text and speech, specifically linguistic information, is inevitable.
To bridge the information gap between text and speech, we adopt supervvsd speech repre- sentations as additional linguistic representations. Trained with large-scale speech dataset, these representations can learn useful information without using labeled data. Previous studies (Shah et al., 2021 Choi et al. 2021) also reveal that the representations from the pre-trained model contain rich information trained from a large-scale speech dataset. In particular, the representations from the middle layer of the pre-trained model contain rich linguistic information which has a characteristic of pronunciation. As result, it has been successfully utilized for various speech tasks such as speech recognition (Baevski et al. 2020 2021 voice conversion (Choi et al. [2021] Lee et al., 2021a), and speech resynthesis (Polyak al. 102T). However, these useful representations have not yet received significant attention in TTS systems due to the difficulty to utilize in generative model.
In this paper, we present HierSpeech, which is hierarchical conditional variational autoencoder using self-supervised speech representations for end-to-end TTS. We leverage self-supervised speech representations (Baevski et al. (2020) to enrich the linguistic information in latent representations, and to learn each attribute hierarchically from linguistic representations to acoustic representations. Although the self-supervised representations of text were previously studied for TTS (Jia eJi el. 2022) to the best of our knowledge, this is the first study that involves the investigation of self-supervised speech representations for single-stage end-to-end TTS. Based on the state-of-the- TTS model (Kim) et al. 202T). we demonstrate that self-supervised speech representations can reduce the information gap between text and speech by significantly improving the reconstruction quality. Compared with the sfate-of-the-ant TTS model, HierSpeech achieves +0.303 comparative mean opinion score (CMOS) and reduces the phoneme error rate from 9.16 5.78 for the VCTK dataset.
Based on the pre-trained HierSpeech, we also present novel adaptive TTS frameworks. Specifically, extend HierSpeech to HierSpeech-U, which can adapt the pre-trained model to synthesize the voice of novel speakers with untranscribed speech data. To extract the linguistic representations without text transcripts, HierSpeech-U can utilize the self-supervised speech representations, and learn the acoustic representations from untranscribed speech data. The results show that the synthetic quality of HierSpeech-U U trained only with the speech data is comparable that of HierSpeech trained with the text-speech pairs. The main contributions of this paper are as follows:
We propose HierSpeech, which is hierarchical conditional variational autoencoder using self-supervised representations that can improve the linguistic information in the latent repre- sentations, and learn attributes hierarchically. This significantly improves the reconstruction quality by bridging the gap between text and speech.
We investigate self-supervised speech representations for the TTS system by thoroughly conducting more than 30,000 GPU hours of experiments. Audio samples are available at httpp://shh- prm1. gi wt/iii/iiiee wwtp/hhrsseech-eeeeee demo/
To utilize untranseribed speech data, we extend the model to HierSpeech-U U which can adapt the TTS model without text transcripts. The results also reveal that the adaptation quality without text transcripts is comparable that of the baseline model using text transcripts.
2

---

XLS-R (wav2vec 2.0)
Text sequence
Self-supervised linguistic representations
Acoustic representations
Figure 1: Hierarchical text-to-speech pipeline.
2 HierSpeech
In this paper, we propose the hierarchical conditional variational autoencoder using self- supervieee representations for TTS. We use the self-supervised speech representations to improve the linguistic information in the latent representations, and to learn each representation hierarchically. Moreover, we extend HierSpeech to HierSpeech-U, which can adapt the model without text transcripts. The details of the speech representations, architecture, and untranscribed TTS method are described in the following subsections.
2.1 Speech representations
Acoustic representations In TTS systems, Mel-spectrogram widely used intermediate acoustic feature, which is converted from the waveform using the short-time Fourier transform (STFT). However, this acoustie feature consists of various attributes such as linguistic information (e.g., pronunciation) and style information (e.g., rhythm, intonation, and timbre). Hence, synthesizing this rich feature from only the text simultancously exacerbates the one-to-many mapping problem, and it is difficult to extract the expressive linguistic information from the spectrogram. To mitigate this issue, we adopt additional linguistic features to map the text and acoustic features as follows.
Linguistic representations To bridge the gap between text and speech, we use self self-supervised speech representations for additional intermediate linguistic features as shown in Fig. [ Previous studies reveal that the extracted features from the pre-trained model, such wav2vee 2.0, contain rich linguistic information, and this can improve various tasks such as automatic speech recognition (ASR) and speech translation. (Shah et al. 2021 Choi al. 2021) also shows the representations from the middle layer of wav2vee 2.0 contain a large proportion of linguistic information relevant to pronunciation. Specifically,
use the self supervuee speech representations from the 12th layer of the XLS-R (Babu et al.) 202T) which is pre-trained wav2vec 2.0 with a large-scale cross-lingun speech dataset. In addition, we also conduct various experiments investigate these representations for the TTS system as detailed in Section 3.3 and Appendix C.
2.2 Hierarchical variational inference
To connect the two parts of TTS systems, the previous end-to-end TTS model, VITS (Kim al. 2021) adopts conditional variational autoencoder that maximizes the evidence lower bound (ELBO) over the intractable marginal likeliiiddood of data log Po (zlcc):
log (og()()))) 2 (x)x)) log (x(2)) - log lo (2(2)))
(1)
po(2)c)
where po(=|c) is prior distribution over latent variables = given condition c, (=(2)) is the likelihood function that generates data v given latent variables z as decoder, and (=(x))) the approximate posterior. Subsequently, VITS uses the normalizing flow to improve the expressiveness of the prior distribution and adversarial training the waveform domain. Based on VITS, HierSpeech uses hierarchical conditional variational autoencoder to connect multi-level intermediate representations via disentangled latent variables speech representations, and learns them in an end-to-end manner. Unlike recently proposed hierarchical VAE (Vahdat and Kautz, 2020: Lee et al. (022a) which uses top-down path networks conditioning each other, we approximate them separately from different representations of speech. As shown in Fig 2 the acoustic posterior distribution and linguistic posterior distribution are encoded separately by acoustic encoder Φa and linguistic encoder di respectively. To disentangle each latent variable, we use the linear-scale spectrogram of the target speech Expec for the rich acoustic representations E,, and the output of the 12th layer of XL.SRR Tw22 for the rich linguistic representations 21. The optimization objective of HierSpeech can be expressed
3

---

Training
Wavelörit
Inference
Planene
Waveform
Flow
Decoder
Flow f
Decoder
Prodicter
D||0
Flow fa"
Sinne
Monotonic Aliymmen
fi
KL
Sampling
Acoustic
Linguialic Enerddrr
Aupuce
Projectoon
Projection
STFT
Dunsion Profictor
Stoclendiz Duntion Predictor
Soap grddien
Wavellon
Text Enconer
Test Ficendr
Waveform
Noizz
Noise
Phonone
Phonene
Figure 2: Overall framework of HierSpeech.
as follows:
log 2 (= log Po. (x)2z) - log Id. (Za ([apec))
log Lov (2
(2)
pa. (Za (21)
pe, (2)lc)
where = [ 21. 0 [0d, fa. 01), Φ [0a, Φ1), qo., (#a ( spec)) and Jo, (21 (ww2e) are the approximate posterior for acoustic and linguistic representation, and po, (2) is prior distribution over linguistic latent variables given condition c, Pu., (za (z) is prior distribution over acoustic latent variables za. where 2 is sampled from qs, (z 22w)). and pe, (ar za) is the likelihood function that generates data x given latent variables za as a decoder Oa. For the reconstruction loss, we use Mel-reconstruction loss free which minimizes the !1 distance of the Mel-spectrogram between the ground truth and reconstructed waveform using STFT and Mel-scale transformation.
Acoustic encoder and waveform decoder The acoustic encoder &a composed of non-casual WaveNet residual blocks which are layers of dilated convolutions with gated activation unit and skip connection. Thereafter, the output then fed the projection layer to sample the acoustic representations Za from the mean and variance of the posterior distribution using the reparametrization trick. During training, the sliced 2n fed a waveform decoder reconstruct the raw audio æ. We use HiFi-GAN generator G (Kong et al. 2020) as waveform decoder 9g which consists of stack of transposed convolution and recepteeeeptive field fusion module. For adversarial feedback, also use the multi-period discriminator D) to capture the different periodic features of the waveform.
Lodv(D) = (( (D(x) - 1)a D(G(2a))
(3)
Lodv 02) E(2.) (D(G(z))) 1)²] 1)}
(4)
where X is the ground truth waveform. matching loss Lym which minimizes the 11 distance each discriminator's features between the ground truth and reconstructed waveform.
To ensure stable training, we use the additional feature
Linguistic encoder and phoneme predictor The linguistic encoder Oi has the same structure as the acoustic encoder. However, we use self-supervised speech representations Tw2e as the input of the linguistic encoder which is extracted from the pre-trained XLS-R, and the linguistic representations is extracted. To enforce linguistic characteristics, is fed to the auxiliary phoneme predictor. We minimize the connectionist temporal classification (CTC) loss Late to optimize the linguistic encoder and phoneme predictor. However, to remove the additional projection layer, the projected mean and variance from the linguistic encoder are also directly used as the acoustic prior distribution with weight-sharing between 0g and Φi. To maintain hierarchy, the representation 2, is transformed by normalizing flow. Hence, the KI divergence between acoustic prior and posterior is minimized as:
Lai log 9o. (Zal (ppec) logg pe,, (2a (w2))
(5)

---

Because the acoustic prior distribution is obtained from the linguistic information, to bridge the gap between distributions, we use the normalizing flow to disentangle the information from the acoustic posterior and increase the expressiveness of the acoustic prior distribution.
po. (Zu Twze) = Ma((((aaa) ne, (Iwze), 0o, Za qa. (Za [Espec) (zai Mob (Fspec) , 4b. (Xspee)
(6)
Text encoder The text encoder 9 consists of a stacked feed-forwart Transformer network using relative positional representation. The phoneme sequence Ctext fed the text encoder, and the projection layer is used to produce the mean and variance for linguistic prior distribution. To align the text with the linguistic representations of speech, we use monotonic alignment search (MAS) of (Kim et al., 2020) which searches the alignment A satisfied with maximizing the likelihood of data:
L log go, (21 - log (ZilCrent, A)
(7)
We also use the normalizing flow to increase the expressiveness of the linguistic prior distribution.
po, (zalCreat, A) N (f( no, (Crents A), de, (Ctezt, A)))d((00((z))00))
(8)
qd (w2e) = (2z Mos (Iw2o), Jo (Iw2.))
To sample the duration given phonemes, we adopt a flow-based stochastic duration predictor of (Kim et al.] (02T) trained with maximum likelihood estimation. We use the negative variational lower bound of it as duration loss Ldur- For the multi-speaker settings, we add the global speaker embedding the residual block of the acoustic/linguisti encoder, residual block in normalizing flow, stochastic duration predictor, and decoder.
Total loss
The final objectives for HierSpeech can be expressed as follows:
Ceotal = Lat + AuzCra + AreeCree + detehote Adur Audo Ladu 0a) fmmLim (9)
2.3
Untranscribed text-to-speech
For the untranscribed TTS model (HierSpeech-UD) we train the model using a style encoder (Min
et al. 2021) which extracts style embedding from speech as global conditioning (Jia et al. 201 8). We use a linear-scale spectrogram as the input of the style encoder. After pre-training with the multi-speaker dataset, we adapt the model to novel speaker without text transcripts. Through self- supervised speech representations, the pre-trained linguistic encoder is able to extract rich linguistic representations from speech without text transcripts. Hence, HierSpeech-U synthesizes speech with the style novel speaker by fine-tuning the acoustic encoder, the normalizing flow blocks of the acoustic prior, and decoder with only speech data.
3 Experiment and Result
3.1 Experimental setup
Datasets We train the models using the VCTK and LibriTTS datasets. The VCTK dataset contains 46 hours of audio for 108 speakers (Veaux et al. (017). We use the train-clean subsets of the LibriTTS dataset (Zen al. 2019) which contains 110 hours of audio for 1,151 speakers. For both datasets, we downsample the audio at 22,050Hz. Following the (Lee et al. 2021a), we use only non-parallel data for the training dataset which consists of different utterances for each speaker, and we use the parallel data for the test dataset which consists of 25 same utterances in the VCTK dataset. For the LibriTTS dataset, we randomly select two samples from each speaker for the test dataset. For speaker adaptation, we randomly select 98 speakers as the base speakers and 10 speakers (5 males and females) as the novel speakers. To evaluate the speaker adaptation performance for the number of speakers, we also train the model using 1, 1,249 speakers from the VCTK and LibriTTS datasets.
Preprocessing For the input of the acoustic and speaker encoder, we use linear-scale spectrogram with 513 bins which is transformed from the audio. For reconstruction loss, we use Mel-spectoogram with 80 bins. For the input of the linguistic encoder, we use the output from the middle layer of the XLS-R (0.3B) (Babu al. 202 which is pre-trained wav2vec 2.0 with large-scale cross-lingut
5

---

speech dataset To extract the self- speech representations, we use the downsampled audio 16,000Hz as an imput to the XLS-R, and the extracted representations are upsampled to map the spectrogram by interpolation 3] For the input of the text encoder, we use the text sequences converted to International Phonetic Alphabet (IPA) sequences using the open-source Phonemizer] Following (Kim et al. 202T), we intersperse the phoneme sequences with a blank token. However, we use phoneme sequences without a blank token for the target sequence of the phoneme predictor
Training We train HierSpeech using the AdamW optimizer (Loshchilov and Hutter 2019) with 3 = 0.8, $2 = 0.99, and weight decay X = 0.01, and apply the learning rate schedule by the decay of 0.9991/8 / at an initial learning rate of 2 x 10 for the generator and discriminator. We train HierSpeech with batch size of 256 for 600k steps using mixed precision training four NVIDIA A100 GPUs. For speaker adaptation, we fine-ture tune-tue novel speakers for only 2000 steps. For the ablation study, we train the model with batch size of 64 for 300k steps.
U with few samples of
3.2 Evaluation metrics
Subjective metries We conduct two mean opinion score (MOS) tests for naturalness and similarity. For the naturalness MOS (nMOS), each sample from the target and synthesized speech are rated by at minimum of 20 listeners on a scale of 1-5. For the similarity MOS (sMOS), the synthesized and target speech are presented to minimum of 20 listeners on scale of 1-4. The nMOS and sMOS are reported with 95% confidence intervals.
Objective metrics For the naturalness measurement, we calculate the phoneme error rate (PER) and word error rate (WER) of the synthesized speech. We used the fined-tuned wav2vec 2.0 (Bavsski et al. 2020) without language model for automatic speech recognition. For the similarity measurement, we conduct three metrics: the equal error rate (EER) of the automatic speaker verification (ASV), the Mel-cepstral distortion (MCD), and the FO root mean square error (RMSE,o). yo). We used a pre-trained ASV model (Chung et al. 2020). The ASV model is trained using a large-scale speech dataset, VoxCeleb2 (Chung a 20188) via online data augmentation (Heo al., (020). We calculate the EER where both acceptance and rejection rates are equal for the sample pairs from the synthesized and target speech (2, 700 108 291, 600). We compute the MCD and RMSE,0 by applying dynamic time warping (DTW) between the synthesized speech and target speech. We conducted a duration prediction performance evaluation using the average absolute differences of the utterance duration (DDUR) (Zhang et al.) 2019). To compare the inference speeds, we calculate the Speed which is the synthetic waveform frame per second, and the synthesis speed over real-time (Real-time).
3.3 Analysis of self-supervised representations
Previous researches (Fan et al., 2020; Choi et al. 2021 reveal that the representations from the front layer of wav2vec 2.0
Table 1: Results of different input
are clustered by each speaker. Hence, (Choi et al. 2021)
features for the speaker encoder.
uses the 1st layer of pre-trained wav2vee 2.0 as an input to
the speaker encoder. As input of the speaker encoder, we compared three different features; lincar-scale spectrogram (linear spectrogram), the representations from the 1st layer of XLS-R and 12th layer of XLS-R. Table [ shows that the linear spectrogram has better transfer performance than the representations of XLS-R. Although the speaker embedding extracted from the 1st layer of the XLS-R has better performance than one of the 12th layer, the results show that the linear spectrogram contains a significant amount of high-resolution information for speaker characteristics. In this respect, we use linear spectrogram the input for the speaker encoder, and can conclude that there is a little speaker information to transfer speaking style in the representations of the middle layer of XLS-R.
Inpur feature
PER
WER
FER
Linear spectrogram
5.45
20.13
2.77
1st layer of XLS R
5.67
20.36
4.62
12th layer of XI.S-R
6.29
21.50
10.23
"Specifically, we use the 12th layer of the XL.S- The results detailed in Section 3.3 and Appendix C. We simply interpolate with respect to the time-domain. The ASR model uses the audio with sampling rate of 16,000 Hz. However, the TTS model use higher resolution audio for high-quality audio synthesis. //ttp://giitatti wawaaa..cm/bbaaa cwm/ httmm/bbottooo/hooemmi..eo
We observe that the blank token degraded the classification performance under the experimental conditions.

---

To evaluate the speech disentanglement performance with re- spect to the layer of XLS-R, we conduct frame-level speaker classification on the representations from specific layer of the XL.SR and the linguistic latent variables Ej respectively. Table 2 shows that the representations from each layer con- tain some speaker information to be classified. However, the speaker information is reduced through the linguistic encoder, and the žσ from the 12th layer of XLS-R has the lowest accu- racy. Hence, we representations from the 12th layer for the linguistic encoder. It should be noted that all representations, with the exception of the 23th layer, improve the TTS performance of HierSpeech when compared with that of VITS, which means that all representations contain rich information trained with large-scale speech dataset.
Table 2: Speaker classification ac-
curacy on linguistic representations from the different layer of XI.SR.
layer
Accuracy (w2v)
Accuracy (a))
1st layer 12th layer
79.12%
15.09%
59.46%
8.95%
24th layer
48.73%
11.35%
3.4 Evaluation on TTS
Table 3:
The TTS evaluation results on the VCTK dataset.
Method
iMOS
sMOS
PER
WER
EER
MCD
RMSE,o
DDUR
Speed (kIlk)
Real time
GT
4.06=00.22
3.34+0.03
5.64
18.
4.08
GT (HIFi-GAN)
4.03++0.02
3.30+-0.33
5.94
19:52
5.04
1.25
28.32
6,484.09
x294.06
Tacotron2
3.76:20.02
3.1620.03
22.18
4.18
263.94
x11.97
Glow-TTS
3.95=00022
3.09++0.03
11.77
26.40
5.33
4.31
32.98
0.38
1,410.75
x63.97
PortaSpeach
3.97-40.02
3.154-0.3
11:35
25.46
5.48
4.34
32.89
0.43
1,163.21
x52.75
VITS
4.02:#0.02
3. 3.19++0.03
9. 9.16
25.54
3.83
4.27
32.93
0.37
1,610.77
x72.83
HierSpeech (Ours)
4.04--002
3.22-40.03
5.78
19.55
3.74
4,05
32.15
0.33
1,459.95
x66.21
Table 4:
The speaker transfer evaluation results on the LibriTTS dataset.
Method
aMOS
sMOS
PER
WER
EER
MCD
RMSE,s
DDUR Speed (kH+)
Real-time
GT
4.0420.03
3.40:0.03
7.01
18.28
4.45
VITS
3.96
3.26.10.03
13.62
29.83
5.00
4.37
34.18
1.09
1,781.40
x80.78
HierSpeech (Ours)
3.98=0003
3.26++0.03
7.47
20.34
5.00
4.42
32.95
0.72
1,678.99
X76.13
Table shows that our model outperforms the other models with respect to the nMOS and sMOS for both datasets. In terms of the ASR evaluation, our model shows lower PER and WER than the other models by synthesizing speech with more accurate pronunciation. In terms of EER, all models have similar performance, which indicates the target speaker embedding (ID) is useful supervision for multi-speaker TTS. For the speaker transfer, we evaluate each model using the same speaker encoder in Table and subsection 3.5) In terms of the MCD and RMSE our model has the lowest error distance. Although VITS has better performance in inference speed, our model has a faster inference speed than two-stage end-to-ent TTS models. We also compare the models using speaker encoder, which are trained with VCTK and LibriTTS dataset in Table 4 Our model outperforms VITS in terms of nMOS and ASR evaluation. However, our model has slightly lower transfer performance in MCD. We found that transferring the speaker from the long sentence or noisy audio results in low performance of ASR for VITS and low speaker transfer quality for HierSpeech. We also conduct the comparative mean opinion score (CMOS) tests between the models trained with each dataset; VCTK and LibriTTS, Table 5 also shows that our model has a better performance than VITS in CMOS evaluation with r-test valuee. Also, our model achieves -0.096 CMOS compared to ground truth (GT) audio on the VCTK dataset. However, Table 5bbshoss that our model has -0.505 CMOS compared to GT, and it means that there is a room for improvement in TTS system by improving the expressiveness and robustness of the model.
Table 5: CMOS comparison. Positive score indicates that HierSpeech is rated better than the baseline.
(a) The CMOS results the VCTK dataset.
(b) The CMOS results the LibrITTS dataset.
Method
CMOS
p-value
Method
CMOS
p value
HierSpeech (Ours)
HierSpeech (Ours)
-0.09
0.003
GT
-0.505
<10-104
VITS
+0.303
<10 24
VITS
+0,297
<10-
7

---

Table 6: Results for untranscribed text-to- sexteto-ppeech. We compare few-shot speaker adaptation perfor- mance of HierSpeech-U with that of HierSpeech. Both models use the pre-trained HierSpeech which
trained using VCTK and LibriTTS datasets. We used 10 unseen speakers of VCTK dataset as novel speakers, and fine-tuned each model with 20 samples from each speaker.
Method
Transcript
nMOS
sMOS
PER
WER
EER
MCD
RMSE,a
DDUR
GT
4.13++.110
3.38-0.10
4.26
16,69
4.14
HierSpeech
/
4.09+0.10
3.18-0.11
4.40
16.95
6.40
3.96
29.56
0.28
HierSpesch-U
x
4.08:10.09
3.15-0.12
3.71
15.85
6.40
4.09
30.64
0.36
3.5
Untranscribed text-to-speech
To evaluate the untranscribed text-to- sox-eto-peeech performance of HierSpeech-U, we compare the few-shot speaker adapta- tion performance of HierSpeech-U with one of peerSp which is fine-tuned by unseen speakers using few audio- text pairs (Arik et al. 2018). Both models use the pre- trained HierSpeech which trained using VCTK and Lib- riTTS datasets. For the speaker adaptation of HierSpeech, we fine-tune the entire model. Although HierSpeech-U is fine-tuned without text transcripts, Table O shows that
HerSppach HerSpeach-U QT
Equal enernat (8)
HierSpeech- has comparable performance to HierSpeech in terms of nMOS and sMOS. Moreover, HierSpeech-U achieves a lower PER and WER than that of HierSpeech, which is slightly increased compared to PER 3.58 and WER 15,50 for the pre-trained model. Although there is limita- tion in adapting duration, HierSpeech-U U is able adapt the speaker in terms of voice by utilizing the linguistic repre- sentations from the self-supervised speech representations without text transcripts. Note that we fail to fine-tune VITS without text transcripts in that the PER for VITS increases from 7.47 to 12.27. Furthermore, we evaluate the adaptation quality with different numbers of adaptation samples (1, 5, 10, and 20) in Figure 3 We also investigate the effectiveness for different numbers of pre-trained speakers (98 from VCTK and 1,151 from LibriTTS) and the evaluation results described in Appendix C.
samples
The EER results for different numbers of adaptation samples.
Figure 3:
3.6 Ablation study and hyperparameter search
To compare the effectiveness for the number of normalizing flow, we train VITS which has the same number of flow blocks (4 8) with HierSpeech. However, we observe that increasing the number of flow blocks does not always indicate an improvement model performance, resulting in the degradation of the pronunciation and audio quality as shown in Table 7 Adding a phoneme predictor (PP) the posterior encoder of VITS improves the PER and WER by guiding the alignment search. Removing the acoustic encoder (AF) and synthesizing waveform directly from the linguistic representations from the 12th layer of XLS-R degrade the performance, and it also has higher Mel-spectrogram reconstruction error.
Table 7: Results of the ablation study.
Method
PER
WER
EER
MCD
VITS flow w PP
12.24
30.62
3.85
4.36
13.42
32.77
4.00
4.36
7.60
22.98
3.74
4.17
HierSpeach AE w.o AE and PP
6.25
20.89
3.48
4.15
8.00
23.84
3.63
4.25
16.70
37.43
3.89
4.39
We conduct hyperparameter search and ablation study of the phoneme predictor in Appendix C. The proper value of Acte 45 increases the overall performance in all the objective metrics. In contrast, when using an excessively large Acte- the KL divergence between the acoustic and linguistic distribution increases, which decreases the performance of audio quality with noisy sound. For better generalization, we attempted to combine label smoothing with CTC loss (Kim et al.) 2018) and conduct experiments on data augmentation for the effective speech disentanglement of linguistic information following (Choi et al. 202T). However, label smoothing and data augmentation lead to the problem for KL divergence optimization, thus resulting in the degradation of the synthesis quality.
8

---

3.7 Evaluation on VC
To evaluate speech disentanglement, we compared the voice conversion (VC) task of our model with three models; VC (Qian et al. 2019). VoiceMixer (Lee et al.) 2021a), and VITS (Kim et al., 2021). ). For fair comparison, we train all models using the same dataset. However, note that Vu and VoiceMixer do not use the phoneme information. Following (Lee) et 2021aa,, we randomly select 20 speakers. Thereafter, a single audio sample is selected from each speaker, and then all the possible pairs of samples (20 x 20 = 400) are produced for evaluation. Table shows shows that model outperforms the other models in nMOS and ASR evaluation metrics. Although VITS has similar performance in the EER evaluation, our model has better performance in ASR evaluation, which means that our model can disentangle content and style information with small loss of content information. We also observe that using the representations from the middle layer of XLS-R improves the PER performance when compared with that of the previous layer, and the auxiliary phoneme predictor also helps retain the linguistic information.
Table 8: The VC evaluation results on the VCTK.
Model
#MOS
sMOS
PER
WER
EER
GT
4.19:10.03
3.37:10.04
5.20
14.05
0.98
AutoVC
3.74:10.05
3.04:10.05
59.42
89.01
12.50
VoiceMixer
4.00.10.05
3.11:10.05
12.67
31.80
9.35
VITS
4.06110.04
3.16:0.00
6.46
18.73
2.75
HierSpeech
1.004
3.16::0.05
5.51
16.25
2.75
4 Broader Impact
Recently, self- supervised speech representations have been utilized in TTS tasks (Siuzdak et al. 2022,
Kim et al. 2022 Du et al. 2022) and we see that our hierarchical speech synthesis structure using self-supervised speech representations can be also utilized for various tasks, First, the application of a low-resource language can utilize self-supervised speech representations to improve the synthesis quality. Second, cross-lingual speaker adaptation for dubbing can be used the film industry. However, recent speech synthesis systems such as TTS and VC can generate the audio with realistic sound, there an increased potential risk of harm, malicious use, and ethical issues. Specifically, these systems could be misused in various manners, such fake news generation, voice spoofing, and unauthorized use of web crawl speech data. To mitigate these issues, fake audio detection is studied (Singhal et al. 2019 Tak et al.) 2021). Moreover, provide mitigation strategy for the proposed system by releasing the fake audio detection model with an ensemble of discriminators that can distinguish real and fake audio based results of multiple discriminators. Although it is difficult to differentiate in human evaluation score, the model achieves a 16.59% EER for fake audio detection on 1,852 test samples of the ground-truth and synthesized speech from our model.
5 Conclusion
We presented an end-to-end TTS model, HierSpeech, which can learn and synthesize speech from text through hierarchical intermediate representations an end-to-end manner. By bridging the gap between text and speech through self-supervised speech representations, the proposed model significantll improved the reconstruction quality. We successfully demonstrated that our hierarchical conditional variational autoencoder can improve linguistic capability in latent representations, and learn each attribute hierarchically using self-supervised speech representations. We thoroughly conducted more than 30,000 GPU hours of experiments on self-supervised representations for the TTS system, and we hope that these results can serve as a basis for future speech research. Furthermore, we also demonstrate the effectiveness of novel speaker adaptation framework without text transcripts. We see our hierarchical structure extending to cross-lingual TTS systems or other low-resource TTS systems. Also, we will try improve the expressive and robustness of model by modeling prosody (Im et al.) (202)) and noise from speech (Sacki et al., (022). However, the single-stage end-to-end TTS model is limited in terms of computational complexity in that the training process requires 20 days using four A00 GPUs Hence, in future works, an attempt will be made to decrease the computational cost without quality degradation by adopting iDWT (Lee al.; (022h) in the decoder and simplifying the discriminator to facilitate more rapid training (Andreev et al. 2002).
and replace the decoder with an diffusion- frfdueiio-bbaeedd neural vocoder (Koizumi et al. 2022b a). In addition,
we discussed the potential positive and negative impact of our model in Section 4 To mitigate the negative impact, we will release fake audio detector that can distinguish between real and fake audio.

---

Acknowledgements
This work was supported by Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea governmen((MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University). No. 2019-0-01371, Development of Brain- inspired AI with Human-like Intelligence, No. 2021 100-0000, Artificial Intelligence Innovation Hub, and No. 2022-0-00984, Development of Artificial Intelligence Technology for Personalized Plug- and-Play Explanation and Verification of Explanation) and Clova Voice, NAVER Corp., Seongnam, Korea.
References
Pavel Andreev, Aibek Alanov, Oleg Ivanov, and Dmitry Vetrov. Hifi++: a unified framework for neural vocoding, bandwidth extension and speech enhancement. arXiv preprint arXiv:2203. 2022.
Sercan Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with few samples. Advances Neural Information Processing Systems, 31, 2018.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick Platen, Yatharth Saraf, Juan Pino, et al. XIs-r: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint 1rXiv:2111.09996 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework self-supervised learning speech representations. Advances in Neural Information Processing Systems, 33, 2020.
Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recognition. Advances in Neural Information Processing Systems, 34, 2021.
Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. Neural analysis and synthesis: Reconstructing speech from self-supervised representations. Advances in Neural Information Processing Systems, 34, 2021.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. In Proc. Interspeech 2018, pages 1086-1090, 2018.
Joon Son Chung, Jaesung Huh, Seongkyu Mun, Minjae Lee, Hee Soo Heo, Soyeon Choe, Chiheon Ham, Sunghwan Jung, Bong-Jin Lee, and lcksang Han. In defence of metric learning for speaker recognition. In Proc. Interspeech 2020, pages 2977-2981, 2020.
Chenpeng Du, Yiwei Guo, Xie Chen, and Kai Yu. Vqits: High-fidelit text-to-speech synthesis with self-supervised vq acoustic feature. arXiv preprint
2022.
Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu. Exploring wav2vec 2.0 on speaker verification and language identification. arXiv preprint arXiv:2012.06788, 2020.
Hee Soo Heo, Bong-Jin Lee, Jaesung Huh, and Joon Son Chung. Clova bascline system for the voxceleb speaker recognition challenge 2020. arXiv preprint 1XXi:20009.11133, 2020.
Chae-Bin Im, Sang-Hoon Lee, Seung- Boung-Bin e Kim, and Seong- Whan Lee. Emoq-tts: Emotion intensity quantization for fine-gre graingr controllable emotional speeeh. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6317-6321. IEEE, 2022.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen. Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in Neural Information Processing Systems, 31, 2018.
Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu. Png bert: Augmented bert on phonemes and graphemes for neural tts. In Proc. Interspeech 2021, pages 151-155, 2021.
10

---

Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances Neural Information Processing Systems, 33:8067-8077,, 2020.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pages 5530. PMLR, 2021.
Minchan Kim, Myeonghun Jeong, Byoung Jin Choi, Sunghwan Ahn, Joun Yeop Lee, and Nam Soo Kim. Transfer learning framework for low-resource text-to-speech using large-scale unlabeled speech corpus. In Proc. Interspeech 2022, pages 788-792, 2022. doi: 10.21437/mttesseecch. 2022-225.
Suyoun Kim, Michael Seltzer, Jinyu Li, and Rui Zhao. Improved training for online end-to-end speech recognition systems. pages 2913-2917, 2018.
Yuma Koizumi, Kohei Yatabe, Heiga Zen, and Michiel Bacchiani. Wavefit: An iterative and non- autoregressive neural vocoder based on fixed-point iteration. arXiv preprint arXiv:2210.01022 2022a.
Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. Specgrad: Diffusion probabilistic model based neural vocoder with adaptive noise spectral shaping. In Proc. Interspeech 2022, pages 803-807, 2022b. doi: 20.114///eerspecchh22222- 2022-3330
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances in Neural Information Processing Systems, 2020.
Ji-Hi Hyun Lee, Sang-Hoon Lee, Ji-Hoon Kim, and Seong- Whan Lee. Pvae-tts: Adaptive text-to- speech via progressive style adaptation. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6312-6316. IEEE, 2022a.
Sang-Hoon Lee, Ji-Hoon Kim, Hyunseung Chung, and Seong- Seong-Whan Lee. Voicemixer: Adversarial voice style mixup. Advances in Neural Information Processing Systems, 34, 2021a.
Sang-Hoon Lee, Ji-Hoon Kim, Kang-Eun Lee, and Seong- Whaan Lee. Fre-gan 2: Fast and efficient trequency-consistee audio synthesis. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022b.
Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. Bidirectional variational inference for non- autoregressive text-to-speech. In International Conference on Learning Representations, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. Meta-stylespeech: Multi-speaker adaptive text-to-speech generation. In International Conference on Machine Learning, pages 7748-7759. PMLR, 2021.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03490 2016
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proc. Interspeech 2021, 2021.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pages 5210-5219. PMLR, 2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang. Mark Hasegawa-Johnson, and David Cox. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pages 7836-7846. PMLR, 2020.

---

Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in Neural Information Processing Systems, 32, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. In International Conference on Learning Representations, 2021a.
Yi Ren, Jinglin Liu, and Zhou Zhao. Portaspeech: Portable and high-quality generative text-to-speech. Advances in Neural Information Processing Systems, 34, 202
Takaaki Sacki, Kentaro Tachibana, and Ryuichi Yamamoto. Drspeech: Degradation-robust text-to- speech synthesis with frame-level and utterance-level acoustic representation learning. In Proc. Interspeech 2022, pages 793-797, 2022. doi: 222447//meerspecch22222222999
Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah. What all do audio transformer models hear? probing acoustic representations for language delivery and its structure. arXiv preprint arXiv:2101.0003 2021.
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang. Rj Syan, et al. Natural tts synthesis by conditioning wavenet mel spectrogram predictions. In 2018 IEEE international conference acoustics, speech and signal processing (ICASSP), pages 4779-4783 IEEE, 2018.
Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin'ich Satoh. Spotfake: A multi-modal framework for fake news detection. In 2019 IEEE fifth interna- tional conference on multimedia big data (BigMM), pages 39-47. IEEE, 2019.
Hubert Siuzdak, Piotr Dura, Pol van Rijn, and Nori Jacoby. Wavthruvec: Latent speech representation as intermediate features for neural speech synthesis. In Proc. Interspeech 2022, pages 833-837, 2022. doi: 10.21 10.21445//mterspeehhh000 2000:eeeeseeeee22220770099999 10797.
RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A. Saurous. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In International Conference on Machine Learning, pages 4700-4709, 2018.
Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher. End-to-end anti-spoofing with rawnet2. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6369-63666 6373. IEEE, 2021.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 2020.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Superseded-cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit. 2017.
Yuxuan Wang, RJ Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv: 1703. 10/35, 2017.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. pages 1526-1530, 2019.
Jing- Jing-Xuan Zhang, Zhen- Hua Ling, Li-Juan Liu, Yuan Jiang, and Li- Li-Rong Dai. Sequence-to-seqee sequence acoustic modeling for voice conversion. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(3):631-644 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 4
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The details is described at Secotion 3.3 and the supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 3 and the supplemental material
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [Yes] See Appendix E
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes] See Appendix E
13


Reviews:
[{"review_id": "qbD_Sqw1y8h", "reviewer": "Reviewer_oiLb", "summary": "This paper studies end-to-end text-to-speech synthesis and few-shot speaker adaptation using untranscribed speech. In particular, the authors extend from the VITS model and leverage self-supervised speech representations as the intermediate latent variable to bridge the gap between text and acoustic features. An additional phoneme prediction loss is used to refine self-supervised representation, making it closer to flow-transformed acoustic latent z_a. In addition, the injection of the intermediate latent inferred from self-supervised model enables speaker adaptation using untranscribed data.", "questions": "1. Adding PP to the VITS posterior encoder is helpful. The posterior encoder in VITS is analogous to the acoustic encoder in HierSpeech. Have the authors tried adding PP loss to z_a also?\n2. What is “sliced” z_a?\n3. What is the “global conditioning” in Section 2.3 and how is it used? Is it different from speaker embedding? If it is different, does training on transcribed data require it?\n4. What is the speaker encoder and how is it trained?\n", "limitations": "Yes, the authors discussed it. It is nicely addressed and mitigation strategies are presented.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. An interesting approach to integrate linguistic representations learned by SSL models into the end-to-end TTS pipeline\n2. Provides a neat way of speaker adaptation with untranscribed data\n3. Phoneme prediction loss seems to help not only HierSpeech but also VITS\n4. Good ablation studies comparing different layers of wav2vec representations for HierSpeech and favorable results compared to the baselines (VITS).\n\nWeaknesses\n1. Some details are missing. See the question section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NeCDaJDx83n", "reviewer": "Reviewer_SZqL", "summary": "The authors propose a novel speech synthesis model which utilizes untranscribed speech data to improve the representation of the linguistic information. The authors combine the representation of wave2vec (self-supervision) as an input to a linguistic encoder which is used as a prior for an acoustic VAE. The authors augments the VAE usual ELBO with multiple terms to improve the audio quality, and also add KL divergence term between the acoustic prior and the linguistic prior (which is conditioned on text) to support speech synthesis. \nThe authors perform multiple experiments to demonstrate the effectiveness of their method, and ablation studies to justify their design choices. The authors also demonstrate how the proposed method allows untranscribed text-to-speech by fine tuning the acoustic encoder over audio-only data.", "questions": "* Lines 85-92 :This paragraph repeats the previous paragraph?\n\n* Line 121: NVAE is not a general hierarchical VAE but rather a particular hierarchical VAE for images\n\n* Line 127: Eq. 2 does not make sense. As it currently stands, the rightmost term has no effect on the generative process or the acoustic posterior.\nIn practice, this effect is established via the weight sharing of p_theta_a(z_a|x_w2v) = q_phi_l(z_lx_w2v) but this critical information is rather easy to miss. Please fix it by adapting equation 2 to explicitly show the effect of the weight sharing.\n\nI add here an example of formulating Eq. 2 with explicit weight -sharing: replace q_phi_l(z_l|w_w2v) with  p_theta_a(z_a|x_w2v) and separate Eq. (2) into two terms, ELBO + and KL( p(z_a|x_wv2) || p(z_a|c) )\nThis formulation explicitly shows that the conditional prior p(z_a|x_w2v) is regularized to match the linguistic conditional prior p(z_l |c)\n\n* Line 128: It could be useful to define A here\n\n* Line 283: extra \"The\"\n\n* Line 306: Which KL divergence problem did you encounter? Posterior collapse? Other problems?", "limitations": "The authors discussed the limitations of the proposed method.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The audio quality of the provided samples is very good compared to competing methods.\n* The proposed model demonstrates consistent improvement over existing methods.\n* The proposed model supports untranscribed text-to-speech.\n\nWeaknesses:\n* The formulation of the acoustic VAE is misleading, since the \"weights sharing\" between the acoustic prior and the linguistic posterior is critical, but is not part of Eq. 2 (the ELBO)\n* The system overview lacks clarity, and adding all the loss terms (instead of a single KL term)  could help.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NIFbXTtAwKz", "reviewer": "Reviewer_Pops", "summary": "In order to address the challenge in TTS that it is hard to infer both the linguistic and acoustic attributes from the text directly, the paper proposes HierSpeech, which is a high-quality end-to-end TTS system based on a hierarchical conditional variational autoencoder (VAE) utilizing self-supervised speech representations. The pipeline firstly extracts self-supervised linguistic representation from text, and then converts it to acoustic representations, and finally generates waveform.", "questions": "I think the author can discuss the difference of propose method with method in [1], which also makes use of self-supervised linguistic representation. Whether the only difference in the pipeline is that the method in [1] leverages a spectrum synthesizer and hifi-gan but the proposed method incorporates hifi-gin into HierSpeech pipeline (since the HierSpeech also utilizes GAN loss for waveform synthesis)?\n\n\n[1] VQTTS: High-Fidelity Text-to-Speech Synthesis with Self-Supervised VQ Acoustic Feature\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces self-supervised linguistic representation from wav2vec to help the TTS.\n2. The experiments verify the effectiveness of proposed pipeline. The paper also discusses the potential of HierSpeech on few-shot TTS and voice conversion.\n3. The paper is well-written and easy to understand.\n\nWeaknesses:\nThere is no obvious weakness.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept