More diﬀerentiated pause insertion forphoneme-based multi-speaker TTS models ∗☆ Dong Yang1 , Tomoki Koriyama2 , Yuki Saito1 , Takaaki Saeki1 ,Detai Xin1 , Hiroshi Saruwatari1 (1 The University of Tokyo, 2 CyberAgent, Inc.)

1Introduction

Pause insertion is an essential part of text-tospeech (TTS) systems because proper pauses withnatural duration signiﬁcantly enhance the rhythm ofsynthetic speech. The position prediction of pauses
is called phrase break prediction or phrasing, which
is mainly solved by using neural networks with wordrepresentations nowadays [1, 2, 3]. However, conventional phrasing models ignore various speakers’
diﬀerent styles of inserting silent pauses, which degrades the performance of the model trained on amulti-speaker speech corpus. Besides, most mainstream TTS models (e.g., FastSpeech 2 [4]) mainly
use phonemes as input. They treat all silent pauses
as one phoneme, which leads to the duration of all
silent pauses in synthetic speech following the same
distribution and not being suﬃciently diﬀerentiated.
Therefore, we propose a pause insertion framework
for multi-speaker phoneme-based TTS models, inwhich silent pauses are categorized by duration. In
our previous work, bidirectional encoder representations from transformers (BERT) [5] pre-trainedon a large-scale text corpus performed very well in
predicting English phrase breaks [6]. Our approachin this study improves on the previous model byadding multi-task learning and injecting speaker embeddings to capture various speaker features.

2

Proposed Method

We ﬁrst categorize the silent pauses by duration
through the Gaussian mixture model-based method
in [7]. We specify the pauses as three categories:
brief (< 300 ms), medium (300–700 ms), and long(> 700 ms). Besides, there are two main types ofsilent pause: respiratory pauses (RPs) [8, 1] andpunctuation-indicated pauses (PIPs). The former
is inserted at word transitions without punctuation
to utter long sentences ﬂuently, and the latter isinserted at punctuation marks. Since their position, frequency, and duration distribution are diﬀerent, we design the categorized pause insertion (CPI)
model with a multi-task learning framework shownin Fig. 1. We use the encoder-decoder structure andtake BERT as the encoder. Two sets of BiLSTMlayers decode the output of the last layer (hiddensequence) in BERT and speaker embeddings (initialized randomly) that correspond to the predictions of RPs and PIPs. Probability and Category in Fig. 1 represent position prediction andcategory prediction, respectively. In practice, the∗RP

PIP

(WCE)

(BCE)

Probability

Category

Probability

Category

Sigmoid
LinearSoftmax
LinearSigmoid
LinearSoftmax
Linear(BCE)(WCE)

BiLSTM
BiLSTM

BiLSTM
BiLSTM
Hidden sequence
BERT
TextFig. 1

Speaker embedding

Architecture of proposed method.

model ﬁrst predicts Probability that represents theoccurrence of pauses and then outputs Category
with the highest probability among the three categories. These predicted pauses are input into TTS
models as phonemes, the three categories of which
can be denoted as “sp1”, “sp2”, and “sp3” whereas
we usually denote silent pauses as ”sp” in TTS tasks.

3Experiments

3.1

DatasetWe constructed the dataset from LibriTTS [9],a multi-speaker English corpus derived from audiobooks. LibriTTS includes plenty of long-form sentences containing multiple silent pauses uttered bymore than 2,000 speakers, and thus ﬁts our purpose of evaluating the performance of multi-speaker
pause prediction. We used the Montreal Forced
Aligner (MFA) [10] to align text and speech andobtain pause durations. Because MFA recognizes
the silence at word transitions over 30 ms as silent
pauses, we regarded silent pauses over 30 ms atpunctuation marks as PIPs and those over 50 ms
at word transitions without punctuation as RPs.
3.2

Baseline

Klimkov et al. provided the mainstream framework of a phrasing model based on English audiobooks [1], which was implemented as our baseline. In the baseline model, the input word2vec
embeddings were processed by BiLSTM projection
with peephole connection (BiLSTMP) [11] layers
and splicing windows. The splicing window stacked
together seven frames before and after and fed themto the next layer. The model output the probabilityof the occurrence of a pause after each token.
3.3Experimental Conﬁguration

For the baseline model, the hidden size and projection size for each BiLSTMP layer were 512 and音素ベース複数話者 TTS モデルにおけるより差別化されたポーズ挿入, 楊 棟 (東大院・情報理工) , 郡山 知樹(サイバーエージェント), 齋藤 佑樹, 佐伯 高明, 辛 徳泰, 猿渡 洋 (東大院・情報理工)Table 1

Results of position prediction of RPs.
Precision Recall
FβBaseline
0.393
0.187 F0.5 = 0.322
CPI
0.575
0.261 F0.5 = 0.463

Table 2

Table 3
Method A
CPI(RPs)
CPI(RPs)
CPI
CPI(RPs)
CPI(RPs)

Confusion matrix of category prediction.
Prediction of RPs
1232,565 885
0300 513
014200Subjective performance of CPI.
Score
Method B
0.560 vs. 0.440
FastSpeech2
0.537 vs. 0.463
Baseline
0.557 vs. 0.443
Baseline
0.488 vs. 0.512
CPI(Position)
0.460 vs. 0.540
CPI

Prediction of PIPs
Label12
3
1
6,155
1,766
2,058 and one with categorized pauses. We selected 162
2,258 3,186
2,509 speakers (eight males, eight females) with good syn3335
352 2,735 thetic sound quality and 123 long sentences withmore than 1 RPs from the test set. Each sentence corresponds to several speakers, and then 277
128, respectively. The dimension of word2vec emtext-speaker pairs were used in our listening tests.
beddings was 300. For the proposed model, we con1
We asked native listeners from Amazon Mechanicalﬁgured the BERT as BERTBASE and set the hidden
Turk to participate in the tests. Every test was comsize of each BiLSTM layer to 512. The dimensions ofpleted by 30 listeners, each of whom listened to ten
the hidden sequence and speaker embeddings werepairs of synthetic speech and was asked to choose
768. Word2vec embeddings and BERTBASE were
the one with better rhythm.
pre-trained on BookCorpus [11]. The loss function
The results are shown in Table. 3, in which Basewas binary cross-entropy (BCE) loss for Probabilline and CPI(PRs) only predicted the position of
ity and weighted cross-entropy (WCE) loss for CatRPs, and CPI(Position) predicted the position ofegory. The weights of categories 1–3 in WCE loss
RPs and PIPs. Test scores with p-values below 0.05
function equaled the total number of non-pause toare bold-faced. From the results, listeners perceived
kens divided by their total number in the training
the diﬀerence between CPI(RPs) and Baseline toset. As an exception, we set the weight of category
be insigniﬁcant, which shows people are insensitive
3 of RPs to 1.0 due to its sparseness.
to the position of RPs. People only became aware
During training, each mini-batch had 32 senwhen listening to a long sentence without a pause,
tences. We used the Adam optimizer and set thethat was why CPI(RPs) performed a little better
initial value of the learning rate to 5 × 10−5 . The
than Baseline. Listeners were more likely to recoglearning rate dropped by 0.2 times when the model’s
nize that CPI performed better than Baseline andperformance on the validation set did not improve
CPI(RPs). Coupled with the inability of the liswithin 5000 iterations. We took 200,000 as the maxteners to distinguish signiﬁcantly between CPI(RPs)
imum iterations (about 16 epochs). The models thatand CPI (Position), we can conclude that inputting
performed best on the validation set during these itcategorized pause phonemes to phoneme-based TTSerations were saved to make comparisons.
models makes the rhythm of synthetic speech better.
3.4 Objective Evaluations
For phrasing models, their performance is usually
measured by position prediction of RPs because position prediction of PIPs is a relatively simple task.
The results are shown in Table. 1, in which our
proposed CPI model performed much better thanthe baseline. Word representations from pre-trained
BERT and speaker embedding bring a huge boost
to the model. Table 2 shows the results of category
prediction of CPI model. In category prediction,
the accuracy of category 2 was lower than that of
the other two categories, which suggests that there
is still some room for improvement in the choice ofthresholds for categorization.
3.5

Subjective Evaluations

To explore the performance of our proposed models in multi-speaker TTS, especially to show the improvement of inputting categorized pause phonemes,
we performed AB preference tests using FastSpeech
2 as our TTS model with HiFi-GAN [12] as thevocoder. We trained two TTS models with silent
pause phonemes, one with non-categorized pauses,
1 https://huggingface.co/bert-base-uncased

4Conclusion

In this paper, we proposed a categorized pause insertion model. The results of objective evaluations
showed that word representations from pre-trained
BERT and speaker embeddings can bring a large improvement to phrasing models trained with a multispeaker dataset. The results of subjective evaluations showed that by inserting categorized pauses,
the synthetic speech had better rhythm.

References[1] V. Klimkov et al., in INTERSPEECH, 2017.
[2] K. Futamata et al., in INTERSPEECH, 2021.
[3] A. Abbas et al., in INTERSPEECH, 2022.
[4] Y. Ren et al., in ICLR, 2021.
[5] J. Devlin et al., in NAACL-HLT, 2019.
[6] D. Yang et al., in ASJ, 2022.
[7] E. Campione, J. Véronis, in Speech Prosody, 2002.
[8] G. Bailly, C. Gouvernayre, in INTERSPEECH, 2012.
[9] H. Zen et al., in INTERSPEECH, 2019.
[10] M. McAuliﬀe et al., in INTERSPEECH, 2017.
[11] Y. Zhu et al., in ICCV, 2015.
[12] J. Kong et al., in NeurIPS, 2020.