Interspeech 2022
18-22 September 2022, Incheon, Korea

End-to-End Text-to-Speech Based on Latent Representation of
Speaking Styles Using Spontaneous Dialogue
Kentaro Mitsui1 , Tianyu Zhao1 , Kei Sawada1 , Yukiya Hono2 , Yoshihiko Nankaku2 , Keiichi Tokuda2
1

rinna Co., Ltd., Japan, 2 Nagoya Institute of Technology, Japan{kemits,tianyuz,keisawada}@rinna.co.jp, {hono,nankaku,tokuda}@sp.nitech.ac.jp

Abstract

alogue history. Guo et al. used the bidirectional encoder representations from Transformers (BERT) [11] to compute encodings of current text and chat history and fed them to the encoderof the acoustic model to improve the naturalness of the synthetic
speech [12]. Cong et al. considered the acoustic informationof the previous utterance as well as the linguistic informationby predicting the Global Style Token [13] of the current utterance from the mel-spectrogram of the previous utterance [14].
These two studies used predefined transcript to record spoken
dialogue, which may differ from actual dialogue without transcript, in terms of the frequency of the spontaneous behaviors
and the presence or absence of backchannels.
In this study, we record a free-form dialogue on a given
topic without preparing a transcript to achieve more human-like
dialogue speech synthesis. Of the three aforementioned features of spontaneous dialogue, (1) we use VITS [4], an end-toend TTS which robustly estimates alignment between text andspeech by monotonic alignment search (MAS) and blank tokens. (2) We incorporate an utterance-level latent variable into
VITS to facilitate the modeling of one-to-many relationship between text and speech. Following the framework of VAE [15],we propose two methods: VAE-VITS that assumes a standard
normal distribution for the prior distribution of the latent variable, and GMVAE-VITS, which assumes a Gaussian Mixture
Model (GMM) for the prior. Furthermore, by sharing the latent
space among speakers, training is encouraged to make similar
speaking styles between speakers close in the latent space. (3)We introduce a style predictor that predicts the speaking styleof current speech based on dialogue history to realize an entrainment that is close to actual dialogue. Speech sequencesin dialogue history are difficult to handle directly because theirlength is extremely long. Therefore, we adopt a two-stage training framework: first, VAE/GMVAE-VITS is trained using a single utterance and then style predictor is trained using a sequenceof style representations extracted from past utterances.

The recent text-to-speech (TTS) has achieved quality comparable to that of humans; however, its application in spoken dialogue has not been widely studied. This study aims to realize a TTS that closely resembles human dialogue. First, werecord and transcribe actual spontaneous dialogues. Then, theproposed dialogue TTS is trained in two stages: first stage,variational autoencoder (VAE)-VITS or Gaussian mixture variational autoencoder (GMVAE)-VITS is trained, which introduces an utterance-level latent variable into variational inference with adversarial learning for end-to-end text-to-speech(VITS), a recently proposed end-to-end TTS model. A style
encoder that extracts a latent speaking style representation fromspeech is trained jointly with TTS. In the second stage, a style
predictor is trained to predict the speaking style to be synthesized from dialogue history. During inference, by passing the
speaking style representation predicted by the style predictor to
VAE/GMVAE-VITS, speech can be synthesized in a style appropriate to the context of the dialogue. Subjective evaluation
results demonstrate that the proposed method outperforms theoriginal VITS in terms of dialogue-level naturalness.
Index Terms: end-to-end TTS, spontaneous dialogue, speaking
style, variational autoencoder, BERT

1. Introduction
Dialogue is a conversation between two or more people. In
recent years, the development of natural language processing
has greatly improved the quality of text-based dialogue generation resulting in human-computer or computer-computer dialogue [1, 2]. On the other hand, speech is essential for humandialogue. Therefore, TTS has an important role in facilitating
communication between humans and computers.
The development of deep learning has resulted in synthesizing speech in a quality comparable to that of humans [3, 4].
However, dialogue speech often has characteristics that are different from those of the recited speech. First, while recited
speech has transcript beforehand, dialogue speech is a spontaneous speech. Therefore, dialogue speech is more difficult tomodel than recited speech because of repetition, fillers, prolongation, and breaths. Second, dialogues are frequently accompanied by backchannels, also known as aizuchi [5] in Japanese,and laughter. These factors transcribed in the same way can beuttered in various styles. Thus, it is necessary to appropriately
model the one-to-many relationship between text and speech.
Finally, several factors of speech such as pitch [6], energy [7],and speech rate [8] can be in sync with the dialogue partner,
which is called entrainment [7]. Considering these features,
TTS can resemble more natural human-human dialogue.
Several studies have focused on conversational TTS.
Yokoyama et al. used Utsunomiya University spoken dialogue
database [9] to control paralinguistic information [10]. They
utilized paralinguistic information tags and did not consider di-

Copyright © 2022 ISCA

2. Spontaneous dialogue corpus
To record speech that is close to actual human-human conversation, the following method is used for speech recording andpost-processing. First, two or more speakers are given a topic
and asked to talk freely and their voices are recorded on independent channels. Automatic speech recognition (ASR) automatically transcribes the recorded speech. Transcripts are thenmanually modified and given time information (start and end
time of each utterance) to produce the final transcript with time
information. Using this time information, the audio file is split
to obtain utterance-level speech. Although it is time-consuming
to transcribe and assign time information to free dialogue, theuse of ASR can greatly reduce the burden of post-processing.
In addition, the lack of predefined transcript allows the speakers to produce more spontaneous speech which contains repetition, fillers, prolongation, and also backchannels. Speech data

2328

10.21437/Interspeech.2022-259

𝑿
Style
Encoder

idence lower bound (ELBO) of the following log-likelihood (forsimplicity, we omit s in the equation below):
log p(x|c) ≥ Eq(zf |x)q(zu |zf ) [log p(x|zf , zu )]LinearSpectrogram
Posterior
Encoder

Waveform
Decoder

Frame-level𝒛𝑓 Latent
Variable𝒙− Eq(zu |zf ) [DKL (q(zf |x)||p(zf |c, zu )]− DKL (q(zu |zf )||p(zu ))FlowUtterance
Encoder

Frame-level Feature

Utterance-levelMAS𝒛𝑢 Latent Variable≈

Speaker

Projection

M ′′
X

(m′ )

log p(x|zf

(2)

′′

, zu(m ) )m′ =1 m′′ =1

M
′′
1 X
DKL (q(zf |x)||p(zf |c, zu(m ) )
′′
M
′′

−

Projection𝒔 Embedding1
M ′ M ′′′′

Mean / Var

Stochastic
Duration
Predictor

M′
X

m =1
M′

1 X
(m′ )
− ′
DKL (q(zu |zf )||p(zu ))M ′Text Encoding
Text Encoder

(3)

m =1

Noise

where M ′ , M ′′ denote the numbers of Monte Carlo samplingfor zf , zu , respectively. The first and second terms of eq. (3)can be calculated in the same way as in the original VITS. Assuming p(zu ) = N (zu ; 0, I), the third term is the KL divergence between two multivariate normal distributions, which canbe calculated analytically. We call the proposed method definedby the above model and objective function as VAE-VITS.
This study further examines the use of a Gaussian mixture model (GMM) with equal mixture weights p(zu |yu ) =N (zu ; µyu , diag(σy2 u )) for the prior distribution, following
GMVAE-Tacotron [17], where yu denotes the discrete latentclass corresponding to zu , and the number of latent classes isdefined as K. In this case, the ELBO of the log-likelihood isapproximated as follows, instead of eq. (3):𝒄Figure 1: Training procedure of VITS incorporating utterancelevel latent variable.
Phonemerecorded in the aforementioned method enables to model thecharacteristics of actual dialogues more accurately.

3. Two-stage training-based dialogue TTS
Let x1 , . . . , xN and s1 , . . . , sN be the sequence of dialogue speech and speaker ID of each utterance, respectively. The purpose of this study is to synthesize the speech
xn corresponding to the nth text tn and speaker ID sn
by considering the past dialogue speech x1 , . . . , xn−1 andspeaker ID s1 , . . . , sn−1 , that is, to model the distributionp(xn |tn , sn , x1 , . . . , xn−1 , s1 , . . . , sn−1 ). In spoken dialogue, each of x1 , · · · , xn−1 is an extremely long time series,
and it is difficult to model them directly when n is large. Therefore, we propose a two-step training framework which is described in section 3.1 and section 3.2.

′

log p(x|c) ≥3.1. Speaking style modeling using VAE/GMVAE-VITS

m =1 m =1

′′

M
X

−

1
M ′′

−

1 X
M′ ′

m′′ =1
M′

The first training stage models the utterance-level relationshipbetween text and speech, that is, pθ (xn |tn , sn ). In this study,
we model this relationship using VITS [4]. VITS is an endto-end TTS model that learns the relationship between thephoneme sequence c and speech waveform x via frame-level latent variable zf . VITS estimates monotonic alignment betweenc and zf during training using MAS algorithm [16]. Thus, it
can be trained more stably than fully attention-based models
such as Tacotron 2 [3].
The proposed method introduces an utterance-level latentvariable zu to represent the speaking style of each utterance.
We also introduce an utterance encoder that predicts the mean
µu and variance σu2 of zu using zf and speaker embedding s.
That is, the posterior distribution of zu is given as follows:q(zu |zf , s) = N (zu ; µu , diag(σu2 )).
(1)
Hereafter, the utterance encoder and the posterior encoder,which predicts zf from the linear spectrogram X of speechx, will be called a style encoder together. We condition thestochastic duration predictor, flow, and decoder on zu to predict duration, acoustic feature, and waveform considering given
speaking style, respectively. Concretely, we apply a linear projection to zu and add it to the speaker embedding s, which is fed
to each module. A conceptual diagram of the proposed method
is depicted in Fig. 1.
The proposed method can be trained by maximizing the ev-

+

m =1

K
X

′′

M
M
X
X
′′
1
(m′ )
log p(x|zf , zu(m ) )
′
′′
M M
′′
′

yu =1

′′

DKL (q(zf |x)||p(zf |c, zu(m ) ))



(m′ )

DKL (q(yu |zf(m)q(yu |zf)||p(yu ))(m))DKL (q(zu |zf)||p(zu |yu )) . (4)Some utterances such as backchannels are short and uttered indiverse styles, while others are longer and uttered in a relatively
consistent style in dialogue speech. By assuming GMM for theprior distribution, the various styles of dialogue speech are expected to be represented more accurately. We call this method
GMVAE-VITS.
3.2. Style transition modeling using style predictor
In the second stage of training, a style predictor which predicts the distribution of zn using sequences of speaking style
z1 , . . . , zn−1 and speaker ID s1 , . . . , sn is trained, with thestyle encoder trained in the first stage fixed. An outline ofthe proposed style predictor is shown in Fig. 2 (a). A simple unidirectional LSTM is employed to model the transitionof speaking styles during a dialogue. When we directly use
zu described in section 3.1 as a speaking style representation,
the variation caused by sampling from the posterior distribution hinders the training of style predictor. Therefore, we define⊤ ⊤vu = [µ⊤u , σu ] as a style vector and used it as an input/outputof the style predictor. Hereafter, we replace the subscript u with
the index n in the dialogue for simplicity.

2329

𝒗1

𝒗2

𝒗3

𝒗4

⋯

Unidirectional LSTM
0

𝑠1

𝒗1
𝑠2

𝒗2
𝑠3

𝒄sent
𝒄1sent 𝒄sent
2
3
cont
𝒄cont
𝒄1cont 𝒄2
3

𝒗3
𝑠4

𝒄sent
4

𝒄cont
4

𝒄sent
𝑛

Table 1: RMSE between the style vector predicted using thestyle predictor and one extracted from the target speech.

Pre-trained BERT

[CLS] 𝑡𝑛,1

𝑡𝑛,2

⋯

(b) Sentence Encoding⋯(a) Style Predictor

𝒄cont
𝑛

None

S

C

S+C

VAE-VITSGMVAE-VITS

0.329
0.713

0.256
0.484

0.291
0.610

0.250
0.470

Table 2: Objective evaluation results.

Pre-trained BERT

[CLS] 𝒕𝑛−𝑘+1 𝒕𝑛−𝑘+2 ⋯Method𝒕𝑛(c) Context Encoding

Figure 2: Conceptual diagram of (a) style predictor, (b) sentence encoding, and (c) context encoding.
Style predictor takes two sequences as inputs, the stylevector sequence v1 , . . . , vn−1 and speaker ID sequences1 , . . . , sn , and predicts the style vector vn of the current utterance. The model is trained to minimize the mean squared
error ∥ṽn − vn ∥2 between the predicted and target style vector ṽn and vn , respectively. In addition, following Guo etal. [12], two types of linguistic features extracted using pretrained BERT [11] are input supplementally: (1) sentence encoding csent
n , an output vector corresponding to the [CLS] token, where the text tn = (tn,1 , tn,2 , . . . ) is prefixed with [CLS]
token and input to the BERT, and (2) context encoding ccont, annoutput vector corresponding to the [CLS] token, where a seriesof k text sequences tn−k+1 , . . . , tn are concatenated, prefixedwith a [CLS] token, and input to the BERT. The procedures ofcomputing these encodings are shown in Fig. 2 (b) and (c).

Method

MCD

MSD

DUR

VITS
VAE-oracle
GMVAE-oracle
VAE-predicted
GMVAE-predicted

7.70
7.06
7.04
7.54
7.51

9.50
8.18
8.17
9.22
9.18

0.50
0.44
0.41
0.50
0.47

e−2 , respectively. All the models were trained for 200k steps
using AdamW optimizer [18] with β1 = 0.8, β2 = 0.99 andweight decay λ = 0.01. The batch size was set to 48 and thelearning rate was scheduled in the same manner as in the previous study [4]. KL annealing [19] was introduced for training
VAE/GMVAE-VITS: KL weights of terms newly introduced by
the proposed method were increased from 0 to 1 by cosine annealing over the initial 50k steps.
Three unidirectional LSTM layers with 256 cells anddropout [20] rate 0.5 were used as the style predictor. The target style vector was obtained using the style encoder of trained
VAE/GMVAE-VITS. We trained BERT [11] from scratch on
approximately 400 GB of Japanese text and used it to compute 1024-dimensional sentence encoding csent and context encoding ccont . The text sequence length k for obtaining ccont
was set to 10. While the dialogues in the training set consisted of 160–693 utterances, we (1) randomly selected a sequence length l from the range [10, 30], and (2) randomly extracted consecutive l utterances from each dialogue. Thereby,
we avoided overfitting caused by memorizing the entire series.
The model was trained up to 2,000 steps with batch size 32 using the same AdamW optimizer used in training VITS and thecheckpoint with the smallest validation loss was used.

4. Experiments
4.1. Experimental conditions
4.1.1. Datasets
Japanese dialogues between two females, who can see eachother’s faces through glass and can hear each other’s voice witha headphone, were recorded in isolated soundproof chambersby the method described in section 2 and used for the experiment. The speakers were colleagues working in radio, which
made their conversation more friendly. The recorded data contained dialogues of 55 topics, which were transcribed and divided into 18,385 utterances. Azure speech to text was used for
ASR. 45 dialogues (15,739 utterances), 5 dialogues (1,284 utterances), and 5 dialogues (1,362 utterances) were used as training, development, and evaluation set, respectively. All the experiments were conducted using 24 kHz/16 bit speech signals.
A 186-dimensional linguistic feature extracted using Japanese
text frontend, Open JTalk 1 , was used as an input of VITS (c inFig. 1), instead of phoneme sequences.

4.2. Results
4.2.1. Objective evaluation of style predictor
To evaluate the effectiveness of providing additional linguistic information to the style predictor, we trained the following
four models for each of VAE/GMVAE-VITS: (1) None: neither csent nor ccont was used, (2) S: only csent was used, (3)C: only csent was used, and (4) S+C: both csent and ccont wereused. The root mean squared error (RMSE) between predicted
and target style vectors is presented in Table 1. The RMSE of S
was significantly smaller than None for both VAE and GMVAE,
which indicates the effectiveness of csent in style prediction. In
addition, by comparing None and C, or S and S+C, we cansee that ccont also contributed to improved prediction accuracy.
These results suggest that it is effective to use not only acoustic
but also linguistic history in predicting transition of speaking
styles during a dialogue. In the following experiments, we used
the style predictor trained in the S+C condition.

4.1.2. Model and training details
We trained three models: the original VITS [4], which does
not explicitly consider speaking styles, and the proposed VAEVITS and GMVAE-VITS described in section 3.1. The hyperparameters of VITS were set to be the same as in the previous
study. Following GMVAE-Tacotron [17], the utterance encoder
was composed of two 1D-convolutional layers with 512 filters
and a kernel size of three, two bidirectional LSTM layers with
256 cells at each direction, and a mean pooling layer followed
by a linear projection layer. The number of Monte Carlo sampling was set to 1 and the dimension of zu was set to 16. For
GMVAE-VITS, the number of latent classes K was set to 10,
and the initial value and lower bound of σyu were set to e−1 and4.2.2. Objective evaluation of overall system
We conducted an objective evaluation to compare the performance of the proposed methods with baseline VITS. For
the proposed methods, we evaluated two cases: one is touse zu obtained from the target speech using the style encoder (VAE/GMVAE-oracle) and the other is to use zu sam-

1 http://open-jtalk.sourceforge.net/

2330

(a) Prior distribution (b) Posterior distribution(c) Duration(d) Pitch(e) Loudness

Figure 3: Analysis of latent space learned by GMVAE-VITS in terms of (a) prior distribution, (b) posterior distribution, (c) average
duration, (d) average pitch, and (e) average loudness.
Table 3: Results of MOS evaluation on utterance-level anddialogue-level naturalness with 95% confidence intervals.
MethodUtterance

Dialogue

VITS
GMVAE-oracle
GMVAE-predicted

3.38±0.14
3.51±0.12
3.56±0.12

3.34±0.12
3.59±0.12
3.53±0.11

inal one. The evaluation was conducted on a 5-point scalefrom 1 (bad) to 5 (excellent). Thirty raters participated in theevaluation, and each rater evaluated thirty speech samples.
The results are presented in Table 3. With regard toutterance-level naturalness, although the scores of GMVAEoracle/predicted were slightly higher than VITS, there was no
significant difference between them. Though VITS does not
utilize explicit style representation, the synthetic speech wasevaluated as natural because various speaking styles exist thatsound natural when heard as a single utterance. Regarding
dialogue-level naturalness, the score of GMVAE-oracle was significantly higher than VITS (p = 0.003 in Student’s t-test), confirming that using appropriate speech styles contributed to thenaturalness of dialogue. Furthermore, GMVAE-predicted alsoachieved a significantly higher score than VITS (p = 0.021),
indicating that style predictor was able to predict the appropriate speaking style when heard as a dialogue.

pled from the distribution defined by the predicted style vector ṽu (VAE/GMVAE-predicted). The performance was evaluated in terms of following metrics: (1) mel-cepstral distortion(MCD) [21], the RMSE of a 60-dimensional mel-cepstrum extracted from synthetic and target speech, (2) mel-spectral distortion (MSD), the RMSE of 80-dimensional mel-spectrogram
extracted from synthetic and target speech, and (3) total duration error (DUR), the error of speech length of synthetic andtarget speech. Since the series length differs between synthetic
and target speech, dynamic time warping [22] was used to alignthem before calculating MCD and MSD.
The results are presented in Table 2. Both VAE/GMVAEoracle showed significant improvement in MCD and MSD compared to baseline VITS. DUR was also slightly improved, suggesting that zu represents duration-related features as well as
acoustic features. VAE/GMVAE-predicted also showed improvement in MCD and MSD relative to baseline VITS. This
indicates that the style predictor was able to predict speaking
styles that are close to those of target speech. The performanceof GMVAE-VITS was slightly better than VAE-VITS for both
oracle and predicted. This is probably because the richer priorof GMVAE-VITS could represent the various speaking styles of
dialogue speech more appropriately. In the next section, we further compare the proposed GMVAE-VITS with baseline VITS.

4.2.4. Analysis of latent space
Fig. 3 (a) and (b) illustrates the prior and posterior distribution of trained GMVAE-VITS, respectively, where dimensionality reduction was applied using principal component analysis.
We observed that the latent representations of the two speakers were mixed, indicating that the learned latent space was
speaker-independent. This is because the speaker embedding s
was explicitly used, allowing the latent variable zu to represent
only speaker-independent speaking styles. We also synthesized
all texts in the evaluation set with different speaker IDs and latent classes and described the average duration, pitch of voiced
segments, and loudness of synthetic speech in Fig. 3 (c), (d),and (e). We confirmed that each latent class had different characteristics and they were common across speakers. With thesecharacteristics, the learned prior distribution can be applied tomodify speaking style to the desired one.

4.2.3. Subjective evaluation of overall system
We conducted two mean opinion score (MOS) tests to evaluate
the subjective quality of the synthetic speech2 . In the utterancelevel evaluation, raters were presented only one utterance andasked to evaluate its naturalness. In the dialogue-level evaluation, raters were presented a short dialogue consisting of 6 utterances (approximately 10–20 sec) and asked to evaluate its naturalness as a spoken dialogue (whether natural entrainment occurred, whether the speaking style was suitable for the context,etc.). We used ground truth timing of each utterance to construct dialogue samples because our spontaneous dialogue corpus contained numerous overlaps and simply playing the synthesized speech alternatively resulted in unnatural dialogue. For
dialogue-level evaluation, we also computed text-speech alignment over recorded speech using MAS and used the alignment
information for synthesis to align speech length with the orig-

5. Conclusions
In this study, we aimed to synthesize spoken dialogue that isclose to human spontaneous dialogue and proposed (1) recording and transcription of free-form dialogues without transcripts,(2) VAE/GMVAE-VITS to model various speaking styles, and(3) a style predictor that predicts speaking styles using linguistic and acoustic features from past dialogues. The combination of GMVAE-VITS and the style predictor achieved higher
naturalness than conventional VITS in a dialogue-level evaluation. The latent space acquired by GMVAE-VITS was speakerindependent and had different characteristics for each latentclass. This study assumed that transcriptions of past utterances
and timing of each utterance were available; however, actual
applications will require estimating these as well. Future work
will include introducing a mechanism to automatically estimate
them and unifying the proposed two-stage training framework
into a single end-to-end training framework.

2 Speech samples are available at :https://rinnakk.
github.io/research/publications/DialogueTTS.

2331

6. References[19] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz,
and S. Bengio, “Generating sentences from a continuous space,”in Proc. CoNLL, Berlin, Germany, Aug. 2016, pp. 10–21.

[1] Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao,
J. Gao, J. Liu, and B. Dolan, “DialoGPT: Large-scale generative pre-training for conversational response generation,” in Proc.
ACL, online, Jul. 2020, pp. 270–278.

[20] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov, “Dropout: a simple way to prevent neuralnetworks from overfitting,” The journal of machine learning research, vol. 15, no. 1, pp. 1929–1958, Jun. 2014.

[2] R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du
et al., “LaMDA: Language models for dialog applications,” arXiv
preprint arXiv:2201.08239, Jan. 2022.

[21] R. Kubichek, “Mel-cepstral distance measure for objective speech
quality assessment,” in Proc. IEEE PACRIM, Victoria, Canada,
May 1993, pp. 125–128.

[3] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang,
Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan et al., “Natural TTS
synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP, Calgary, Canada, May 2018, pp. 4779–
4783.

[22] D. J. Berndt and J. Clifford, “Using dynamic time warping to findpatterns in time series,” in Proc. KDD Workshop, vol. 10, Seattle,
U.S.A., 1994, pp. 359–370.

[4] J. Kim, J. Kong, and J. Son, “Conditional variational autoencoderwith adversarial learning for end-to-end text-to-speech,” in Proc.
ICML, online, Jul. 2021, pp. 5530–5540.
[5] S. Kita and S. Ide, “Nodding, aizuchi, and final particles injapanese conversation: How conversation reflects the ideology ofcommunication and social relationships,” Journal of Pragmatics,
vol. 39, no. 7, pp. 1242–1254, 2007.
[6] A. Gravano, Š. Beňuš, R. Levitan, and J. Hirschberg, “Three
ToBI-based measures of prosodic entrainment and their correlations with speaker engagement,” in Proc. SLT, California, U.S.A.,
Dec. 2014, pp. 578–583.
[7] R. Levitan and J. B. Hirschberg, “Measuring acoustic-prosodic
entrainment with respect to multiple levels and dimensions,” inProc. INTERSPEECH, Florence, Italy, Aug. 2011, pp. 3081–
3084.
[8] R. Street, “Speech convergence and speech evaluation in factfinding interviews,” Human Communication Research, vol. 11,
no. 2, pp. 139–169, 1984.
[9] H. Mori, T. Satake, M. Nakamura, and H. Kasuya, “Constructing a
spoken dialogue corpus for studying paralinguistic information in
expressive conversation and analyzing its statistical/acoustic characteristics,” Speech Communication, vol. 53, no. 1, pp. 36–50,
Jan. 2011.
[10] M. Yokoyama, T. Nagata, and H. Mori, “Effects of dimensional
input on paralinguistic information perceived from synthesized
dialogue speech with neural network,” in Proc. INTERSPEECH,
Hyderabad, India, Sep. 2018, pp. 3053–3056.
[11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pretraining of deep bidirectional Transformers for language understanding,” in Proc. NAACL-HLT, Minneapolis, U.S.A., Jun. 2019,
pp. 4171–4186.
[12] H. Guo, S. Zhang, F. K. Soong, L. He, and L. Xie, “Conversational
end-to-end TTS for voice agents,” in Proc. SLT, online, Jan. 2021,
pp. 403–409.
[13] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor,
Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style Tokens: Unsupervised style modeling, control and transfer in end-to-end speechsynthesis,” in Proc. ICML, Stockholm, Sweden, Jul. 2018, pp.
5180–5189.
[14] J. Cong, S. Yang, N. Hu, G. Li, L. Xie, and D. Su, “Controllable
context-aware conversational speech synthesis,” in Proc. INTERSPEECH, online, Sep. 2021, pp. 4658–4662.
[15] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,”in Proc. ICLR, Banff, Canada, Apr. 2014.
[16] J. Kim, S. Kim, J. Kong, and S. Yoon, “Glow-TTS: A generative
flow for text-to-speech via monotonic alignment search,” in Proc.
NeurIPS, vol. 33, online, Dec. 2020, pp. 8067–8077.
[17] W.-N. Hsu, Y. Zhang, R. J. Weiss, H. Zen, Y. Wu, Y. Wang,
Y. Cao, Y. Jia, Z. Chen, J. Shen et al., “Hierarchical generativemodeling for controllable speech synthesis,” in Proc. ICLR, New
Orleans, U.S.A., May 2019.
[18] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, New Orleans, U.S.A., May 2019.

2332