INTERSPEECH 2015

Preserving Word-level Emphasis in Speech-to-speech Translation
using Linear Regression HSMMs
Quoc Truong Do, Shinnosuke Takamichi, Sakriani Sakti,
Graham Neubig, Tomoki Toda, Satoshi Nakamura
Graduate School of Information Science
Nara Institute of Science and Technology, Japan{do.truong.dj3,shinnosuke-t,ssakti,neubig,tomoki,s-nakamura}@is.naist.jp

Abstract

a mapping function. However, this simple method was applied
to only very small vocabularies, specifically the 10 digits. As aresult, it cannot generalize to unseen words, and has difficultyin modeling emphasis in large vocabulary systems. In [7], amethod has been proposed to translate F0 patterns across languages, but other acoustic parameters such as duration, power,or spectrum that are related to emphasis have not being investigated.
In our work, we take one step further to construct an S2Stranslation system that conveys emphasis with a much larger
vocabulary size than tackled before. The idea is based on
the conventional S2S framework, with the incorporation of anadditional component to estimate an emphasis level for eachword in an utterance by applying linear-regression hidden semiMarkov models (LR-HSMMs), which are a simple form ofmulti-regression hidden semi-Markov models (MR-HSMMs)[8]. We choose LR-HSMMs for our S2S model for two reasons.
First, it allows us to build a single model for both emphasis estimation and synthesis of emphasized speech. Second, it is appropriate for tasks with words that do not exist in the training data,
because it allows us to model speech at the phoneme level. By
utilizing LR-HSMMs, we can estimate a real-numbered valueof emphasis at the word-level. Then, the sequence of word-levelemphasis levels is translated to the target language by an emphasis translation model using conditional random fields (CRFs)[9], which allows us flexibly integrate different features to theemphasis translation model. Finally, the text-to-speech system
synthesizes emphasized speech using text and the corresponding emphasis sequence.

In speech, emphasis is an important type of paralinguistic information that helps convey the focus of an utterance, new information, and emotion. If emphasis can be incorporated into aspeech-to-speech (S2S) translation system, it will be possible toconvey this information across the language barrier. However,previous related work focuses only on the translation of particular prosodic features, such as F0 , or works with emphasis but
focuses on extremely small vocabularies, such as the 10 digits. In this paper, we describe a new S2S method that is able totranslate the emphasis across languages and consider multiple
features of emphasis such as power, F0 , and duration over larger
vocabularies. We do so by introducing two new components:
word-level emphasis estimation using linear regression hidden
semi-Markov models, and emphasis translation that translates
the word-level emphasis to the target language with conditionalrandom fields. The text-to-speech synthesis system is also modified to be able to synthesize emphasized speech. The resultshows that our system can translate the emphasis correctly with
91.6% F -measure for objective test, and 87.8% for subjective
test.
Index Terms: speech translation, paralinguistic translation,emphasis estimation, emphasis translation

1. Introduction
Modern speech-to-speech (S2S) translation systems [1] havegreatly improved in accuracy, and computer-aided. Communication across the language barrier is moving closer to reality.
However, most S2S systems can not translate paralinguistic information such as emphasis or emotion, and as a result, communication though S2S systems is more dry and less emotionallyengaging than standard speech communication. If it were possible to translate paralinguistic information along with the content, communication though S2S translation could be a much
more fulfilling experience. Among the various types of paralinguistic information, in our work we focus on emphasis, whichplays an important role in conveying the key words of utterances
to make communication smoother.
In speech, emphasis is manifested by changing the duration, power, or F0 [2]. The challenge in developing an S2S
system that can accurately translate emphasis is that we must
consider these acoustic features in three components: emphasis
extraction, emphasis translation, and synthesis of emphasized
speech. Previous work [3] proposed a binary emphasis detection method to find emphasized parts in a speech. However, thiswork uses only F0 patterns to detect emphasis with a binary
value, it also was strictly mono-lingual. [4] proposed a method
to model the word-level emphasis in HMM-based TTS using
factorized decision trees, but there is no emphasis estimation ortranslation. In [5, 6], duration and power are extracted directlyfrom speech input and then translated to the target language by

Copyright © 2015 ISCA

2. Word-level Emphasis ModelingIn this section, we describe the use of linear regression hidden
semi-Markov models (LR-HSMMs) in modeling word-levelemphasis.
2.1. LR-HSMM definition
We adopt LR-HSMMs, which are a simple form of
MR-HSMMs [8] to model the emphasis speech as follows. We assume a word sequence consists of J words
w = [w1 , · · · , wj , · · · , wJ ], and a length T vector sequence of acoustic features of the input utterance o =
 >

> >
o1 , · · · , o>
. As the observation feature vector ott · · · , oT
at frame t we use a combination of the spectral feature vector(1)(2)
ot and the F0 feature vector ot as described in [10]. The
likelihood function of the LR-HSMMs is given by
P (o|λ, M) =X

P (q|λ, M) P (o|q, λ, M) ,

(1)

all q

3665

September 6- 10, 2015, Dresden, Germany

ASR

where q = {q1 , · · · , qt , · · · , qT } is the HSMM state sequence,
λ = {λ1 , · · · , λj , · · · , λJ } is the word-level emphasis weight
sequence, and M is an HSMM parameter set. Note that in thispaper, the emphasis weight is shared over all HSMM states corresponding to a word as shown in Fig. 1. The state output prob-It

isEmphasis est.
It

is

...

...

Figure 1: Word-level emphasis.
ability density function modeled by a Gaussian distribution1 isgiven byTY

P (ot |qt , ωt , M) ,(2)

P (ot |qt = i, ωt , M)
2


Y
(s)
(s)
(s)
(s)
=
N ot ; µi + ωt bi , Σi,(3)where ωt is frame-level emphasis equivalent to λj , where j is
the word corresponding to frame t, and s is a stream index (i.e.,s = 1 for the spectral feature and s = 2 for the F0 features). At
HSMM state i for the sth stream, the mean vector is given by(s)a linear combination of the vector µi for normal speech and(s)the vector bi expressing the difference between normal speechand emphasized speech using ωt as a weighting value, and the(s)covariance matrix is Σi . Moreover, the duration probability
is given byNY

P (di |ωi , M) ,

hot
0.9

暑い

暑い

です

0.8

0.1

です

TTS

[14]. Because of limitations in training data, we adopt decisiontree-based state tying [15, 16].
Using this decision tree, we can partition the set of Gaussian components into 2 groups, one is normal, and the otheris emphasized Gaussians. Finally, the mean vectors of normal(d)(s)
Gaussians are set to µi and µi , and the difference mean vec(s)tors between normal and emphasized Gaussians are set to bi(d)and bi so that the mean vectors of the LR-HSMMs are equal tothose of emphasized Gaussians if the emphasis weight ωi is set
to 1. The covariance matrices and variances of the LR-HSMMs
are set to those of normal Gaussians.

s=1

=

is
0.2

Figure 2: Emphasis speech-to-speech translationt=1

P (q|λ, M)

It
0.1

Emphasis trans.

λ hot

λ is

P (o|q, λ, M) =

MT

hot

...

λ It

hot

3. Word-level Emphasized Speech
Translation
Our proposed emphasized speech translation model consists ofa conventional S2S system, an emphasis estimation model, andan emphasis translation model as illustrated in Fig. 2. In this paper, we focus on the emphasis modeling and translation. Therefore, we assume that the ASR and MT systems provide correcttranscriptions and translation outputs.

(4)

i=1

P (di |ωi , M)

=



(d)
(d)
(d) 2
N di ; µi + ωi bi , σi
, (5)3.1. Emphasis weight sequence estimation


>
> >
,
Given an observation sequence o = o>
1 , · · · , ot , · · · , oT
and its transcription, the process to estimate the emphasis
weight sequence is as follows: First, an LR-HSMM is constructed by selecting the Gaussian distributions corresponding
to context of the given transcription. Then, emphasis is estimated by determining maximum likelihood estimates of theemphasis weight sequence, which is the same as the adaptation
process in the cluster adaptive training (CAT) algorithm [17].
The word-level emphasis weight sequence is estimated by maximizing the HSMM likelihood as follows:

where {d1 , · · · , di , · · · , dN } is a set of HSMM state durations
corresponding to q, ωi = λj if di ∈ wj , and N is the numberof states in the sentence HSMM sequence (i.e., the sum of diover N HSMM states is equivalent to T ). At HSMM state i,
the mean of the Gaussian distribution is also given by a linear(d)
combination of the mean value µi for normal speech and the(d)value bi expressing the difference between the normal speechand emphasized speech using ωi as a weighting value, and the(d) 2variance is given by σi .

λ̂ = arg max P (o|λ, M) .

2.2. Training of the LR-HSMM

λ(6)

The training process mainly follows the standard HMM-based
speech synthesis training process [12, 13, 14]. First, the training data is labeled with full contextual factors encoding various
features of the sentence. To model emphasis, we use an additional contextual factor encoding the word-level emphasis byadding an emphasis question to the standard question set to cluster context-dependent phoneme HSMM states in each cluster

This maximization process is performed with the EM algorithm[18]. In the E-step, posterior probabilities are calculated as follows:

1 Specifically, a multi-space probability distribution [11] is used for
the F0 component in this paper.

Then, in the M-step, the maximum likelihood estimate of the word-level emphasis weight sequence

3666

γi,t

(s)

=

P (qt = i|o, λ, M),

(7)

(d)
γi,t

=

P (di = t|o, λ, M).

(8)

n
o
λ̂ = λ̂1 , · · · , λ̂j , · · · , λ̂J is determined as
λ̂jgj−1 kj ,=where θk is the weight parameter. The feature function fk
(e)0
(e)0
combines the word-level emphasis bi-gram λn−1 , λn , and(k)the input feature xn . The CRFs model parameters are optimized by maximizing the likelihood function in Equation (12)using Limited-memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) [19] as implemented in CRFSuite [20].

(9)where gj and kj are calculated by

gj

=

X

" 2 T
XX

i∈q(j)

s=1 t=1+

TX(s) (s) >γi,t bi(d) (d) 2 (d) −2γi,t bi σi(s) −1 (s)bi

3.3. Speech synthesis

Σi

As described in Section 2, the TTS system relies on LRHSMMs to synthesize the emphasized speech. Unlike the wordbased model used in previous works [5, 6], our system uses aphoneme based model, allowing the proposed method to workwith larger vocabularies.
The output speech parameter vector sequence o(e) is determined by maximizing the likelihood function given the state
sequence q = [q1 , · · · , qT ], the word-level emphasis sequence

#
,

(10)

t=1

kj

=

X

" 2
X

i∈q(j)

s=1

(s) >

bi

(d) (d) −2+bi σi(s) −1

Σi

T
X

(s)



(d)
dt − µi



γi,t

(s)

(s)

ot − µi



t=1
T
X

(d)
γi,t



#

(e)0

,

λ̂

(11)

t=1

(e)0

ô(e) = argmax P (W o(e) |q, λ̂

where q(j) indicates a set of HSMM states corresponding toword wj . It should be noted that in this framework we can easily
control the effect of individual acoustic features on emphasis
estimation by selecting Gaussian components used in the Mstep as shown in Eqs. 10 and 11.

(f )0

into buckets, giving us a discrete sequence λ̂(13)where W is the weighting matrix for calculating the dynamic
features [21]. We also adopt the Global Variance method [22]to alleviate the over-smoothness of the generated parameters.
The STRAIGHT analysis-synthesis system [23] was employedfor parameter extraction and waveform generation. The feature
vector consists of spectral and excitation parameters and theirdelta and delta-delta features. Five-state left-to-right HSMMs
were used.

Our next step is emphasis translation, or to take word-level em(f )phasis estimates in the source languages λ̂ , and convert them(e)to emphasis estimates in the target language λ̂ . We perform
estimation of emphasis in the target language using conditionalrandom fields (CRFs) [9], a standard method for discriminative
sequential prediction. To train CRFs to predict target side emphasis, we create training data consisting of source and targetwords w(f ) and w(e) , and the corresponding estimated empha(e)sis values. As λ̂ is a sequence of continuous values, and(f )CRFs requires discrete state sequences, we first quantize λ̂
(e)

, M),

o(e)

3.2. Emphasis translation with conditional random fields

and λ̂

, and the HSMM model set M

4. Experiments
4.1. Experimental Setup
Experiments were performed using a bilingual EnglishJapanese emphasis corpus [24]. The corpus consists of 1015
parallel utterances of English and Japanese. In each language, atleast one of the content words in the sentence is emphasized. In
our experiments, we use the data from 2 speakers, a native English speaker, and a Japanese native speaker as the training andtesting data. After filtering out long sentences over 10 words,
we obtained 966 utterances, which we divided into 916 sentences with 1,186 emphasized words for training and 50 sentences with 62 emphasized words for testing. The word-levelemphasis is quantized to the closest of {0, 0.3, 0.6, 0.9} unlessstated otherwise, which we describe in detail in Section 4.3.
To measure the accuracy, we calculate emphasis F measure, the harmonic mean of the precision and recall withwhich the system detects emphasis.

and

0

(e)

λ̂ . We then create CRFs training data that consists of N
(e)0
(e)0
(e)0
samples D = [(x1 , λ1 ), · · · , (xn , λn ), · · · , (xN , λN )],(e)where xn is a feature vector for each word in wn consistingof:(f )• source word-level emphasis λj , and its context,(f )• source word wj , and word context,(f )• source word part of speech (PoS) pos(wj ), and PoS
context,(e)• target word wn , and word context,

4.2. Emphasis translation evaluation
In our first experiment, we evaluate the ability of the proposed
method to reproduce emphasis in the target language. In addition, we also evaluate the effect of the combination of input
features described in Section 3.2 to find out which features gives
the highest F -measure.
The translated word-level emphasis is classified into binary
values (1 and 0) using a threshold 0.5. We have evaluated different thresholds, and found out that 0.5 is the best value to classify
emphasized and normal words. We also use a baseline “All Emphasis” that predicts that every word is emphasized. The resultis shown in Table 1.
Comparing the first row with the others, we can see that theemphasis prediction model outperforms the chance rate by approximately 40%. This indicates that our proposed system canproduce emphasis in the target language relatively accurately.
Next, we can see that the model that use the features including the source language information (3rd-7th row) is better than(e)target word PoS pos(wn ), and PoS context,•
where context means the information of one succeeding and one
preceding words.
To decide which source features correspond to a target word(e)(f )wn , we use one-to-one word alignments between wj and(e)wn . The likelihood of CRFs is given by
(K
)
N
Y
X
(e)0
(e)0
(k)
exp
θk fk (λn−1 , λn , xn )0P (λ(e) |x) =

n=1
N
X Y
(e)0 n=1

λ̃

k=1
(K
X

exp

(e)0
(e)0
θk fk (λ̃n−1 , λ̃n, x(k)
n )

),

k=1

(12)

3667

4.4. Manual evaluation

Table 1: F -measure for different combinations of input features. e en and e ja denote word-level emphasis, w en and w ja
denote word information, and t en and t ja denote PoS tag ofEnglish, and Japanese, respectively.
Feature typeF -measure (%)
All Emphasis
42.5
w ja, t ja, t ja c, e ja
81.6
e en, e ja
82.8
e en, e ja, e en c
82.8
e en, e ja, w en, w ja
84.8
e en, e ja, w en, w ja, t en, t ja
90.0
e en, e ja, w en, w ja, t en, t ja, t ja c
91.6

In the final experiment, we performed a manual evaluation todetermine how well the end-to-end system can translate emphasis. We asked to 6 native Japanese speakers to listen to the emphasis translated utterances, and select the words that they think
are emphasized in 150 randomized testing utterances from thefollowing 3 systems.
Baseline: No emphasis translation is performed. The TTS istrained using a normal decision tree.
CRF-based: Emphasis is translated from English to Japaneseusing the CRF model, which is trained using the best
features in Table 1.
Natural: Natural speech spoken speech by a Japanese speaker.
Fig. 3 shows the accuracy for all 3 systems. We can see thatthe proposed emphasis translation model achieves a large improvement over the baseline system by 11.8% F -measure. The
audio generated by the baseline system have many words thatare randomly emphasized, because there is no emphasis control
based on the source utterance.
Comparing these results with the automatic evaluation ofSection 4.3, we can still see a gap of approximately 4% between the results. This is likely due to problems of speech synthesis. When listening to the natural and synthetic audio, we
found that there are often pauses inserted in natural speech inorder to emphasize words, which the synthetic audio does nothave. This problem can be addressed by introducing a pause
prediction model in the target language.

Table 2: F -measure for different quantization methods.
SystemF -measure (%)
0/1 Quant
85.5
0.1 Quant
90.8
0.3 Quant
91.6
Labels90.7

the model use only target information (2nd row). This demonstrates that our model is effectively translating emphasis fromthe source, as opposed to simply predicting based on the target.
Looking at the third and fourth rows, we can see that theemphasis context in the source language does not help the wordlevel emphasis translation, indicating that word-level emphasisin the target language depends mainly on the emphasis of thecorresponding source word. By adding the word information inboth languages, the accuracy increased by 2%, and further increased when adding the PoS tag information by approximately
6%. Finally, we add the context of the PoS tags in Japanese,yielding the best system with 91.6% accuracy. This is consistent with the characteristic of the corpus that content words areusually emphasized. We also tested with other combinationsof the features, but none of them gave the accuracy higher than
91.6%. Overall, the result indicates that along with acoustic features (emphasis level), the linguistic features such as word, PoS
tag are also contributing to the improvement of the translation
model.

95 % confidence interval

100

98.0

F-measure

95
90

87.8

85
80

77.0

75
70

4.3. Word-level emphasis quantization evaluation

Baseline

CRF-based

NaturalFigure 3: Emphasis prediction F -measure for manual evaluation

Next, we examine the effect of word-level emphasis quantization to the translation model. When creating the CRFs model,
we use 4 different quantization schemes.

5. Conclusion

0/1 Quant: The word-level emphasis is quantized into theclosest of 1 and 0

In this paper, we present a new speech translation method thatis able to translate the emphasis across languages. Compared
to previous works, our proposed method can works with larger
vocabularies, and consider many factors of emphasis such as
duration, power, and F0 . Although the experiment data is notreally big, the method can works with unseen words without
any modifications. Future work will improve the emphasized
speech synthesis by adding a pause prediction model, improvethe emphasis translation, and include an investigation of ASR
and MT errors.

0.3 Quant: The word-level emphasis is quantized into thecloset of {0, 0.3, 0.6, 0.9}.
0.1 Quant: The word-level emphasis is quantized into bucketof 0.1
Labels: The word-level emphasis in source language is quantized according to “0.3 Quant” and the emphasis in target
language is derived from the labels from the corpus andhas binary values, 1 for emphasis, 0 for normal.
From the result, we can see that the quantization scheme“0.3 Quant” gives the best result, likely because it provides anappropriate amount of training data. And more importantly,
it even outperforms the manually created “Labels,” suggestingthat training the system using quantized word-level emphasis
can be more effective than binary values.

6. Acknowledgements
Part of this work was supported by JSPS KAKENHI Grant
Number 24240032 and by the Commissioned Research of National Institute of Information and Communications Technology(NICT), Japan.

3668

7. References[21] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Kitamura, “Speech parameter generation algorithms for HMM-based
speech synthesis,” in Proceedings of ICASSP, vol. 3, 2000, pp.
1315–1318 vol.3.

[1] S. Nakamura, “Overcoming the language barrier with speechtranslation technology,” Science & Technology Trends - Quarterly
Review No.31, April 2009.

[22] T. Toda and K. Tokuda, “A speech parameter generation algorithm
considering global variance for HMM-based speech synthesis,”
IEICE, pp. 816 – 824, 2007.

[2] H. Fujisaki, “Information, prosody, and modeling - with emphasis
on tonal features of speech,” in Proceedings of Speech Prosody,
2004, pp. 1–10.

[23] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigné, “Restructuring speech representations using a pitch-adaptive timefrequency
smoothing and an instantaneous-frequency-based F0 extraction:
Possible role of a repetitive structure in sounds,” Speech Communication, vol. 27, no. 34, pp. 187 – 207, 1999.

[3] B. Arons, “Pitch-based emphasis detection for segmenting speechrecordings,” in Proceedings of ICSLP, 1994, pp. 1931–1934.
[4] K. Yu, F. Mairesse, and S. Young, “Word-level emphasis modelling in HMM-based speech synthesis,” in Processing of ICASSP,March 2010, pp. 4238–4241.

[24] D. Q. Truong, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Collection and analysis of a Japanese-English emphasized
speech corpus,” in Proceedings of Oriental COCOSDA, September 2014.

[5] T. Kano, S. Sakti, S. Takamichi, G. Neubig, T. Toda, and S. Nakamura, “A method for translation of paralinguistic information,” inProceedings of IWSLT, 2012.
[6] T. Kano, S. Takamichi, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Generalizing continuous-space translation of paralinguistic information.” in Proceedings of INTERSPEECH, 2013, pp.
2614–2618.
[7] P. D. Aguero, J. Adell, and A. Bonafonte, “Prosody generation forspeech-to-speech translation,” in Proceedings of ICASSP, vol. 1,
2006.
[8] H. Zen, K. Tokuda, T. Masuko, T. Kobayasih, and T. Kitamura,“A hidden semi-Markov model-based speech synthesis system,”
IEICE, vol. E90-D, no. 5, pp. 825 – 834, 2007.
[9] J. D. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditionalrandom fields: Probabilistic models for segmenting and labelingsequence data,” in Proceedings of ICML, 2001, pp. 282–289.
[10] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, “Simultaneous modeling of spectrum, pitch and duration in HMM-based speech synthesis,” in Proceedings of EUROSPEECH. ISCA, 1999.
[11] K. Tokuda, Y. Nankaku, T. Toda, H. Zen, J. Yamagishi, andK. Oura, “Speech synthesis based on hidden Markov models,”
Proceedings of the IEEE, vol. 101, no. 5, pp. 1234–1252, 2013.
[12] H. Zen, K. Tokuda, and A. Black, “Statistical parametric speechsynthesis,” Speech Communication, vol. 51, no. 11, pp. 1039 –
1064, 2009.
[13] K. Yu, H. Zen, F. Mairesse, and S. Young, “Context adaptive training with factorized decision trees for HMM-based statistical parametric speech synthesis,” Speech Communication, vol. 53, no. 6,
pp. 914 – 923, 2011.
[14] K. Morizane, K. Nakamura, T. Toda, H. Saruwatari, andK. Shikano, “Emphasized speech synthesis based on hidden
Markov models,” in Proceedings of Oriental COCOSDA, 2009,
pp. 76–81.
[15] S. J. Young, J. J. Odell, and P. C. Woodland, “Tree-based statetying for high accuracy acoustic modelling,” in Proceedings of
Human Language Technology Workshop, 1994, pp. 307–312.
[16] K. Shinoda and T. Watanabe, “MDL-based context-dependent
subword modeling for speech recognition,” ASJ (E), vol. 21, pp.
79–86, 2000.
[17] M. Gales, “Cluster adaptive training of hidden Markov models,”IEEE Transactions on Speech and Audio Processing, vol. 8, no. 4,
pp. 417–428, 2000.
[18] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” The royalstatistical society, vol. 39, no. 1, pp. 1–38, 1977.
[19] J. Nocedal, “Updating Quasi-Newton Matrices with Limited Storage,” Mathematics of Computation, vol. 35, no. 151, pp. 773–782,
1980.
[20] N. Okazaki, “Crfsuite:a fast implementation of conditional random fields (crfs),” 2007. [Online]. Available:
http://www.chokkan.org/software/crfsuite/3669