3-1-6
Study on Word-Level Emphasis Across English and Japanese ∗☆ Do Quoc Truong, Shinnosuke Takamichi, Sakriani Sakti, Graham Neubig,Tomoki Toda, Satoshi Nakamura (NAIST)

1

Introduction

Speech-to-speech (S2S) translation systems [1] allow us to communicate with other people in different languages. However, conventional S2S systems ignore emphasis, which is an important factor of paralinguistic information. In order to construct a S2S system considering emphasis, it is important to study on how emphasis is expressed ineach language and also across languages. In thispaper, we analyzed emphasis is word-level. We ﬁrst
estimate a real-numbered value of word-level emphasis using many speech features such as F0 , durationand power. Based on this estimation we perform ananalysis of how emphasis is expressed in individual
languages and also across languages.

2Word-level emphasis modeling andestimation with LR-HSMM

In this section, we describe about the use of linearregression hidden semi-Markov models (LR-HSMM)[2] in modeling and estimating word-level emphasis.
2.1 LR-HSMM
The observation feature vector sequence for eachsentence consisting of W words is given by o =][ ⊤⊤ ⊤, where the observation feao1 , · · · , o⊤t , · · · , oT
ture vector ot at frame t consists of the spectral(1)(2)
feature vector ot and the F0 feature vector ot inthis paper. The number of frames in the sequence
is T . The likelihood function of the LR-HSMM isgiven by∑
P (o|λ, M) =P (q|λ, M) P (o|q, λ, M) , (1)all q
where q = {q1 , · · · , qt , · · · , qT } is the HSMM state
sequence, λ = {λ1 , · · · , λw , · · · , λW } is the wordlevel emphasis weight sequence, and M is an HSMM
parameter set. Note that in this paper, the emphasis
weight is shared over all HSMM states corresponding to a word. The state output probability density
function modeled by a Gaussian distribution is givenbyT∏
P (o|q, λ, M) =P (ot |qt , λt , M) ,(2)t=1where s is a stream index (i.e., s = 1 for the spectral(s)features and s = 2 for the F0 features), µi is the(s)mean vector for normal speech and bi is the diﬀerence vector between normal speech and emphasized
speech using λw as a weighting value, and the co(s)variance matrix is Σi . The duration probability
function is also derived in the same fashion with thestate output probability function. The emphasis sequence λ is estimated by maximizing the likelihood
function 1.
The LR-HSMM model is trained by following thestandard HMM-based speech synthesis training process [3]. To model the emphasis and normal speech,we use an additional contextual factor encoding theword-level emphasis, i.e., “current word is emphasized?”, as also used in [4].

3Experiment set-up and evaluation

The experiments were performed using a bilingualEnglish-Japanese emphasis corpus [5], in which theemphasized words were carefully selected to maintain the naturalness of the emphasized utterances.
The corpus consists of 966 pairs of utterances thatwere spoken by 3 bilingual speakers; 6 mono-lingual
Japanese and 1 mono-lingual English speakers. TheLR-HSMMs were trained using 916 utterances foreach speaker. All 966 emphasis sequences were usedfor the analysis. We adopt STRAIGHT [6] for thespeech analysis.
3.1

Word-level emphasis estimation evalua-tion
First, we validate that the proposed method isable to detect emphasis and ﬁnd which acoustic features (spectral, F0, duration) are more useful toestimate the emphasis or distinguish between theemphasized and normal words. We do so by optimizing the emphasis weight sequence using diﬀerent
settings of acoustic features:• dur: using only the duration feature.
• lf0: using only log F 0 (lf0) feature.
• sp: using only spectral features.
• sp dur: using spectral and duration features.
• sp lf0: using spectral and lf0 features.

P (ot |qt = i, λt = λw , M) =2
()
∏(s)
(s)
(s)
(s)
N ot ; µi + λw bi , Σi,

• lf0 dur: using lf0 and duration features.
(3)• sp lf0 dur: combine all features.

s=1∗

日本語と英語間における単語レベルの強調に関する調査

日本音響学会講演論文集- 241 -

2015年9月

The word-level emphasis is then classiﬁed into labels of 0 and 1 indicating normal and emphasizedwords by using emphasis threshold 0.5. Then, wecalculate the F -measure to show how accurately thesystem can detect the emphasis. The result is shownin Figure 1. First, by looking at the duration column, the duration feature works to some extent inEnglish, but does not work well in Japanese. We
can also observed this situation when we combine
lf0 and duration. This is a reasonable ﬁnding, as
the Japanese discourages people from using long duration to emphasise words. Looking at the performance of individual features, we can see that for
English all three features duration, F0 , and spectral play the same role in terms of emphasis prediction. However, for Japanese, the spectral features
are more signiﬁcant compared to the other two.
By combining all features together. We achieved
the best performance for both languages. The
F -measure for English is 75.63% and Japanese is80.36%. Therefore, we will use this combination infurther analysis.
90JapaneseEnglishnot so high. It might possible to develop a emphasis
translation by using a linear function. Of course, amore sophisticated method could be used to make abetter generalization.

Fig. 2 Relationship between English words andJapanese words emphasis.

480Conclusion

F-Measure7060504030
20100durlf0

sp

lf0_dursp_lf0sp_dursp_lf0_dur

Fig. 1 F -measure of emphasis prediction

3.2 Emphasis across English and JapaneseIn this experiment, we analyse the correlation ofemphasis weights in English and Japanese. The idea
is if we can ﬁnd a strong correlation of word-levelemphasis between English and Japanese, we can develop a simple linear function to translate it. ThePearson correlation coeﬃcient is calculated between
the English and Japanese emphasis weight to measure the strength of the linear association between
them∑ (en)(ja)− λ̄(en) )(λi − λ̄(ja) )i (λi√r= √.
22∑ (en)∑ (ja)
(en)
(ja)
− λ̄)
− λ̄ )i (λii (λi(4)(en)
where r is the Pearson correlation coeﬃcient, λi
is the emphasis level for the word i-th in English(ja)and λiis the emphasis level for the corresponding Japanese word which is determined by one-toone word alignment, λ̄(en) and λ̄(ja) is the mean ofemphasis level of English and Japanese, respectively.
As the result, the correlation of the word-level emphasis between English and Japanese is shown in
Figure 2, the Pearson correlation coeﬃcient is 0.625,
indicating that there is some correlation coeﬃcientof emphasis level between two languages, but it is

日本音響学会講演論文集

This paper has conducted a bilingual analysis ofemphasis. We found that the combination of all
speech features outperforms other combination interm of emphasis detection. The analysis of emphasis across languages found a relatively high correlation of emphasis vectors between English andJapanese. The future works will construct an emphasis speech translation that translates the wordlevel emphasis vectors across languages.
Acknowledgements: Part of this work was
supported by JSPS KAKENHI Grant Number
24240032 and by the Commissioned Research of National Institute of Information and Communications
Technology (NICT), Japan.

References[1] S. Nakamura, “Overcoming the language barrier withspeech translation technology,” Science & Technology
Trends - Quarterly Review No.31, April 2009.
[2] H. Zen, K. Tokuda, T. Masuko, T. Kobayasih, and T. Kitamura, “A hidden semi-Markov model-based speech synthesis system,” IEICE - Trans. Inf. Syst., vol. E90-D, no.
5, pp. 825–834, 2007.
[3] H. Zen, K. Tokuda, and A.W. Black, “Statistical parametric speech synthesis,” Speech Communication, vol. 51,
no. 11, pp. 1039 – 1064, 2009.
[4] K. Morizane, K. Nakamura, T. Toda, H. Saruwatari, andK. Shikano, “Emphasized speech synthesis based on hidden Markov models,” in Proceedings of Oriental COCOSDA, 2009, pp. 76–81.
[5] D. Q. Truong, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Collection and analysis of a Japanese-English
emphasized speech corpus,” in Proceedings of Oriental
COCOSDA, September 2014.
[6] H. Kawahara, I. Masuda-Katsuse, and A.de Cheveigné,“Restructuring speech representations using a pitchadaptive timefrequency smoothing and an instantaneousfrequency-based F0 extraction: Possible role of a repetitive structure in sounds,” Speech Communication, vol.
27, no. 34, pp. 187 – 207, 1999.

- 242 -2015年9月