Study on Word-Level Emphasis Across English and Japanese ∗◎ Do Quoc Truong, Shinnosuke Takamichi, Sakriani Sakti, Graham Neubig,
Tomoki Toda, Satoshi Nakamura (NAIST)

1

Introduction

Speech-to-speech (S2S) translation systems [1] allow us to communicate with other people in different languages. However, conventional S2S systems ignore the emphasis, which is an importantfactor of paralinguistic information. In order to construct such a S2S system, it is important to study
on how emphasis is expressed in each language andalso across languages. In this paper, the emphasis
is analyzed in word-level. We ﬁrst estimate a realnumbered value of word-level emphasis using many
speech features such as F0 , duration and power.
Based on this estimation we perform an analysis of
how emphasis is expressed in individual language
and also across languages.

2Word-level emphasis modeling andestimation with LR-HSMM

In this section, we describe about the use of linearregression hidden semi-Markov models (LR-HSMM)[2] in modeling and estimating word-level emphasis.
2.1

LR-HSMMThe observation feature vector sequence for eachsentence that consists of W words is given by o =][ ⊤⊤ ⊤, where the observation feao1 , · · · , o⊤t , · · · , oT
ture vector ot at frame t consists of the spectral(1)(2)
feature vector ot and the F0 feature vector ot inthis paper. The number of frames in the sequence
is T . The likelihood function of the LR-HSMM isgiven by∑
P (o|λ, M) =P (q|λ, M) P (o|q, λ, M) , (1)all q
where q = {q1 , · · · , qt , · · · , qT } is the HSMM state
sequence, λ = {λ1 , · · · , λw , · · · , λW } is the wordlevel emphasis weight sequence, and M is an HSMM
parameter set. Note that in this paper, the emphasis
weight is shared over all HSMM states corresponding to a word. The state output probability density
function modeled by a Gaussian distribution is givenbyT∏
P (o|q, λ, M) =P (ot |qt , λt , M) ,(2)t=1

P (ot |qt = i, λt = λw , M) =(s)(s)+ λw bi , Σi∗)2∏(where s is a stream index (i.e., s = 1 for the spectral(s)features and s = 2 for the F0 feature), µi is the(s)mean vector for normal speech and bi is the diﬀerence vector between normal speech and emphasized
speech using λw as a weighting value, and the co(s)variance matrix is Σi . The duration probability
function is also derived in the same fashion with thestate output probability function. The emphasis sequence λ is estimated by maximizing the likelihood
function 1.
The LR-HSMMs model is trained by following thestandard HMM-based speech synthesis training process [3]. To model the emphasis and normal speech,
we use an additional contextual factor encoding theword-level emphasis, i.e., “current word is emphasized?”, as also used in [4].

3Experiment set-up and evaluation

The experiments were performed using a bilingualEnglish-Japanese emphasis corpus [5], in which theemphasized words were carefully selected to maintain the naturalness of the emphasized utterances.
The corpus consists of 966 pairs of utterances thatwere spoken by 3 bilingual speakers; 6 mono-lingualJapanese and 1 mono-lingual English speakers. TheLR-HSMMs were trained using 916 utterances foreach speaker. All 966 emphasis sequences were usedfor the analysis. We adopt STRAIGHT [6] for thespeech analysis.
3.1

Word-level Emphasis Estimation Evaluation

First, we validate that the proposed method isable to detect emphasis and ﬁnd which acoustic features (spectral, F0, duration) are more useful toestimate the emphasis or distinguish between theemphasized and normal words. We do so by optimizing the emphasis weight sequence using diﬀerent
settings of acoustic features:• dur: using only the duration feature.
• lf0: using only log F 0 (lf0) feature.
• sp: using only spectral features.
• sp dur: using spectral and duration features.
• sp lf0: using spectral and lf0 features.

(s)

(s)

N ot ; µis=1,

日英対訳コーパスにおける強調音声の分析(3)• lf0 dur: using lf0 and duration features.
• sp lf0 dur: combine all features.

The word-level emphasis is then classiﬁed into labels of 0 and 1 indicating normal and emphasizedwords by using emphasis threshold 0.5. Then, wecalculate the F -measure to show how accurate thesystem can detect the emphasis. The result is shownin Figure 1. First, by looking at the duration column, the duration feature works not so bad in English, but does not work well in Japanese. We canalso observed this situation when combine lf0 andduration. This problem caused by the characteristic Japanese where people can not use long duration
to emphasise words. Looking at the performanceof individual feature, we can see that for English
all three feature duration, F0 , and spectral play thesame role in term of emphasis prediction. However,for Japanese, the spectral feature is more signiﬁcantcompared to the other two.
By combining all features together. We archived
the best performance for both languages. TheF -measure for English is 75.63% and Japanese is80.36%. Therefore, we will use this combination for
the emphasis translation experiments.
90JapaneseEnglish80

F-Measure

70
60
50
40
30
20
100durlf0

sp

lf0_dursp_lf0sp_dursp_lf0_dur

Fig. 1 F -measure of emphasis prediction

3.2 Emphasis across English and JapaneseIn this experiment, we analyse the correlation ofemphasis weights in English and Japanese. The
Pearson correlation coeﬃcient is calculated between
the English and Japanese emphasis weight to measure the strength of the linear association betweenthem∑ (en)(ja)− λ̄(en) )(λi − λ̄(ja) )i (λi√√r=.
22∑ (en)∑ (ja)(en) )(ja) )(λ(λ−λ̄−λ̄iiii(4)(en)
where r is the Pearson correlation coeﬃcient, λi
is the emphasis level for the word i-th in English(ja)and λiis the emphasis level for the corresponding Japanese word which is determined by one-toone word alignment, λ̄(en) and λ̄(ja) is the mean ofemphasis level of English and Japanese, respectively.
As the result, the correlation of the word-level emphasis between English and Japanese is shown in
Figure 2, the Pearson correlation coeﬃcient is 0.625,indicating that there is some correlation coeﬃcientof emphasis level between two languages, but it is
not so high. It might possible to develop a emphasis
translation by using a linear function. Of course, amore sophisticated method could be used to make abetter generalization.

Fig. 2

Relationship between English words andJapanese words emphasis.

4Conclusion

This paper has conducted a bilingual analysis ofemphasis. We found that the combination of all
speech features outperforms other combination interm of emphasis detection. The analysis of emphasis across languages found a relatively high correlation of emphasis vectors between English andJapanese. The future works will construct an emphasis speech translation that translates the wordlevel emphasis vectors across languages.

5Acknowledgements

Part of this work was supported by JSPS KAKENHI Grant Number 24240032.

参考文献[1] S. Nakamura, “Overcoming the language barrier withspeech translation technology,” Science & TechnologyTrends - Quarterly Review No.31, April 2009.
[2] H. Zen, K. Tokuda, T. Masuko, T. Kobayasih, and T. Kitamura, “A hidden semi-Markov model-based speech synthesis system,” IEICE - Trans. Inf. Syst., vol. E90-D, no.
5, pp. 825–834, 2007.
[3] H. Zen, K. Tokuda, and A.W. Black, “Statistical parametric speech synthesis,” Speech Communication, vol. 51,no. 11, pp. 1039 – 1064, 2009.
[4] K. Morizane, K. Nakamura, T. Toda, H. Saruwatari, andK. Shikano, “Emphasized speech synthesis based on hidden Markov models,” in Proceedings of Oriental COCOSDA, 2009, pp. 76–81.
[5] D. Q. Truong, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Collection and analysis of a Japanese-English
emphasized speech corpus,” in Proceedings of Oriental
COCOSDA, September 2014.
[6] H. Kawahara, I. Masuda-Katsuse, and A.de Cheveigné,“Restructuring speech representations using a pitchadaptive timefrequency smoothing and an instantaneousfrequency-based F0 extraction: Possible role of a repetitive structure in sounds,” Speech Communication, vol.
27, no. 34, pp. 187 – 207, 1999.